← index #16091Issue #3754
Related · medium · value 1.441
QUERY · ISSUE

requests::Response::content hangs forever for Socket::read

openby der-brecheropened 2024-10-27updated 2024-11-05
bug

Port, board and/or hardware

ESP32/ESP8266/Unix

MicroPython version

MicroPython v1.23.0 on 2024-06-02; ESP module with ESP8266

Reproduction

I experienced the issue with an ESP running micropython wanting to access an AhoyDTU via simple HTTP GET. After updating the AhoyDTU (latest (ahoy_v0.8.140) https://fw.ahoydtu.de/fw/release%2Fahoy_v0.8.140/) the call got stuck and the ESP hangs there forever.
As other HTTP clients do not have any issues requesting the server, I see the issue independent from AhoyDTU and request micropython to be more robust here.

Expected behaviour

No response

Observed behaviour

Depending of the server's message requests::Response::content hangs as the internal Socket::read() blocks and hangs forever. As the documentation for Socket::read() says it waits until EOF is recognized I assume this is the issue here.

Additional Information

In case a Content-Length is being transmitted Response::content should only read the amount of bytes as denoted by Content-Length. I case the message is being streamed, read() shall be used to read until EOF is recognized.

I was able to fix this locally by doing the following:
Try to read Content-Length from HTTP headers and conditionally set it at the response object. In case Response::content is requested and the socket is used to read the payload, check if Content-Length is available and conditionally only read the denoted amount of bytes via Socket::read(int) which does not wait for EOF. In case no Content-Length is available, do the Socket::read() as before.
fix-requests-hangs-on-read.zip

Code of Conduct

Yes, I agree

CANDIDATE · ISSUE

ESP32: socket.recv() and socket.read() sometimes hang forever

closedby chrisb2opened 2018-05-02updated 2018-05-05

I have implemented Response.iter_content() in urequests so that I can deal with large responses without running out of memory. I based my code on the equivalent in Python requests. My new method is:

    def iter_content(self, chunk_size=1):
        def generate():
            while True:
                chunk = self.raw.read(chunk_size)
                if not chunk:
                    break
                yield chunk
            self._content_consumed = True

        if self._content_consumed:
            raise RuntimeError("response already consumed")
        elif chunk_size is not None and not isinstance(chunk_size, int):
            raise TypeError("chunk_size must be an int, it is instead a %s."
                % type(chunk_size))

        return generate()

I copied urequests and renamed to requests before adding iter_content() to requests, so that I could have both modules for testing.

My test program that executes this is:

import requests

for x in range(1,20):
    r = requests.get('http://jsonplaceholder.typicode.com/users')
    try:
        for chunk in r.iter_content(chunk_size=256):
            print(chunk.decode('UTF-8'))
    finally:
        r.close()

Running this with either recv() or read() in iter_content() I get the following results:

  • Loboris Micropython for ESP32 - always hangs before 20 iterations is reached
  • Micropython for ESP32 - always hangs before 20 iterations is reached
  • Micropython for ESP8266 - never hangs
  • Python3 (std requests on Raspbian on RPi) - never hangs

The hang always occurs when the request looks complete in terms of what is printed out. If I press cntl-C when a hang has occurred I get:

Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "<stdin>", line 4, in <module>
  File "requests.py", line 36, in generate
KeyboardInterrupt:

where line 36 is
chunk = self.raw.read(chunk_size)

regards,
Chris

Keyboard

j / / n
next pair
k / / p
previous pair
1 / / h
show query pane
2 / / l
show candidate pane
c
copy suggested comment
r
toggle reasoning
g i
go to index
?
show this help
esc
close overlays

press ? or esc to close

copied