requests::Response::content hangs forever for Socket::read
Port, board and/or hardware
ESP32/ESP8266/Unix
MicroPython version
MicroPython v1.23.0 on 2024-06-02; ESP module with ESP8266
Reproduction
I experienced the issue with an ESP running micropython wanting to access an AhoyDTU via simple HTTP GET. After updating the AhoyDTU (latest (ahoy_v0.8.140) https://fw.ahoydtu.de/fw/release%2Fahoy_v0.8.140/) the call got stuck and the ESP hangs there forever.
As other HTTP clients do not have any issues requesting the server, I see the issue independent from AhoyDTU and request micropython to be more robust here.
Expected behaviour
No response
Observed behaviour
Depending of the server's message requests::Response::content hangs as the internal Socket::read() blocks and hangs forever. As the documentation for Socket::read() says it waits until EOF is recognized I assume this is the issue here.
Additional Information
In case a Content-Length is being transmitted Response::content should only read the amount of bytes as denoted by Content-Length. I case the message is being streamed, read() shall be used to read until EOF is recognized.
I was able to fix this locally by doing the following:
Try to read Content-Length from HTTP headers and conditionally set it at the response object. In case Response::content is requested and the socket is used to read the payload, check if Content-Length is available and conditionally only read the denoted amount of bytes via Socket::read(int) which does not wait for EOF. In case no Content-Length is available, do the Socket::read() as before.
fix-requests-hangs-on-read.zip
Code of Conduct
Yes, I agree
ESP32: socket.recv() and socket.read() sometimes hang forever
I have implemented Response.iter_content() in urequests so that I can deal with large responses without running out of memory. I based my code on the equivalent in Python requests. My new method is:
def iter_content(self, chunk_size=1):
def generate():
while True:
chunk = self.raw.read(chunk_size)
if not chunk:
break
yield chunk
self._content_consumed = True
if self._content_consumed:
raise RuntimeError("response already consumed")
elif chunk_size is not None and not isinstance(chunk_size, int):
raise TypeError("chunk_size must be an int, it is instead a %s."
% type(chunk_size))
return generate()
I copied urequests and renamed to requests before adding iter_content() to requests, so that I could have both modules for testing.
My test program that executes this is:
import requests
for x in range(1,20):
r = requests.get('http://jsonplaceholder.typicode.com/users')
try:
for chunk in r.iter_content(chunk_size=256):
print(chunk.decode('UTF-8'))
finally:
r.close()
Running this with either recv() or read() in iter_content() I get the following results:
- Loboris Micropython for ESP32 - always hangs before 20 iterations is reached
- Micropython for ESP32 - always hangs before 20 iterations is reached
- Micropython for ESP8266 - never hangs
- Python3 (std requests on Raspbian on RPi) - never hangs
The hang always occurs when the request looks complete in terms of what is printed out. If I press cntl-C when a hang has occurred I get:
Traceback (most recent call last):
File "<stdin>", line 7, in <module>
File "<stdin>", line 4, in <module>
File "requests.py", line 36, in generate
KeyboardInterrupt:
where line 36 is
chunk = self.raw.read(chunk_size)
regards,
Chris