requests::Response::content hangs forever for Socket::read
Port, board and/or hardware
ESP32/ESP8266/Unix
MicroPython version
MicroPython v1.23.0 on 2024-06-02; ESP module with ESP8266
Reproduction
I experienced the issue with an ESP running micropython wanting to access an AhoyDTU via simple HTTP GET. After updating the AhoyDTU (latest (ahoy_v0.8.140) https://fw.ahoydtu.de/fw/release%2Fahoy_v0.8.140/) the call got stuck and the ESP hangs there forever.
As other HTTP clients do not have any issues requesting the server, I see the issue independent from AhoyDTU and request micropython to be more robust here.
Expected behaviour
No response
Observed behaviour
Depending of the server's message requests::Response::content hangs as the internal Socket::read() blocks and hangs forever. As the documentation for Socket::read() says it waits until EOF is recognized I assume this is the issue here.
Additional Information
In case a Content-Length is being transmitted Response::content should only read the amount of bytes as denoted by Content-Length. I case the message is being streamed, read() shall be used to read until EOF is recognized.
I was able to fix this locally by doing the following:
Try to read Content-Length from HTTP headers and conditionally set it at the response object. In case Response::content is requested and the socket is used to read the payload, check if Content-Length is available and conditionally only read the denoted amount of bytes via Socket::read(int) which does not wait for EOF. In case no Content-Length is available, do the Socket::read() as before.
fix-requests-hangs-on-read.zip
Code of Conduct
Yes, I agree
requests do not works in mutithread, breaks the whole script/device
A simple script like this, breaks the whole script or device when trying to make requests from two works ¿? I dont know exactly how to call the problem but you can replicate it easily.
Im using MicroPython v1.21.0 on 2023-10-06; Raspberry Pi Pico W with RP2040
The main loop can work fine forever (pinging the url) but as soon as i press the button it breaks with this random behaviors:
- Main loop requests hangs and remains until timeout, i can raise timeout to hours and request still hanged forever, 2nd thread request go out fine for one time but if i press button again it just hangs too and it just hangs forever until reboot device
- Main loop request hangs forever and button requests works fine every-time i press button
- Button requests pass fine for one time, then it hangs but Main loop request works fine forever
I have been trying to make it works for 1 week and it just don work, what i tried:
Using thread.allocate_lock()
Making two request functions (one for main loop one for button press request)
make a clone of request library and import it with a different name, then calling the requests library from main loop and the cloned library from button request
using other gpio button (not the bootsel button)
use the 2nd thread function without button (making it automatically with a sleep times)
All tries ended on the same problem, also i have use a longer sleep times for the main loop and press the button to call the request function some minutes after the main loop ended and it do the same
It seems (at my poor knowledge on python) that requests library do not "release" (or close) at all the requests and it leave only available for the first core that use it?
How can i make it works with this two threads functions?
import utime, _thread, ujson, requests, network
ssid_name = 'xxx'
ssid_pass = 'xxx'
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
def make_http_request(url):
print("request entry")
r = None
response_data = None
try:
print("trying block")
r = requests.get(url, timeout=10)
response_data = ujson.loads(r.text)
r.close()
return response_data
except Exception as e:
print("Error:", e)
raise
finally:
if r is not None:
r.close()
def bootsel():
import rp2
while True:
if rp2.bootsel_button() == 1:
print("pressed")
but_data = make_http_request("https://httpbin.org/get")
print(f"But: {but_data}")
_thread.start_new_thread(bootsel,())
while True:
if wlan.isconnected():
print("connected.. pinging")
ping = make_http_request("https://httpbin.org/get")
print(f"Ping: {ping}")
print("sleeping")
utime.sleep(45)
else:
time=0
while not wlan.isconnected():
print ('connecting...')
wlan.connect(ssid_name, ssid_pass)
while not wlan.isconnected():
time = time + 1
print(f'waiting connection {time}')
utime.sleep_ms(1000)