requests::Response::content hangs forever for Socket::read
Port, board and/or hardware
ESP32/ESP8266/Unix
MicroPython version
MicroPython v1.23.0 on 2024-06-02; ESP module with ESP8266
Reproduction
I experienced the issue with an ESP running micropython wanting to access an AhoyDTU via simple HTTP GET. After updating the AhoyDTU (latest (ahoy_v0.8.140) https://fw.ahoydtu.de/fw/release%2Fahoy_v0.8.140/) the call got stuck and the ESP hangs there forever.
As other HTTP clients do not have any issues requesting the server, I see the issue independent from AhoyDTU and request micropython to be more robust here.
Expected behaviour
No response
Observed behaviour
Depending of the server's message requests::Response::content hangs as the internal Socket::read() blocks and hangs forever. As the documentation for Socket::read() says it waits until EOF is recognized I assume this is the issue here.
Additional Information
In case a Content-Length is being transmitted Response::content should only read the amount of bytes as denoted by Content-Length. I case the message is being streamed, read() shall be used to read until EOF is recognized.
I was able to fix this locally by doing the following:
Try to read Content-Length from HTTP headers and conditionally set it at the response object. In case Response::content is requested and the socket is used to read the payload, check if Content-Length is available and conditionally only read the denoted amount of bytes via Socket::read(int) which does not wait for EOF. In case no Content-Length is available, do the Socket::read() as before.
fix-requests-hangs-on-read.zip
Code of Conduct
Yes, I agree
esp32: Webserver Socket stops responding after a minute.
Port, board and/or hardware
esp32 port, ESP32 and ESP32C3
ESP32 (lolin32-Lite, esp32 devkit v1) and ESP32C3 (esp32c3 supermini)
both board have same issues
MicroPython version
MicroPython v1.23.0 on 2024-06-02; ESP32C3 module with ESP32C3
in older versions than MicroPython v1.20.0 code works fine withou issues
MicroPython v1.19.1 works OK
MicroPython v1.20.0 works OK
MicroPython v1.21.0 NOT working
MicroPython v1.22.0 NOT working
MicroPython v1.23.0 NOT working
Reproduction
Copy content of zip file to your ESP device. Set correct SSID and Password and reset your board.
issue.zip
The web server will offer you index.html, which cyclically send requests to server and server answers back.
in terminal you can see positions of virtual joysticks on webpage.
After about a minute, the server stops responding. There is enough free RAM, I don't know where the problem is, everything works correctly on older versions.
There is another version of the web server available in the Zip archive, without the use of threads, but the problem is exactly the same.
Expected behaviour
I expect that the server will behave the same in all versions. It should respond indefinitely to web browser requests.
Observed behaviour
In Chrome, using developer tools, I check the responses from the server. After 1-5 minutes, the server stops responding and sometimes restarts.
Additional Information
Web server must run in background, without blocking of REPL and WebRepl
Code of webserver with threads:
import network
import usocket
import _thread
from os import stat, listdir
from time import sleep_ms
import gc
class WebServer:
def __init__(self, web_folder='/www', port=80):
self.WEB_FOLDER = web_folder
self.MIMETYPES = {
"txt" : "text/plain",
"htm" : "text/html",
"html" : "text/html",
"css" : "text/css",
"csv" : "text/csv",
"js" : "application/javascript",
"xml" : "application/xml",
"xhtml" : "application/xhtml+xml",
"json" : "application/json",
"zip" : "application/zip",
"pdf" : "application/pdf",
"ts" : "application/typescript",
"ttf" : "font/ttf",
"jpg" : "image/jpeg",
"jpeg" : "image/jpeg",
"png" : "image/png",
"gif" : "image/gif",
"svg" : "image/svg+xml",
"ico" : "image/x-icon",
"cur" : "application/octet-stream",
"tar" : "application/tar",
"tar.gz": "application/tar+gzip",
"gz" : "application/gzip",
"mp3" : "audio/mpeg",
"wav" : "audio/wav",
"ogg" : "audio/ogg"
}
self.webserv_sock = None
self.url_handlers = {}
self.port = port
def _file_exists(self, path):
try:
stat(path)
return True
except:
return False
def get_mime_type(self, filename):
try:
_, ext = filename.rsplit(".", 1)
return self.MIMETYPES.get(ext, "application/octet-stream")
except:
return "application/octet-stream"
def read_in_chunks(self, file_object, chunk_size=1024):
while True:
data = file_object.read(chunk_size)
if not data:
break
yield data
def serve_file(self, client, path):
try:
if path.startswith("/*GET_FILE"):
file_path = path.replace("/*GET_FILE", "")
else:
if path == "/":
path = "/index.html"
if '?' in path:
path = path.split('?')[0]
file_path = self.WEB_FOLDER + path
mime_type = self.get_mime_type(file_path)
filestatus = 0 # 0=Not found 1=Found 2=found in GZip
if self._file_exists(file_path + '.gz'):
filestatus = 2
file_path += '.gz'
elif self._file_exists(file_path):
filestatus = 1
if filestatus > 0:
with open(file_path, 'rb') as file:
client.write(b'HTTP/1.1 200 OK\r\n')
client.write(b"Content-Type: " + mime_type.encode() + b"\r\n")
if filestatus == 2:
client.write(b'Content-Encoding: gzip\r\n')
client.write(b'\r\n')
for piece in self.read_in_chunks(file):
client.write(piece)
else:
client.write(b"HTTP/1.0 404 Not Found\r\n\r\nFile not found.")
except OSError as e:
print("OSError:", e)
client.write(b"HTTP/1.0 500 Internal Server Error\r\n\r\nInternal error.")
except Exception as e:
print("Exception:", e)
client.write(b"HTTP/1.0 500 Internal Server Error\r\n\r\nInternal error.")
def handle(self, pattern):
"""Decorator to register a handler for a specific URL pattern."""
def decorator(func):
self.url_handlers[pattern] = func
return func
return decorator
def client_handler(self, client):
try:
gc.collect()
sleep_ms(0)
request = client.recv(2048)
if request:
_, path, _ = request.decode("utf-8").split(" ", 2)
for pattern, handler in self.url_handlers.items():
if path.startswith(pattern):
try:
gc.collect()
handler(client, path, request)
except Exception as e:
print("Handler Exception:", e)
client.close()
return
# Default file serving if no handler matches
self.serve_file(client, path)
except Exception as e:
sleep_ms(0)
#print("Webserver Exception:", e)
finally:
client.close()
def web_thread(self):
while True:
try:
cl, addr = self.webserv_sock.accept()
cl.settimeout(2) # time in seconds
self.client_handler(cl)
except Exception as ex:
sleep_ms(0)
def start(self):
addr = usocket.getaddrinfo('0.0.0.0', self.port)[0][-1]
self.webserv_sock = usocket.socket()
self.webserv_sock.setsockopt(usocket.SOL_SOCKET, usocket.SO_REUSEADDR, 1)
self.webserv_sock.bind(addr)
self.webserv_sock.listen(5)
_thread.start_new_thread(self.web_thread, ())
for interface in [network.AP_IF, network.STA_IF]:
wlan = network.WLAN(interface)
if not wlan.active():
continue
ifconfig = wlan.ifconfig()
print("Web server spusten na adrese {}:{}".format(ifconfig[0], self.port))
def stop(self):
if self.webserv_sock:
self.webserv_sock.close()
Code of webserver without threads:
import network
import socket
from os import stat, listdir
from time import sleep_ms
import gc
from micropython import alloc_emergency_exception_buf
# Define constant for registering the handler function
_SO_REGISTER_HANDLER = 20
class WebServer:
def __init__(self, web_folder='/www', port=80):
self.WEB_FOLDER = web_folder
self.MIMETYPES = {
"txt" : "text/plain",
"htm" : "text/html",
"html" : "text/html",
"css" : "text/css",
"csv" : "text/csv",
"js" : "application/javascript",
"xml" : "application/xml",
"xhtml" : "application/xhtml+xml",
"json" : "application/json",
"zip" : "application/zip",
"pdf" : "application/pdf",
"ts" : "application/typescript",
"ttf" : "font/ttf",
"jpg" : "image/jpeg",
"jpeg" : "image/jpeg",
"png" : "image/png",
"gif" : "image/gif",
"svg" : "image/svg+xml",
"ico" : "image/x-icon",
"cur" : "application/octet-stream",
"tar" : "application/tar",
"tar.gz": "application/tar+gzip",
"gz" : "application/gzip",
"mp3" : "audio/mpeg",
"wav" : "audio/wav",
"ogg" : "audio/ogg"
}
self.webserv_sock = None
self.url_handlers = {}
self.port = port
self.client_sockets = []
def _file_exists(self, path):
try:
stat(path)
return True
except:
return False
def get_mime_type(self, filename):
try:
_, ext = filename.rsplit(".", 1)
return self.MIMETYPES.get(ext, "application/octet-stream")
except:
return "application/octet-stream"
def read_in_chunks(self, file_object, chunk_size=1024):
while True:
data = file_object.read(chunk_size)
if not data:
break
yield data
def serve_file(self, client, path):
try:
if path.startswith("/*GET_FILE"):
file_path = path.replace("/*GET_FILE", "")
else:
if path == "/":
path = "/index.html"
if '?' in path:
path = path.split('?')[0]
file_path = self.WEB_FOLDER + path
mime_type = self.get_mime_type(file_path)
filestatus = 0 # 0=Not found 1=Found 2=found in GZip
if self._file_exists(file_path + '.gz'):
filestatus = 2
file_path += '.gz'
elif self._file_exists(file_path):
filestatus = 1
if filestatus > 0:
with open(file_path, 'rb') as file:
client.write(b'HTTP/1.1 200 OK\r\n')
client.write(b"Content-Type: " + mime_type.encode() + b"\r\n")
if filestatus == 2:
client.write(b'Content-Encoding: gzip\r\n')
client.write(b'\r\n')
for piece in self.read_in_chunks(file):
client.write(piece)
else:
client.write(b"HTTP/1.0 404 Not Found\r\n\r\nFile not found.")
except OSError as e:
print("OSError:", e)
client.write(b"HTTP/1.0 500 Internal Server Error\r\n\r\nInternal error.")
except Exception as e:
print("Exception:", e)
client.write(b"HTTP/1.0 500 Internal Server Error\r\n\r\nInternal error.")
def handle(self, pattern):
"""Decorator to register a handler for a specific URL pattern."""
def decorator(func):
self.url_handlers[pattern] = func
return func
return decorator
def accept_websocket(self, sock):
try:
client_sock, client_addr = sock.accept()
#client_sock.setblocking(False)
client_sock.settimeout(2)
client_sock.setsockopt(socket.SOL_SOCKET, _SO_REGISTER_HANDLER, self.client_handler)
self.client_sockets.append(client_sock)
# Debug output
#print("Received new connection from:", client_addr)
#print(self.client_sockets)
except Exception as e:
print("Error accepting connection:", e)
def client_handler(self, client_sock):
try:
request = client_sock.recv(2048)
#print(request)
if request:
_, path, _ = request.decode("utf-8").split(" ", 2)
for pattern, handler in self.url_handlers.items():
if path.startswith(pattern):
try:
gc.collect()
handler(client_sock, path, request)
except Exception as e:
print("Handler Exception:", e)
self.close_client(client_sock)
return
# Default file serving if no handler matches
self.serve_file(client_sock, path)
self.close_client(client_sock)
except Exception as e:
print("Error in client_handler:", e)
self.close_client(client_sock)
def close_client(self, client_sock):
try:
client_sock.setsockopt(socket.SOL_SOCKET, _SO_REGISTER_HANDLER, None)
client_sock.close()
except Exception as e:
print("Error closing client socket:", e)
finally:
if client_sock in self.client_sockets:
self.client_sockets.remove(client_sock)
def start(self):
alloc_emergency_exception_buf(100)
addr = socket.getaddrinfo('0.0.0.0', self.port)[0][-1]
self.webserv_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.webserv_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.webserv_sock.bind(addr)
self.webserv_sock.listen(1)
self.webserv_sock.settimeout(2)
self.webserv_sock.setsockopt(socket.SOL_SOCKET, _SO_REGISTER_HANDLER, self.accept_websocket)
for interface in [network.AP_IF, network.STA_IF]:
wlan = network.WLAN(interface)
if not wlan.active():
continue
ifconfig = wlan.ifconfig()
print("Web server started at address {}:{}".format(ifconfig[0], self.port))
def stop(self):
if self.webserv_sock:
self.webserv_sock.setsockopt(socket.SOL_SOCKET, _SO_REGISTER_HANDLER, None)
self.webserv_sock.close()
for client_sock in self.client_sockets:
self.close_client(client_sock)
Code of Conduct
Yes, I agree