Fixed by b95ca2491aa405199a429fc24e227a32cb6564f8
QUERY · ISSUE
aiohttp: ws: frame not fully received/read if large
I noticed how large payload are cut-off when using the high-level receive-functions provided by the websocket implementation of aiohttp.
The following debug statements show the observed truncation:
diff --git a/python-ecosys/aiohttp/aiohttp/aiohttp_ws.py b/python-ecosys/aiohttp/aiohttp/aiohttp_ws.py
index 6e0818c..34c7b08 100644
--- a/python-ecosys/aiohttp/aiohttp/aiohttp_ws.py
+++ b/python-ecosys/aiohttp/aiohttp/aiohttp_ws.py
@@ -203,7 +203,9 @@ class WebSocketClient:
if has_mask: # pragma: no cover
mask = await self.reader.read(4)
+ print("READING BYTES:", length)
payload = await self.reader.read(length)
+ print("READ BYTES:", len(payload))
if has_mask: # pragma: no cover
payload = bytes(x ^ mask[i % 4] for i, x in enumerate(payload))
return opcode, payload
Output from test run supposed to receive a >1.5KiB payload:
READING BYTES: 1594
READ BYTES: 1436
CANDIDATE · ISSUE
Fix attached for aiohttp's aiohttp_ws.py partial read on ESP32
On ESP32, after receiving 5-10 full websocket frames, this read returns only a partial frame:
payload = await self.reader.read(length)
This is to be expected, as the read(length) call in MicroPython's asyncio.Stream is not guaranteed to return the full length bytes in a single call, especially for large payloads, due to non-blocking I/O or buffer constraints.
I didn't observe this issue on unix/desktop MicroPython, only on the ESP32.
The proper way is to re-do the read() for the remaining length until the entire frame has been received.
The patch below fixes that, as well as adding some error handling.
--- old/aiohttp_ws.py 2025-05-20 14:06:16.111521205 +0200
+++ aiohttp/aiohttp_ws.py 2025-05-20 14:16:28.985286423 +0200
@@ -197,13 +199,31 @@
return opcode, payload
fin, opcode, has_mask, length = self._parse_frame_header(header)
if length == 126: # Magic number, length header is 2 bytes
- (length,) = struct.unpack("!H", await self.reader.read(2))
+ length_data = await self.reader.read(2)
+ if len(length_data) != 2:
+ print("WARNING: aiohttp_ws.py failed to read 2-byte length, closing")
+ return self.CLOSE, b""
+ (length,) = struct.unpack("!H", length_data)
elif length == 127: # Magic number, length header is 8 bytes
- (length,) = struct.unpack("!Q", await self.reader.read(8))
-
+ length_data = await self.reader.read(8)
+ if len(length_data) != 8:
+ print("WARNING: aiohttp_ws.py failed to read 8-byte length, closing")
+ return self.CLOSE, b""
+ (length,) = struct.unpack("!Q", length_data)
if has_mask: # pragma: no cover
mask = await self.reader.read(4)
- payload = await self.reader.read(length)
+ if len(mask) != 4:
+ print("WARNING: aiohttp_ws.py failed to read mask, closing")
+ return self.CLOSE, b""
+ payload = b""
+ remaining_length = length
+ while remaining_length > 0:
+ chunk = await self.reader.read(remaining_length)
+ if not chunk: # Connection closed or error
+ print(f"WARNING: aiohttp_ws.py connection closed while reading payload, got {len(payload)}/{length} bytes, closing")
+ return self.CLOSE, b""
+ payload += chunk
+ remaining_length -= len(chunk)
if has_mask: # pragma: no cover
payload = bytes(x ^ mask[i % 4] for i, x in enumerate(payload))
return opcode, payload
1 comment
The issue probably lies in that read(n) does not guarantee to actually read n bytes (all at once). Normally I'd iterate over reading until result is null/None or read and count received bytes in a loop until everything is received as expected.
Neither approach appears to work in this scenario, though.
Attempted fix: https://github.com/micropython/micropython-lib/pull/1038
Should be fixed by #1034 (which is merged).