asyncio: MicroPython vs. CPython create_task() garbage collection behavior
(Not sure if I should label this a docs bug, feature request, or discussion. Apologies if I missed the mark.)
CPython exhibits a bit of a counterintuitive behavior, which requires users to store the result of asyncio.create_task(), otherwise face the danger of the task being garbage collected.
Will McGugan of Textual blogged about this here:
https://textual.textualize.io/blog/2023/02/11/the-heisenbug-lurking-in-your-async-code/
Vincent Bernat reported this as a documentation bug against CPython, https://github.com/python/cpython/issues/88831 last year, and subsequently the CPython documentation was adjusted to say:
Important
Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn’t referenced elsewhere may get garbage collected at any time, even before it’s done. For reliable “fire-and-forget” background tasks, gather them in a collection:
background_tasks = set()
for i in range(10):
task = asyncio.create_task(some_coro(param=i))
# Add task to the set. This creates a strong reference.
background_tasks.add(task)
# To prevent keeping references to finished tasks forever,
# make each task remove its own reference from the set after
# completion:
task.add_done_callback(background_tasks.discard)
Is MicroPython exhibiting the same behavior? If so, should the documentation be adjusted?
- The MicroPython documentation provides an example that does not store a reference, basically what CPython warns users not to do. Is this a misleading example or a non-issue in MicroPython? Should the example be CPython-friendly anyway though? Should the documentation explain this difference?
- Even if one wanted to be better-safe-than-sorry and write portable code, the CPython-recommended code above does not work under MicroPython, resulting in a
TypeError: unsupported type for __hash__: 'Task'. Should__hash__be added for Tasks? Or should the documentation mention some other way to do this?
Not a MicroPython bug per se, but also worth mentioning: @peterhinch's (excellent!) async tutorial also has examples where Tasks are not stored. At one point it's mentioned that "[t]he .create_task method returns the Task instance which may be saved for status checking or cancellation" (emphasis mine). May, or should?
asyncio: Calling .run() within a task does not behave as per CPython
Port, board and/or hardware
Unix build
MicroPython version
MicroPython v1.23.0-preview.72.g4a2e510a8.dirty on 2024-02-04; linux [GCC 11.4.0] version
Reproduction
import asyncio
async def foo(s):
while True:
print(f"Task {s}")
await asyncio.sleep(1)
async def main():
asyncio.create_task(foo(1))
await asyncio.sleep(2)
asyncio.run(foo(2)) # CPython throws a RuntimeError
await asyncio.sleep(2)
asyncio.create_task(foo(1))
await asyncio.sleep(5)
print("Done") # never happens in MP
asyncio.run(main())
Expected behaviour
On CPython issuing run() in a running task throws a RuntimeError:
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rats17
Task 1
Task 1
Task 1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/adminpete/rats17.py", line 17, in <module>
asyncio.run(main())
File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/adminpete/rats17.py", line 11, in main
asyncio.run(foo(2)) # CPython throws a RuntimeError
File "/usr/lib/python3.10/asyncio/runners.py", line 33, in run
raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
>>>
Observed behaviour
The run() command appears to work, starting the second foo instance. However main never terminates.
MicroPython v1.23.0-preview.72.g4a2e510a8.dirty on 2024-02-04; linux [GCC 11.4.0] version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import rats17
Task 1
Task 1
Task 1
Task 2
Task 1
Task 2
Task 1
Task 2
Task 1
Task 2
... continues forever
Additional Information
This arose from this issue.
A possible fix to asyncio.core.py is
def run(coro):
if cur_task is None:
return run_until_complete(create_task(coro))
else:
raise RuntimeError("asyncio.run() cannot be called from a running event loop")
Code of Conduct
Yes, I agree