asyncio: MicroPython vs. CPython create_task() garbage collection behavior
(Not sure if I should label this a docs bug, feature request, or discussion. Apologies if I missed the mark.)
CPython exhibits a bit of a counterintuitive behavior, which requires users to store the result of asyncio.create_task(), otherwise face the danger of the task being garbage collected.
Will McGugan of Textual blogged about this here:
https://textual.textualize.io/blog/2023/02/11/the-heisenbug-lurking-in-your-async-code/
Vincent Bernat reported this as a documentation bug against CPython, https://github.com/python/cpython/issues/88831 last year, and subsequently the CPython documentation was adjusted to say:
Important
Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn’t referenced elsewhere may get garbage collected at any time, even before it’s done. For reliable “fire-and-forget” background tasks, gather them in a collection:
background_tasks = set()
for i in range(10):
task = asyncio.create_task(some_coro(param=i))
# Add task to the set. This creates a strong reference.
background_tasks.add(task)
# To prevent keeping references to finished tasks forever,
# make each task remove its own reference from the set after
# completion:
task.add_done_callback(background_tasks.discard)
Is MicroPython exhibiting the same behavior? If so, should the documentation be adjusted?
- The MicroPython documentation provides an example that does not store a reference, basically what CPython warns users not to do. Is this a misleading example or a non-issue in MicroPython? Should the example be CPython-friendly anyway though? Should the documentation explain this difference?
- Even if one wanted to be better-safe-than-sorry and write portable code, the CPython-recommended code above does not work under MicroPython, resulting in a
TypeError: unsupported type for __hash__: 'Task'. Should__hash__be added for Tasks? Or should the documentation mention some other way to do this?
Not a MicroPython bug per se, but also worth mentioning: @peterhinch's (excellent!) async tutorial also has examples where Tasks are not stored. At one point it's mentioned that "[t]he .create_task method returns the Task instance which may be saved for status checking or cancellation" (emphasis mine). May, or should?
uasyncio gather() return_exceptions behaviour differs from CPython
I think this is a regression. The following script produces an unexpected result in MicroPython:
try:
import uasyncio as asyncio
except ImportError:
import asyncio
async def will_stop(n):
print('Runs to completion')
for _ in range(6):
await asyncio.sleep(1)
return 2 * n
async def forever(n):
print('forever() will be cancelled')
while True:
await asyncio.sleep(1)
n += 1
return n
async def do_cancel(task):
await asyncio.sleep(3)
print('About to cancel forever')
task.cancel()
async def main():
tasks = [asyncio.create_task(forever(70))]
tasks.append(will_stop(21))
asyncio.create_task(do_cancel(tasks[0]))
res = None
try:
res = await asyncio.gather(*tasks, return_exceptions=True)
except asyncio.CancelledError:
print('Cancelled')
print('Result: ', res)
asyncio.run(main())
CPython (expected behaviour):
adminpete@debian8:/mnt/qnap2/temp$ python3
Python 3.8.0 (default, Dec 6 2019, 16:20:13)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rats40
forever() will be cancelled
Runs to completion
About to cancel forever
Result: [CancelledError(), 42]
>>>
MicroPython (behaves as if return_exceptions==False):
MicroPython v1.17 on 2021-09-02; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import rats40
forever() will be cancelled
Runs to completion
About to cancel forever
Cancelled
Result: None
>>>