Discussion about allocating memory from within an irq
So I've been thinkng about this, and I think it would be possible to have some type of scratch allocator which could be used inside interrupt handlers.
The actual scratch buffer would be allocated outside of the interrupt and somehow associated with the handler.
When the handler runs it would put the registered handler on top of a list of allocators. When a memory allocation is required, it would either use the scratch allocator, or if none were present, use the heap allocator. The scrath allocator would basically just increment a pointer by the number of bytes being allocated, as long as that number of bytes still exists in the scratch buffer,
Each time the interrupt fires, it would reset the scratch buffer.
I just thought I'd throw this out as an idea.
REVIEW ONLY: first steps making GC reentrant
This is totally not ready to merge, but I did a fair bit of work on it and want to see what others think before continuing (or abandoning).
The idea is that you want to be able to allocate memory during an interrupt. If we allow this, then you should be able to rely on allocating memory during an interrupt, so long as there is memory available. You can arrange for memory to be available by calling gc.collect() explicitly and often enough (outside the interrupt).
Allowing gc_alloc/gc_realloc/gc_free to work in an interrupt means that these 3 functions, and the garbage collector routine itself, must be reentrant (can be suspended, executed again at a higher priority interrupt, then resumed).
This means making all global state operations atomic, and always leaving the heap in a consistent state.
It is difficult because you can't simply disable IRQs for the whole gc_alloc: it could take some time to find a free piece of the heap, and in this time you might get another IRQ (eg UART character comes in). So you can only disable IRQs for a very short time.
I've tested this patch by creating 3 timers running at 1Hz, 10Hz and 1000Hz, all allocating large chunks of memory. Then I do gc.collect() in the foreground in an infinite loop. It seems to work, but needs a lot more testing.
This really opens a whole can of worms, since other issues start to become apparent. Eg, we need to atomically store to the global dictionary (mp_store_global). And, ctrl-C handler probably needs to be completely redesigned and rewritten so that you can't ctrl-C while in the middle of a gc collection/alloc/realloc/free (in fact, you shouldn't be able to ctrl-C in any HAL function either since they modify global state of the peripherals).