In #3462 it was suggested that adding a hook after every opcode execution would not "solve" determinism because the length of time of an "execute function" opcode is determined by the length of time of the C-function.

@dpgeorge said:

As @stinos mentions, even doing this won't be enough to get each opcode executing within strict time limits. For example sorting a list is a single opcode (the function/method call) and that can take an arbitrary time depending on the size of the list (eg on pyboard sorting a sorted list of 100 elements takes 2.6ms, and 1000 elements takes 249ms). Also inserting into a dict may require a rehash of the dict and then a garbage collection which can make the insertion time vary from between 30us to 1500us (approx).

And I agree but there's a difference between sort (and other utilities) and gc

1. You don't need sort and other functions for Python language to run. If you wanted, you could even write sort function in Python!
   However, practically speaking you need a garbage collector and cannot write it in Python.
2. It's far simpler to write your own sort in C that has necessary RTOS.yield() hooks than it is to write a garbage collector.

My request

In gc_collect_root() and its callees, wherever there's a loop, insert a hook to allow a call to RTOS.yield()

https://github.com/micropython/micropython/blob/9ebc037eee575fd951dea92c82ed9704d9101924/py/gc.c#L315

modified:

void gc_collect_root(void **ptrs, size_t len) {
    for (size_t i = 0; i < len; i++) {
        void *ptr = ptrs[i];
        VERIFY_MARK_AND_PUSH(ptr);
        gc_drain_stack();
        GC_COLLECT_ROOT_HOOK();
    }
}

modified:

STATIC void gc_drain_stack(void) {
    while (MP_STATE_MEM(gc_sp) > MP_STATE_MEM(gc_stack)) {
        GC_DRAIN_STACK_OUTER_LOOP_HOOK();
        // pop the next block off the stack
        size_t block = *--MP_STATE_MEM(gc_sp);

        // work out number of consecutive blocks in the chain starting with this one
        size_t n_blocks = 0;
        do {
            n_blocks += 1;
            GC_DRAIN_STACK_INNER_WHILE_HOOK();
        } while (ATB_GET_KIND(block + n_blocks) == AT_TAIL);

        // check this block's children
        void **ptrs = (void**)PTR_FROM_BLOCK(block);
        for (size_t i = n_blocks * BYTES_PER_BLOCK / sizeof(void*); i > 0; i--, ptrs++) {
            void *ptr = *ptrs;
            VERIFY_MARK_AND_PUSH(ptr);
            GC_DRAIN_STACK_INNER_FOR_LOOP_HOOK();
        }
    }
}

Yes it's inefficient so most people won't use it but for those that cannot wait >600us for the gc to execute, these macros are necessary.

Optimizations

• Immediately, I can see you can refactor each loop such that it executes x times (configurable) before checking whether it has to yield.

• The "configuration interface" should be simple for the dev/user. If we use the macros as I've currently defined them, the user has to have knowledge of the gc code to understand how the macros will impact the timing. Instead, the config should be something like: setting GC_COOP_MULTI to 0 will result in gc not calling GC_HOOK_CHECK() at all. Each increase, up to MAX, increases frequency of calls to GC_HOOK_CHECK() proportionally