docs: explain optional global-interpreter-lock
Documentation URL
https://micropython.org/
Description
I saw this on the homepage:
multithreading via the "_thread" module, with an optional global-interpreter-lock (still work in progress, only available on selected ports)
My question:
So by default micropython does not have GIL? then does it use native multithreading instead of green thread? E.g on a Linux multicore machine?
And how to do mutex between different native threads?
Code of Conduct
Yes, I agree
esp32: Fix uneven GIL allocation between Python threads.
Summary
Explicitly yield each time a thread mutex is unlocked. Closes #15423.
Key to understanding this bug is that Python threads run at equal RTOS priority, and although ESP-IDF FreeRTOS (and I think vanilla FreeRTOS) scheduler will round-robin equal priority tasks in the ready state it does not make a similar guarantee for tasks moving between ready and waiting.
The pathological case of this bug is when one Python thread task is busy (i.e. never blocks) it will hog the CPU more than expected, sometimes for an unbounded amount of time. This happens even though it periodically unlocks the GIL to allow another task to run.
Assume T1 is busy and T2 is blocked waiting for the GIL. T1 is executing and hits a condition to yield execution:
- T1 calls MP_THREAD_GIL_EXIT
- FreeRTOS sees T2 is waiting for the GIL and moves it to the Ready list (but does not preempt, as T2 is same priority, so T1 keeps running).
- T1 immediately calls MP_THREAD_GIL_ENTER and re-takes the GIL.
- Pre-emptive context switch happens, T2 wakes up, sees GIL is not available, and goes on the waiting list for the GIL again.
To break this cycle step 4 must happen before step 3, but this may be a very narrow window of time so it may not happen regularly - and quantisation of the timing of the tick interrupt to trigger a context switch may mean it never happens.
Yielding at the end of step 2 maximises the chance for another task to run.
Testing
Adds a test case which is based on the code in the linked bug report. This test fails consistently on esp32 without this fix, and passes afterwards. Test case also passes on rp2, which is the only other port where thread tests are currently enabled, and passes if run manually on the unix port.
This PR also includes a commit to enable the thread tests on the esp32 port.
Trade-offs and Alternatives
Could make a port-specific MP_THREAD_GIL_EXIT macro that yields, to avoid the overhead of yielding any time a thread mutex is unlocked. However almost all of the thread mutex lock/unlock events in the esp32 port are the GIL (the other thread mutexes are the GC mutex and the mutex that protects thread creation and cleanup).
This work was funded through GitHub Sponsors.