Pyboard SysTick losing ticks: interrupts sometimes locked out for 170ms
I've spent a couple of days chasing a timing bug and I think I've nailed the cause: the SysTick handler appears to sporadically miss out a chunk of time (about 170ms). I checked this by modifying the SysTick handler to monitor itself:
uint32_t tick_timestamp;
uint32_t last_tick_timestamp;
uint32_t tick_overruns = 0;
uint32_t last_overrun;
/**
* @brief This function handles SysTick Handler.
* @param None
* @retval None
*/
void SysTick_Handler(void) {
// Instead of calling HAL_IncTick we do the increment here of the counter.
// This is purely for efficiency, since SysTick is called 1000 times per
// second at the highest interrupt priority.
extern volatile uint32_t uwTick;
last_tick_timestamp = tick_timestamp;
tick_timestamp = DWT->CYCCNT;
if((tick_timestamp - last_tick_timestamp) > 336000U) { /* 2ms late */
last_overrun = tick_timestamp - last_tick_timestamp;
tick_overruns++;
}
uwTick += 1;
Basically it uses the debug CPU clock counter to see how long since it last ran. Obviously this is normally around 168000 clocks (i.e. 1ms), sometimes a bit less, sometimes a bit more (because of ISR jitter). But every now and then it misses a huge chunk of time. Here's the output of one of my test programs:
ERROR: elapsed_ms and ms_delta mismatch, margin=169
overruns=11, overrun by=168001(28862777)
There are a bunch of overruns at boot then everything is fine for a while. Then (in the above example) overrun 11 hit. The key number above is 28862777: when divided by / 168000 it is 171.8ms. Basically the SysTick handler itself sees that it hasn't run for a big chunk of time. Which can only happen if something is locking out all interrupts for a massive amount of time or SysTick has been disabled. I'm pretty sure there's nothing disabling it.
I wondered what might be locking out interrupts for such a huge chunk of time so I wrote a minimal test program:
from pyb import delay, rng
from time import ticks_cpu
def timestamps():
start = ticks_cpu()
print("Starting with {}".format(start))
while True:
delay_time = rng() % 200
print("Delay time = {}ms".format(delay_time))
before = ticks_cpu() & 0x3fffffff
delay(delay_time)
after = ticks_cpu() & 0x3fffffff
realtime = (after - before) & 0x3fffffff
realtime_ms = realtime / 168000
if abs(realtime_ms - delay_time) > 1:
print("ERROR: delay of {}ms actually took {}ms ({:08x},{:08x} = {})".format(delay_time, realtime_ms, after, before, after - before))
return
I apply masks above to use only the bottom 30 bits to get modulo arithmetic working right for calculating delays.
I had to modify the ticks_cpu() function to return a longer range than just a small int (the clock runs so fast 16 bits enough):
STATIC mp_obj_t time_ticks_cpu(void) {
static bool enabled = false;
if (!enabled) {
CoreDebug->DEMCR |= CoreDebug_DEMCR_TRCENA_Msk;
DWT->CYCCNT = 0;
DWT->CTRL |= DWT_CTRL_CYCCNTENA_Msk;
enabled = true;
}
return mp_obj_new_int_from_uint(DWT->CYCCNT);
}
STATIC MP_DEFINE_CONST_FUN_OBJ_0(time_ticks_cpu_obj, time_ticks_cpu);
Here's an example run:
Delay time = 22ms
Delay time = 105ms
Delay time = 199ms
Delay time = 170ms
ERROR: delay of 170ms actually took 340.4643ms (1b10cd27,17a80774 = 57198003)
This affects all timing that is based on SysTick. So delay(), millis(), etc. all go wrong.
I investigated further into what is happening and I know what at least one of the culprits is: the flash file system mounted on USB. If you run this test program then copy something to the file system it trips the bug. But it also trips at random (which may be the host OS - Ubuntu in my case - touching the file system for indexing or something).
stm32: UART overrun race condition locks up system
There is a race condition present in the stm32 port that relates to the way the UART overrun (ORE bit in USART status register) is handled in the IRQ service routine.
If this condition is met, the pyboard completly locks up and cannot be recovered (you have to hard-reset or power cycle).
The reason is because the UART RX ISR function does not clear the ORE flag if the receive buffer is empty and hence the irq is starting again as soon as the handler exits (ORE is also triggering the IRQ). It results in a 100% CPU utilisation.
This situation is explained in the STM32 reference manual page chapter "Overrun error" in the USART description.
Steps to reproduce:
- Wire up a pyboard UART Rx pin (e.g. X10) to an USB->serial adapter's TX pin
- on the pyboard enter:
from pyb import UART
u=UART(1,230400)
while True: u.read(10)
- on the PC enter:
import time, serial, os
u=serial.serial_for_url('COMxx',230400)
while True: u.write(os.urandom(100)); time.sleep(0.01)
you should now see a lot of random data being transferred to pyboard. Since the window time of the critical section (https://github.com/micropython/micropython/blob/master/ports/stm32/uart.c#L486-L494) is somewhere smaller than 1usec it's very hard to hit the bug and normally you won't see any troubles.
However, the raise condition can be forced if another IRQ can be triggered on the pyboard, hence delaying the calling of the UART service routine.
One way that worked for me was to load a file, e.g. boot.py, in a text editor and hit several times "save" (normally 10-20 times should be ok) -> BOOM. The pyboard freezes and continously fires the UART RX IRQ
The way to resolve is to put the following code
if (__HAL_UART_GET_FLAG(&self->uart, UART_FLAG_ORE) != RESET) {
if (__HAL_UART_GET_FLAG(&self->uart, UART_FLAG_RXNE) == RESET) {
// overrun and empty data, just do a dummy read to clear ORE
// and prevent a raise condition where a continous interrupt stream (due to ORE set) occurs
// (see chapter "Overrun error" ) in STM32 reference manual
self->uart.Instance->DR;
}
}
just before https://github.com/micropython/micropython/blob/master/ports/stm32/uart.c#L486
On the L4, the overrun can be disabled in the UART's control register, so it doesn't need that check.