Background

.mpy files are precompiled .py files, built using mpy-cross, that contain compiled bytecode functions (and can also contain machine code). The benefit of using an .mpy file over a .py file is that they are faster to import and take less memory when importing. They are also smaller on disk.

But the real benefit of .mpy files comes when they are frozen into the firmware. This is done by loading the .mpy file during compilation of the firmware and turning it into a set of big C data structures, which are then compiled and downloaded into the ROM of a device. These C data structures can be executed in-place, ie directly from ROM. This makes importing even faster because there is very little to do, and also means such frozen modules take up much less RAM (because their bytecode stays in ROM).

The downside of frozen code is that it requires recompiling and reflashing the entire firmware. This can be a big barrier to entry, slows down development time, and makes it harder to do OTA updates of frozen code (because the whole firmware must be updated).

An often requested feature is dynamic frozen modules where .mpy files are frozen (written) into ROM at runtime, or at least after the firmware has already been flashed. Such a feature would allow freezing and refreezing Python code without building/compiling firmware.

Static and memory-mappable .mpy files

This PR attempts to solve the problem of dynamic frozen modules but in a different way to the existing freezing mechanism. The idea (and what's implemented in this PR) is to rework the .mpy file format so that it consists of data/bytecode which is mostly static and ready to run. Then if these .mpy files are located in flash/ROM which is memory addressable, the .mpy file can be executed in-place.

With this approach there is still a small amount of unpacking and linking of the .mpy file that needs to be done when it's imported, but it's still going to be a lot better than loading an .mpy from disk. In particular it will save a lot of RAM because the bytecode can stay in flash/ROM and be executed from there.

The main trick to make static .mpy files is for qstrs to go through a lookup table to convert from local qstr number in the module to global qstr number in the firmware. That means the bytecode does not need linking/rewriting of qstrs when it's loaded. Instead only a small qstr table needs to be built (and put in RAM) at import time.

In more detail, in the VM what used to be (schematically):

    qst = DECODE_QSTR_VALUE;

is now (schematically):

    idx = DECODE_QSTR_INDEX;
    qst = code_state->fun_bc->qstr_table[idx];

Surprisingly this qstr indirection in the bytecode has only a small impact on VM performance. On PYBv1.0 one measures:

    diff of scores (higher is better)
    N=100 M=100          p0bc ->       p1bc         diff      diff% (error%)
    bm_chaos.py        363.10 ->     361.74 :      -1.36 =  -0.375% (+/-0.00%)
    bm_fannkuch.py      78.19 ->      77.88 :      -0.31 =  -0.396% (+/-0.00%)
    bm_fft.py         2570.96 ->    2543.43 :     -27.53 =  -1.071% (+/-0.00%)
    bm_float.py       5836.26 ->    5937.80 :    +101.54 =  +1.740% (+/-0.01%)
    bm_hexiom.py        48.30 ->      47.12 :      -1.18 =  -2.443% (+/-0.01%)
    bm_nqueens.py     4460.11 ->    4395.46 :     -64.65 =  -1.450% (+/-0.00%)
    bm_pidigits.py     646.59 ->     649.52 :      +2.93 =  +0.453% (+/-0.54%)
    misc_aes.py        417.04 ->     411.82 :      -5.22 =  -1.252% (+/-0.01%)
    misc_mandel.py    3562.82 ->    3557.88 :      -4.94 =  -0.139% (+/-0.01%)
    misc_pystone.py   2421.36 ->    2355.83 :     -65.53 =  -2.706% (+/-0.01%)
    misc_raytrace.py   378.48 ->     375.83 :      -2.65 =  -0.700% (+/-0.01%)
    viper_call0.py     576.70 ->     576.79 :      +0.09 =  +0.016% (+/-0.02%)
    viper_call1a.py    550.37 ->     550.30 :      -0.07 =  -0.013% (+/-0.01%)
    viper_call1b.py    438.24 ->     436.01 :      -2.23 =  -0.509% (+/-0.00%)
    viper_call1c.py    442.84 ->     441.30 :      -1.54 =  -0.348% (+/-0.22%)
    viper_call2a.py    536.31 ->     536.26 :      -0.05 =  -0.009% (+/-0.00%)
    viper_call2b.py    382.34 ->     378.46 :      -3.88 =  -1.015% (+/-0.11%)

What's done in this PR is the reworking of .mpy files. What is yet to come is being able to store them in flash and import them directly.

Summary

This PR reworks the .mpy file format to be simpler and allow .mpy files to be executed in-place. Performance is not impacted much. Eventually it will be possible to store such .mpy files in a linear, read-only, memory-mappable filesystem so they can be executed from flash/ROM. This will essentially be able to replace frozen code for most applications, making it much simpler to take advantage of memory savings from frozen code.