Make ROMFS Bootable.
Description
The new ROMFS feature allows users to deploy their application with all necessary resources as a package. This is hugely valuable. However, while ROMFS partitions can contain code, they are not actively searched by default for code. I'd like to suggest that ROMFS partition(s) are searched whenever before/after frozen scripts.
E.g. pyexec_file_if_exists() searches frozen scripts first before the current path. Searching ROMFS should be added in between. When used from main.c to run "boot.py" and "main.py" this would ensure that the ROMS partition is considered and executed first.
Similarly, when searching for an import, before searching for frozen scripts, ROMFS is also interrogated.
...
Generally, the goal here is one where you can put your whole application on ROMFS and then /flash or /sdcard don't matter except for storing logs or config files.
Code Size
This can be an optional feature.
Implementation
I hope the MicroPython maintainers or community will implement this feature
Code of Conduct
Yes, I agree
Define `vfs.rom_ioctl()`, add romfs commands to mpremote, and implement ROM partition support on stm32, rp2, esp32, esp8266
Summary
This is a continuation of #8381 to add general ROMFS support to the ports.
So far, we have the vfs.VfsRom filesystem format defined and driver implemented. That was merged via #16446. That allows creating and mounting a ROMFS filesystem if you have the ROMFS image in memory somewhere.
But there's not yet any way to use the ROMFS on the ports, because they don't have a way to get the ROMFS image on them, or access it.
This PR adds support for ROMFS on stm32, rp2, esp32 and esp8266.
I wanted to keep this PR relatively self-contained and minimal (although it's still rather large), so it's easier to review. Then further work can be done later on, eg to add support to more ports. PR #8381 does contain implementations of the remaining ports.
This PR has the following:
- Defines a new function
vfs.rom_ioctl()which is used as a general interface to access the read-only memory of a device. It can query, erase and write this memory. - A general helper function to mount a ROMFS at
/romin the filesystem. - When
MICROPY_VFS_ROMis enabled, there is automatic mounting of the first ROM partition (if found usingvfs.rom_ioctl) at the end of the call tomp_init(), so that all ports behave the same way. This also makes it much easier to enable the ROMFS feature, without adding anything to_boot.py(which is hard to do conditionally). - New
mpremotecommands which usevfs.rom_ioctl()to query, build and deploy ROMFS images. - Documentation for the new
mpremotecommands. - Implementations of
vfs.rom_ioctl()in the following ports: stm32, rp2, esp32, esp8266. - PYBD_SF2/3/6 boards have a ROMFS partition defined, using previously inaccessible QSPI flash.
- A new esp8266 board variant is defined for 2M+ flash parts that adds a 320k ROMFS partition.
Note that no other boards have a ROMFS partition enabled by default, although it's possible to enable it yourself by modifying the board definition (or creating your own board definition with it enabled). The reason for not enabling it by default on more boards is because it's a breaking change for the user, since it allocates a lot of flash to the ROMFS and takes it away from the firmware and possible frozen code. In the future we can work out how to migrate all boards to have a ROMFS, but for now that's left as an unsolved problem.
Testing
Tested on PYBD_SF2, PYBD_SF6, esp8266 using the new variant.
Also tested on rp2 and esp32 boards by modifying the board definition as described in the commit messages for those ports.
Trade-offs and Alternatives
This is a big new feature! It's hard to get big things right first go so that's why I wanted to take smaller steps to add independent parts at a time. Also thanks to @projectgus for advice in this regard.
For now ROMFS is an advanced feature and most users would need to modify board definitions, or create custom ones, to use it. Alternatives would be to just enable it on all boards and provide some docs for users freezing a lot of code to migrate to the new system. Or, postpone this feature even longer until we can work out how to better configure the ROMFS partitions. But that will delay things even more and this feature has been in the works for years now.