← index #5654Issue #4001
Related · high · value 0.125
QUERY · ISSUE

RFC: Do we need (better) support for runtime initialization of C modules

openby stinosopened 2020-02-17updated 2021-10-21

All (?) modules in the MicroPython source code and external C modules are created by defining a const mp_obj_module_t object which basically gets populated at compile time by specifying the globals table. This was done so because it allows putting everything in ROM, I assume. The module object then somehow has to make it into mp_builtin_module_table so it can be found by the import logic. However another way to make modules available within MicroPython is creating an mp_obj_module_t at runtime and register it. Essentially something like

const qstr module_name = qstr_from_str("mymodule");
mp_obj_module_t *module = (mp_obj_module_t*) mp_obj_new_module(module_name);
//Populate the module with variables/functions/classes/....
mp_obj_dict_store(module->globals, "global_var", mp_const_true);
...
//Register module so 'import mymodule' in Python now works
mp_module_register(module_name, module);

this offers some flexibility for making certain things available or not based on runtime conditions for instance. Or if you're short on ROM it allows creating your module in RAM instead (well, just a small part i.e. the globals table not sure if that really helps). It's similar to what gets done in the mpy_init function of native modules, it's also something which is standard in CPython (C modules have a PyInit_xxx which gets called upon first import) and actually there's an example of something similar in unix/main.c (although that just registers a type globally not in a specific module, but the principle is the same).

However there isn't really (see below) any build support which allows doing this: there's no way to say 'here's an initialization function for a module X, call it to create the module X when encountering import X. Is that something we need? I know I could use it because literally all my modules work like that, just not sure if other people ever felt any need for it. Or even thought about it. Implementation could be relatively simple I think: a table similar to how mp_builtin_module_table works but with key/value pairs of module name and a pointer to their init function which gets called to initialize the module once, and a cache for the loaded ones.

Addendum: currently there are some ways to have these 'runtime initialized modules':

  • you can just patch e.g. main.c and add whatever code you need to create a module
  • I found a way to leverage the external C module stuff to do it but it's a bit hacky and needs some boilerplate for each module. Idea is that you enable MICROPY_MODULE_BUILTIN_INIT, create a non-const mp_obj_module_t where the globals just contain one __init__ entry, pointing to a function which then replaces the module globals with a RAM dict populated with whatever the module needs. See https://github.com/stinos/micropython-wrap/blob/master/tests/cmodule.c for example
CANDIDATE · ISSUE

[discussion] Sharing custom C modules is hard

closedby sdfgeoffopened 2018-08-01updated 2020-06-10

Currently, when writing a custom C module, you have to make edits to the Makefile and mpconfigport.h.

This means that, that to distribute a custom C module, you can only do so within the context of a fork of the micropython repository. If I find one person who's written a micropython module for vectormath, and another person who's done a particular communication protocol, then in order to use both of these modules, I have to have a fork of the micropython repository and somehow merge in the code from both of these. There's no easy way to share custom C modules libraries.

I think this contributes hugely to why there are very few custom C modules publicly available for micropython - it's simply too hard to distribute them in a meaningful manner.

Ideally there would be some way of loading compiled libraries (similar to how ctypes allows importing .so files). This is probably impossible in a useful way, but it would be ideal ... allowing drag-drop loading of C modules.
A more realistic way would be to allow some way for code in subdirectories of the ports folder to be able to add their files to the Makefile and mpconfigport.h file at compile time.

I'm willing to invest some time into solving this problem, but was wondering if there are other solutions I missed, or if someone has a better idea about how to solve it.

Keyboard

j / / n
next pair
k / / p
previous pair
1 / / h
show query pane
2 / / l
show candidate pane
c
copy suggested comment
r
toggle reasoning
g i
go to index
?
show this help
esc
close overlays

press ? or esc to close

copied