While working on c_module() manifest support (#18229) I noticed that user C module code on ESP32 is compiled without optimisations across all architectures (esp32, esp32-s3, esp32-c3, esp32-p4). On rp2 (also cmake-based) user modules compile with optimisations as expected.

ESP-IDF applies optimisation flags via directory-level add_compile_options() during idf_component_register(). These flags don't propagate to sources that enter the build through the usermod INTERFACE library link. RP2 only works because CMAKE_BUILD_TYPE=MinSizeRel sets flags via CMAKE_C_FLAGS which is truly global.

I found a workaround on ESP32:

idf_build_get_property(_idf_compile_opts COMPILE_OPTIONS)
target_compile_options(usermod INTERFACE ${_idf_compile_opts})

Investigating further though, this is a symptom of a broader architectural mismatch between MicroPython's usermod cmake pattern and how platform build systems actually work.

How it got this way

The INTERFACE library pattern was introduced in 0cf12dd59c (2021, PR #6960) by Graham Sanderson (Raspberry Pi). The dual-path design was deliberate from day one — both the INTERFACE library linking and usermod_gather_sources() appeared in the same commit.

The rationale was sound at the time: INTERFACE library linking is idiomatic cmake for aggregating sources across targets, and when an executable links against an INTERFACE library cmake compiles the INTERFACE_SOURCES as part of that executable target. The separate usermod_gather_sources() function was written specifically because MicroPython's QSTR code generation pipeline uses add_custom_command which needs explicit file paths at configure time — INTERFACE library properties can contain generator expressions that only resolve at build time. The original comment says it explicitly: "Recursively gather sources for QSTR scanning - doesn't support generator expressions."

So the gather function was never intended for compilation. The INTERFACE library was supposed to handle that correctly, and it did on rp2 (the first cmake port). The assumption was that the consuming target's compilation context would propagate to INTERFACE sources, which works when flags are set globally (CMAKE_C_FLAGS on rp2) but breaks when platform build systems apply compilation context at the directory/component level (ESP-IDF's add_compile_options during idf_component_register). Nobody connected these issues back to the same root cause until now because they surfaced gradually over years through different symptoms.

The problem

The usermod system uses a CMake INTERFACE library as a cross-platform abstraction layer. Each user module creates its own INTERFACE target with sources/includes/definitions, links to the parent usermod target, then the port links against usermod to compile everything.

flowchart TD
    subgraph "User Module cmake files"
        M1["usermod_cexample<br/><i>INTERFACE library</i><br/>INTERFACE_SOURCES<br/>INTERFACE_INCLUDE_DIRECTORIES"]
        M2["usermod_cppexample<br/><i>INTERFACE library</i><br/>INTERFACE_SOURCES<br/>INTERFACE_INCLUDE_DIRECTORIES"]
        M3["usermod_subpackage<br/><i>INTERFACE library</i><br/>INTERFACE_SOURCES<br/>INTERFACE_INCLUDE_DIRECTORIES<br/>INTERFACE_COMPILE_DEFINITIONS"]
    end

    UM["usermod<br/><i>INTERFACE library</i>"]
    M1 -->|target_link_libraries| UM
    M2 -->|target_link_libraries| UM
    M3 -->|target_link_libraries| UM

    subgraph "py/usermod.cmake"
        UGS["usermod_gather_sources()<br/>Recursively walks INTERFACE tree"]
    end

    UM --> UGS

    UGS -->|"flat list"| SRC["MICROPY_SOURCE_USERMOD"]
    UGS -->|"flat list"| INC["MICROPY_INC_USERMOD"]

    subgraph "Port build system"
        QSTR["QSTR scanning<br/><i>uses flat list</i>"]
        INCD["Include directories<br/><i>uses flat list</i>"]
        COMP["Source compilation<br/><i>uses INTERFACE library link</i>"]
    end

    SRC -->|"added to MICROPY_SOURCE_QSTR"| QSTR
    INC -->|"added to INCLUDE_DIRS"| INCD
    UM -->|"target_link_libraries"| COMP

    style COMP fill:#f96,stroke:#333
    style QSTR fill:#9f9,stroke:#333
    style INCD fill:#9f9,stroke:#333

There's a dual-path architecture here: QSTR scanning and include directories use gathered flat lists (which go through the native build system), but actual source compilation uses the INTERFACE library link (which bypasses it).

Every other source domain in MicroPython (py, extmod, shared, lib, drivers, port, board) is registered directly as flat source lists in the port's native build system. For example on ESP32, esp32_common.cmake:

idf_component_register(
    SRCS
        ${MICROPY_SOURCE_PY}
        ${MICROPY_SOURCE_EXTMOD}
        ${MICROPY_SOURCE_SHARED}
        ${MICROPY_SOURCE_LIB}
        ${MICROPY_SOURCE_DRIVERS}
        ${MICROPY_SOURCE_PORT}
        ${MICROPY_SOURCE_BOARD}
        ${MICROPY_SOURCE_TINYUSB}
        # MICROPY_SOURCE_USERMOD is notably absent
    ...
)

usermod is the only domain that uses INTERFACE library indirection for compilation. CMake INTERFACE libraries are designed for header-only libraries and property propagation, not for carrying compilable source files. Using INTERFACE_SOURCES to transport source files across build system boundaries means the compilation context (flags, optimisation, definitions) depends entirely on how the consuming platform target is configured, and each platform does this differently.

The usermod_gather_sources() function in py/usermod.cmake already acknowledges this gap — it walks the INTERFACE library dependency tree and extracts flat source/include lists (MICROPY_SOURCE_USERMOD, MICROPY_INC_USERMOD). But these lists are only used for QSTR scanning and include directories. Source compilation still goes through the INTERFACE library link.

This is possibly the common thread through several existing issues

1. Optimisation flags not propagating (this issue, discovered via #18229) — ESP-IDF applies flags at directory level during idf_component_register(). Sources entering via target_link_libraries(${MICROPY_TARGET} usermod) don't inherit these. RP2 works by accident because CMAKE_BUILD_TYPE is global.

2. Double compilation (#16284) — adding target_compile_options to a per-module usermod INTERFACE target caused sources to compile in two contexts, producing duplicate .o files and linker errors.

3. User modules cannot add IDF component dependencies (#8041, still open) — idf_component_register(REQUIRES ...) runs during the early expansion phase before user module cmake files are processed. Workaround: use idf_component_get_property() for include paths only (#12972).

4. QSTR generation blind to user module compile definitions (Discussion #13538) — micropy_gather_target_properties in py/py.cmake collects INTERFACE_COMPILE_DEFINITIONS but not INTERFACE_COMPILE_OPTIONS. User module conditional compilation via #ifdef breaks QSTR extraction.

Fix: register usermod sources through native build systems

The fix is to align usermod compilation with every other source domain by using the already-gathered flat lists for compilation, not just QSTR scanning.

flowchart TD
    subgraph "User Module cmake files (unchanged)"
        M1["usermod_cexample"]
        M2["usermod_cppexample"]
    end

    UM["usermod<br/><i>INTERFACE library</i>"]
    M1 -->|target_link_libraries| UM
    M2 -->|target_link_libraries| UM

    UGS["usermod_gather_sources()"]
    UM --> UGS

    UGS -->|"flat list"| SRC["MICROPY_SOURCE_USERMOD"]
    UGS -->|"flat list"| INC["MICROPY_INC_USERMOD"]

    subgraph "Port build system"
        QSTR["QSTR scanning"]
        INCD["Include directories"]
        COMP["Source compilation"]
    end

    SRC -->|"added to MICROPY_SOURCE_QSTR"| QSTR
    SRC -->|"added to native SRCS"| COMP
    INC -->|"added to INCLUDE_DIRS"| INCD

    UM -.->|"target_link_libraries<br/>(non-source properties only)"| COMP

    style COMP fill:#9f9,stroke:#333
    style QSTR fill:#9f9,stroke:#333
    style INCD fill:#9f9,stroke:#333

What this involves:

• Each port adds MICROPY_SOURCE_USERMOD to its native source registration (e.g. idf_component_register(SRCS ...) for ESP32, target_sources() for RP2)
• Strip INTERFACE_SOURCES from the usermod target chain after gathering to prevent double compilation
• Keep target_link_libraries(${MICROPY_TARGET} usermod) for propagating non-source INTERFACE properties (compile definitions, compile options, link dependencies)
• No change to user-facing API — existing micropython.cmake files work as-is

The main thing to watch for is edge cases where user modules rely on undocumented INTERFACE source propagation behaviours, though I'd expect that to be rare since the current behaviour is already broken in several ways.

This applies to CMake-based ports only (ESP32, RP2). Make-based ports (stm32, unix, alif, etc.) handle user modules through py/usermod.mk which doesn't have these issues since Make doesn't have the INTERFACE library concept.