Micropython rp2 fails to build on windows systems.
When attempting to build micropython for rp2040 in a windows environment, the make/cmake environment generates a command line that exceeds the windows10/cmd.exe line length limit of 8191 characters.
This is because the RP2 SDK requires about 3k worth of C include paths, and the list of files compiled that is generated on a single command line is about 6k (depending on exact path names.)
Extensive discussion and whining here: https://www.raspberrypi.org/forums/viewtopic.php?f=144&t=306333
I'll add that compiling all the C files for micropython with a single command line seems ... dangerously likely not to scale well on other build platforms as well.
Fix "Argument list too long" error in qstr generation for large projects
Problem
When building MicroPython projects with a large number of source files, the qstr generation step fails with shell errors like:
/bin/sh: Argument list too long
This occurs because the makeqstrdefs.py script receives all source file paths as command-line arguments, which can exceed the shell's ARG_MAX limit. This issue was first encountered when including large user C modules like LVGL, which adds 1200+ files to the qstr scanning list.
The ARG_MAX limit varies significantly between systems:
- Typical Linux systems: 128KB - 2MB depending on kernel version
- This explains why builds succeed on some machines but fail on others with identical source code
- The same project may build successfully on a system with a newer kernel but fail on older systems with lower limits
The issue manifests during the qstr preprocessing step in py/mkrules.mk when generating $(HEADER_BUILD)/qstr.i.last.
Solution
This PR implements support for response files (argument files) in the qstr generation toolchain. Response files allow passing the large file list via a file instead of on the command line, completely avoiding ARG_MAX limitations for the problematic argument.
Changes
-
py/mkrules.mk: Modified the qstr generation rule to write only the source file list to a temporary file (
qstr_sources.txt). All other arguments (CPP, flags, dependencies) remain as command-line arguments. -
py/mkrules.cmake: Modified to write the semicolon-separated source list to a file at configure time. This avoids passing the huge list as command-line arguments to any build-time process.
-
py/makeqstrdefs.py: Enhanced the argument parser to detect and read response files when an argument starts with
@. The parser now handles both newline-separated format (from Make builds) and semicolon-separated format (from CMake builds).
Implementation Details
- Response files are indicated by prefixing the filename with
@(e.g.,sources @qstr_sources.txt) - Only the large source file list is moved to a response file; other arguments stay on the command line
- The format is simple: one file path per line
- Maintains backward compatibility with existing command-line invocations
- Minimal, targeted code changes focused on the specific problem (file list too long)
Testing
This fix has been tested with:
- Standard MicroPython ports (mimxrt, unix, stm32) using Make builds
- ESP32 and RP2 ports using CMake builds
- LVGL-based projects with 1200+ source files where the previous implementation would fail
- Existing projects to verify backward compatibility
Related Issues
This addresses build failures in projects with extensive use of external libraries, particularly those using LVGL bindings or other large codebases that push against shell argument limits.