QUERY · ISSUE

RFC: URL/package.json support in frozen module manifests

openby fill1890opened 2024-09-04updated 2024-09-04

enhancement

Description

manifest.py files currently support installing from micropython-lib with require(); I propose extending the syntax to match that of mip, namely by adding support for a package.json hosted elsewhere (URL, GitHub, or GitLab) and individual .py files by URL.

My initial implementation thoughts are:

Allow github:, gitlab:, and http:// links to be passed to require() in addition to normal package names
Add an optional version=<x> keyword argument to match the mip.install() interface
Port the existing mip functionality either from the module or from mpremote to download modules to a local cache directory during manifest processing
Add these downloaded files to the ManifestFile database so they can be frozen into the firmware with the existing manifest machinery

Please let me know if you have any thoughts/comments/suggestions.

Justification: I'm currently working on a project that uses a number of external libraries I freeze into firmware. I'm using git submodules to pull in library dependencies within the project and package()/modules() to include these into the firmware, however:

I anticipate expanding to multiple firmware variations, which may use different sets of libraries; it may become tedious to set up all required submodules and fetch the correct ones for the firmware being built
This doesn't provide support for freezing a single .py file (unlike mip) from a repository without cloning the entire repository. For example, I'd like to include the typing.mpy file in my firmware for the libraries I use, but I would rather not include the entire repository for it as a submodule.

Code Size

No response

Implementation

I intend to implement this feature and would submit a Pull Request if desirable

Code of Conduct

Yes, I agree

CANDIDATE · PULL REQUEST

tools/manifestfile.py: Add a library for working with manifests.

mergedby jimmoopened 2022-07-15updated 2022-09-05

tools

Work in progress... comments/thoughts welcome!

This is the first step towards updating MicroPython's package management and deployment system in a way that combines freezing, on-device-install and host-based-install.

This PR does seven things:

Splits a library to process manifest files out of makemanifest.py
Extends manifests to be more about "I require this package/module", and make freezing a specific case of that.
Simplify the common case of defining a Python module or package (this is especially the case for libraries in micropython-lib and drivers from the main repo).
Simplify common uses of the include() function ("/manifest.py" is implied if the path is a directory).
Make micropython-lib dependencies able to be expressed with require() rather than include. This will later generalise to use cases where the resource is fetched from the network rather than having the repo checked out locally.
~Update all manifests in this repo to use the new features.~ (Moved to #9037)
Add a Python wrapper for mpy-cross (based on https://gitlab.com/alelec/mpy_cross by @andrewleech), and update makemanifest.py to use it. This will be used to generate pre-compiled binaries for modules to be published for on-device installation.

Part of this is inspired by #8231 (@andrewleech) and the idea is that this "manifestfile" library can be used by tools like mpremote to manage a project and its dependencies that can be deployed to a board. Similarly, the mpy-cross library can be used by mpremote to compile "on-the-fly" during deployment.

In more detail, the idea is to use this to support the following use case:

$ mpremote connect /dev/ttyACM0 install foo

where foo is a package from micropython-lib, and "foo" and its dependencies will be installed on the device. For more sophisticated uses-cases:

$ mpremote connect /dev/ttyACM0 deploy app_manifest.py

where the whole app and dependencies (as described by module/package/include/require commands in the manifest) will be deployed in an rsync-like process. Additionally, it would integrate with #8381 in order to build memory-mappable images -- i.e. the benefits of freezing, but without recompiling or even needing to access the main repo at all).

The next step after this PR is to update the process that builds https://micropython.org/pi (this is where upip downloads resources from) to use this new library, and the subsequently update upip.py to work with this. The result will be:

upip starts working again
upip will download compiled .mpy files rather than .py files

There are some details to still sort out, most importantly the future of PyPI in this picture. Supporting PyPI from upip is quite complicated. Producing libraries that can be deployed to PyPI and deploy correctly via upip is actually quite difficult and unnecessarily burdensome for third-party developers, and quite inefficient from the device side. Randomly sampling a few projects confirmed that this is causing problems.

Personally I would prefer to see more libraries added to micropython-lib, but for most projects that are a single file or a package of a few files I would rather invest the effort in having upip be able to install directly from the author's GitHub repo e.g. >>> upip.install('github:foo/bar') where the repo contained some simple descriptor/metadata file that was generated from a manifest.py. In many cases, a single-file library with no dependencies could be >>> upip.install('github:foo/bar/bar.py'). The same commands would work from mpremote as described above too.