QUERY · ISSUE

mip: github branch is incorrectly used for secondary github: sources in package.json

openby ned-pcsopened 2023-06-12updated 2023-06-12

mip version 0.2.0

When you specify a version to mip.install() to reference a particular GitHub branch, this branch is used for every github: source in the package.json, even those coming from a different repository.

The best solution to this is probably:

provide a way (perhaps an optional third field) for a URL to specify a different branch
default the branch to HEAD for secondary github: urls if they don't provide an explicit branch

In my PR #676 I have done the second part of this (default to HEAD) but not the first part yet.

CANDIDATE · PULL REQUEST

mip: Support relative source file paths in package.json.

openby ned-pcsopened 2023-06-10updated 2023-06-13

This PR adds the ability for mip to load files from relative source paths given in the urls section of package.json files.

Previously, the only accepted forms for source paths were http://, https://, or github:.

These changes allow a source path to be given without one of these prefixes, in which case the URL of the parent of the package.json is prepended.

So instead of someone forking a repository and having to change source lines that start with
github:upstream_repository/filename
into
github:my_repository/filename
they can now just say
filename

which will survive forking or cloning without change.

This also allows doing mip.install() from a file:// location. This makes it easy to do a git clone of a repository and test installation of a package from the local filesystem, especially using the unix port.

To install this package from my fork for testing, (I think you can) use commands such as:

$ mpremote connect /dev/ttyUSB0 mip install --index https://ned-pcs.github.io/micropython-lib/mip/feature/mip-allow-relative-sources mip

Or from a networked device:

import mip
mip.install(mip, index="https://ned-pcs.github.io/micropython-lib/mip/feature/mip-allow-relative-sources")

And then restart your device, as you have now added a new version of mip. To test this, you will need to make sure that your lib directory comes before .frozen in your sys.path before importing mip.

5 comments

andrewleech · 2023-06-11

Adding support for installation from a local folder has been on my to-do list, thanks for adding it here.

I'm also keen to support direct install from manifest rather than needing the json file but it feels right to add just the relative & local path stuff as a clean change first. I'll still try to look into inline manifest support myself on top of this eventually!

ned-pcs · 2023-06-11

Actually, directly creating a package.json from a manifest.py doesn't really cover the entire requirement.

As a package developer, I want to ignore .mpy files and other intermediate/implementation details, concentrating on the source files. However, the source files may well include:

Python files, which get handled pretty well presently, both in micropython-lib distribution and in the micropython build process
Binary resources, which need to be converted to Python using mpy_bin2res.py and loaded using pkg_resources
Other files that need special processing (like the conversion of utemplate .tpl files into _tpl.py generators using utemplate_util.py

The problem is that the manifest.py doesn't allow for these other conversion steps (some of which may need to be specified in a package-dependent manner, depending on the nature of the source files).

And whatever scheme is developed will have to deal both with building into frozen flash as well as distribution via files or networking.

I guess this is something that needs a broader discussion with the community.

andrewleech · 2023-06-11

I know support for freezing non-python resources is included in this PR: https://github.com/micropython/micropython/pull/8381

This will presumably include the manifest support for describing said resources, however I don't know if conversion across to json or mip handling has been considered.

The trick of converting/wrapping them in a python file is a neat workaround for now, but ultimately inefficient and fiddly.

That being said it can be quite handy, I've actually just used the same binary 2 python concept to wrap up libraries and files to build a cpython app as a single static exe on desktop with pyoxidizer :-)

ned-pcs · 2023-06-12

OK, I tested some additional cases and found that I was getting OSError -113 (and/or 118?) from time to time.
I added retries around these errors after a gc.collect() and a sleep, which seems to have increased reliability on my ESP32.

ned-pcs · 2023-06-12

I would like to add an optional third field for the contents of urls that would allow for secondary github: URLs (those not from the same repo as the package.json) to specify a non-HEAD branch. My current PR just sets the branch for secondary github: URLs to HEAD.

mip: github branch is incorrectly used for secondary github: sources in package.json

mip: Support relative source file paths in package.json.

Keyboard