QUERY · ISSUE

micropython-lib version included in official downloads is unclear

openby jasegopened 2020-01-20updated 2020-01-20

The official downloads page lists pre-built images by micropython version, but does not give any information on which micropython-lib version is used. I've had a few issues where I got stack traces into micropython-lib but was unable to find the matching source version.

I think either the micropython-lib repo should contain tags pointing to the corresponding micropython versions, or the micropython-lib version/commit/tag should be linked on the downloads page and somewhere in the micropython repo.

Thanks!

CANDIDATE · PULL REQUEST

micropython/mip: Add a new `mip` library for on-device installation.

mergedby jimmoopened 2022-09-28updated 2022-09-30

This PR adds:

A library (mip) that can be frozen (or manually installed) on network-capable boards for on-device installation of packages from micropython-lib (or other similar package indices).
A tool (tools/build.py) to deploy micropython-lib to a static web server suitable for access from mip (or mpremote when similar functionality is added).

The structure of the static web server output is explained in the comments at the top of deploy.py.

mip.install() supports the following use cases:

mip.install("foo") -- this will download the latest version of the foo package (and dependencies) from micropython-lib.
mip.install("foo", version="x.y") -- install foo version x.y, including dependencies at the version when foo x.y was published
mip.install("foo", target="path/to/lib") -- install foo and dependencies in specified path (otherwise defaults to the first "lib" directory in sys.path)
mip.install("foo", index="https://example.com/") -- install foo from example.com's index
mip.install("http://example.com/x/y/foo.py") -- download foo.py directly from the URL
mip.install("http://example.com/x/y/foo.json") -- download a package, including dependencies, described by foo.json (see below)
mip.install("http://example.com/x/y") -- implicitly http://example.com/x/y/package.json
mip.install("github:org/user/path/foo.py") -- shortcut for github-hosted content (fetches user/path/foo.py from the default branch)
mip.install("github:org/user/path/mypackage.json") -- as above
mip.install("github:org/user") -- this will hopefully be the common case for "self-hosted" packages.
mip.install("github:org/user", version="devel") -- as above, but using the "devel" branch
`mip.install("foo", mpy=False) -- fetch foo but as .py files (i.e. for debugging).

Major differences in mip compared to upip:

Packages installed from the index are bytecode compiled to .mpy by default. The request includes the device's supported version.
pypi is not supported as a source.
It uses urequests rather than having its own HTTP client. (Most boards that freeze mip also freeze urequests anyway)

The package.json looks like

{
  "hashes": [
    ["aioble/server.mpy", "e39dbf64"],
    ...
  ],
  "urls": [
    ["target/path.py", "http://url/foo/bar/path.py"],
    ...
  ],
  "deps": [
    ["name", "version"],
    ...
  ]
  "version": "0.1"
}

The "hashes" are for referencing .mpy files directly from micropython-lib. This is used by micropython-lib packages, but shouldn't be used for self-hosted ones. The "urls" allows arbritrary URLs to be fetched, and "deps" allows recursive dependencies, e.g. a self-hosted package might want to provide some urls to their own files, and then depend on some micropython-lib packages (or a package json in another github repo).

Note: deploy.py requires https://github.com/micropython/micropython/pull/9437

This work was funded through GitHub Sponsors.

11 comments

jimmo · 2022-09-28

I've written a small demo of what writing a "self-hosted" (on GitHub) package will look like, including how to specify dependencies: https://github.com/jimmo/micropython-mlx90640

This can be installed by

>>> mip.install("jimmo/micropython-mlx90640") # installs from default branch (main)
>>> mip.install("jimmo/micropython-mlx90640", version="devel") # use the devel branch (equivalently, tag)

Of course I'd prefer for a driver like this one to end up in micropython-lib and therefore get all the benefits of being bytecode compiled etc, but this third-party publishing (to github or other web server) is still very much a use case we want to support.

jimmo · 2022-09-28

Another artifact of the build is a top-level index.json (which will be https://micropython.org/pi/v2/index.json) which contains the full list of available packages from micropython-lib. mip doesn't consume this, but other tools that want to discover available packages can. ~What's included in this for now is pretty minimal, but unlike the package metadata I don't imagine this being consumed by low-memory devices so this could have more info added (possibly version history etc).~ (Edit: added full version history for each bytecode version, as well as description, author, license)

dpgeorge · 2022-09-29

I checked the frozen version of this new upip, compared to the old one (compiled with mpy-cross v6 using -O0, ie no optimisations). This is to see how the new one compares to the old one in terms of flash size. Note that the new upip relies on urequests, so the total size of the new upip needs to include the size of urequests.

Old upip size:

esp8266: 6056 bytes frozen
rp2: 5832 bytes frozen

New upip+urequests combined size:

esp8266: 5536 bytes frozen
rp2: 5360 bytes frozen

So the new upip combined with urequests is about 500 bytes smaller (around 10% smaller) than the old upip. Even bigger savings are made when urequests already exists for other reasons. In that case upip on its own is just 2464 bytes frozen (on rp2).

Furthermore the dependencies on built-in C modules are reduced with the new upip. The dependencies are:

Old upip: sys, gc, os, errno, json, zlib, uctypes, ssl, socket
New upip: sys, os, binascii (optional), hashlib (optional), and then json, ssl, socket via urequests.

So gc, errno, zlib and uctypes are no longer required by the new upip, compared to the old one.

jimmo · 2022-09-29

I've pushed an updated version that adds two features requested via private feedback:

Ability to install "source only" versions of packages. Alongside the supported bytecode versions, there's now a "py" version. This can be requested using mip.install(..., mpy=False).
Added version history to the index.json. This allows you to discover the available versions for each bytecode version.

dpgeorge · 2022-09-29

What about the ability to do, on the unix port, micropython -m upip install foo? That could be done with the old upip. Do we need to support that?

Edit: maybe we can have a separate upip-cli that supports command-line for unix?

jimmo · 2022-09-29

Addressed comments above.

Edit: maybe we can have a separate upip-cli that supports command-line for unix?

Yep, I want to do this like the change we made for unittest where if you install unittest-discover then you also get micropython -m unittest. (Separate PR)

jimmo · 2022-09-29

As discussed, renamed upip to mip, and deploy.py to build.py.

dpgeorge · 2022-09-29

Thank you, this is awesome!

peterhinch · 2022-09-30

Are there any ideas on how to support platforms like Pico that aren't network-enabled?

jimmo · 2022-09-30

Are there any ideas on how to support platforms like Pico that aren't network-enabled?

@peterhinch
Yep! https://github.com/micropython/micropython/pull/9467

jimmo · 2022-09-30

And https://github.com/micropython/micropython/pull/9463 included a significant overhaul of https://docs.micropython.org/en/latest/reference/packages.html (which hasn't gone live yet, but will hopefully update soon).

micropython-lib version included in official downloads is unclear

micropython/mip: Add a new `mip` library for on-device installation.

Keyboard