QUERY · ISSUE

mip / package.json: Allow 'deps' to come from another repository

openby hmaerkiopened 2023-10-14updated 2023-10-18

Problem

In repo https://github.com/hmaerki/experiment_micropython_mip, I create a mip-package which may be install as follows:

import mip
mip.install("dryer2023", version="main", index="https://hmaerki.github.io/experiment_micropython_mip")

This is the package.json file:

{
    "version": "0.1",
    "hashes": [
        ["dryer2023/statemachine.mpy", "a47f24f2064b"],
        ["dryer2023/__init__.mpy", "c4104085da6d"]
    ],
    "deps": [
        ["umqtt.simple",  "latest"]     <== ERROR A
        ["https://micropython.org/pi/v2/package/6/umqtt.simple/latest.json",  "dummy"]  <== ERROR B
    ]
}

The installation of the hashes works nice.

In the deps section, I would like to install umqtt.simple from another index (the micropython default) but I could not make it working.

ERROR A: Will search in my index (https://hmaerki.github.io/experiment_micropython_mip) which is wrong.
ERROR B: This will correctly download 'latest.json' but then again will try to get the mpy-files from my index which is wrong.

Proposed: Solution

["umqtt.simple", "latest"] should always search in https://micropython.org/pi/v2
["github:org/repo/path/latest.json", "latest"] should search in github:org/repo/path/latest.json

Are there other suggestions/thoughs?

1 comment

ned-pcs · 2023-10-18

Take a look at my work in #676

CANDIDATE · PULL REQUEST

mip: Support relative source file paths in package.json.

openby ned-pcsopened 2023-06-10updated 2023-06-13

This PR adds the ability for mip to load files from relative source paths given in the urls section of package.json files.

Previously, the only accepted forms for source paths were http://, https://, or github:.

These changes allow a source path to be given without one of these prefixes, in which case the URL of the parent of the package.json is prepended.

So instead of someone forking a repository and having to change source lines that start with
github:upstream_repository/filename
into
github:my_repository/filename
they can now just say
filename

which will survive forking or cloning without change.

This also allows doing mip.install() from a file:// location. This makes it easy to do a git clone of a repository and test installation of a package from the local filesystem, especially using the unix port.

To install this package from my fork for testing, (I think you can) use commands such as:

$ mpremote connect /dev/ttyUSB0 mip install --index https://ned-pcs.github.io/micropython-lib/mip/feature/mip-allow-relative-sources mip

Or from a networked device:

import mip
mip.install(mip, index="https://ned-pcs.github.io/micropython-lib/mip/feature/mip-allow-relative-sources")

And then restart your device, as you have now added a new version of mip. To test this, you will need to make sure that your lib directory comes before .frozen in your sys.path before importing mip.

5 comments

andrewleech · 2023-06-11

Adding support for installation from a local folder has been on my to-do list, thanks for adding it here.

I'm also keen to support direct install from manifest rather than needing the json file but it feels right to add just the relative & local path stuff as a clean change first. I'll still try to look into inline manifest support myself on top of this eventually!

ned-pcs · 2023-06-11

Actually, directly creating a package.json from a manifest.py doesn't really cover the entire requirement.

As a package developer, I want to ignore .mpy files and other intermediate/implementation details, concentrating on the source files. However, the source files may well include:

Python files, which get handled pretty well presently, both in micropython-lib distribution and in the micropython build process
Binary resources, which need to be converted to Python using mpy_bin2res.py and loaded using pkg_resources
Other files that need special processing (like the conversion of utemplate .tpl files into _tpl.py generators using utemplate_util.py

The problem is that the manifest.py doesn't allow for these other conversion steps (some of which may need to be specified in a package-dependent manner, depending on the nature of the source files).

And whatever scheme is developed will have to deal both with building into frozen flash as well as distribution via files or networking.

I guess this is something that needs a broader discussion with the community.

andrewleech · 2023-06-11

I know support for freezing non-python resources is included in this PR: https://github.com/micropython/micropython/pull/8381

This will presumably include the manifest support for describing said resources, however I don't know if conversion across to json or mip handling has been considered.

The trick of converting/wrapping them in a python file is a neat workaround for now, but ultimately inefficient and fiddly.

That being said it can be quite handy, I've actually just used the same binary 2 python concept to wrap up libraries and files to build a cpython app as a single static exe on desktop with pyoxidizer :-)

ned-pcs · 2023-06-12

OK, I tested some additional cases and found that I was getting OSError -113 (and/or 118?) from time to time.
I added retries around these errors after a gc.collect() and a sleep, which seems to have increased reliability on my ESP32.

ned-pcs · 2023-06-12

I would like to add an optional third field for the contents of urls that would allow for secondary github: URLs (those not from the same repo as the package.json) to specify a non-HEAD branch. My current PR just sets the branch for secondary github: URLs to HEAD.

mip / package.json: Allow 'deps' to come from another repository

Problem

Proposed: Solution

mip: Support relative source file paths in package.json.

Keyboard