mip: package installation using GitLab URLs for repos in subgroups
This is more of an awareness piece and something to trigger a discussion. There is also the possibility that I am wrong and the installation process is working as intended.
I'm currently developing MicroPython packages on GitLab, which sit inside a 'Libraries' subgroup, which is inside a parent 'MicroPython IoT Projects' group.
Due to the way that the mip _rewrite_url function parses GitLab and GitHub URLs, the assumption is made that the first component after splitting the URL, is the org and the second is the repository slug, which doesn't account for subgroups.
https://github.com/micropython/micropython-lib/blob/96bd01ec047923c15bf936d2f77043968745542d/micropython/mip/mip/init.py#L63
As a result, I had to change the way I reference my packages and package extension urls and deps in the package.json files. My network-utils 'package.json' has to have the following structure:
{
"urls": [
["network_utils/__init__.py", "https://gitlab.com/micropython-iot-projects/libraries/micropython-network-utils/-/raw/HEAD/network-utils/network_utils/__init__.py"]
],
"deps": [
["logging", "latest"],
["github:josverl/micropython-stubs/mip/typing.mpy", "main"],
["github:josverl/micropython-stubs/mip/typing_extensions.mpy", "main"]
],
"version": "0.0.1"
}
In order to install network-utils with mpremote, I would have to use the following command:
mpremote mip install https://gitlab.com/micropython-iot-projects/libraries/micropython-network-utils/-/raw/HEAD/network-utils/package.json
The command below will not work, resulting in a 403 error:
mpremote mip install gitlab:micropython-iot-projects/libraries/micropython-network-utils/network-utils/package.json
The network-utils-mqtt extension package, which requires network-utils, had to have its package.json written like so:
{
"urls": [
["network_utils/mqtt.py", "https://gitlab.com/micropython-iot-projects/libraries/micropython-network-utils/-/raw/HEAD/network-utils-mqtt/network_utils/mqtt.py"]
],
"deps": [
["https://gitlab.com/micropython-iot-projects/libraries/micropython-network-utils/-/raw/HEAD/network-utils/", "develop"]
],
"version": "0.0.1"
}
The URL for the deps, which is the network-utils package this one extends, had to be rewritten so that the package.json could be found correctly.
micropython/mip/mip/__init__.py: Add an alias for codeberg repos.
Summary
This PR introduces an alias to access codeberg repos from within the mip embedded package manager.
Right now packages hosted on codeberg could only be referenced by their full URL, unlike other packages hosted on either GitHub or GitLab. To make access those packages easier, now they can be referenced as "codeberg:org/repo@branch".
This is the micropython-lib counterpart of https://github.com/micropython/micropython/pull/18989.
Testing
The same package mentioned in the other repo's PR (https://codeberg.org/agatti/micropython-package-test) was installed successfully via import mip; mip.install("$URL") on an ESP32C3.
The ESP8266 is not able to access codeberg repos due to a SSL cipher incompatibility (see #400), but there isn't much that can be done for that right now.
Trade-offs and Alternatives
The same remarks about whether adding a new provider or not, mentioned in the micropython PR still apply.
A similar refactoring has been applied to the on-device python code - albeit with a few more aggressive size optimisations, having a final size impact of 31 bytes (for reference, the new codeberg URL string is 29 bytes long...).
The extra code size be brought down to 29 bytes by taking out the /{}/{} suffix, rebuild that string fragment inside _rewrite_url, and append that to the final rewritten URL. Not sure if all that extra complexity is worth for just two bytes. A more approachable size reduction could be had by renaming _PROVIDERS into something shorter. Suggestions for the new name are welcome :)
Generative AI
I did not use generative AI tools when creating this PR.