Hi, I can see a lot of work's been put into this! Can you give some details of your use case / needs for wanting to keep pypi support?

In the early days of mip development a lot of discussion was had in https://github.com/micropython/micropython/pull/8914 regarding pypi support vs new clean alternatives.

Currently on pypi it's very hard to find packages that are actually intended for use on micropython, even then most of them are old / obsolete / dummy or outright broken. It was determined that this would be nigh on impossible to clean all up, leading to many users getting broken installations due to these or other cpython packages being installed.

Hi @andrewleech
this work was somehow necessary to keep my established system alive with newer MicroPython versions. I fully automated the package development including tagging and upload to PyPI or test PyPI.

I've read through all the comments of the mentioned PR and can see the reasons why it was implemented like it is. I don't want to open the discussion on these topics again, but I want to mention that having PyPI in place with all its drawbacks and changes over time, it provides in my opinion THE first entry point for packages. You point with having new users without knowledge of packaging is fair, I'm also not an expert yet there, but we could create a MicroPython specific tutorial for that.
I also agree with you that it is hard to find packages for MicroPython on PyPI but I feel that now opening the new possibilities with micropython-lib and it's way of working might not be the solution to the issue. With PyPI you had one unique name for a package, with the current approach you can have one name for tons of different packages, like with each fork. Of course only the things merged to the main branch are official, but I feel like non are actually merged back? Additionally dependency management might become really hard on larger packages or even create conflicts, but this is also the case on PyPI.
My last argument for having the PyPI support back, is having it back. I agree to cut off old stuff from time to time, but phasing it out might be a better solution than removing it completly. And really last argument is scalability: with PyPI the files and artifacts are served, with the micropython-lib each new fork, as far as I understood, creates a new GitHub page serving the 99% as before plus the one new module added, unless you merge it to your fork, but still it is then your fork. Sorry for opening the discussion ...

Thanks @brainelectronics

One of the goals of mip was to make it just do the minimum possible to support the primary use case (i.e. downloading packages from micropython-lib and simple files hosted on github/static-http) without using up too much flash space etc.

As outlined in earlier discussions, the problem with PyPI is that it's difficult to publish packages for micropython that get installed correctly (e.g. requires custom sdist) and there isn't a great story for automatic compilation to .mpy, etc. It just isn't a good solution to the thing we want, nor does it integrate into the rest of our workflows (e.g. freezing).

That said - I'm not against the idea of supporting PyPI as a source for packages, but I don't think this belongs in mip.
So how would you feel about turning this PR into a standalone tool (you could even call it pip or upip -- i would suggest python-ecosys/pip). So someone who really wants to use PyPI packages, could first mip-install-pip, then use pip from there.

Note that this new code depends on utarfile (from micropython-lib). We don't want to make utarfile a default dependency for mip, but it's confusing for users to know they need to install it to use the PyPI-specific functionality. However, if they're explicitly installing a pip package, it's fine for that to require utarfile.

(Another option would be to make a mip-pypi extension package, that adds this support to mip ... i.e. you make the minimal possible change to mip to import mip_pipi if it's available and use it if it is). We are already doing this with other packages to add optional functionality. (Even mip-cmdline is a simple example, but aioble` does it too).

Worth mentioning: is on-device installation of PyPI packages actually the feature you need. If you haven't seen it, pipkin (from the author of Thonny) is a tool to install PyPI packages to a connected device. (It's kind of conceptually like mpremote mip install but for PyPI) -- https://github.com/aivarannamaa/pipkin

it provides in my opinion THE first entry point for packages.

I agree that this does sound good, but in practice it doesn't seem to work. We constantly see "why doesn't package X from PyPI work on MicroPython", where the answer is some combination of "it only works on {CPython, CircuitPython, other-MicroPython-fork}" or "it's unmaintained" or "it never worked at all" or "it installed incorrectly".

With PyPI you had one unique name for a package, with the current approach you can have one name for tons of different packages, like with each fork.

The idea is that micropython-lib provides a single namespace of packages. Not sure I follow what you mean by "one name for tons of different packages".

each new fork, as far as I understood, creates a new GitHub page serving the 99% as before plus the one new module added, unless you merge it to your fork, but still it is then your fork.

I think there might be a misunderstanding here... Forks of micropython-lib (other than for the purpose of sending PRs) are not part of the plan. Packages are either:

• Upstreamed into micropython-lib (in which case they can be installed with mip.install("name")
• Not in micropython-lib (i.e. custom package.json listing the files and dependencies explicitly).

It is definitely not the intention that people create their own micropython-lib forks with just one or two additional packages. The purpose of the index argument to install is either for:
a) testing/development -- it's useful to run a custom deployment (e.g. on my local PC)
b) completely standalone libraries (i.e. not a fork of micropython-lib, but an entirely independent ecosystem)

(I guess you could argue that there's a variant of (b) which is that someone might create a fork of micropython-lib with sweeping changes across many packages, which I would sort of view as a independent ecosystem).

FWIW,we are also working on support for adding third-party packages to micropython-lib -- i.e. if you want to maintain your own package outside of micropython-lib, and provide a package.json, then we will scoop that up and include it in the index (including compilation to .mpy). (Just need to get v1.20 released first :D ).

Thanks for the deeper insight @jimmo I now also agree that having this change in mip makes no sense at all.

I'll move the content and changes, as you suggested, into a dedicated python-ecosys/pip package.

Worth mentioning: is on-device installation of PyPI packages actually the feature you need. If you haven't seen it, pipkin (from the author of Thonny) is a tool to install PyPI packages to a connected device. (It's kind of conceptually like mpremote mip install but for PyPI) -- https://github.com/aivarannamaa/pipkin

I already know that package, but it does not satisfy my needs, as I have several boards out in the wild which can do some kind of OTA. The user can select to use either officially release packages from PyPI or test releases from Test PyPI. That's why I need on device installations.

The idea is that micropython-lib provides a single namespace of packages. Not sure I follow what you mean by "one name for tons of different packages".

Maybe I'm wrong here, but users can opt to run the GitHub pages which serves the static files to do installations right, like --index https://USERNAME.github.io/micropython-lib/mip/BRANCH_NAME. By that, as you also mentioned, users can host custom packages in their forks. So a package named "carl" can contain lama.py on user A, but if user B creates the package "carl" with mouse.py it could get complex. I've worked in places where users used all possibilities they got, which became quite chaotic over time, that's most likly why I'm a bit over the top on some things 😄

I'm actually interested in getting some packages in (and pushed more into the light). I'll also willing to contribute here with some docs and guides. But one thing at a time, I hope to finish the new package during the weekend