QUERY · ISSUE

urequests still using HTTP/1.0

openby linusoftopened 2024-04-11updated 2024-05-06

The urequests module is still using HTTP/1.0 for HTTP requests:

line 39 in micropython-lib/micropython/urllib.urequest/urllib/urequest.py on the master branch:

s.write(b" HTTP/1.0\r\nHost: ")

Since a host header is included, can this be updated to use HTTP/1.1 or even HTTP/2.0?

Some sites (e.g. mongodb) are now responding with '426 Update Required' for HTTP/1.0 requests.

Thanks

3 comments

th4ruka · 2024-05-05

Hey! I also think it is better to update the header in line 39 of micropython-lib/micropython/urllib.urequest/urllib/urequest.py as,

- s.write(b" HTTP/1.0\r\nHost: ")
+ s.write(b" HTTP/2.0\r\nHost: ")

As I know, most major web servers and CDNs provide backward compatibility. Therefore there will be no harm by updating it.

sosi-deadeye · 2024-05-05

HTTP/2 is a binary protocol and uses HPACK for header compression. Changing this, require additional code to handle the protocol and header compression.

I also think the change to HTTP/1.1 may require some additional code, to handle the different connection types : close, keep-alive.

In addition, there are different Transfer-Endocings.

mattytrentini · 2024-05-06

It would be good to merge back some of the changes from the CircuitPython requests library (they've updated it since originally starting with the MicroPython implementation). In it, they respond with v1.1, though I'm not sure it 100% conforms to spec...

https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/main/adafruit_requests.py

CANDIDATE · PULL REQUEST

bundle-networking: Include urllib.urequest

closedby aveaoopened 2023-09-03updated 2023-09-13

urllib.urequest.urlopen provides an easy way to stream HTTP requests (which urequests can't do), and is seemingly generally recommended over urequests for anything that is more involved than doing a small HTTP request.

it's already included by default with esp8266 but somehow isn't with the rest of network-capable boards.

this PR changes it so that all boards that install bundle-networking get urllib.urequest too.

3 comments

jimmo · 2023-09-13

Thanks @aveao -- Unfortunately, although I agree it's definitely convenient to have as much stuff installed by default as possible, I don't think we're likely to make this change.

it's already included by default with esp8266 but somehow isn't with the rest of network-capable boards.

This is a bit of a historical accident with the esp8266, it includes too much stuff by default, I think at the time there was excitement about having the first network-capable board and providing features for the kickstarter. It's always easy to add stuff but then we can almost never remove it.

We've also made it easier to install packages since then (e.g. mip).

provides an easy way to stream HTTP requests (which urequests can't do)

Can you explain more what you mean by this? If you have a Response object from requests, you can use resp.raw to access the underlying socket. (This is true of upstream requests too). Not sure if that's what you meant by streaming though?

this PR changes it so that all boards that install bundle-networking get urllib.urequest too.

This is the problem -- it's a huge decision to add the additional firmware size to every single board, especially when there's already requests, when people who need it can just install it from mip.

andrewleech · 2023-09-13

For what it's worth my vote is not not include it by default; I regularly use network capable boards and my primary two use cases are:

Based on mqtt to communicate with home automation gear
With microdot to host a small web server interface to the hardware

I for one almost never call out to external http servers so never use requests.
On the rare occasion I do want to make a http request I'll use an asyncio library like uaiohttpclient

aveao · 2023-09-13

ah, didn't realize urequests had that functionality, it's not documented on micropython side and I tend to not rely too much on upstream python docs as micropython tends to implement a limited selection of it.

also: it's weird that urllib.urequest.urlopen exists at all, it's pretty much a two-line change to go from urllib.urequest.urlopen to using the .raw and it's not much smaller than urequests.

anyhow, closing this.

urequests still using HTTP/1.0

bundle-networking: Include urllib.urequest

Keyboard