QUERY · ISSUE

urequests: TypeError: object with buffer protocol required

openby GM-Script-Writer-62850opened 2022-10-14updated 2022-10-22

r=urequest.post("http://www.example.com",headers={"Content-Length":19},data='{"hello":"world"}')
print(r.status_code,r.content)

Seems that that 19 is the issue and it needs to be a string, should the user of the lib be expected to know this or should this:

        for k in headers:
            s.write(k)
            s.write(b": ")
            s.write(headers[k])
            s.write(b"\r\n")

be changed to this:

        for k in headers:
            s.write(k)
            s.write(b": ")
            s.write(str(headers[k]))
            s.write(b"\r\n")

on a side note, why does this not happen with header={}
https://forums.raspberrypi.com/viewtopic.php?t=341355

12 comments

jimmo · 2022-10-14

I agree that's confusing.

However, the "Content-Length" header is set automatically (a few lines below the bit you linked to), there's no need to include it in the headers dict.

on a side note, why does this not happen with header={}
https://forums.raspberrypi.com/viewtopic.php?t=341355

I'm not sure what you're asking here sorry...

GM-Script-Writer-62850 · 2022-10-14

i am sure there are other headers that will have a numeric value, that was the 1st header i though about

on the side note, notice how headers is defined as {} here
Why does that work as expected where as when i did that my variable eg json was still populated with old data on the next function call, my solution was to use a boolean and if it is false change that value to {} inside the function

jimmo · 2022-10-17

@GM-Script-Writer-62850 I think this is what you're referring to
https://stackoverflow.com/questions/1132941/least-astonishment-and-the-mutable-default-argument

GM-Script-Writer-62850 · 2022-10-18

yes, i was wondering why this is not the behavior for the header parameter in the request function, i saw that and was going to report it as a bug, but the issue does happen somehow

jimmo · 2022-10-18

@GM-Script-Writer-62850 Oh I see. In the specific case where it's called with default headers and auth is set, then it will modify the default headers dict. This was introduced in https://github.com/micropython/micropython-lib/commit/e7e8eff86b219bec2d6e19baeb8f0d85f7b77e47

This should be done like Content-Length etc and handled later with direct writes to the socket rather than modifying the headers dict.

GM-Script-Writer-62850 · 2022-10-18

i was wondering how/why does header always starts as a empty dict in this lib, but not when i do it
if i put print(header) it is either empty or has what i just defined in my call to get/post

jimmo · 2022-10-18

i was wondering how/why does header always starts as a empty dict in this lib, but not when i do it if i put print(header) it is either empty or has what i just defined in my call to get/post

@GM-Script-Writer-62850 Could you share a full example of what you mean? I'm not quite sure what you're asking or what you're saying isn't working? Or are you asking why something does work when you think it shouldn't?

I assume by header you mean headers ?

GM-Script-Writer-62850 · 2022-10-18

yes, i have a cat on my lap and should have zoomed in to see the s

GM-Script-Writer-62850 · 2022-10-18

if i call post("http://www.example.com",data,headers={"DNT":1})
then if i call post("http://www.example.com",data)
when post calls requests what do you think headers is? {"DNT":1} or {}
the answer is {}, but when I make a function with a variable set as {} (eg def test(var1,var2={})) it's content persist between calls (var2), for what reason is the request function able to get around that annoying behavior with its headers variable

jimmo · 2022-10-18

@GM-Script-Writer-62850 you might need to post more of your test function.

def test(var1, var2={}):
    print(var1, var2)

test('a')
test('b', {'b': 2})
test('c')

prints (as expected)

a {}
b {'b': 2}
c {}

(Other than the specific case of when auth is enabled that I highlighted above, this is how urequests works -- it never modifies the headers dict).

Maybe you are doing this (i.e. modifying the argument):

def test(var1, var2={}):
    var2[var1] = True
    print(var2)

test('a')
test('b', {'b': 2})
test('c')

{'a': True}
{'b': True}
{'a': True, 'c': True}

This is explained in detail in the stackoverflow link above.

sosi-deadeye · 2022-10-22

It's a well-known problem, and there is PEP671.

In short terms:

def foo(data={}):
    ...

should be ...

def foo(data=None):
    data = data or {}

and maybe in near future

def foo(data=>{}):
    ...

With the cost of 2 pointers.

GM-Script-Writer-62850 · 2022-10-22

Thanks jimmo, that behavior makes it even more confusing (only keeping data if it is set in the function but dropping it when it is supplied)

sorry for skimming the stackoverflow stuff and being too blind to see it, how i know why headers is not affected

this side question when on way longer than i expected, maybe this report should be purged of the side question

CANDIDATE · ISSUE

urequests - header dict not being passed through correctly?

closedby eymasopened 2018-10-22updated 2018-10-22

I'm at a loss regarding the following:

import urequests as requests
import ujson

method = "get"
url = "https://api.example.com"
headers = {
    "authorization": "Bearer" + token,
    "accept": "application/json"
    }
result = requests.request(method, url, headers)
result_obj = ujson.loads(result.json())

Looking at the request function within urequests.py, I believe I do have to pass a dictionary onto headers. However; the following is returned when I do so: TypeError: object with buffer protocol required

If I pass on the headers within the function call. I get a Bad Request error and no other way to read the response object to determine the exact cause.

4 comments

SpotlightKid · 2018-10-22

I think all header keys and values need to be byte strings, i.e. bytes not str.

eymas · 2018-10-22

Having added a quick encoder the error still repeats, unfortunately:
header = ujson.dumps(headers).encode('utf-8')

SpotlightKid · 2018-10-22

The headers must be a dict, not a JSON string. However, I was incorerct, the dict keys and values can be strings. Your error was trying to JSON-decode an already decode response. response.json() returns the decoded JSON response body, i.e. usually a dict or a list. You don't need to decode this manually.

Example:

import urequests as requests

url = "https://httpbin.org/get"
token = "Abracadabra"
headers = {
    "authorization": "Bearer" + token,
    "accept": "application/json"
}
result = requests.get(url, headers=headers)
print(result.text)
print(result.json())

Output:

{
  "args": {}, 
  "headers": {
    "Accept": "application/json", 
    "Authorization": "BearerAbracadabra", 
    "Connection": "close", 
    "Host": "httpbin.org"
  }, 
  "origin": "87.78.173.172", 
  "url": "https://httpbin.org/get"
}

{'url': 'https://httpbin.org/get', 'headers': {'Host': 'httpbin.org', 'Accept': 'application/json', 'Authorization': 'BearerAbracadabra', 'Connection': 'close'}, 'args': {}, 'origin': '87.78.173.172'}

Note that I use the requests.get function. If you use requests.request, the HTTP method needs to be in uppercase, e.g. GETnot get.

eymas · 2018-10-22

Decoding the decoded response was indeed it. I must've thought result.json() would have returned a JSON object that needed to be turned into a dict; which was already done by the .json() function.
As for the method declaration, I agree that it's better to use the get/post function lest your own formatting betrays yourself.

Thanks for the help!

urequests: TypeError: object with buffer protocol required

urequests - header dict not being passed through correctly?

Keyboard