QUERY · ISSUE

json lib module, syntax error

openby permaBoxopened 2023-11-21updated 2024-09-03

needs-info

The json lib module seems to have an issue. That was not observed in 2018.

To reproduce the error take:
r = requests.get('https://aa.usno.navy.mil/api/rstt/oneday?date=2023-11-21 &coords=48.441832,9.890139 &tz=2').json()

Generated error message:
File "requests/init.py", line 33, in json
ValueError: syntax error in JSON

4 comments

ricksorensen · 2023-11-22

This returns status_code==400 on my ESP32C3 , here is REPL :

>>> r = requests.get('https://aa.usno.navy.mil/api/rstt/oneday?date=2023-11-21 &coords=48.441832,9.890139 &tz=2')
>>> r.json()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/__init__.py", line 33, in json
ValueError: syntax error in JSON
>>> r.text
'<html>\r\n<head><title>400 Bad Request</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'
>>> r.status_code
400
>>>

The request works fine on my Linux box - using ipython- and returns a good json string. The text is not JSON. Maybe the difficulty is in requests module?

On the ESP32C3 micropython:

>>> r.headers
{'Server': 'nginx', 'Connection': 'close', 'Date': 'Wed, 22 Nov 2023 17:16:01 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Content-Length': '150'}
>>>

On my linux box:

In [5]: r.headers
Out[5]: {'Server': 'nginx', 'Date': 'Wed, 22 Nov 2023 17:15:20 GMT', 'Content-Type': 'application/json', 'Content-Length': '1274', 'Connection': 'keep-alive', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': "default-src 'self';", 'Access-Control-Allow-Origin': '*'}

ricksorensen · 2023-11-22

If I remove the space between date and &coords and remove the space abetween coordinates and &tz it works:

>>> r=requests.get('https://aa.usno.navy.mil/api/rstt/oneday?date=2023-11-21&coords=48.441832,9.890139&tz=2')
>>> r.json()
{'apiversion': '4.0.1', 'properties': {'data': {'year': 2023, 'month': 11, 'fracillum': '61%', 'closestphase': {'phase': 'First Quarter', 'month': 11, 'year': 2023, 'time': '12:50', 'day': 20}, 'isdst': False, 'day': 21, 'day_of_week': 'Tuesday', 'label': None, 'sundata': [{'phen': 'Begin Civil Twilight', 'time': '08:02'}, {'phen': 'Rise', 'time': '08:37'}, {'phen': 'Upper Transit', 'time': '13:06'}, {'phen': 'Set', 'time': '17:35'}, {'phen': 'End Civil Twilight', 'time': '18:10'}], 'tz': 2.0, 'moondata': [{'phen': 'Set', 'time': '00:52'}, {'phen': 'Rise', 'time': '15:09'}, {'phen': 'Upper Transit', 'time': '20:34'}], 'curphase': 'Waxing Gibbous'}}, 'type': 'Feature', 'geometry': {'coordinates': [9.890139, 48.44183], 'type': 'Point'}}
>>>

permaBox · 2023-11-23

It works ... sometimes, means it is not stable.
I often get this error:
...
File "requests/init.py", line 180, in get
File "requests/init.py", line 76, in request
OSError: -202

jonnor · 2024-08-25

In order to be able to debug the issue, you need to provide examples of valid JSON that the library does not parse.

CANDIDATE · ISSUE

the json library in python-stdlib requires some constants in re module which is not available in the embeded re module of micropython

closedby versaloonopened 2023-10-01updated 2023-10-07

/ # mpy
MicroPython v1.20.0-348-g24a6e951e-dirty on 2023-08-13; linux [] version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import mip
>>> mip.install('json')
Installing json (latest) from https://micropython.org/pi/v2 to /usr/micropython/lib
Copying: /usr/micropython/lib/json/__init__.mpy
Copying: /usr/micropython/lib/json/encoder.mpy
Copying: /usr/micropython/lib/json/scanner.mpy
Copying: /usr/micropython/lib/json/tool.mpy
Copying: /usr/micropython/lib/json/decoder.mpy
Done
>>> import json
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "json/__init__.py", line 114, in <module>
  File "json/decoder.py", line 6, in <module>
  File "json/scanner.py", line 13, in <module>
AttributeError: 'module' object has no attribute 'VERBOSE

4 comments

andrewleech · 2023-10-01

This micropython-lib json library is quite old, I'm guessing it was written against the ffi re module (unix only version).

Are there features in this json that the built-in one is missing?

versaloon · 2023-10-01

This micropython-lib json library is quite old, I'm guessing it was written against the ffi re module (unix only version).

Are there features in this json that the built-in one is missing?

the json library depends on re library, and need constants below, which is not in embedded re module implementation:
re.VERBOSE
re.MULTILINE
re.DOTALL

andrewleech · 2023-10-02

I meant are you trying to use the mip installed json to do something that doesn't work on the builtin import json?

versaloon · 2023-10-07

No, I just do a test.
I have switched to embedded json.

json lib module, syntax error

the json library in python-stdlib requires some constants in re module which is not available in the embeded re module of micropython

Keyboard