QUERY · ISSUE

68c28174 breaks utf8 decode with ignore

openby ryannathansopened 2017-12-05updated 2026-01-16

bugpy-coreunicode

b'\xff\xfe'.decode('utf8', 'ignore')

used to work but now produces UnicodeError exception due to commit https://github.com/micropython/micropython/commit/68c28174d0e0ec3f6b1461aea3a0b6a1b84610bb

Tested on STM32

On CPython this still works as expected:

>>> b'\xff\xfe'.decode('utf8', 'ignore')
''

CANDIDATE · ISSUE

Crash printing exception detail when source code is not valid UTF-8

openby jepleropened 2025-08-07updated 2026-01-20

bugunicode

Port, board and/or hardware

unix port, coverage build, x86_64 linux

MicroPython version

MicroPython v1.26.0-preview.524.g255d74b5a8 on 2025-08-06; linux [GCC 12.2.0] version

Reproduction

# Slight changes (like removing the derived exception type) move the misbehavior
# around. For instance, in my local build, not having this triggers
# 'NotImplementedError: opcode' instead.
class Dummy(BaseException):
    pass

# Smuggle invalid UTF-8 string into decompress_error_text_maybe
# This invalid UTF-8 string acts matches the test MP_IS_COMPRESSED_ROM_STRING
# This can also happen if the input file is not a valid UTF-8 file.
b = eval(b"'\xff" + b"\xfe" * 4096 + b"'")
try:
    raise BaseException(b)
except BaseException as good:
    print(type(good), good.args[0])

Expected behaviour

CPython fails the eval() with SyntaxError: (unicode error) 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte.

Observed behaviour

The invalid utf-8 string can be successfully created. When fetching the exception's args property, a crash occurs inside of mp_decompress_rom_string (which should never have been called). The call occurs because the first byte of the invalid UTF-8 string is \xff, the marker for compressed ROM strings.

Additional Information

Found via fuzzer, manually minimized.

Code of Conduct

Yes, I agree