JSON module does not throw exception when object is not serialisable

[peterhinch]

This came up in http://forum.micropython.org/viewtopic.php?f=6&t=953&p=7551#p7551
cPython throws an exception on an attempt to serialise a bytes object (regardless of object contents):

>>> a = bytes(x for x in range(256))
>>> z = json.dumps(a).encode('utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.4/json/encoder.py", line 192, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.4/json/encoder.py", line 250, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.4/json/encoder.py", line 173, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff' is not JSON serializable

MicroPython does not throw an exception but does not work: the restored object is of type str and its contents differs from that saved:

>>> a = bytes(x for x in range(256))
>>> z = json.dumps(a)
>>> x = json.loads(z)
>>> len(x)
192
>>> len(a)
256
>>> type(a)
<class 'bytes'>
>>> type(x)
<class 'str'>
>>> 

[dpgeorge]

This was a design decision. It's undefined behaviour if you serialise something that can't be serialised. I guess the bigger issue is: do we need to provide a mechanism to serialise a larger set of types?

[peterhinch]

pickle works on everything I've tried so far, and it is the classic Python solution. The version in the MicroPython library is very micro, and doubtless less efficient than cPython's C coded version but it does work.

It seems a lot to ask of someone new to Python or JSON to know what datatypes are valid.

[pfalcon]

This was a design decision. It's undefined behaviour if you serialise something that can't be serialised.

+1. It's also should be pointed out that MicroPython provides "ujson" module, which can't be compared one-to-one to CPython. Users may write actual "json" module which would work as they desire.
But as a convenience to user, some ports automatically map "json" to "ujson" if no real "json" is available, and that may be a source of confusion.

[pfalcon]

I guess the bigger issue is: do we need to provide a mechanism to serialise a larger set of types?

I lately had a case when I wanted extensibility. But turned out, I want extensibility which even CPython doesn't provide: I wanted to get instances of particular namedtuple type for a JSON arrays. So, I skipped following that further.

[jonnor]

Serializing a bytes object still does not seem to raise and exception - just silently create invalid JSON.

import json
a = bytes(x for x in range(256))
z = json.dumps(a)
print(z)

"\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008\t\n\u000b\u000c\r\u000e\u000f\u0010\u0011\u0012\u0013\u0014\u0015\u0016\u0017\u0018\u0019\u001a\u001b\u001c\u001d\u001e\u001f !"#$%&'()*+,-./0123456789:;<=>?@abcdefghijklmnopqrstuvwxyz[\]^_`abcdefghijklmnopqrstuvwxyz{|}~��������������������������������������������������������������������������������������������������������������������������������"