<story><title>Parsing JSON Is a Minefield (2016)</title><url>http://seriot.ch/projects/parsing_json.html</url></story><parent_chain><item><author>q3k</author><text>Some other fun facts about JSON, its mainstream implementations and using it reliably:&lt;p&gt;1. json.dump(s) in Python by default emits non-standards-compliant JSON, ie. will happily serialize NaN&amp;#x2F;Inf&amp;#x2F;-Inf. You want to set allow_nan=False to be compliant. Otherwise this _will_ annoy someone who has to consume your shoddy pseudo-JSON from a standards-compliant library.&lt;p&gt;2. JSON allows for duplicate&amp;#x2F;repeated keys, and allows for the parser to basically do anything when that happens. Do you know how the parser implementation you use handles this? Are you sure there&amp;#x27;s no differences between that implementation and other implementations used in your system (eg. between execution and validation)? What about other undefined behaviour, like permitted number ranges?&lt;p&gt;3. Do you pass around user-provided JSON data accross your system? How many JSON nesting levels does your implementation allow? What happens if it&amp;#x27;s exceeded? What happens if different parts of your processing system have different limits? What about other unspecified limits like serialized size, string length?&lt;p&gt;My general opinion is that it&amp;#x27;s extremely hard to reliably use JSON as an interchange format reliably when multiple systems and&amp;#x2F;or parser implementations are involved. It&amp;#x27;s based on a set of underdefined specifications that leaves critical behaviour undefined, effectively making it impossible to have 100% interoperable implementations. It doesn&amp;#x27;t help that one of the mainstream implementations (in Python) is just non-compliant by default.&lt;p&gt;I highly encourage any greenfield project to look into well designed and better specified alternatives.</text></item></parent_chain><comment><author>makeitdouble</author><text>It’s a pyramid.&lt;p&gt;At the bottom you have CSV which is popular beyond belief, and has no real specification, with common cases wildly handled differently across libraries.&lt;p&gt;In the middle you have JSON which isn’t 100% interoperable, but goes 98% of the way.&lt;p&gt;And you have XML and protobufs at the top tip, who have strong mechanisms available for interoperability but at an operational cost that rarely justifies the upgrade from JSON.&lt;p&gt;I suspect it will take a lot more that “well designed” and “better specified” to justify moving away from JSON as the default stepup from chaotic CSV like formats.</text></comment>
<story><title>Parsing JSON Is a Minefield (2016)</title><url>http://seriot.ch/projects/parsing_json.html</url></story><parent_chain><item><author>q3k</author><text>Some other fun facts about JSON, its mainstream implementations and using it reliably:&lt;p&gt;1. json.dump(s) in Python by default emits non-standards-compliant JSON, ie. will happily serialize NaN&amp;#x2F;Inf&amp;#x2F;-Inf. You want to set allow_nan=False to be compliant. Otherwise this _will_ annoy someone who has to consume your shoddy pseudo-JSON from a standards-compliant library.&lt;p&gt;2. JSON allows for duplicate&amp;#x2F;repeated keys, and allows for the parser to basically do anything when that happens. Do you know how the parser implementation you use handles this? Are you sure there&amp;#x27;s no differences between that implementation and other implementations used in your system (eg. between execution and validation)? What about other undefined behaviour, like permitted number ranges?&lt;p&gt;3. Do you pass around user-provided JSON data accross your system? How many JSON nesting levels does your implementation allow? What happens if it&amp;#x27;s exceeded? What happens if different parts of your processing system have different limits? What about other unspecified limits like serialized size, string length?&lt;p&gt;My general opinion is that it&amp;#x27;s extremely hard to reliably use JSON as an interchange format reliably when multiple systems and&amp;#x2F;or parser implementations are involved. It&amp;#x27;s based on a set of underdefined specifications that leaves critical behaviour undefined, effectively making it impossible to have 100% interoperable implementations. It doesn&amp;#x27;t help that one of the mainstream implementations (in Python) is just non-compliant by default.&lt;p&gt;I highly encourage any greenfield project to look into well designed and better specified alternatives.</text></item></parent_chain><comment><author>jerf</author><text>&amp;quot;My general opinion is that it&amp;#x27;s extremely hard to reliably use JSON as an interchange format reliably when multiple systems and&amp;#x2F;or parser implementations are involved.&amp;quot;&lt;p&gt;I suspect one of the reasons that JSON has been so successful is precisely this fuzziness, though. Every language can do something a little slightly different and it&amp;#x27;ll work at first when you send it to somebody else. You get up and off the ground really quickly, and can fix up issues as you go.&lt;p&gt;If you try to specify something with a stronger schema right off the bat, I find a number of problems immediately emerge that tend to slow the process down. It may be foreign to programmers on HN who have embraced a strong static type mindset, or dynamic programmers who have learned the hard way that sometimes you need to be more precise about your types, but there&amp;#x27;s still a lot of programmers out there who will wonder why you&amp;#x27;re asking them whether this is an int or a float is relevant. I came in to work this morning to an alert system telling me that a field that a particular system has been sending as an integer for a couple of months now over many thousands of pushes, &amp;quot;number of bytes transferred&amp;quot;, is apparently capable of being a float once every several thousand times for some reason. There&amp;#x27;s a lot of programmers who will send a string, or a null, or maybe a float, or maybe it&amp;#x27;s always an integer, and deeply don&amp;#x27;t understand why you care what it&amp;#x27;s getting serialized as.&lt;p&gt;And that&amp;#x27;s just an example of some of the issues, not a complete list. Trying to specify with some stronger system moves a lot of these issues up front.&lt;p&gt;(If your organization has internalized that&amp;#x27;s just how it has to be done, great! I bet you encountered a lot of these bumps on the way, though.)&lt;p&gt;This isn&amp;#x27;t a celebration of JSON per se... this is really a rather cynical take. I don&amp;#x27;t know that we need to type everything to the n&amp;#x27;th degree in the first meeting, but &amp;quot;why can&amp;#x27;t we just let our dynamically-typed language send this number as a string sometimes?&amp;quot; is definitely something I&amp;#x27;ve had to discuss. (Now, I don&amp;#x27;t get a lot of resistance per se, but it&amp;#x27;s something I have to bring up.) I&amp;#x27;m not presenting this as a good thing, but as a theory that JSON&amp;#x27;s success is actually in large part &lt;i&gt;because&lt;/i&gt; of its loosey-gooseyness, and not despite it, regardless of how we may feel about it.</text></comment>