<commit_before>================================================
Readable Serial Object Notation  (RSON)
================================================

Version 0.03

.. contents::

Introduction
=============

RSON is a superset of the JSON data format.

RSON syntax is relaxed compared to JSON:

- Any valid JSON file is also a valid RSON file as long as
  it is encoded in UTF-8 or ASCII.  (External conversion functions
  could detect other files and pass UTF-8 or Unicode to the RSON
  decoder.)
- Comments are allowed with a leading # (in any column)
  if the comment is the only non-whitespace on the line.
- String quoting is not required unless the string contains
  any RSON special characters.
- Integer formats include hex, binary, and octal, and embedded
  underscores are allowed.
- Trailing commas are allowed inside **[]** and **{}**.
- Dictionary keys do not have to be strings.
- Python-style triple-quoted (**"""**) strings are supported for
  practically arbitrary embedded data.

In addition to the relaxed JSON syntax, RSON supports Python-style
indentation to create structures.  When inside any **[]** or **{}**
pair, the syntax is JSON syntax, with the enhancements described
above.  *Outside* any **[]** or **{}** pairs, RSON indented syntax
is used to describe the structure.

RSON indentation controls the nesting levels of dicts or lists.  As
with Python, spaces or tabs may be used but it is inadvisable to mix them.

Examples
============

Any valid JSON is valid RSON, so no pure JSON examples are given.

The simple array::

    1
    2
    3

.. page::

The two-dimensional array::

    [1, 2, 3]
    [4, 5, 6]
    [7, 8, 9]
    [a, b, c]
    [d, e, f]

Nested dictionaries::

    George:
        age:     42
        height:  6'2"
        weight:  232
    Sam:
        age:     13
        height:  5'1"
        weight:  103.5
    Morrie:
        age:     0.7
        height:  0.5"
        weight:  1.5oz

Arbitrary strings::

    George:
        age:     42
        height:  6'2"
        weight:  232
        comment: """
      Python style triple quotes leave data intact"""


    Sam:
        age:     13
        height:  5'1"
        weight:  103.5

    Morrie:
        age:     0.7
        height:  0.5"
        weight:  1.5oz
        comment = Equals strings have some trimming.

          Morrie is a goldfish.


.. page::

Windows-registry-style keys::

    evilness:high:starter
        cmd1 = cd /
        cmd2 = rm -Rf *

    evilness:high:more subtle
        cmd1 = cd /etc
        cmd2 = rm *pass*

    evilness:low:silly:
        cmd1 = cat < /dev/random > /dev/null


Nested arrays and dicts::

    1
    2
    []
       a
       b
       c
    {}
       z: x
       w: []
           m


Syntax rules
===============

Inside any **[]** or **{}** pairs, the syntax is extended JSON, as described in
the introduction.

Outside these delimiters, the following rules apply:

- A group of lines at the same indentation level (which continues past any lines
  which are indented more, but stops on the first line which is indented less),
  either comprises:

      * A single scalar (must be all on one line, or a triple quoted, or equal
        prefixed string)
      * A list (JSON array)
      * A dict (JSON object)

  The determination is made by examining the first two lines of tokens.  It
  is an error to mix key:value pairs with single array elements.

- An equal sign is used to denote the start of a string.  The string extends to
  the end of the line, and through any more-indented lines until a line at the
  same or lesser indentation as the line containing the equal sign is found.

  The string is trimmed both horizontally and vertically.

- A colon is used to separate a key from its value.  This can be done repeatedly,
  and the resultant dictionaries are merged.

- A colon will be assumed if not present between a key and an equal sign
  denoting the start of a value string.

- A colon will be assumed if not present between a key and group of lines that
  is more indented than the line the key is on.  (But see complex structure syntax,
  below, for exceptions.)

Complex syntax structure
-------------------------

Sometimes, you need an array that has some array elements and some non-array elements,
e.g. *[1, 2, 3, [a, b, c]]*.

To support this, whenever a text file line is more deeply indented than the previous
line, RSON examines the last value on the previous line.  If that value is an *empty*
dict or array, then the indented lines are added into that structure.  In all other
cases, the previous line is considered to be a key (or set of keys for a nested
dict structure), and the indented lines comprise the value for a dictionary.

Implementation
==================

Currently, there is a Python RSON decoder available at `http://code.google.com/p/rson/`__.

The decoder is made available as a single file download, and works like the JSON decoder:

    from rson import loads
    result = loads(source)

The decoder is built in a manner that facilitates replacing parts of the decoder, as
necessary.  For example, you could easily use the Decimal class instead of floats.  It
supports many of the same options as json/simplejson.  In fact, a `subclassed version of
the decoder`__ passes the simplejson regression.

__ http://code.google.com/p/rson/
__ http://code.google.com/p/rson/source/browse/trunk/py2x/testbench/cjsontest.py

Future plans
==================

RSON is a work in progress.  An encoder will be built at some point, but most changes
will be driven by user feedback to the rson-discuss google group.
<commit_msg>Added copyright.<commit_after>================================================
Readable Serial Object Notation  (RSON)
================================================

Version 0.03

Copyright (c) 2010 <NAME>

.. contents::

Introduction
=============

RSON is a superset of the JSON data format.

RSON syntax is relaxed compared to JSON:

- Any valid JSON file is also a valid RSON file as long as
  it is encoded in UTF-8 or ASCII.  (External conversion functions
  could detect other files and pass UTF-8 or Unicode to the RSON
  decoder.)
- Comments are allowed with a leading # (in any column)
  if the comment is the only non-whitespace on the line.
- String quoting is not required unless the string contains
  any RSON special characters.
- Integer formats include hex, binary, and octal, and embedded
  underscores are allowed.
- Trailing commas are allowed inside **[]** and **{}**.
- Dictionary keys do not have to be strings.
- Python-style triple-quoted (**"""**) strings are supported for
  practically arbitrary embedded data.

In addition to the relaxed JSON syntax, RSON supports Python-style
indentation to create structures.  When inside any **[]** or **{}**
pair, the syntax is JSON syntax, with the enhancements described
above.  *Outside* any **[]** or **{}** pairs, RSON indented syntax
is used to describe the structure.

RSON indentation controls the nesting levels of dicts or lists.  As
with Python, spaces or tabs may be used but it is inadvisable to mix them.

Examples
============

Any valid JSON is valid RSON, so no pure JSON examples are given.

The simple array::

    1
    2
    3

.. page::

The two-dimensional array::

    [1, 2, 3]
    [4, 5, 6]
    [7, 8, 9]
    [a, b, c]
    [d, e, f]

Nested dictionaries::

    George:
        age:     42
        height:  6'2"
        weight:  232
    Sam:
        age:     13
        height:  5'1"
        weight:  103.5
    Morrie:
        age:     0.7
        height:  0.5"
        weight:  1.5oz

Arbitrary strings::

    George:
        age:     42
        height:  6'2"
        weight:  232
        comment: """
      Python style triple quotes leave data intact"""


    Sam:
        age:     13
        height:  5'1"
        weight:  103.5

    Morrie:
        age:     0.7
        height:  0.5"
        weight:  1.5oz
        comment = Equals strings have some trimming.

          Morrie is a goldfish.


.. page::

Windows-registry-style keys::

    evilness:high:starter
        cmd1 = cd /
        cmd2 = rm -Rf *

    evilness:high:more subtle
        cmd1 = cd /etc
        cmd2 = rm *pass*

    evilness:low:silly:
        cmd1 = cat < /dev/random > /dev/null


Nested arrays and dicts::

    1
    2
    []
       a
       b
       c
    {}
       z: x
       w: []
           m


Syntax rules
===============

Inside any **[]** or **{}** pairs, the syntax is extended JSON, as described in
the introduction.

Outside these delimiters, the following rules apply:

- A group of lines at the same indentation level (which continues past any lines
  which are indented more, but stops on the first line which is indented less),
  either comprises:

      * A single scalar (must be all on one line, or a triple quoted, or equal
        prefixed string)
      * A list (JSON array)
      * A dict (JSON object)

  The determination is made by examining the first two lines of tokens.  It
  is an error to mix key:value pairs with single array elements.

- An equal sign is used to denote the start of a string.  The string extends to
  the end of the line, and through any more-indented lines until a line at the
  same or lesser indentation as the line containing the equal sign is found.

  The string is trimmed both horizontally and vertically.

- A colon is used to separate a key from its value.  This can be done repeatedly,
  and the resultant dictionaries are merged.

- A colon will be assumed if not present between a key and an equal sign
  denoting the start of a value string.

- A colon will be assumed if not present between a key and group of lines that
  is more indented than the line the key is on.  (But see complex structure syntax,
  below, for exceptions.)

Complex syntax structure
-------------------------

Sometimes, you need an array that has some array elements and some non-array elements,
e.g. *[1, 2, 3, [a, b, c]]*.

To support this, whenever a text file line is more deeply indented than the previous
line, RSON examines the last value on the previous line.  If that value is an *empty*
dict or array, then the indented lines are added into that structure.  In all other
cases, the previous line is considered to be a key (or set of keys for a nested
dict structure), and the indented lines comprise the value for a dictionary.

Implementation
==================

Currently, there is a Python RSON decoder available at `http://code.google.com/p/rson/`__.

The decoder is made available as a single file download, and works like the JSON decoder:

    from rson import loads
    result = loads(source)

The decoder is built in a manner that facilitates replacing parts of the decoder, as
necessary.  For example, you could easily use the Decimal class instead of floats.  It
supports many of the same options as json/simplejson.  In fact, a `subclassed version of
the decoder`__ passes the simplejson regression.

__ http://code.google.com/p/rson/
__ http://code.google.com/p/rson/source/browse/trunk/py2x/testbench/cjsontest.py

Future plans
==================

RSON is a work in progress.  An encoder will be built at some point, but most changes
will be driven by user feedback to the rson-discuss google group.

{'fraction_non_alphanumeric': 0.10031988626266217, 'fraction_numerical': 0.012795450506486582, 'mean_word_length': 2.754169446297532, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5953583', 'n_tokens_mistral': 3276, 'n_tokens_neox': 3087, 'n_words': 1567}