<commit_before>/*
    Copyright (c) 2015
    vurtun <<EMAIL>>
    MIT licence

    JSON
    -----------------------
    This file provides both the interface and implementation for a JSON parser.
    The parser does not allocate memory or create a complex DOM object.
    Instead it reads either token or token pairs. Each token represents one child
    of the current node. Therefore you only reach the first depth of the DOM-Tree
    in your first parsing pass. Deeper levels are reached by recursivly parsing
    each previously read token.

    It is important to note that while every valid JSON tree should be parsable
    without problems, there is no guarantee that any invalid input is detected as
    such, since the parser has only minimal validation.

    USAGE
    ------------------------
    To instantiate the implementation part of library you have to define
        #define JSON_IMPLEMENTATION
    in *ONE* of your source files, before #include'ing this file. This expands out the
    actual implementation into that C/C++ file.

    To make the implementation private to the file that generates the implementation use
    #define JSON_STATIC

    First you have to initialize the library over the function `json_init` which
    does not take any arguments and sets up look up tables for the ANSI C version.

    To begin the parsing procces the json parser iterator needs to be created by using
    the `json_begin` function which takes a UTF-8 string + length in bytes
    and returns the newly created iterator.

    Parsing function API (based around an iterator)
        json_begin()   -- creates a parsing lexer iterator from a json string and a string length
        json_read()    -- parses a token and returns the iterator to the next token
        json_parse()   -- parses a token pair and returns the iterator to the next token
        json_num()     -- converts a previously parsed token into a number
        json_type()    -- returns the type of a previously read token

    Utility function API (not neccessary for the actual parsing process)
        json_cpy()     -- copies a prev read token into a user provided buffer and returns the length
        json_cmp()     -- compares a prev read token with a user provided string
        json_deq()     -- dequotes a string in a prev read token

*/

#if 0<commit_msg>fixed typo<commit_after>/*
    Copyright (c) 2015
    vurtun <<EMAIL>>
    MIT licence

    JSON
    -----------------------
    This file provides both the interface and implementation for a JSON parser.
    The parser does not allocate memory or create a complex DOM object.
    Instead it reads either token or token pairs. Each token represents one child
    of the current node. Therefore you only reach the first depth of the DOM-Tree
    in your first parsing pass. Deeper levels are reached by recursivly parsing
    each previously read token.

    It is important to note that while every valid JSON tree should be parsable
    without problems, there is no guarantee that any invalid input is detected as
    such, since the parser has only minimal validation.

    USAGE
    ------------------------
    To instantiate the implementation part of the library you have to define
        #define JSON_IMPLEMENTATION
    in *ONE* of your source files, before #include'ing this file. This expands out the
    actual implementation into that C/C++ file.

    To make the implementation private to the file that generates the implementation use
    #define JSON_STATIC

    First you have to initialize the library over the function `json_init` which
    does not take any arguments and sets up look up tables for the ANSI C version.

    To begin the parsing procces the json parser iterator needs to be created by using
    the `json_begin` function which takes a UTF-8 string + length in bytes
    and returns the newly created iterator.

    Parsing function API (based around an iterator)
        json_begin()   -- creates a parsing lexer iterator from a json string and a string length
        json_read()    -- parses a token and returns the iterator to the next token
        json_parse()   -- parses a token pair and returns the iterator to the next token
        json_num()     -- converts a previously parsed token into a number
        json_type()    -- returns the type of a previously read token

    Utility function API (not neccessary for the actual parsing process)
        json_cpy()     -- copies a prev read token into a user provided buffer and returns the length
        json_cmp()     -- compares a prev read token with a user provided string
        json_deq()     -- dequotes a string in a prev read token

*/

#if 0
{'fraction_non_alphanumeric': 0.05412316215640316, 'fraction_numerical': 0.0025569997869166845, 'mean_word_length': 2.652918287937743, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2412016', 'n_tokens_mistral': 1126, 'n_tokens_neox': 1073, 'n_words': 663}