<commit_before><commit_msg>Added documentation.<commit_after>\documentclass[11pt]{article}

\usepackage[latin1]{inputenc}
\usepackage{times}
\usepackage[T1]{fontenc}

\usepackage{underscore}
\usepackage{verbatim}
\usepackage{enumitem}

\usepackage[nofancy]{latex2man}

\usepackage[margin=1.25in,letterpaper]{geometry}

\begin{document}

\setDate{May 2013}
\setVersion{1.0}

\begin{Name}{3}{yajl_fort}{<NAME>}{}{The \texttt{yajl_fort} Module}
The \texttt{yajl_fort} module defines a Fortran interface to the YAJL library.
YAJL is an event-driven parser for JSON (JavaScript Object Notation) format
data streams written in C.  Despite its name, the JSON data format is language
independent and makes an excellent data interchange format. It is light weight,
flexible, easy for humans to read and write, all of which make it a superior
format to XML for many I/O uses.
\end{Name}

\section{Synopsis}
\begin{description}[style=nextline]\raggedright
\item[Derived Types]
  \texttt{fyajl_callbacks} (abstract),
  \texttt{fyajl_parser},
  \texttt{fyajl_status}.
\item[Parameters]
  \texttt{FYAJL_TERMINATE_PARSING},
  \texttt{FYAJL_CONTINUE_PARSING},
  \texttt{FYAJL_INTEGER_KIND},
  \texttt{FYAJL_REAL_KIND},
  \texttt{FYAJL_ALLOW_COMMENTS},
  \texttt{FYAJL_DONT_VALIDATE_STRINGS},
  \texttt{FYAJL_ALLOW_TRAILING_GARBAGE},
  \texttt{FYAJL_ALLOW_MULTIPLE_DOCUMENTS},
  \texttt{FYAJL_ALLOW_PARTIAL_DOCUMENT},
  \texttt{FYAJL_STATUS_OK},
  \texttt{FYAJL_STATUS_CLIENT_CANCELED},
  \texttt{FYAJL_STATUS_ERROR}.
\item[Procedures]
  \texttt{fyajl_get_error},
  \texttt{fyajl_status_to_string}.
\end{description}

\section{Prerequisites}
The \texttt{yajl_fort} module uses YAJL version 2.0.  The source code
for this library can be obtained from \URL{http://lloyd.github.io/yajl}.
The library is also available as a standard system package in all major
Linux distributions.

\section{Parser Callback Functions}
The JSON data language is quite simple.  It is built on two basic data
structures.  An \emph{array} is an ordered list of comma separated
\emph{values} enclosed in brackets (\texttt{[} and \texttt{]}).
An \emph{object} is an unordered list of comma separated
\emph{name}\texttt{:}\emph{value} pairs enclosed in braces (\texttt{\{}
and \texttt{\}}).  A \emph{name} is a string enclosed in double quotes,
and a \emph{value} is one of the following: a string in double quotes,
a number (integer or real), a boolean (\texttt{true} or
\texttt{false}), \texttt{null}, an \emph{object} or an \emph{array}.
Note how the data structures can be nested.
Whitespace is insignificant except in strings.
At the top level, a valid JSON document consists of a single \emph{object}.
See \URL{http://www.json.org} for a detailed description of the syntax.

The C language YAJL parser operates by calling application-defined callback
functions in response to the various events encountered while parsing the
input stream.  The callback functions communicate with each other through
a common, application-defined, context data struct, and a void pointer to
that data struct is passed to each of the callbacks.  In this Fortran
interface, this application-defined code/data is implemented by the abstract
derived type \texttt{fyajl_callbacks}:
\begin{verbatim}
type, abstract :: fyajl_callbacks
contains
  procedure(cb_no_args), deferred :: start_map
  procedure(cb_no_args), deferred :: end_map
  procedure(cb_string),  deferred :: map_key
  procedure(cb_no_args), deferred :: null_value
  procedure(cb_logical), deferred :: logical_value
  procedure(cb_integer), deferred :: integer_value
  procedure(cb_double),  deferred :: double_value
  procedure(cb_string),  deferred :: string_value
  procedure(cb_no_args), deferred :: start_array
  procedure(cb_no_args), deferred :: end_array
end type fyajl_callbacks
\end{verbatim}
Application code extends this type, adding the desired context data
components and providing concrete implementations of the callback functions.
The required interfaces for the deferred type bound callback functions are:
\begin{verbatim}
integer function cb_no_args (this)
  class(fyajl_callbacks) :: this
integer function cb_integer (this, value)
  class(fyajl_callbacks) :: this
  integer(FYAJL_INTEGER_KIND), intent(in) :: value
integer function cb_double (this, value)
  class(fyajl_callbacks) :: this
  real(FYAJL_REAL_KIND), intent(in) :: value
integer function cb_logical (this, value)
  class(fyajl_callbacks) :: this
  logical, intent(in) :: value
integer function cb_string (this, value)
  class(fyajl_callbacks) :: this
  character(*,kind=c_char), intent(in) :: value
\end{verbatim}
The return value of each function must be either
\texttt{FYAJL_CONTINUE_PARSING} or \texttt{FYAJL_TERMINATE_PARSING}.
The latter return value will trigger the parser to terminate with an error.
The kind parameters for integer and real values, \texttt{FYAJL_INTEGER_KIND}
and \texttt{FYAJL_REAL_KIND}, correspond to C's long long and double, and
are dictated by the YAJL library.  The callbacks are used as follows.
\begin{description}\setlength{\itemsep}{0pt}
\item[\texttt{start_map}]
  is called when a \texttt{\{} is parsed, marking the start of an \emph{object}.
\item[\texttt{end_map}]
  is called when a \texttt{\}} is parsed, marking the end of an \emph{object}.
\item[\texttt{start_array}]
  is called when a \texttt{[} is parsed, marking the start of an \emph{array}.
\item[\texttt{end_array}]
  is called when a \texttt{]} is parsed, marking the end of an \emph{array}.
\item[\texttt{map_key}]
  is called when the name of a \emph{name}\texttt{:}\emph{value} pair is
  parsed, and the parsed name string is passed to the function.
\item[\texttt{integer_value}]
  is called when an integer \emph{value} is parsed.
  The value is passed to the function.
\item[\texttt{double_value}]
  is called when a real \emph{value} is parsed, and
  the value is passed to the function.
\item[\texttt{string_value}]
  is called when a string \emph{value} is parsed, and
  the value is passed to the function.
\item[\texttt{logical_value}]
  is called when the \emph{value} token \texttt{true} or \texttt{false} is
  parsed, and the corresponding Fortran logical value is passed to the function.
\item[\texttt{null_value}]
  is called when the \emph{value} token \texttt{null} is parsed.
\end{description}

\section{Parsing}
The derived type \texttt{fyajl_parser} and its type bound procedures
implement the JSON parser.  
\begin{verbatim}
type, extends(fyajl_callbacks) :: my_callbacks
  ...
end type
type(my_callbacks), target :: callbacks
...
\end{verbatim}
As described in the previous section, an application-specific extension of
the abstract type must be defined and an instance (here \texttt{callbacks})
of that extension initialized if it contains data components.
\begin{verbatim}
type(fyajl_parser) :: parser
call parser%init (callbacks)
\end{verbatim}
Initializes the parser, configuring it to use the specified callbacks.
Note that proper finalization of the parser object occurs automatically
when the object is deallocated or goes out of scope.  Finalization of
the callback object is the responsibility of the application.
\begin{verbatim}
call parser%parse (buffer, stat)
  character(kind=c_char), intent(in) :: buffer(:)
  type(fyajl_status), intent(out) :: stat
\end{verbatim}
Parses the chunk of JSON text passed in the \texttt{buffer} array. Parsing
can be done incrementally via repeated calls to this function, passing
successive chunks from the JSON input stream in the \texttt{buffer} array.
The status of the parser is returned in \texttt{stat}; see Error Handling.
\begin{verbatim}
call parser%parse_complete (stat)
  type(fyajl_status), intent(out) :: stat
\end{verbatim}
Parses any remaining internally buffered JSON text.  Because the parser is
stream-based it needs an explicit end-of-input signal to force it to parse
content at the end of the stream that sometimes may exist.  The status of
parser is returned in \texttt{stat}; see Error Handling.
\begin{verbatim}
parser%bytes_consumed()
\end{verbatim}
Returns the number of characters consumed from the buffer in the last
call to parse.

\subsection{Error Handling}
The \texttt{parse} and \texttt{parse_complete} methods return a
\texttt{type(fyajl_status)} status value, which equals one of the
following module parameters:
\begin{description}[style=nextline]\setlength{\itemsep}{0pt}
\item[FYAJL_STATUS_OK]
  No error.
\item[FYAJL_STATUS_ERROR]
  A parsing error was encountered;
  use \texttt{fyajl_get_error} to get information about it.
\item[FYAJL_STATUS_CLIENT_CANCELLED]
  One of the callback procedures returned \texttt{FYAJL_TERMINATE_PARSING}.
\end{description}
The comparison operators \texttt{==} and \texttt{/=} are defined for
\texttt{type(fyajl_status)} values.

Several additional functions (not type bound) are provided for error handling.
\begin{verbatim}
fyajl_get_error(parser, verbose, buffer)
  logical, intent(in) :: verbose
  character(kind=c_char), intent(in) :: buffer(:)
\end{verbatim}
Returns a character string describing the the error encountered by the parser.
If \texttt{verbose} is true, the message will include the portion of the input
stream where the error occurred together with an arrow pointing to the specific
character.  The \texttt{buffer} array should contain the chunk of JSON input
passed in the last call to \texttt{parse}. 
\begin{verbatim}
fyajl_status_to_string(code)
  type(fyajl_status), intent(in) :: code
\end{verbatim}
Returns a character string describing the specified status value.

\subsection{Parsing Options}
The parser supports several options provided by the YAJL library.
They are set and unset using the \texttt{set_option} and
\texttt{unset_option} methods after the parser has been initialized:
\begin{verbatim}
call parser%set_option (option)
call parser%unset_option (option)
\end{verbatim}
where \texttt{option} is one of the following module parameters.  The
default for all is unset.
\begin{description}[style=nextline]\setlength{\itemsep}{0pt}
\item[FYAJL_ALLOW_COMMENTS]
  JSON does not provide for comments.  Setting this option causes the
  parser to ignore java\-script style comments in the input stream.  This
  includes single-line comments that begin with \texttt{//} and continue
  to the end of the line.  This is a very useful extention to the JSON
  standard, but one that is not supported by many JSON parsers.
\item[FYAJL_DONT_VALIDATE_STRINGS]
  By default, the parser verifies that all strings are valid UTF-8.  This
  option disables this check, resulting in slightly faster parsing.
\item[FYAJL_ALLOW_TRAILING_GARBAGE]
  By default, \texttt{parse_complete} verifies that the entire input text has
  been consumed and will return an error if it finds otherwise.  Setting this
  option will disable this check.  This can be useful when parsing a input
  stream that contains more than one JSON document.  In such scenarios, the
  \texttt{bytes_consumed} method is useful for identifying the trailing
  portion of the input text for subsequent handling.
\item[FYAJL_ALLOW_MULTIPLE_DOCUMENTS]
  An instance of a parser normally expects that the input stream consists
  of a single JSON document.  Setting this option changes that behavior and
  allows an instance to parse an input stream containing multiple documents
  that are separated by whitespace.
\item[FYAJL_ALLOW_PARTIAL_DOCUMENT]
  By default, \texttt{parse_complete} verifies that the top level \emph{object}
  is complete; that is, the closing \texttt{\}} has been parsed.  If it finds
  otherwise it returns an error.  Setting this option disables this check.
\end{description}

\section{Bugs}
Bug reports and improvement suggestions should be directed to
\Email{<EMAIL>}

\LatexManEnd

\end{document}

{'fraction_non_alphanumeric': 0.08481800358025744, 'fraction_numerical': 0.0017901287187793028, 'mean_word_length': 5.230483271375465, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2529476', 'n_tokens_mistral': 3613, 'n_tokens_neox': 3459, 'n_words': 1343}