<commit_before>\section{Our technique}
\label{sec-our-technique}

\subsection{Parsing clauses}
\label{sec-our-technique-parsing-clauses}

In order to parse \texttt{loop} clauses, we use a simplified version
of a parsing technique known as \emph{combinator parsing}
\cite{Wadler:1985:RFL:5280.5288}.  It is simplified in that we do not
need the full backtracking power of this technique.  It is fairly easy
to structure the individual clause and sub-clause parsers so that a
parser either succeeds or fails, simply because the \texttt{loop}
grammar is unambiguous at this level.  When a parser succeeds, the
result is correct and unambiguous, and when it fails, other parsers
are tried in sequence.  Obtaining the correct result requires the
individual parsers to be tried in a particular order, which is a
disadvantage of the simplification.  As discussed in
\refSec{sec-using-external-parser-framework}, we plan to avoid this
restriction by using an external parsing framework.

With this parsing technique, client code defines \emph{elementary
  parsers} that are then combined using combinators such as
\emph{alternative} and \emph{sequence}.  The resulting parser code is
\emph{modular} in that individual parsers do not have to be listed in
one single place.  For the \texttt{loop} clauses, this modularity
means that each type of clause can be defined in a different module.

In our parsing framework, an individual parser is an ordinary
\commonlisp{} function that takes a list of \commonlisp{} expressions
and that returns three values:

\begin{enumerate}
\item A generalized Boolean%
  \footnote{The term \emph{generalized Boolean} is part of the
    \commonlisp{} \cite{ansi:common:lisp} terminology.  It means any
    value where \texttt{nil} stands for \emph{false} and any other
    value stands for \emph{true}.}
  indicating whether the parse succeeded.
\item The result of the parse.  If the parse does not succeed, then
  this value is unspecified.
\item A list of the tokens that remain after the parse.  If the
  parse does not succeed, then this list contains the original
  list of tokens passed as an argument.
\end{enumerate}

Consider the following example:

{\small\begin{verbatim}
       (define-parser arithmetic-up-1-parser
         (consecutive
          (lambda (var type-spec from to by)
            (make-instance 'for-as-arithmetic-up
              :order '(from to by)
              :var-spec var
              :type-spec type-spec
              :start-form from
              :end-form (cdr to)
              :by-form by
              :termination-test (car to)))
          'simple-var-parser
          'optional-type-spec-parser
          (alternative 'from-parser
                       'upfrom-parser)
          (alternative 'to-parser
                       'upto-parser
                       'below-parser)
          'by-parser))
\end{verbatim}}
<commit_msg>Do not indent a small paragraph after a code example.<commit_after>\section{Our technique}
\label{sec-our-technique}

\subsection{Parsing clauses}
\label{sec-our-technique-parsing-clauses}

In order to parse \texttt{loop} clauses, we use a simplified version
of a parsing technique known as \emph{combinator parsing}
\cite{Wadler:1985:RFL:5280.5288}.  It is simplified in that we do not
need the full backtracking power of this technique.  It is fairly easy
to structure the individual clause and sub-clause parsers so that a
parser either succeeds or fails, simply because the \texttt{loop}
grammar is unambiguous at this level.  When a parser succeeds, the
result is correct and unambiguous, and when it fails, other parsers
are tried in sequence.  Obtaining the correct result requires the
individual parsers to be tried in a particular order, which is a
disadvantage of the simplification.  As discussed in
\refSec{sec-using-external-parser-framework}, we plan to avoid this
restriction by using an external parsing framework.

With this parsing technique, client code defines \emph{elementary
  parsers} that are then combined using combinators such as
\emph{alternative} and \emph{sequence}.  The resulting parser code is
\emph{modular} in that individual parsers do not have to be listed in
one single place.  For the \texttt{loop} clauses, this modularity
means that each type of clause can be defined in a different module.

In our parsing framework, an individual parser is an ordinary
\commonlisp{} function that takes a list of \commonlisp{} expressions
and that returns three values:

\begin{enumerate}
\item A generalized Boolean%
  \footnote{The term \emph{generalized Boolean} is part of the
    \commonlisp{} \cite{ansi:common:lisp} terminology.  It means any
    value where \texttt{nil} stands for \emph{false} and any other
    value stands for \emph{true}.}
  indicating whether the parse succeeded.
\item The result of the parse.  If the parse does not succeed, then
  this value is unspecified.
\item A list of the tokens that remain after the parse.  If the
  parse does not succeed, then this list contains the original
  list of tokens passed as an argument.
\end{enumerate}

Consider the following example:

{\small\begin{verbatim}
       (define-parser arithmetic-up-1-parser
         (consecutive
          (lambda (var type-spec from to by)
            (make-instance 'for-as-arithmetic-up
              :order '(from to by)
              :var-spec var
              :type-spec type-spec
              :start-form from
              :end-form (cdr to)
              :by-form by
              :termination-test (car to)))
          'simple-var-parser
          'optional-type-spec-parser
          (alternative 'from-parser
                       'upfrom-parser)
          (alternative 'to-parser
                       'upto-parser
                       'below-parser)
          'by-parser))
\end{verbatim}}

\noindent
{'fraction_non_alphanumeric': 0.06942544459644323, 'fraction_numerical': 0.004445964432284542, 'mean_word_length': 2.9979494190020506, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4414857', 'n_tokens_mistral': 1637, 'n_tokens_neox': 1549, 'n_words': 696}