<commit_before>\section{\module{re} ---
         Regular expression operations}
\declaremodule{standard}{re}
\moduleauthor{<NAME>}{<EMAIL>}
\sectionauthor{<NAME>}{<EMAIL>}


\modulesynopsis{Regular expression search and match operations with a
                Perl-style expression syntax.}


This module provides regular expression matching operations similar to
those found in Perl.  Regular expression pattern strings may not
contain null bytes, but can specify the null byte using the
\code{\e\var{number}} notation.  Both patterns and strings to be
searched can be Unicode strings as well as 8-bit strings.  The
\module{re} module is always available.

Regular expressions use the backslash character (\character{\e}) to
indicate special forms or to allow special characters to be used
without invoking their special meaning.  This collides with Python's
usage of the same character for the same purpose in string literals;
for example, to match a literal backslash, one might have to write
\code{'\e\e\e\e'} as the pattern string, because the regular expression
must be \samp{\e\e}, and each backslash must be expressed as
\samp{\e\e} inside a regular Python string literal.

The solution is to use Python's raw string notation for regular
expression patterns; backslashes are not handled in any special way in
a string literal prefixed with \character{r}.  So \code{r"\e n"} is a
two-character string containing \character{\e} and \character{n},
while \code{"\e n"} is a one-character string containing a newline.
Usually patterns will be expressed in Python code using this raw
string notation.

\begin{seealso}
  \seetitle{Mastering Regular Expressions}{Book on regular expressions
            by <NAME>, published by O'Reilly.  The second 
            edition of the book no longer covers Python at all, 
            but the first edition covered writing good regular expression
            patterns in great detail.}
\end{seealso}


\subsection{Regular Expression Syntax \label{re-syntax}}

A regular expression (or RE) specifies a set of strings that matches
it; the functions in this module let you check if a particular string
matches a given regular expression (or if a given regular expression
matches a particular string, which comes down to the same thing).

Regular expressions can be concatenated to form new regular
expressions; if \emph{A} and \emph{B} are both regular expressions,
then \emph{AB} is also a regular expression.  If a string \emph{p}
matches A and another string \emph{q} matches B, the string \emph{pq}
will match AB if \emph{A} and \emph{B} do no specify boundary
conditions that are no longer satisfied by \emph{pq}.  Thus, complex
expressions can easily be constructed from simpler primitive
expressions like the ones described here.  For details of the theory
and implementation of regular expressions, consult the Friedl book
referenced above, or almost any textbook about compiler construction.

A brief explanation of the format of regular expressions follows.  For
further information and a gentler presentation, consult the Regular
Expression HOWTO, accessible from \url{http://www.python.org/doc/howto/}.

Regular expressions can contain both special and ordinary characters.
Most ordinary characters, like \character{A}, \character{a}, or
\character{0}, are the simplest regular expressions; they simply match
themselves.  You can concatenate ordinary characters, so \regexp{last}
matches the string \code{'last'}.  (In the rest of this section, we'll
write RE's in \regexp{this special style}, usually without quotes, and
strings to be matched \code{'in single quotes'}.)

Some characters, like \character{|} or \character{(}, are special.
Special characters either stand for classes of ordinary characters, or
affect how the regular expressions around them are interpreted.

The special characters are:

\begin{list}{}{\leftmargin 0.7in \labelwidth 0.65in}

\item[\character{.}] (Dot.)  In the default mode, this matches any
character except a newline.  If the \constant{DOTALL} flag has been
specified, this matches any character including a newline.<commit_msg>SF bug #803842:  Wrong description of regexp concatenation<commit_after>\section{\module{re} ---
         Regular expression operations}
\declaremodule{standard}{re}
\moduleauthor{<NAME>}{<EMAIL>}
\sectionauthor{<NAME>}{<EMAIL>}


\modulesynopsis{Regular expression search and match operations with a
                Perl-style expression syntax.}


This module provides regular expression matching operations similar to
those found in Perl.  Regular expression pattern strings may not
contain null bytes, but can specify the null byte using the
\code{\e\var{number}} notation.  Both patterns and strings to be
searched can be Unicode strings as well as 8-bit strings.  The
\module{re} module is always available.

Regular expressions use the backslash character (\character{\e}) to
indicate special forms or to allow special characters to be used
without invoking their special meaning.  This collides with Python's
usage of the same character for the same purpose in string literals;
for example, to match a literal backslash, one might have to write
\code{'\e\e\e\e'} as the pattern string, because the regular expression
must be \samp{\e\e}, and each backslash must be expressed as
\samp{\e\e} inside a regular Python string literal.

The solution is to use Python's raw string notation for regular
expression patterns; backslashes are not handled in any special way in
a string literal prefixed with \character{r}.  So \code{r"\e n"} is a
two-character string containing \character{\e} and \character{n},
while \code{"\e n"} is a one-character string containing a newline.
Usually patterns will be expressed in Python code using this raw
string notation.

\begin{seealso}
  \seetitle{Mastering Regular Expressions}{Book on regular expressions
            by <NAME>, published by O'Reilly.  The second 
            edition of the book no longer covers Python at all, 
            but the first edition covered writing good regular expression
            patterns in great detail.}
\end{seealso}


\subsection{Regular Expression Syntax \label{re-syntax}}

A regular expression (or RE) specifies a set of strings that matches
it; the functions in this module let you check if a particular string
matches a given regular expression (or if a given regular expression
matches a particular string, which comes down to the same thing).

Regular expressions can be concatenated to form new regular
expressions; if \emph{A} and \emph{B} are both regular expressions,
then \emph{AB} is also a regular expression.  In general, if a string
\emph{p} matches \emph{A} and another string \emph{q} matches \emph{B},
the string \emph{pq} will match AB.  This holds unless \emph{A} or
\emph{B} contain low precedence operations; boundary conditions between
\emph{A} and \emph{B}; or have numbered group references.  Thus, complex
expressions can easily be constructed from simpler primitive
expressions like the ones described here.  For details of the theory
and implementation of regular expressions, consult the Friedl book
referenced above, or almost any textbook about compiler construction.

A brief explanation of the format of regular expressions follows.  For
further information and a gentler presentation, consult the Regular
Expression HOWTO, accessible from \url{http://www.python.org/doc/howto/}.

Regular expressions can contain both special and ordinary characters.
Most ordinary characters, like \character{A}, \character{a}, or
\character{0}, are the simplest regular expressions; they simply match
themselves.  You can concatenate ordinary characters, so \regexp{last}
matches the string \code{'last'}.  (In the rest of this section, we'll
write RE's in \regexp{this special style}, usually without quotes, and
strings to be matched \code{'in single quotes'}.)

Some characters, like \character{|} or \character{(}, are special.
Special characters either stand for classes of ordinary characters, or
affect how the regular expressions around them are interpreted.

The special characters are:

\begin{list}{}{\leftmargin 0.7in \labelwidth 0.65in}

\item[\character{.}] (Dot.)  In the default mode, this matches any
character except a newline.  If the \constant{DOTALL} flag has been
specified, this matches any character including a newline.
{'fraction_non_alphanumeric': 0.0704461188853008, 'fraction_numerical': 0.002392058366224136, 'mean_word_length': 4.412297734627832, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1629349', 'n_tokens_mistral': 2142, 'n_tokens_neox': 2062, 'n_words': 1107}