<commit_before>// bdlpcre_regex.h                                                    -*-C++-*-
#ifndef INCLUDED_BDLPCRE_REGEX
#define INCLUDED_BDLPCRE_REGEX

#ifndef INCLUDED_BSLS_IDENT
#include <bsls_ident.h>
#endif
BSLS_IDENT("$Id$ $CSID$")

//@PURPOSE: Provide a mechanism for regular expression pattern matching.
//
//@CLASSES:
//  bdlpcre::RegEx: mechanism for compiling and matching regular expressions
//
//@SEE_ALSO: http://www.pcre.org/
//
//@DESCRIPTION: This component provides a mechanism, 'bdlpcre::RegEx', for
// compiling (or "preparing") regular expressions, and subsequently matching
// subject strings against a prepared expression.  The regular expressions
// supported by this component correspond approximately with Perl 5.10.  See
// the appendix entitled "Perl Compatibility" below for more information.
//
// Upon construction, a 'bdlpcre::RegEx' object is initially not associated
// with a regular expression.  A regular expression pattern is compiled for use
// by the object using the 'prepare' method.  Subject strings may then be
// matched against the prepared pattern using the set of overloaded 'match'
// methods.
//
// The component provides the following groups of 'match' overloads:
//: 1  The first group of 'match' overloads simply returns 0 if a given subject
//:    string matches the prepared regular expression, and returns a non-zero
//:    value otherwise.
//:
//: 2  The second group of 'match' overloads returns the substring of the
//:    subject that was matched, ether as a 'bslstl::StringRef', or as a
//:    'bsl::pair<size_t, size_t>' holding the (offset, length) pair.
//:
//: 3  The third group of 'match' overloads returns a vector of either
//:    'bslstl::StringRef' or 'bsl::pair<size_t, size_t>' holding the matched
//:    substrings.  The first element of the vector indicate the substring of
//:    the subject that matched the entire pattern.  Subsequent elements
//:    indicate the substrings of the subject that matched respective
//:    sub-patterns.
//
///"Prepared" State
///----------------
// A 'bdlpcre::RegEx' object must first be prepared with a valid regular
// expression before attempting to match subject strings.  We say that an
// instance of 'bdlpcre::RegEx' is in the "prepared" state if the object holds
// a valid regular expression, in which case calls to the overloaded 'match'
// methods of that instance are valid.  Otherwise, the object is in the
// "unprepared" state.  Upon construction, an 'bdlpcre::RegEx' object is in the
// "unprepared" state.  A successful call to the 'prepare' method puts the
// object into the "prepared" state.  The 'clear' method, as well as an
// unsuccessful call to 'prepare', puts the object into the "unprepared" state.
// The 'isPrepared' accessor may be used to determine whether an object is
// prepared.
//
///Prepare-Time Flags
///------------------
// A set of flags may be optionally supplied to the 'prepare' method to affect
// specific pattern matching behavior.  The flags recognized by 'prepare' are
// defined in an enumeration declared within the 'bdlpcre::RegEx'.  The
// following describes these flags and their effects.
//
///Case-Insensitive Matching
///- - - - - - - - - - - - -
// If 'RegEx::k_FLAG_CASELESS' is included in the flags supplied to 'prepare',
// then letters in the regular expression pattern supplied to 'prepare' match
// both lower- and upper-case letters in subject strings subsequently supplied
// to 'match'.  This is equivalent to Perl's '/i' option, and can be turned off
// within a pattern by a '(?i)' option setting.
//
///Multi-Line Matching
///- - - - - - - - - -
// By default, a subject string supplied to 'match' is treated as consisting of
// a single line of characters (even if it actually contains '\n' characters).
// The start-of-line meta-character '^' matches only at the beginning of the
// string, and the end-of-line meta-character '$' matches only at the end of
// the string (or before a terminating '\n', if present).  This matches the
// behavior of Perl.
//
// If 'RegEx::k_FLAG_MULTILINE' is included in the flags supplied to 'prepare',
// then start-of-line and end-of-line meta-characters match immediately
// following or immediately before any '\n' characters in subject strings
// supplied to 'match', respectively (as well as at the very start and end of
// subject strings).  This is equivalent to Perl's '/m' option, and can be
// turned off within a pattern by a '(?m)' option setting.  If there are no
// '\n' characters in the subject string, or if there are no occurrences of '^'
// or '$' in the prepared pattern, then including 'k_FLAG_MULTILINE' has no
// effect.
//
///UTF-8 Support
///- - - - - - -
// If 'RegEx::k_FLAG_UTF8' is included in the flags supplied to 'prepare', then
// the regular expression pattern supplied to 'prepare', as well as the subject
// strings subsequently supplied to 'match', are interpreted as strings of
// UTF-8 characters instead of strings of ASCII characters.  The behavior of
// 'match' methods is undefined if 'pattern()' was prepared with 'k_FLAG_UTF8',
// but 'subject' is not a valid UTF-8 string.
//
///Dot Matches All
///- - - - - - - -
// If 'RegEx::k_FLAG_DOTMATCHESALL' is included in the flags supplied to
// 'prepare', then a dot metacharacter in the pattern matches a character of
// any value, including one that indicates a newline.  However, it only ever
// matches one character, even if newlines are encoded as '\r\n'.  If
// 'k_FLAG_DOTMATCHESALL' is not used to prepare a regular expression, a dot
// metacharacter will *not* match a newline; hence, patterns expected to match
// across lines will fail to do so.  This flag is equivalent to Perl's '/s'
// option, and can be changed within a pattern by a '(?s)' option setting.  A
// negative class such as '[^a]' always matches newline characters, independent
// of the setting of this option.
//
///JIT Compiling optimization
///----------------------------
// Just-in-time compiling is a heavyweight optimization that can greatly speed
// up pattern matching. However, it comes at the cost of extra processing
// before the match is performed, so it is of most benefit when the same
// pattern is going to be matched many times. This does not necessarily mean
// many calls of a matching function; if the pattern is not anchored, matching
// attempts may take place many times at various positions in the subject, even
// for a single call. Therefore, if the subject string is very long, it may
// still pay to use JIT even for one-off matches.
//
// If 'RegEx::k_FLAG_JIT' is included in the flags supplied to 'prepare', then
// all following matchings will be performed with JIT optimization.  To turn
// off JIT optimization support, regular expression should be prepared again,
// without using 'k_FLAG_JIT'.
//
// The table below demonstrates the benefit of the 'match' method with JIT
// optimization over the basic 'match' method:
//..
//  Legend
//  ------
//  'SIMPLE_PATTERN':
//      Pattern - (abc)*
//      Subject -  XXXabcZZZ
//
//  'EMAIL_PATTERN':
//      Pattern - [A-Za-z0-9._-]+@[[A-Za-z0-9.-]+
//      Subject - <EMAIL>
//
//  'IP_ADDRESS_PATTERN':
//      Pattern - (?:[0-9]{1,3}\.){3}[0-9]{1,3}
//      Subject - 255.255.255.255
//
//   Each pattern/subject returns 1 match.  Measurements (in seconds) have been
//   tallied over the 100000 matchings in a row.
//
//  +--------------------+---------------------+---------------------+
//  | Pattern            | 'match' without-JIT |  'match' using-JIT  |
//  +====================+=====================+=====================+
//  | SIMPLE_PATTERN     |        0.0161       |    0.0052 (~3.1x)   |
//  +--------------------+---------------------+---------------------+
//  | EMAIL_PATTERN      |        0.0222       |    0.0086 (~2.6x)   |
//  +--------------------+---------------------+---------------------+
//  | IP_ADDRESS_PATTERN |        0.0331       |    0.0062 (~5.3x)   |
//  +--------------------+---------------------+---------------------+
//..
// Note that the test was run on Linux / Intel Xeon CPU (3.47GHz, 64-bit),
// compiled with gcc-4.8.2 in optimized mode.
//
///Usage
///-----
// The following snippets of code illustrate using this component to extract
// the text of the "Subject:" field from an Internet e-mail message (RFC822).
// The following 'parseSubject' function accepts an RFC822-compliant message of
// a specified length and returns the text of the message's subject in the
// 'result' "out" parameter:
//..
//  int parseSubject(bsl::string *result,
//                   const char  *message,
//                   bsl::size_t  messageLength)
//      // Parse the specified 'message' of the specified 'messageLength' for
//      // the "Subject:" field of 'message'.  Return 0 on success and load the
//      // specified 'result' with the text of the subject of 'message'; return
//      // a non-zero value otherwise with no effect on 'result'.
//  {
//..
// The following is the regular expression that will be used to find the
// subject text of 'message'.  The "?P<subjectText>" syntax, borrowed from
// Python, allows us later to refer to a particular matched sub-pattern (i.e.,
// the text between the ':' and the '\r' in the "Subject:" field of the header)
// by the name "subjectText":
//..<commit_msg>Code review comments have been fixed.<commit_after>// bdlpcre_regex.h                                                    -*-C++-*-
#ifndef INCLUDED_BDLPCRE_REGEX
#define INCLUDED_BDLPCRE_REGEX

#ifndef INCLUDED_BSLS_IDENT
#include <bsls_ident.h>
#endif
BSLS_IDENT("$Id$ $CSID$")

//@PURPOSE: Provide a mechanism for regular expression pattern matching.
//
//@CLASSES:
//  bdlpcre::RegEx: mechanism for compiling and matching regular expressions
//
//@SEE_ALSO: http://www.pcre.org/
//
//@DESCRIPTION: This component provides a mechanism, 'bdlpcre::RegEx', for
// compiling (or "preparing") regular expressions, and subsequently matching
// subject strings against a prepared expression.  The regular expressions
// supported by this component correspond approximately with Perl 5.10.  See
// the appendix entitled "Perl Compatibility" below for more information.
//
// Upon construction, a 'bdlpcre::RegEx' object is initially not associated
// with a regular expression.  A regular expression pattern is compiled for use
// by the object using the 'prepare' method.  Subject strings may then be
// matched against the prepared pattern using the set of overloaded 'match'
// methods.
//
// The component provides the following groups of 'match' overloads:
//: 1  The first group of 'match' overloads simply returns 0 if a given subject
//:    string matches the prepared regular expression, and returns a non-zero
//:    value otherwise.
//:
//: 2  The second group of 'match' overloads returns the substring of the
//:    subject that was matched, ether as a 'bslstl::StringRef', or as a
//:    'bsl::pair<size_t, size_t>' holding the (offset, length) pair.
//:
//: 3  The third group of 'match' overloads returns a vector of either
//:    'bslstl::StringRef' or 'bsl::pair<size_t, size_t>' holding the matched
//:    substrings.  The first element of the vector indicate the substring of
//:    the subject that matched the entire pattern.  Subsequent elements
//:    indicate the substrings of the subject that matched respective
//:    sub-patterns.
//
///"Prepared" State
///----------------
// A 'bdlpcre::RegEx' object must first be prepared with a valid regular
// expression before attempting to match subject strings.  We say that an
// instance of 'bdlpcre::RegEx' is in the "prepared" state if the object holds
// a valid regular expression, in which case calls to the overloaded 'match'
// methods of that instance are valid.  Otherwise, the object is in the
// "unprepared" state.  Upon construction, an 'bdlpcre::RegEx' object is in the
// "unprepared" state.  A successful call to the 'prepare' method puts the
// object into the "prepared" state.  The 'clear' method, as well as an
// unsuccessful call to 'prepare', puts the object into the "unprepared" state.
// The 'isPrepared' accessor may be used to determine whether an object is
// prepared.
//
///Prepare-Time Flags
///------------------
// A set of flags may be optionally supplied to the 'prepare' method to affect
// specific pattern matching behavior.  The flags recognized by 'prepare' are
// defined in an enumeration declared within the 'bdlpcre::RegEx'.  The
// following describes these flags and their effects.
//
///Case-Insensitive Matching
///- - - - - - - - - - - - -
// If 'RegEx::k_FLAG_CASELESS' is included in the flags supplied to 'prepare',
// then letters in the regular expression pattern supplied to 'prepare' match
// both lower- and upper-case letters in subject strings subsequently supplied
// to 'match'.  This is equivalent to Perl's '/i' option, and can be turned off
// within a pattern by a '(?i)' option setting.
//
///Multi-Line Matching
///- - - - - - - - - -
// By default, a subject string supplied to 'match' is treated as consisting of
// a single line of characters (even if it actually contains '\n' characters).
// The start-of-line meta-character '^' matches only at the beginning of the
// string, and the end-of-line meta-character '$' matches only at the end of
// the string (or before a terminating '\n', if present).  This matches the
// behavior of Perl.
//
// If 'RegEx::k_FLAG_MULTILINE' is included in the flags supplied to 'prepare',
// then start-of-line and end-of-line meta-characters match immediately
// following or immediately before any '\n' characters in subject strings
// supplied to 'match', respectively (as well as at the very start and end of
// subject strings).  This is equivalent to Perl's '/m' option, and can be
// turned off within a pattern by a '(?m)' option setting.  If there are no
// '\n' characters in the subject string, or if there are no occurrences of '^'
// or '$' in the prepared pattern, then including 'k_FLAG_MULTILINE' has no
// effect.
//
///UTF-8 Support
///- - - - - - -
// If 'RegEx::k_FLAG_UTF8' is included in the flags supplied to 'prepare', then
// the regular expression pattern supplied to 'prepare', as well as the subject
// strings subsequently supplied to 'match', are interpreted as strings of
// UTF-8 characters instead of strings of ASCII characters.  The behavior of
// 'match' methods is undefined if 'pattern()' was prepared with 'k_FLAG_UTF8',
// but 'subject' is not a valid UTF-8 string.
//
///Dot Matches All
///- - - - - - - -
// If 'RegEx::k_FLAG_DOTMATCHESALL' is included in the flags supplied to
// 'prepare', then a dot metacharacter in the pattern matches a character of
// any value, including one that indicates a newline.  However, it only ever
// matches one character, even if newlines are encoded as '\r\n'.  If
// 'k_FLAG_DOTMATCHESALL' is not used to prepare a regular expression, a dot
// metacharacter will *not* match a newline; hence, patterns expected to match
// across lines will fail to do so.  This flag is equivalent to Perl's '/s'
// option, and can be changed within a pattern by a '(?s)' option setting.  A
// negative class such as '[^a]' always matches newline characters, independent
// of the setting of this option.
//
///JIT Compiling optimization
///--------------------------
// Just-in-time compiling is a heavyweight optimization that can greatly speed
// up pattern matching. However, it comes at the cost of extra processing
// before the match is performed, so it is of most benefit when the same
// pattern is going to be matched many times. This does not necessarily mean
// many calls of a matching function; if the pattern is not anchored, matching
// attempts may take place many times at various positions in the subject, even
// for a single call. Therefore, if the subject string is very long, it may
// still pay to use JIT even for one-off matches.
//
// If 'RegEx::k_FLAG_JIT' is included in the flags supplied to 'prepare', then
// all following matchings will be performed with JIT optimization.  To turn
// off JIT optimization support, regular expression should be prepared again,
// without using 'k_FLAG_JIT'.
//
// The table below demonstrates the benefit of the 'match' method with JIT
// optimization over the basic 'match' method:
//..
//  Legend
//  ------
//  'SIMPLE_PATTERN':
//      Pattern - (abc)*
//      Subject -  XXXabcZZZ
//
//  'EMAIL_PATTERN':
//      Pattern - [A-Za-z0-9._-]+@[[A-Za-z0-9.-]+
//      Subject - <EMAIL>
//
//  'IP_ADDRESS_PATTERN':
//      Pattern - (?:[0-9]{1,3}\.){3}[0-9]{1,3}
//      Subject - 255.255.255.255
//
//   Each pattern/subject returns 1 match.  Measurements (in seconds) have been
//   tallied over the 100000 matchings in a row.
//
//  +--------------------+---------------------+---------------------+
//  | Pattern            | 'match' without-JIT |  'match' using-JIT  |
//  +====================+=====================+=====================+
//  | SIMPLE_PATTERN     |        0.0161       |    0.0052 (~3.1x)   |
//  +--------------------+---------------------+---------------------+
//  | EMAIL_PATTERN      |        0.0222       |    0.0086 (~2.6x)   |
//  +--------------------+---------------------+---------------------+
//  | IP_ADDRESS_PATTERN |        0.0331       |    0.0062 (~5.3x)   |
//  +--------------------+---------------------+---------------------+
//..
// Note that the test was run on Linux / Intel Xeon CPU (3.47GHz, 64-bit),
// compiled with gcc-4.8.2 in optimized mode.
//
///Note on memory allocation exceptions
///------------------------------------
// PCRE2 library supports memory allocation/deallocation functions supplied by
// the client.  'bdlpcre_regex' provides wrappers around bslma allocators which
// are called from the context of the PCRE2 library (C linkage).  Any
// exceptions thrown during memory allocation are catched by the wrapper
// functions and are not propagated to the PCRE2 library.
//
///Usage
///-----
// The following snippets of code illustrate using this component to extract
// the text of the "Subject:" field from an Internet e-mail message (RFC822).
// The following 'parseSubject' function accepts an RFC822-compliant message of
// a specified length and returns the text of the message's subject in the
// 'result' "out" parameter:
//..
//  int parseSubject(bsl::string *result,
//                   const char  *message,
//                   bsl::size_t  messageLength)
//      // Parse the specified 'message' of the specified 'messageLength' for
//      // the "Subject:" field of 'message'.  Return 0 on success and load the
//      // specified 'result' with the text of the subject of 'message'; return
//      // a non-zero value otherwise with no effect on 'result'.
//  {
//..
// The following is the regular expression that will be used to find the
// subject text of 'message'.  The "?P<subjectText>" syntax, borrowed from
// Python, allows us later to refer to a particular matched sub-pattern (i.e.,
// the text between the ':' and the '\r' in the "Subject:" field of the header)
// by the name "subjectText":
//..
{'fraction_non_alphanumeric': 0.1515183191344797, 'fraction_numerical': 0.01008728375058799, 'mean_word_length': 2.6204351939451276, 'pattern_counts': {'":': 2, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5811923', 'n_tokens_mistral': 5397, 'n_tokens_neox': 5175, 'n_words': 2370}