<commit_before>// bdlb_tokenizer.h                                                   -*-C++-*-
#ifndef INCLUDED_BDLB_TOKENIZER
#define INCLUDED_BDLB_TOKENIZER

#ifndef INCLUDED_BSLS_IDENT
#include <bsls_ident.h>
#endif
BSLS_IDENT("$Id: $")

//@PURPOSE: Provide access to user-described tokens via string references.
//
//@CLASSES:
//  bdlb::Tokenizer: lexer for tokens defined via hard and/or soft delimiters
//  bdlb::TokenizerIterator: input iterator for delimited tokens in a string
//
//@SEE_ALSO: bslstl_stringref
//
//@DESCRIPTION: This component defines a mechanism, 'bdlb::Tokenizer', that
// provides non-destructive sequential (read-only) access to tokens in a given
// input string as characterized by two disjoint sets of user-specified
// delimiter characters, each of which is supplied at construction via either a
// 'const bslstl::StringRef&' or (for efficiency, when only the leading
// characters of the input string may need to be parsed) a 'const char *'.
// Note that each character (including '\0') that is not explicitly designated
// as a delimiter character is assumed to be *token* character.
//
///Soft versus Hard Delimiters
///---------------------------
// The tokenizer recognizes two distinct kinds of delimiter characters, *soft*
// and *hard*.
//
// A *soft* *delimiter* is a maximal (non-empty) sequence of soft-delimiter
// characters.  Soft delimiters, typically whitespace characters, are used to
// separate (rather than terminate) tokens, and thus never result in an empty
// token.
//
// A *hard* *delimiter* is a maximal (non-empty) sequence of delimiter
// characters consisting of exactly one hard-delimiter character.  Hard
// delimiters, typically printable punctuation characters such ('/') or colon
// (':' ), are used to terminate (rather than just separate) tokens, and thus a
// hard delimiter that is not preceded by a token character results in an empty
// token.
//
// Soft delimiters are used in applications where multiple consecutive
// delimiter characters are to be treated as just a single delimiter.  For
// example, if we want the input string '"Sticks  and stones"'' to parse into a
// sequence of three non-empty tokens ["Sticks", "and", "stones"], rather than
// the five-token sequence ["Sticks", "", "", "and", "stones"], we would make
// the space (' ') a soft-delimiter character.
//
// Hard delimiters are used in applications where consecutive delimiter
// characters are to be treated as separate delimiters, giving rise to the
// possibility of empty tokens.  Making the slash ('/') in the standard date
// format a hard delimiter for the input string "15//9" yields the three-token
// sequence ["15", "", "9"], rather than the two-token one ["15", "9"] had it
// been made soft.
//
// All members within each respective character set are considered equivalent
// with respect to tokenization.  For example, making '/' and ':' *soft*
// *delimiter* characters on the questionably formatted date "2015/:10:/31"
// would yield the token sequence ["2015", "10", "31"], whereas making '/' and
// ':' *hard* *delimiter* characters would result in the token sequence
// ["2015", "", "10", "", "31"].  Making either of these two delimiter
// characters hard and the other soft would, in this example, yield the former
// (shorter) sequence of tokens.  The details of how soft and hard delimiters
// interact is illustrated in more detail in the following section (but also
// see, later on, the section on "Comprehensive Detailed Parsing
// Specification").
//
///The Input String to be Tokenized
///--------------------------------
// Each input string consists of an optional leading sequence of soft-delimiter
// characters called the *leader*, followed by an alternating sequence of
// tokens and delimiters (the final delimiter being optional):
//..
//  Input String:
//  +--------+---------+-------------+---...---+---------+-------------+
//  | leader | token_1 | delimiter_1 |         | token_N | delimiter_N |
//  +--------+---------+-------------+---...---+---------+-------------+
//  (optional)                                              (optional)
//..
// The tokenization of a string can also be expressed as pseudo-Posix regular
// expression notation:
//..
//   delimiter = [[:soft:]]+ | [[:soft:]]* [[:hard:]] [[:soft:]]*
//   token     = [^[:soft:][:hard:]]*
//   string    = [[:soft:]]* (token delimiter)* token?
//..
// Parsing is from left to right and is *greedy* -- i.e., the longest sequence
// satisfying the regular expression is the one that matches.  For example, let
// 's' represent the start of a soft delimiter, 'd' the start of a hard
// delimiter, '^" the start of a token, and '~' the continuation of that same
// delimiter or token.  Using '.' as a soft delimiter and "/" as a hard one,
// the string
//..
//         s~   h~  h~~  h~     s~   hh  s  h~h    h~~~        Delimiters
//
//        "..One/.if./.by./land,..two//if.by/./sea!./.."
//
//           ^~~  ^~   ^~  ^~~~   ^~~ ^^~ ^~  ^^~~             Tokens
//                                    |       |
//                                 (empty)  (empty)
//..
// yields the tokenization
//..
//     [One]  [if]   [by]  [land,]  [two] [] [if] [by]  [] [sea]       Tokens
//
//  (..)   (/.)  (./.)  (./)     (..)   (/)(/)  (.)  (/.)(/)   (./..)  Delims
//..
// Notice that in pair of hard delimiters "/./" before the token "sea", the
// soft token character between the two hard ones binds to the earlier
// delimiter.
//
///Iterating using a 'TokenizerIterator' object (ACCESS TO TOKENS ONLY)
///--------------------------------------------------------------------
// This component provides two separate mechanisms by which a user may iterate
// over a sequence of tokens.  The first mechanism is as a *token* *range*,
// exposed by the 'TokenizerIterator' objects returned by the 'begin' and 'end'
// methods on a 'Tokenizer' object.  A 'TokenizerIterator' supports the concept
// of a standard *input* *iterator*, returning each successive token as a
// 'bslstl::StringRef', making it suitable for generic use -- e.g., in a
// range-based 'for' loop:
//..
//  void parse_1(bsl::ostream& output, const char *input)
//      // Print, to the specified 'output' stream, each whitespace-delimited
//      // token in the specified 'input; string on a separate line following
//      // a vertical bar ('|') and a hard space (' ').
//  {
//      const char softDelimiters[] = " \t\n";  // whitespace
//
//      for (bslstl::StringRef token : bdlb::Tokenizer(input, softDelimiters)) {
//          bsl::cout << "| " << token << bsl::endl;
//      }
//  }
//..
// The 'parse_1' function above produces each (non-whitespace) token in the
// supplied input string on a separate line.  So, were 'parse_1' to be given a
// reference to 'bsl::cout' and the input string
//..
//  " Times  like \tthese\n  try \n \t men's\t \tsouls.\n"
//..
// we would expect
//..
//  | Times
//  | like
//  | these
//  | try
//  | men's
//  | souls.
//..
// to be displayed on 'bsl::cout'.  Note that there is no way to access the
// delimiters from a 'TokenizerIterator' directly, for that we will need to
// use the 'tokenizer' as a non-standard "iterator" directly.
//
///Iterating using a 'Tokenizer' object (ACCESS TO TOKENS AND DELIMITERS)
///----------------------------------------------------------------------
// The second mechanism, not intended for generic use, provides direct access
// to the previous and current (trailing) delimiters as well as the current
// token:
//..
//  void parse_2(bsl::outstream, const char *input)
//      // Print, to the specified 'output' stream the leader of the specified
//      // 'input', on a singly line, followed by subsequent current token and
//      // (trailing) delimiter pairs on successive lines, each line beginning
//      // with a vertical bar ('|') followed by a tab ('\t') character.
//  {
//      const char softDelimiters[] = " ";
//      const char hardDelimiters[] = ":/";
//
//      bdlb::Tokenizer it(input, softDelimiters, hardDelimiters);
//      bsl::cout << "| " << '"' << it.previousDelimiter() << '"' << bsl::endl;
//
//      for (; it.isValid(); ++it) {
//          bsl::cout << "|\t"
//                    << '"' << it.token() << '"'
//                    << "\t"
//                    << '"' << it.trailingDelimiter() '"'
//                    << bsl::endl;
//      }
//..
// The parse_2 function above produces the *leader* on the first line,
// followed by each *token* along with its current (trailing) delimiter on
// successive lines.  So, were 'parse_2' to be given a reference to
// 'bsl::cout' and the input string
//..
//  " I've :been: a : :bad:/ boy! / "
//..
// we would expect
//..
//  |       " "
//  |       "I've"  " :"
//  |       "been"  ": "
//  |       "a :"   " : "
//  |       ""      ":"
//  |       "bad"   ":"
//  |       ""      "/ "
//  |       "boy!"  " / "
//..
// to be displayed on 'bsl::cout'.
//
///Token and Delimiter Lifetimes
///-----------------------------
// All tokens and delimiters are returned efficiently by value as
// 'bslstl::StringRef' objects, which naturally remain valid so long as the
// underlying input string remains unchanged -- irrespective of the validity
// of the 'tokenizer' or any of its dispensed token iterators.  Note, however,
// that all such token iterators are invalidated if the parent tokenizer object<commit_msg>Add missing argument to example function.<commit_after>// bdlb_tokenizer.h                                                   -*-C++-*-
#ifndef INCLUDED_BDLB_TOKENIZER
#define INCLUDED_BDLB_TOKENIZER

#ifndef INCLUDED_BSLS_IDENT
#include <bsls_ident.h>
#endif
BSLS_IDENT("$Id: $")

//@PURPOSE: Provide access to user-described tokens via string references.
//
//@CLASSES:
//  bdlb::Tokenizer: lexer for tokens defined via hard and/or soft delimiters
//  bdlb::TokenizerIterator: input iterator for delimited tokens in a string
//
//@SEE_ALSO: bslstl_stringref
//
//@DESCRIPTION: This component defines a mechanism, 'bdlb::Tokenizer', that
// provides non-destructive sequential (read-only) access to tokens in a given
// input string as characterized by two disjoint sets of user-specified
// delimiter characters, each of which is supplied at construction via either a
// 'const bslstl::StringRef&' or (for efficiency, when only the leading
// characters of the input string may need to be parsed) a 'const char *'.
// Note that each character (including '\0') that is not explicitly designated
// as a delimiter character is assumed to be *token* character.
//
///Soft versus Hard Delimiters
///---------------------------
// The tokenizer recognizes two distinct kinds of delimiter characters, *soft*
// and *hard*.
//
// A *soft* *delimiter* is a maximal (non-empty) sequence of soft-delimiter
// characters.  Soft delimiters, typically whitespace characters, are used to
// separate (rather than terminate) tokens, and thus never result in an empty
// token.
//
// A *hard* *delimiter* is a maximal (non-empty) sequence of delimiter
// characters consisting of exactly one hard-delimiter character.  Hard
// delimiters, typically printable punctuation characters such ('/') or colon
// (':' ), are used to terminate (rather than just separate) tokens, and thus a
// hard delimiter that is not preceded by a token character results in an empty
// token.
//
// Soft delimiters are used in applications where multiple consecutive
// delimiter characters are to be treated as just a single delimiter.  For
// example, if we want the input string '"Sticks  and stones"'' to parse into a
// sequence of three non-empty tokens ["Sticks", "and", "stones"], rather than
// the five-token sequence ["Sticks", "", "", "and", "stones"], we would make
// the space (' ') a soft-delimiter character.
//
// Hard delimiters are used in applications where consecutive delimiter
// characters are to be treated as separate delimiters, giving rise to the
// possibility of empty tokens.  Making the slash ('/') in the standard date
// format a hard delimiter for the input string "15//9" yields the three-token
// sequence ["15", "", "9"], rather than the two-token one ["15", "9"] had it
// been made soft.
//
// All members within each respective character set are considered equivalent
// with respect to tokenization.  For example, making '/' and ':' *soft*
// *delimiter* characters on the questionably formatted date "2015/:10:/31"
// would yield the token sequence ["2015", "10", "31"], whereas making '/' and
// ':' *hard* *delimiter* characters would result in the token sequence
// ["2015", "", "10", "", "31"].  Making either of these two delimiter
// characters hard and the other soft would, in this example, yield the former
// (shorter) sequence of tokens.  The details of how soft and hard delimiters
// interact is illustrated in more detail in the following section (but also
// see, later on, the section on "Comprehensive Detailed Parsing
// Specification").
//
///The Input String to be Tokenized
///--------------------------------
// Each input string consists of an optional leading sequence of soft-delimiter
// characters called the *leader*, followed by an alternating sequence of
// tokens and delimiters (the final delimiter being optional):
//..
//  Input String:
//  +--------+---------+-------------+---...---+---------+-------------+
//  | leader | token_1 | delimiter_1 |         | token_N | delimiter_N |
//  +--------+---------+-------------+---...---+---------+-------------+
//  (optional)                                              (optional)
//..
// The tokenization of a string can also be expressed as pseudo-Posix regular
// expression notation:
//..
//   delimiter = [[:soft:]]+ | [[:soft:]]* [[:hard:]] [[:soft:]]*
//   token     = [^[:soft:][:hard:]]*
//   string    = [[:soft:]]* (token delimiter)* token?
//..
// Parsing is from left to right and is *greedy* -- i.e., the longest sequence
// satisfying the regular expression is the one that matches.  For example, let
// 's' represent the start of a soft delimiter, 'd' the start of a hard
// delimiter, '^" the start of a token, and '~' the continuation of that same
// delimiter or token.  Using '.' as a soft delimiter and "/" as a hard one,
// the string
//..
//         s~   h~  h~~  h~     s~   hh  s  h~h    h~~~        Delimiters
//
//        "..One/.if./.by./land,..two//if.by/./sea!./.."
//
//           ^~~  ^~   ^~  ^~~~   ^~~ ^^~ ^~  ^^~~             Tokens
//                                    |       |
//                                 (empty)  (empty)
//..
// yields the tokenization
//..
//     [One]  [if]   [by]  [land,]  [two] [] [if] [by]  [] [sea]       Tokens
//
//  (..)   (/.)  (./.)  (./)     (..)   (/)(/)  (.)  (/.)(/)   (./..)  Delims
//..
// Notice that in pair of hard delimiters "/./" before the token "sea", the
// soft token character between the two hard ones binds to the earlier
// delimiter.
//
///Iterating using a 'TokenizerIterator' object (ACCESS TO TOKENS ONLY)
///--------------------------------------------------------------------
// This component provides two separate mechanisms by which a user may iterate
// over a sequence of tokens.  The first mechanism is as a *token* *range*,
// exposed by the 'TokenizerIterator' objects returned by the 'begin' and 'end'
// methods on a 'Tokenizer' object.  A 'TokenizerIterator' supports the concept
// of a standard *input* *iterator*, returning each successive token as a
// 'bslstl::StringRef', making it suitable for generic use -- e.g., in a
// range-based 'for' loop:
//..
//  void parse_1(bsl::ostream& output, const char *input)
//      // Print, to the specified 'output' stream, each whitespace-delimited
//      // token in the specified 'input; string on a separate line following
//      // a vertical bar ('|') and a hard space (' ').
//  {
//      const char softDelimiters[] = " \t\n";  // whitespace
//
//      for (bslstl::StringRef token : bdlb::Tokenizer(input, softDelimiters)) {
//          bsl::cout << "| " << token << bsl::endl;
//      }
//  }
//..
// The 'parse_1' function above produces each (non-whitespace) token in the
// supplied input string on a separate line.  So, were 'parse_1' to be given a
// reference to 'bsl::cout' and the input string
//..
//  " Times  like \tthese\n  try \n \t men's\t \tsouls.\n"
//..
// we would expect
//..
//  | Times
//  | like
//  | these
//  | try
//  | men's
//  | souls.
//..
// to be displayed on 'bsl::cout'.  Note that there is no way to access the
// delimiters from a 'TokenizerIterator' directly, for that we will need to
// use the 'tokenizer' as a non-standard "iterator" directly.
//
///Iterating using a 'Tokenizer' object (ACCESS TO TOKENS AND DELIMITERS)
///----------------------------------------------------------------------
// The second mechanism, not intended for generic use, provides direct access
// to the previous and current (trailing) delimiters as well as the current
// token:
//..
//  void parse_2(bsl::outstream& output, const char *input)
//      // Print, to the specified 'output' stream the leader of the specified
//      // 'input', on a singly line, followed by subsequent current token and
//      // (trailing) delimiter pairs on successive lines, each line beginning
//      // with a vertical bar ('|') followed by a tab ('\t') character.
//  {
//      const char softDelimiters[] = " ";
//      const char hardDelimiters[] = ":/";
//
//      bdlb::Tokenizer it(input, softDelimiters, hardDelimiters);
//      output << "| " << '"' << it.previousDelimiter() << '"' << "\n";
//
//      for (; it.isValid(); ++it) {
//          output << "|\t"
//                 << '"' << it.token() << '"'
//                 << "\t"
//                 << '"' << it.trailingDelimiter() '"'
//                 << "\n";
//      }
//  }
//..
// The parse_2 function above produces the *leader* on the first line,
// followed by each *token* along with its current (trailing) delimiter on
// successive lines.  So, were 'parse_2' to be given a reference to
// 'bsl::cout' and the input string
//..
//  " I've :been: a : :bad:/ boy! / "
//..
// we would expect
//..
//  |       " "
//  |       "I've"  " :"
//  |       "been"  ": "
//  |       "a :"   " : "
//  |       ""      ":"
//  |       "bad"   ":"
//  |       ""      "/ "
//  |       "boy!"  " / "
//..
// to be displayed on 'bsl::cout'.
//
///Token and Delimiter Lifetimes
///-----------------------------
// All tokens and delimiters are returned efficiently by value as
// 'bslstl::StringRef' objects, which naturally remain valid so long as the
// underlying input string remains unchanged -- irrespective of the validity
// of the 'tokenizer' or any of its dispensed token iterators.  Note, however,
// that all such token iterators are invalidated if the parent tokenizer object
{'fraction_non_alphanumeric': 0.18217095470975972, 'fraction_numerical': 0.0044652349564108015, 'mean_word_length': 2.210957501280082, 'pattern_counts': {'":': 8, '<': 69, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4989297', 'n_tokens_mistral': 5478, 'n_tokens_neox': 5323, 'n_words': 2119}