<commit_before>trespass
========

Regular Expression Engine

Almost POSIX extended regular expressions, plus:

1. Search multiple patterns simultaneously
    ```python
    pattern = Pattern()
    pattern.addRegExp(r'(ab+c*)', 1)
    pattern.addRegExp(r'bf(ab+)*', 2)
    pattern.addRegExp(r'^(a(bc)?)*$', 3)
    pattern.addRegExp(r'01x?(ab+)*2', 'red')
    assert pattern.match('abbbabf') == ((0, 4), 1)
    assert pattern.match('ddg01abb2s') == ((3, 9), 'red')
    ```

2. Add text in chunks
    ```python
    pattern = Pattern('hello$', 'done')
    matcher = Matcher(pattern)
    assert matcher.addChunk('hello, world') is None
    assert matcher.addChunk(' and again hello') is None
    assert matcher.addFinal('') == ((23, 28), 'done')
    ```

3. Flexible capturing using tags (#)
    ```python
    # positions 1,15 are start and end of matching segment
    # positions 3,9 are start of alphanumeric sequences
    # positions 8,14 are end of alphanumeric sequences
    # Note: use \# to match a literal #
    pattern = Pattern('([[:space:]]+#[[:alnum:]]+#)+!')
    assert pattern.match('!\t\thello world!') \
                == ((1, 3, 8, 9, 14, 15), None)
    ```

4. Greedy and reluctant matching
    ```python
    pattern = Pattern('a{2,4}?#a??#a?#a+#a*')
    assert pattern.match('aaaaaaaa') == ((0, 2, 2, 3, 8, 8), None)
    ```<commit_msg>Update README to match 2.0.0 examples<commit_after>trespass
========

Regular Expression Engine

Almost POSIX extended regular expressions, plus:

1. Search multiple patterns simultaneously - return longest-leftmost result - 
   if results have same length and startpos, pattern added earlier succeeds
    ```python
    pattern = Pattern()
    pattern.addRegExp(r'(ab+c*)', 1)
    pattern.addRegExp(r'bf(ab+)*', 2)
    pattern.addRegExp(r'^(a(bc)?)*$', 3)
    pattern.addRegExp(r'01x?(ab+)*2', 'red')
    match = pattern.match('abbbabf')
    assert (match.start(), match.end(), match.value()) == (0, 4, 1)
    match = pattern.match('ddg01abb2s')
    assert (match.start(), match.end(), match.value()) == (3, 9, 'red')
    ```

2. Add text in chunks
    ```python
    pattern = Pattern('hello$', 'done')  # 2nd param can be anything
    pattern.addRegExp('hello', addLinks) # e.g. addLinks is a function
    matcher = Matcher(pattern)
    match = matcher.addChunk('hello')
    # matches second pattern, but may also match first pattern
    assert match is None
    match = matcher.addChunk(', world')
    # we now know text does not match first pattern so match to
    # second pattern succeeds
    assert (match.start(), match.end()) == (0, 5)
    assert match.value() == addLinks
    # After any successful match we need to create a new Matcher
    # Indexes start from 0 again
    matcher = Matcher(pattern)
    match = matcher.addChunk(' and again hello')
    # again, no match yet since text may match the first pattern
    assert match is None
    match = matcher.addFinal('')
    # we now know that both patterns match - same start and length
    # so pattern added first wins
    assert (match.start(), match.end()) == (11, 16)
    assert match.value() == 'done'
    ```

3. Flexible capturing using tags (#)
    ```python
    # positions 3,8 are limits of first alphanumeric sequence
    # positions 9,14 are limits of second alphanumeric sequence
    # Note: use \# to match a literal #
    pattern = Pattern('([[:space:]]+#[[:alnum:]]+#)+!')
    match = pattern.match('!\t\thello world!')
    assert match.start() == 1
    assert match.end() == 15
    assert match.tags() == (3, 8, 9, 14)
    ```

4. Greedy and reluctant matching
    ```python
    # a{2,4}? matches as few characters as possible (2)
    # a?? matches as few characters as possible (0)
    # a? matches as many characters as possible (1)
    # a* matches as many characters as possible (4)
    # a+ matches as many characters as possible (1)
    # (a* gets to be greedy before a+, so a* eats more characters,
    # however the requirement for a+ to match at least one, in order
    # that the entire match is successful, causes a* to match 4, not 5)
    pattern = Pattern('a{2,4}?#a??#a?#a*#a+#')
    assert pattern.match('aaaaaaaa').tags() == (2, 2, 3, 7, 8)
    ```    
{'fraction_non_alphanumeric': 0.15340638399237733, 'fraction_numerical': 0.024773701762744165, 'mean_word_length': 3.31551901336074, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2665082', 'n_tokens_mistral': 1402, 'n_tokens_neox': 1318, 'n_words': 456}