<jupyter_start><jupyter_text>*This notebook contains an excerpt from the [Whirlwind Tour of Python](http://www.oreilly.com/programming/free/a-whirlwind-tour-of-python.csp) by <NAME>; the content is available [on GitHub](https://github.com/jakevdp/WhirlwindTourOfPython).*  Flexible Pattern Matching with Regular Expressions Regular expressions generalize this "wildcard" idea to a wide range of flexible string-matching sytaxes.The Python interface to regular expressions is contained in the built-in ``re`` module; as a simple example, let's use it to duplicate the functionality of the string ``split()`` method:<jupyter_code>line = 'the quick brown fox jumped over a lazy dog'
line.split()
import re
regex = re.compile('\s+')
regex.split(line)<jupyter_output><empty_output><jupyter_text>In this case, the input is ``"\s+"``: "``\s``" is a special character that matches any whitespace (space, tab, newline, etc.), and the "``+``" is a character that indicates *one or more* of the entity preceding it.Thus, the regular expression matches any substring consisting of one or more spaces.<jupyter_code>for s in ["     ", "abc  ", "  abc", "   abc"]:
    if s.startswith('  a'):
        print(repr(s), "matches")
    else:
        print(repr(s), "does not match")
regex = re.compile('\s+')
for s in ["     ", "abc  ", "  abc"]:
    if regex.match(s):
        print(repr(s), "matches")
    else:
        print(repr(s), "does not match")
regex = re.compile('\s+a')
for s in ["     ", "abc  ", "  abc", "   abc"]:
    if regex.match(s):
        print(repr(s), "matches")
    else:
        print(repr(s), "does not match")
line = 'the quick brown fox jumped over a lazy dog'
line.index('fox')
line.find('fox')
regex = re.compile('fox')
match = regex.search(line)
match.start()
regex = re.compile('[a-z]+o[a-z]+')
match = regex.search(line)
match.start()
regex.findall(line)
line.replace('fox', 'BEAR')
regex.sub('BEAR', line)<jupyter_output><empty_output><jupyter_text>A more sophisticated exampleBut, you might ask, why would you want to use the more complicated and verbose syntax of regular expressions rather than the more intuitive and simple string methods?The advantage is that regular expressions offer *far* more flexibility.<jupyter_code>email = re.compile('\w+@\w+\.[a-z]{3}')
text = "To email Guido, try <EMAIL> or the older address <EMAIL>."
email.findall(text)
email.sub('--@--.--', text)
email.findall('<EMAIL>')<jupyter_output><empty_output><jupyter_text>Basics of regular expression syntaxThe syntax of regular expressions is much too large a topic for this short section.Still, a bit of familiarity can go a long way: I will walk through some of the basic constructs here, and then list some more complete resources from which you can learn more.My hope is that the following quick primer will enable you to use these resources effectively.  Simple strings are matched directlyIf you build a regular expression on a simple string of characters or digits, it will match that exact string:<jupyter_code>regex = re.compile('ion')
regex.findall('Great Expectations')<jupyter_output><empty_output><jupyter_text>Some characters have special meaningsWhile simple letters or numbers are direct matches, there are a handful of characters that have special meanings within regular expressions. They are:```. ^ $ * + ? { } [ ] \ | ( )```We will discuss the meaning of some of these momentarily.In the meantime, you should know that if you'd like to match any of these characters directly, you can *escape* them with a back-slash:<jupyter_code>print(r'Hi\nThere')
regex = re.compile(r'\\')
regex.findall("the \ cost \ is $20")
regex = re.compile(r'\$')
regex.findall("the cost is $20")
regex = re.compile(r'\w\s\w')
regex.findall('the fox is 9 years old')<jupyter_output><empty_output><jupyter_text>The following table lists a few of these characters that are commonly useful:| Character | Description                 |.| Character | Description                     ||-----------|-----------------------------|-|-----------|---------------------------------|| ``"\d"``  | Match any digit             |-| ``"\D"``  | Match any non-digit             || ``"\s"``  | Match any whitespace        |-| ``"\S"``  | Match any non-whitespace        || ``"\w"``  | Match any alphanumeric char |-| ``"\W"``  | Match any non-alphanumeric char |This is *not* a comprehensive list or description; for more details, see Python's [regular expression syntax documentation](https://docs.python.org/3/library/re.htmlre-syntax).  Square brackets match custom character groupsIf the built-in character groups aren't specific enough for you, you can use square brackets to specify any set of characters you're interested in.For example, the following will match any lower-case vowel:<jupyter_code>regex = re.compile('[aeiou]')
regex.split('consequential')<jupyter_output><empty_output><jupyter_text>Similarly, you can use a dash to specify a range: for example, ``"[a-z]"`` will match any lower-case letter, and ``"[1-3]"`` will match any of ``"1"``, ``"2"``, or ``"3"``.For instance, you may need to extract from a document specific numerical codes that consist of a capital letter followed by a digit. You could do this as follows:<jupyter_code>regex = re.compile('[A-Z][0-9]')
regex.findall('1043879, G2, H6, Z44')<jupyter_output><empty_output><jupyter_text>Wildcards match repeated charactersIf you would like to match a string with, say, three alphanumeric characters in a row, it is possible to write, for example, ``"\w\w\w"``.Because this is such a common need, there is a specific syntax to match repetitions – curly braces with a number:<jupyter_code>regex = re.compile(r'\w{3}')
regex.findall('The quick brown fox')
regex = re.compile(r'\w+')
regex.findall('The quick brown fox')<jupyter_output><empty_output><jupyter_text>The following is a table of the repetition markers available for use in regular expressions:| Character | Description | Example ||-----------|-------------|---------|| ``?`` | Match zero or one repetitions of preceding  | ``"ab?"`` matches ``"a"`` or ``"ab"`` || ``*`` | Match zero or more repetitions of preceding | ``"ab*"`` matches ``"a"``, ``"ab"``, ``"abb"``, ``"abbb"``... || ``+`` | Match one or more repetitions of preceding  | ``"ab+"`` matches ``"ab"``, ``"abb"``, ``"abbb"``... but not ``"a"`` || ``{n}`` | Match ``n`` repetitions of preeeding | ``"ab{2}"`` matches ``"abb"`` || ``{m,n}`` | Match between ``m`` and ``n`` repetitions of preceding | ``"ab{2,3}"`` matches ``"abb"`` or ``"abbb"`` |<jupyter_code>email = re.compile(r'\w+@\w+\.[a-z]{3}')
email2 = re.compile(r'[\w.]+@\w+\.[a-z]{3}')
email2.findall('<EMAIL>')<jupyter_output><empty_output><jupyter_text>We have changed ``"\w+"`` to ``"[\w.]+"``, so we will match any alphanumeric character *or* a period.With this more flexible expression, we can match a wider range of email addresses (though still not all – can you identify other shortcomings of this expression?).  Parentheses indicate *groups* to extractFor compound regular expressions like our email matcher, we often want to extract their components rather than the full match. This can be done using parentheses to *group* the results:<jupyter_code>email3 = re.compile(r'([\w.]+)@(\w+)\.([a-z]{3})')
text = "To email Guido, try <EMAIL> or the older address <EMAIL>."
email3.findall(text)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16154891304347826, 'fraction_numerical': 0.004891304347826087, 'mean_word_length': 4.3417997097242385, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '601443', 'license': 'MIT', 'n_tokens_mistral': 2275, 'n_tokens_neox': 2200, 'n_words': 817, 'path': '11-introduction_regular_expressions.ipynb', 'repo_name': 'hanisaf/advanced-data-management-and-analytics', 'size': '7445'}