<jupyter_start><jupyter_text>Regular Expression: Analyzing Text in Python  What is Regular Expression?Working with text is HARD. Do we consider "dog" to be different from "dogs"? How do we explain to a computer if we want those words to be considered the same word? In written language, most rules are not really rules, but patterns within text. Regular expression is a paradigm for describing patterns in text in order to accomodate the flexibility of written language and our attempts to find information within larger bodies of text.For example, I might want to find a series of keywords within a body of text in order to determine whether or not the topic of the text has been correctly identified (or even in order to determine what the topic of the text actually is!). Regular expression has the ability to help express patterns in text that we may want to examine, and the flexibility to determine exactly how we want to capture it. Some common uses of regular expression:- Validating emails, phone numbers, addresses, and other text blocks- Finding subsets of a document- Determining the presence or count of particular words, phrases or patterns- Web Scraping- Grammar suggestionsThere are, of course many more ways to use regular expression. Let's start looking at how we can use it in our personal data analysis.  Basics of Regular ExpressionRemember, regular expression is way to describe patterns. Below, we will explore how to create various kinds of patterns. To get started, we need to import our regular expression library, called `re`. We can import `re` just like any other library by including the following line of code:```pythonimport re```The first function that we will utilize is `re.search`. This function will allow us to search through a string of text in order to try and find patterns of interest. Let's begin by importing some text. We will utilize two chapters (44 and 45) from the Jane Austen novel "Pride and Prejudice" (one of my favorites). Below is the code to import these chapters as a (very long) string:<jupyter_code>import requests
url = "https://raw.githubusercontent.com/dustywhite7/pythonMikkeli/master/exampleData/prideAndPrejudiceChapters.txt"
document = requests.get(url).text

print(document)<jupyter_output>Chapter 44


Elizabeth had settled it that Mr. Darcy would bring his sister to visit
her the very day after her reaching Pemberley; and was consequently
resolved not to be out of sight of the inn the whole of that morning.
But her conclusion was false; for on the very morning after their
arrival at Lambton, these visitors came. They had been walking about the
place with some of their new friends, and were just returning to the inn
to dress themselves for dining with the same family, when the sound of a
carriage drew them to a window, and they saw a gentleman and a lady in
a curricle driving up the street. Elizabeth immediately recognizing
the livery, guessed what it meant, and imparted no small degree of her
surprise to her relations by acquainting them with the honour which she
expected. Her uncle and aunt were all amazement; and the embarrassment
of her manner as she spoke, joined to the circumstance itself, and many
of the circumstances of the preceding day, opened to them a new ide[...]<jupyter_text>Finding textNow that we have our text imported, we can begin to explore it. We will start by looking for the word "Elizabeth" (she is the main character of the novel, and is also sometimes called "Lizzy" or "Eliza"). In order to look for the word "Elizabeth", we will use the `re.search` function mentioned earlier. This function takes two arguments: a regular expression, and a string in which to search for matches to that pattern. Regular expression patterns will be stored as strings, as well, but we will use **raw strings**. To create a raw string, we prepend the string with a single letter `r`, so that our string looks something like this: `r''`. By using raw strings, we allow ourselves to use escape characters (character sequences beginning with `\` such as `\n` or `\r`) within our string to describe possible patterns that we would like to match.Once we have created our regular expression pattern, we can call the `re.search` function. It will look for the first match to your pattern. Here is what happens when we look for a match to "Elizabeth":<jupyter_code>import re

re.search(r'Elizabeth', document)<jupyter_output><empty_output><jupyter_text>This is a successful search! The document does mention Elizabeth, and we see that a match to the expression `r'Elizabeth'` happened. The result of the function is an `re.Match` object, which contains information about where the match was found, as well as the text that matched our stated pattern. When we include a string like `r'Elizabeth'` as our pattern, we are telling regular expression to find a **capital** "E", followed by lower case "l", followed by lower case "i", followed by lower case "z", followed by lower case "a", followed by lower case "b", followed by lower case "e", followed by lower case "t", followed by lower case "h". No other text will match this pattern.We will often want to be more expressive than this. For example, if we read the document, we can see that some characters mention Elizabeth as Eliza. We can of course search for Eliza separately, or we can try to combine Eliza into our expression pattern, so that we can detect all matches of either Elizabeth or Eliza. To see the difference, we will use the `re.finditer` function to look for all matches to our pattern:<jupyter_code># Number of results for only Elizabeth 
print("Number of results for only Elizabeth")
print(len([i for i in re.finditer(r'Elizabeth', document)]))

# Number of results for Elizabeth OR Eliza
print("Number of results for only Elizabeth OR Eliza")
print(len([i for i in re.finditer(r'Eliza(beth)?', document)]))<jupyter_output>Number of results for only Elizabeth
32
Number of results for only Elizabeth OR Eliza
34<jupyter_text>What did we just do? A couple things. First, we used the `re.finditer` function, which generates an **iterable**. In order to get back all of our results, we need to create a list by iterating over the results. We used a list comprehension to do that. (Remember that list comprehensions are statements like `[i for i in some_iterable]`). At that point, we had a list of results. We then checked the length of the list for each of two different regular expressions. One for just Elizabeth, and one for Eliza/Elizabeth. Now let's break down the difference in the regular expressions. There are two differences between our first and second expressions. First, we used `()` to encapsulate a portion of the word Elizabeth that may not be present in some cases. Parens (`()`) create **groups** in regular expression. Sometimes we will want to capture groups from within an expression, and other times we will only want to indicate that a group may or may not be present. In this case, we indicate that the sequence `beth` is not strictly required for a match by using a `?` character after the group. This doesn't mean that our text needs to include a `?`. The `?` symbol says "I want either 1 or 0 matches to the "beth" sequence.In other words, "beth" may be present (have one match) or not present (have zero matches) while still allowing us to have a successful match to our regular expression pattern. There are several similar expressions:| Symbol | Meaning || --- | :---: || `?` | 1 or 0 repetitions|| `+` | 1 or more repetitions | | `*` | 0 or more repetitions ||`{n}` | exactly `n` repetitions||`{m,n}` | between `m` and `n` repetitions|We can use these expressions with groups, or with single characters. What if we wanted to find the end of sentences in our text? Let's try and express that:<jupyter_code>[i for i in re.finditer(r'[.?!][ ]*', document)]<jupyter_output><empty_output><jupyter_text>This is a good start! We know that sentences can end with "?", "!", or ".", so we create a set of characters that can be matched: `[.?!]`. This expression says that a character can be a period, question mark, or exclamation mark and still match our pattern. We then use a character set including only the space character followed by a star (`[ ]*`) to indicate that there should be 0 or more spaces following each period.We can use character sets where spelling is uncertain, or where there is a possibility of letters being upper or lower case. Let's look for the word "on", both with and without a character set to accomodate case ambiguity:<jupyter_code># Number of results for "on"
print("Number of results for 'on'")
print(len([i for i in re.finditer(r'on', document)]))

# Number of results for "on" or "On"
print("Number of results for 'on' or 'On'")
print(len([i for i in re.finditer(r'[Oo]n', document)]))<jupyter_output>Number of results for 'on'
219
Number of results for 'on' or 'On'
221<jupyter_text>If we weren't careful, we would miss some matches to the word we are looking for!  Finding numbersWe won't always be looking for words. Sometimes, we want to find or validate numbers. In the United States, phone numbers are commonly expressed with a three-digit area code, a hyphen (`-`), and a seven-digit phone number with the first three numbers separated from the final four by another hyphen. Let's try to detect this pattern in text that is provided by the user. Before we get started, let's talk about how we can identify numbers.When we look for numbers, we typically are not looking for a single specific number (like 42). We are instead looking for **some** number. It might be 2, it might be 142, or it might be 1000. Let's look for a number in the following sentence:<jupyter_code>sentence = "My favorite number is 7"

re.search(r'7', sentence)<jupyter_output><empty_output><jupyter_text>Okay, we can find an exact number. Can you think of a way that we have discussed above to find **any** single-digit number?**Solve it (2 points)**:The user will enter their favorite single-digit number. You need to make sure that there is a number in the sentence. Store the results of your `re.search` function in the variable `hasNumber` in the cell below:<jupyter_code>import re

# Ask user for favorite number (DON'T DELETE)
myNumber = input("Please enter your favorite single-digit number: ")

# Create the sentence (DON'T DELETE)
sentence = "My favorite number is {}".format(int(myNumber))

# Write a regular expression to determine if there is a number in the sentence
hasNumber = re.search(r'\b\d\b', sentence)

print(bool(hasNumber))<jupyter_output>Please enter your favorite single-digit number: 10
False<jupyter_text>When we want to find a number, we could use character sets: `[0123456789]`. This character set specifies that we will accept any number character as a match to our pattern. If we want to specify that it be a number that has more than one character, we could provide a pattern such as `[0123456789]+`. It will get really old, really fast to keep typing `[0123456789]`. Instead, we can use the shorthand `\d` representation for a **digit** or number. Now, if we want to find a number character, we can just write `\d` in place of `[0123456789]`. For one or more number characters, we write `\d+`.Now, let's get back to trying to break down phone number validation. Using what we know about digit representation, combined with our knowledge of ways to express repeated patterns from before, we can express a phone number! If we rewrite our goal as words, we would end up with the following:- Three numbers- A hyphen- Three numbers- A hyphen- Four numbersLet's write this as a regular expression:<jupyter_code># Ask user for phone number
myNumber = input("Please enter your phone number: ")

# Check if the phone number is valid
valid = re.search(r'\d{3}-\d{3}-\d{4}', myNumber)

# Print a boolean value indicating whether or not the string contains a valid phone number
print(bool(valid))<jupyter_output>Please enter your phone number: 123-456-7890 turtles
True<jupyter_text>Awesome! Try different valid and invalid phone numbers in the cell above to see if they work with our code.  More general patternsWhat other patterns might come in handy? Here is a short table of helpers for making your expressions easier to write:| Pattern | Meaning | Pattern | Meaning || --- | --- | --- | --- || `\s` | any whitespace character | `\S` | any non-whitespace character || `\d` | any digit character | `\D` | any non-digit character | | `\w` | any word character (a number, a letter, or a `_`) | `\W` | non-word characters|| `\b` | word boundaries (the start or end of a sequence of word characters) ||| `^` | start of a string | `$` | end of a stringGoing back to our phone number exercise, what happens if you provide the string `123-456-7890 turtles` as a phone number? Is it considered valid? **It is!** We can use our new patterns to protect against this:<jupyter_code># Add start and end of string characters
valid = re.search(r'^\d{3}-\d{3}-\d{4}$', "123-456-7890 turtles")

# Print an improved boolean value indicating whether or not the string contains a valid phone number
print(bool(valid))<jupyter_output>True<jupyter_text>Now we won't be fooled so easily! Our number has to be formatted correctly, AND our string cannot contain any other characters. Play with the code above and check for yourself!When we think about how to create an expression to validate text or numbers, we should be particularly careful to think of any **possible** ways that a value might be invalid. It is not sufficient to think only of ways in which text is **likely** to be invalid. If we are not careful, our validation will not truly validate, and any work that we do with the "validated" text may be broken by cases we have not accounted for.  Looking BehindSpeaking of not accounting for all the possibilities, we made this mistake earlier when we were looking for the ends of sentences earlier. We looked for all periods (`.`), question marks (`?`), and exclamation marks (`!`). Unfortunately, not all periods come at the end of sentences! In *Pride and Prejudice*, many characters are referred to as "Mr." or "Mrs.". Those periods are followed by spaces, and so would have counted as the end of a sentence. Negative LookbehindIn order to solve this problem, we need a new tool: the **negative lookbehind**. A negative lookbehind is a pattern that must be matched while simultaneously NOT being preceded by another pattern. In this case, we would want `[.?!]` wherever that character set is NOT preceded by `Mr` or `Mrs`. First, let's make sure that we can find `Mr` and `Mrs` with a single expression:<jupyter_code>[i for i in re.finditer(r'Mrs*', document)]<jupyter_output><empty_output><jupyter_text>So far, so good! Now that we can find all of the cases that we **don't** want to count as ends of sentences, let's use a negative lookbehind. The expression we want to avoid will be put in a group (between `()`), and we will put the character sequence `?<!` at the start of the group:<jupyter_code># Count of sentences without negative lookbehind (incorrect)
print("Count of sentences without negative lookbehind (incorrect)")
print(len([i for i in re.finditer(r'[.!?]', document)]))

# Count of sentences with negative lookbehind (correct)
print("Count of sentences with negative lookbehind (correct)")
print(len([i for i in re.finditer(r'(?<!Mr)(?<!Mrs)[.!?]', document)]))<jupyter_output>Count of sentences without negative lookbehind (incorrect)
161
Count of sentences with negative lookbehind (correct)
128<jupyter_text>We have to use **two** negative lookbehinds to solve this problem, because we cannot allow for `Mr` or `Mrs` to precede our punctuation. You might have tried `(?<!Mrs*)` (as I did at first). Unfortunately, we cannot use this structure because we are not permitted to use a negative lookbehind that is of uncertain length (as is any pattern with `*`, `+`, `?`, or `{}`). Instead, we simply check for the first string, then check for the second string.  Positive LookbehindWe can also use a positive lookbehind to **require** that our pattern be preceded by another pattern. For example, we may wish to determine which chapters are covered in a text document (we know it is Chapters 44 and 45, but we can use this as an exercise anyway). A positive lookbehind is indicated within parens (`()`) just like the negative lookbehind, but uses the character sequence `?<=`. Let's try to find numbers that are preceded by the pattern `Chapter `:<jupyter_code>[i for i in re.finditer(r'(?<=Chapter )\d+', document)]<jupyter_output><empty_output><jupyter_text>Excellent! Just as we know *should* be the case, we can see that we found the beginnings of both chapters 44 and 45.  OptionsWhen we are creating patterns, we sometimes will encounter multiple acceptable patterns that we wish to account for. With phone numbers, area codes in the United States are often written in parens rather than with hypens. Thus, we might need to account for two patterns:- `xxx-xxx-xxxx`- `(xxx) xxx-xxxx`Let's try to write expressions for the second option (since we can already accomodate the first):<jupyter_code># Ask user for phone number
myNumber = input("Please enter your phone number: ")

# Check if the phone number is valid
valid = re.search(r'^[(]\d{3}[)][ ]\d{3}-\d{4}$', myNumber)

# Print a boolean value indicating whether or not the string contains a valid phone number
print(bool(valid))<jupyter_output>Please enter your phone number: (425) 443-1880
True<jupyter_text>Our new rule is written `[(]\d{3}[)][ ]`. There are a lot of square brackets! In order to look for the `()` characters, we have to wrap them in a character set, so that our program knows that they are not there to indicate a group, but are instead literal characters. In between the parens, we have our `\d{3}` three-digit number set. Finally, at the end, we have a single space character `[ ]` that must separate the area code `(xxx)` from the rest of the phone number `xxx-xxxx`.You can try various inputs to see if the code continues to validate phone numbers correctly. It should work on our new format with parens, but not on our original phone number format. Luckily, now that we have a validity check for both phone number formats, we can incorportate them into a single expression to accept either valid format:<jupyter_code># Ask user for phone number
myNumber = input("Please enter your phone number: ")

# Check if the phone number is valid
valid = re.search(r'^([(]\d{3}[)][ ]|\d{3}-)\d{3}-\d{4}$', myNumber)

# Print a boolean value indicating whether or not the string contains a valid phone number
print(bool(valid))<jupyter_output>Please enter your phone number: (123) 456-7890
True<jupyter_text>In this case, we have taken the original area code rule (`\d{3}-`) as well as the new area code rule (`[(]\d{3}[)][ ]`), and we have put them inside of a group using parens `()`. They are separated inside this group by the `|` character. Just like it does in our boolean conditions, the `|` character represents the logical statement "or". Essentially, we are saying that our pattern must match `\d{3}-` OR `[(]\d{3}[)][ ]`, followed by `\d{3}-\d{4}`.For good measure, we have wrapped the whole expression in string start and end characters (`^` and `$`) in order to disallow any other text in our valid phone numbers.  Extracting resultsWhen we use the `re.search` or `re.finditer` functions, matches are returned to us through an `re.Match` object. We can extract the text from a match object by using the `.string` attribute.<jupyter_code>valid.string<jupyter_output><empty_output><jupyter_text>Also useful is the `.span` function of the `re.Match` object, which returns to us the start and end points (within the original string) of the match to our pattern:<jupyter_code>valid.span()<jupyter_output><empty_output><jupyter_text>Finally, we also have the ability to transform an `re.Match` object into a `bool`-type object, which is useful in various contexts such as text validation. To evaluate an `re.Match` object, we can simply convert the variable into which it is stored:<jupyter_code>bool(valid)<jupyter_output><empty_output><jupyter_text>If, on the other hand, we fail to find a match to our pattern, our `re.search` or `re.finditer` function will return a `None` object, and it's boolean value will be `False`:<jupyter_code>value = re.search(r'happy', "I am sad")

print(value)
print(bool(value))<jupyter_output>None
False<jupyter_text>**Solve it (2 points)**:Write a function called `dateCheck` that will accept a string as an argument to determine whether or not that string contains ONLY a valid date. The date should be of the format DD-MM-YYYY. In other words, the day of the month should come first, followed by a hyphen (`-`), followed by the month itself, followed by another hyphen (`-`), and finally the year. Days and months should always have two digits, and years should have four digits.For the purpose of this exercise, pretend that all months have 31 days, and remember that there are twelve months in any year.When the string is a correctly formatted date, your function should return `True`. Where the string is not a valid date, your function should return `False`.The blue cell below should contain ALL NECESSARY CODE, including import statements, for your function to work properly.<jupyter_code>import re

def dateCheck(date):
    check = re.search(r'^(3[01]|[012]\d)-(0\d|1[012])-\d{4}$', date)
    return bool(check)<jupyter_output><empty_output><jupyter_text>**Solve it (2 points)**:Write a function called `emailCheck` that will accept a string as an argument, and determine whether or not that string is a valid email. Emails take the following form: a string that contains numbers, letters, and periods followed by the `@` symbol and a domain. We will accept domains that contain numbers and letters, ending in `.com`, `.edu`, or `.org`.When the string is a correctly formatted email address, your function should return `True`. Where the string is not a valid email address, your function should return `False`.The blue cell below should contain ALL NECESSARY CODE, including import statements, for your function to work properly.<jupyter_code>import re

def emailCheck(email):
    check = re.search(r'^[a-zA-Z0-9.]+[@][a-zA-Z0-9]+([.]com|[.]edu|[.]org)$', email)
    return bool(check)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07882500223553608, 'fraction_numerical': 0.008226772780112671, 'mean_word_length': 4.8126299376299375, 'pattern_counts': {'":': 1, '<': 77, '<?xml version=': 0, '>': 71, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '460765', 'license': 'MIT', 'n_tokens_mistral': 6063, 'n_tokens_neox': 5746, 'n_words': 3368, 'path': 'Notebooks/5 - Regular Expression in Python.ipynb', 'repo_name': 'dustywhite7/pythonMikkeli', 'size': '22366'}