<jupyter_start><jupyter_text>Introduction to Regular Expressions**Regular Expressions** give us a way to do much more powerful string manipulation. A regular expression, or simply **regex**, is a special string that describes a specific pattern that you would like to match in another string. Examples of questions that regexes can answerIt might be helpful to see a list of questions that a regex pattern can match:* Match all words that begin with 'S' and end in 'y'* Match the word 'friend' or 'freind'* Match a word with at least 3 digits in it* Match all Gmail email addresses* Capture the word immediately following the word 'Author'* Capture the word immediately following the third occurrence of the word 'coffee' The `contains` and `extract` methodsWe will be primarily concerned with finding matching patterns within string values of a Pandas Series. We will then select all values within the Series that match the pattern via boolean indexing. The `contains` string Series method will be used for this.Eventually, we will use the `extract` string Series method to extract particular substrings from the strings within the Series. A simple example without regular expressionsLet's match all movie titles that contain either an 'x', 'y', or 'z'. Without using a regex, we would use multiple `contains` string methods separating them with the logical **or** symbol:<jupyter_code>import pandas as pd
movie = pd.read_csv('../data/movie.csv')
title = movie['title']
title.head(3)
has_xyz = title.str.contains('x') | title.str.contains('y') | title.str.contains('z')
title[has_xyz].head()<jupyter_output><empty_output><jupyter_text>We can sum up this boolean Series to determine the number of values that have either an 'x', 'y', or 'z' in them.<jupyter_code>has_xyz.sum()<jupyter_output><empty_output><jupyter_text>Use a regex insteadInstead, we can use the regex `'[xyz]'`, which matches the pattern for any string that contains an 'x', 'y', or 'z'. We can verify that we get the same total. This regex plus many more will be covered in detail below.<jupyter_code>title.str.contains('[xyz]').sum()<jupyter_output><empty_output><jupyter_text>Mini-Programming LanguageRegular expressions are a miniature programming language that have their own strict set of rules just like any other language. The syntax is written as a string mixing both **literal** and **special** characters.  Literal vs Special CharactersThere are two distinct categories of characters within a regex string - **Literal** and **Special*** **Literal** - these characters don't have any special meaning. They simply represent themselves. They are also referred to as **regular** characters.* **Special** - these characters do have a special meaning. Each special character represents something very specific. They are also referred to as **metacharacters**. Matching with only Literal CharactersThe simplest regex patterns you can write contain only literal characters. These strings will look like any ordinary string. Let's search for movies that have the word `'Star'` in them.`'Star'` is a valid regular expression. We will use the `contains` Series string method which accepts a regular expression as its first argument. It returns a boolean Series.<jupyter_code>pattern = 'Star'
title.str.contains(pattern).head()<jupyter_output><empty_output><jupyter_text>Filter for only movies containing `Star`Let's take this resulting Series and use it for boolean indexing. The result should be the movie titles that have **`Star`** in them.<jupyter_code>pattern = 'Star'
filt = title.str.contains(pattern)
title[filt].head()<jupyter_output><empty_output><jupyter_text>Regular Expressions are case sensitiveRegexes are case sensitive by default. `'Star'` only matches movie titles with an uppercase `'S'` followed immediately by lowercase `'tar'`. Let's search for lowercase `'star'`:<jupyter_code>pattern = 'star'
filt = title.str.contains(pattern)
title[filt]<jupyter_output><empty_output><jupyter_text>Find all movies containing exact string `'Star Wars'`<jupyter_code>pattern = 'Star Wars'
filt = title.str.contains(pattern)
title[filt]<jupyter_output><empty_output><jupyter_text>Find all movies containing exact string `'hine'`:<jupyter_code>pattern = 'hine'
filt = title.str.contains(pattern)
title[filt].head()<jupyter_output><empty_output><jupyter_text>Special CharactersThe following characters are the **special** or **metacharacters**`. ^ $ * + ? { } [ ] \ | ( )`  Details and examples with special charactersThe rest of this notebook is devoted to examples that explain each of the special characters above. This will not be an exhaustive coverage of regular expressions as they can get quite complex. There are even entire books written on the subject. The dot metacharacter `.`The **dot** or **period** is a special character that matches any character. For example the regex `'m.le'` will match any string that has an **`m`** followed by any character followed by `le`. It will match 'male', 'mile', 'mole', 'thimble', 'tumble', etc... Let's see how many movie titles have this pattern:<jupyter_code>pattern = 'm.le'
filt = title.str.contains(pattern)
title[filt]<jupyter_output><empty_output><jupyter_text>The caret metacharacter `^`The caret, `^`, is a special character that forces the pattern to match from the beginning of the string. Let's take a look at the difference between the regexes `War` and `^War`. The first matches the word 'War' anywhere in the string. The second matches the word 'War' only at the beginning. Let's output the differences:<jupyter_code>pattern = 'War'
filt = title.str.contains(pattern)
title[filt].head()
pattern = '^War'
filt = title.str.contains(pattern)
title[filt]<jupyter_output><empty_output><jupyter_text>The dollar sign metacharacter `$`The dollar sign metacharacter, **`$`** works analogously to the caret but instead forces a match to the **end** of the string. Let's find all the movies that end in 'War':<jupyter_code>pattern = 'War$'
filt = title.str.contains(pattern)
title[filt]<jupyter_output><empty_output><jupyter_text>Start and End Anchor tagsThe caret and dollar metacharacters are also know as **anchor** tags since they anchor the pattern to either the beginning or end.  Combining special charactersA regex can have any number of literal and meta special characters. The following regex matches movies that begin with `S`, followed by any character followed `n`.<jupyter_code>pattern = r'^S.n'
filt = title.str.contains(pattern)
title[filt].head(3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09153633854645814, 'fraction_numerical': 0.00045998160073597056, 'mean_word_length': 5.888067581837381, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '625676', 'license': 'MIT', 'n_tokens_mistral': 1785, 'n_tokens_neox': 1743, 'n_words': 804, 'path': 'jupyter_notebooks/pandas/mastering_data_analysis/08. Regular Expressions/01. Introduction to Regular Expressions.ipynb', 'repo_name': 'manual123/Nacho-Jupyter-Notebooks', 'size': '6522'}