<jupyter_start><jupyter_text>Python: Regular expressions (Regex)  Introduction **Regex** are sequences of characters that describe searched elements. They are widely used in data science to **extract and search information from data**.<jupyter_code># Example
ai = ["data science", "big data", "statistics", "computer science"]
regex = "data"
ai_after_regex = ["data science", "big data"]<jupyter_output><empty_output><jupyter_text>Generic characters One of the most important generic characters in regex is the **dot(.)**. It allows to take any element in the regex.<jupyter_code># Example
strings = ["but", "bat", "robotics"]
strings_regex = "b.t" # the matching regex<jupyter_output><empty_output><jupyter_text>Find the beginning and the end of a string There are two symbols that allow you to search for the beginning and the end of a character string. These are the **symbol (^) for the beginning** of a string and the **symbol ($) for the end** of a string.<jupyter_code># Example
strings = ["he throws his bat", "robot"]
bad_string = "it is good but not generous"
regex_for_strings = "b.t$"<jupyter_output><empty_output><jupyter_text>Dataset<jupyter_code>import csv

posts = list(csv.reader(open("askreddit_2015.csv", encoding="utf-8")))
print(posts[0:5])
posts = posts[1:]

for post in posts[:10]:
    print(post)<jupyter_output>['What\'s your internet "white whale", something you\'ve been searching for years to find with no luck?', '11510', '1433213314.0', '1', '26195']
["What's your favorite video that is 10 seconds or less?", '8656', '1434205517.0', '4', '8479']
['What are some interesting tests you can take to find out about yourself?', '8480', '1443409636.0', '1', '4055']
["PhD's of Reddit. What is a dumbed down summary of your thesis?", '7927', '1440188623.0', '0', '13201']
['What is cool to be good at, yet uncool to be REALLY good at?', '7711', '1440082910.0', '0', '20325']
['[Serious] Redditors currently in a relationship, besides dinner and a movie, what are your favorite activities for date night?', '7598', '1439993280.0', '2', '5389']
["Parents of Reddit, what's something that your kid has done that you pretended to be angry about but secretly impressed or amused you?", '7553', '1439161809.0', '0', '11520']
['What is a good subreddit to binge read the All Time Top Posts of?', '7498', '1438822288.0',[...]<jupyter_text>Counting matches with the re module In this section, we will discuss the **re** module. A useful method of this module is the **search()** function. It allows to check if there is a **match between a string and a regex**.<jupyter_code>import re
# Example 1

if re.search("artificial intelligence", "big data") is not None:
    print("Match found!")
else:
    print("Match not found!")
# Example 2

if re.search("b.", "big data") is not None:
    print("Match found!")
else:
    print("Match not found!")<jupyter_output>Match found!<jupyter_text>Training Let's count the number of posts that contain the matcher "of Reddit" in the titles.<jupyter_code>of_reddit_count = 0

for row in posts:
    if re.search("of Reddit", row[0]) is not None:
        of_reddit_count += 1
print(of_reddit_count)<jupyter_output>76<jupyter_text>Hooks to match several letters To match several characters, we use the brackets **\[ \]**.  Training Let's count the number of posts that contain the matchers "of Reddit" and "of reddit" in the titles.<jupyter_code>of_reddit_count = 0

for row in posts:
    if re.search("of [Rr]eddit", row[0]) is not None:
        of_reddit_count += 1
print(of_reddit_count)<jupyter_output>102<jupyter_text>Ignore special characters To ignore special characters, use the **\** character.  Training Let's count the number of posts whose title contains the tag [Serious].<jupyter_code>serious_count = 0

for row in posts:
    if re.search("\[Serious\]", row[0]) is not None:
        serious_count += 1
print(serious_count)<jupyter_output>69<jupyter_text>Improve our regex  Training Let's go count the posts with the following tags in the title: (Serious) - (serious) - [Serious] - [serious].<jupyter_code>serious_count = 0

for row in posts:
    if re.search("[\[\(][Ss]erious[\)\]]", row[0]) is not None:
        serious_count += 1
print(serious_count)<jupyter_output>80<jupyter_text>Combine multiple regex To combine several regex, we use the **|** character.  Training Let's go count for all the posts, all the tags that are at the beginning, at the end and both at the same time.<jupyter_code>serious_start_count = 0
serious_end_count = 0
serious_final_count = 0

for row in posts:
    if re.search("^[\[\(][Ss]erious[\)\]]", row[0]) is not None:
        serious_start_count += 1
    if re.search("[\[\(][Ss]erious[\)\]]$", row[0]) is not None:
        serious_end_count += 1
    if re.search("^[\[\(][Ss]erious[\)\]]|[\[\(][Ss]erious[\)\]]$", row[0]) is not None:
        serious_final_count += 1
print(serious_start_count)
print(serious_end_count)
print(serious_final_count)<jupyter_output>69
11
80<jupyter_text>Editing strings with regex The approach that allows to modify one element by another is the **sub()** method of the **re** module.<jupyter_code># Example 1
re.sub("hello", "hi", "hello world!") 
# Example 2
re.sub("hey", "hi", "hello world!")<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>posts_new = []

for row in posts:
    row[0] = re.sub("[\[\(][Ss]erious[\)\]]", "[Serious]", row[0])
    posts_new.append(row)
print(posts_new[0:10])<jupyter_output>[['What\'s your internet "white whale", something you\'ve been searching for years to find with no luck?', '11510', '1433213314.0', '1', '26195'], ["What's your favorite video that is 10 seconds or less?", '8656', '1434205517.0', '4', '8479'], ['What are some interesting tests you can take to find out about yourself?', '8480', '1443409636.0', '1', '4055'], ["PhD's of Reddit. What is a dumbed down summary of your thesis?", '7927', '1440188623.0', '0', '13201'], ['What is cool to be good at, yet uncool to be REALLY good at?', '7711', '1440082910.0', '0', '20325'], ['[Serious] Redditors currently in a relationship, besides dinner and a movie, what are your favorite activities for date night?', '7598', '1439993280.0', '2', '5389'], ["Parents of Reddit, what's something that your kid has done that you pretended to be angry about but secretly impressed or amused you?", '7553', '1439161809.0', '0', '11520'], ['What is a good subreddit to binge read the All Time Top Posts of?', '7498', '143882[...]<jupyter_text>Matching years with regex To match numbers and alphabetical letters, the following regexes are used respectively: **[0-9]** and **[a-z]** or **[A-Z]** for capital letters.<jupyter_code># Example
strings = ['he throws his bat in 1995', 'robot']
year_string = []

for string in strings:
    if re.search("[1-2][0-9]{3}", string) is not None: # [1-2][0-9]{3} <==> [1-2][0-9][0-9][0-9]
        year_string.append(string)
year_string<jupyter_output><empty_output><jupyter_text>Extract all years To extract characters, it is also possible to use the **findall()** method of the **re** module.<jupyter_code># Example 1
re.findall("[a-z]", "abcd1234")
# Example 2
years_string = "We are already in 2022, one year more than 2021 and less than 2023!"
years = re.findall("[2][0-9][0-9][0-9]", years_string)
years<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14528276050937972, 'fraction_numerical': 0.05792140216349445, 'mean_word_length': 5.189830508474576, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '646203', 'license': 'MIT', 'n_tokens_mistral': 2659, 'n_tokens_neox': 2389, 'n_words': 873, 'path': 'courses/14. Regular expressions in Python.ipynb', 'repo_name': 'Niangmohamed/The-Fundamentals-of-Python', 'size': '7303'}