<jupyter_start><jupyter_text>Regular ExpressionsRegular expression is a set of characters, called as the pattern, which helps in finding substrings in a given string. The pattern is used to detect the substringsFor example, suppose you have a dataset of customer reviews about your restaurant. Say, you want to extract the emojis from the reviews because they are a good predictor os the sentiment of the review.Take another example, the artificial assistants such as Siri, Google Now use information retrieval to give you better results. When you ask them for any query or ask them to search for something interesting on the screen, they look for common patterns such as emails, phone numbers, place names, date and time and so on. This is because then the assitant can automatically make a booking or ask you to call the resturant to make a booking.Regular expressions are very powerful tool in text processing. It will help you to clean and handle your text in a much better way.  Let's import the regular expression library in python.<jupyter_code>import re<jupyter_output><empty_output><jupyter_text>Let's do a quick search using a pattern.<jupyter_code>re.search('Ravi', 'Ravi is an exceptional student!')
# print output of re.search()
match = re.search('Ravi', 'Ravi is an exceptional student!')
print(match.group())<jupyter_output>Ravi<jupyter_text>Let's define a function to match regular expression patterns<jupyter_code>def find_pattern(text, patterns):
    abc = re.search(patterns, text)
    if abc:
        return abc
    else:
        return 'Not Found!'<jupyter_output><empty_output><jupyter_text>Quantifiers<jupyter_code># '*': Zero or more 
print(find_pattern("ca", "a*"))
print(find_pattern("abc", "a*"))
print(find_pattern("abbc", "ab*"))
# '?': Zero or one (tells whether a pattern is absent or present)
print(find_pattern("ac", "ab?"))
print(find_pattern("abc", "ab?"))
print(find_pattern("abbc", "ab?"))
# '+': One or more
print(find_pattern("ac", "ab+"))
print(find_pattern("abc", "ab+"))
print(find_pattern("abbc", "ab+"))
# {n}: Matches if a character is present exactly n number of times
print(find_pattern("abbc", "ab{2}"))

# {m,n}: Matches if a character is present from m to n number of times
print(find_pattern("aabbbbbbc", "ab{3,5}"))   # return true if 'b' is present 3-5 times
print(find_pattern("aabbbbbbc", "ab{7,10}"))  # return true if 'b' is present 7-10 times
print(find_pattern("aabbbbbbc", "ab{,10}"))   # return true if 'b' is present atmost 10 times
print(find_pattern("aabbbbbbc", "ab{10,}"))   # return true if 'b' is present from at least 10 times<jupyter_output>Found a match!
Not Found!
Found a match!
Not Found!<jupyter_text>Anchors<jupyter_code># '^': Indicates start of a string
# '$': Indicates end of string

print(find_pattern("James", "^J"))   # return true if string starts with 'J' 
print(find_pattern("Pramod", "^J"))  # return true if string starts with 'J' 
print(find_pattern("India", "a$"))   # return true if string ends with 'c'
print(find_pattern("Japan", "a$"))   # return true if string ends with 'c'<jupyter_output><_sre.SRE_Match object; span=(0, 1), match='J'>
Not Found!
<_sre.SRE_Match object; span=(4, 5), match='a'>
Not Found!<jupyter_text>Wildcard<jupyter_code># '.': Matches any character
print(find_pattern("a", "."))
print(find_pattern("#", "."))<jupyter_output>Found a match!
Found a match!<jupyter_text>Character sets<jupyter_code># Now we will look at '[' and ']'.
# They're used for specifying a character class, which is a set of characters that you wish to match.
# Characters can be listed individually as follows
print(find_pattern("a", "[abc]"))

# Or a range of characters can be indicated by giving two characters and separating them by a '-'.
print(find_pattern("c", "[a-c]"))  # same as above
# '^' is used inside character set to indicate complementary set
print(find_pattern("a", "[^abc]"))  # return true if neither of these is present - a,b or c<jupyter_output>Not Found!<jupyter_text>Character sets| Pattern  | Matches                                                                                    ||----------|--------------------------------------------------------------------------------------------|| [abc]    | Matches either an a, b or c character                                                      || [abcABC] | Matches either an a, A, b, B, c or C character                                             || [a-z]    | Matches any characters between a and z, including a and z                                  || [A-Z]    | Matches any characters between A and Z, including A and Z                                  || [a-zA-Z] | Matches any characters between a and z, including a and z ignoring cases of the characters || [0-9]    | Matches any character which is a number between 0 and 9                                    |  Meta sequences| Pattern  | Equivalent to    ||----------|------------------|| \s       | [ \t\n\r\f\v]    || \S       | [^ \t\n\r\f\v]   || \d       | [0-9]            || \D       | [^0-9]           || \w       | [a-zA-Z0-9_]     || \W       | [^a-zA-Z0-9_]    |  Greedy vs non-greedy regex<jupyter_code>print(find_pattern("aabbbbbb", "ab{3,5}")) # return if a is followed by b 3-5 times GREEDY
print(find_pattern("aabbbbbb", "ab{3,5}?")) # return if a is followed by b 3-5 times GREEDY
# Example of HTML code
print(re.search("<.*>","<HTML><TITLE>My Page</TITLE></HTML>"))
# Example of HTML code
print(re.search("<.*?>","<HTML><TITLE>My Page</TITLE></HTML>"))<jupyter_output><_sre.SRE_Match object; span=(0, 6), match='<HTML>'><jupyter_text>The five most important re functions that you would be required to use most of the times arematch() Determine if the RE matches at the beginning of the stringsearch() Scan through a string, looking for any location where this RE matchesfinall() Find all the substrings where the RE matches, and return them as a listfinditer() Find all substrings where RE matches and return them as asn iteratorsub() Find all substrings where the RE matches and substitute them with the given string<jupyter_code># - this function uses the re.match() and let's see how it differs from re.search()
def match_pattern(text, patterns):
    if re.match(patterns, text):
        return re.match(patterns, text)
    else:
        return ('Not found!')
print(find_pattern("abbc", "b+"))
print(match_pattern("abbc", "b+"))
## Example usage of the sub() function. Replace Road with rd.

street = '21 Ramakrishna Road'
print(re.sub('Road', 'Rd', street))
print(re.sub('R\w+', 'Rd', street))
## Example usage of finditer(). Find all occurrences of word Festival in given sentence

text = 'Diwali is a festival of lights, Holi is a festival of colors!'
pattern = 'festival'
for match in re.finditer(pattern, text):
    print('START -', match.start(), end="")
    print('END -', match.end())
# Example usage of findall(). In the given URL find all dates
url = "http://www.telegraph.co.uk/formula-1/2017/10/28/mexican-grand-prix-2017-time-does-start-tv-channel-odds-lewisl/2017/05/12"
date_regex = '/(\d{4})/(\d{1,2})/(\d{1,2})/'
mr = '/\d{4}/\d{2}/\d{2}'
print(re.findall(date_regex, url))
## Exploring Groups
m1 = re.search(date_regex, url)
print(m1.group())  ## print the matched group
print(m1.group(1)) # - Print first group
print(m1.group(2)) # - Print second group
print(m1.group(3)) # - Print third group
print(m1.group(0)) # - Print zero or the default group<jupyter_output>/2017/10/28/
{'fraction_non_alphanumeric': 0.14631197097944376, 'fraction_numerical': 0.01262931613596668, 'mean_word_length': 3.5197328476017002, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '222538', 'license': 'MIT', 'n_tokens_mistral': 2361, 'n_tokens_neox': 2266, 'n_words': 833, 'path': 'Regular+Expressions.ipynb', 'repo_name': 'aryancodify/NaturalLanguageProcessing', 'size': '7443'}