<jupyter_start><jupyter_text>Unit 1  Chapter 1  ---<jupyter_code>import re
import logging
from importlib import reload
reload(logging)
import sys
logging.basicConfig(format='Explanation | %(levelname)s : %(message)s', level=logging.INFO, stream=sys.stdout)
log = logging.getLogger("Zero to Hero in NLP")<jupyter_output><empty_output><jupyter_text>Regular Expressions> Regular expressions can be used to specify pattern of strings we might want to extract from a document. Text Normalization> Text normalization is a set of tasks used to convert text into a more convenient, and standard form.* Tokenization    * A method to seperate words from running texts.        *But tokenization is much more than just seperating words*        * For processing tweets or texts we’ll need to tokenize emoticons like :) or hashtags like nlproc.    ---* Lemmatization    * The task of determining that two words have the same root, despite their surface differences. For example, sang, sung, sing have the same root sing.     ℹ️ The word *sing* is known as a lemma    ---* Stemming     * Stemming refers to a simpler version of lemmatization in which we mainly just strip suffixes from the end of the word.    ---* Sentence segmentation    * Sentence segmentation is the method of breaking up individual sentences using cues like point(.), exclamation mark(!) and more.    ---     Edit Distance> Edit distance a metric that measures similarity between two strings based on the number of edits(insertion, deletion, substitution).    ---English words are often seperated by whitespaces, but it's not always the case. For example, 'New York' & 'rock 'n' roll' shall be treated as single large words rather than splitting on *New* & *York*. A good reason to not rely on python's split() method for tokenization --- Regular Expressions * Regular expressions are useful when we have a *pattern* to search within a *corpus* of text.---Corpus - A collection of documents or a single document.---* A regular expression search function will search through the corpus, returning all texts that match the pattern.* Here, regular expressions are delimited by backslashes(/). ---A simple regular expression would be a sequence of characters.For example, /woodchuck/, /buttercup/> Couple of things to pay attention to.    * Regular expressions are case-sensitive. 'Woodchuck' does not match the regex /woodchuck/    * Note how the regex is delimited by /.   Square brackets [ ]> The string of characters inside square braces denote disjunction of characters to match. In other words, specifies an OR condition.For example,/[wW]oodchuck/ - This regex matches woodchuck or Woodchuck./[abc]/ - Matches sequence having a, b, or c. Matches 'Hi my name is John', but not 'A good night'  Dash -> Characters seperated by dash denote a range.For example,/[2-5]/ - A range of 2 to 5. 2, 3, 4, or 5./[a-z]/ - A range of lowercase English alphabets. a,b,c,...,z. Matches 'a good day.'/[A-Z]/ - A range of uppercase English alphabets. A,B,C,...,Z. Matches 'Best day of my life.'  Caret ^> If caret is the first character inside a square bracket, the sequence after it is negated. In other words, all the characters are matched except the ones after caret.For example,/[^A-Z]/ - Matches lowercase letters but not uppercase. Matches 'I WAS GOING TO the market'./[^Ss]/ - Neither S nor s. Matches 'I had a bad day.''Sometimes I wish to escape'./[A-Z]/ - A range of uppercase English alphabets. A,B,C,...,Z. Matches 'Best day of my life.'  Question mark ?> If specified after a character, it says match or nothing. In other words character before ? is optionalFor example,/[woodchucks?]/ - Matches with string either having 's' after woodchuck or not at all. Matches woodchuck or woodchucks/[colou?r]/ - Matches with string either having 'u' in colour or not having 'u' at all. Matches color or colour.<jupyter_code>print(re.findall(r'[Ss]ome','sometimes I feel very energetic. Somedays not.')) 
log.info(' Matches Some and some')
log.info("The findall() method returns all the words that match a pattern in a corpus. search() returns the first point of match")
print(re.findall(r'h[abc][vd]','I have had a good day in the cabin ha of my backyard.')) 
log.info(' Matches words starting with h, having a,b, or c and ending with v or d')

print(re.findall(r'[A-Z]','This is the Besttttt day of my Liffeee')) 
log.info(' Matches uppercase letters.')

print(re.findall(r'[^A-Z]','This is  the Besttttt day of my Liffeee')) 
log.info(' Matches everything except uppercase letters.')
print(re.findall(r'[^T]','This is  the Besttttt day of my Liffeee')) 
log.info(' Matches everything except uppercase T.')
print(re.findall(r'colou?r','Bright colors and colours')) 
log.info(' Matches if string has color or colour')
print(re.findall(r'[A-Za-z]unny days?','sunny day vs sunny days vs Sunny day vs Sunny days vs funny day')) 
log.info(' Matches if string has anything with unny day or unny days')<jupyter_output>['sunny day', 'sunny days', 'Sunny day', 'Sunny days', 'funny day']
Explanation | INFO :  Matches if string has anything with unny day or unny days<jupyter_text>Kleene *> The Kleene star means “zero or more occurrences of the immediately previous character or regular expression”.For example,/a*/ - Matches zero or more 'a'. Matches 'baaaaaaa' and 'Hello'/aa*/ - Matches one or more 'a'. Matches 'baaaaaa' but not 'Hello' because atleast one 'a' should be there<jupyter_code>print(re.findall(r'a*','Hello')) 
log.info(' Zero \'a\' found')
print(re.findall(r'aa*','Hello')) 
log.info(' No \'a\' found')
print(re.findall(r'[ab]*','Baaaaaa & aaaaaaa and a')) 
log.info('Matches if string has zero or more \'a\' or \'b\' ')
print(re.findall(r'[ab]*','bbbbbb')) 
log.info(' Matches if string has zero or more \'a\' or \'b\' ')<jupyter_output>['bbbbbb', '']
Explanation | INFO :  Matches if string has zero or more 'a' or 'b'<jupyter_text>Kleene +> The Kleene + means “one or more occurrences of the immediately previous character or regular expression”.For example,/a+/ - Matches one or more 'a'. Matches 'baaaaaaa' but not 'Hello'/[0-9]+/ - Matches sequence of digits<jupyter_code>print(re.findall(r'[0-9]+','My phone number is 9457068769')) 
log.info(' Matches sequence of digits')<jupyter_output>['9457068769']
Explanation | INFO :  Matches sequence of digits<jupyter_text>Period .> The period specifies any single character (wildcard).For example,/beg.n/ - Matches for any single character betwee'beg' and 'n'<jupyter_code>print(re.findall(r'beg.n','begin vs begun vs beg\'n')) 
log.info(' Matches anything with "beg" and "n"')
print(re.findall(r't.*d','tday vs timid vs ticked vs topped vs top')) 
log.info(' Matches sequence starting with "t" and ending with "d"')
print(re.findall(r'beg.n','It is the beginning')) 
log.info(' Matches anything with "beg" and "n"')<jupyter_output>['begin']
Explanation | INFO :  Matches anything with "beg" and "n"<jupyter_text>Anchors > Anchors are special characters that anchor regular expressions to particular places in a string.* ^ (caret) specifies the start of line.* $ (dollar) matches end of line.* \b matches a word boundary.* \B matches a non-word boundary<jupyter_code>print(re.findall(r'^The','The is a very common word in English')) 
log.info(' Matches if "The" is at the beginning of string')
print(re.findall(r'^the','The is the most common word in English')) 
log.info(' Matches if "the" is at the beginning of string. Note that "the" is present but not matched.')
print(re.findall(r'the$','Pattern you are looking for is -> the')) 
log.info(' Matches if "the" is at the end of string.')
print(re.findall(r'the$','You are the winner')) 
log.info(' Matches if "the" is at the end of string. Notice that "the" is present but not matched')
print(re.findall(r'\bthe\b','the vs other')) 
log.info(' Matches words having only "the". Notice that "the" is present in "other" but not matched')
print(re.findall(r'\bthe\b','the vs other'))<jupyter_output>['the']<jupyter_text>Disjunction, Grouping, & Precedence  Disjunction operator |> Disjunction operator is used to search for more than one string. Example, you want to search for "cats" or "dogs". You cannot use /[catsdogs]/, instead use /cats|dogs/For example,/cats|dogs/ - This regex matches cats or dogs.<jupyter_code>re.findall(r'cats|dogs','I have two cats and 3 dogs')
re.search(r'cats|dogs','I have two cats and 3 dogs')
print(re.findall(r'[0-9]+|[A-Z]','this will return empty list'))
log.info(" It returns empty list because there are no string of digits or(|) uppercase letters")
print(re.findall(r'[0-9]+|[A-Z]','this will return something because it has num3er5'))
log.info(" It returns where it finds a sequence of one or more digits")
print(re.findall(r'[0-9]+|[A-Z]','this will return something because it has num3er5 as well as UPPERCASE LETTERS'))
log.info(" It returns where it finds a sequence of one or more digits or uppercase letters")
print(re.findall(r'[0-9]+|[A-Z]+','this will return something because it has num3er5 as well as UPPERCASE LETTERS'))
log.info(" Notice the differece in output due to Kleene+")<jupyter_output>['3', '5', 'UPPERCASE', 'LETTERS']
Explanation | INFO :  Notice the differece in output due to Kleene+<jupyter_text>Precedence ( )> Enclosing a pattern inside () makes it act like a single character. For example, If you want to match 'guppy' or 'guppies', you can use /gupp(y | ies)/ instead of /guppy | guppies/.<jupyter_code>print(re.search(r'gupp(y|ies)','I have a lot of guppies and she has one guppy'))
log.info(" Using search() here to demonstrate. search() returns when the first match is found.")<jupyter_output><re.Match object; span=(16, 23), match='guppies'>
Explanation | INFO :  Using search() here to demonstrate. search() returns when the first match is found.<jupyter_text>--- Operator precedence hierarchy<jupyter_code>from prettytable import PrettyTable
table = PrettyTable()
table.field_names = ('Operator Name','Operator')
table.add_row(["Paranthesis", '()'])
table.add_row(["Counters", '* + ? {}'])
table.add_row(["Disjunction", '|'])
print(table)<jupyter_output>+---------------+----------+
| Operator Name | Operator |
+---------------+----------+
|  Paranthesis  |    ()    |
|    Counters   | * + ? {} |
|  Disjunction  |    |     |
+---------------+----------+<jupyter_text>* It's easy to miss some strings while using regular expressions because one patterns match may miss another type of pattern.---> There are two kinds of errors: * false positives : strings that we incorrectly matched.* false negatives, strings that we incorrectly missed. Reducing the overall error rate for an application thus involves two antagonistic efforts:• Increasing precision (minimizing false positives)     • Increasing recall (minimizing false negatives)  Substitution operator s/> Substitution operator is used to substitute a regex with a given pattern.For example, If you want to substitute 'guppy' with 'guppies', you can use s/guppy/guppies/.*Python code below*<jupyter_code>re.sub(pattern=r'guppy',string='I have 4 guppy',repl='guppies')<jupyter_output><empty_output><jupyter_text>--- Capture group> This use of parentheses to store a pattern in memory is called a capture group. Every time a capture group is used (i.e., parentheses surround a pattern), the re- sulting match is stored in a numbered register. If you match two different sets of parentheses, \2 means whatever matched the second capture group.For example, /the (.*)er they (.*), the \1er we \2/The above regex will match 'the faster they ran, the faster we ran' but not 'the faster they ran, the faster we ate'. ![capture group](../images/capture_group.png) * The first (.)er means anything that ends with er will match and stored in \1er.* The second (.) means whatever is stored in (.) will have to match exactly at \2.<jupyter_code>re.findall(r'^(?!Volcano)[A-Za-z]+','Volcano has erupted')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12375115439509697, 'fraction_numerical': 0.005205272437242885, 'mean_word_length': 4.5301764159702875, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '64914', 'license': 'Apache-2.0', 'n_tokens_mistral': 3713, 'n_tokens_neox': 3539, 'n_words': 1513, 'path': 'Unit 1 - Introduction/.ipynb_checkpoints/Chapter 1 | Regular Expressions-checkpoint.ipynb', 'repo_name': 'samacker77/NLP-Starter-Bundle', 'size': '11911'}