<jupyter_start><jupyter_text>Regular Expressions (RegEx) A regular expression, regex or regexp is a sequence of characters that define a search pattern. ![regex](https://miro.medium.com/max/1200/1*ZVlIZ1ZYC6rASz-dYPzhZQ.jpeg) **references**- https://docs.python.org/3/howto/regex.html- https://www.datacamp.com/community/tutorials/python-regular-expression-tutorial- https://www.dataquest.io/blog/regular-expressions-data-scientists/- https://www.kaggle.com/rtatman/fraudulent-email-corpus**may save your life**- https://regex101.com/  First things first For the standard case **import re** should be enough. For the later case **pip3/pip install regex** should install it.<jupyter_code>import re
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Sintax Special Characters:- `.` Matches any character except a newline.- `^` Matches the start of the string.- `$` Matches the end of the string or just before the newline at the end of the string.- `*` Matches 0 or more repetitions of the preceding RE.- `+` Matches 1 or more repetitions of the preceding RE.- `?` Matches 0 or 1 repetitions of the preceding RE.- `? https://www.regular-expressions.info/lookaround.html- NOTA: re.M -> modo multilinea  Special Sequences:- **Literals** `a` - **Alternation** `a|b`- **Character sets** `[ab]`, `[^ab]` <- we use the hat between brackets to indicate that we want the opposite- **Wildcards** `.`- **Escape special characters** `\` (?,*,+,^,$)- **Ranges** `[a-d]`, `[1-9]`, `[A-D]`- **Quantifiers** `{2}`, `{2,}`, `{2,4}`, `?`, `*`, `+`- **Grouping** `()`- **Anchors** `^`, `$`- **Character classes** `\w`, `\d`, `\s`, `\n`, `\W`, `\D`, `\S`**\w** - Matches any alphanumeric character (digits and alphabets). Equivalent to `[a-zA-Z0-9_]`. By the way, underscore _ is also considered an alphanumeric character. **\d** - Matches any digit. Equivalent to `[0-9]` **\s** - Matches where a string contains any whitespace character. Equivalent to `[ \t\n\r\f\v]`**\W** - Matches any non-alphanumeric character. Equivalent to `[^a-zA-Z0-9_]`**\D** - Matches any non digit. Equivalent to `[^0-9]` **\S** - Matches where a string contains any non-whitespace character. Equivalent to `[^ \t\n\r\f\v]`  Methods  re.sub(pattern, repl, string, count=0)Replaces one or many matches with a string<jupyter_code>txt = "gabriel, DDio & CClara are TA's??"
a = '2'
#re.sub
#Literals
txt = re.sub('[A-Z]{2}', a, txt)
txt
#Ranges
re.sub('[A-Z]','',txt)
#Escape special character, quantifiers
re.sub('\?{2}','.',txt)<jupyter_output><empty_output><jupyter_text>re.search(pattern, string, flags=0)Scan through a string, looking for any location where this RE matches. If the search is succesful, `re.search()` returns a match object. Otherwise, it returns `None`.<jupyter_code>#re.search
txt = "The rain in Spain and the sun in England, Tower"
x = re.search("^The.*Spain", txt) 
x.span()
txt = "The rain in Spain"
#\b whole words only
x = re.search(r"\bS\w+", txt)
print(x)
print(x.span())
#returns a tuple containing the start-, and end positions of the match
print(x.start())
#contains the start position of the match
print(x.end())
#contains the end position of the match
print(x.string)
#print the string passed into the function (variable 'txt')
print(x.group())
#Print the part of the string where there was a match
for i in range(x.span()[0], x.span()[1]):
    print(x.string[i])
print(re.search(r'r\w+\s\w+', txt))
print(re.search(r'R\w*', txt))
print(re.search(r'T\w*', txt))
print(re.search(r'^t\w*', txt))<jupyter_output><re.Match object; span=(4, 11), match='rain in'>
None
<re.Match object; span=(0, 3), match='The'>
None<jupyter_text>re.match(pattern, string)Determine if the RE matches at the beginning of the string.<jupyter_code>#re.match
pattern = r"Cookie"
sequence = "I want a Cookie"
sequence2= "Cookie, I want you!"
if re.match(pattern, sequence2):
    print("Match!")
else: 
    print("Not a match!")
txt = "The rain in Spain"
#matches at the beginning of the string
print(re.match(r'r\w*', txt))
print(re.match(r'^r\w*', txt))

print(re.search(r'^T\w*', txt)) # search with ^ is the same as match

print(re.match(r'T\w*', txt))
email_address = 'Please contact us at: <EMAIL>'
matchi = re.search(r'(\w+)@([\w\.]+)', email_address)
if matchi:
    print(matchi.group(0)) # The whole matched text
    print(matchi.group(1)) # The username (group 1)
    print(matchi.group(2)) # The host (group 2)<jupyter_output><EMAIL>
support
thebridge.com<jupyter_text>re.fullmatch(pattern, string)<jupyter_code>class_names = ["Andrea", "Ariadna", "Estela", "Anais", "Xeles", "Maria", "Mar"]
for name in class_names:
    if re.fullmatch("Maria", name):
        print(f"{name} is desired name")
    else:
        print(f"{name} is not desired name")<jupyter_output>Andrea is not desired name
Ariadna is not desired name
Estela is not desired name
Anais is not desired name
Xeles is not desired name
Maria is desired name
Mar is not desired name<jupyter_text>re.findall (pattern, string)Find all substrings where the RE matches, and returns them as a list.<jupyter_code>#re.findall
email_address = "Please contact us at: <EMAIL>, <EMAIL>"

#'addresses' is a list that stores all the possible match
addresses = re.findall(r'\w+\.?\w+@[\w\.-]+', email_address)
addresses
print(re.findall('[^aeiou\s]',email_address))
print(re.findall('\sc\w*',email_address))
print(re.findall('^P\w*',email_address))
with open('info.txt', 'r') as file:
    client_info = file.read()
emails_clients=re.findall(r"[\w.]+@[\w.]+", client_info)
print(emails_clients[:5])
client_numbers_normal = re.findall(r"[0-9]{2}-\d{3}-\d{3}-\d{3}", client_info)
print(client_numbers_normal[:5])
client_numbers = re.findall(r"(([0-9]{2}-)\d{3}-\d{3}-\d{3})", client_info)
print(client_numbers[:5])<jupyter_output>[('34-739-941-941', '34-'), ('34-278-870-242', '34-'), ('34-999-876-292', '34-'), ('34-345-887-949', '34-'), ('34-905-089-682', '34-')]<jupyter_text>re.split(pattern, string, maxsplit=0)Returns a list where the string has been split at each match<jupyter_code>#re.split
sente = "Hello,\n Please, contact me the sooner.\n Thank you,\n Me"
reg = re.split("\n", sente)
reg
"".join(reg)<jupyter_output><empty_output><jupyter_text>https://stackoverflow.com/questions/10804732/what-is-the-difference-between-and-in-regex<jupyter_code>client_list = re.split(r"(?<=[0-9]{2}-\d{3}-\d{3}-\d{3})",client_info)
print(client_list[:3])<jupyter_output>['<NAME> <EMAIL> 2518 Nulla Road 34-739-941-941', ' <NAME> <EMAIL> 8453 Nostra, St. 34-278-870-242', ' Zephania Copeland <EMAIL> P.O. Box 733, 3179 Ligula. Av. 34-999-876-292']<jupyter_text>re.compile(patternCompiles a RE into a regular expression object.<jupyter_code>name_check = re.compile(r"[^A-Za-z ]")
name = input("Please insert your name:")
while name_check.search(name):
    # it loops while if finds a match
    print("Please enter your name correctly!")
    name = input("Please insert your name:")
print("Finally mate, I thought you'd never do it")<jupyter_output>Please insert your name:<NAME>
Please enter your name correctly!
Please insert your name:clara+piniella
Please enter your name correctly!
Please insert your name:Clara.piniella
Please enter your name correctly!
Please insert your name:7835djfa
Please enter your name correctly!
Please insert your name:898593*+
Finally mate, I thought you'd never do it<jupyter_text>-----------------------------------------------------------------------------------------------------------  Some practice Now is your turn.  Simple validation of a usernamehttps://www.codewars.com/kata/56a3f08aa9a6cc9b75000023    Write a simple regex to validate a username. Allowed characters are:- lowercase letters,- numbers,- underscoreLength should be between 4 and 16 characters (both included).<jupyter_code># Maria team
def validate_usr(username):
    namecheck = re.compile(r"[^a-z0-9_]{4,16}")
    while namecheck.search(username):
        print("Please enter your name correctly!")
        username = input("Please insert your name:")
    print("Finally mate, I thought you'd never do it")
    
validate_usr(username=input("Enter a username: "))
# Alex team
def validate_usr(username):
    str_list = re.findall("[a-z]|[0-9]|_",username)
    compare = ""
    compare = compare.join(str_list)
    if compare == username and len(compare)<=16 and len(compare)>4:
        return True
    else:
        return False

username = "clara"
test = validate_usr(username)
test
# your solution
def validate_usr(username):
    if re.match(r'^[a-z\d_]{4,16}$', username):
        return True
    else:
        return False
validate_user = lambda string: bool(re.match('^[a-z\d_]{4,16}$', string))
check_usernames = ["unusuario", "UNUSUARIO", "UN_usuario", "un_usuario76", "8934_38875aa"]
check_usernamesbool = [validate_user(e) for e in check_usernames]
check_usernamesbool<jupyter_output><empty_output><jupyter_text>Regex validate PIN code https://www.codewars.com/kata/55f8a9c06c018a0d6e000132ATM machines allow 4 or 6 digit PIN codes and PIN codes cannot contain anything but exactly 4 digits or exactly 6 digits.If the function is passed a valid PIN string, return true, else return false.Examples:```python"1234"   -->  True"12345"  -->  False"a234"   -->  False```<jupyter_code># Javi Gil team
pin = re.compile(r"^[0-9]{4,6}$")
enter_pin= input("Please insert your pin:")
while not (pin.search(enter_pin)):
    # it loops while if finds a match
    print("venga zoquete a ver si saca el regex regex")
    enter_pin = input("Please insert your pin:")
print("Finally mate, I thought you'd never do it")

# esta no valdría porque coge los de 5
# Alfon team
pin_check = re.compile(r"^[0-9]{4}$|^[0-9]{6}$")

def validate_usr(pin):
    while not pin_check.search(pin):
        print("'Pin incorrecto'")
        pin = input('Dame un pin')
    print("Pin correcto")


validate_usr(pin=input("Insert a number"))

#aprobado!
# your solution
def validate_pin(pin):
    if re.fullmatch("\d{4}|\d{6}", str(pin)):
        return True
    else:
        return False
pin = 7847
pins = ["8475", 8735, 9858473485, "fklajkfd", pin]
checkpins = [validate_pin(pi) for pi in pins]
checkpins
list(filter(validate_pin, pins))<jupyter_output><empty_output><jupyter_text>-----------------------------------------------------------------------------------------------------------  The FBI challenge - https://www.fbi.gov/scams-and-safety/common-fraud-schemes/nigerian-letter-or-419-fraud- https://www.kaggle.com/rtatman/fraudulent-email-corpus It's your first day at the FBI office and your boss has just send you an `txt` file: `emails.txt`, she asked you to make some analysis but first of all, you need to get a dataframe like the following. You'll need some python knowledge and some regex for that goal.<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>---------------------------------------------------------------------------------------------------------------------------  Since we are good people, here you have a proposed solution<jupyter_code>emails_info={}
fh = open("emails.txt", "r").read()
fh.count("From r")
contents = re.split(r"From r", fh)
contents[0]
contents.pop(0)<jupyter_output><empty_output><jupyter_text>- sender_email- sender_name- date_sent- time_sent- subject  Info Sender<jupyter_code>info_sender=[]
for i,e in enumerate(contents):
    try:
        info_sender.append(re.search("From:.*", e).group())
    except: 
        info_sender.append("not found")
len(info_sender)
#sender_email
emails_info['sender_email']=[]
for line in info_sender:
    res=re.findall(r'[\w\.]+@[\w\.-]+', line)
    if res:
        emails_info['sender_email'].append(res[0])
    else:
        emails_info['sender_email'].append(np.nan)
        
len(emails_info['sender_email'])
#sender name
emails_info['sender_name']=[]
for line in info_sender:
    res=re.findall(r':.*<', line)
    if res:
        emails_info['sender_name'].append(res[0][1:-1])
    else:
        emails_info['sender_name'].append(np.nan)
len(emails_info['sender_name'])<jupyter_output><empty_output><jupyter_text>Info Dates<jupyter_code>#DATES
dates=[]
for i,e in enumerate(contents):
    try:
        dates.append(re.search("Date:.*", e).group())
    except: 
        dates.append("not found")
len(dates)
#email date
emails_info['date_sent']=[]
for dat in dates:
    res=re.findall(r"\d+\s\w{3}\s\d+", dat)
    if res:
        emails_info['date_sent'].append(res[0])
    else:
        emails_info['date_sent'].append(np.nan)

len(emails_info['date_sent'])
emails_info['time_sent']=[]
for dat in dates:
    res=re.findall(r"\d{2}:\d{2}:\d{2}", dat)
    if res:
        emails_info['time_sent'].append(res[0])
    else:
        emails_info['time_sent'].append(np.nan)

len(emails_info['time_sent'])<jupyter_output><empty_output><jupyter_text>Subject<jupyter_code>subject=[]
for i,e in enumerate(contents):
    try:
        subject.append(re.search("Subject:.*", e).group())
    except: 
        subject.append("not found")
len(subject)
emails_info['subject']=[]
for sub in subject:
    res=re.findall(r":.*", sub)
    if res:
        emails_info['subject'].append(res[0][2:])
    else:
        emails_info['subject'].append(np.nan)

len(emails_info['subject'])<jupyter_output><empty_output><jupyter_text>Creating DataFrame<jupyter_code>df=pd.DataFrame(emails_info)
df.isnull().sum()
df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1852434456928839, 'fraction_numerical': 0.0250187265917603, 'mean_word_length': 4.182841614906832, 'pattern_counts': {'":': 1, '<': 79, '<?xml version=': 0, '>': 80, 'https://': 12, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '435573', 'license': 'Apache-2.0', 'n_tokens_mistral': 4891, 'n_tokens_neox': 4624, 'n_words': 1201, 'path': 'week4/day5/theory/regex/regex_nov2020.ipynb', 'repo_name': 'JavierOlcoz/data_science_nov_2020', 'size': '13527'}