<jupyter_start><jupyter_text>Parse the scripts, retrieve a list of all the Characters and then Create a list for every character.Exercise two has a function for getting all the episodes that returns a two column data frame of Character / Dialogue.   Grab the Character Series pull all the unique values, iterate through that list and pull separate dialog out.<jupyter_code>import pandas as pd
import re
season_episode_list = [13, 18, 17, 17, 14, 17, 15, 14, 14, 14, 14, 14, 14, 14, 14, 14, 10, 10, 10, 10, 10]
script_urls = []

for season in range(21):
    for episode in range(season_episode_list[season]):
        script_urls.append("http://southparkwillie.com/Season{0}/E{0}{1:0>2}script.htm".format(season + 1, episode + 1))
#This is our function for getting all the scripts and returning a dataframe
def create_script_dataframe(script_urls):
    '''
    Function to create a pandas dataframe with two columns - "Character" / "Dialogue"
    script_urls - list of URLS
    
    return: pandas dataframe
    '''
    #Create a returnable Dataframe
    all_scripts_df = pd.DataFrame()
    
    #Loop through the URLs, grab the scripts and clean up
    for url in script_urls:
        script_list = pd.read_html(url)
        script_df = script_list[0]
        script = script_df.iloc[:,0:2]
        script = script.dropna()
        all_scripts_df = all_scripts_df.append(script, ignore_index = True)
        
    all_scripts_df.columns = ['Character', 'Dialogue']
    return all_scripts_df
#This is our function for 
def get_character_dialogue(character, script):
    '''
       Retrieve the 'character' dialogue from the 'script'
       character: string
       script: pandas data frame
       
       Return: list of dialogue strings
    '''
    #TODO: Once more cleaning has been done on the Panda's side, this next line shoule not be required.
    character = character.title() + ":"
    
    #Initialize a list to return
    dialogue = []
    pattern = '\[.*?\]'
    
    for row_index in script.index.values:
        if script.loc[row_index, 'Character'] == character:
            dialogue.append(re.sub(pattern, '', script.loc[row_index, 'Dialogue']))
            
    return dialogue
#List all the characters
from pprint import pprint

df = create_script_dataframe(script_urls)
pprint(len(df.Character.unique()))
for character in df.Character.unique():
    pprint(character)<jupyter_output>4497
'Boys:'
'Kyle:'
'Ike:'
'Cartman:'
'Stan:'
'Kenny:'
'Chef:'
'Ms. Crabtree:'
'<NAME>. Crabtree:'
'Farmer:'
'Officer Barbrady:'
'Mr. Garrison:'
'Mr. Hat:'
'DEPARTS'
'Conductor:'
'THE FOUR FOOD GROUPS'
'Kid 1:'
'Kid 2:'
'Wendy:'
'<NAME>:'
'Ms. Cartman:'
'Reporter:'
'Kitty:'
'Alien:'
'Moo… Moo…Moo…Moo…'
'Cows:'
'Moo??'
'Moo moo, moo.'
'Moo moo, moo. Moo.'
'Cow:'
'Moo moo?'
'Alien Carl:'
'Moo…moo…moo.'
'Moo moo.'
'Moo moo. Moo.'
'Moo moo. Moo moo.'
'Blonde:'
'<NAME>:'
'Mayor:'
'Aide 1:'
'Aide 2:'
'<NAME>:'
'<NAME>:'
'Show Announcer:'
'<NAME>:'
'TV Announcer:'
'Beefcake:'
'Mrs. Cartman:'
'Clyde:'
'Bebe:'
'Townsman:'
'Jimbo:'
'Townswoman:'
'<NAME>:'
'Agent 1:'
'Agent 2:'
'TV Crew Director:'
'Townsman 1:'
'Townsman 2:'
'Geraldo:'
'Ned:'
'Randy:'
'Johnson:'
'Ted:'
'Newscaster:'
'Crowd:'
'Individual in crowd:'
'Host:'
'<NAME>:'
'<NAME>:'
'Scuzzlebutt:'
'Denver citizens:'
'Sylvester:'
'Pip:'
'Rex:'
'Fosse:'
'Bill:'
'Sparky:'
'Bookie:'[...]<jupyter_text>Looking at the Character list, it appears that I need to do a lot more cleaning up.Starting with removing the `:` at the end of the Character names as I had `TODO`'d in a previous Exercise.Though it could be that the characters without the `:` could not actually be characters.Then I need to look at some of the the dialogue for characters and confirm that they are actual characters with dialogue.Like:     * THE FOUR FOOD GROUPS    * DEPARTS    * PHOTO DOJO    * MLKKK    * URFATANDDUM    Probably easier to parse all the characters with names in ALL CAPS.Other caracter<jupyter_code>#Count the Characters as is
pprint(len(df.Character.unique()))

#Find all the Characters without a `:`
for character in df.Character.unique():
    if not character.endswith(":"):
        pprint(character)
#Need a slightly different character/dialog function that does not add the `:` when looking up the character
def get_character_dialogue_sans_colon(character, script):
    '''
       Retrieve the 'character' dialogue from the 'script'
       character: string
       script: pandas data frame
       
       Return: list of dialogue strings
    '''
    #TODO: Once more cleaning has been done on the Panda's side, this next line shoule not be required.
    #character = character.title() + ":"
    
    #Initialize a list to return
    dialogue = []
    pattern = '\[.*?\]'
    
    for row_index in script.index.values:
        if script.loc[row_index, 'Character'] == character:
            dialogue.append(re.sub(pattern, '', script.loc[row_index, 'Dialogue']))
            
    return dialogue
pprint(get_character_dialogue_sans_colon('DEPARTS', df))
pprint(get_character_dialogue_sans_colon('THE FOUR FOOD GROUPS', df))
#Print all the dialogue
for character in df.Character.unique():
    if not character.endswith(":"):
        pprint(character)
        pprint(get_character_dialogue_sans_colon(character, df))<jupyter_output>'DEPARTS'
['12:30']
'THE FOUR FOOD GROUPS'
['THE FOUR FOOD GROUPS']
'Moo… Moo…Moo…Moo…'
['GREETINGS, COWS OF EARTHWE COME IN PEACE.']
'Moo??'
['REALLY??']
'Moo moo, moo.'
['WE HAVE EXPERIMENTED WITH ALLTHE BEINGS OF EARTH,']
'Moo moo, moo. Moo.'
['AND WE HAVE LEARNED THAT YOU ARE THE MOST INTELLIGENTAND WISE.',
 "OH, THAT WAS CARL'S FAULT. HE'S NEW."]
'Moo moo?'
['WHY DID YOU TURN SOME OF USINSIDE OUT?']
'Moo…moo…moo.'
['YEAH, SORRY ABOUT THAT, MY BAD!']
'Moo moo.'
['TAKE THIS DEVICE.']
'Moo moo. Moo.'
["IT'S A GIFT FROM US."]
'Moo moo. Moo moo.'
['FAREWELL COWS, PEACE BE WITH YOU!']
'NEWS 4'
['SPECIAL REPORT', 'SPECIAL REPORT']
'Commercial'
['Who will win our souls? . Our Savior and Lord?  Or the Prince of Darkness?  '
 "It's the final battle between Good and Evil  and it's only on Pay Per View! "
 'Jesus Versus Satan! Live, from the South Park Forum on Saturday. Call now to '
 'order, only $49.95']
'BAR'
['Free BEER in REAR']
'Cartman'
["Hey! What the hell do you think you're doing h[...]<jupyter_text>Cleaning up things todo:* Remove Characters that are numbers (12:30)* Remove Characters in ALL UPPER CASE* Remove colon at end of Character* Remove Characters that don't have dialogue<jupyter_code>pprint(get_character_dialogue_sans_colon('Colorado Springs20mi Pueblo75mi Cheyenne MountainSecret Military Base18mi', df))
pprint(get_character_dialogue('Alien', df))
pprint(get_character_dialogue('Alien Carl', df))
print(get_character_dialogue('Cows', df))
print(get_character_dialogue('Cow', df))<jupyter_output>['Moo??REALLY??', 'Mooo.', ' Mooooo?', ' Mooooo?!']
['Moo moo?WHY DID YOU TURN SOME OF USINSIDE OUT?', 'Moooo!', 'Mooo!', ' Mooooo!', 'Moo! ']
{'fraction_non_alphanumeric': 0.13235505682635593, 'fraction_numerical': 0.012228456337217667, 'mean_word_length': 4.089311859443631, 'pattern_counts': {'":': 4, '<': 19, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '270700', 'license': 'MIT', 'n_tokens_mistral': 2530, 'n_tokens_neox': 2410, 'n_words': 793, 'path': 'Southpark Scripts [Exercise 3] (Multiple Episodes, Multiple Characters).ipynb', 'repo_name': 'KevinHeathHigh/southparkscriptdownloader', 'size': '7010'}