<jupyter_start><jupyter_text>NEW CONTINUING: After [finding errors](https://github.com/Data-Science-for-Linguists-2019/Animated-Movie-Gendered-Dialogue/blob/master/code/Disney_code/Messy_Disney_Movies.ipynb) in Frozen and a successful [scraping of Tangled](https://github.com/Data-Science-for-Linguists-2019/Animated-Movie-Gendered-Dialogue/blob/master/code/Disney_code/Scraping_Tangled.ipynb), I've done the same for Frozen. Scraping FrozenBasically, I lost my patience with the Frozen dataframe. I'm doing this myself. (I guess I couldn't *let it go*...sorry, sleep-deprived me lives for puns)This webpage (link below in code) is actually formatted much more neatly than Tangled's was. Because of this, it was pretty easy to for loop through the html file, make a string out of each line, and go from there!<jupyter_code>from bs4 import BeautifulSoup
import requests
import re
%pprint
link = "https://transcripts.fandom.com/wiki/Frozen"
page_response = requests.get(link, timeout=5)
page_content = BeautifulSoup(page_response.content, "html.parser")
#I did this to help find where the script was located, but I can't flash the whole thing
#page_content
script_html = page_content.find("div", {"class": "mw-content-ltr mw-content-text"} ) 
#this portion contains the actual script!!
#I looked at this to see how things were formatted, but I can't show you the whole script
# or else Disney might get angry!
#script_html
#The script is split up at random points, so this is kind of annoying.
# At first I thought that meant I had to for loop through each of these, but that
# actually gave me 5 copies of my script! YIKES!
script_chunks = script_html.find_all("ul")
len(script_chunks)
#here's the smallest chunk in the bunch, so you can see what this formatting looks like!
script_chunks[3]<jupyter_output><empty_output><jupyter_text>Building a list of tuplesHere, I for loop through each next element in my first chunk (which goes through the whole script), turn each line in the chunk into a string, take out all those html characters, and split on the colons (unlike Tangled, I can do this becauce colons only appear after actual lines!!)<jupyter_code>just_lines_list = []
for line in script_chunks[0].next_elements:
    line_str = str(line)
    if line_str.startswith("<li><b>") or line_str.startswith("<li> <b>"): #all lines start with these
        line_str = re.sub(r"<li> ?<b>", '', line_str)
        line_str = line_str.replace("\n</li>", '', 1).replace('"', '').replace("</b>", '')
        line_str_real = re.sub(r"\[[^\]]*\]", '', line_str)
        speaker_line = line_str_real.split(":")
        speaker_line[0] = speaker_line[0].strip().lower()
        speaker_line[1] = speaker_line[1].strip().lower()   
        just_lines_list.append(tuple(speaker_line))
len(just_lines_list)
just_lines_list[-1]
just_lines_list[0]<jupyter_output><empty_output><jupyter_text>List of tuples created! Time to create a dataframe!!<jupyter_code>import pandas as pd
frozen = pd.DataFrame(just_lines_list, columns=["Speaker", "Text"])
frozen.head()
frozen.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 854 entries, 0 to 853
Data columns (total 2 columns):
Speaker    854 non-null object
Text       854 non-null object
dtypes: object(2)
memory usage: 6.7+ KB<jupyter_text>Sweet, sweet victory!!!!!For those of you who are curious, the actual webpage has 856 lines...I tried to capture as many as possible, and I'm still not sure where those last two lines got to. But I figure this data is better than what I had, so I think I can live with two missing lines!Let's pickle it!<jupyter_code>frozen.to_pickle((r'..\..\Animated-Movie-Gendered-Dialogue\private\frozen_lines.pkl'))
sorted(frozen.Speaker.unique())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11105193075898802, 'fraction_numerical': 0.010652463382157125, 'mean_word_length': 5.107317073170732, 'pattern_counts': {'":': 2, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '360314', 'license': 'RSA-MD', 'n_tokens_mistral': 1256, 'n_tokens_neox': 1190, 'n_words': 406, 'path': 'code/Disney_code/Scraping_Frozen.ipynb', 'repo_name': 'Data-Science-for-Linguists-2019/Animated-Movie-Gendered-Dialogue', 'size': '3755'}