<jupyter_start><jupyter_text>Streamlining the Line Extraction Process for two scriptsIn this code I will...* further analyze the white space in How To Train Your Dragon 2 and Rise of the Guardians* I will see if the white space patterns match in each script* I will write a general code that can be applied to both scripts to extract character dialogue line by lineNOTE: this is NEW CONTINUATION FILE, stemming from my work on [white space](https://github.com/Data-Science-for-Linguists-2019/Animated-Movie-Gendered-Dialogue/blob/master/code/DreamWorks_code/Analyzing_White_Space.ipynb).The Final Streamline can be found [here](https://github.com/Data-Science-for-Linguists-2019/Animated-Movie-Gendered-Dialogue/blob/master/code/DreamWorks_code/Streamline_RotG_HTTYD2_Croods_DataFrames.ipynb).<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>1) Importing Scripts<jupyter_code>#importing each script
httyd2 = open(r'C:\Users\cassi\Desktop\Data_Science\Animated-Movie-Gendered-Dialogue\private\imsdb_raw_nov_2015\Animation\howtotrainyourdragon2.txt')
httyd2_script = httyd2.read()
httyd2.close()

rotg = open(r'C:\Users\cassi\Desktop\Data_Science\Animated-Movie-Gendered-Dialogue\private\imsdb_raw_nov_2015\Animation\riseoftheguardians.txt')
rotg_script = rotg.read()
rotg.close()

croods = open(r'C:\Users\cassi\Desktop\Data_Science\Animated-Movie-Gendered-Dialogue\private\imsdb_raw_nov_2015\Animation\croodsthe.txt')
croods_script = croods.read()
croods.close()
%pprint
#Shaving off excess at beginning of each script
httyd2_script[:351]
httyd2_script = httyd2_script[351:]
rotg_script[:222]
rotg_script = rotg_script[222:]
croods_script[:190]
croods_script = croods_script[190:]
print(httyd2_script[:500])
print('\n')
print(rotg_script[:500])
print('\n')
print(croods_script[:500])
import re<jupyter_output><empty_output><jupyter_text>2) Analyzing White SpaceHere, we'll see how consistent the space distribution is in each of these scripts<jupyter_code># A general function for white space analysis
def white_space_count(script_name):
    white_space = re.findall(" {3,}", script_name)
    len_w_s = [len(x) for x in white_space]
    print(len_w_s[:100])
    print(set(len_w_s))
    for num in set(len_w_s):
        print(num, "white spaces appear", len_w_s.count(num), "times")<jupyter_output><empty_output><jupyter_text>HTTYD2 White Space Clean Up<jupyter_code>#HTTYD2
white_space_count(httyd2_script)
httyd2_script[:2000]
# 11 before the first scene header
# 11 after a scene header and before scene description
# 25 before a two new line chars
# 11 before hiccup's character name
# 11 between hiccup's lines
# 10 after hiccup's lines end
# 11 before next scene header
# 10 between scene descriptions
# 10 before hiccups next line
# 10 between hiccup's lines<jupyter_output><empty_output><jupyter_text>HTTYD2 has these pesky random "FINAL DRAFT" Lines that aren't necessary. And they're preceded by the same number of white space characters as actual lines. So, it's best to get rid of them.<jupyter_code>httyd2_titles = re.findall(r"\n\n           HTTYD 2 \(D. DeBlois\) - FINAL DRAFT - 05/05/14 [0-9]+\.", httyd2_script)
len(httyd2_titles)
httyd2_titles[:10]
httyd2_script = re.sub(r"\n\n           HTTYD 2 \(D. DeBlois\) - FINAL DRAFT - 05/05/14 [0-9]+\.",'', httyd2_script)
#reanalyze white space
white_space_count(httyd2_script)
1440 - 1343 #Yay! matched the 97 not-lines<jupyter_output><empty_output><jupyter_text>RotG White Space Clean Up<jupyter_code>#RotG
white_space_count(rotg_script)
rotg_script[:1000]
rotg_script[:3500]
# 10 before a sequence
# 25 after sequence/before scene header
# 11 after scene header/before char name
# 11 after char name/before line
# 11 between char lines
#10 at the end of Jack's first lines/before scene description
#10 between scene description lines/before Jack's next line
# 11 after char name and after each line
#26 before some lines.
# 11 before a 

#10 at the end of a line of dialogue
#11 or 26 before a speakername
#11 before this strange chunk: "\n\n           Guardians Screening Script 9/6/12 Avid 2."

line = "\n\n                          JACK\n\n"
len(line)
len(line[2:-6])
titles = re.findall(r"\n\n           Guardians Screening Script 9/6/12 Avid [0-9]+\.", rotg_script)
print(len(titles))
print(titles[:10])
#remove the weird chunk
rotg_script_2 = re.sub(r"\n\n           Guardians Screening Script 9/6/12 Avid [0-9]+\.", '', rotg_script)
#analyze white space again, see if 11 white spaces went down by 134
white_space_count(rotg_script_2)
1873 - 1739 #Yay!<jupyter_output><empty_output><jupyter_text>Croods White Space Clean Up<jupyter_code>white_space_count(croods_script)
croods_script[:2500]
# 10 before SEQ
# 25 before FADE IN
# 25 before TO
# 26 before Eep's line
# 11 between Eep's lines
# 10 before scene descriptions<jupyter_output><empty_output><jupyter_text>It seems that, in general,* 11 or 26 spaces precede a speaker's name* 10 spaces or 25 spaces precede things that aren't actual lines  3) Testing This spacing method<jupyter_code>#try to remove parentheticals
httyd2_par = re.findall(r" *\([^\)]*\)", httyd2_script)
croods_par = re.findall(r" *\([^\)]*\)", croods_script)
len(croods_par)
len(httyd2_par)
croods_par_2 = re.findall(r" {1,}\([^\)]*\)", croods_script)
len(croods_par_2)
httyd2_par
httyd2_script_2 = re.sub(r" {1,}\([^\)]*\)", '', httyd2_script)
white_space_count(httyd2_script_2)
(760-690) + (1571-1547) + (1343-1318)
24 + 70 + 25
httyd2_script_2[:500]
httyd2_lines = re.findall(r"\n\n {26}(\b[A-Z]['A-Z ]{1,})+\n\n", httyd2_script_2) #THIS IS GOOD!!!!
print(len(httyd2_lines))
print(set(httyd2_lines))
httyd2_morelines = re.findall(r"(\n\n {11}([A-Z]['A-Z]{1,}[ ]?){1,4}\n\n)", httyd2_script_2) 
print(len(httyd2_morelines))
print(set(httyd2_morelines))<jupyter_output>{('\n\n           VALKA\n\n', 'VALKA'), ('\n\n           GOBBER\n\n', 'GOBBER'), ('\n\n           STOICK\n\n', 'STOICK'), ('\n\n           HICCUP\n\n', 'HICCUP'), ('\n\n           UG\n\n', 'UG'), ('\n\n           ERET\n\n', 'ERET')}<jupyter_text>Let's try to write a function that will do this for all three movies, and see how well it works<jupyter_code>def line_identifier(script):
    script_lines = re.findall(r"\n\n {26}(\b[A-Z]['A-Z ]{1,})+\n\n", script)
    more_lines = re.findall(r"(\n\n {11}([A-Z]['A-Z]{1,}[ ]?){1,4}\n\n)", script)
    return len(script_lines), set(script_lines), len(more_lines), set(more_lines)
#First, take out parentheticals of other scripts:
def no_parentheses(script):
    new_script = re.sub(r" *\([^\)]*\)", '', script)
    return new_script
croods_updated = no_parentheses(croods_script)
croods_script[:500]
croods_updated[:500]
white_space_count(croods_updated)
len_croods_lines, set_croods_lines, len_more_lines, set_more_lines = line_identifier(croods_updated)
print(len_croods_lines)
print(set_croods_lines)
print(len_more_lines)
print(set_more_lines)
#double check this
more_lines = re.findall(r"\n\n {11}\b[A-Z]['A-Z ]{1,}\n\n", croods_updated)
#we can live without the collective line
#need to tweek the spacing here
set(more_lines)
rotg_updated = no_parentheses(rotg_script)
rotg_updated[:500]
rotg_script[:500]
white_space_count(rotg_updated)
len_rotg_lines, set_rotg_lines, len_more_rotg_lines, set_more_rotg_lines = line_identifier(rotg_updated)
print(len_rotg_lines)
print(set_rotg_lines)
print(len_more_rotg_lines)
print(set_more_rotg_lines)
#test that this is okay:
more_lines_rotg = re.findall(r"\n\n {11}\b[A-Z]['A-Z ]{1,}\n\n", rotg_updated)
more_lines_rotg = re.findall(r"(\n\n {11}([A-Z]['A-Z]{1,}[ ]?){1,4}\n\n)", rotg_updated)
print(len(more_lines_rotg))
print(set(more_lines_rotg))
set([x[0] for x in set(more_lines_rotg)]) - set_more_rotg_lines
#FADE TO WHITE IS THE ONLY PROBLEM HERE!!!
#This can be found later in, when dealing with the dataframe
re.findall(r"\n\n {25}[\S]+", croods_updated)
#creating a function to replace these lines!!!
def create_lines(script):
    id_lines = re.sub(r"\n\n {26}(\b[A-Z]['A-Z ]{1,})+\n\n", r"_NEWLINE_\1_", script)
    id_more_lines = re.sub(r"\n\n {11}([A-Z]['A-Z]{1,}[ ]?){1,4}\n\n", r"_NEWLINE_\1_", id_lines)
    script_lines = id_more_lines.split("_NEWLINE_")
    return script_lines
#test on croods
crood_lines = create_lines(croods_updated)
len(crood_lines)
crood_lines[:10]
#now, the end of a line should be \n\n {10}
#and, we still need to deal with 25 white spaces
for line in crood_lines:
    if r"                         " in line:
        print(line)
def remove_not_lines(script_lines):
    new_script_lines = []
    for line in script_lines:
        new_line = re.sub(r" {25}([A-Z]{3,} )+", r"_NOTLINE_", line)
        if "_NOTLINE_" in new_line:
            new_line = new_line.split("_NOTLINE_")
            new_line = new_line[0]
            new_script_lines.append(new_line)
        else:
            new_script_lines.append(new_line)
    new_new_script_lines = []
    for line in new_script_lines:
        new_line = re.sub("\n\n {10}[\S]", '_HEADER_', line)
        real_line = new_line.split('_HEADER_')[0]
        real_line = re.sub("\n\n", ' ', real_line)
        final_line = re.sub(" {3,}", ' ', real_line)
        new_new_script_lines.append(final_line)
    final_lines = new_new_script_lines[1:]
    return final_lines

final_crood_lines = remove_not_lines(crood_lines)
len(final_crood_lines)
#create the dataframe!
def create_DF(final_lines):
    lines_list = []
    for line in final_lines:
        speaker_text_tu = tuple(re.split('_ ', line))
        lines_list.append(speaker_text_tu)
    script_DF = pd.DataFrame(lines_list, columns=["Speaker", "Text"])
    return script_DF
test_croods_DF = create_DF(final_crood_lines)
test_croods_DF.head()
test_croods_DF.tail()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13527414875012858, 'fraction_numerical': 0.03600452628330419, 'mean_word_length': 4.913625304136253, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '607592', 'license': 'RSA-MD', 'n_tokens_mistral': 3956, 'n_tokens_neox': 3669, 'n_words': 833, 'path': 'code/old_code/Rough_Streamline.ipynb', 'repo_name': 'Data-Science-for-Linguists-2019/Animated-Movie-Gendered-Dialogue', 'size': '9721'}