<jupyter_start><jupyter_text>Commands, Attempt 2!I'm going to try regular expressions this time!<jupyter_code>import pandas as pd
import re
movie_df = pd.read_pickle(r"C:/Users/cassi/Desktop/Data_Science/Animated-Movie-Gendered-Dialogue/private/all_tagged_dialogue.pkl")
movie_df.info()
#should, shall, will should be in different regex

commands_4 = []
for line in movie_df.Text:
    if re.search(r'you (must|could)(?! be )', line) or re.search(r'you (should|shall|will|have to)', line):
        commands_4.append(line)
len(commands)
len(commands_4)
%pprint
commands_4
len(commands_2)
len(commands_3)
commands
set(commands) - set(commands_2)
set(commands_3) - set(commands_2)
set(commands) - set(commands_3) #some "must be"'s missed
#Some commands to try...
#let me
#come
#look
#stop
#don't
#quiet
#shut up

#Modals:
#Suggestions: can, could, may, should
#Commands: must, should, shall, ought, need, have to
#What about: you('d') better

commands_8 = []
for line in movie_df.Text:
    if re.search(r'you must(?! be )', line) or re.search(r"you (shall|will|have to|ought|need to|may not|can't|cannot|won't)", line):
        commands_8.append(line)
#you can seems more like a suggestion...
#while you can't is more a command...
suggestions_3 = []
for line in movie_df.Text:
    if re.search(r'you may (?!be )', line) or re.search(r'you (should|could|better|can )', line):
        suggestions_3.append(line)
len(commands_5) #was 134 before ought and need to
#150 before can't, cannot, and won't
len(commands_8) #THIS ONE!!!!!
len(commands_7) #go with this one!
len(commands_6)
%pprint<jupyter_output>Pretty printing has been turned ON<jupyter_text>Though the 6th try overcaptures a bit--notice the differences between "you can't go", "you won't have to", and "you can't get something for nothing, you know?" and "how do you know you won't like it if you don't try it?"The general use of you is captured with this regular expression, along with if clauses. But overall, it doesn a pretty good job of capturing what I'm looking for!<jupyter_code>set(commands_5) - set(commands_6)
len(suggestions_1)
len(suggestions_2)
len(suggestions_3)
set(suggestions_1) - set(suggestions_3)
set(suggestions_1) - set(suggestions_2) #I think I'll opt to keep suggestions 1?
#though some of these are conditional statements, you may remove your helmet
suggestions_1
commands_5<jupyter_output><empty_output><jupyter_text>Commands as Verb Base FormsMy first attempt, which only involved finding base forms, didn't prove very useful because it overcaptured.In this attempt, I will only look at base forms that come after punctuation.<jupyter_code>movies_df = pd.read_pickle(r'C:/Users/cassi/Desktop/Data_Science/Animated-Movie-Gendered-Dialogue/private/all_tagged_movies_2.pkl')
test_sents = movies_df.POS.iloc[:100]
test_sents[0][0][1]
sent = movie_df.POS.iloc[0]
sent.index(('slave', 'NN'))
sent[0]
commands_count = 0
i = 0
for line in test_sents:
    print("Searching...", i)
    if line[0][1] == 'VB':
        commands_count += 1
        print("command found!")
    next_lines = line[1:]
    for x in line[1:]:
        if x[1] == 'VB':
            place = line.index(x)
            if line[place-1][1] == "." or line[place-1][1] == "," or line[place-1][1] == "CC":
                commands_count += 1
                print("command found!")
    i += 1
test_sents.iloc[35]
test_sents.iloc[11]
test_sents.iloc[18]
test_sents.iloc[19]
test_sents.iloc[25]
%pprint<jupyter_output>Pretty printing has been turned OFF<jupyter_text>As you can see, it's still not perfect (especially with those pesky incorrect tags...but it does a pretty good job!<jupyter_code>test_sents.iloc[45]
# Expanding this to the whole dataframe
commands_count = 0
i = 0
for line in test_sents[0]:
    print("Searching...", i)
    if line[0][1] == 'VB':
        commands_count += 1
        print("command found!")
    next_lines = line[1:]
    for x in line[1:]:
        if x[1] == 'VB':
            place = line.index(x)
            if line[place-1][1] == "." or line[place-1][1] == ",":
                commands_count += 1
                print("command found!")
    i += 1
def find_commands(tagged_sent):
    commands_count = 0
    if tagged_sent[0][1] == 'VB':
        commands_count += 1
    for x in tagged_sent[1:]:
        if x[1] == 'VB':
            place = tagged_sent.index(x)
            if tagged_sent[place-1][1] == "." or tagged_sent[place-1][1] == "," or tagged_sent[place-1][1] == 'CC':
                commands_count += 1
    return commands_count
movies_df["Command_Count"] = movies_df.POS.map(find_commands)
movies_df.Command_Count[:100]
movies_df.Command_Count.describe()
movies_df[movies_df.Command_Count == 10].POS.iloc[0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1338132542875291, 'fraction_numerical': 0.023078551767944103, 'mean_word_length': 4.107027027027027, 'pattern_counts': {'":': 2, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '663328', 'license': 'RSA-MD', 'n_tokens_mistral': 1741, 'n_tokens_neox': 1633, 'n_words': 455, 'path': 'code/old_code/command_try_2.ipynb', 'repo_name': 'Data-Science-for-Linguists-2019/Animated-Movie-Gendered-Dialogue', 'size': '4723'}