<jupyter_start><jupyter_text><NAME>: The BasicsTo begin looking at how gender affects dialogue in these movies let's get some basic stats first. Specifically:* Token count per line* Type count per lineHow do these vary based on gender? Later on I hope to look at these character by character:* Token count per character* Type counter per character* TTR per characterI'm not doing TTR line by line, because some of these lines are incredibly short, creating huge TTRs. It's best to look at a character's overall TTR  1) Token count<jupyter_code>import nltk
import pandas as pd
disney_df = pd.read_pickle(r"C:/Users/cassi/Desktop/Data_Science/Animated-Movie-Gendered-Dialogue/private/all_disney_annotated.pkl")
disney_df.info()
disney_df["Tokens"] = disney_df.Text.map(nltk.word_tokenize)
disney_df.head()
disney_df["Types"] = disney_df.Tokens.map(set)
disney_df.head()
disney_df["Token_Count"] = disney_df.Tokens.map(len)
disney_df["Type_Count"] = disney_df.Types.map(len)
disney_df.head()<jupyter_output><empty_output><jupyter_text>A look at the overall averages...Looks like most lines are 14 tokens long, but with a very big standard deviation. The average type count per line is 11 tokens, again with a big standard deviation.<jupyter_code>disney_df[["Token_Count", "Type_Count"]].describe()
disney_df[disney_df.Token_Count == 0] #a blank line for flynn--indicates a silent response
disney_df[disney_df.Token_Count == 480]<jupyter_output><empty_output><jupyter_text>The large standard deviation could be due to song lyrics being included. Let's create a new dataframe, with just dialogue.<jupyter_code>disney_df_dialogue = disney_df[disney_df.Song == 'D']
disney_df_dialogue.reset_index(drop=True, inplace=True)
disney_df_dialogue.head()
print(disney_df.Movie.value_counts()) #lines in each movie w/ songs
print(disney_df_dialogue.Movie.value_counts()) #lines in each movie w/o songs
print(disney_df.Movie.value_counts().describe()) # shows average number of lines in each movie w/ songs
print(disney_df_dialogue.Movie.value_counts().describe()) # shows average number of line in each movie w/o songs
print(disney_df[["Token_Count", "Type_Count"]].describe()) #token/type counts in song df
print('\n')
print(disney_df_dialogue[["Token_Count", "Type_Count"]].describe()) #token/type counts in dialogue df<jupyter_output>Token_Count   Type_Count
count  7422.000000  7422.000000
mean     14.005794    11.148612
std      21.246459    11.914911
min       0.000000     0.000000
25%       5.000000     4.000000
50%       9.000000     8.000000
75%      16.000000    14.000000
max     480.000000   181.000000


       Token_Count   Type_Count
count  6787.000000  6787.000000
mean     12.482245    10.364520
std      14.433096     9.458817
min       0.000000     0.000000
25%       5.000000     4.000000
50%       8.000000     8.000000
75%      15.000000    13.000000
max     300.000000   140.000000<jupyter_text>Now, the average token count is 12.5, and the average type count is 10. The standard deviations are also lower!Let's look at the overall stats for each movie!<jupyter_code>disney_df.groupby(['Year', 'Movie'], sort = 'False')[["Token_Count", "Type_Count"]].describe()
disney_df_dialogue.groupby(['Year', 'Movie'], sort = 'False')[["Token_Count", "Type_Count"]].describe()<jupyter_output><empty_output><jupyter_text>Now that we have the overall averages, let's see how these differ depending on certain filters.  Disney Era?<jupyter_code>disney_df_dialogue.groupby('Disney_Period', sort = False)[["Token_Count", "Type_Count"]].describe()<jupyter_output><empty_output><jupyter_text>Interesting. Looks like the early period has longer lines than the middle period, and the later period has longer lines than both the early and late period. However, average type count doesn't vary as much.  Speaker Gender?<jupyter_code>disney_df_dialogue.groupby('Gender', sort = False)[["Token_Count", "Type_Count"]].describe()<jupyter_output><empty_output><jupyter_text>Female speakers tend to speak one token less than male speakers. Females also seem to speak one type less than male speakers. Does this change by movie/era? Well, that doesn't look too readable. Time to make some charts!!<jupyter_code>from matplotlib import pyplot as plt
f_lines = disney_df_dialogue[disney_df_dialogue.Gender == 'f']
m_lines = disney_df_dialogue[disney_df_dialogue.Gender == 'm']
f_in_year = f_lines.groupby('Year')['Speaker'].describe()['unique']
m_in_year = m_lines.groupby('Year')['Speaker'].describe()['unique']

f_tok_by_year = f_lines.groupby('Year')["Token_Count"].describe()["mean"]
tok_per_f_by_year = f_tok_by_year / f_in_year

f_type_by_year = f_lines.groupby('Year')["Type_Count"].describe()["mean"]
print(f_tok_by_year)
print(f_type_by_year)<jupyter_output>Year
1937    17.650407
1950    13.184615
1959    10.132075
1989    15.017094
1991    10.010050
1992     8.177966
1995     9.231579
1998     9.168919
2009    14.244444
2010    12.538983
2012    12.492063
2013     9.697143
2016    15.795918
Name: mean, dtype: float64
Year
1937    13.406504
1950    10.803846
1959     8.908805
1989    11.846154
1991     8.914573
1992     7.652542
1995     8.157895
1998     8.128378
2009    12.025926
2010    10.006780
2012    10.292063
2013     8.042857
2016    11.954082
Name: mean, dtype: float64<jupyter_text>Average Token Count per Line in Each Movie by Gender<jupyter_code>f_lines = disney_df_dialogue[disney_df_dialogue.Gender == 'f']
m_lines = disney_df_dialogue[disney_df_dialogue.Gender == 'm']

f_tok_by_year = f_lines.groupby('Year')["Token_Count"].describe()["mean"]

m_tok_by_year = m_lines.groupby('Year')["Token_Count"].describe()["mean"]  

f_tok_by_year.plot(figsize = (15,6))
m_tok_by_year.plot(figsize = (15,6))
plt.legend(['female', 'male'])
plt.show()<jupyter_output><empty_output><jupyter_text>Average Type Count per Line in Each Movie By Gender<jupyter_code>#type counts?
f_lines = disney_df_dialogue[disney_df_dialogue.Gender == 'f']
m_lines = disney_df_dialogue[disney_df_dialogue.Gender == 'm']

f_type_by_year = f_lines.groupby('Movie', sort=False)["Type_Count"].describe()['mean']
m_type_by_year = m_lines.groupby('Movie', sort=False)["Type_Count"].describe()['mean']  

#FIND A WAY TO GET LABELS ON AXES!!!

f_type_by_year.plot(figsize = (15, 6))
m_type_by_year.plot(figsize = (15, 6))
plt.legend(['female', 'male'])
plt.show()<jupyter_output><empty_output><jupyter_text>With the exception of Snow White, Cinderella, and Brave, male speakers use a great number of word tokens/types per line. This is interesting line by line, but I eventually also want to look into it OVERALL and by character. I feel that this may be more informative.Also, this may not be the best visual for these stats. Let's try a bar graph  Average Token Count/Line by Gender in Each Movie<jupyter_code>subgroup_2 = disney_df_dialogue.groupby(['Movie','Gender'], sort = False)["Token_Count"].describe()['mean']
subgroup_2.unstack().plot.bar(color=['magenta', 'blue', 'green'], figsize = (15,6))<jupyter_output><empty_output><jupyter_text>This is a graph of average token count by gender in each film. (A different visualization of the same stats found in the line plots above)  Average Type Count/Line by Gender in Each Movie<jupyter_code>subgroup_3 = disney_df_dialogue.groupby(['Movie','Gender'], sort = False)["Type_Count"].describe()['mean']
subgroup_3.unstack().plot.bar(color = ['magenta', 'blue', 'green'], figsize = (15,6))
#Seeing number of lines by gender for each movie
#aka, total number of lines per gender in each movie
#ex: in Snow White, male characters total 214 of the movie's lines; females utter 123 of all lines
total_lines_per_gen = disney_df_dialogue.groupby(['Movie'], sort=False)['Gender'].value_counts()
total_lines_per_gen
#The total number of female/male characters per movie
num_char_per_gen = disney_df_dialogue.groupby(['Movie', 'Gender'], sort=False)["Speaker"].unique().map(len)
num_char_per_gen
#total number of lines / total number of characters
avg_lines_per_gen = total_lines_per_gen.divide(num_char_per_gen)
# This returns movies in alphabetical order, not chronological order. Need to reindex
avg_lines_per_gen = avg_lines_per_gen.reindex(index = total_lines_per_gen.index)
avg_lines_per_gen<jupyter_output><empty_output><jupyter_text>Total Lines per Gender in Each Movie<jupyter_code>total_lines_per_gen.unstack().plot.bar(color = ['magenta', 'blue', 'green'], figsize = (15, 6))<jupyter_output><empty_output><jupyter_text>Except for Cinderella, Sleeping Beauty, Brave, Moana, and Tangled, overall Male characters have more lines than female characters  Average Number of Lines per Gender in Each MovieHere, I divide the total number of lines per gender by the number of male and female characters in each movie, to get the average number of lines uttered by men and women in each movie<jupyter_code>### Oh...interesting....
avg_lines_per_gen.unstack().plot.bar(color = ['magenta', 'blue', 'green'], figsize=(15,6))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11108632612090118, 'fraction_numerical': 0.06926165514164621, 'mean_word_length': 5.661961367013373, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '114242', 'license': 'RSA-MD', 'n_tokens_mistral': 3483, 'n_tokens_neox': 2991, 'n_words': 836, 'path': 'code/Analysis_code/Preliminary Analysis/Disney_Analysis_Basic.ipynb', 'repo_name': 'Data-Science-for-Linguists-2019/Animated-Movie-Gendered-Dialogue', 'size': '8975'}