<jupyter_start><jupyter_text>Calculating Gender Composition from List of WritersAuthor: <NAME>Date: 4/25/18 (updated 6/8/18)<jupyter_code>import pandas as pd
import numpy as np

episodesData = pd.read_csv("top100ComediesWithWriters.csv", encoding='latin-1')
episodesData = episodesData.replace(np.nan, '', regex=True)
episodesData.head(3)
import json

from urllib.request import urlopen

# to determine writer gender based on name, we use the Gender API found at https://gender-api.com/

# key obtained from Gender API
myKey = ""

def writerGender(writerFirstName):
    """
    Given a first name (ex: "Oliver"), call the Gender API and return 
    a list containing the predicted gender of name and confidence level
    """
    url = "https://gender-api.com/get?key=" + myKey + "&name=" + writerFirstName.strip(" ")
    response = urlopen(url)
    decoded = response.read().decode('utf-8')
    data = json.loads(decoded)

    return [data['gender'], data['accuracy']]<jupyter_output><empty_output><jupyter_text>Although we now have an API and accompanying function to derive gender from a writer's name, we do not want to apply these to the episodesData data frame itself. Because unique writer values often repeat (as in most writers write multiple episodes), this would result in calling the API over 11,000 times, which is problematic. Instead, we want to create a new dataframe, in which each writer and their corresponding gender is observed once. This smaller dataframe will act as a key.<jupyter_code>writerDict = {}

# because each episodes' writers are stored as a single string, 
# seperated by commas, create a list of writers by using str.split()
for episode in range(0, len(episodesData)): 
    writersNoSpace = episodesData['writers'][episode].replace(', ', ',')
    episodesWriters = writersNoSpace.split(',')
    
    # for each writer in the list, add to the dictionary of writers
    for writer in episodesWriters:
        if writer not in writerDict:
            writerDict[writer] = 1
        else: 
            writerDict[writer] = writerDict[writer] + 1    

# convert dictionary into a dataframe
writerDictDF = pd.DataFrame(list(writerDict.items()), columns=['Writer', 'Episodes Written'])
writerDictDF = writerDictDF.sort_values(by = 'Episodes Written', ascending = False)
writerDictDF = writerDictDF.reset_index()
del writerDictDF['index']

def firstName(name):
    return name.split(' ')[0]

writerDictDF['first'] = writerDictDF['Writer'].apply(firstName)

writerDictDF.head()
# Create a dataframe including only unique first names, i.e. all the 'John' writers take up just one row
firstNameList = writerDictDF['first'].unique()
writersFirstNames = pd.DataFrame(firstNameList, columns = ['first'])<jupyter_output><empty_output><jupyter_text>This new dataframe contains a little over 500 observations, one for each unique writer found in the episodesData dataframe. We can now apply the writerGender() function defined above to determine the gender of each writer. Once calculated, two supporting functions seperate the returned gender and prediction accuracy list into seperate values, attaching each as columns to the dataframe.<jupyter_code>writersFirstNames['genderAndAccuracy'] = ''

writersFirstNames['genderAndAccuracy'] = writersFirstNames['first'].apply(writerGender)

def getGender(value):
    return value[0]

def getAccuracy(value):
    return value[1]

writersFirstNames['gender'] = ''
writersFirstNames['gender'] = writersFirstNames['genderAndAccuracy'].apply(getGender)
writersFirstNames['accuracy'] = ''
writersFirstNames['accuracy'] = writersFirstNames['genderAndAccuracy'].apply(getAccuracy)
writersFirstNames.tail()
# Now that we have an estimated gender for each first name, 
# merge back with the original writer's dataframe to attribute gender to each unique writer
del writersFirstNames['genderAndAccuracy']
writerDictDF2 = writersFirstNames.merge(writerDictDF, how = "right")

del writerDictDF2['first']
writerDictDF2.head()<jupyter_output><empty_output><jupyter_text>Quality Control - Updating Errors Made by the Gender API (we double checked every name that had less than 97% accuracy)<jupyter_code>#loop to correct writers' genders that the API mislabeled
writersToUpdate = ['<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>',"<NAME>",'<NAME>','<NAME>','<NAME>-Rollins','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>','<NAME>']
indexOfWriters = []

for writer in writersToUpdate:
    indexOfWriters.append(writerDictDF2.loc[writerDictDF2['Writer'] == writer].index.values[0])

for writer in indexOfWriters:
    if writerDictDF2.loc[writer,'gender'] == 'female':
        writerDictDF2.loc[writer,'gender'] = 'male'
    if writerDictDF2.loc[writer,'gender'] == 'male':
        writerDictDF2.loc[writer,'gender'] = 'female' 
#unknown genders loop - for writers whom we could not find a definite gender for
noKnownGender = ['<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>', "<NAME>"]
indexOfWriters = []

for writer in noKnownGender:
    indexOfWriters.append(writerDictDF2.loc[writerDictDF2['Writer'] == writer].index.values[0])

for writer in indexOfWriters:
    writerDictDF2.loc[writer,'gender'] = 'N/A'
# Updating all the writers who received a 'unknown' gender from the API
maleIDs = [2115,1759,2025,1759,2041,1531,1034,1535,2160]
femaleIDs = [1385,1155,2089,1907,883,1608]

for writer in maleIDs:
    writerDictDF2.loc[writer,'gender'] = 'male'
    
for writer in femaleIDs:
    writerDictDF2.loc[writer,'gender'] = 'female'
writerDictDF2.to_csv("C:\\Users\\Oliver\\Documents\\imbd episodes\\genderOfWriters.csv")<jupyter_output><empty_output><jupyter_text>Determining Gender Composition of Each Episode Now that we have obtained the gender for each writer, the next step is to add genders to the episodesData dataframe. A merge between the two dataframes would not be successful because multiple writers are recorded single strings in the episodesData dataframe. Instead, we create a third dataframe, in which each writer of each episode is listed individually in a row. This will allow for a merge with the writerDictDF dataframe.<jupyter_code>writingStaff = []

for index, row in episodesData.iterrows():
    
    if row['writers'] != r'\N':
        episodes = row['tconst']
        writerGenders = row['writers']
        
        # for each writer included in the writers column of episodesData, create a new row for that writer
        for writer in writerGenders.split(','):
            newRow = [episodes, writer]
            writingStaff.append(newRow)

writingStaffDF = pd.DataFrame(writingStaff, columns = ['tconst', 'Writer'])
writingStaffDF2 = writingStaffDF.sort_values(by = 'Writer')
writerDictDF2 = writerDictDF2.sort_values(by = 'Writer')

def stripSpace(value):
    return value.strip(' ')
    
writingStaffDF2['Writer'] = writingStaffDF2['Writer'].apply(stripSpace)
writerDictDF2['Writer'] = writerDictDF2['Writer'].apply(stripSpace)
writingStaffDF2.head(10)
writerDictDF2.head(10)
writingStaffWithGender = writingStaffDF2.merge(writerDictDF2, how = 'left')
writingStaffWithGender = writingStaffWithGender.sort_values(by = 'tconst')
writingStaffWithGender = writingStaffWithGender.reset_index()
del writingStaffWithGender['index']
writingStaffWithGender.head()
# add a new column of blank strings
writingStaffWithGender['writerGenders'] = ''

# iterate over the length of the data frame in order to create full lists of writers involved, rather
# than having writing staffs seperated over multiple rows
for writer in range(24425, len(writingStaffWithGender) - 1):
    
    # create a variable holding the string of the writers name for current row
    genders = writingStaffWithGender['gender'][writer]
    count = 1
    
    # while subsequent rows contain information about the current row's same movie, add the writer's
    # names to the 'names' variable. 
    while writingStaffWithGender['tconst'][writer] == writingStaffWithGender['tconst'][writer + count]:
        genders = genders + "," + writingStaffWithGender['gender'][writer + count]
        count = count + 1
        
    # once the last row for the given movie is reached and all writers have been added to the 
    # 'names' variable, replace the empty string in the 'writers' column with the string held by 'names'
    writingStaffWithGender['writerGenders'][writer] = genders<jupyter_output><empty_output><jupyter_text>We now have a dataframe of episode title IDs and corresponding gender compositions, in string list form. However, each episode is listed multiple times, once for each writer. Additionally, the full gender compositions (including the gender of every writer for a given episode) are only included in the first instance of each episode. Thus, the final step is to drop duplicate episode IDs, keeping only the first instance, and deleting the unhelpful columns.<jupyter_code># drop duplicate episodes in the data frame, always keeping the first instance of each repeated movie
# because the full writing staffs are only included in the first instance
writingStaffWithGender2 = writingStaffWithGender.drop_duplicates(subset = 'tconst', keep = 'first')

writingStaffWithGender2 = writingStaffWithGender2.drop(['gender', 'Episodes Written', 'Writer', 'gender', 'accuracy'], axis = 1)

writingStaffWithGender2 = writingStaffWithGender2.reset_index()
del writingStaffWithGender2['index']

writingStaffWithGender2.head()<jupyter_output><empty_output><jupyter_text>Merging Gender Compositions and Episode Data, Calculating Measures, and Dropping Missing ValuesFinally, we have a proper dataframe containing each TV episode and the genders of each writer for that episode. This is merged with the episodesData dataframe, pulling all necessary data together.<jupyter_code># attach each episode's writing staff to our main dataframe, 'episodesData'
episodesDataWithGenderComp = episodesData.merge(writingStaffWithGender2, how = "inner", on = "tconst")

episodesDataWithGenderComp.head(3)
def calculateGenderComp(value):
    """
    Given a list of individuals' genders in comma-seperated string format, 
    return the proportion of males included in the list
    """
    
    listOfGender = value.split(',')
    
    numMaleWriters = 0
    
    for writerGender in listOfGender:
        if writerGender == 'N/A':
            return -1 # these episodes are to be deleted later
        if writerGender == 'male':
            numMaleWriters = numMaleWriters + 1
            
    return numMaleWriters / len(listOfGender)

def calculateNumWriters(value):
    """
    Given a list of individuals' genders in comma-seperated string format, 
    return the proportion of males included in the list
    """
    
    listOfGender = value.split(',')
            
    return len(listOfGender)


episodesDataWithGenderComp['perMaleWrite'] = 0
episodesDataWithGenderComp['perMaleWrite'] = episodesDataWithGenderComp['writerGenders'].apply(calculateGenderComp)

episodesDataWithGenderComp['numWriters'] = 0
episodesDataWithGenderComp['numWriters'] = episodesDataWithGenderComp['writerGenders'].apply(calculateNumWriters)
del episodesDataWithGenderComp['Unnamed: 0']
episodesDataWithGenderComp = episodesDataWithGenderComp[episodesDataWithGenderComp['perMaleWrite'] != -1]
episodesDataWithGenderComp.head()
episodesDataWithGenderComp.to_csv("C:\\Users\\Oliver\\Documents\\imbd episodes\\episodesDataWithGenderComp-6-8-18.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12530040606613077, 'fraction_numerical': 0.012513466478826552, 'mean_word_length': 5.548019533369506, 'pattern_counts': {'":': 0, '<': 126, '<?xml version=': 0, '>': 126, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '633029', 'license': 'MIT', 'n_tokens_mistral': 3690, 'n_tokens_neox': 3469, 'n_words': 1052, 'path': 'writer-gender-comp-calculate.ipynb', 'repo_name': 'OGladfelter/IMDd-Comedies', 'size': '12700'}