<jupyter_start><jupyter_text>IntroductionI downloaded a CSV of [City of Chicago employee salary data](https://data.cityofchicago.org/Administration-Finance/Current-Employee-Names-Salaries-and-Position-Title/xzkq-xp2w), which includes the names, titles, departments and salaries of Chicago employees. I wasinterested to see whether men and women earn similar salaries for similar roles.City data don't report the gender of employees, so I used an employee's first nameas a proxy, which is explained in more detail below.<jupyter_code>workers = pd.read_csv('Current_Employee_Names__Salaries__and_Position_Titles.csv')<jupyter_output><empty_output><jupyter_text>To simplify the analysis, I restricted my attention to full-time employees with a salary.<jupyter_code>workers = workers[(workers['Salary or Hourly']=='Salary') & (workers['Full or Part-Time']=='F')]<jupyter_output><empty_output><jupyter_text>To make grouping and matching first names easier, I extracted the first name from each employee record and lower-cased it:<jupyter_code>workers['Name'] = workers['Name'].apply(lambda s: s.split(',')[1].split()[0].lower())<jupyter_output><empty_output><jupyter_text>Annual salary is represented as a string with a leading $ sign, which I converted to floats.<jupyter_code>workers['Annual Salary'] = workers['Annual Salary'].apply(lambda s: float(s.strip('$')))<jupyter_output><empty_output><jupyter_text>The first ten rows of the data set look like this now:<jupyter_code>workers.head(10)<jupyter_output><empty_output><jupyter_text>Gender predictionTo estimate the gender of an employee based on his or her first name, I used a data set of [baby names](https://www.ssa.gov/oact/babynames/limits.html). For each unique name, I countedhow many times, from years 1940 to 2016, that name had been given to a boy versus a girl. If the name was more frequently given to boys, then I predicted the gender associated with thename to be male, and vice-versa for female.<jupyter_code># Data are in seperate CSV files per year, and are concatenated here
name_data = []
for yob in range(1940, 2017):
    df = pd.read_csv('names/yob' + str(yob) + '.txt', 
                     header=0, names=['Name', 'Gender', 'Count'])
    name_data.append(df)
names = pd.concat(name_data, axis=0)
# Lower-case first name so that it can be joined with the workers dataframe
names['Name'] = names['Name'].str.lower()
names.head(5)
# Count how often a name is given to boys and girls
gender_frequency = names.groupby(['Name', 'Gender']).sum().reset_index()
gender_frequency.sample(5)
def predict_gender(df):
    max_idx = df['Count'].idxmax()
    return df.loc[max_idx]
# Select the more frequent gender for each name
gender_guess = gender_frequency.groupby('Name').agg(predict_gender).reset_index()
gender_guess.sample(10)<jupyter_output><empty_output><jupyter_text>The above list of names and associated genders can be combined with the worker data to predict the gender of each Chicago employee using a join:<jupyter_code>workers = pd.merge(workers, gender_guess, on='Name', how='inner')
workers[['Name', 'Job Titles', 'Department', 'Gender', 'Annual Salary']].sample(10)<jupyter_output><empty_output><jupyter_text>AnalysisI wanted to know wether men and women were paid equally if they shared the same job title anddepartment. To answer this, I specifically looked at full-time, salaried employees, and jobsfor which both men and women were employed under the same title and department.For example, given the job title POLICE OFFICER in the POLICE department, a position for whichboth men and women are employed, do male and female officers have similar salaries? Moregenerally, are men and women paid equally across all job titles and departments?<jupyter_code># Focus on these columns
workers = workers[['Job Titles', 'Department', 'Gender', 'Annual Salary']]
# Remove jobs for which only men or only women are employed
job_groups = workers[['Job Titles', 'Gender', 'Annual Salary']].groupby(['Job Titles'])
def male_and_female(grp):
    return np.any(grp['Gender']=='M') and np.any(grp['Gender']=='F')
job_groups = job_groups.filter(male_and_female)
# Look at the maximum salary of each gender for each job title
job_group_maximums = job_groups.groupby(['Job Titles', 'Gender']).agg(np.max)
job_group_maximums.head(30)
higher_max_male_salary_count = 0
total_jobs = 0
for job_title, df in job_group_maximums.groupby(level=0):
    assert len(df) == 2
    if df.loc[(job_title, 'M')][0] > df.loc[(job_title, 'F')][0]:
        higher_max_male_salary_count += 1
    total_jobs += 1
higher_max_male_salary_percentage = 100 * higher_max_male_salary_count / total_jobs
higher_max_male_salary_percentage
ax = sns.stripplot(x="Gender", y="Annual Salary", data=job_group_maximums.reset_index(), jitter=True)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10880398671096346, 'fraction_numerical': 0.00872093023255814, 'mean_word_length': 6.211077844311378, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '380472', 'license': 'MIT', 'n_tokens_mistral': 1543, 'n_tokens_neox': 1453, 'n_words': 487, 'path': 'ChicagoEmployeeGenderSalary.ipynb', 'repo_name': 'hobgreenson/chicago_employees', 'size': '4816'}