<jupyter_start><jupyter_text>Project to simulate a data set using numpy.random  IntroductionThe underrepresentation of women at management level in third-level institutions is an international problem. In 2015 in Ireland 81% of professorial positions were held by men and, while women represented 62% of non-academic staff, men represented 72% of the highest paid non-academic staff. While there are currently 2 female presidents of Institutes of Technology in Ireland, there has never been a female university president. This issue was brought to the fore in the media when four academics from NUIG took a case against the University alleging gender discrimination. This followed the landmark case taken by Dr <NAME> against NUIG in 2014. Dr <NAME> won her case against the university when the Equality Tribunal ruled that she was not promoted to senior lecturer because of her gender. The ruling described the university’s interview process as “ramshackle” and noted that one of the men promoted was not even eligible to apply. In light of this I have decided to look at the gender breakdown at senior management level in Galway Mayo Institute of Technology (GMIT) to see if there is a similar pattern.       Process  I will look at the following variables:   Age   Qualification   Years of Experience   Grade   Gender      The age variable will be a non-negative real number between 25 & 66.The Qualification variable will be a real number with one of 3 different values: 8, 9, 10, to represent levels on the National Framework of Qualifications, (Level 8 = Honours Degree, Level 9 = Masters Degree, Level 10 = PhD)   .The years of experience variable will be a non-negative real number, and will be based on age and qualification.The Grade variable will be a categorical variable with one of 7 different values: Assistant Lecturer, Lecturer, Senior Lecturer 1, Senior Lecturer 2, Senior Lecturer 3, Registrar, PresidentThe Gender variable will be either male or femaleThe determinant of whether a staff member has reached senior management level will be their Grade - those at SL2, SL3, Registrar & President will be considered at Senior Management levelOnly staff categorised as Academic will be considered in this project.  I will use the HEA publication 'Higher Education Institutional Staff Profiles by Gender', July 2018 to get the current statistics for gender breakdown in GMIT. This publication tells me that as at July 2018 46% of Academic staff in GMIT were female, and 54% male.  I will look at statistics on qualifications provided by GMIT, and use a uniform distribution with certain parameters as described further on to generate age data.   Years of experience will be a calculation based on age & qualification, and Grade will be randomly assigned using current proportions as at July 2018 as published in the HEA document referenced above.<jupyter_code>#Import pandas & numpy.
import pandas as pd
import numpy as np

# Set the seed so that the numbers can be reproduced.
np.random.seed(0)<jupyter_output><empty_output><jupyter_text>Pandas is a software library for data maniupulation & analysis.    Numpy is a python library useful for large, multi-dimensional arrays & matrices, along with functions to operate on arrays. Numpy.random is part of the numpy library and is what I will use to generate random data to simulate my dataframe.   np.random.seed sets the 'seed' for the random data. If the same seed is set the same numbers are generated each time. Changing the seed means a different set of numbers is generated.    The Dataset<jupyter_code>#Insert age, gender & qualification columns to dataframe
gender = ['male', 'female']
grade = ['Assistant Lecturer', 'Lecturer', 'Senior Lecturer 1', 'Senior Lecturer 2', 'Senior Lecturer 3', 'Registrar', 'President']
df = pd.DataFrame({'Age': np.random.randint(25,66,450),
                   'Gender': np.random.choice(gender, 450,p=[0.54, 0.46]), 
                   'Grade': np.random.choice(grade, 450, p=[.1573, .7584, .0225, .0393, .0169, .0028, .0028])})

#Function to assign qual depending on age, i.e. <27 cannot hold PhD
def qual(row):

                if row['Age']>= 27:

                    return np.random.choice (np.arange (8,11), p=[.62, .21, .17])
                
                else: 
                    
                    return np.random.choice (np.arange (8,10), p =[.666, .334])
                                  
qual =df.apply (lambda row: qual(row),axis=1)
df['Qualification'] = qual

#Function to determine experience based on age & qual
def exp(row):

                if row['Qualification']== 10:

                    return row['Age']-27
            
                elif row['Qualification']== 9:

                    return row['Age']-23
                
                elif row['Qualification']== 8: 
                    
                    return row['Age']-22
                                  
exp =df.apply (lambda row: exp(row),axis=1)
df['Experience'] = exp
df<jupyter_output><empty_output><jupyter_text>Procedure for generating variables    Age:I used the function random.randint, a discrete uniform distribution, to generate ages of academic staff, with a minimum age of 25 and a maxiumum age of 66. It is appropriate to use this type of distribution when you have a minimum & maximum value, but no insight into where within this range a particular value might fall. I made the assumption, based on minimum requirements for academic posts, that the youngest one can be appointed is 25, and currently one must retire on 31st August after 65th birthday, making 66 the maximum age.   Gender: I used np.random.choice to generate the data on gender, giving a 46%, 54% female/male breakdown as per the July 2018 HEA Institutional Staff Profiles by Gender report. This function generates a random sample from a given one dimensional array, the parameters entered being gender, size of output, and probabilities associated with each entry of gender, i.e. 54% male, 46% female.    Qualification:   Initially I had used a random normal distribution to assign qualification, however having ran the code I realised that I need to look at the relationship between age & qualification, as, for example, a 25 year old staff member is unlikely to hold a PhD, given that the academic minimum requirements are a qualification to at least Level 8 plus three years experience. A Level 8 qualification generally takes 3/4 years, a Masters 1 year and a PhD at least 4 years. I therefore made the assumption that the youngest a PhD holder could be is 27. The IUA report on PhD students in the Universities found that the average age of a PhD student is 26 years (data collected in 2008), and a 1997 OECD report found that the typical graduation age from PhDs in Ireland is 24-27 years (Organization for Economic Co-operation and Development, Education at a Glance: OECD Indicators, 1997).   I then had to create an if function to look at the age column, and if that was less than or equal to 27, randomly choose either Level 8 or Level 9 based on the current proportions of academic staff holding those qualifications - after researching the GMIT staff data I discovered that currently 21% of academic staff hold a PhD, 62% hold a Master's Degree, and the remainder an Honours Degree.  I applied the same function to those in the age column that were greater than 27, randomly choosing either a Level 8, 9 or 10 again using the current proportions of GMIT staff holding these qualifications.     Grade   To assign Grade I used the same function as for Gender; np.random.choice. This function was used as it allows you to assign probabilities for the various outcomes. I assigned seven variables to incorporate the current academic grades - Assistant Lecturer, Lecturer, Senior Lecturer 1, Senior Lecturer 2, Senior Lecturer 3, Registrar, President. The breakdown was again taken from the HEA publication 'Higher Education Institutional Staff Profiles by Gender'.     Years of Experience    This is a calculated variable based on age & qualifications. I took the age for each row, and if the 'Qualification' column was Level 10 subtracted 27 (18 years as school leaving age plus 4 years for completion of primary degree, 1 year for Masters & 4 years for PhD). Similarly if the 'Qualification' column was Level 9, subtracted 23, and Level 8 subtracted 22, using an 'if' statement and Pandas apply function to apply the calculation to each row.   Conclusions   Objective: To focus on just Senior Management Grades as per the real world phenomenon as stated in the introduction.<jupyter_code>#Import libraries to plot results, matplotlib & seaborn
import matplotlib as plt
import seaborn as sns
%matplotlib inline
#Show histogram based on Grade & Gender, just for Senior Management Grades
#First drop non-Senior Management Grades 
array = ['President', 'Registrar', 'Senior Lecturer 3', 'Senior Lecturer 2']
ndf=df.loc[df['Grade'].isin(array)]
ndf
#Display histogram by Grade & Gender
sns.countplot(x='Grade', hue='Gender',data= ndf, palette = "Set2")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.053675741210184064, 'fraction_numerical': 0.024468202358646534, 'mean_word_length': 3.795983086680761, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '58295', 'license': 'Apache-2.0', 'n_tokens_mistral': 2426, 'n_tokens_neox': 2113, 'n_words': 1301, 'path': 'Simulated data set using numpy.random.ipynb', 'repo_name': 'paulacduffy/simulatedatasetproject', 'size': '9107'}