<jupyter_start><jupyter_text>ADA final exam (Fall 2021)This exam consists of 2 parts.The first part has 3 tasks.The second part has 2 tasks.The two parts are completely independent of each other. Important: The data used for this exam **is not entirely real.** A great deal of it has been artificially created or modified. Conclusions drawn from this data have no real link to reality!<jupyter_code>%matplotlib inline

# General
import pandas as pd
import numpy as np

# For plotting
import matplotlib.pyplot as plt
import seaborn as sns

# For graph-related stuff
import networkx as nx

from scipy.stats import ttest_ind, bartlett, levene
# For regression analyses and statistical tests
import scipy.stats as stats
import statsmodels.stats
import statsmodels.formula.api as smf

# For classification
from sklearn import linear_model as lm
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score<jupyter_output><empty_output><jupyter_text>Part 1 — Faculty hiring networkFor the first part of the exam, you will analyze data from a faculty hiring network. After completing their Ph.D., researchers who want to pursue an academic career must compete for relatively few permanent positions (as “faculty”, i.e., professors) in a highly competitive job market. You will analyze a directed graph that captures the dynamics of this hiring process in the United States:- Each node $u$ in this graph represents a different American university.- For each university, we also provide a score that captures how highly  this university is ranked (e.g., Stanford University is ranked more highly than the University of Alabama). The higher the score, the more highly ranked the university.- Each directed edge $(u,v)$ in this graph represents a hiring decision where someone who did their Ph.D. at university $u$ subsequently was hired by university $v$. Each edge is also associated with a gender label, which specifies if the person hired was a man (M) or a woman (F).- Note that, when multiple researchers were trained at a given university $u$ and subsequently hired by university $v$, there will be multiple edges pointing from $u$ to $v$, each of which corresponds to a different person. Files (in the `./data/part-1/` folder)- `nodelist.tsv` a tab-separated file containing the nodes of the graph. Recall that each node represents a different university. The file has these columns:    - `u` — the node index;    - `score` — the rank of the university (the higher the better);    - `name` — the name of the university.- `edgelist.tsv` a tab-separated file containing the edges of the graph. Recall that each edge represents a hiring decision. The file has columns:    - `u` — the source university, where the person got their Ph.D.;    - `v` — the target university, where the person was hired;    - `gender` — the gender of the person. Task 1 — (20 points)1. Using the above files, create the directed graph representing the hiring network using `networkx`.Each node should contain the attributes `score` and `name`, and each edge should contain the attribute `gender`.Print the total numbers of nodes and edges in the graph.2. Create a plot that allows you to easily visualize **(a)** what fraction of all researchers in the graph are hired by the $N$ universities that hire the most; and  **(b)** what fraction of all researchers in the graph are trained by the $N$ universities whose students get hired the most.Here, $N$ could be any number between 1 and the number of universities.3. **/Multiple choice:/** Which centrality metric(s) is/are illustrated in the plot you made for the previous subtask? Select all that apply from the list below and **justify your answer.**    1. Betweenness centrality    2. In-degree centrality    3. Out-degree centrality    4. PageRank centrality4. **/True or false:/** Search engines such as Google have historically used PageRank centrality as a signal to determine the importance of pages on the Web. Is the following statement **true** or **false**? Justify. **"In the faculty hiring network, PageRank centrality is positively correlated with university score, that is, universities with higher scores are more likely to also have higher PageRank centrality."**<jupyter_code>df_task1_nodelist = pd.read_csv('data/part-1/nodelist.tsv', sep='\t')
# print(df_task1_nodelist.head())

df_task1_edgelist = pd.read_csv('data/part-1/edgelist.tsv', sep='\t')
df_task1_edgelist.head()


#1.1
edge_list = df_task1_edgelist[['u', 'v']].apply(tuple, axis=1)
digraph = nx.DiGraph()
digraph.add_edges_from(edge_list.values)


nodes = digraph.nodes()
names = df_task1_nodelist['name'].tolist()
scores = df_task1_nodelist['score'].tolist()
attr1 = {}
for node in nodes:
    attr1[node] = {'name' : names[node - 1], 'score' : scores[node - 1]}
nx.set_node_attributes(digraph, attr1)

attr2 = {}
genders = df_task1_edgelist['gender'].tolist()
for i, edge in enumerate(digraph.edges()):
    attr2[edge] = {'gender' : genders[i]}
nx.set_edge_attributes(digraph, attr2)

print(digraph.nodes[3])
print(digraph.edges[1, 2])

#1.2
hire_counts = df_task1_edgelist['v'].value_counts()
train_counts = df_task1_edgelist['u'].value_counts()
df_hire = pd.DataFrame(hire_counts)
df_train = pd.DataFrame(train_counts)
df_hire['hired_fraction'] = df_hire.v.apply(lambda x: x/df_hire['v'].sum())
df_hire = df_hire.rename_axis('num').reset_index()
df_train['trained_fraction'] = df_train.u.apply(lambda x: x/df_train['u'].sum())

sns.set_style('dark')
sns.set_context(rc = {'patch.linewidth': 0.0})
fig, ax = plt.subplots(figsize=(10,5), dpi= 300)
fig.suptitle('Barplot of Hired Fraction and N University')
sns.barplot(x=df_hire['num'], y=df_hire['hired_fraction'], color='b')
ax.tick_params(axis='x', labelsize=3, rotation=-90)

fig, ax = plt.subplots(figsize=(10,5), dpi= 300)
fig.suptitle('Barplot of Trained Fraction and N University')
sns.barplot(x=df_hire['num'], y=df_train['trained_fraction'], color='r')
ax.tick_params(axis='x', labelsize=3, rotation=-90)<jupyter_output><empty_output><jupyter_text>1.3for the plot (a)Bfor the plot (b)C 1.4False  Task 2 — (15 points)Each hiring decision $(u,v)$ connects two different universities with different scores.1. Divide the universities into quintiles according to their scores ($Q_1$ to $Q_5$, where $Q_5$ contains the highest-ranked universities, i.e., those with the highest scores). Create a succinct plot that captures, for each pair of quintiles ($Q_i$, $Q_j$), what fraction of people in the dataset who got their Ph.D. from a university in $Q_i$ were hired by a university in $Q_j$.2. **/Multiple choice:/** What does the plot made in the previous subtask reveal about the faculty hiring process? Select one of the following options and justify your answer.    1. Ph.D. graduates who are hired as professors usually go to universities that are better than the universities where they were trained.    2. Ph.D. graduates who are hired as professors usually go to universities that are around the same or worse than the universities where they were trained.3. **/True or false:/** Let $E$ be the set of edges from the hiring network, and let $X$ be defined as follows:$$X = \frac{|(u,v) \in E : u \in Q_1 \text{ and } v \in Q_5 |}{{|(u,v) \in E : u \in Q_1|}}$$ *Thinking critically about the data that was given to you,* is the following statement **true** or **false**? Justify your answer. **"If I get my Ph.D. in a 1st-quintile university, I have a chance of around X * 100%  of getting a job in a 5th-quintile university."**---**Hints:**- [pandas.qcut](https://pandas.pydata.org/docs/reference/api/pandas.qcut.html) makes splitting data into quantiles easy!<jupyter_code>#2.1

df_task1_nodelist.sort_values(by='score')
num = int(len(df_task1_nodelist) / 5)
q1 = df_task1_nodelist[0:num]['u'].tolist()
q2 = df_task1_nodelist[num:num*2]['u'].tolist()
q3 = df_task1_nodelist[num*2:num*3]['u'].tolist()
q4 = df_task1_nodelist[num*3:num*4]['u'].tolist()
q5 = df_task1_nodelist[num*4:num*5]['u'].tolist()
fractions = []
pairs = []
countx = 0
county = 0
for q_x in [q1,q2,q3,q4,q5]:
    countx += 1
    county = 0
    for q_y in [q1,q2,q3,q4,q5]:
        county += 1
        frac = df_task1_edgelist.loc[df_task1_edgelist['u'].isin(q_x) & df_task1_edgelist['v'].isin(q_y)].value_counts().sum() / len(df_task1_edgelist)
        fractions.append(frac)
        pairs.append(f'Q{countx}Q{county}')

fig, ax = plt.subplots(figsize=(10,5), dpi= 300)
fig.suptitle('Barplot of Hired fraction of QiQj pairs')
sns.barplot(x=pairs, y=fractions, color='g')
ax.tick_params(axis='x', labelsize=8)
ax.set_ylabel('Fraction')<jupyter_output><empty_output><jupyter_text>2.2B 2.3True  Task 3 — (25 points)Let the "source university" of a person be the university at which the person got their Ph.D., and let their "target university" be the university at which they were hired after their Ph.D.1. For each hiring decision $(u, v)$, let the score gain ($d$) be defined as the difference between the scores of the target university and the source university (target university score minus source university score; see hint for example). Calculate the average difference for men ($\bar{d_{M}}$) and for women ($\bar{d_{F}}$) in the dataset. Determine whether the difference between these two averages ($\bar{d_{M}}$ and $\bar{d_{F}}$) is statistically significant.2. **/True or false:/** Is the following statement **true** or **false**? Justify your answer. **"The results you obtained in Task 3.1 do not refute the hypothesis that women are disadvantaged in the faculty hiring process."**3. For each quintile $Q$ from Task 2.1, calculate the following: **(a)** the percentage of women among hiring decisions $(u, v)$ with $u \in Q$ ; and **b)** the average score gain for hiring decisions $(u, v)$ with $u \in Q$.4. **/Discuss:/** Considering the results obtained in Task 3.3, why is comparing the average score gain for men ($\bar{d_{M}}$) and women ($\bar{d_{F}}$) insufficient for determining the causal effect of gender on score gain?5. Use `statsmodels` to run a linear regression where hiring decisions (edges of the hiring network) are the data points, gender and source-university score are the predictors, and score gain is the outcome. Report the summary of your model.6. **/True or false:/** Considering the following statement, answer **true** or **false** and justify your answer. **"The results you obtained in Task 3.5 suggest that academic hiring decisions are unfair to women."**---**Hint** — The university with the best score is Stanford University (score 100), and the university with the 10th best score is the University of Washington (score 95.26). If someone was trained at Stanford and then got a job at the University of Washington, their score gain is $95.26 - 100 = -4.74$.<jupyter_code>#3.1
score_gain = []
for index in df_task1_edgelist.index:
    score_gain.extend((df_task1_nodelist['score'][df_task1_nodelist['u'] == df_task1_edgelist.loc[index]['v']].values - df_task1_nodelist['score'][df_task1_nodelist['u'] == df_task1_edgelist.loc[index]['u']].values).tolist())
df_task1_edgelist['score_gain'] = score_gain
print('gender: M')
print(df_task1_edgelist['score_gain'].loc[df_task1_edgelist['gender'] == 'M'].describe())
print('\t')
print('gender: F')
print(df_task1_edgelist['score_gain'].loc[df_task1_edgelist['gender'] == 'F'].describe())

print('Bartlett test: ', bartlett(df_task1_edgelist['score_gain'].loc[df_task1_edgelist['gender'] == 'M'], df_task1_edgelist['score_gain'].loc[df_task1_edgelist['gender'] == 'F']))
print('Levene test: ', levene(df_task1_edgelist['score_gain'].loc[df_task1_edgelist['gender'] == 'M'], df_task1_edgelist['score_gain'].loc[df_task1_edgelist['gender'] == 'F']))

print('T-test: ', ttest_ind(df_task1_edgelist['score_gain'].loc[df_task1_edgelist['gender'] == 'M'], df_task1_edgelist['score_gain'].loc[df_task1_edgelist['gender'] == 'F'], equal_var=False))<jupyter_output>gender: M
count    2379.000000
mean      -23.255171
std        25.625527
min       -96.213640
25%       -40.618060
50%       -18.444288
75%        -3.120091
max        66.031093
Name: score_gain, dtype: float64
	
gender: F
count    2009.000000
mean      -25.201988
std        25.571818
min       -94.182330
25%       -42.673745
50%       -19.570987
75%        -3.818861
max        42.462488
Name: score_gain, dtype: float64
Bartlett test:  BartlettResult(statistic=0.00958197282994193, pvalue=0.9220215846053008)
Levene test:  LeveneResult(statistic=0.7725435261037057, pvalue=0.3794794900265609)
T-test:  Ttest_indResult(statistic=2.5101515006112165, pvalue=0.012104657096317136)<jupyter_text>3.1pvalue = 0.012104657096317136 < 0.05So difference between these two averages is statistically significant. 3.2True<jupyter_code>#3.3

countx = 0
for q_x in [q1,q2,q3,q4,q5]:
    countx += 1
    print(f'Q{countx}:')
    print(df_task1_edgelist.loc[df_task1_edgelist['v'].isin(q_x)][df_task1_edgelist['gender'] == 'F']['gender'].value_counts() / len(df_task1_edgelist[df_task1_edgelist['gender'] == 'F']))
    print(f'average score_gain:')
    print(df_task1_edgelist.loc[df_task1_edgelist['v'].isin(q_x)][df_task1_edgelist['gender'] == 'F']['score_gain'].mean())
    print('\t')<jupyter_output>Q1:
F    0.372822
Name: gender, dtype: float64
average score_gain:
-4.156989848118447
	
Q2:
F    0.222001
Name: gender, dtype: float64
average score_gain:
-20.89413799980229
	
Q3:
F    0.169238
Name: gender, dtype: float64
average score_gain:
-36.43165082510985
	
Q4:
F    0.125436
Name: gender, dtype: float64
average score_gain:
-49.54378298996681
	
Q5:
F    0.110503
Name: gender, dtype: float64
average score_gain:
-60.0298608559266<jupyter_text>3.4Average score_gain only represents part of the effect. Still need to investigate other factor such as variance, max scores, min scores.<jupyter_code>#3.5<jupyter_output><empty_output><jupyter_text>Part 2 — Tennis interviewsFor the second part of the exam, you will work with text data extracted from questions asked during post-match tennis interviews and fromTV commentary done during the tennis match.Examples of interview questions:> That last set seemed like a faultless performance. Is that the way you saw it?> How disappointing is that loss for you?> As a spectator, would you rather watch a guy with big aces or a match with lots of rallies?Example of commentary text:>  Serena is desperate to shake herself into action. She lets out an almighty shriek when she goes 30-15 up, but Azarenka responds with a brilliant forehand. The level of Williams' scream goes up and up during the next point, and she lets out another roar when she belts a forehand cross court win the point. She wraps the game up next time round, and she looked something like back to her best there! Files (in the `data` folder)- `questions.tsv.gz` a tab-separated compressed file containing interview questionsand some metadata. We refer to this dataset as the "questions dataset."Each row contains a question made to a specific player after a match.The file has the following columns:    - `questions` — the text of the question asked; **Note that you will never be asked to use this row; it is merely meant to help you understand the data better**.    - `gender` — the gender of the player;    - `ranking` — the ranking of the player at the time;    - `similarity` — this column will be explained later in the exam.- `commentary.tsv.gz` a tab-separated compressed file containing TV commentary done duringthe tennis match. We refer to this dataset as the "commentary dataset." Each line containsa piece of commentary done in a tennis match.The file has the following columns:    - `commentary` — the text of the commentary; **Note that you will never be asked to use this row; it is merely meant to help you understand the data better**.    - `gender` — the gender of the players in the match;- `TFIDF.npz`, a file containing 2D numpy arrays saved in the `.npz` format thatwill be used in Task 4.Both datasets are **balanced regarding gender,** i.e., they have the same number of rows about men as about women.Commentary text, as well as interview questions, have been preprocessed to make your life easier, as follows:- We replaced all words starting with upper-case letters with the special string `'--ENT--'`. This was done to remove personal names (e.g., Alice, Bob), which could help predict whether the text is associated with a male or a female tennis player.- We removed the first word in each sentence. This was done because, for the first word of each sentence, it is difficult to know whether it is a personal name or simply a regular word.- We replaced all gendered pronouns in the text (he/him/his and she/her/hers) with the special string `'--PRON--'`. This, too, was done because these words would make it easy to predict whether the text is associated with a male or a female tennis player.- We removed punctuation and stop words (to reduce the dimensionality of the data).The above example commentary text thus appears as follows in the data provided to you:> desperate shake action lets almighty shriek --PRON-- goes 30 15 --ENT-- responds brilliant forehand level --ENT-- scream goes point --PRON-- lets roar --PRON-- belts forehand cross court win point wraps game time round --PRON-- looked like --PRON-- bestWe also provide some code for loading and displaying the dataset:<jupyter_code>#  Part 2 ! PROVIDED CODE !

questions = pd.read_csv("./data/part-2/questions.tsv.gz", sep="\t", compression="infer")
commentary = pd.read_csv("./data/part-2/commentary.tsv.gz", sep="\t", compression="infer")<jupyter_output><empty_output><jupyter_text>Task 4  (20 points)Your goal now is to investigate whether there is gender bias in the questions being asked to female vs. male players, i.e., whether men vs. women are asked different questions on average. You start your analysis with a classification framing.To make your life easier, we already provided a TF-IDF matrix for each dataset (`commentary.tsv.gz` and `questions.tsv.gz`) and with code to load these matrices (which are numpy arrays).- Each row in each matrix $M$ corresponds to a row in the corresponding dataframe, i.e., each row corresponds to a different question or a different piece of TV commentary.- Each column in each matrix corresponds to a different word from the vocabulary.- Each position $(i,j)$ in the matrix contains the TF-IDF score for word $j$ from the vocabulary in question or commentary $i$. That is, each term $M[i,j]$ corresponds to the term frequency (TF) of word $j$ in text $i$, multiplied by the inverse document frequency (IDF) of word $j$:$$M[i,j] = \text{TF}(i,j) * \text{IDF}(j)$$1. **/Multiple Choice:/** Which of the following is the correct way to calculate the term frequency and the inverse document frequency? Justify your answer, explaining what each of the two components should capture.    - Option A:    $$\text{TF}(i,j) = \text{number of times the $j$-th word occurs in all documents} $$    $$\text{IDF}(j) =  \log \frac{\text{number of documents the $j$-th word occurs in}}{\text{number of documents}}$$    - Option B:    $$ \text{TF}(i,j) = \text{number of times the $j$-th word occurs in the $i$-th document} $$    $$\text{IDF}(j) =  \log \frac{\text{number of documents the $j$-th word occurs in}}{\text{number of documents}}$$    - Option C:    $$ \text{TF}(i,j) = \text{number of times the $j$-th word occurs in all documents} $$    $$\text{IDF}(j) =  \log \frac{\text{number of documents}}{\text{number of documents the $j$-th word occurs in}}$$    - Option D:    $$ \text{TF}(i,j) = \text{number of times the $j$-th word occurs in the $i$-th document} $$    $$\text{IDF}(j) =  \log \frac{\text{number of documents}}{\text{number of documents the $j$-th word occurs in}}$$2. Leveraging the matrices described above, train two logistic regression classifiers (one for interview questions, the other for TV commentary) that use the TF-IDF vectors as features to predict the gender of the athletes. You should split each dataset into training and test sets using sklearn (see [sklearn.model_selection.train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)) using parameters `test_size=0.4` and `random_state=42`. You should use sklearn's logistic regression model (see [here](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.htmlsklearn.linear_model.LogisticRegression)) using parameters `C=10` (which controls regularization) and `max_iter=2000`. For each dataset, train the model on the training set (with 60% of the data) and report the accuracy on the test set (with 40% of the data).3. **/True or false:/** Considering the following statement, answer **true** or **false** and justify your answer. **"Based on the outcome of Task 4.2, the text from tennis commentary text is more gender-biased than post-match questions."**4. The parameter `C` above controls a regularization penalty. The lower `C`, the stronger the regularization penalty (caveat: sklearn’s convention is the reverse of the convention you saw in class, where a *higher* $\lambda$ parameter meant a stronger regularization penalty ). For the questions dataset, re-run the classification task using the same setup as in Task 4.2, but now with `C=2000`. Report the accuracy.5. **/Discuss:/** Explain why increasing `C` changed the results.<jupyter_code># 4.2 ! PROVIDED CODE !
tmp = np.load("./data/part-2/TFIDF.npz")
TFIDF_commentary = tmp['commentary']
TFIDF_questions = tmp['questions']

TFIDF_questions<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10842256524359152, 'fraction_numerical': 0.031567236823807535, 'mean_word_length': 4.815094339622641, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 36, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '228783', 'license': 'MIT', 'n_tokens_mistral': 6878, 'n_tokens_neox': 6389, 'n_words': 2658, 'path': 'plots/345737.ipynb', 'repo_name': 'WeiSHI7/datastory_template', 'size': '21573'}