<jupyter_start><jupyter_text>You can find dataset description here https://archive.ics.uci.edu/ml/datasets/Student+Performance. Data is in "Data Folder" name "student.zip"  How to complete this assignment First download the data and extract two tables from the link above. Then complete 9 graded tasks below, the score is given in brackets. Finally submit the resulting `.ipynb` file to rs-app Auto-test.- **Do not delete or rename the variables** given before the inscription 'your code here', they are needed for correct verification.- **Do not change the code** in the last Notebook cell, it is required for server side.- Your **Notebook must run completely without errors** in order to be graded! Please check everything before submission by going _Cell_ -> _Run All_  Here goes!  Import necessary libraries<jupyter_code>import pandas as pd
from numpy import mean<jupyter_output><empty_output><jupyter_text>Read the data<jupyter_code>mat = pd.read_csv("student-mat.csv", sep=";")
por = pd.read_csv("student-por.csv", sep=";")<jupyter_output><empty_output><jupyter_text>Look at first five rows of table with students math grades. In all other tasks, unless otherwise stated, we use it.<jupyter_code>mat.head()<jupyter_output><empty_output><jupyter_text>**Task 1 (0.25 points).** Count number of boys and girls in the dataset.<jupyter_code>mat[mat['sex'] == 'M'].iloc[:, [1]]
print(len(mat[mat['sex'] == 'M'].iloc[:, [1]]), len(mat[mat['sex'] == 'F'].iloc[:, [1]]))
boys = len(mat[mat['sex'] == 'M'].iloc[:, [1]])
girls = len(mat[mat['sex'] == 'F'].iloc[:, [1]])<jupyter_output><empty_output><jupyter_text>**Task 2 (0.25 points).** Count number of 17 year old students having math grade G3 between 10 and 15 inclusively<jupyter_code>mat[(mat['age'] == 17) & ((mat['G3'] > 9) & (mat['G3'] < 16))].head()
students_17_year_old = len(mat[(mat['age'] == 17) & ((mat['G3'] > 9) & (mat['G3'] < 16))])<jupyter_output><empty_output><jupyter_text>**Task 3 (0.5 points).** List top-10 students having biggest average grade (G1, G2, G3)  in math. Count number of boys from that list.<jupyter_code>col = mat.loc[: , "G1": "G3"]
mat['G123'] = col.mean(axis=1)
top = mat.sort_values(by=['G123', 'sex'], ascending=(False, False)).head(10)
print(len(top[top['sex'] == 'M']))
boys_from_top_10 = len(top[top['sex'] == 'M'])<jupyter_output><empty_output><jupyter_text>**Task 4 (1 point).** Count mean and standard deviation of absences for students with romantic relationship and access to internet<jupyter_code>target = mat[(mat['romantic'] == 'yes') & (mat['internet'] == 'yes')]
target["absences"].mean()
target["absences"].std()
mean = target["absences"].mean()
std = target["absences"].std()<jupyter_output><empty_output><jupyter_text>**Task 5 (1.5 points).** Count number of students for each existing age. Return list with counts sorted by students age in ascending order. Note: On 0 index in list we have number of 15 year old students, on 1st index number of 16 year old students, ...<jupyter_code>mat.groupby(['age']).size().values.tolist()
student_counts_for_each_age = mat.groupby(['age']).size().values.tolist()<jupyter_output><empty_output><jupyter_text>**Task 6* (2 points).** Find most frequent pair (Fjob, Mjob) of students from math grade table. Note 1: Do not take student into account if at least one of parents job name is "other". Note 2: Pairs (Fjob: teacher, Mjob: at_home) and (Fjob: at_home, Mjob: teacher) are different and their occurrences should be counted separately.<jupyter_code>mat.head()<jupyter_output><empty_output><jupyter_text>Hint: groupby (Fjob, Mjob), count rows, descending sort by count, reset_index, take first row values.<jupyter_code>almost = mat[(mat['Mjob'] != 'other') & (mat['Fjob'] != 'other')].groupby(['Fjob', 'Mjob'])\
.size().reset_index(name = 'size').sort_values(['size'], ascending=False).head(1)
almost = almost[['Fjob', 'Mjob']].values.tolist()
print(almost[0])
father = almost[0][0]
mother = almost[0][1]<jupyter_output><empty_output><jupyter_text>**Task 7 (1 point).** Count number of students presenting both in mat and por tables. _Note 1_: That's a pity that tables haven't foreign keys for joining together. The uniqueness of the student is determined by the uniqueness of the set of all fields except subject fields (G1, G2, G3). _Note 2_: Finally you'd better to come up with table having all student personal information and non-null grades in math ["G1_mat", "G2_mat", "G3_mat"] and ["G1_por", "G2_por", "G3_por"]. Joined dataframe could be helpful for further usage. Note 3: probably you would like to use "merge" to avoid duplicating column names from right and left dataframe.<jupyter_code>por = por.rename(columns={'G1': 'G1_por', 'G2': 'G2_por', 'G3': 'G3_por'})
mat = mat.rename(columns={'G1': 'G1_mat', 'G2': 'G2_mat', 'G3': 'G3_mat'})
por.duplicated(keep='first')
both = mat.merge(por,  how='inner', suffixes=('_mat', '_por'))
both.head()
both_present = len(both)<jupyter_output><empty_output><jupyter_text>**Task 8 (1.5 points).** Find absolute differences between average mat and por student final grades for each school (GP & MS)<jupyter_code>mat_list = mat.groupby('school')['G3_mat'].mean().sort_values(ascending=False).tolist()

por_list = por.groupby('school')['G3_por'].mean().sort_values(ascending=False).tolist()
GP_diff = abs(mat_list[0] - por_list[0])
MS_diff = abs(mat_list[1] - por_list[1])
print(GP_diff, MS_diff)<jupyter_output>2.0868608045953643 0.8026163909195851<jupyter_text>**Task 9* (2 points).** Find top-5 linearly dependent pairs of numeric student features from mat table. _Note 1_: Use absolute Pearson correlation value for measuring the linear relationship between features. Correlation coefficients are used in statistics to measure how strong a relationship is between two variables. There are several types of correlation coefficients, but the most popular is Pearson’s. It can take value from -1 to 1 inclusive.  - A correlation coefficient of 1 means that for every positive increase in one variable, there is a positive increase of a fixed proportion in the other. For example, shoe sizes go up in (almost) perfect correlation with foot length. - A correlation coefficient of -1 means that for every positive increase in one variable, there is a negative decrease of a fixed proportion in the other. For example, the amount of gas in a tank decreases in (almost) perfect correlation with speed. - Zero means that for every increase of one variable we do not expect always increase or decrease of another variable. It is not a proof, that variables are not related with each other. That only means NO LINEAR relationship between them. Suppose we have variables x and y connected by a sinusoidal law $y=sin(x)$. There is obvious no linear relationship between them (for $x \in [0, \frac{\pi}{2}]$ y increases unevenly ant later for $x \in [\frac{\pi}{2}, \pi]$ y decreases unevenly). Even though they relate to each other by more complex law. More mathematical explanation https://en.wikipedia.org/wiki/Correlation_and_dependence.<jupyter_code># Just a short example
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

x = np.array(np.arange(0, 314, 10)) / 100
y1 = -x + 10
y2 = np.sin(x)

fig, (ax1, ax2) = plt.subplots(2, 1)
ax1.scatter(x, -x + 10)
ax1.set_title("Pearson correlation coefficient for variables x and y=(-x + 10) is %.2f. Negative correlation."
      % np.corrcoef(x, y1)[0][1])
ax1.set_xlabel("X")
ax1.set_ylabel("Y =-X + 10")
ax2.scatter(x, y2)
ax2.set_title("Pearson correlation coefficient for variables x and y=sin(x) is %.2f. No correlation."
      % np.corrcoef(x, y2)[0][1])
ax2.set_xlabel("X")
ax2.set_ylabel("Y =sin(X)")

plt.subplots_adjust(hspace=1.0)
plt.show()<jupyter_output><empty_output><jupyter_text>_Note 2_: Use Pandas _.corrcoef_  for calculating Pearson correlation https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.corr.html. Return list of lists of column name pairs by absolute correlation value in descending order. Each pair of names should follow the alphabet order.<jupyter_code>mat.drop('G123', inplace=True, axis=1)
mat = mat.rename(columns={'G1_mat': 'G1', 'G2_mat': 'G2', 'G3_mat': 'G3'})
correlation_data = mat.corr(method='pearson').unstack().sort_values(ascending=False).head(40)
print(correlation_data)
answer = correlation_data[correlation_data != 1].head(20).index[[i for i in range(0, 20)]].tolist()
answer
answer
real_answer = list()
for tuple_ in answer:
    real_answer.append(list(tuple_))
real_answer
new_answer = []
for list_ in real_answer:
    new_answer.append(sorted(list_))
new_answer = new_answer[::2]
new_answer[0:5]
top5_correlations = new_answer[0:5]<jupyter_output><empty_output><jupyter_text>**Task 10 (bonus no grading).** Plot 5 scatterplots showing relationship between most correlated pairs of variables from the previous task.  Warning! Do not change anything in this area.<jupyter_code>with open('student_answers.txt', 'w') as file:
    file.write(f"{boys} {girls}\n")
    file.write(f"{students_17_year_old}\n")
    file.write(f"{boys_from_top_10}\n")
    file.write(f"{round(mean, 3)} {round(std, 3)}\n")
    file.write(f"{student_counts_for_each_age}\n")
    file.write(f"{father} {mother}\n")
    file.write(f"{both_present}\n")
    file.write(f"{round(GP_diff, 3)} {round(MS_diff, 3)}\n")
    file.write(f"{top5_correlations}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14017695341648012, 'fraction_numerical': 0.02537042959172796, 'mean_word_length': 5.848175182481752, 'pattern_counts': {'":': 1, '<': 62, '<?xml version=': 0, '>': 63, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '100749', 'license': 'Apache-2.0', 'n_tokens_mistral': 3202, 'n_tokens_neox': 3039, 'n_words': 999, 'path': 'first_week/Pandas_data_manipulations.ipynb', 'repo_name': 'myusernameisuseless/rolling_scopes_ml', 'size': '9381'}