<jupyter_start><jupyter_text>Challenge - What Do You See? Data Fundamentals Lesson 2 Part 4I have chosen a dataset from the University of Minho in Portugal. The dataset is a collection of student grades for two subjects (math and Portuguese) and numerous demographic and social data points for each student.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Can they be merged?First I will create data frames for the two datasets (one for math and one for Portuguese) and look at some of the values to see if I can merge them into one data frame.<jupyter_code>math_df = pd.read_csv('student-mat.csv', delimiter=';')

math_df.head()
port_df = pd.read_csv('student-por.csv', delimiter=';')

port_df.head()<jupyter_output><empty_output><jupyter_text>From these subsets of the data, it looks like I will be able to merge these data frames after some small adjustments. The grades for each term are labeled the same for both subjects and the number of absences is different for each class. To adjust for this before trying to merge, I will rename those columns in each data frame to be subject specific.<jupyter_code>math_df.rename(columns={'paid': 'math_paid',
                        'absences': 'math_absences', 
                        'G1': 'math_grades1', 
                        'G2': 'math_grades2', 
                        'G3': 'math_grades3'}, inplace=True)

port_df.rename(columns={'paid': 'port_paid',
                        'absences': 'port_absences', 
                        'G1': 'port_grades1', 
                        'G2': 'port_grades2', 
                        'G3': 'port_grades3'}, inplace=True)

math_df.head()
merged_df = math_df.merge(port_df)

merged_df.head()<jupyter_output><empty_output><jupyter_text>After making some plots, I realized that the values in study time are non-descriptive categoricals, and rather than try to relabel my axes, I am going to change the values to be more descriptive strings.Edit: I tried using string values, but they did not order themselves appropriately so I commented out this code and will use descriptive text in the axis label.<jupyter_code>#merged_df['studytime'].replace({1: '<2', 2: '2-5', 3: '5-10', 4: '>10'}, inplace=True)<jupyter_output><empty_output><jupyter_text>After some work with different plots, I realized I would like a new column that better summarizes math growth and one that summarizes Portuguese growth.<jupyter_code>merged_df['math_growth'] = (merged_df['math_grades2'] - merged_df['math_grades1']) + (merged_df['math_grades3'] - merged_df['math_grades2'])

merged_df['port_growth'] = (merged_df['port_grades2'] - merged_df['port_grades1']) + (merged_df['port_grades3'] - merged_df['port_grades2'])

merged_df.head()<jupyter_output><empty_output><jupyter_text>What questions are we trying to answer?Now that all the data is in a single data frame, we can begin to create visualizations. But what questions are we trying to answer with this data. I don't have a ton of context for these schools and this region, but I can explore some areas of personal curiosity as a teacher seeking correlation. I think the first question I would like to answer is whether absolute study time correlates to greater success in both subjects or not.<jupyter_code>sns.set()
sns.set_style('whitegrid')
plt.figure(figsize=(10, 5))

plt.subplot(1, 2, 1)
plt.hist2d(x=merged_df['studytime'], y=merged_df['math_growth'], bins=4)
plt.ylabel('Change in Math Grades')
plt.xlabel('Total Weekly Study Time (<2, 2-5, 5-10, >10 hours)')
plt.title('Math Grades as a Function of Study Time')

plt.subplot(1, 2, 2)
plt.hist2d(x=merged_df['studytime'], y=merged_df['port_growth'], bins=4)
plt.ylabel('Change in Portuguese Grades')
plt.xlabel('Total Weekly Study Time (<2, 2-5, 5-10, >10 hours)')
plt.title('Portuguese Grades as a Function of Study Time')

plt.show()<jupyter_output><empty_output><jupyter_text>Comments on Plot 1Without controlling for other variables, there does not appear to be the expected correlation. It appears that students experience diminishing returns when studying more than 5 hours per week. Most students seem to study 2-5 hours per week and their grades are  more likely to drop in math and rise in Portuguese over the course of the year.   So what should we look into next?Parents may be interested in this next one. I will be exploring whether there is a correlation between paid support in a subject and growth over the course of the year.<jupyter_code>plt.figure(figsize=(10, 5))

plt.subplot(1, 2, 1)
plt.bar(x=merged_df['math_paid'], height=merged_df['math_growth'])
plt.ylabel('Change in Math Grades')
plt.xlabel('Received Paid Support')
plt.title('Math Growth as a Function of Paid Support')

plt.subplot(1, 2, 2)
plt.bar(x=merged_df['port_paid'], height=merged_df['port_growth'])
plt.ylabel('Change in Portuguese Grades')
plt.xlabel('Received Paid Support')
plt.title('Portuguese Growth as a Function of Paid Support')

plt.show()
merged_df.boxplot(column='math_growth', by='math_paid')
plt.title('')
plt.xlabel('Received Paid Support')
plt.ylabel('Change in Math Grades')
plt.show()
merged_df.boxplot(column='port_growth', by='port_paid')
plt.title('')
plt.xlabel('Received Paid Support')
plt.ylabel('Change in Portuguese Grades')
plt.show()<jupyter_output><empty_output><jupyter_text>Comments on Plot 2The bar plot was easier to make but did not communicate the message I was hoping to convey. Using the boxplots, I can see that the interquartile ranges for students that paid a tutor and those that did not are comparable. The only striking difference is the lack of outliers for students who received tutoring in Portuguese.Still this comparison is not telling in itself as there are many other variables that have not been controlled for. One could assume that students receiving tutoring have a history of poor performance and so keeping on track with the rest of the class could be an improvement.   School RivalryLet's see how one school does compared to another in final grades for math and for Portuguese.<jupyter_code>sch_gp_df = merged_df.loc[merged_df['school'] == 'GP']
sch_ms_df = merged_df.loc[merged_df['school'] == 'MS']
plt.figure(figsize=(10, 5))

plt.subplot(1, 2, 1)
plt.hist(sch_gp_df['math_grades3'], density=True, alpha=.5, color='green')
plt.hist(sch_ms_df['math_grades3'], density=True, alpha=.5, color='red')
plt.title('Distribution of Final Math Grades by School')
plt.xlabel('Final Math Grades')

plt.subplot(1, 2, 2)
plt.hist(sch_gp_df['port_grades3'], density=True, alpha=.5, color='green')
plt.hist(sch_ms_df['port_grades3'], density=True, alpha=.5, color='red')
plt.title('Distribution of Final Portuguese Grades by School')
plt.xlabel('Final Portuguese Grades')

plt.show()<jupyter_output><empty_output><jupyter_text>Comments on Plot 3It appears that <NAME> (green) is doing a little bit better than <NAME> (red). Silveira has a pretty normal distribution for math grades centered on the median grade. Pereira seems skewed a bit to the right.In Portuguese, again just from the looks of it, Pereira is skewed right and Silveira is concentrated closer to the median.  Who wants higher education?Let's see if there is any correlation between the slightly better end of year performance at Gabriel Pereira school and the number of students that are planning to move on to higher education.<jupyter_code>plt.figure(figsize=(10,5))

plt.subplot(1, 2, 1)
sns.countplot(x='higher', data=sch_gp_df)
plt.title('Higher Ed Choices at <NAME>')
plt.xlabel('Continuing to Higher Ed')

plt.subplot(1, 2, 2)
sns.countplot(x='higher', data=sch_ms_df)
plt.title('Higher Ed Choices at <NAME>')
plt.xlabel('Continuing to Higher Ed')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09973080374310986, 'fraction_numerical': 0.013331624150749904, 'mean_word_length': 5.196981731532962, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '369226', 'license': 'Apache-2.0', 'n_tokens_mistral': 2490, 'n_tokens_neox': 2294, 'n_words': 899, 'path': 'Kessler_Jeff_2-2-4_DataVizChallenge.ipynb', 'repo_name': 'tjeffkessler/eda_port_ed_data', 'size': '7847'}