<jupyter_start><jupyter_text>Part 0: Preprocessing- remove any superfluous data. - check for null values- check for repetition. - check for outliers Part 1: Analysis- Describe the demographic details of people most likely to pass the test-- plot each data point against likelihood to pass. This could be done with bar charts/scatter plots/histograms/check my viz document for ideas. Each should be plotted against pass to see the impact. -- this is referring to age/nationality/language- Describe the efficacy of the two interventions - the test prep course and the Dojo class-- i should look at my data vis doc to get ideas. i could plot dc/tp and pass together.-- compare these two features - Identify any other interesting trends from the data set and offer some analysis as to their importance or cause.-- here i can talk about hours studied. -- run through some of the other techniques in data viz or my useful code.-- the relationship between age and time studied is somewhat correlated. -- this may indicate why age appears to have some impact on the pass rate.Part 2: Model- i could try imputing for the hours studied- i should also encode country, language and sex<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import seaborn as sns
import os

plt.rcParams['font.size'] = 14
plt.figure(figsize=(12,5))

path = f"{os.getcwd()}/../raw_data/test_data.tsv"
df = pd.read_csv(path, sep='\t')
df.head()
df.nunique()
# all appears to be normal. 

df.isnull().sum()
# only hours studied and notes are lacking. 

df.notes.unique()
# The notes do not appear to be relevant to studying so I will ignore them at this stage. 
df = df.drop(['notes','first','last'],axis=1)
df.describe()
df.describe(include=['O'])<jupyter_output><empty_output><jupyter_text>Graphing The above pairplot gives some indication of the nature of the relationships between the aforementioned primary features and pass rate. These are as follows1. Dojo Class attendance with Pass Rate- The majority of students did not attend the Dojo course- The majority of students who attended the Dojo passed- The majority of students who did not attend the Dojo failed2. Test Prep attendance with Pass Rate- The majority of students did not attend the test prep course- For students who attended the test prep course, an approximately equal number passed and failed- The majority of students who who did not attend the test prep course failed, this difference was less than with the dojo. 3. Age with Pass Rate- The lower the age, the lower the pass rate became- Students younger than 36 were slightly more likely to fail than pass- Students older than 36 slightly more likely to pass than fail  Analytics 1.1 Demographics- Describe the demographic details of people most likely to pass the test- plot each data point against likelihood to pass. This could be done with bar charts/scatter plots/histograms/check my viz document for ideas. Each should be plotted against pass to see the impact. - this is referring to age/nationality/language<jupyter_code>#pass rate by country
country_pr = df[['country','pass']].groupby('country').sum()/df[['country','pass']].groupby('country').count()
print(country_pr.sort_values(by='pass'))
country_pr.sort_values(by='pass').plot(kind='bar')


# all countries had a pass rate of between 0.38 and 0.51. 
# The lowest pass rate was Finland and the highest was USA. 
#pass rate by language
lang_pr = (df[['lang','pass']].groupby('lang').sum()/df[['lang','pass']].groupby('lang').count())
print(lang_pr.sort_values(by='pass'))
lang_pr.sort_values(by='pass').plot(kind='bar')

# all languages had a pass rate of between 0.38 and 0.49. 
# The lowest pass rate was Finnish and the highest was French. 

sns.kdeplot(data=df.age[df['pass']],shade='red')
# blue is a pass
sns.kdeplot(data=df.age[df['pass']==False], shade='blue')
# orange is a fail.

# Age with Pass Rate
# - The pass rate was lowest for students aged 20 years old. The pass rate then gradually increases with age, peaking for students aged 50. 
# - Students younger than 36 were slightly more likely to fail than pass
# - Students older than 36 were slightly more likely to pass than fail<jupyter_output><empty_output><jupyter_text>Analytics 1.2 Intervention Comparison- Describe the efficacy of the two interventions - the test prep course and the Dojo class-- i should look at my data vis doc to get ideas. i could plot dc/tp and pass together.-- compare these two features  1. Dojo Class attendance with Pass Rate- The majority of students did not attend the Dojo course- The majority of students who attended the Dojo passed- The majority of students who did not attend the Dojo failed2. Test Prep attendance with Pass Rate- The majority of students did not attend the test prep course- For students who attended the test prep course, an approximately equal number passed and failed- The majority of students who who did not attend the test prep course failed, this difference was less than with the dojo.<jupyter_code>sns.kdeplot(data=df.dojo_class[df['pass']],shade='red')
sns.kdeplot(data=df.dojo_class[df['pass']==False],shade='blue')


sns.kdeplot(data=df.test_prep[df['pass']],shade='red')
sns.kdeplot(data=df.test_prep[df['pass']==False], shade='blue')<jupyter_output><empty_output><jupyter_text>Analytics 1.3 Interesting Trends- Identify any other interesting trends from the data set and offer some analysis as to their importance or cause.-- here i can talk about hours studied. -- run through some of the other techniques in data viz or my useful code.-- the relationship between age and time studied is somewhat correlated. -- this may indicate why age appears to have some impact on the pass rate.<jupyter_code>sns.heatmap(df.corr())<jupyter_output><empty_output><jupyter_text>The above heatmap shows us that most variables are not correlated. The highest correlated variables are as follows (descending order):1. Dojo Class attendance with Pass Rate2. Hours Studied with Pass Rate3. Age with Pass RateThese would appear to be my primary features influencing Pass Rate.I will also investigate the role of the test prep course in relation to pass rate, as the challenge instructs.<jupyter_code>plt.Figure(figsize=(10,10))
sns.pairplot(data=df,hue='pass', palette=['red','blue'])<jupyter_output><empty_output><jupyter_text>Hours Studied with Pass Rate- The lower the number of hours studied, the lower the pass rate became- Students who studied less than 8 hours were slightly more likely to fail than pass- Students who studied more than 8 hours were slightly more likely to pass than failHours Studied with Age- The lower the number of hours studied, the lower the pass rate became- Students who studied less than 8 hours were slightly more likely to fail than pass- Students who studied more than 8 hours were slightly more likely to pass than fail  Interesting Trends<jupyter_code>import plotly.express as px
plt.figure(figsize=(20, 10))
fig = px.scatter(
    df,
    x="hours_studied",
    y="age",
    color="pass",
    hover_name="country",
    opacity=0.2
    )
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07992695603315073, 'fraction_numerical': 0.008568619188088215, 'mean_word_length': 4.793327908868999, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '421265', 'license': 'MIT', 'n_tokens_mistral': 1983, 'n_tokens_neox': 1887, 'n_words': 951, 'path': 'notebooks/analysis_wp.ipynb', 'repo_name': 'xvarc/educational_analytics_submission', 'size': '7119'}