<jupyter_start><jupyter_text>Tirgul 9 A sample project analysis - Read the data- Filter data- Determine features- Determine label<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>The DatasetThe dataset contanins information on students and their grades in math, reading and writing.<jupyter_code>data = pd.read_csv('StudentsPerformance.csv')
data.tail()
# Clear N/A
data = data.dropna(axis=0)
data.tail()<jupyter_output><empty_output><jupyter_text>Apearantly there weren't any 'na's' in the data<jupyter_code># let's see some summary
data.pivot_table(['math score','reading score','writing score' ],'gender') # note the default is mean<jupyter_output><empty_output><jupyter_text>Looks like the male students are leading in Math, but are behind on Reading and Writing  Ploting a histogramLet's display a histogram for each subject by gender.Top is female.[pandas_hist](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.hist.html)<jupyter_code>data.groupby('gender').hist(figsize=(10,5))<jupyter_output><empty_output><jupyter_text>Let's calculate the variance and the range of scores<jupyter_code>data.groupby('gender').var()<jupyter_output><empty_output><jupyter_text>We can see that male students tend to have a smaller variance then the female students.  Correlation between scores<jupyter_code>scoreData = data[['math score','reading score','writing score']]
scoreData.tail()
scoreData.corr()<jupyter_output><empty_output><jupyter_text>The corrolation across subjects is quite high, but between reading and writing is near perfect.<jupyter_code># cmap='jet' refers to table colors
#  vmin=0.0 , vmax = 1 indicate the lower and upper bounderies of legend 
# annot=True display the value of each square
sns.heatmap(scoreData.corr(), vmin=0.0 , vmax = 1,cmap='jet' , annot=True)<jupyter_output><empty_output><jupyter_text>$\pi$-Charts[pie charts docs](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.plot.pie.html)<jupyter_code>parentEducData = data[["parental level of education"]]
parentEducData.tail()
parentEducData.value_counts() # counts the amount from each categorized value
parentEducData.value_counts().plot.pie(autopct='%1.1f%%')
# autopct display percents for each part<jupyter_output><empty_output><jupyter_text>Looking for more correlation<jupyter_code>data['mean score'] = scoreData.mean(axis=1)
data.tail()<jupyter_output><empty_output><jupyter_text>Is there any connection between parental level of education and lunch to grades?<jupyter_code>EducLunchMean_ScoreData = data[['parental level of education','lunch','mean score']].copy(deep=True)
EducLunchMean_ScoreData.tail()
EducLunchMean_ScoreData.pivot_table('mean score','parental level of education').sort_values("mean score")
sns.regplot(x='reading score', y='writing score', data=data);
# Plot data and a linear regression model fit
sns.regplot(x='reading score', y='math score', data=data);
# higher spread for decreased correlation
EducLunchMean_ScoreData.groupby('parental level of education')['mean score'].hist(alpha=0.5,legend=True,figsize=(10,10))
# We cannot use 'pivot_table' here since we do not wish to aggregate the data<jupyter_output><empty_output><jupyter_text>We can see that from "some college" to "master's degree" the difference is small<jupyter_code>EducLunchMean_ScoreData.pivot_table('mean score','lunch')
EducLunchMean_ScoreData.groupby('lunch')['mean score'].hist(alpha=0.5,legend=True)<jupyter_output><empty_output><jupyter_text>The lunch type tells us more about the student potential<jupyter_code>EducLunchMean_ScoreData['lunch_cat'] = EducLunchMean_ScoreData['lunch'].astype('category').cat.codes
EducLunchMean_ScoreData<jupyter_output><empty_output><jupyter_text>Standard = 1free/reduced = 0<jupyter_code>EducLunchMean_ScoreData.pivot_table('lunch_cat','parental level of education').sort_values(by='lunch_cat')<jupyter_output><empty_output><jupyter_text>It is interesting to see, that the lunch type is spread more or less equaly between the education levelsLet's check that if getting a free/reduced lunch is a good indecator for you score<jupyter_code>EducLunchMean_ScoreData.pivot_table('mean score','parental level of education','lunch',margins=True)<jupyter_output><empty_output><jupyter_text>- The bottom margin shows the score according to the lunch (free/standard)- The right margin shows the score accordint to the parents degree<jupyter_code>EducLunchMean_ScoreData.pivot_table('mean score','parental level of education','lunch',margins=True,aggfunc='std')
mean_parent = EducLunchMean_ScoreData.groupby('parental level of education')['mean score'].mean()

# We manually orderd the plot according to the degrees
order = [5,2,4,0,1,3]
plt.figure(figsize=(10,5))
plt.scatter(mean_parent.index[order],mean_parent.values[order])<jupyter_output><empty_output><jupyter_text>Boxplot for mean score<jupyter_code>EducLunchMean_ScoreData['mean score'].plot(kind='box')<jupyter_output><empty_output><jupyter_text>Trees!!<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error as mse

data = pd.read_csv('StudentsPerformance.csv')
data.tail()<jupyter_output><empty_output><jupyter_text>Preparing the data for learning<jupyter_code>X = pd.get_dummies(data[['gender','race/ethnicity','lunch','test preparation course']])
y = data[['math score','reading score', 'writing score']]

X.head()<jupyter_output><empty_output><jupyter_text>Let's remove the reduntant fields<jupyter_code>X = X.drop(columns=['gender_male','lunch_standard','test preparation course_none'])
X.head()
X_train,X_test,y_train,y_test = train_test_split(X,y['math score'],test_size=0.3,random_state=42)
model = DecisionTreeRegressor(random_state=42)

model.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Eval Function<jupyter_code>def eval(x_test,y_test,model):
    pred = model.predict(x_test)
    print("MSE: {:.3f}".format(mse(pred,y_test,squared=False)))
eval(X_test,y_test,model)<jupyter_output>MSE: 14.610<jupyter_text>Draw Tree function[plot_tree docs](https://scikit-learn.org/stable/modules/generated/sklearn.tree.plot_tree.html)<jupyter_code>import sklearn.tree as tree
def plot_tree(tree_model,feat,size=(15,10)):
    fig = plt.figure(figsize=size)
    tree.plot_tree(tree_model, 
                   feature_names = feat, 
                   filled=True,
                  fontsize=15)  
    plt.show()<jupyter_output><empty_output><jupyter_text>Tree Pruning<jupyter_code>model = DecisionTreeRegressor(max_depth=3,random_state=42)
model.fit(X_train,y_train)

eval(X_test,y_test,model)
plot_tree(model,X_test.columns,size=(30,20))
model = DecisionTreeRegressor(min_samples_split=100,random_state=42)
model.fit(X_train,y_train)

eval(X_test,y_test,model)
plot_tree(model,X_test.columns,size=(60,20))<jupyter_output>MSE: 14.577
{'fraction_non_alphanumeric': 0.11933208579242839, 'fraction_numerical': 0.010364185979559522, 'mean_word_length': 8.636615811373092, 'pattern_counts': {'":': 0, '<': 87, '<?xml version=': 0, '>': 87, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '60344', 'license': 'MIT', 'n_tokens_mistral': 2446, 'n_tokens_neox': 2316, 'n_words': 465, 'path': 'shai_yona_tirgol/9/tirgul9.ipynb', 'repo_name': 'Liron-Firuz/Data-analsis-lesson-2021A', 'size': '6947'}