<jupyter_start><jupyter_text>Creating a regression model that predicts the writing score using XGBoost<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

dataset = pd.read_csv('StudentsPerformance.csv')
dataset.head()
import seaborn as sns
fig,ax = plt.subplots(ncols=3,figsize=(20,5))
sns.boxplot(x='gender',y='writing score',data=dataset,ax=ax[0])
sns.boxplot(x='race/ethnicity',y='writing score',data=dataset,ax=ax[1])
sns.boxplot(x='parental level of education',y='writing score',data=dataset,ax=ax[2])
fig,ax = plt.subplots(ncols=2,figsize=(20,5))
sns.boxplot(x='lunch',y='writing score',data=dataset,ax=ax[0])
sns.boxplot(x='test preparation course',y='writing score',data=dataset,ax=ax[1])<jupyter_output><empty_output><jupyter_text>Here it shows that each of the non test score parameters affect the writing score. It seems that test preparation course and parental level of education plays the biggest part among all five of the parameters. Now we check whether the reading score and math score also affects the writing score.<jupyter_code>plt.scatter(dataset['math score'],dataset['writing score'],color='red')
plt.xlabel('math score')
plt.ylabel('writing score')
plt.show()
plt.scatter(dataset['reading score'],dataset['writing score'],color='red')
plt.xlabel('reading score')
plt.ylabel('writing score')
plt.show()<jupyter_output><empty_output><jupyter_text>Based on both of the graphs it seems that there is a strong correlation between reading score and math score on the writing score. Based on this we can assume that the students who score well on reading and math test are more academical and thus perform well on their writing test<jupyter_code>from sklearn import preprocessing

label = preprocessing.LabelEncoder()
df_categorical_encoded = pd.DataFrame()
for i in dataset.columns:
    df_categorical_encoded[i]=label.fit_transform(dataset[i])

from scipy.stats import chi2_contingency

def cramers_V(var1,var2):
    crosstab = np.array(pd.crosstab(var1,var2,rownames=None,colnames=None))
    stat = chi2_contingency(crosstab)[0]
    obs = np.sum(crosstab)
    mini = min(crosstab.shape)-1
    return(stat/(obs*mini))
rows =[]

for var1 in df_categorical_encoded:
    col = []
    for var2 in df_categorical_encoded:
        cramers = cramers_V(df_categorical_encoded[var1],df_categorical_encoded[var2])
        col.append(round(cramers,2))
    rows.append(col)

cramers_results = np.array(rows)
cramerv_matrix = pd.DataFrame(cramers_results, columns=df_categorical_encoded.columns,index=df_categorical_encoded.columns)
mask=np.triu(np.ones_like(cramerv_matrix,dtype=np.bool))
cat_heatmap = sns.heatmap(cramerv_matrix,mask=mask,vmin=-1,vmax=1,annot=True,cmap='BrBG')
cat_heatmap.set_title('Correlation Heatmap', fontdict={'fontsize':12},pad=12);<jupyter_output><empty_output><jupyter_text>The heatmap shows that the most significant parameter to affect the writing score is the reading score. We decide based on the results to use all the other parameters to help in creating the XGBoost model.<jupyter_code>X_writing = dataset.drop(['writing score'],axis=1)
y_writing = dataset.iloc[:,-1].values
X_writing.head()<jupyter_output><empty_output><jupyter_text>We then have to encode the parameters with string values (from gender to test preparation course)<jupyter_code>from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(sparse = False), list(range(5)))], remainder='passthrough')
X_writing = np.array(ct.fit_transform(X_writing))
print(X_writing)<jupyter_output>[[ 1.  0.  0. ...  1. 72. 72.]
 [ 1.  0.  0. ...  0. 69. 90.]
 [ 1.  0.  0. ...  1. 90. 95.]
 ...
 [ 1.  0.  0. ...  0. 59. 71.]
 [ 1.  0.  0. ...  0. 68. 78.]
 [ 1.  0.  0. ...  1. 77. 86.]]<jupyter_text>With the string values encoded we can proceed to train our model however we would also like to feature scale the math score and reading score parameter since the value range is rather wide<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_writing,y_writing,test_size=0.2,random_state=1)
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train[:,-2:]=sc.fit_transform(X_train[:,-2:])
X_test[:,-2:]=sc.transform(X_test[:,-2:])
print(X_train[0])<jupyter_output>[0.   1.   0.   0.   1.   0.   0.   0.   0.   0.   1.   0.   0.   1.
 0.   0.   1.   0.84 0.82]<jupyter_text>Now we can proceed to creating the XGBoost regression model<jupyter_code>from xgboost import XGBRegressor
regressor = XGBRegressor()
regressor.fit(X_train,y_train)
y_pred = regressor.predict(X_test)
np.set_printoptions(precision=2)
print(np.concatenate((y_pred.reshape(len(y_pred),1),y_test.reshape(len(y_test),1)),1))<jupyter_output>[[ 51.52  54.  ]
 [ 68.98  68.  ]
 [ 73.66  74.  ]
 [ 59.87  58.  ]
 [ 53.93  55.  ]
 [ 53.65  51.  ]
 [ 49.14  51.  ]
 [ 30.72  36.  ]
 [ 66.52  69.  ]
 [ 83.59  82.  ]
 [ 66.52  60.  ]
 [ 56.58  54.  ]
 [ 72.96  67.  ]
 [ 82.52  75.  ]
 [ 70.27  69.  ]
 [ 83.58  84.  ]
 [ 88.74  90.  ]
 [ 92.98  92.  ]
 [ 65.16  63.  ]
 [ 51.22  52.  ]
 [ 84.06  87.  ]
 [ 84.25  84.  ]
 [ 75.99  73.  ]
 [ 58.56  54.  ]
 [ 53.71  50.  ]
 [ 72.15  65.  ]
 [ 69.93  70.  ]
 [ 61.56  68.  ]
 [ 62.24  65.  ]
 [ 68.32  73.  ]
 [ 40.07  48.  ]
 [ 73.07  74.  ]
 [ 48.58  47.  ]
 [ 63.93  68.  ]
 [ 55.98  62.  ]
 [ 84.23  76.  ]
 [ 76.74  72.  ]
 [ 60.37  59.  ]
 [ 61.54  60.  ]
 [ 86.76  93.  ]
 [ 81.19  78.  ]
 [ 75.84  78.  ]
 [ 74.3   74.  ]
 [ 75.23  74.  ]
 [ 82.69  92.  ]
 [ 85.22  89.  ]
 [ 63.29  62.  ]
 [ 67.95  68.  ]
 [ 55.81  49.  ]
 [ 98.53  95.  ]
 [ 73.57  84.  ]
 [ 80.88  82.  ]
 [ 73.39  73.  ]
 [ 77.64  79.  ]
 [ 80.9   81.  ]
 [ 50.55  47.  ]
 [ 78.46  80.  ]
 [ 81.91  76.  ]
 [ 48.88  56. [...]<jupyter_text>Here is the result of the model against the test set, based on the results the model is rather accurate at predicting the scores<jupyter_code>from sklearn.metrics import r2_score
r2_score(y_test,y_pred)
from sklearn.model_selection import cross_val_score
accuracies = cross_val_score(estimator = regressor, X = X_train, y = y_train, cv = 10)
print("Accuracy: {:.2f} %".format(accuracies.mean()*100))
print("Standard Deviation: {:.2f} %".format(accuracies.std()*100))<jupyter_output>Accuracy: 91.59 %
Standard Deviation: 1.48 %
{'fraction_non_alphanumeric': 0.14249803613511391, 'fraction_numerical': 0.07682639434406913, 'mean_word_length': 3.8521341463414633, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '95235', 'license': 'MIT', 'n_tokens_mistral': 2761, 'n_tokens_neox': 2435, 'n_words': 580, 'path': 'XGBoost Predicting Writing Score.ipynb', 'repo_name': 'eseb96/Student-Exam', 'size': '6365'}