<jupyter_start><jupyter_text>Student's Final Grade Prediction  Introduction The material for this mini-project is taken from [Here](https://archive.ics.uci.edu/ml/datasets/student+performance). The objective is to gain insight into the problems of low scoring students and construct a machine learning model to predict their final grade (third period grade).  Attribute Information - **school** - student's school *(binary: "GP" - <NAME> or "MS" - Mousinho da Silveira)*- **sex** - student's sex *(binary: "F" - female or "M" - male)*- **age** - student's age *(numeric: from 15 to 22)*- **address** - student's home address type *(binary: "U" - urban or "R" - rural)*- **famsize** - family size *(binary: "LE3" - less or equal to 3 or "GT3" - greater than 3)*- **Pstatus** - parent's cohabitation status *(binary: "T" - living together or "A" - apart)*- **Medu** - mother's education *(numeric: 0 - none,  1 - primary education (4th grade), 2 – 5th to 9th grade, 3 – secondary education or 4 – higher education)*- **Fedu** - father's education *(numeric: 0 - none,  1 - primary education (4th grade), 2 – 5th to 9th grade, 3 – secondary education or 4 – higher education)*- **Mjob** - mother's job *(nominal: "teacher", "health" care related, civil "services" (e.g. administrative or police), "at_home" or "other")*- **Fjob** - father's job *(nominal: "teacher", "health" care related, civil "services" (e.g. administrative or police), "at_home" or "other")*- **reason** - reason to choose this school *(nominal: close to "home", school "reputation", "course" preference or "other")*- **guardian** - student's guardian *(nominal: "mother", "father" or "other")*- **traveltime** - home to school travel time *(numeric: 1 - 1 hour)*- **studytime** - weekly study time *(numeric: 1 - 10 hours)*- **failures** - number of past class failures *(numeric: n if 1<=n<3, else 4)*- **schoolsup** - extra educational support *(binary: yes or no)*- **famsup** - family educational support *(binary: yes or no)*- **paid** - extra paid classes within the course subject (Math or Portuguese) *(binary: yes or no)*- **activities** - extra-curricular activities *(binary: yes or no)*- **nursery** - attended nursery school *(binary: yes or no)*- **higher** - wants to take higher education *(binary: yes or no)*- **internet** - Internet access at home *(binary: yes or no)*- **romantic** - with a romantic relationship *(binary: yes or no)*- **famrel** - quality of family relationships *(numeric: from 1 - very bad to 5 - excellent)*- **freetime** - free time after school *(numeric: from 1 - very low to 5 - very high)*- **goout** - going out with friends *(numeric: from 1 - very low to 5 - very high)*- **Dalc** - workday alcohol consumption *(numeric: from 1 - very low to 5 - very high)*- **Walc** - weekend alcohol consumption *(numeric: from 1 - very low to 5 - very high)*- **health** - current health status *(numeric: from 1 - very bad to 5 - very good)*- **absences** - number of school absences *(numeric: from 0 to 93)*- **G1** - first period grade *(numeric: from 0 to 20)*- **G2** - second period grade *(numeric: from 0 to 20)*- **G3** - final grade *(numeric: from 0 to 20, output target)*  Load the Data<jupyter_code>mat_data = pd.read_csv('datasets/student-mat.csv', sep=';')
por_data = pd.read_csv('datasets/student-por.csv', sep=';')
mat_data.head()
por_data.head()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>from collections import defaultdict
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()<jupyter_output><empty_output><jupyter_text>Choosing Dataset to Use<jupyter_code>print('Math Course Dataset: {} X {}'.format(mat_data.shape[0], mat_data.shape[1]))
print('Portuguese Course Dataset: {} X {}'.format(por_data.shape[0], por_data.shape[1]))<jupyter_output>Math Course Dataset: 395 X 33
Portuguese Course Dataset: 649 X 33<jupyter_text>The Portuguese Course Dataset has more rows than the Mathematics Course Dataset, both of which have the same number of columns. I will cover this issue using Portuguese Data Sets.<jupyter_code>data = por_data.copy()<jupyter_output><empty_output><jupyter_text>My intention in making a copy of **por_data** to **data** is to shorten the variable name. **data** is the common variable name I use for the main data.  Dataset's Information<jupyter_code>data.describe()
data.info()
data.isnull().sum().sum()<jupyter_output><empty_output><jupyter_text>The observation shows that:- There is a student with very high rates of absenteeism.- Both numeric and categorical features are present.- There are no missing values in the data set.  Correlation<jupyter_code>corr = data.corr()
mask = np.triu(np.ones_like(corr, dtype=bool))
plt.figure(figsize=(15,12))
sns.heatmap(corr, cmap='Greens', mask=mask, annot=True);<jupyter_output><empty_output><jupyter_text>All features appear to have a fairly low correlation with the target label, except that G1 and G2 are highly correlated.  Categorical Features<jupyter_code>for col in data.columns:
    if col not in list(data._get_numeric_data()):
        print('{}: {}'.format(col, sorted(data[col].unique())))<jupyter_output>school: ['GP', 'MS']
sex: ['F', 'M']
address: ['R', 'U']
famsize: ['GT3', 'LE3']
Pstatus: ['A', 'T']
Mjob: ['at_home', 'health', 'other', 'services', 'teacher']
Fjob: ['at_home', 'health', 'other', 'services', 'teacher']
reason: ['course', 'home', 'other', 'reputation']
guardian: ['father', 'mother', 'other']
schoolsup: ['no', 'yes']
famsup: ['no', 'yes']
paid: ['no', 'yes']
activities: ['no', 'yes']
nursery: ['no', 'yes']
higher: ['no', 'yes']
internet: ['no', 'yes']
romantic: ['no', 'yes']<jupyter_text>This section has been displayed in the Attribute Information section. I computed this to make it easier for me to take care of it.  Additional Information<jupyter_code>in_trouble = data[data['G3'] < 10]<jupyter_output><empty_output><jupyter_text>I will screen students with a G3 lower than 10.<jupyter_code>threshold = defaultdict(dict)
for feature in data._get_numeric_data():
    threshold[feature]['lower'] = data[feature].quantile(0.25)
    threshold[feature]['upper'] = data[feature].quantile(0.75)
    threshold[feature]['mean'] = round(data[feature].mean(), 2)<jupyter_output><empty_output><jupyter_text>DataFrame<jupyter_code>trouble_df = defaultdict(list)
for index, row in in_trouble.iterrows():
    for feature in data._get_numeric_data():
        if row[feature] < threshold[feature]['lower']:
            trouble_df['Student ID'].append(index)
            trouble_df['Feature'].append(feature)
            trouble_df['Value'].append(row[feature])
            trouble_df['Mean'].append(threshold[feature]['mean'])
            
        elif row[feature] > threshold[feature]['upper']:
            trouble_df['Student ID'].append(index)
            trouble_df['Feature'].append(feature)
            trouble_df['Value'].append(row[feature])
            trouble_df['Mean'].append(threshold[feature]['mean'])
            
trouble_df = pd.DataFrame(trouble_df)
def frame(index):
    temp_df = trouble_df[trouble_df['Student ID'] == index].reset_index().drop(['index', 'Student ID'], axis=1)
    temp_df.columns.name = 'Student ID: {}'.format(index)
    return temp_df<jupyter_output><empty_output><jupyter_text>Plot<jupyter_code>trouble_plot = {}
for index, row in in_trouble.iterrows():
    features = trouble_df[trouble_df['Student ID'] == index]['Feature'].values
    
    student = pd.DataFrame(row[features]).T
    student['Legend'] = 'Student'
    
    mean = pd.DataFrame(data[features].mean()).T
    mean['Legend'] = 'Mean'
    
    bind = pd.concat([student, mean]).melt('Legend')
    trouble_plot[index] = bind
def plot(index):
    print('Student ID: {}'.format(index))
    sns.catplot(x='variable', y='value', hue='Legend', data=trouble_plot[index], kind='bar',
                palette=sns.color_palette(['blue', 'green']))
    plt.plot()<jupyter_output><empty_output><jupyter_text>Evaluate Student's Problem<jupyter_code>trouble_id = trouble_df['Student ID'].unique()
student_id = trouble_id[0]

# Use this to randomly issue student ID.
# student_id = np.random.choice(trouble_id)
frame(student_id)
plot(student_id)<jupyter_output>Student ID: 18<jupyter_text>The student with ID 18 has the following problems compared to average students:- Feel more failures in the previous class.- Get more free time after school.- Hang out with friends more often.- Consume more alcohol on weekends.- Has a lower value in the previous period.  Modeling<jupyter_code>import sys; sys.path.append('../')
# Model
from sklearn.linear_model import Lasso
from sklearn.svm import SVR
from sklearn.ensemble import ExtraTreesRegressor
from xgboost import XGBRegressor
# Feature Engineering
from fd_lib.feature_engineering import BestPair
# Model Selection
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import train_test_split
# Pipeline
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.model_selection import RandomizedSearchCV
from fd_lib.feature_engineering import CustomFeature<jupyter_output><empty_output><jupyter_text>I will try different algorithms to find out which one performs best, I am considering the following models:- **Lasso** *(linear model with regularization terms)*- **SVR** *(Support Vector Regressor)*- **ExtraTreesRegressor** *(similar to RandomForestRegressor)*- **XGBRegressor** *(Extreme Gradient Boosting Regressor)*  Train Test Split<jupyter_code>X = data.drop('G3', axis=1)
y = data['G3']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Feature Engineering<jupyter_code>BestPair(X, y).mold()<jupyter_output>[Product]   age & G2         | 0.882
[Product]   age & failures   | -0.391<jupyter_text>The process found that the pair of:- **age** and **G2** provide a high **positive** correlation.- **age** and **failure** provide a good **negative** correlation.<jupyter_code>features = Pipeline([
    ('new', CustomFeature([
        ('product', 'age', 'G2'),
        ('product', 'age', 'failures')
    ])),
    ('categorical', ColumnTransformer([
        ('encode', OneHotEncoder(), ['school', 'sex', 'address', 'famsize', 'Pstatus', 'Mjob',
                                     'Fjob', 'reason', 'guardian', 'schoolsup', 'famsup', 'paid',
                                     'activities', 'nursery', 'higher', 'internet', 'romantic'])
    ], remainder='passthrough'))
])<jupyter_output><empty_output><jupyter_text>Model Selection<jupyter_code>estimators = {
    'Lasso': Lasso(),
    'SVR': SVR(),
    'ET': ExtraTreesRegressor(),
    'XGB': XGBRegressor(),
}
for name, estimator in estimators.items():
    cv = KFold(n_splits=10, shuffle=True, random_state=0)
    scores = cross_val_score(estimator, features.fit_transform(X), y, cv=cv, n_jobs=-1)
    print('{}: {:.3f}'.format(name, scores.mean()))<jupyter_output>Lasso: 0.786
SVR: 0.774
ET: 0.824
XGB: 0.824<jupyter_text>The observation shows that **ExtraTrees** and **XGB** are the best performers.  Fitting Model I'll be using 3 times more **n_iter** on *XGB* than *ExtraTrees*. This is because **XGB** typically will work **faster**.  1. ExtraTreesRegressor<jupyter_code>params = {
    'n_estimators': [100, 250, 500],
    'max_features': ['auto', 'sqrt', 'log2'],
    'max_depth': [3, 4, 6, 8, 10, None],
    'min_samples_split': [2, 5, 10, 15, 20],
    'min_samples_leaf': [1, 2, 5, 10, 15]
}
est_ET = Pipeline([
    ('features', features),
    ('regressor', RandomizedSearchCV(ExtraTreesRegressor(), param_distributions=params,
                                     n_iter=150, cv=5, n_jobs=-1, random_state=0, verbose=1))
])
est_ET.fit(X_train, y_train);
best_ET = est_ET.named_steps['regressor'].best_params_
best_ET<jupyter_output><empty_output><jupyter_text>2. XGBRegressor<jupyter_code>params = {
    'eta': [0.05, 0.1, 0.15, 0.2, 0.25, 0.3],
    'max_depth': [3, 4, 6, 8, 10],
    'subsample': [0.5, 0.75, 0.9],
    'min_child_weight': [3, 5, 7, 10],
    'colsample_bytree': [0.3, 0.4, 0.5, 0.7],
    'gamma': [0, 0.1, 0.2, 0.3, 0.4],
    'objective': ['reg:squarederror']
}
est_XGB = Pipeline([
    ('features', features),
    ('regressor', RandomizedSearchCV(XGBRegressor(), param_distributions=params,
                                     n_iter=450, cv=5, n_jobs=-1, random_state=0, verbose=1))
])
est_XGB.fit(X_train, y_train);
best_XGB = est_XGB.named_steps['regressor'].best_params_
best_XGB<jupyter_output><empty_output><jupyter_text>Scoring<jupyter_code>score = pd.DataFrame({'Train': [est_ET.score(X_train, y_train),
                                est_XGB.score(X_train, y_train)],
                      'Test': [est_ET.score(X_test, y_test),
                               est_XGB.score(X_test, y_test)]},
                    
                     index=['ExtraTrees', 'XGB'])

score.columns.name = 'R2'
score.index.name = 'Model'
score
columns = [('Predict', 'ET'), ('Predict', 'XGB'), ('True', '')]

predict = pd.DataFrame({'ET': est_ET.predict(X_test).round(0).astype(int),
                        'XGB': est_XGB.predict(X_test).round(0).astype(int),
                        'True': y_test}).sample(5)

predict.columns = pd.MultiIndex.from_tuples(columns)
predict<jupyter_output><empty_output><jupyter_text>Both models have done a great job. Their predictions are very close to the true values.  Time  1. ExtraTreesRegressor<jupyter_code>%%timeit
Pipeline([
    ('features', features),
    ('regressor', ExtraTreesRegressor(**best_ET))
]).fit(X_train, y_train)
%%timeit
est_ET.predict(X_test)<jupyter_output>509 ms ± 134 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>2. XGBRegressor<jupyter_code>%%timeit
Pipeline([
    ('features', features),
    ('regressor', XGBRegressor(**best_XGB))
]).fit(X_train, y_train)
%%timeit
est_XGB.predict(X_test)<jupyter_output>87.9 ms ± 19.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)<jupyter_text>Overview **XGBRegressor** has done a better job in terms of time complexity. The training time is about **10 times faster** and the prediction time is about **5 times faster**. Note that each loop can perform differently.  Another Dataset<jupyter_code>X_2 = mat_data.drop('G3', axis=1)
y_2 = mat_data['G3']
pd.DataFrame({'Predict': est_XGB.predict(X_2).round(0).astype(int), 'True': y_2}).sample(5)
est_XGB.score(X_2, y_2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15394664466446645, 'fraction_numerical': 0.018770627062706272, 'mean_word_length': 4.6573317775184755, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 88, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '339538', 'license': 'MIT', 'n_tokens_mistral': 4992, 'n_tokens_neox': 4684, 'n_words': 1358, 'path': "Student's Final Grade Prediction/Student's Final Grade Prediction.ipynb", 'repo_name': 'fdavidsen/Personal-Data-Science-Projects', 'size': '14553'}