<jupyter_start><jupyter_text>The objective with this notebook is to convert the preprocessed structured input data into a form usable by our model. This includes one-hot encoding categorical data, normalizing value ranges, partitioning training vs testing datasets, class balancing, etc.<jupyter_code>import pandas as pd
import numpy as np
from keras.utils import to_categorical
from keras.utils import normalize
import matplotlib.pyplot as plt


categorical_feature_keys = [
    'PREV_GPA', 'HIGH_SCHOOL_GPA'
]

numeric_feature_keys = [
    'FM_UNMET_NEED', 'CURRENT_BALANCE', 'WI_GRADE_COUNT', 'DFW_GRADE_COUNT'
]
# Your path may vary, change this appropriately
datasets = '../../../data/'
path_to_stu_risk = f'{datasets}/interim/01-preprocess-structured-data-fine.csv'

# One of the things we love about pandas is that it's easy to load CSV data
# into a "data frame"
risk_raw_df = pd.read_csv(path_to_stu_risk)
risk_raw_df.describe()
# Separate the data into input and output sets

x = risk_raw_df.copy()

# Elliminate the output data from the input dataset
x = x.drop('LAST_GPA', axis=1)

y = pd.DataFrame(risk_raw_df['LAST_GPA'])
x.head(5)
# Normalize the numeric fields 

for k in numeric_feature_keys:
    x[k] = normalize(x[k].values)[0]

# Let's name the columns of data to help keep track
grade_decode = {
    0: "F",  1: "D", 2: "D+", 3: "C-", 4: "C",
    5: "C+", 6: "B-", 7: "B", 8: "B+", 9: "A-", 10: "A", 11: 'GAP'
}
num_classes = len(grade_decode)

# Replacing the categorical value with the categorical features for our input dataset
for field in categorical_feature_keys:
    cat = to_categorical(x[field].values, num_classes=num_classes)
    for k, v in grade_decode.items():
        column = field + "_" + v
        x.insert(value=cat[:,k], loc=len(x.columns), column=column)
    x = x.drop(field, axis=1)

# Replacing the categorical value with the categorical features for our output dataset
cat = to_categorical(y['LAST_GPA'].values, num_classes=num_classes)
for k, v in grade_decode.items():
    column = 'LAST_GPA_' + v
    y.insert(value=cat[:,k], loc=len(y.columns), column=column)
y = y.drop('LAST_GPA', axis=1)

y.head(5)<jupyter_output><empty_output><jupyter_text>The data formatting is now readable by our model. Now it's time to seperate the data into training and testing collections. We will start with 80% for training, and 20% for testing. There should be an even proportion of classes divided between the two collections. Let's get a baseline of the distribution in the whole dataset.<jupyter_code>risk_raw_df.hist(column='LAST_GPA',bins=22)
plt.show()
# Splitting into train and test sets with an 80/20 proportion

total_rows = x.shape[0]
msk = np.random.rand(total_rows) < 0.8

x_train = x[msk]
y_train = y[msk]

x_test = x[~msk]
y_test = y[~msk]
x_train.head(5)
y_train.head(5)
x_train.to_csv(f'{datasets}/interim/x_train-fine.csv', index=False)
y_train.to_csv(f'{datasets}/interim/y_train-fine.csv', index=False)
x_test.to_csv(f'{datasets}/interim/x_test-fine.csv', index=False)
y_test.to_csv(f'{datasets}/interim/y_test-fine.csv', index=False)
(x_train.shape, y_train.shape, x_test.shape, y_test.shape)
x_test.head(5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12288401253918495, 'fraction_numerical': 0.011912225705329153, 'mean_word_length': 5.066539923954372, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '70972', 'license': 'MIT', 'n_tokens_mistral': 1203, 'n_tokens_neox': 1134, 'n_words': 326, 'path': 'notebooks/01-student-risk-fine-detail/01-preprocessing-data/02-convert-for-model.ipynb', 'repo_name': 'sekondusg/bandu', 'size': '3190'}