<jupyter_start><jupyter_text>Importing & Loading Required Packages<jupyter_code>import json # will be needed for saving preprocessing details
import numpy as np # for data manipulation
import pandas as pd # for data manipulation
from sklearn.model_selection import train_test_split # will be used for data split
from sklearn.preprocessing import LabelEncoder # for preprocessing
from sklearn.preprocessing import OneHotEncoder
from sklearn.ensemble import RandomForestClassifier # for training the algorithm
from sklearn.ensemble import ExtraTreesClassifier # for training the algorithm
import joblib # for saving algorithm and preprocessing objects
import pickle
from pandas.api.types import CategoricalDtype
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import FeatureUnion
from sklearn.model_selection import cross_val_score
import seaborn as sns
from matplotlib import pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Loading dataIn this tutorial, I will use Adult Income data set. In this data set, the ML will be used to predict whether income exceeds $50K/year based on census data. Code to load data and show first rows of data:<jupyter_code># load dataset
df = pd.read_csv('https://raw.githubusercontent.com/pplonski/datasets-for-start/master/adult/data.csv', skipinitialspace=True)
#x_cols = [c for c in df.columns if c != 'income']
# set input matrix and target column
X = df.loc[:, df.columns != 'income']
y = df['income']
# show first rows of data
df.head()<jupyter_output><empty_output><jupyter_text>The X matrix has 32,561 rows and 14 columns. This is input data for our algorithm, each row describes one person. The y vector has 32,561 values indicating whether income exceeds 50K per year.Before starting data preprocessing we will split our data into training, and testing subsets. We will use 30% of the data for testing.<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Before starting data preprocessing we will split our data into training, and testing subsets. We will use 30% of the data for testing.<jupyter_code># data split train / test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=1234)<jupyter_output><empty_output><jupyter_text>Data pre-processingIn our data set, there are missing values and categorical columns. For ML algorithm training we will use the Random Forest algorithm from the sklearn package. In the current implementation it can not handle missing values and categorical columns, that's why we need to apply pre-processing algorithms.To fill missing values we will use the most frequent value in each column (there are many other filling methods, the one I select is just for example purposes).<jupyter_code>#Checking which features have null values.
X_train.isnull().sum()
# fill missing values
train_mode = dict(X_train.mode().iloc[0]) # Gets the most frequent value in each column
X_train = X_train.fillna(train_mode)
X_test = X_test.fillna(train_mode)
print(train_mode)<jupyter_output>{'age': 36.0, 'workclass': 'Private', 'fnlwgt': 123011, 'education': 'HS-grad', 'education-num': 9.0, 'marital-status': 'Married-civ-spouse', 'occupation': 'Prof-specialty', 'relationship': 'Husband', 'race': 'White', 'sex': 'Male', 'capital-gain': 0.0, 'capital-loss': 0.0, 'hours-per-week': 40.0, 'native-country': 'United-States'}<jupyter_text>From train_mode you see, that for example in the age column the most frequent value is 31.0.Let's convert categoricals into numbers. I will use LabelEncoder from sklearn package:<jupyter_code># convert categoricals
encoders = {}
for column in ['workclass', 'education', 'marital-status',
                'occupation', 'relationship', 'race',
                'sex','native-country']:
    categorical_convert = LabelEncoder()
    X_train[column] = categorical_convert.fit_transform(X_train[column])
    X_test[column] = categorical_convert.fit_transform(X_test[column])
    #X_test[column] = categorical_convert.fit_transform(X_test[column])
    encoders[column] = categorical_convert<jupyter_output><empty_output><jupyter_text>Algorithms TrainingData is ready, so we can train our Random Forest algorithm.<jupyter_code># train the Random Forest algorithm
rf = RandomForestClassifier(n_estimators = 100)
rf = rf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>We will also train Extra Trees & Logistic regression algorithm:<jupyter_code># train the Extra Trees algorithm
et = ExtraTreesClassifier(n_estimators = 100)
et = et.fit(X_train, y_train)

# train the logistic regression
LR = LogisticRegression(random_state=0)
LR = LR.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Prediction<jupyter_code>predicted_classes_RF = rf.predict(X_test)
print('Random Forest Model Accuracy = ' , accuracy_score(predicted_classes_RF, y_test)*100 , '%')
predicted_classes_et = et.predict(X_test)
print('Extra Trees Model Accuracy = ' , accuracy_score(predicted_classes_et, y_test)*100 , '%')
predicted_classes_LR = LR.predict(X_test)
print('Logistic Regression Model Accuracy = ' , accuracy_score(predicted_classes_LR, y_test)*100,'%')<jupyter_output>Logistic Regression Model Accuracy =  79.53324120988792 %<jupyter_text>Let’s plot the confusion matrixChoosing RF as a base model as it has the best accuracy<jupyter_code>cfm = confusion_matrix(predicted_classes_RF, y_test.values)
sns.heatmap(cfm, annot=True)
plt.xlabel('Predicted classes')
plt.ylabel('Actual classes')<jupyter_output><empty_output><jupyter_text>Saving Pre-Processing Steps & Trained ModelsAs you see, training the algorithm is easy, just 2 lines of code - much less than data reading and pre-processing. Now, let's save the algorithm that we have created. The important thing to notice is that the ML algorithm is not only the rf and et variable (with model weights), but we also need to save pre-processing variables train_mode and encoders as well. For saving, I will use joblib package.<jupyter_code># save preprocessing objects and RF algorithm
joblib.dump(train_mode, "./Pre_Processing_Step_A_Filling_Missing_train_mode.joblib", compress=True)
joblib.dump(encoders, "./Pre_Processing_Step_B_Label_encoders.joblib", compress=True)
joblib.dump(rf, "./ML_Model_Random_Forest.joblib", compress=True)
joblib.dump(et, "./ML_Model_extra_trees.joblib", compress=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08611442193087009, 'fraction_numerical': 0.012067938021454111, 'mean_word_length': 6.5005586592178775, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '584635', 'license': 'MIT', 'n_tokens_mistral': 2102, 'n_tokens_neox': 1983, 'n_words': 658, 'path': 'research/Train_Adult_Income.ipynb', 'repo_name': 'MohammedSamirMahmoud/Django_ML_WebService', 'size': '6712'}