<jupyter_start><jupyter_text>ABALONE YEARS PREDICTION Abalone is the common name in the US for *Haliotis*, other terms are Paua (NZ) or Omer (UK). They all refer to a type molluscs with big shells The aim of this notebook is to predict the age of abalone from physical measurements, by working in different scenarios. The age (in years) is obtained by adding 1.5 to the number of rings (last column of the dataset.  Atribute selection with PCA Libraries import<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
from sklearn import tree
from sklearn.model_selection import GridSearchCV
import numpy as np
import pandas as pd
from sklearn import decomposition
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler<jupyter_output><empty_output><jupyter_text>Load and split the dataset<jupyter_code>Abalone = pd.read_csv("abalone.csv", sep=",", decimal=".")
X = Abalone.iloc[:,:-1]
y = Abalone.iloc[:,-1]
X.head()
y.head()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=33)
# Get list of categorical attributes

s = (X.dtypes == 'object')
object_cols = list(s[s].index)
print(object_cols)<jupyter_output>['Sex']<jupyter_text>Turn the categorical attribute into several numerical attributes, to be able to process the data with Scikit Learn models<jupyter_code>from sklearn.preprocessing import OneHotEncoder

Encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)
Encoder_train = pd.DataFrame(Encoder.fit_transform(X_train[object_cols]))
Encoder_test = pd.DataFrame(Encoder.fit_transform(X_test[object_cols]))

# Put back the index
Encoder_train.index = X_train.index
Encoder_test.index = X_test.index

# Remove original categorical attributes
num_X_train = X_train.drop(object_cols, axis=1)
num_X_test = X_test.drop(object_cols, axis=1)

# Add the one-hot encoded columns
Encoded_X_train = pd.concat([num_X_train, Encoder_train], axis=1)
Encoded_X_test = pd.concat([num_X_test, Encoder_test], axis=1)
Encoded_X_train.head()
Encoded_X_test.head()<jupyter_output><empty_output><jupyter_text>Because PCA is affected by standarization, it is necessary to scale the attributes before applying PCA<jupyter_code>scaler = StandardScaler()

# First fit the training set only

scaler.fit(Encoded_X_train)

# Then apply transform to train and test sets

Encoded_X_train = scaler.transform(Encoded_X_train)
Encoded_X_test = scaler.transform(Encoded_X_test)
print(Encoded_X_train.shape)<jupyter_output>(2923, 10)<jupyter_text>Let's use the maximum number of PCA components for the moment(which implies 10 PCA components)<jupyter_code>pca_init = decomposition.PCA(n_components=10)
pca_init.fit(Encoded_X_train)<jupyter_output><empty_output><jupyter_text>Let´s see how much variance explains each of the ten components. It´s clear to see that just the four first components alone explain more than 90%. This is because PCA procedure orders the components from the most important to the least important<jupyter_code>pca_init.explained_variance_ratio_<jupyter_output><empty_output><jupyter_text>This can be seen graphycally as well<jupyter_code>a = np.array([0])
concat = np.concatenate((a, pca_init.explained_variance_ratio_), axis=0)
accumulated_variance = np.cumsum(concat)
plt.plot(accumulated_variance)
plt.show()<jupyter_output><empty_output><jupyter_text>Therefore, let's compute just 4 PCA components and apply them to train and test.<jupyter_code>pca_init = decomposition.PCA(n_components=4)
pca_init.fit(Encoded_X_train)
X_train_red = pca_init.transform(Encoded_X_train)
X_test_red = pca_init.transform(Encoded_X_test)<jupyter_output><empty_output><jupyter_text>Tree Regressor Now, we can apply a regression tree model to the new, reduced, training set, and test it on the transformed test set<jupyter_code>clf = tree.DecisionTreeRegressor(random_state=0)
clf = clf.fit(X_train_red, y_train)
y_train_pred = clf.predict(X_train_red)
y_pred = clf.predict(X_test_red)
print ("MAE for the training set using 3 pca is {}".format(mean_absolute_error(y_train, y_train_pred)))
print ("MAE for the test set using 3 pca is {}".format(mean_absolute_error(y_test, y_pred)))<jupyter_output>MAE for the training set using 3 pca is 0.0
MAE for the test set using 3 pca is 2.3309409888357258<jupyter_text>It is possible to reduce the dimesion of the problem from 10 to 4, however, while obtaining a MAE of 2.3 approximately. Do not forget that we are predicting  the age of the abalone (rings on the shell plus 1.5), so the unit of MAE is years Let´s see what would happen if we use all the original attributes<jupyter_code>clf = tree.DecisionTreeRegressor(random_state=0)
clf = clf.fit(Encoded_X_train, y_train)
y_train_pred = clf.predict(Encoded_X_train)
y_pred = clf.predict(Encoded_X_test)
print ("Accuracy for the training set using all the attributes is {}".format(mean_absolute_error(y_train, y_train_pred)))
print ("Accuracy for the test set using all the attributes is {}".format(mean_absolute_error(y_test, y_pred)))<jupyter_output>Accuracy for the training set using all the attributes is 0.0
Accuracy for the test set using all the attributes is 2.148325358851675<jupyter_text>MAE is pretty similar using 6 more attributes, which means that attribute selection with PCA works reasonably well (with a tree regressor). In practical terms, the years of abalones are predicted just with 3 attributes, which means that in the case of large dataset (big data approach), the number of values will be reduced in a 60%, thus optimizing the process substantially. As, the attribute reduction seems to work pretty well, let´s try it now with a different model. A Random Forest.  Random Forest Regressor<jupyter_code>from sklearn.ensemble import RandomForestRegressor
clf = RandomForestRegressor(random_state=0)
clf = clf.fit(X_train_red, y_train)
y_train_pred = clf.predict(X_train_red)
y_pred = clf.predict(X_test_red)
print ("Accuracy for the training set using 3 pca is {}".format(mean_absolute_error(y_train, y_train_pred)))
print ("Accuracy for the test set using 3 pca is {}".format(mean_absolute_error(y_test, y_pred)))<jupyter_output>Accuracy for the training set using 3 pca is 0.698323640095792
Accuracy for the test set using 3 pca is 1.9051834130781502<jupyter_text>The increase in test accuracy using 4 components in a regressor tree versus random forest it is not very significative (2.33 vs 1.90 years). However the training accuracy has decreased (0 vs 0.69 years)<jupyter_code>clf = RandomForestRegressor(random_state=0)
clf = clf.fit(Encoded_X_train, y_train)
y_train_pred = clf.predict(Encoded_X_train)
y_pred = clf.predict(Encoded_X_test)
print ("Accuracy for the training set using 3 pca is {}".format(mean_absolute_error(y_train, y_train_pred)))
print ("Accuracy for the test set using 3 pca is {}".format(mean_absolute_error(y_test, y_pred)))<jupyter_output>Accuracy for the training set using 3 pca is 0.6207321245295928
Accuracy for the test set using 3 pca is 1.6453748006379587<jupyter_text>However, when using all the attributes, the changes in test accuracy is slightly more relevant if regressor tree is compared with a random forest (2.14 vs 1.64 years)  Tunning the number of PCA The number of PCA components can also be considered as a hyper-parameter, and it is possible to use a pipeline in order to select the right number of components. Let´s use that approach, so results can be analyzed later on<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
pca = decomposition.PCA()

# Define the paramter grid with the number of components
# to be used by PCA from 1 to 10
param_grid = {'attribute_selection__n_components': range(1,11)}

clf = Pipeline([
    ('attribute_selection', pca),
    ('tree', tree.DecisionTreeRegressor(random_state=0))
])

optPcaTree = GridSearchCV(clf, 
                              param_grid, 
                              cv=5,
                              n_jobs=1,
                              verbose=1,
                              scoring='neg_mean_absolute_error')

# Fit is done with the complete training set 

optPcaTree.fit(Encoded_X_train, y_train) 
print (optPcaTree.best_params_, -optPcaTree.best_score_)<jupyter_output>{'attribute_selection__n_components': 10} 2.041737940472118<jupyter_text>The optimal number of PCA components for the tree model is equal to 10 (the maximum possible value). So the hyper-parameter tunning of PCA, results on larger number of attributes than "selecting" manually the minimum number of attributes with maximum variance explained<jupyter_code>pca = decomposition.PCA()

# Define the paramter grid with the number of components
# to be used by PCA from 1 to 10
param_grid = {'attribute_selection__n_components': range(1,11)}

clf = Pipeline([
    ('attribute_selection', pca),
    ('forest', RandomForestRegressor(random_state=0))
])

optPcaForest = GridSearchCV(clf, 
                              param_grid, 
                              cv=5,
                              n_jobs=1,
                              verbose=1,
                              scoring='neg_mean_absolute_error')

# Fit is done with the complete training set 

optPcaForest.fit(Encoded_X_train, y_train) 
print (optPcaForest.best_params_, -optPcaForest.best_score_)<jupyter_output>{'attribute_selection__n_components': 9} 1.5432774546698598
{'fraction_non_alphanumeric': 0.07578499201703033, 'fraction_numerical': 0.025013304949441192, 'mean_word_length': 5.289156626506024, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '662507', 'license': 'MIT', 'n_tokens_mistral': 3139, 'n_tokens_neox': 2926, 'n_words': 963, 'path': 'Abalone.ipynb', 'repo_name': 'Victorlavado/Abalone_PCA_Pipeline', 'size': '9395'}