<jupyter_start><jupyter_text>Prediction on Animal Type  SummaryThe data set we will be using is Zoo (1990) provided by UC Irvine Machine Learning Repository. It stores data with 7 classes of animals and their related characteristics including animal name, hair, feathers and other attributes. In this project, we will use classification as our method to predict a most likely type of a given animal. IntroductionThe earth is an amazing planet that cultivates branches of animals. In general, scholars split them into 12 classes including mammals, birds, reptiles, amphibians, fishes, insects, crustaceans, arachnids, echinoderms, worms, mollusks and sponges(BioExploer.net., 2022). The traditional way in animal classification is manually identifying the characteristics and attributing it the mostly close class (Manohar, Sharath, & Kumar, 2016). However, it is tedious and time consuming, especially when the data set is very huge. A question hereby comes to us, if we can apply K-nearest neighbors (KNN) algorithms in predicting the type an animal belongs to given its related characteristics, such as hair, feathers, etc.? Therefore, in this project, we will show how we use KNN to do classification in animals based on data set Zoo(1990) which contains 1 categorical attribute, 17 Boolean-valued attributes and 1 numerical attribute. The categorical attribute appears to be the class attribute. Detailed breakdowns are as follows:1. animal name: Unique for each instance 2. hair: Boolean 3. feathers: Boolean 4. eggs: Boolean 5. milk: Boolean 6. airborne: Boolean 7. aquatic: Boolean 8. predator: Boolean 9. toothed: Boolean 10. backbone: Boolean 11. breathes: Boolean 12. venomous: Boolean 13. fins: Boolean 14. legs: Numeric (set of values: {0,2,4,5,6,8}) 15. tail: Boolean 16. domestic: Boolean 17. catsize: Boolean 18. type: Numeric (integer values in range [1,7]) Methods & ResultsWe are going to use multiple analysis to classify the type of the animals using 16 variables including hair, feathers, eggs, milk, airborne, aquatic, predator, toothed, backbone, breathes, venomous, fins, legs, tail, domestic, catsize as our predictors. To predict the class of a new observation, the algorithms of each type will be further explained before implementation.<jupyter_code>from sklearn.model_selection import train_test_split, GridSearchCV
import numpy as np
import matplotlib.pyplot as plt
import sys
sys.path.append( '..' )
from src.pre_processing import *
from src.train_and_predict_model import *
from src.line_plot import *
from src.para_optimize import *
from src.std_acc import *
from src.line_plot import *
# reading the data as a csv from the uci web server, with header = false as the data contains no header
# Adding column names to the data
colm = ["animalName", "hair", "feathers", "eggs", "milk", "airborne", "aquatic", 
        "predator", "toothed", "backbone", "breathes", "venomous", "fins", 
        "legs", "tail", "domestic", "catsize", "type"]

zoo_data = pre_process("https://archive.ics.uci.edu/ml/machine-learning-databases/zoo/zoo.data", colm)
#uncomment this line if the server is down for fetching the data
#zoo_data = pd.read_csv("./data/zoo.csv")
zoo_data.head()<jupyter_output><empty_output><jupyter_text>The first thing is to import the data. The data set is downloaded from [UCI repository]("https://archive-beta.ics.uci.edu/ml/datasets/zoo"). It is then saved as a csv file in this project repository. Some exploratory data analysis needs to be run before running the actual analyses on the data set.<jupyter_code># saving the data as a csv file in our data directory
# zoo_data.to_csv(r'./data/zoo.csv')
# Check if there are missing values
print("Whether the dataset contains missing value: " + str(zoo_data.isna().any().any()))
# drop the first column
#zoo_data = zoo_data.drop(zoo_data.columns[[0,1]], axis=1)
# Create a summary of the data set, including descriptive statistics
zoo_data.describe()<jupyter_output><empty_output><jupyter_text>After checking whether there are missing values in the data set, we can clearly deduce that the data set is clean according to the data summary we generated above. Since most features are binary and categorical, there is no need to do normalization and standardization.<jupyter_code># Create a visualization of the dataset
zoo_data.hist(figsize=(15,15));
plt.suptitle("fig.1 Histogram of all features");<jupyter_output><empty_output><jupyter_text>As shown in fig.1, the histograms of each feature are generated. The ones with skewed distribution might be more decisive in the prediction. However, since the data set is relatively small, all the features except the `animalName` are going to be used to predict. In the next part, we are going to split the data, into the training set and testing set. After that, different classification models will be trained and evaluated.  ClassificationNow we will use the training set to build an accurate model, whereas the testing set is used to report the accuracy of the models. Here is a list of algorithms we will use in the following section:K Nearest Neighbor(KNN)Decision TreeSupport Vector MachineLogistic Regression<jupyter_code># extracting the feature that will predict
feature = zoo_data[["hair", "feathers", "eggs", "milk", "airborne", 
                   "aquatic", "predator", "toothed", "backbone", "breathes", 
                   "venomous", "fins", "legs", "tail", "domestic", "catsize"]]
# making it as a X
X = feature
X.head()
# taking the y values, the type
y = zoo_data['type']
y[0:5]
# splitting the dataset 80-20 for train and test
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)
print('Training set:', X_train.shape,  y_train.shape)
print('Test set:', X_test.shape,  y_test.shape)<jupyter_output>Training set: (80, 16) (80,)
Test set: (21, 16) (21,)<jupyter_text>KNNKNN captures the idea of similarity (sometimes called distance, proximity, or closeness) with some basic mathematics we might have learned earlier. Basically in terms of geometry we can always calculate the distance between points on a graph. Similarly, using KNN we can group similar points together and predict the target with our feature variables(x).<jupyter_code>#training the model for different set of K values and finding the best K value
Ks = 81
mean_acc = np.zeros((Ks-1))
std_acc = np.zeros((Ks-1))
ConfustionMx = [];
for n in range(1,Ks):
    
    #Train Model and Predict  
    neigh = KNeighborsClassifier(n_neighbors = n).fit(X_train,y_train)
    yhat=neigh.predict(X_test)
    mean_acc[n-1] = metrics.accuracy_score(y_test, yhat)

std_acc = stdAcc(yhat,y_test,Ks)
mean_acc
# plotting the accuracy for different K values
# plt.plot(range(1,Ks),mean_acc,'g')
# plt.fill_between(range(1,Ks), mean_acc - 1 * std_acc, mean_acc + 1 * std_acc, alpha=0.10)
# plt.legend(('Accuracy ', '+/- 3xstd'))
# plt.ylabel('Accuracy ')
# plt.xlabel('Number of Neighbors (K)')
# plt.title('fig.2 Number of Neighbors vs. Accuracy')
# plt.tight_layout()
line_plot(Ks, mean_acc, std_acc, "Number of Neighbors (K)", "Accuracy", "fig.2 Number of Neighbors vs. Accuracy")
plt.show()
print("The best accuracy was with the values", mean_acc.max(), "with k=", mean_acc.argmax()+1 )
# Finding the K value using Grid Search
knn = KNeighborsClassifier()
k_vals = list(range(1, 21))
param_grid = dict(n_neighbors=k_vals)

para_optimize(knn, param_grid, 3, X_train, y_train)<jupyter_output><empty_output><jupyter_text>KNN final model & Evaluation<jupyter_code># as the best accuracy was with K = 1
# using K = 1 for the final KNN model
# Final KNN model is here used the splited test part to train again for better training, and better prediction
# KNN evaluation is also here scroll through the output
final_knn_model = finalModel("KNN", 1, X_train, X_test, y_train, y_test, X, y)<jupyter_output>KNN INFO: So the accuracy score for K = 1 is 1.0

KNN Cross Validate: 

fit_time       0.000539
score_time     0.000792
test_score     0.936847
train_score    1.000000
dtype: float64

KNN Classification report: 

              precision    recall  f1-score   support

           1       1.00      1.00      1.00         7
           2       1.00      1.00      1.00         5
           4       1.00      1.00      1.00         1
           5       1.00      1.00      1.00         1
           6       1.00      1.00      1.00         3
           7       1.00      1.00      1.00         4

    accuracy                           1.00        21
   macro avg       1.00      1.00      1.00        21
weighted avg       1.00      1.00      1.00        21<jupyter_text>Decision TreeA decision tree is a decision support tool that uses a tree-like model of decisions and their possible consequences, including chance event outcomes, resource costs, and utilityThe goal of using a Decision Tree is to create a training model that can use to predict the class or value of the target variable by learning simple decision rules inferred from prior data(training data).<jupyter_code>Ks = 50
mean_acc = np.zeros((Ks-1))
std_acc = np.zeros((Ks-1))
ConfustionMx = [];
for n in range(1,Ks):
    # Train Model and Predict  
    decTree = DecisionTreeClassifier(criterion="entropy", max_depth = n)
    decTree.fit(X_train,y_train)
    yhat=decTree.predict(X_test)
    mean_acc[n-1] = metrics.accuracy_score(y_test, yhat)

std_acc = stdAcc(yhat,y_test,Ks)
mean_acc
# plt.plot(range(1,Ks),mean_acc,'g')
# plt.fill_between(range(1,Ks),mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10)
# plt.legend(('Accuracy ', '+/- 3xstd'))
# plt.ylabel('Accuracy ')
# plt.xlabel('Max_Depth')
# plt.title('fig.3 Max_depth vs. Accuracy')
# plt.tight_layout()
line_plot(Ks, mean_acc, std_acc, "Maximum Depth", "Accuracy", "fig.2 Relationship between Number of Neighbors and Accuracy")
plt.show()
print("The best accuracy was with the values", mean_acc.max(), "with max_depth =", mean_acc.argmax()+1)<jupyter_output>The best accuracy was with the values 0.9047619047619048 with max_depth = 5<jupyter_text>Decision Tree final model & evaluation<jupyter_code># As Best is max depth = 5
# using max depth = 5 for the final decision tree
# Final decision tree is here used the split test part to train again for better training, and better prediction
# DT evaluation is also here scroll through the output
Final_dec_Tree = finalModel("DT", 5, X_train, X_test, y_train, y_test, X, y)<jupyter_output>DT INFO: So the accuracy score for max depth = 5 is 0.9523809523809523

DT Cross Validate: 

fit_time       0.000673
score_time     0.000414
test_score     0.937500
train_score    0.995833
dtype: float64

DT Classification report: 

              precision    recall  f1-score   support

           1       1.00      1.00      1.00         7
           2       1.00      1.00      1.00         5
           4       1.00      1.00      1.00         1
           5       1.00      1.00      1.00         1
           6       0.75      1.00      0.86         3
           7       1.00      0.75      0.86         4

    accuracy                           0.95        21
   macro avg       0.96      0.96      0.95        21
weighted avg       0.96      0.95      0.95        21<jupyter_text>Support Vector MachineSVM or Support Vector Machine is a linear model for classification and regression problems. It can solve linear and non-linear problems and work well for many practical problems. The idea of SVM is simple: The algorithm creates a line or a hyperplane which separates the data into classes.  SVM training model Jaccard Score, final model and evaluation<jupyter_code>#Final SVM is here used the splited test part to train again for better training, and better prediction
#svm evaluation train and final model is also here scroll through the output
svec = finalModel("SVM", -1, X_train, X_test, y_train, y_test, X, y)<jupyter_output>SVM INFO: So the Training Jaccard score for SVM is: 0.8260869565217391

SVM Evaluation:

              precision    recall  f1-score   support

           1       1.00      1.00      1.00         7
           2       1.00      1.00      1.00         5
           4       0.33      1.00      0.50         1
           5       1.00      1.00      1.00         1
           6       1.00      1.00      1.00         3
           7       1.00      0.50      0.67         4

    accuracy                           0.90        21
   macro avg       0.89      0.92      0.86        21
weighted avg       0.97      0.90      0.91        21<jupyter_text>Logistic RegressionLogistic Regression is a "Supervised machine learning" algorithm that can be used to model the probability of a certain class or event. It is used when the data is linearly separable and the outcome is binary or dichotomous in nature. That means Logistic regression is usually used for Binary classification problems.  Logistic Regression training model Jaccard Score, final model and evaluation<jupyter_code># final LR model is here used the splited test part to train again for better training, and better prediction
# LR evaluation train and final model is also here scroll through the output
LR = finalModel("LR", -1, X_train, X_test, y_train, y_test, X, y)<jupyter_output>LR INFO: So the Training Jaccard score for Logistic Regression is: 0.68

LR Evaluation: 

              precision    recall  f1-score   support

           1       0.88      1.00      0.93         7
           2       0.83      1.00      0.91         5
           4       0.50      1.00      0.67         1
           5       0.00      0.00      0.00         1
           6       0.75      1.00      0.86         3
           7       1.00      0.25      0.40         4

    accuracy                           0.81        21
   macro avg       0.66      0.71      0.63        21
weighted avg       0.81      0.81      0.76        21
{'fraction_non_alphanumeric': 0.07918928700687658, 'fraction_numerical': 0.04567499095186391, 'mean_word_length': 2.967834577828834, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 33, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '258066', 'license': 'MIT', 'n_tokens_mistral': 4503, 'n_tokens_neox': 4074, 'n_words': 1653, 'path': 'analysis/_build/jupyter_execute/old_report/zoo_analysis.ipynb', 'repo_name': 'DSCI-310/DSCI-310-Group-7', 'size': '13815'}