<jupyter_start><jupyter_text>Handwritten digit recognition using SVM<jupyter_code># import all necessary libraries
import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

from sklearn.preprocessing import scale
from sklearn.model_selection import train_test_split

from sklearn import metrics
from sklearn.metrics import confusion_matrix
from sklearn.svm import SVC

from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV<jupyter_output><empty_output><jupyter_text>Data Import<jupyter_code>#import file and reading few lines
numbers = pd.read_csv(r'C:/Users/<NAME>/Downloads/train.csv',encoding = "latin")
numbers.head(10)
numbers.shape<jupyter_output><empty_output><jupyter_text>Data understanding and exploration<jupyter_code>#checking datatype
numbers.info()<jupyter_output><empty_output><jupyter_text>All the columns are of dtype - int64.<jupyter_code>numbers.describe(percentiles = [0.05,0.10,0.25,0.50,0.75,0.90,0.99])
#checking for null values
round(100*(numbers.isnull().sum()/(len(numbers.index))),2).sort_values(ascending = False)<jupyter_output><empty_output><jupyter_text>There are no null values.  Column- `Label`<jupyter_code># let us check unique entries of label column
np.unique(numbers['label'])
numbers['label'].value_counts()
#visualising the column - label
sns.countplot(numbers['label'],palette = 'icefire')<jupyter_output><empty_output><jupyter_text>Let us examine few `pixels`<jupyter_code>y = pd.value_counts(numbers.values.ravel()).sort_index()
N = len(y)
x = range(N)
width =0.9
plt.figure(figsize=[8,8])
plt.bar(x, y, width, color="blue")
plt.title('Pixel Value Frequency (Log Scale)')
plt.yscale('log')
plt.xlabel('Pixel Value (0-255)')
plt.ylabel('Frequency')
plt.figure(figsize=(5,5))
sns.distplot(numbers['pixel656'])
plt.show()
plt.figure(figsize=(5,5))
sns.distplot(numbers['pixel684'])<jupyter_output><empty_output><jupyter_text>`label` vs `pixel`<jupyter_code>sns.barplot(x='label', y='pixel683', data=numbers)
sns.barplot(x='label', y='pixel572', data=numbers)<jupyter_output><empty_output><jupyter_text>Let us visualize few numbers:<jupyter_code>one = numbers.iloc[2, 1:]
one = one.values.reshape(28,28)
plt.imshow(one)
plt.title("Digit 1")
zero = numbers.iloc[1, 1:]
zero = zero.values.reshape(28,28)
plt.imshow(zero)
plt.title("Digit 0")
nine = numbers.iloc[11, 1:]
nine = nine.values.reshape(28,28)
plt.imshow(nine)
plt.title("Digit 9")<jupyter_output><empty_output><jupyter_text>Let us check heatmap<jupyter_code>plt.figure(figsize=(15,15))
sns.heatmap(data=numbers.corr(),annot=False)<jupyter_output><empty_output><jupyter_text>Almost all nearby pixel values are correlated, which is expected as well.  Data Preparation<jupyter_code># average feature values
pd.set_option('display.max_rows', 999)
round(numbers.drop('label', axis=1).mean(), 2).sort_values(ascending = False)<jupyter_output><empty_output><jupyter_text>We see that average varies between 140 to 0. It is better to scale them.<jupyter_code># splitting into X and y
X = numbers.drop("label", axis = 1)
y = numbers['label']
# scaling the features
X_scaled = scale(X)

# train test split
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, train_size=0.2,test_size = 0.8, random_state = 101)
print('X_train shape:',X_train.shape)
print('y_train shape:',y_train.shape)
print('X_test shape:',X_test.shape)
print('y_test shape:',y_test.shape)<jupyter_output><empty_output><jupyter_text>Model Building 1. Let us first try - `Linear` model:<jupyter_code># linear model
model_linear = SVC(kernel='linear')
model_linear.fit(X_train, y_train)

# predict
y_pred = model_linear.predict(X_test)
# confusion matrix and accuracy, precision, recall

# accuracy
print("accuracy:", metrics.accuracy_score(y_true=y_test, y_pred=y_pred), "\n")

# cm
print(metrics.confusion_matrix(y_true=y_test, y_pred=y_pred))
#precision, recall and f1-score
scores=metrics.classification_report(y_test, y_pred, labels=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
print(scores)<jupyter_output><empty_output><jupyter_text>The linear model gives approx. 91.31% accuracy. Let's look at a non-linear model with randomly chosen hyperparameters. Let us try `Non- linear` models: 2. `Poly` kernel<jupyter_code># non-linear model
# using poly kernel, C=1, default value of gamma

# model
non_linear_model_poly = SVC(kernel='poly')

# fit
non_linear_model_poly.fit(X_train, y_train)

# predict
y_pred = non_linear_model_poly.predict(X_test)
# confusion matrix and accuracy, precision, recall

# accuracy
print("accuracy:", metrics.accuracy_score(y_true=y_test, y_pred=y_pred), "\n")

# cm
print(metrics.confusion_matrix(y_true=y_test, y_pred=y_pred))<jupyter_output><empty_output><jupyter_text>The accuracy dropped to 87.12%, so obviously no point in going with polynomial. Let us try 'rbf'. 3. `rbf` kernel<jupyter_code># non-linear model
# using rbf kernel, C=1, default value of gamma

# model
non_linear_model = SVC(kernel='rbf')

# fit
non_linear_model.fit(X_train, y_train)

# predict
y_pred = non_linear_model.predict(X_test)
# confusion matrix and accuracy, precision, recall

# accuracy
print("accuracy:", metrics.accuracy_score(y_true=y_test, y_pred=y_pred), "\n")

# cm
print(metrics.confusion_matrix(y_true=y_test, y_pred=y_pred))
#precision, recall and f1-score
scores=metrics.classification_report(y_test, y_pred, labels=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
print(scores)<jupyter_output><empty_output><jupyter_text>As we clearly see that the non-linear `rbf` model gives approx. 94% accuracy. And most of the precision is above 90%.Thus, going forward, let's choose hyperparameters corresponding to non-linear `rbf` models.  Grid Search: Hyperparameter Tuning<jupyter_code># creating a KFold object with 5 splits 
folds = KFold(n_splits = 5, shuffle = True, random_state = 101)

# specify range of hyperparameters
# Set the parameters by cross-validation
hyper_params = [ {'gamma': [0.01, 0.001,0.0001],
                     'C': [1, 10, 100]}]


# specify model
model = SVC(kernel="rbf")

# set up GridSearchCV()
model_cv = GridSearchCV(estimator = model, 
                        param_grid = hyper_params, 
                        scoring= 'accuracy', 
                        cv = folds, 
                        verbose = 1,
                        return_train_score=True,n_jobs = -1)      

# fit the model
model_cv.fit(X_train, y_train)  
# cv results
cv_results = pd.DataFrame(model_cv.cv_results_)
cv_results
# converting C to numeric type for plotting on x-axis
cv_results['param_C'] = cv_results['param_C'].astype('int')

# # plotting
plt.figure(figsize=(20,7))

# subplot 1/3
plt.subplot(131)
gamma_01 = cv_results[cv_results['param_gamma']==0.01]

plt.plot(gamma_01["param_C"], gamma_01["mean_test_score"])
plt.plot(gamma_01["param_C"], gamma_01["mean_train_score"])
plt.xlabel('C')
plt.ylabel('Accuracy')
plt.title("Gamma=0.01")
plt.ylim([0.50, 1.1])
plt.legend(['test accuracy', 'train accuracy'], loc='upper left')
plt.xscale('log')

# subplot 2/3
plt.subplot(132)
gamma_001 = cv_results[cv_results['param_gamma']==0.001]

plt.plot(gamma_001["param_C"], gamma_001["mean_test_score"])
plt.plot(gamma_001["param_C"], gamma_001["mean_train_score"])
plt.xlabel('C')
plt.ylabel('Accuracy')
plt.title("Gamma=0.001")
plt.ylim([0.50, 1.1])
plt.legend(['test accuracy', 'train accuracy'], loc='upper left')
plt.xscale('log')


# subplot 3/3
plt.subplot(133)
gamma_0001 = cv_results[cv_results['param_gamma']==0.0001]

plt.plot(gamma_0001["param_C"], gamma_0001["mean_test_score"])
plt.plot(gamma_0001["param_C"], gamma_0001["mean_train_score"])
plt.xlabel('C')
plt.ylabel('Accuracy')
plt.title("Gamma=0.0001")
plt.ylim([0.50, 1.1])
plt.legend(['test accuracy', 'train accuracy'], loc='upper left')
plt.xscale('log')
# printing the optimal accuracy score and hyperparameters
best_score = model_cv.best_score_
best_hyperparams = model_cv.best_params_

print("The best test score is {0} corresponding to hyperparameters {1}".format(best_score, best_hyperparams))<jupyter_output><empty_output><jupyter_text>Let us take the best value ({'C': 10, 'gamma': 0.001}) as suggested by the sklearn.  Building and Evaluating the Final Model<jupyter_code># model with optimal hyperparameters

# model
model = SVC(C=10, gamma=0.001, kernel="rbf")

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# metrics
print("accuracy", metrics.accuracy_score(y_test, y_pred), "\n")
print(metrics.confusion_matrix(y_test, y_pred), "\n")
# different class-wise accuracy - #precision, recall and f1-score
scores=metrics.classification_report(y_test, y_pred, labels=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
print(scores)<jupyter_output><empty_output><jupyter_text>We see that with hyperparameter - C = 10 and gamma = 0.001, we see overall accuracy of the model is 95% and also precision for each label is above 94%.<jupyter_code># Let us visualize our final model on unseen training dataset

df = np.random.randint(1,y_pred.shape[0]+1,5)

plt.figure(figsize=(16,4))
for i,j in enumerate(df):
    plt.subplot(150+i+1)
    d = X_test[j].reshape(28,28)
    plt.title(f'Predicted Label: {y_pred[j]}')
    plt.imshow(d)
plt.show()<jupyter_output><empty_output><jupyter_text>Let us use our final model on test data (test.csv)<jupyter_code>#import file and reading few lines
test_df = pd.read_csv(r'C:/Users/<NAME>/Downloads/test.csv',encoding = "latin")
test_df.head(10)
test_df.shape
test_df.info()
# scaling the features
test_scaled = scale(test_df)
#model.predict
test_predict = model.predict(test_scaled)
# Plotting the distribution of prediction
a = {'ImageId': np.arange(1,test_predict.shape[0]+1), 'Label': test_predict}
data_to_export = pd.DataFrame(a)
sns.countplot(data_to_export['Label'], palette = 'icefire')
# Let us visualize few of predicted test numbers

df = np.random.randint(1,test_predict.shape[0]+1,5)

plt.figure(figsize=(16,4))
for i,j in enumerate(df):
    plt.subplot(150+i+1)
    d = test_scaled[j].reshape(28,28)
    plt.title(f'Predicted Label: {test_predict[j]}')
    plt.imshow(d)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14021995286724273, 'fraction_numerical': 0.03260015710919089, 'mean_word_length': 6.401889534883721, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 75, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '354442', 'license': 'Unlicense', 'n_tokens_mistral': 3990, 'n_tokens_neox': 3640, 'n_words': 801, 'path': '18BCS6182 Practical 5&6 .ipynb', 'repo_name': 'vardhanreddy7/Digit-recogninzer-', 'size': '10188'}