<jupyter_start><jupyter_text>Brief Overview: Entering high school students make program choices among vocational program and academic program. Their choice might be modeled using their writing score and their social economic status.**Academic Programs** are more likely to demand essays and logical reasoning, whereas **vocational Programs** are about practical hands-on training and learning tangible skills.The basic difference between these two is theory versus practice. An academic school generally teaches theoretical knowledge and broad-based research skills, while a vocational school teaches hands-on, applied skills such as carpentry, auto mechanics and hairdressing. **University of Tesas wants to launch some new programs and want to know there target audience(Which student should be be pitched for which course/program). They hired you to do the analysis and predict which program a student will choose out of Academic Programs and Vocational Programs on the bases of the information given such as socio-economic status, the type of school attended (public or private), gender and their prior reading, writing, maths and science scores.**  Upload Required Libraries<jupyter_code># Pandas and Numpy libraries
import pandas as pd
import numpy as np
# For preprocessing the data
from sklearn import preprocessing
# To split the dataset into train and test datasets
from sklearn.model_selection import train_test_split
# To model the Gaussian Navie Bayes classifier
from sklearn.naive_bayes import GaussianNB
# To calculate the accuracy score of the model
from sklearn.metrics import accuracy_score
import seaborn as sns
import matplotlib.pyplot as plt
from scipy.stats import zscore
import warnings 
warnings.filterwarnings( "ignore")<jupyter_output><empty_output><jupyter_text>Importing data<jupyter_code>df=pd.read_csv("Program_Choice.csv",index_col=0)
df.head()<jupyter_output><empty_output><jupyter_text>EDA **Shape**<jupyter_code># Shape of the dataframe
print("no. of rows: ",df.shape[0], "\n""no. of columns: ",df.shape[1])<jupyter_output>no. of rows:  200 
no. of columns:  13<jupyter_text>**Data type of data features**<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 200 entries, 0 to 199
Data columns (total 13 columns):
 #   Column   Non-Null Count  Dtype  
---  ------   --------------  -----  
 0   id       200 non-null    float64
 1   female   200 non-null    object 
 2   ses      200 non-null    object 
 3   schtyp   200 non-null    object 
 4   prog     200 non-null    object 
 5   read     200 non-null    float64
 6   write    200 non-null    float64
 7   math     200 non-null    float64
 8   science  200 non-null    float64
 9   socst    200 non-null    float64
 10  honors   200 non-null    object 
 11  awards   200 non-null    float64
 12  cid      200 non-null    int64  
dtypes: float64(7), int64(1), object(5)
memory usage: 21.9+ KB<jupyter_text>**Null value Check**<jupyter_code># Handling missing data
# Test whether there is any null value in our dataset or not. We can do this using isnull() method.
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Droping "id" as it is insignificant<jupyter_code>df.drop("id",axis=1,inplace =True)<jupyter_output><empty_output><jupyter_text>**Making different list for categorical columns and numerical columns**<jupyter_code>cat=[]
num=[]
for i in df.columns:
    if df[i].dtype=="object":
        cat.append(i)
    else:
        num.append(i)
print(cat) 
print(num)<jupyter_output>['female', 'ses', 'schtyp', 'prog', 'honors']
['read', 'write', 'math', 'science', 'socst', 'awards', 'cid']<jupyter_text>The above output shows that there is no “null” value in our dataset.Let’s try to test whether any categorical attribute contains a “?” in it or not. At times there exists “?” or ” ” in place of missing values. Using the below code snippet we are going to test whether adult_df data frame consists of categorical variables with values as “?”.<jupyter_code>for variable in cat:
    print(variable,":", sum(df[variable] == '?'))<jupyter_output>female : 0
ses : 0
schtyp : 0
prog : 0
honors : 0<jupyter_text>The output of the above code snippet shows that there are "?" or " " present in the data set  Data preprocessing we need some summary statistics of our dataframe. For this, we can use describe() method. It can be used to generate various summary statistics.  Describe for numerical and categorical columns<jupyter_code>df[num].describe().T
df[cat].describe().T<jupyter_output><empty_output><jupyter_text>**Are there any duplicate records, Check for duplicate data**<jupyter_code>dups = df.duplicated()
print('Number of duplicate rows = %d' % (dups.sum()))

df[dups]
## df.drop_duplicates(inplace=True) If present we can drop them like this<jupyter_output><empty_output><jupyter_text>unique values for categorical variables<jupyter_code>for column in df.columns:
    if df[column].dtype == 'object':
        print(column.upper(),': ',df[column].nunique())
        print(df[column].value_counts().sort_values())
        print('\n')<jupyter_output>FEMALE :  2
male       91
female    109
Name: female, dtype: int64


SES :  3
low       47
high      58
middle    95
Name: ses, dtype: int64


SCHTYP :  2
private     32
public     168
Name: schtyp, dtype: int64


PROG :  2
vocation     95
academic    105
Name: prog, dtype: int64


HONORS :  2
enrolled         53
not enrolled    147
Name: honors, dtype: int64<jupyter_text>**Univariate Analysis**<jupyter_code>fig, axes = plt.subplots(nrows=4,ncols=2)
fig.set_size_inches(10,20)
a = sns.distplot(df['read'] , ax=axes[0][0])
a.set_title("read Distribution",fontsize=10)

a = sns.boxplot(df['read'] , orient = "v" , ax=axes[0][1])
a.set_title("read Boxplot",fontsize=10)

a = sns.distplot(df['write'] , ax=axes[1][0])
a.set_title("write Distribution",fontsize=10)

a = sns.boxplot(df['write'] , orient = "v" , ax=axes[1][1])
a.set_title("write Boxplot",fontsize=10)


a = sns.distplot(df['math'] , ax=axes[2][0])
a.set_title("math Distribution",fontsize=10)

a = sns.boxplot(df['math'] , orient = "v" , ax=axes[2][1])
a.set_title("math Boxplot",fontsize=10)


a = sns.distplot(df['science'] , ax=axes[3][0])
a.set_title("science Distribution",fontsize=10)

a = sns.boxplot(df['science'] , orient = "v" , ax=axes[3][1])
a.set_title("science Boxplot",fontsize=10)

plt.show()
fig, axes = plt.subplots(nrows=3,ncols=2)
fig.set_size_inches(10,18)
a = sns.distplot(df['socst'] , ax=axes[0][0])
a.set_title("socst Distribution",fontsize=10)

a = sns.boxplot(df['socst'] , orient = "v" , ax=axes[0][1])
a.set_title("socst Boxplot",fontsize=10)

a = sns.distplot(df['awards'] , ax=axes[1][0])
a.set_title("awards Distribution",fontsize=10)

a = sns.boxplot(df['awards'] , orient = "v" , ax=axes[1][1])
a.set_title("awards Boxplot",fontsize=10)

a = sns.distplot(df['cid'] , ax=axes[2][0])
a.set_title("cid Distribution",fontsize=10)

a = sns.boxplot(df['cid'] , orient = "v" , ax=axes[2][1])
a.set_title("cid Boxplot",fontsize=10)
plt.show()<jupyter_output><empty_output><jupyter_text>Bivariate and Multivariate Analysis<jupyter_code>plt.figure(figsize=(8,8))
sns.stripplot(df["prog"], df['read'], jitter=True) 
plt.show()<jupyter_output><empty_output><jupyter_text>**Students who got marks above 70 in Reading Skills have chosen Academic Program**<jupyter_code>plt.figure(figsize=(8,8))
sns.stripplot(df["prog"], df['write'], jitter=True) 
plt.show()<jupyter_output><empty_output><jupyter_text>**Students who got marks below 35 in writing Skills have mostly chosen vocation Program, Rest distribution is similar**<jupyter_code>plt.figure(figsize=(8,8))
sns.stripplot(df["prog"], df['math'], jitter=True) 
plt.show()<jupyter_output><empty_output><jupyter_text>**Students having higher marks in Maths tends to choose Academic as there Program(>60)** With high capital_gain probability of getting income >50K is also high.<jupyter_code>plt.figure(figsize=(8,8))
sns.stripplot(df["prog"], df['science'], jitter=True) 
plt.show()<jupyter_output><empty_output><jupyter_text>**Here students having the highest marks and lowest marks in science have chosen vocation program, Rest of the distribution is similar**<jupyter_code>plt.figure(figsize=(8,8))
sns.stripplot(df["prog"], df['socst'], jitter=True) 
plt.show()<jupyter_output><empty_output><jupyter_text>**Students who got marks above 70 in socst have chosen Academic Program and below 30 have chosen vocation**<jupyter_code>plt.figure(figsize=(8,5))
sns.stripplot(df["prog"], df['cid'], jitter=True) 
plt.show()<jupyter_output><empty_output><jupyter_text>**Nearly Similar trend**<jupyter_code>sns.pairplot(df,hue="prog")
plt.figure(figsize=(10,8))
sns.heatmap(df.corr(),annot=True)
plt.show()<jupyter_output><empty_output><jupyter_text>Outlier Check<jupyter_code># construct box plot for continuous variables
plt.figure(figsize=(10,10))
df[num].boxplot(vert=0)
plt.show()<jupyter_output><empty_output><jupyter_text>**There are nearly no outliers in most of the numerical columns, only outlier is in award variable. In Gaussian Naive Bayes, outliers will affect the shape of the Gaussian distribution and have the usual effects on the mean etc. So depending on our use case, it makes sense to remove outlier .**  outlier treatment<jupyter_code>def remove_outlier(col):
    sorted(col)
    Q1,Q3=np.percentile(col,[25,75])
    IQR=Q3-Q1
    lower_range= Q1-(1.5 * IQR)
    upper_range= Q3+(1.5 * IQR)
    return lower_range, upper_range
lr,ur=remove_outlier(df["awards"])
df["awards"]=np.where(df["awards"]>ur,ur,df["awards"])
df["awards"]=np.where(df["awards"]<lr,lr,df["awards"])
plt.figure(figsize=(10,10))
df[num].boxplot(vert=0)<jupyter_output><empty_output><jupyter_text>**get_dummies of the object variables**<jupyter_code>## Excluding prog
cat
cat1 = ['female', 'ses', 'schtyp', 'honors']<jupyter_output><empty_output><jupyter_text>**Drop First is used to ensure that multiple columns created based on the levels of categorical variable are not included else it will result in to multicollinearity . This is done to ensure that we do not land in to dummy trap.**<jupyter_code>df =pd.get_dummies(df, columns=cat1,drop_first=True)
df.shape<jupyter_output><empty_output><jupyter_text>**Train-Test Split**Split X and y into training and test set in 70:30 ratio with random_state=1<jupyter_code># Arrange data into independent variables and dependent variables
X = df.drop("prog",axis=1)  ## Features
y = df["prog"]  ## Target

# Split X and y into training and test set in 70:30 ratio
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30 , random_state=1)<jupyter_output><empty_output><jupyter_text>**Gaussian Naive Bayes**<jupyter_code>from sklearn.naive_bayes import GaussianNB
from sklearn import metrics
NB_model = GaussianNB()
NB_model.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Now GaussianNB classifier is built. The classifier is trained using training data. We can use fit() method for training it. After building a classifier, our model is ready to make predictions. We can use predict() method with test set features as its parameters. **Performance Matrix on train data set**<jupyter_code>y_train_predict = NB_model.predict(X_train)
model_score = NB_model.score(X_train, y_train)                      ## Accuracy
print(model_score)
print(metrics.confusion_matrix(y_train, y_train_predict))          ## confusion_matrix
print(metrics.classification_report(y_train, y_train_predict))     ## classification_report
## Performance Matrix on test data set
y_test_predict = NB_model.predict(X_test)
model_score = NB_model.score(X_test, y_test)                    ## Accuracy
print(model_score)
print(metrics.confusion_matrix(y_test, y_test_predict))         ## confusion_matrix
print(metrics.classification_report(y_test, y_test_predict))    ## classification_report<jupyter_output>0.7333333333333333
[[25 10]
 [ 6 19]]
              precision    recall  f1-score   support

    academic       0.81      0.71      0.76        35
    vocation       0.66      0.76      0.70        25

    accuracy                           0.73        60
   macro avg       0.73      0.74      0.73        60
weighted avg       0.74      0.73      0.74        60<jupyter_text>Accuracy of our Gaussian Naive Bayes model<jupyter_code>NB_model.score(X_train, y_train)
NB_model.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>**Looking at Recalls, Training accuracy and Test accuracy. Model seems to be performing well  KNN Model For naive bayes algorithm while calculating likelihoods of numerical features it assumes the feature to be normally distributed and then we calculate probability using mean and variance of that feature only and also it assumes that all the predictors are independent to each other. Scale doesn’t matter. Performing a features scaling in this algorithms may not have much effect. **Generally, good KNN performance usually requires preprocessing of data to make all variables similarly scaled and centered**Now lets apply zscore on continues columns and see the performance for KNN<jupyter_code># Arrange data into independent variables and dependent variables
X =  df.drop("prog",axis=1) ## Features
y = df["prog"]  ## Target
X[["read","write","math","science","socst","awards","cid"]]=X[["read","write","math","science","socst","awards","cid"]].apply(zscore)
X.head()
# Split X and y into training and test set in 70:30 ratio
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30 , random_state=10)
from sklearn.neighbors import KNeighborsClassifier

KNN_model=KNeighborsClassifier()
KNN_model.fit(X_train,y_train)
## Performance Matrix on train data set
y_train_predict = KNN_model.predict(X_train)
model_score = KNN_model.score(X_train, y_train)
print(model_score)
print(metrics.confusion_matrix(y_train, y_train_predict))
print(metrics.classification_report(y_train, y_train_predict))
## Performance Matrix on test data set
y_test_predict = KNN_model.predict(X_test)
model_score = KNN_model.score(X_test, y_test)
print(model_score)
print(metrics.confusion_matrix(y_test, y_test_predict))
print(metrics.classification_report(y_test, y_test_predict))<jupyter_output>0.7333333333333333
[[21  6]
 [10 23]]
              precision    recall  f1-score   support

    academic       0.68      0.78      0.72        27
    vocation       0.79      0.70      0.74        33

    accuracy                           0.73        60
   macro avg       0.74      0.74      0.73        60
weighted avg       0.74      0.73      0.73        60<jupyter_text>**Default value n_neighbors=5, lets check the performance for K=7**<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

KNN_model=KNeighborsClassifier(n_neighbors=7)
KNN_model.fit(X_train,y_train)
## Performance Matrix on train data set
y_train_predict = KNN_model.predict(X_train)
model_score = KNN_model.score(X_train, y_train)
print(model_score)
print(metrics.confusion_matrix(y_train, y_train_predict))
print(metrics.classification_report(y_train, y_train_predict))
## Performance Matrix on test data set
y_test_predict = KNN_model.predict(X_test)
model_score = KNN_model.score(X_test, y_test)
print(model_score)
print(metrics.confusion_matrix(y_test, y_test_predict))
print(metrics.classification_report(y_test, y_test_predict))<jupyter_output>0.7166666666666667
[[20  7]
 [10 23]]
              precision    recall  f1-score   support

    academic       0.67      0.74      0.70        27
    vocation       0.77      0.70      0.73        33

    accuracy                           0.72        60
   macro avg       0.72      0.72      0.72        60
weighted avg       0.72      0.72      0.72        60<jupyter_text>**Run the KNN with no of neighbours to be 1,3,5..19 and *Find the optimal number of neighbours from K=1,3,5,7....19 using the Mis classification error**Hint: Misclassification error **(MCE) = 1 - Test accuracy score.** Calculated MCE for each model with neighbours = 1,3,5...19 and find the model with lowest MCE<jupyter_code># empty list that will hold accuracy scores
ac_scores = []

# perform accuracy metrics for values from 1,3,5....19
for k in range(1,20,2):
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train, y_train)
    # evaluate test accuracy
    scores = knn.score(X_test, y_test)
    ac_scores.append(scores)

# changing to misclassification error
MCE = [1 - x for x in ac_scores]
MCE<jupyter_output><empty_output><jupyter_text>**Plot misclassification error vs k (with k value on X-axis) using matplotli**<jupyter_code>import matplotlib.pyplot as plt
# plot misclassification error vs k
plt.plot(range(1,20,2), MCE)
plt.xlabel('Number of Neighbors K')
plt.ylabel('Misclassification Error')
plt.show()<jupyter_output><empty_output><jupyter_text>For K = 15 it is giving the best test accuracy lets check train and test for K=17 with other evaluation metrics<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

KNN_model=KNeighborsClassifier(n_neighbors=15)
KNN_model.fit(X_train,y_train)
## Performance Matrix on train data set
y_train_predict = KNN_model.predict(X_train)
model_score = KNN_model.score(X_train, y_train)
print(model_score)
print(metrics.confusion_matrix(y_train, y_train_predict))
print(metrics.classification_report(y_train, y_train_predict))
## Performance Matrix on test data set
y_test_predict = KNN_model.predict(X_test)
model_score = KNN_model.score(X_test, y_test)
print(model_score)
print(metrics.confusion_matrix(y_test, y_test_predict))
print(metrics.classification_report(y_test, y_test_predict))
0.8166666666666667 - 0.6928571428571428<jupyter_output><empty_output><jupyter_text>**As the difference between train and test accuracies is 12.38 % which is more than 10%(Industry standard).So, we will look for the next best value of K. which is K=19**<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

KNN_model=KNeighborsClassifier(n_neighbors=19)
KNN_model.fit(X_train,y_train)
## Performance Matrix on train data set
y_train_predict = KNN_model.predict(X_train)
model_score = KNN_model.score(X_train, y_train)
print(model_score)
print(metrics.confusion_matrix(y_train, y_train_predict))
print(metrics.classification_report(y_train, y_train_predict))
## Performance Matrix on test data set
y_test_predict = KNN_model.predict(X_test)
model_score = KNN_model.score(X_test, y_test)
print(model_score)
print(metrics.confusion_matrix(y_test, y_test_predict))
print(metrics.classification_report(y_test, y_test_predict))
0.8 - 0.7142857142857143<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10902759174926333, 'fraction_numerical': 0.0330565229038307, 'mean_word_length': 5.072218607677294, 'pattern_counts': {'":': 2, '<': 127, '<?xml version=': 0, '>': 129, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '347370', 'license': 'MIT', 'n_tokens_mistral': 6647, 'n_tokens_neox': 6150, 'n_words': 1694, 'path': 'M6_Machine_Learning/M6_Machine_Learning_W1 Supervised Learning - Classification Models - Naive Bayes and KNN/Week-1 Mentoring session (NB & KNN) Updated.ipynb', 'repo_name': 'fborrasumh/greatlearning-pgp-dsba', 'size': '18665'}