<jupyter_start><jupyter_text>MACHINE LEARNING 1 - BASICS  ML Tasks<jupyter_code>Type of tasks
  - Classification
  - Regression
  - Structured annotation
  - Clustering
  - Transcription
Challenges
  - Quality of data 
  - Time-Consuming task − Another challenge faced by ML models is the consumption of time especially for data acquisition, feature extraction and retrieval. 
  - Lack of specialist persons − As ML technology is still in its infancy stage, availability of expert resources is a tough job.
  - No clear objective for formulating business problems 
  - Issue of overfitting & underfitting 
  - Curse of dimensionality − Another challenge ML model faces is too many features of data points. This can be a real hindrance.
  - Difficulty in deployment − Complexity of the ML model makes it quite difficult to be deployed in real life.
Applications
  - Emotion analysis
  - Sentiment analysis
  - Error detection and prevention
  - Weather forecasting and prediction
  - Stock market analysis and forecasting
  - Speech synthesis
  - Speech recognition
  - Customer segmentation
  - Object recognition
  - Fraud detection
  - Fraud prevention
  - Recommendation of products to customer in online shopping<jupyter_output><empty_output><jupyter_text>Categorisation of the problem<jupyter_code>Categorize by input:
- If you have labelled data, it’s a supervised learning problem.
- If you have unlabelled data and want to find structure, it’s an unsupervised learning problem.
- If you want to optimize an objective function by interacting with an environment, it’s a reinforcement learning problem.

Categorize by output.
- If the output of your model is a number, it’s a regression problem.
- If the output of your model is a class, it’s a classification problem.
- If the output of your model is a set of input groups, it’s a clustering problem.
- Do you want to detect an anomaly ? That’s anomaly detection<jupyter_output><empty_output><jupyter_text>Learning Classification  Supervised Learning<jupyter_code>- The majority of practical machine learning uses supervised learning.
- Supervised learning is where you have input variables (x) and an output variable (Y) and you use an algorithm to learn the mapping function from the input to the output.
- Y = f(X)
- It is called supervised learning because the process of an algorithm learning from the training dataset can be thought of as a teacher supervising the learning process.
Supervised learning problems can be further grouped into regression and classification problems.
  - Classification: A classification problem is when the output variable is a category, such as “red” or “blue” or “disease” and “no disease”.
  - Regression: A regression problem is when the output variable is a real value, such as “dollars” or “weight”.

Some common types of problems built on top of classification and regression include recommendation and time series prediction respectively.

Some popular examples of supervised machine learning algorithms are:
  - Linear regression for regression problems.
  - Random forest for classification and regression problems.
  - Support vector machines for classification problems.<jupyter_output><empty_output><jupyter_text>Unsupervised Learning<jupyter_code>Unsupervised learning is where you only have input data (X) and no corresponding output variables.
The goal for unsupervised learning is to model the underlying structure or distribution in the data in order to learn more about the data.

These are called unsupervised learning because unlike supervised learning above there is no correct answers and there is no teacher. 
Algorithms are left to their own devises to discover and present the interesting structure in the data.

Unsupervised learning problems can be further grouped into clustering and association problems.
  - Clustering: A clustering problem is where you want to discover the inherent groupings in the data, such as grouping customers by purchasing behavior.
  - Association:  An association rule learning problem is where you want to discover rules that describe large portions of your data, such as people that buy X also tend to buy Y.

Some popular examples of unsupervised learning algorithms are:
  - k-means for clustering problems.
  - Apriori algorithm for association rule learning problems.<jupyter_output><empty_output><jupyter_text>Semi Supervised<jupyter_code>Problems where you have a large amount of input data (X) and only SOME of the data is labeled (Y) are called semi-supervised learning problems.

These problems sit in between both supervised and unsupervised learning.
  - A good example is a photo archive where only some of the images are labeled, (e.g. dog, cat, person) and the majority are unlabeled.
  - Many real world machine learning problems fall into this area.
  - This is because it can be expensive or time-consuming to label data as it may require access to domain experts. Whereas unlabeled data is cheap and easy to collect and store.

You can use unsupervised learning techniques to discover and learn the structure in the input variables.
You can also use supervised learning techniques to make best guess predictions for the unlabeled data, feed that data back into the supervised learning algorithm as training data and use the model to make predictions on new unseen data.<jupyter_output><empty_output><jupyter_text>Basic Stats<jupyter_code>import pandas as pd
import statsmodels.formula.api as sm
df = pd.DataFrame({"A": [10,20,30,40,50], "B": [20, 30, 10, 40, 50], "C": [32, 234, 23, 23, 42523]})
result = sm.ols(formula="A ~ B + C", data=df).fit()
print(result.params)
print(result.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                      A   R-squared:                       0.579
Model:                            OLS   Adj. R-squared:                  0.158
Method:                 Least Squares   F-statistic:                     1.375
Date:                Tue, 16 Jun 2020   Prob (F-statistic):              0.421
Time:                        10:44:00   Log-Likelihood:                -18.178
No. Observations:                   5   AIC:                             42.36
Df Residuals:                       2   BIC:                             41.19
Df Model:                           2                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]<jupyter_text>Interpretation of a regression Output<jupyter_code>R-squared        - coeff of determination. How well the regression line approximates real data points
Adj. R-squared   - same as above, adjsuted for number of observations and degrees of freedom of residuals
F-Stat           - measure of how significant the fit is. Mean sq error / mean sq error of residuals
Prob(F-stat)     - prob to get F-stat, given the null hypothesis they are unrelated
Log-Likelihood   - value of the likelihood function of the fitted model
AIC              - Akaike Information Criterion: adjusts log-likelihood based on number of observations and complexity of model   
BIC              - Bayesian Information Criterion: same as AIC, but with higher penalty for models wit more parameters
Df Residuals     - degrees of freedom of the residulas. Number of observations - number of parameters
Df Model         - number of parameters in the model (not including the constant)
Coef             - estimated value of coeff
Std err          - basic standard error of the estimate of coeff. 
t                - t-stat (how statistically significant the coeff is)
P>|t|            - p-value that null hypothesis that the coeff = 0 is true. if < 0.05: strong relationship between term and response
95% Conf.Int     - lower and upper value of the 95% confidence interval
Omnibus          - Angostino test: provides a combined statistical test of the presence of skewness and kurtosis
Prob(Omnibus)    - same as above, turned into prob
Skew             - measure of symmetry of data around mean
Kurtosis         - measure of shape distribution
Durbin-Watson    - test for autocorrelation (important in time series)
Jarque=Bera      - different test of skewness and kurtosis
Prob (JB)
Cond.No          - test for multicolinearity (parameters are related to each other)
Log Likelihood
  - not possible to compare raw log-lieklihoods between models (better to use AIC or BIC)
  - Likelihood is the likelihood of the entire model given a set of parameter estimates.
  - It is calculated by 
    - taking a set of parameter estimates
    - calculating the probability density for each one
    - multiplying the probability densities for all the observations together 
    - >> this follows from probability theory in that P(A and B) = P(A)P(B) if A and B are independent)
  - In practice, what this means for linear regression:
    - you take a set of parameter estimates (beta, sd)
    - plug them into the normal pdf
    - calculate the density for each observation y at that set of parameter estimates
    - multiply them all together. 
  - Typically, we choose to work with the log-likelihood because 
    - it is easier to calculate because instead of multiplying we can sum (log(a*b) = log(a) + log(b)), which is computationally faster. 

Log likelihood is used for almost everything. 
  - It is the basic quantity that we use to find parameter estimates (Maximum Likelihood Estimates) for a huge suite of models. 
  - For simple linear regression, these estimates turn out to be the same as those for least squares, but for more complicated models least squares may not work.

AIC
  - Lower value of AIC suggests "better" model, but it is a relative measure of model fit 
  - It is used for model selection (only), i.e. it lets you to compare different models estimated on the same dataset
  - Lower indicates a more parsimonious model, relative to a model fit with a higher AIC.
  - Model selection conducted with the AIC will choose the same model as leave-one-out cross validation 
  - Dont compare too many models with the AIC (like with p-values) because lowest AIC does not mean that it is the most appropriate mod
Omnibus          
  - We want something close to zero, which means normalcy of residuals
        
Prob(Omnibus)    
  - statistical test that residuals are normally distributed
  - we want something close to 1
Skew
  - closer to 0 means symetric residual distribution
  - If skewness is less than −1 or greater than +1, the distribution is highly skewed.
  - If skewness is between −1 and −½ or between +½ and +1, the distribution is moderately skewed.
  - If skewness is between −½ and +½, the distribution is approximately symmetric.
    
Kurtosis
  - a uniform distribution has a kurtosis of 1.8 (excess -1.2) (lowest is discrete with 2 outcomes: kurto 1)
  - a normal distribution has a kurtosis of 3 (excess 0)
  - a logistic distribution has a kurtosis of 4.2 (excess 1.2)
  - highest kurtosis is a student distribution
Durbin-Watson
  - Value between zero and 4.0
  - we hope to get a value between 1 and 2. ideally 2
  - A value of 2.0 means there is no autocorrelation detected in the sample. 
  - values from zero to 2.0 indicate positive autocorrelation
  - values from 2.0 to 4.0 indicate negative autocorrelation.
Condition Number 
  – This test measures the sensitivity of a function output as compared to its input.
  - When we have multicollinearity, we can expect much higher fluctuations to small changes in the data
  - We want a relatively small number (something below 30)<jupyter_output><empty_output><jupyter_text>Train Accuracy vs Test Accuracy vs Confusion matrix<jupyter_code>Accuracy: 
  - The amount of correct classifications / the total amount of classifications.
  - The train accuracy: The accuracy of a model on examples it was constructed on.
  - The test accuracy is the accuracy of a model on examples it hasn't seen.

Confusion matrix: 
  - Confusion matrix returns the testing accuracy 
  - A tabulation of the predicted class (usually vertically) against the actual class (thus horizontally).
from sklearn.neighbors import KNeighborsClassifier

# KNN Classifier
knn = KNeighborsClassifier(n_neighbors = 3)
knn.fit(X_train, y_train)
Y_pred = knn.predict(X_test)

acc_knn_train = round(knn.score(X_train, y_train) * 100, 2)
print(acc_knn_train)

acc_knn_test = round(knn.score(X_test, y_test) * 100, 2)
print(acc_knn_test)<jupyter_output><empty_output><jupyter_text>Confusion Matrix<jupyter_code>Confusion matrix 
  - matrix (table) that can be used to measure the performance of an machine learning algorithm, usually a supervised learning one

By convention here
  - row = instances of an actual class 
  - column = instances of a predicted class
# 2 Class looks like this
-----------------------------------------------------------------
                    Predicted Negative      Predicted Positive
Actual Negative       True Negative            False Positive
Actual Positive       False NEgative           True Positive
-----------------------------------------------------------------

Accuracy = (TN + TP) / (Total)
Precision = TP / (FP + TP)<jupyter_output><empty_output><jupyter_text>Implementation in Scikit-Learn<jupyter_code>from sklearn.metrics import classification_report, confusion_matrix

y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]
y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]

cm = confusion_matrix(y_actu, y_pred)
print(cm)
from sklearn.metrics import accuracy_score
accuracy_score(y_actu, y_pred)

print(classification_report(y_actu, y_pred))
Possible to use pyCM for richer output, but beed to make sure that we feed pyCM numpy arrays:
    
# to check 
print('Type of Y_actu: {}'.format(type(y_actu)))
print('Type of Y_[pred]: {}'.format(type(y_pred)))

# then use this
from pycm import *

cm = ConfusionMatrix(actual_vector=y_actu.values, predict_vector=y_pred)    # putting .values will transfrom the df into an array
print(cm)<jupyter_output><empty_output><jupyter_text>Models Adequacy<jupyter_code># Good Article
https://www.hackernoon.com/choosing-the-right-machine-learning-algorithm-68126944ce1f
    
# Good Cheatsheet 
http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html<jupyter_output><empty_output><jupyter_text>Classification<jupyter_code>Classification is the task of predicting the type or class of an object within a finite number of options. 
The output variable for classification is always a categorical variable. 

- K-Nearest neighbors algorithm – simple but computationally exhaustive.
- Naive Bayes – Based on Bayes theorem.
- Logistic Regression – Linear model for binary classification.
- SVM – can be used for binary/multiclass classifications.
- Decision Tree – ‘If Else’ based classifier, more robust to outliers.
- Ensembles – Combination of multiple machine learning models clubbed together to get better results.<jupyter_output><empty_output><jupyter_text>Regression<jupyter_code>Regression is a set of problems where the output variable can take continuous values

- Linear Regression – Simplest baseline model for regression task, works well only when data is linearly separable and very less or no multicollinearity is present.
- Lasso Regression – Linear regression with L2 regularization.
- Ridge Regression – Linear regression with L1 regularization.
- SVM regression
- Decision Tree Regression etc.<jupyter_output><empty_output><jupyter_text>Clustering<jupyter_code>Clustering is the task of grouping similar objects together. 
Machine learning models help to identify similar objects automatically without manual intervention. 
We can not build effective supervised machine learning models (models that need to be trained with manually curated or labeled data) without homogeneous data.

- K means – Simple but suffers from high variance.
- K means++ – Modified version of K means.
- K medoids.
- Agglomerative clustering – A hierarchical clustering model.
- DBSCAN – Density-based clustering algorithm etc.<jupyter_output><empty_output><jupyter_text>Dimensionality Reduction<jupyter_code>Dimensionality is the number of predictor variables used to predict the independent variable or target.
Often in the real world datasets the number of variables is too high. 
Too many variables also bring the curse of overfitting to the models.
In practice among these large numbers of variables, not all variables contribute equally towards the goal.
In a large number of cases, we can actually preserve variances with a lesser number of variables

- PCA – It creates lesser numbers of new variables out of a large number of predictors. The new variables are independent of each other but less interpretable.
- TSNE – Provides lower dimensional embedding of higher-dimensional data points.
- SVD – Singular value decomposition is used to decompose the matrix into smaller parts in order to efficient calculation.<jupyter_output><empty_output><jupyter_text>Deep Learning<jupyter_code>Deep learning is a subset of machine learning which deals with neural networks. 
Based on the architecture of neural networks let’s list down important deep learning models:
- Multi-Layer perceptron
- Convolution Neural Networks
- Recurrent Neural Networks
- Boltzmann machine
- Autoencoders etc.<jupyter_output><empty_output><jupyter_text>Cross Validation<jupyter_code>There is always a need to validate the stability of the machine learning model and need some kind of assurance that:
  - the  model has got most of the patterns from the data correct
  - the model is not picking up too much on the noise
  - the model is low on bias and variance.
Validation
  - process of deciding whether the numerical results quantifying hypothesized relationships between variables, are acceptable as descriptions of the data.

Residiuals
  - evaluation of residuals = error estimation for the model is made after training 
  - a numerical estimate of the difference in predicted and original responses is done, also called the training error. 
  - However, this only gives us an idea about how well our model does on the data used to train it. 
  - It possible that the model is underfitting or overfitting the data. 

Cross Validation:
  - Pupose: get an indication of how well the learner will generalize to an independent / unseen data set
  - How: discussed below
Model Bias / Variance

Bias
  - In an ideal scenario, these error values should sum up to zero. 
  - To return the model’s bias, we take the average of all the errors. 
  - The lower the average value, better the model.

Variance
  - Similarly for calculating the model variance, we take standard deviation of all the errors. 
  - A low value of standard deviation suggests our model does not vary a lot with different subsets of training data.

We should focus on achieving a balance between bias and variance. 
  - This can be done by reducing the variance and controlling bias to an extent.
  - This will result in a better predictive model.
  - This trade-off usually leads to building less complex predictive models as well.<jupyter_output><empty_output><jupyter_text>K-Fold Cross Validation<jupyter_code>The Problem
  - As there is never enough data to train your model, removing a part of it for validation poses a problem of underfitting. 
  - By reducing the training data, we risk losing important patterns/ trends in data set, which in turn increases error induced by bias. 

The Solution
  - What we require is a method that provides ample data for training the model and also leaves ample data for validation. 
  - K-Fold cross validation does exactly that.

K Fold cross validation
  - the data is divided into k subsets. 
  - the holdout method is repeated k times, such that each time:
      - one of the k subsets is used as the test set / validation set
      - the other k-1 subsets are put together to form a training set. 
  - The error estimation is averaged over all k trials to get total effectiveness of our model. 
  - As can be seen, every data point gets to be in a validation set exactly once, and gets to be in a training set k-1 times. 
  - This significantly reduces
      - bias as we are using most of the data for fitting
      - variance as most of the data is also being used in validation set. 
  - Interchanging the training and test sets also adds to the effectiveness of this method. 
  - As a general rule and empirical evidence, K = 5 or 10 is generally preferred, but nothing’s fixed and it can take any value.
Example of 5 Fold Cross Validation

Validation  XXXXXXXXXX  XXXXXXXXXX  XXXXXXXXXX  XXXXXXXXXX
XXXXXXXXXX  Validation  XXXXXXXXXX  XXXXXXXXXX  XXXXXXXXXX 
XXXXXXXXXX  XXXXXXXXXX  Validation  XXXXXXXXXX  XXXXXXXXXX 
XXXXXXXXXX  XXXXXXXXXX  XXXXXXXXXX  Validation  XXXXXXXXXX 
XXXXXXXXXX  XXXXXXXXXX  XXXXXXXXXX  XXXXXXXXXX  Validation<jupyter_output><empty_output><jupyter_text>Stratified K-Fold Cross Validation<jupyter_code>In some cases, there may be a large imbalance in the response variables. 
  - For example, in dataset concerning price of houses, there might be large number of houses having high price. 
  - Or in case of classification, there might be several times more negative samples than positive samples. 

For such problems, a slight variation in the K-Fold cross validation technique is made:
  - Each fold contains approximately the same percentage of samples of each target class as the complete set
  - in case of prediction problems, the mean response value is approximately equal in all the folds. 

This variation is also known as Stratified K Fold.<jupyter_output><empty_output><jupyter_text>Leave P-Out Cross Validation (exhaustive method)<jupyter_code>Exhaustive Methods computes all possible ways the data can be split into training and test sets.
Leave P-Out
  - Leaves p data points out of training data
  - Meaning: 
      - if there are n data points in the original sample then, n-p samples are used to train the model and p points are used as the validation set. 
      - This is repeated for all combinations in which original sample can be separated this way
      - Then the error is averaged for all trials, to give overall effectiveness.

This method is exhaustive in the sense that:
  - it needs to train and validate the model for all possible combinations
  - for moderately large p, it can become computationally infeasible.
    
A particular case of this method is when p = 1. 
  - This is known as Leave one out cross validation. 
  - This method is generally preferred over the previous one because it does not suffer from the intensive computation
  - Number of possible combinations is equal to number of data points in original sample or n.<jupyter_output><empty_output><jupyter_text>Python Implementation<jupyter_code># K-Fold
from sklearn import model_selection

kfold = model_selection.KFold(n_splits=10, random_state=100)                                    # chose the number of folds
model_kfold = LinearRegression()                                                                # chose the model
results_kfold = model_selection.cross_val_score(model_kfold, X_test, y_test, cv=kfold)
print("Accuracy: %.2f%%" % (results_kfold.mean()*100.0))
print(results_kfold)  
# Stratified k-fold cross validation

from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, random_state=None)

# X is the feature set and y is the target
for train_index, test_index in skf.split(X,y): 
    print("Train:", train_index, "Validation:", val_index) 
    X_train, X_test = X[train_index], X[val_index] 
    y_train, y_test = y[train_index], y[val_index]
# k-fold cross validation with repetition (if the train set does not adequately represent the entire population, strtified is not good)
# In repeated cross-validation, the cross-validation procedure is repeated n times, yielding n random partitions of the original sample

from sklearn.model_selection import RepeatedKFold
rkf = RepeatedKFold(n_splits=5, n_repeats=10, random_state=None)

# X is the feature set and y is the target
for train_index, test_index in rkf.split(X):
     print("Train:", train_index, "Validation:", val_index)
     X_train, X_test = X[train_index], X[val_index]
     y_train, y_test = y[train_index], y[val_index]
# Using Cross Val SCore

from sklearn import datasets, linear_model
from sklearn.model_selection import cross_val_score

diabetes = datasets.load_diabetes()
X = diabetes.data[:150]
y = diabetes.target[:150]
lasso = linear_model.Lasso()
print(cross_val_score(lasso, X, y, cv=10))<jupyter_output>[ 0.34557351  0.34848715  0.26654262 -0.01126674  0.24875619  0.08731544
  0.13386583  0.14000888  0.2873109   0.00960079]<jupyter_text>Optimization<jupyter_code>There are different optimization methods, some are model dependent, such as optimizing the K-neighbor value for a KNN (see KNN)
One generic methodology, that adapts to any model is GridSearchCV
A Machine Learning model is defined as a mathematical model with a number of parameters that need to be learned from the data. 
However, there are some parameters, known as Hyperparameters and those cannot be directly learned. 
They are commonly chosen by human based on some intuition or hit and trial before the actual training begins. 
These parameters exhibits their importance by improving performance of the model such as its complexity or its learning rate. 

Models can have many hyper-parameters and finding the best combination of parameters can be treated as a search problem.
But it can be found by just trying all combinations and see what parameters work best. 
The main idea behind it is to create a grid of hyper-parameters and just try all of their combinations with GridSearchCV.
GridSearchCV takes a dictionary that describes the parameters that could be tried on a model to train it. 
The grid of parameters is defined as a dictionary, where the keys are the parameters and the values are the settings to be tested.

Use GridsearchCV
One of the great things about GridSearchCV is that it is a meta-estimator. 
It takes an estimator like SVC, and creates a new estimator, that behaves exactly the same – in this case, like a classifier. 
You should add refit=True and choose verbose (verbose just means the text output describing the process).<jupyter_output><empty_output><jupyter_text>Example 1 with Iris on SVM<jupyter_code>import pandas as pd  
import seaborn as sns
import numpy as np  
from sklearn.svm import SVC  
from sklearn.metrics import classification_report, confusion_matrix  

iris = sns.load_dataset('iris')
iris = pd.DataFrame(iris,columns=['sepal_length','sepal_width','petal_length','petal_width', 'species']  )

# separate the data from the target attributes
X = iris[['sepal_length', 'sepal_width','petal_length','petal_width']]
y = iris['species']

# Prepare the test / train sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
from sklearn.model_selection import GridSearchCV

param_grid = {'C': [0.1,1, 10, 100], 
              'gamma': [1,0.1,0.01,0.001],
              'kernel': ['rbf', 'poly', 'sigmoid']}

# defining parameter range 
grid = GridSearchCV(SVC(),param_grid,refit=True,verbose=1)

# fitting the model for grid search
grid.fit(X_train,y_train)
# What fit does is a bit more involved then usual. 
# First, it runs the same loop with cross-validation, to find the best parameter combination. 
# Once it has the best combination, it runs fit again on all data passed to fit (without cross-validation), to built a single new model
# You can inspect the best parameters found by GridSearchCV in the best_params_ attribute, and the best estimator in the best_estimator_ attribute:

# print best parameter after tuning 
print(grid.best_params_) 
  
# print how our model looks after hyper-parameter tuning 
print(grid.best_estimator_) 
# Then you can re-run predictions and see classification report on this grid object just like you would with a normal model.
grid_predictions = grid.predict(X_test)

# print confusion matrix and classification report 
print(confusion_matrix(y_test,grid_predictions))
print(classification_report(y_test,grid_predictions))<jupyter_output>[[11  0  0]
 [ 0 13  0]
 [ 0  0  6]]
              precision    recall  f1-score   support

      setosa       1.00      1.00      1.00        11
  versicolor       1.00      1.00      1.00        13
   virginica       1.00      1.00      1.00         6

    accuracy                           1.00        30
   macro avg       1.00      1.00      1.00        30
weighted avg       1.00      1.00      1.00        30<jupyter_text>Example 2 with Iris on KNN<jupyter_code>import pandas as pd  
import seaborn as sns
import numpy as np  

from sklearn.neighbors import KNeighborsClassifier  
from sklearn.metrics import classification_report, confusion_matrix  

iris = sns.load_dataset('iris')
iris = pd.DataFrame(iris,columns=['sepal_length','sepal_width','petal_length','petal_width', 'species']  )

# separate the data from the target attributes
X = iris[['sepal_length', 'sepal_width','petal_length','petal_width']]
y = iris['species']

# Prepare the test / train sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
from sklearn.model_selection import GridSearchCV

# Using shift/tab, the hyperparameters for KNeighborsClassifier are the following:
#     n_neighbors=5,
#     weights='uniform',
#     algorithm='auto',
#     leaf_size=30,
#     p=2,
#     metric='minkowski',
#     metric_params=None,
#     n_jobs=None,

param_grid = {'n_neighbors': [1,2,3,4,5,6,7,8,9,10,11,12], 
              'weights': ['uniform', 'distance']}

# defining parameter range
knn = KNeighborsClassifier
grid = GridSearchCV(knn(),param_grid,refit=True,verbose=1)

# fitting the model for grid search
grid.fit(X_train,y_train)

# print best parameter after tuning 
print(grid.best_params_) 
  
# print how our model looks after hyper-parameter tuning 
print(grid.best_estimator_)
# Then you can re-run predictions and see classification report on this grid object just like you would with a normal model.
grid_predictions = grid.predict(X_test)

# print confusion matrix and classification report 
print(confusion_matrix(y_test,grid_predictions))
print(classification_report(y_test,grid_predictions))<jupyter_output>[[11  0  0]
 [ 0 12  1]
 [ 0  0  6]]
              precision    recall  f1-score   support

      setosa       1.00      1.00      1.00        11
  versicolor       1.00      0.92      0.96        13
   virginica       0.86      1.00      0.92         6

    accuracy                           0.97        30
   macro avg       0.95      0.97      0.96        30
weighted avg       0.97      0.97      0.97        30
{'fraction_non_alphanumeric': 0.06298780881119784, 'fraction_numerical': 0.015222860091595175, 'mean_word_length': 3.609335513601903, 'pattern_counts': {'":': 3, '<': 94, '<?xml version=': 0, '>': 97, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 20, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '218559', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 8773, 'n_tokens_neox': 8098, 'n_words': 3873, 'path': 'Learning Notes/Learning Notes ML - 1a Intro.ipynb', 'repo_name': 'k21k/Python-Notes', 'size': '31006'}