<jupyter_start><jupyter_text>Cross-validation methodsIn this notebook you will consider K-Fold cross validation method to estimate quality of a claasifier.<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np
import numpy.testing as np_testing
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Load MAGIC Data Set Source: https://magic.mpp.mpg.de/<jupyter_code>!wget https://archive.ics.uci.edu/ml/machine-learning-databases/magic/magic04.data<jupyter_output><empty_output><jupyter_text>Features description:- **Length:** continuous  major axis of ellipse [mm]- **Width:** continuous  minor axis of ellipse [mm]- **Size:** continuous  10-log of sum of content of all pixels [in phot]- **Conc:** continuous  ratio of sum of two highest pixels over fSize [ratio]- **Conc1:** continuous  ratio of highest pixel over fSize [ratio]- **Asym:** continuous  distance from highest pixel to center, projected onto major axis [mm]- **M3Long:** continuous  3rd root of third moment along major axis [mm]- **M3Trans:** continuous  3rd root of third moment along minor axis [mm]- **Alpha:** continuous  angle of major axis with vector to origin [deg]- **Dist:** continuous  distance from origin to center of ellipse [mm]- **Label:** g,h  gamma (signal), hadron (background)g = gamma (signal): 12332 \h = hadron (background): 6688<jupyter_code>f_names = np.array(["Length", "Width", "Size", "Conc", "Conc1", "Asym", "M3Long", "M3Trans", "Alpha", "Dist"])

data = pd.read_csv("magic04.data", header=None, names=list(f_names)+["Label"])
data.head()<jupyter_output><empty_output><jupyter_text>Data preparation<jupyter_code># prepare a matrix of input features
X = data[f_names].values

# prepare a vector of true labels
y = 1 * (data['Label'].values == "g")
# print sizes of X and y
X.shape, y.shape
X[:2]
y[:5]<jupyter_output><empty_output><jupyter_text>Preprocessing Scale input data using StandardScaler:$$X_{new} = \frac{X - \mu}{\sigma}$$<jupyter_code># Import StandardScaler
from sklearn.preprocessing import StandardScaler

# Create object of the class and set up its parameters
ss = StandardScaler()

# Estimate mean and sigma values
ss.fit(X)

# Scale the sample
X = ss.transform(X)<jupyter_output><empty_output><jupyter_text>Train / test split Split the dataset into train and test samples.<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, stratify=y, random_state=11)<jupyter_output><empty_output><jupyter_text>Define a model Now let's create a neural network and fit it.<jupyter_code>!pip install pytorch-lightning 
import torch
from torch.nn import functional as F
from torch import nn
import pytorch_lightning as pl

class Model(pl.LightningModule):

    def __init__(self):
        super().__init__()
        
        # define all layers of the netwrok
        self.net = nn.Sequential(
                                nn.Linear(10, 10), 
                                nn.Tanh(), 
                                nn.Linear(10, 1), 
                                nn.Sigmoid())

    
    def forward(self, x):
        # make a prediction for x
        return self.net(x)

    # calculate loss function values
    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = F.binary_cross_entropy(y_hat, y)
        return loss

    # define optimizer to fit the network
    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)
model = Model()
model<jupyter_output><empty_output><jupyter_text>Data loader creationWe will define a helping function for converting `X_train` and `y_train` into PyTorch `DataLoader`.<jupyter_code>from torch.utils.data import TensorDataset, DataLoader

def create_data_loader(X_train, y_train, batch_size=128):
    # combine X and y into one pytorch tensor dataset
    dataset_train = TensorDataset(torch.tensor(X_train, dtype=torch.float), 
                                  torch.tensor(y_train.reshape(-1, 1), dtype=torch.float))
    # loader divides our train data into batches
    train_loader = DataLoader(dataset_train, batch_size=batch_size, num_workers=4)
    return train_loader
# example of usage
create_data_loader(X[:5], y[:5], 1)<jupyter_output><empty_output><jupyter_text>Quality metricsWe will use the function below to calculate quality metrics for out model.<jupyter_code>from sklearn import metrics

def quality_metrics_report(y_true, y_pred, y_proba):
    """
    Parameters
    ----------
    y_true: array-like of shape (n_samples,)
        Ground truth (correct) target values.
    y_pred: array-like of shape (n_samples,)
        Estimated targets as returned by a classifier.
    y_proba : array, shape = [n_samples]
        Target scores, can be probability estimates of the positive
        class.
        
    Returns
    -------
    List of metric values: [accuracy, precision, recall, f1, roc_auc]
    """
    
    accuracy  = metrics.accuracy_score(y_true, y_pred)
    precision = metrics.precision_score(y_true, y_pred)
    recall    = metrics.recall_score(y_true, y_pred)
    f1        = metrics.f1_score(y_true, y_pred)
    roc_auc   = metrics.roc_auc_score(y_true, y_proba)
    
    return [accuracy, precision, recall, f1, roc_auc]
# example of usage
quality_metrics_report(y_true=[0, 0, 1, 1], 
                       y_pred=[0, 1, 1, 1], 
                       y_proba=[0.1, 0.6, 0.8, 0.9])<jupyter_output><empty_output><jupyter_text>Fit model<jupyter_code># init model and trainer
model = Model()
trainer = pl.Trainer(max_epochs=10)

# create pytorch dataloader
train_loader = create_data_loader(X_train, y_train, batch_size=128)

# fit the model
trainer.fit(model, train_loader)<jupyter_output><empty_output><jupyter_text>Test model<jupyter_code># make prediction
y_test_proba = model(torch.tensor(X_test, dtype=torch.float))[:, 0].detach().numpy()
y_test_pred  = 1 * (y_test_proba > 0.5)
# example
y_test_proba[:5]
# example
y_test_pred[:5]
# example
y_test[:5]
# compute quaility metrics
report = quality_metrics_report(y_test, y_test_pred, y_test_proba)
report
# transform report into a table
df = pd.DataFrame()
df['Metrics'] = ['Accuracy', 'Precision', 'Recall', 'F1', 'ROC AUC']
df['Mean']    = report
df<jupyter_output><empty_output><jupyter_text>Uncertainty estimation with bootstrap We will estimate uncertainties of the quality metrics using the bootstrap method. Algorithm:    - Given a model fitted on a train sample- N – number of objects in a test sample- For i=1, …, B do    - Sample with **replacement** a subsample with N objects from the test sample    - Calculate quality metrics on this subsample- Estimate statistics of the metrics: mean, **variance**, confidence intervals  Task 1Using `y_test_pred`, `y_test_proba` predictions, and the bootstrap estimate means and standard deviation of the quality metrics for the classifier above. **Hint:** use `resample(y_test, y_test_pred, y_test_proba, replace=True, n_samples=len(y_test))` to make bootstrap samples. Use function `quality_metrics_report` above to compute the quality metrics.<jupyter_code>from sklearn.utils import resample

def boostrap_uncertainties(y_test, y_test_pred, y_test_proba, n_iterations=100):
    
    metrics = []
    
    # go through each iteration of KFold
    for it in range(n_iterations):
        
        ### BEGIN SOLUTION
        
        ### END SOLUTION
        
    metrics = np.array(metrics)
    df = pd.DataFrame()
    df['Metrics'] = columns=['Accuracy', 'Precision', 'Recall', 'F1', 'ROC AUC']
    df['Mean']    = metrics.mean(axis=0)
    df['Std']     = metrics.std(axis=0)
    
    return df
# run uncertainty estimation
report = boostrap_uncertainties(y_test, y_test_pred, y_test_proba, n_iterations=100)
report<jupyter_output><empty_output><jupyter_text>Expected approximate output:       ```pythonMetrics       Mean     Std0 Accuracy    0.862501 0.0033021 Precision   0.861022 0.0038302 Recall      0.939593 0.0034353 F1          0.898587 0.002760```<jupyter_code>### BEGIN HIDDEN TESTS
actual  = report.values[-1, 1:].astype(np.float)
desired = np.array([0.920540, 0.002441])
np_testing.assert_allclose(actual, desired, atol=0.005)
### END HIDDEN TESTS<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1083071721806327, 'fraction_numerical': 0.02100941801497223, 'mean_word_length': 4.013922518159807, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 54, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '207748', 'license': 'Apache-2.0', 'n_tokens_mistral': 2811, 'n_tokens_neox': 2642, 'n_words': 759, 'path': 'course_1/4_topic/4_uncertainty_estimation/4_uncertainty_estimation.ipynb', 'repo_name': 'HSE-LAMBDA/ML-IDS', 'size': '8282'}