<jupyter_start><jupyter_text>**Assignment 2 - IN-STK5000/9000 - Autumn21****Students**: - <NAME>- <NAME>- <NAME>**Data**: - Ionosphere- URL for description: https://archive.ics.uci.edu/ml/datasets/ionosphere- URL for data: https://archive.ics.uci.edu/ml/machine-learning-databases/ionosphere/ionosphere.data- URL for names: https://archive.ics.uci.edu/ml/machine-learning-databases/ionosphere/ionosphere.names**Task type**:- Binary Classification  TasksIn this file, we will set up a pipeline for1. Collecting data for an experiment **(DONE)**2. Processing the data **(DONE)**3. Running a machine learning algorithm **(DONE)**4. Evaluating the algorithm **(DONE)**5. Choosing the best algorithm and hyperparameters **(DONE)**The main questions we need to answer are1. How will data be collected?2. How much data would we need? **(2x224 samples was enough to achieve 97% of accuracy)**3. What algorithm would be best? **(rf)**4. How would the amount of data influence algorithm selection? **(yes, oversampling the minority class up to the same qtd of the other class (balanced classes distribution) increased the accuracy from 93 to 97%)**5. How robust is our procedure to assumptions?  Dependencies<jupyter_code>!pip install pipelinehelper
from sklearn.preprocessing import StandardScaler, MaxAbsScaler, MinMaxScaler
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from pipelinehelper import PipelineHelper
from sklearn.pipeline import Pipeline
from sklearn.utils import resample
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
seed = 1
n_jobs = -1

import numpy as np
import abc
import scipy.stats

# This is a base class for generating classification problems
class BaseClassificationGenerator:
    @abc.abstractmethod
    ## generate n_points (x,y) pairs
    def generate(self, n_points):
        pass
    ## generate an (x,y) at x.
    def generate_at_point(self, x):
        pass

# Generates data where $X|Y=i~Normal(\mu_i, \Sigma_i)$
class GaussianClassificationGenerator(BaseClassificationGenerator):
    # Initialise the class centers
    def __init__(self, n_dimensions, class_proportions):
        self.n_dimensions = n_dimensions
        self.class_proportions = class_proportions
        self.n_classes = class_proportions.shape[0]
        self.means = np.zeros([self.n_classes, self.n_dimensions])
        self.covariances = np.zeros([self.n_classes, self.n_dimensions, self.n_dimensions])
        # Generate means   $\mu_i \sim Uniform([0,1]^n)$
        # and covariances  $\Sigma_i \sim Gamma(1)$
        for i in range(self.n_classes):
            self.means[i] = np.random.uniform(size=self.n_dimensions)
            self.covariances[i] = scipy.stats.wishart.rvs(self.n_dimensions, np.identity(self.n_dimensions))
    # generate data
    def generate(self, n_points):
        Y = np.random.choice(self.n_classes, p=self.class_proportions, size=n_points)
        X = np.zeros([n_points, self.n_dimensions])
        for t in range(n_points):
            X[t]= np.random.multivariate_normal(self.means[Y[t]], self.covariances[Y[t]])
        return [X, Y]<jupyter_output>Requirement already satisfied: pipelinehelper in /usr/local/lib/python3.7/dist-packages (0.7.8)
Requirement already satisfied: sklearn in /usr/local/lib/python3.7/dist-packages (from pipelinehelper) (0.0)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from sklearn->pipelinehelper) (0.22.2.post1)
Requirement already satisfied: numpy>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sklearn->pipelinehelper) (1.19.5)
Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sklearn->pipelinehelper) (1.4.1)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sklearn->pipelinehelper) (1.0.1)<jupyter_text>Importing data<jupyter_code># importing data
df = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/ionosphere/ionosphere.data")<jupyter_output><empty_output><jupyter_text>Dealing with raw data and resampling<jupyter_code># checking if there are only 2 labels
df.g.unique()
# converting to binary values 0 and 1
df.g = df.g.apply(lambda x: 0 if x=='b' else 1)

# checking the balance between classes
df.g.value_counts()  # unbalanced
# balancing classes
# oversampling data for minority class from 126 to 224
df_minority = resample(
    df[df.g == 0], 
    replace=True, 
    n_samples=224, 
    random_state=seed, 
    stratify=None
)

# putting both together
df_resampled = pd.concat([df_minority, df[df.g == 1]])
del df_minority

# checking the balance between classes
df_resampled.g.value_counts()  # balanced
total = df.g.value_counts()[0] + df.g.value_counts()[1]
generator = GaussianClassificationGenerator(34, np.array([df.g.value_counts()[0]/total, df.g.value_counts()[1]/total]))
df_artificial = generator.generate(500)
# splitting features and classes
X = df.iloc[:, :-1]
Y = df.g

X_resampled = df_resampled.iloc[:, :-1]
Y_resampled = df_resampled.g
del df, df_resampled
X_artificial = df_artificial[0]
Y_artificial = df_artificial[1]
del df_artificial<jupyter_output><empty_output><jupyter_text>Creating pipelines<jupyter_code># creating pipeline among pre-processings and classifiers
pipe = Pipeline([
    ('scaler', PipelineHelper([
        ('std', StandardScaler()),
        ('maxabs', MaxAbsScaler()),
        ('minmax', MinMaxScaler()),
    ])),
    ('classifier', PipelineHelper([
        ('lr', LogisticRegression(solver='saga', random_state=seed, n_jobs=n_jobs)),
        ('rf', RandomForestClassifier(random_state=seed, n_jobs=n_jobs)),
    ])),
])

params = {
    'scaler__selected_model': pipe.named_steps['scaler'].generate({
        'std__with_mean': [True, False],
        'std__with_std': [True, False],
        'maxabs__copy': [True],
        'minmax__copy': [True],
    }),
    'classifier__selected_model': pipe.named_steps['classifier'].generate({
        'lr__penalty': ['l1', 'l2'],
        'lr__max_iter': [100, 300, 500],
        'rf__n_estimators': [100, 300, 500],
        'rf__criterion': ['gini', 'entropy'],
        'rf__bootstrap': [True, False]
    })
}<jupyter_output><empty_output><jupyter_text>Creating random grid search with cross validation<jupyter_code>rgs = RandomizedSearchCV(
    estimator=pipe,
    param_distributions=params, 
    n_iter=100,
    scoring='accuracy', 
    n_jobs=n_jobs, 
    refit=True, 
    cv=10,
    random_state=seed
)

rgs.fit(X, Y)  # using unbalanced classes
print(rgs.best_params_)
print(rgs.best_score_)
rgs1 = RandomizedSearchCV(
    estimator=pipe,
    param_distributions=params, 
    n_iter=100,
    scoring='accuracy', 
    n_jobs=n_jobs, 
    refit=True, 
    cv=10,
    random_state=seed
)

rgs1.fit(X_resampled, Y_resampled)  # using balanced classes
print(rgs1.best_params_)
print(rgs1.best_score_)
rgs2 = RandomizedSearchCV(
    estimator=pipe,
    param_distributions=params, 
    n_iter=100,
    scoring='accuracy', 
    n_jobs=n_jobs, 
    refit=True, 
    cv=10,
    random_state=seed
)

rgs2.fit(X_artificial, Y_artificial)  # using artificial classes
print(rgs2.best_params_)
print(rgs2.best_score_)<jupyter_output>{'scaler__selected_model': ('minmax', {'copy': True}), 'classifier__selected_model': ('rf', {'bootstrap': False, 'criterion': 'gini', 'n_estimators': 500})}
0.8520000000000001<jupyter_text>Evaluating results<jupyter_code># for unbalanced data
# taking only some importante dimensions for evaluation
# sorting mean test scores in decending fashion

rts = pd.DataFrame(rgs.cv_results_)[['mean_test_score', 'param_scaler__selected_model', 'param_classifier__selected_model']]
rts = rts.sort_values('mean_test_score', ascending=False)
rts.head(50)
# for balanced data
# taking only some importante dimensions for evaluation
# sorting mean test scores in decending fashion

rts_balanced = pd.DataFrame(rgs1.cv_results_)[['mean_test_score', 'param_scaler__selected_model', 'param_classifier__selected_model']]
rts_balanced = rts_balanced.sort_values('mean_test_score', ascending=False)
rts_balanced
# for artificial data

rts_artificial = pd.DataFrame(rgs2.cv_results_)[['mean_test_score', 'param_scaler__selected_model', 'param_classifier__selected_model']]
rts_artificial = rts_artificial.sort_values('mean_test_score', ascending=False)
rts_artificial<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12067955477445812, 'fraction_numerical': 0.020620972466315172, 'mean_word_length': 4.903181189488244, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 36, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '298258', 'license': 'MIT', 'n_tokens_mistral': 2974, 'n_tokens_neox': 2823, 'n_words': 640, 'path': 'assignment2/IN_STK5000_9000_Assignment2.ipynb', 'repo_name': 'fabiorodp/IN_STK5000_Adaptive_methods_for_data_based_decision_making', 'size': '8567'}