<jupyter_start><jupyter_text>Laborator 6: modele de clasificare<NAME>, _grupa 10LF383_ Tătaru Dragoș-Cătălin, _grupa 10LF383_ Dataseturile utilizate pentru antrenarea algoritmilor de clasificare:- Wi-Fi localization- Echocardiogram- Seeds- Dermatology  Modelele de clasificare in a nutshellPentru clasificarea datelor din dataseturile enumerate mai sus vom utiliza următoarele modele de clasificare: kNN, Decision Tree, MLP, Gaussian NB și Random Forest. În paragraful următor vă vom explica modul de funcționare al fiecărui algoritm și evidenția hiperparametrii și formulele care stau la baza acestora."hiperparametrii și formulele care stau la baza acestora"sursă: xkcd 1838: Machine Learning  [1. k-nearest neighbors classifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.htmlsklearn.neighbors.KNeighborsClassifier) class `sklearn.neighbors.KNeighborsClassifier`(n_neighbors=5, weights='uniform', algorithm='auto', leaf_size=30, p=2, metric='minkowski', metric_params=None, n_jobs=None, **kwargs) Într-o problemă de clasificare, algoritmul kNN (_k_-nearest neighbors) identifică cei mai apropiați _k_ vecini ai fiecărui item neclasificat - indiferent de etichetele acestora - vecini localizați în setul de antrenare. Determinarea claselor din care fac parte itemii neclasificați se face prin votare: clasa în care aparțin majoritatea vecinilor se consideră clasa itemului.Exemplu: clasificarea itemului c cu 3NN. În urma votării se determină clasa lui c: o.sursă: YouTube (How kNN algorithm works de <NAME>)Pentru determinarea distanței dintre itemi se pot utiliza mai multe metrici. Scikit-learn admite orice funcție Python ca și metrică, dar implicit folosește metrica _Minkowski_. Iată câteva exemple de metrici des utilizate în kNN:- _distanța Minkowski_: $d_{st} = \sqrt[p]{\sum_{j=1}^n |x_{sj} - y_{tj}|^p}$  (_Obs._: p este un hiperparametru utilizat de Scikit-learn)- _distanța Euclideană_: $d(\textbf{x},\textbf{y}) = \sqrt{\sum_{i=1}^n (y_i - x_i)^2}$- _distanța Manhattan (City block)_: $d_{st} = \sum_{j=1}^n |x_{sj} - y_{tj}|$- _distanța Mahalanobis_: $d(\textbf{x},\textbf{y}) = \sqrt{\sum_{i=1}^n \frac{(x_i - y_i)^2}{s_i^2}}$, unde $s_i$ este deviația standard a lui $x_i$ și $y_i$ în sample  [2. Decision tree classifier](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.htmlsklearn.tree.DecisionTreeClassifier) class `sklearn.tree.DecisionTreeClassifier`(criterion='gini', splitter='best', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=None, random_state=None, max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, class_weight=None, presort='deprecated', ccp_alpha=0.0) Un arbore de decizie (_decision tree_) este o structură arborescentă tip flowchart unde un nod intern reprezintă un feature, ramura este un criteriu de decizie, iar fiecare frunză este un rezultat, o clasificare. Algoritmul Decision tree selectează cel mai bun feature folosind o metrică ASM (_Attribute Selection Measure_), convertește un nod feature la un nod tip criteriu de decizie, și partiționează (splits) datasetul în subseturi. Procesul se execută recursiv până arborele conține numai noduri criterii de decizie și noduri frunză rezultat. Cu cât arborele este mai adânc, cu atât sunt mai complexe criteriile de decizie și modelul are o acuratețe mai mare. Structura unui arbore de decizie.sursă: KDnuggets (Decision Tree Algorithm, Explained)Pentru măsurarea calității unui split, Scikit-learn utilizează două metrici ASM:- _impuritatea Gini_ (cât de des este etichetat greșit un element ales aleator dacă a fost etichetat folosind distribuția etichetelor dintr-un subset; poate determina overfitting-ul modelului): $Gini(p) = 1 - \sum_{j=1}^c p_j^2$ - _entropia_ (similar cu Gini impurity, mai intensă d.p.d.v. computațional din cauza funcției logaritmice): $H(p) = - \sum_{j=1}^c p_j \log p_j$(unde c este numărul de clase (etichete), iar $p_j$ este subsetul etichetat cu clasă i, unde $j \in \{1, 2, ..., c\}$).  [3. Multilayer perceptron (MLP) classifier](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.htmlsklearn.neural_network.MLPClassifier) class `sklearn.neural_network.MLPClassifier`(hidden_layer_sizes=(100, ), activation='relu', solver='adam', alpha=0.0001, batch_size='auto', learning_rate='constant', learning_rate_init=0.001, power_t=0.5, max_iter=200, shuffle=True, random_state=None, tol=0.0001, verbose=False, warm_start=False, momentum=0.9, nesterovs_momentum=True, early_stopping=False, validation_fraction=0.1, beta_1=0.9, beta_2=0.999, epsilon=1e-08, n_iter_no_change=10, max_fun=15000) _Perceptronii_ sunt o clasă de clasificatori utilizați în învățarea supervizată, fiind un model matematic al unui neuron biologic. În particular, _perceptronii multistrat (MLP)_ formează rețele neuronale cu mai multe straturi de perceptroni: un strat de intrare, unul sau mai multe straturi intermediare (ascunse), și un strat de ieșire.Un perceptron multistrat ilustrat.sursă: GitHub (ledell/sldm4-h2o)Într-o rețea neuronală, o _funcție de activare_ definește ieșirea unui perceptron după ce este supus unui set de intrare. În forma lui cea mai simplă, funcția poate returna un rezultat binar (funcție liniară, output 0 sau 1): făcând analogie cu neuronul biologic, dacă trece un impuls electric prin axonul acestuia sau nu. În cazul rețelelor neuronale moderne care utilizează mai multe straturi de perceptroni, funcțiile de activare pot fi și non-binare (non-liniare). Scikit-learn admite funcții de activare de ambele tipuri în implementarea MLP classifier:- _funcția identitate_: $f(x) = x$- _sigmoida logistică_: $f(x) = \frac{1}{1 + \exp(-x)}$- _tangenta hiperbolică_: $f(x) = \tanh(x) = \frac{\sinh(x)}{\cosh(x)} = \frac{e^x - e^{-x}}{e^x + e^{-x}}$- _Rectified Linear Unit (ReLU)_: $f(x) = \max(0, x) = \begin{cases} 0 & \text{dacă } x \leq 0 \\ x & \text{dacă } x > 0 \end{cases}$De asemenea, clasificatorul MLP din Scikit-learn utilizează și algoritmi de optimizare a ponderilor (solvers): _LBFGS_ (algoritm Quasi-Newton), _SGD_ (stochastic gradient descent) și _Adam_ (algoritm derivat din SGD, creat de <NAME> și <NAME>).  [4. Gaussian Naïve Bayes classifier](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.htmlsklearn.naive_bayes.GaussianNB) class `sklearn.naive_bayes.GaussianNB`(priors=None, var_smoothing=1e-09) Algoritmul de clasificare _Gaussian Naïve Bayes_ aparține familiei de clasificatori _Naïve Bayes_, care presupun că prezența unui feature într-o clasă nu este afectată de prezența altor features; pe scurt, proprietățile contribuie independent la probabilitatea apartenenței la o clasă. În particular, algoritmul _Gaussian Naïve Bayes_ urmează funcția de probabilitate (PDF) a unei distribuții normale (Gaussiene):$$\large P(x_i | y) = \frac{1}{\sqrt{2 \pi \sigma_y^2}}\exp\bigg(-\frac{(x_i - \mu_y)^2}{2 \sigma_y^2}\bigg),$$unde parametrii $\sigma_y$ și $\mu_y$, deviația standard și media, sunt determinați folosind maximum likelihood estimation (MLE), o metodă de estimare a parametrilor unei PDF prin maximizarea unei funcții de likelihood (cât de bine se potrivește un sample cu un model statistic).  [5. Random Forest classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.htmlsklearn.ensemble.RandomForestClassifier) class `sklearn.ensemble.RandomForestClassifier`(n_estimators=100, criterion='gini', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features='auto', max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, bootstrap=True, oob_score=False, n_jobs=None, random_state=None, verbose=0, warm_start=False, class_weight=None, ccp_alpha=0.0, max_samples=None) Un clasificator _Random forest_ se folosește de ipotezele emise de mai mulți arbori de decizie aleatori (_random trees_), obținuți în urma unui _random split_. Un random forest se obține prin construirea unui random tree pentru fiecare set de antrenare. Acești arbori funcționează ca un ansamblu; pentru fiecare dată de intrare se aplică modelele din ansamblu, și rezultatul final se obține agregând rezultatele prin votare. Astfel, un random forest este un _meta-estimator_: se obține o predicție în urma mai multor predicții.Un model random forest făcând o predicție; în urma votării se obține rezultatul 1.sursă: Medium (Towards Data Science: Understanding Random Forest)La fel ca la _Decision Tree classifier_, pentru măsurarea calității unui split, Scikit-learn utilizează două metrici:- _impuritatea Gini_: $Gini(p) = 1 - \sum_{j=1}^c p_j^2$ - _entropia_: $H(p) = - \sum_{j=1}^c p_j \log p_j$(unde c este numărul de clase (etichete), iar $p_j$ este subsetul etichetat cu clasă i, unde $j \in \{1, 2, ..., c\}$).  1. Wi-Fi localizationSîrbu Matei-Dan, _grupa 10LF383_Sursă dataset: http://archive.ics.uci.edu/ml/datasets/Wireless+Indoor+LocalizationDescriere dataset: [DOI 10.1007/978-981-10-3322-3_27 via ResearchGate](Docs/chp_10.1007_978-981-10-3322-3_27.pdf)Synopsis: Setul de date _Wireless Indoor Localization_ cuprinde 2000 de măsurători ale puterii semnalului (măsurat în dBm) recepționat de la routerele unui birou din Pittsburgh. Acest birou are șapte routere și patru camere; un utilizator înregistrează cu ajutorul unui smartphone o dată pe secundă puterea semnalelor venite de la cele șapte routere, fiecărei înregistrări fiindu-i asociate camera în care se afla utilizatorul la momentul măsurării (1, 2, 3 sau 4).În figura de mai jos este ilustrat un sample din dataset: ![Sample](./Images/wifi_localization_sample.png)În cele ce urmează, coloana Class (camera) este reprezentată de y, iar coloanele WS1 - WS7 (features: puterea semnalului de la fiecare router), de X.<jupyter_code>import numpy as np
import pandas as pd
from IPython.display import display, HTML
header = ['WS1', 'WS2', 'WS3', 'WS4', 'WS5', 'WS6', 'WS7', 'Class']
data_wifi = pd.read_csv("./Datasets/wifi_localization.txt", names=header, sep='\t')
display(HTML("<i>Dataset overview:</i>"))
display(data_wifi)
X = data_wifi.values[:, :7]
y = data_wifi.values[:, -1]
folds = 5<jupyter_output><empty_output><jupyter_text>Testarea algoritmilor de clasificare pe setul de date cu Scikit-learn<jupyter_code># just a pretty printing function, don't mind me...
def print_stats_cv(model_cv_stats):
    print(f"Test accuracy for each fold: {model_cv_stats['test_accuracy']} \n=> Average test accuracy: {round(model_cv_stats['test_accuracy'].mean() * 100, 3)}%")
    print(f"Train accuracy for each fold: {model_cv_stats['train_accuracy']} \n=> Average train accuracy: {round(model_cv_stats['train_accuracy'].mean() * 100, 3)}%")
    print(f"Test F1 score for each fold: {model_cv_stats['test_f1_macro']} \n=> Average test F1 score: {round(model_cv_stats['test_f1_macro'].mean() * 100, 3)}%")
    print(f"Train F1 score for each fold: {model_cv_stats['train_f1_macro']} \n=> Average train F1 score: {round(model_cv_stats['train_f1_macro'].mean() * 100, 3)}%")<jupyter_output><empty_output><jupyter_text>1. k-nearest neighbors classifier<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_validate
from sklearn.preprocessing import MinMaxScaler

# hiperparametri
knn_neighbors = 4
knn_minkowski_p = 3

# scalare date
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(X)

# implementare KNN
model = KNeighborsClassifier(n_neighbors=knn_neighbors, p=knn_minkowski_p)
model_cv_stats = cross_validate(model, X_scaled, y, cv=folds, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>{folds}-fold cross validation for {knn_neighbors}-nearest neighbors classification:</h4>"))
print_stats_cv(model_cv_stats)<jupyter_output><empty_output><jupyter_text>2. Decision Tree classifier<jupyter_code>from sklearn.tree import DecisionTreeClassifier

# hiperparametri
dt_criterion = 'gini'
dt_splitter = 'best'

# implementare Decision Tree
model = DecisionTreeClassifier(criterion=dt_criterion, splitter=dt_splitter, random_state=42)
model_cv_stats = cross_validate(model, X, y, cv=folds, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>{folds}-fold cross validation for Decision Trees classification:</h4>"))
print_stats_cv(model_cv_stats)<jupyter_output><empty_output><jupyter_text>3. Multilayer Perceptron (MLP) classifier<jupyter_code>from sklearn.neural_network import MLPClassifier

# hiperparametri
mlp_solver = 'adam'
mlp_activation = 'relu'
mlp_alpha=1e-3
mlp_hidden_layer_sizes = (50,50)

# implementare MLP
model = MLPClassifier(solver=mlp_solver, activation=mlp_activation, alpha=mlp_alpha, hidden_layer_sizes=mlp_hidden_layer_sizes, random_state=42)
model_cv_stats = cross_validate(model, X, y, cv=folds, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>{folds}-fold cross validation for MLP classification</h4>"))
display(HTML(f"using hardcoded hyperparameters - Solver: <b>{mlp_solver}</b>, Activation function: <b>{mlp_activation}</b>, Parameter for regularization (α): <b>{mlp_alpha}</b>, Hidden layer sizes: <b>{mlp_hidden_layer_sizes}</b>"))
print_stats_cv(model_cv_stats)<jupyter_output><empty_output><jupyter_text>4. Gaussian Naïve Bayes classifier<jupyter_code>from sklearn.naive_bayes import GaussianNB

# implementare GNB
model = GaussianNB()
model_cv_stats = cross_validate(model, X, y, cv=folds, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>{folds}-fold cross validation for Gaussian NB classification</h4>"))
print_stats_cv(model_cv_stats)<jupyter_output><empty_output><jupyter_text>5. Random Forest classifier<jupyter_code>from sklearn.ensemble import RandomForestClassifier

# hiperparametri
rfc_n_estimators = 150
rfc_criterion = 'gini'

# implementare Random Forest
model = RandomForestClassifier(n_estimators=rfc_n_estimators, criterion=rfc_criterion)
model_cv_stats = cross_validate(model, X, y, cv=folds, scoring=('accuracy', 'f1_macro'), return_train_score=True)

display(HTML(f"<h4>{folds}-fold cross validation for Random Forest classification</h4>"))
print_stats_cv(model_cv_stats)<jupyter_output><empty_output><jupyter_text>Nested Cross Validation pentru optimizarea hiperparametrilor<jupyter_code>from sklearn.model_selection import KFold
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
from sklearn.metrics import accuracy_score

# CVs configuration
inner_cv = KFold(n_splits=4, shuffle=True)
outer_cv = KFold(n_splits=5, shuffle=True)

# outer CV folds:
print("5-fold cross validation: split overview")
splits = outer_cv.split(range(data_wifi.index.size))
subsets = pd.DataFrame(columns=['Fold', 'Train row indices', 'Test row indices'])
for i, split_data in enumerate(splits):
    subsets.loc[i]=[i + 1, split_data[0], split_data[1]]
display(subsets)<jupyter_output>5-fold cross validation: split overview<jupyter_text>1. k-nearest neighbors classifier<jupyter_code>outer_cv_acc = []
param_candidates = {'n_neighbors': np.linspace(start=1, stop=30, num=30, dtype=int),
                    'p': np.linspace(start=1, stop=5, num=4, dtype=int)} 
param_search = RandomizedSearchCV(estimator=KNeighborsClassifier(), param_distributions=param_candidates, scoring='accuracy', cv=inner_cv, random_state=42)
    
for fold in range(5):
    X_train = X_scaled[subsets.loc[subsets.index[fold],'Train row indices']]
    y_train = y[subsets.loc[subsets.index[fold],'Train row indices']]
    X_test = X_scaled[subsets.loc[subsets.index[fold],'Test row indices']]
    y_test = y[subsets.loc[subsets.index[fold],'Test row indices']]
    
    param_search.fit(X_train, y_train)
    y_estimated = param_search.predict(X_test)
    accuracy = accuracy_score(y_test, y_estimated)
    outer_cv_acc.append(accuracy)
    
    print(f'Outer fold {fold+1}, optimal hyperparameters after inner 4-fold CV: {param_search.best_params_}')
    print(f'kNN model accuracy with optimal hyperparameters: {accuracy}')
    
print(f'\nAverage model accuracy: {round(np.mean(outer_cv_acc) * 100, 2)}%')<jupyter_output>Outer fold 1, optimal hyperparameters after inner 4-fold CV: {'p': 3, 'n_neighbors': 7}
kNN model accuracy with optimal hyperparameters: 0.98
Outer fold 2, optimal hyperparameters after inner 4-fold CV: {'p': 3, 'n_neighbors': 3}
kNN model accuracy with optimal hyperparameters: 0.985
Outer fold 3, optimal hyperparameters after inner 4-fold CV: {'p': 3, 'n_neighbors': 3}
kNN model accuracy with optimal hyperparameters: 0.985
Outer fold 4, optimal hyperparameters after inner 4-fold CV: {'p': 1, 'n_neighbors': 11}
kNN model accuracy with optimal hyperparameters: 0.985
Outer fold 5, optimal hyperparameters after inner 4-fold CV: {'p': 1, 'n_neighbors': 12}
kNN model accuracy with optimal hyperparameters: 0.985

Average model accuracy: 98.4%<jupyter_text>2. Decision Tree classifier<jupyter_code>outer_cv_acc = []
param_candidates = {'criterion': ['gini', 'entropy'],
                    'splitter': ['best', 'random']} 
param_search = GridSearchCV(estimator=DecisionTreeClassifier(), param_grid=param_candidates, scoring='accuracy', cv=inner_cv)

for fold in range(5):
    X_train = X[subsets.loc[subsets.index[fold],'Train row indices']]
    y_train = y[subsets.loc[subsets.index[fold],'Train row indices']]
    X_test = X[subsets.loc[subsets.index[fold],'Test row indices']]
    y_test = y[subsets.loc[subsets.index[fold],'Test row indices']]
    
    param_search.fit(X_train, y_train)
    y_estimated = param_search.predict(X_test)
    accuracy = accuracy_score(y_test, y_estimated)
    outer_cv_acc.append(accuracy)
    
    print(f'Outer fold {fold+1}, optimal hyperparameters after inner 4-fold CV: {param_search.best_params_}')
    print(f'Decision Tree model accuracy with optimal hyperparameters: {accuracy}')
    
print(f'\nAverage model accuracy: {round(np.mean(outer_cv_acc) * 100, 2)}%')<jupyter_output>Outer fold 1, optimal hyperparameters after inner 4-fold CV: {'criterion': 'entropy', 'splitter': 'best'}
Decision Tree model accuracy with optimal hyperparameters: 0.975
Outer fold 2, optimal hyperparameters after inner 4-fold CV: {'criterion': 'gini', 'splitter': 'best'}
Decision Tree model accuracy with optimal hyperparameters: 0.9525
Outer fold 3, optimal hyperparameters after inner 4-fold CV: {'criterion': 'entropy', 'splitter': 'best'}
Decision Tree model accuracy with optimal hyperparameters: 0.96
Outer fold 4, optimal hyperparameters after inner 4-fold CV: {'criterion': 'entropy', 'splitter': 'best'}
Decision Tree model accuracy with optimal hyperparameters: 0.9825
Outer fold 5, optimal hyperparameters after inner 4-fold CV: {'criterion': 'entropy', 'splitter': 'best'}
Decision Tree model accuracy with optimal hyperparameters: 0.9675

Average model accuracy: 96.75%<jupyter_text>3. Multilayer Perceptron (MLP) classifier<jupyter_code>outer_cv_acc = []
param_candidates = {'alpha': np.linspace(start=0, stop=1e-1, num=500),
                    'activation': ['identity', 'logistic', 'tanh', 'relu']}
param_search = RandomizedSearchCV(estimator=MLPClassifier(max_iter=1000), param_distributions=param_candidates, scoring='accuracy', cv=inner_cv)

for fold in range(5):
    X_train = X[subsets.loc[subsets.index[fold],'Train row indices']]
    y_train = y[subsets.loc[subsets.index[fold],'Train row indices']]
    X_test = X[subsets.loc[subsets.index[fold],'Test row indices']]
    y_test = y[subsets.loc[subsets.index[fold],'Test row indices']]
    
    param_search.fit(X_train, y_train)
    y_estimated = param_search.predict(X_test)
    accuracy = accuracy_score(y_test, y_estimated)
    outer_cv_acc.append(accuracy)
     
    print(f'Outer fold {fold+1}, optimal hyperparameters after inner 4-fold CV: {param_search.best_params_}')
    print(f'MLP model accuracy with optimal hyperparameters: {accuracy}')
     
print(f'\nAverage model accuracy: {round(np.mean(outer_cv_acc) * 100, 2)}%')<jupyter_output>Outer fold 1, optimal hyperparameters after inner 4-fold CV: {'alpha': 0.001002004008016032, 'activation': 'logistic'}
MLP model accuracy with optimal hyperparameters: 0.975
Outer fold 2, optimal hyperparameters after inner 4-fold CV: {'alpha': 0.08857715430861723, 'activation': 'logistic'}
MLP model accuracy with optimal hyperparameters: 0.97
Outer fold 3, optimal hyperparameters after inner 4-fold CV: {'alpha': 0.08336673346693386, 'activation': 'logistic'}
MLP model accuracy with optimal hyperparameters: 0.975
Outer fold 4, optimal hyperparameters after inner 4-fold CV: {'alpha': 0.021042084168336674, 'activation': 'tanh'}
MLP model accuracy with optimal hyperparameters: 0.9925
Outer fold 5, optimal hyperparameters after inner 4-fold CV: {'alpha': 0.06472945891783567, 'activation': 'tanh'}
MLP model accuracy with optimal hyperparameters: 0.9675

Average model accuracy: 97.6%<jupyter_text>4. Gaussian Naïve Bayes classifier<jupyter_code>outer_cv_acc = []
param_candidates = {'var_smoothing': np.linspace(start=1e-9, stop=1e-2, num=500)} 
param_search = RandomizedSearchCV(estimator=GaussianNB(), param_distributions=param_candidates, scoring='accuracy', cv=inner_cv)

for fold in range(5):
    X_train = X[subsets.loc[subsets.index[fold],'Train row indices']]
    y_train = y[subsets.loc[subsets.index[fold],'Train row indices']]
    X_test = X[subsets.loc[subsets.index[fold],'Test row indices']]
    y_test = y[subsets.loc[subsets.index[fold],'Test row indices']]
    
    param_search.fit(X_train, y_train)
    y_estimated = param_search.predict(X_test)
    accuracy = accuracy_score(y_test, y_estimated)
    outer_cv_acc.append(accuracy)
    
    print(f'Outer fold {fold+1}, optimal hyperparameters after inner 4-fold CV: {param_search.best_params_}')
    print(f'Gaussian Naïve Bayes model accuracy with optimal hyperparameters: {accuracy}')
    
print(f'\nAverage model accuracy: {round(np.mean(outer_cv_acc) * 100, 2)}%')<jupyter_output>Outer fold 1, optimal hyperparameters after inner 4-fold CV: {'var_smoothing': 0.00274549170741483}
Gaussian Naïve Bayes model accuracy with optimal hyperparameters: 0.985
Outer fold 2, optimal hyperparameters after inner 4-fold CV: {'var_smoothing': 0.002404810378757515}
Gaussian Naïve Bayes model accuracy with optimal hyperparameters: 0.9825
Outer fold 3, optimal hyperparameters after inner 4-fold CV: {'var_smoothing': 0.0012024056893787576}
Gaussian Naïve Bayes model accuracy with optimal hyperparameters: 0.985
Outer fold 4, optimal hyperparameters after inner 4-fold CV: {'var_smoothing': 2.0041078156312626e-05}
Gaussian Naïve Bayes model accuracy with optimal hyperparameters: 0.98
Outer fold 5, optimal hyperparameters after inner 4-fold CV: {'var_smoothing': 0.002725451629258517}
Gaussian Naïve Bayes model accuracy with optimal hyperparameters: 0.9875

Average model accuracy: 98.4%<jupyter_text>5. Random Forest classifier<jupyter_code>outer_cv_acc = []
param_candidates = {'criterion': ['gini', 'entropy'],
                   'n_estimators': np.linspace(start=1, stop=500, num=500, dtype=int)} 
param_search = RandomizedSearchCV(estimator=RandomForestClassifier(), param_distributions=param_candidates, scoring='accuracy', cv=inner_cv)

for fold in range(5):
    X_train = X[subsets.loc[subsets.index[fold],'Train row indices']]
    y_train = y[subsets.loc[subsets.index[fold],'Train row indices']]
    X_test = X[subsets.loc[subsets.index[fold],'Test row indices']]
    y_test = y[subsets.loc[subsets.index[fold],'Test row indices']]
    
    param_search.fit(X_train, y_train)
    y_estimated = param_search.predict(X_test)
    accuracy = accuracy_score(y_test, y_estimated)
    outer_cv_acc.append(accuracy)
    
    print(f'Outer fold {fold+1}, optimal hyperparameters after inner 4-fold CV: {param_search.best_params_}')
    print(f'Random Forest Classifier model accuracy with optimal hyperparameters: {accuracy}')
    
print(f'\nAverage model accuracy: {round(np.mean(outer_cv_acc) * 100, 2)}%')<jupyter_output>Outer fold 1, optimal hyperparameters after inner 4-fold CV: {'n_estimators': 404, 'criterion': 'entropy'}
Random Forest Classifier model accuracy with optimal hyperparameters: 0.9875
Outer fold 2, optimal hyperparameters after inner 4-fold CV: {'n_estimators': 386, 'criterion': 'entropy'}
Random Forest Classifier model accuracy with optimal hyperparameters: 0.9675
Outer fold 3, optimal hyperparameters after inner 4-fold CV: {'n_estimators': 246, 'criterion': 'gini'}
Random Forest Classifier model accuracy with optimal hyperparameters: 0.9825
Outer fold 4, optimal hyperparameters after inner 4-fold CV: {'n_estimators': 52, 'criterion': 'gini'}
Random Forest Classifier model accuracy with optimal hyperparameters: 0.99
Outer fold 5, optimal hyperparameters after inner 4-fold CV: {'n_estimators': 333, 'criterion': 'entropy'}
Random Forest Classifier model accuracy with optimal hyperparameters: 0.985

Average model accuracy: 98.25%<jupyter_text>2. EchocardiogramSîrbu Matei-Dan, _grupa 10LF383_Sursă dataset: http://archive.ics.uci.edu/ml/datasets/EchocardiogramSynopsis: Datasetul _Echocardiogram_ cuprinde 132 de înregistrări de date obținute din ecocardiografiile unor pacienți care au suferit un atac de cord în trecut. Datele sunt utilizate pentru a prezice dacă un pacient va supraviețui cel puțin un an după un atac de cord.În cele ce urmează, atributul **alive-at-1** este reprezentată de y, iar coloanele **survival, still-alive, age-at-heart-attack, pericardial-effusion, fractional-shortening, epss, lvdd, wall-motion-score, wall-motion-index, mult**, de X._Verbose attribute info:_1. survival -- the number of months patient survived (has survived, if patient is still alive). Because all the patients had their heart attacks at different times, it is possible that some patients have survived less than one year but they are still alive. Check the second variable to confirm this. Such patients cannot be used for the prediction task mentioned above.2. still-alive -- a binary variable. 0=dead at end of survival period, 1 means still alive3. age-at-heart-attack -- age in years when heart attack occurred4. pericardial-effusion -- binary. Pericardial effusion is fluid around the heart. 0=no fluid, 1=fluid5. fractional-shortening -- a measure of contracility around the heart lower numbers are increasingly abnormal6. epss -- E-point septal separation, another measure of contractility. Larger numbers are increasingly abnormal.7. lvdd -- left ventricular end-diastolic dimension. This is a measure of the size of the heart at end-diastole. Large hearts tend to be sick hearts.8. wall-motion-score -- a measure of how the segments of the left ventricle are moving9. wall-motion-index -- equals wall-motion-score divided by number of segments seen. Usually 12-13 segments are seen in an echocardiogram. Use this variable INSTEAD of the wall motion score.10. mult -- a derivate var which can be ignored11. name -- the name of the patient (I have replaced them with "name")12. group -- meaningless, ignore it13. alive-at-1 -- Boolean-valued. Derived from the first two attributes. 0 means patient was either dead after 1 year or had been followed for less than 1 year. 1 means patient was alive at 1 year.<jupyter_code>import numpy as np
import pandas as pd
from IPython.display import display, HTML
header = ['survival', 'still-alive', 'age-at-heart-attack', 'pericardial-effusion', 'fractional-shortening', 'epss', 'lvdd', 'wall-motion-score', 'wall-motion-index', 'mult', 'name', 'group', 'alive-at-1']
data_echo = pd.read_csv("./Datasets/echocardiogram.csv", names=header)
display(HTML("<i>Dataset overview:</i>"))
display(data_echo)
# ignoring irrelevant columns, as per verbose attribute info:
del data_echo['mult']
del data_echo['name']
del data_echo['group']
folds = 5<jupyter_output><empty_output><jupyter_text>După cum se poate observa în subsolul tabelului, despre unii pacienți nu se știe dacă au supraviețuit la 1 an de la atacul de cord, exact informația pe care vrem să o prezicem după antrenarea modelelor de clasificare. Aceste ecocardiograme se vor elimina din dataset.<jupyter_code>data_echo = data_echo[(data_echo['alive-at-1'] == '0') | (data_echo['alive-at-1'] == '1')]
display(HTML("<i>Last records in sanitized dataset:</i>"))
data_echo.tail()<jupyter_output><empty_output><jupyter_text>De asemenea, unele înregistrări au valori lipsă, valori pe care va trebui să le improvizăm prin _missing value imputation_. Un _imputer_ eficient pe care îl putem utiliza este `KNNImputer` din `sklearn.impute`: acesta aproximează valorile lipsă dintr-o ecocardiogramă analizând _k_ vecini ai acesteia, vecini din punct de vedere al altor parametri existenți. De exemplu, dacă nu cunoaștem variabila **epss**, dar știm valoarea **fractional_shortening**, o putem aproxima pe prima comparând-o pe cea din urmă cu a vecinilor, pentru că amândouă sunt o măsură a contractilității.<jupyter_code>data_echo.replace(to_replace='?', value=np.nan, inplace=True)
display(HTML("<i>Dataset extract with missing values:</i>"))
data_echo[33:39]
from sklearn.impute import KNNImputer
header = ['survival', 'still-alive', 'age-at-heart-attack', 'pericardial-effusion', 'fractional-shortening', 'epss', 'lvdd', 'wall-motion-score', 'wall-motion-index', 'alive-at-1']
imputer = KNNImputer(missing_values=np.nan, n_neighbors=5)
imputer.fit(data_echo)
data_echo = pd.DataFrame(data=imputer.transform(data_echo), columns=header)
display(HTML("<i>Dataset extract with values imputed using KNNImputer:</i>"))
display(data_echo[33:39])
X = data_echo.values[:, :9]
y = data_echo.values[:, -1]<jupyter_output><empty_output><jupyter_text>Testarea algoritmilor de clasificare pe setul de date cu Scikit-learn  1. k-nearest neighbors classifier<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_validate
from sklearn.preprocessing import MinMaxScaler

# hiperparametri
knn_neighbors = 4
knn_minkowski_p = 3

# scalare date
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(X)

# implementare KNN
model = KNeighborsClassifier(n_neighbors=knn_neighbors, p=knn_minkowski_p)
model_cv_stats = cross_validate(model, X_scaled, y, cv=folds, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>{folds}-fold cross validation for {knn_neighbors}-nearest neighbors classification:</h4>"))
print_stats_cv(model_cv_stats)<jupyter_output><empty_output><jupyter_text>2. Decision Tree classifier<jupyter_code>from sklearn.tree import DecisionTreeClassifier

# hiperparametri
dt_criterion = 'gini'
dt_splitter = 'best'

# implementare Decision Tree
model = DecisionTreeClassifier(criterion=dt_criterion, splitter=dt_splitter, random_state=42)
model_cv_stats = cross_validate(model, X, y, cv=folds, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>{folds}-fold cross validation for Decision Trees classification:</h4>"))
print_stats_cv(model_cv_stats)<jupyter_output><empty_output><jupyter_text>3. Multilayer Perceptron (MLP) classifier<jupyter_code>from sklearn.neural_network import MLPClassifier

# hiperparametri
mlp_solver = 'adam'
mlp_activation = 'relu'
mlp_alpha=1e-3
mlp_hidden_layer_sizes = (50,50)

# implementare MLP
model = MLPClassifier(solver=mlp_solver, activation=mlp_activation, alpha=mlp_alpha, hidden_layer_sizes=mlp_hidden_layer_sizes, random_state=42, max_iter=1000)
model_cv_stats = cross_validate(model, X, y, cv=folds, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>{folds}-fold cross validation for MLP classification</h4>"))
display(HTML(f"using hardcoded hyperparameters - Solver: <b>{mlp_solver}</b>, Activation function: <b>{mlp_activation}</b>, Parameter for regularization (α): <b>{mlp_alpha}</b>, Hidden layer sizes: <b>{mlp_hidden_layer_sizes}</b>"))
print_stats_cv(model_cv_stats)<jupyter_output><empty_output><jupyter_text>4. Gaussian Naïve Bayes classifier<jupyter_code>from sklearn.naive_bayes import GaussianNB

# implementare GNB
model = GaussianNB()
model_cv_stats = cross_validate(model, X, y, cv=folds, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>{folds}-fold cross validation for Gaussian NB classification</h4>"))
print_stats_cv(model_cv_stats)<jupyter_output><empty_output><jupyter_text>5. Random Forest classifier<jupyter_code>from sklearn.ensemble import RandomForestClassifier

# hiperparametri
rfc_n_estimators = 150
rfc_criterion = 'gini'

# implementare Random Forest
model = RandomForestClassifier(n_estimators=rfc_n_estimators, criterion=rfc_criterion)
model_cv_stats = cross_validate(model, X, y, cv=folds, scoring=('accuracy', 'f1_macro'), return_train_score=True)

display(HTML(f"<h4>{folds}-fold cross validation for Random Forest classification</h4>"))
print_stats_cv(model_cv_stats)<jupyter_output><empty_output><jupyter_text>Nested Cross Validation pentru optimizarea hiperparametrilor<jupyter_code>from sklearn.model_selection import KFold
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
from sklearn.metrics import accuracy_score

# CVs configuration
inner_cv = KFold(n_splits=4, shuffle=True)
outer_cv = KFold(n_splits=5, shuffle=True)

# outer CV folds:
print("5-fold cross validation: split overview")
splits = outer_cv.split(range(data_echo.index.size))
subsets = pd.DataFrame(columns=['Fold', 'Train row indices', 'Test row indices'])
for i, split_data in enumerate(splits):
    subsets.loc[i]=[i + 1, split_data[0], split_data[1]]
display(subsets)<jupyter_output>5-fold cross validation: split overview<jupyter_text>1. k-nearest neighbors classifier<jupyter_code>outer_cv_acc = []
param_candidates = {'n_neighbors': np.linspace(start=1, stop=30, num=30, dtype=int),
                    'p': np.linspace(start=1, stop=5, num=4, dtype=int)} 
param_search = RandomizedSearchCV(estimator=KNeighborsClassifier(), param_distributions=param_candidates, scoring='accuracy', cv=inner_cv, random_state=42)
    
for fold in range(5):
    X_train = X_scaled[subsets.loc[subsets.index[fold],'Train row indices']]
    y_train = y[subsets.loc[subsets.index[fold],'Train row indices']]
    X_test = X_scaled[subsets.loc[subsets.index[fold],'Test row indices']]
    y_test = y[subsets.loc[subsets.index[fold],'Test row indices']]
    
    param_search.fit(X_train, y_train)
    y_estimated = param_search.predict(X_test)
    accuracy = accuracy_score(y_test, y_estimated)
    outer_cv_acc.append(accuracy)
    
    print(f'Outer fold {fold+1}, optimal hyperparameters after inner 4-fold CV: {param_search.best_params_}')
    print(f'kNN model accuracy with optimal hyperparameters: {accuracy}')
    
print(f'\nAverage model accuracy: {round(np.mean(outer_cv_acc) * 100, 2)}%')<jupyter_output>Outer fold 1, optimal hyperparameters after inner 4-fold CV: {'p': 3, 'n_neighbors': 7}
kNN model accuracy with optimal hyperparameters: 1.0
Outer fold 2, optimal hyperparameters after inner 4-fold CV: {'p': 1, 'n_neighbors': 12}
kNN model accuracy with optimal hyperparameters: 1.0
Outer fold 3, optimal hyperparameters after inner 4-fold CV: {'p': 5, 'n_neighbors': 12}
kNN model accuracy with optimal hyperparameters: 0.8666666666666667
Outer fold 4, optimal hyperparameters after inner 4-fold CV: {'p': 3, 'n_neighbors': 3}
kNN model accuracy with optimal hyperparameters: 0.8666666666666667
Outer fold 5, optimal hyperparameters after inner 4-fold CV: {'p': 5, 'n_neighbors': 12}
kNN model accuracy with optimal hyperparameters: 0.9285714285714286

Average model accuracy: 93.24%<jupyter_text>2. Decision Tree classifier<jupyter_code>outer_cv_acc = []
param_candidates = {'criterion': ['gini', 'entropy'],
                    'splitter': ['best', 'random']} 
param_search = GridSearchCV(estimator=DecisionTreeClassifier(), param_grid=param_candidates, scoring='accuracy', cv=inner_cv)

for fold in range(5):
    X_train = X[subsets.loc[subsets.index[fold],'Train row indices']]
    y_train = y[subsets.loc[subsets.index[fold],'Train row indices']]
    X_test = X[subsets.loc[subsets.index[fold],'Test row indices']]
    y_test = y[subsets.loc[subsets.index[fold],'Test row indices']]
    
    param_search.fit(X_train, y_train)
    y_estimated = param_search.predict(X_test)
    accuracy = accuracy_score(y_test, y_estimated)
    outer_cv_acc.append(accuracy)
    
    print(f'Outer fold {fold+1}, optimal hyperparameters after inner 4-fold CV: {param_search.best_params_}')
    print(f'Decision Tree model accuracy with optimal hyperparameters: {accuracy}')
    
print(f'\nAverage model accuracy: {round(np.mean(outer_cv_acc) * 100, 2)}%')<jupyter_output>Outer fold 1, optimal hyperparameters after inner 4-fold CV: {'criterion': 'gini', 'splitter': 'best'}
Decision Tree model accuracy with optimal hyperparameters: 1.0
Outer fold 2, optimal hyperparameters after inner 4-fold CV: {'criterion': 'entropy', 'splitter': 'random'}
Decision Tree model accuracy with optimal hyperparameters: 1.0
Outer fold 3, optimal hyperparameters after inner 4-fold CV: {'criterion': 'gini', 'splitter': 'random'}
Decision Tree model accuracy with optimal hyperparameters: 1.0
Outer fold 4, optimal hyperparameters after inner 4-fold CV: {'criterion': 'gini', 'splitter': 'random'}
Decision Tree model accuracy with optimal hyperparameters: 0.8666666666666667
Outer fold 5, optimal hyperparameters after inner 4-fold CV: {'criterion': 'gini', 'splitter': 'best'}
Decision Tree model accuracy with optimal hyperparameters: 1.0

Average model accuracy: 97.33%<jupyter_text>3. Multilayer Perceptron (MLP) classifier<jupyter_code>outer_cv_acc = []
param_candidates = {'alpha': np.linspace(start=0, stop=1e-1, num=500),
                    'activation': ['identity', 'logistic', 'tanh', 'relu']}
param_search = RandomizedSearchCV(estimator=MLPClassifier(max_iter=1000), param_distributions=param_candidates, scoring='accuracy', cv=inner_cv)

for fold in range(5):
    X_train = X[subsets.loc[subsets.index[fold],'Train row indices']]
    y_train = y[subsets.loc[subsets.index[fold],'Train row indices']]
    X_test = X[subsets.loc[subsets.index[fold],'Test row indices']]
    y_test = y[subsets.loc[subsets.index[fold],'Test row indices']]
    
    param_search.fit(X_train, y_train)
    y_estimated = param_search.predict(X_test)
    accuracy = accuracy_score(y_test, y_estimated)
    outer_cv_acc.append(accuracy)
     
    print(f'Outer fold {fold+1}, optimal hyperparameters after inner 4-fold CV: {param_search.best_params_}')
    print(f'MLP model accuracy with optimal hyperparameters: {accuracy}')
     
print(f'\nAverage model accuracy: {round(np.mean(outer_cv_acc) * 100, 2)}%')<jupyter_output>Outer fold 1, optimal hyperparameters after inner 4-fold CV: {'alpha': 0.03547094188376754, 'activation': 'tanh'}
MLP model accuracy with optimal hyperparameters: 1.0
Outer fold 2, optimal hyperparameters after inner 4-fold CV: {'alpha': 0.04208416833667335, 'activation': 'tanh'}
MLP model accuracy with optimal hyperparameters: 1.0
Outer fold 3, optimal hyperparameters after inner 4-fold CV: {'alpha': 0.09458917835671343, 'activation': 'relu'}
MLP model accuracy with optimal hyperparameters: 1.0
Outer fold 4, optimal hyperparameters after inner 4-fold CV: {'alpha': 0.06653306613226453, 'activation': 'tanh'}
MLP model accuracy with optimal hyperparameters: 0.9333333333333333
Outer fold 5, optimal hyperparameters after inner 4-fold CV: {'alpha': 0.025851703406813628, 'activation': 'relu'}
MLP model accuracy with optimal hyperparameters: 1.0

Average model accuracy: 98.67%<jupyter_text>4. Gaussian Naïve Bayes classifier<jupyter_code>outer_cv_acc = []
param_candidates = {'var_smoothing': np.linspace(start=1e-9, stop=1e-2, num=500)} 
param_search = RandomizedSearchCV(estimator=GaussianNB(), param_distributions=param_candidates, scoring='accuracy', cv=inner_cv)

for fold in range(5):
    X_train = X[subsets.loc[subsets.index[fold],'Train row indices']]
    y_train = y[subsets.loc[subsets.index[fold],'Train row indices']]
    X_test = X[subsets.loc[subsets.index[fold],'Test row indices']]
    y_test = y[subsets.loc[subsets.index[fold],'Test row indices']]
    
    param_search.fit(X_train, y_train)
    y_estimated = param_search.predict(X_test)
    accuracy = accuracy_score(y_test, y_estimated)
    outer_cv_acc.append(accuracy)
    
    print(f'Outer fold {fold+1}, optimal hyperparameters after inner 4-fold CV: {param_search.best_params_}')
    print(f'Gaussian Naïve Bayes model accuracy with optimal hyperparameters: {accuracy}')
    
print(f'\nAverage model accuracy: {round(np.mean(outer_cv_acc) * 100, 2)}%')<jupyter_output>Outer fold 1, optimal hyperparameters after inner 4-fold CV: {'var_smoothing': 0.0002404819378757515}
Gaussian Naïve Bayes model accuracy with optimal hyperparameters: 1.0
Outer fold 2, optimal hyperparameters after inner 4-fold CV: {'var_smoothing': 0.00048096287575150303}
Gaussian Naïve Bayes model accuracy with optimal hyperparameters: 1.0
Outer fold 3, optimal hyperparameters after inner 4-fold CV: {'var_smoothing': 0.006332665697394791}
Gaussian Naïve Bayes model accuracy with optimal hyperparameters: 0.8666666666666667
Outer fold 4, optimal hyperparameters after inner 4-fold CV: {'var_smoothing': 0.008056112418837675}
Gaussian Naïve Bayes model accuracy with optimal hyperparameters: 0.8666666666666667
Outer fold 5, optimal hyperparameters after inner 4-fold CV: {'var_smoothing': 0.007795591402805612}
Gaussian Naïve Bayes model accuracy with optimal hyperparameters: 0.9285714285714286

Average model accuracy: 93.24%<jupyter_text>5. Random Forest classifier<jupyter_code>outer_cv_acc = []
param_candidates = {'criterion': ['gini', 'entropy'],
                   'n_estimators': np.linspace(start=1, stop=500, num=500, dtype=int)} 
param_search = RandomizedSearchCV(estimator=RandomForestClassifier(), param_distributions=param_candidates, scoring='accuracy', cv=inner_cv)

for fold in range(5):
    X_train = X[subsets.loc[subsets.index[fold],'Train row indices']]
    y_train = y[subsets.loc[subsets.index[fold],'Train row indices']]
    X_test = X[subsets.loc[subsets.index[fold],'Test row indices']]
    y_test = y[subsets.loc[subsets.index[fold],'Test row indices']]
    
    param_search.fit(X_train, y_train)
    y_estimated = param_search.predict(X_test)
    accuracy = accuracy_score(y_test, y_estimated)
    outer_cv_acc.append(accuracy)
    
    print(f'Outer fold {fold+1}, optimal hyperparameters after inner 4-fold CV: {param_search.best_params_}')
    print(f'Random Forest Classifier model accuracy with optimal hyperparameters: {accuracy}')
    
print(f'\nAverage model accuracy: {round(np.mean(outer_cv_acc) * 100, 2)}%')<jupyter_output>Outer fold 1, optimal hyperparameters after inner 4-fold CV: {'n_estimators': 436, 'criterion': 'gini'}
Random Forest Classifier model accuracy with optimal hyperparameters: 1.0
Outer fold 2, optimal hyperparameters after inner 4-fold CV: {'n_estimators': 53, 'criterion': 'entropy'}
Random Forest Classifier model accuracy with optimal hyperparameters: 0.9333333333333333
Outer fold 3, optimal hyperparameters after inner 4-fold CV: {'n_estimators': 164, 'criterion': 'gini'}
Random Forest Classifier model accuracy with optimal hyperparameters: 1.0
Outer fold 4, optimal hyperparameters after inner 4-fold CV: {'n_estimators': 299, 'criterion': 'gini'}
Random Forest Classifier model accuracy with optimal hyperparameters: 0.9333333333333333
Outer fold 5, optimal hyperparameters after inner 4-fold CV: {'n_estimators': 192, 'criterion': 'entropy'}
Random Forest Classifier model accuracy with optimal hyperparameters: 1.0

Average model accuracy: 97.33%<jupyter_text>3. Seeds<NAME>, _grupa 10LF383__Sursă dataset:_ http://archive.ics.uci.edu/ml/datasets/Seeds_Articol relevant:_ <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, 'A Complete Gradient Clustering Algorithm for Features Analysis of X-ray Images', in: Information Technologies in Biomedicine, Ewa Pietka, J<NAME> (eds.), Springer-Verlag, Berlin-Heidelberg, 2010, pp. 15-24._Scurtă descriere:_ Baza de date curentă a fost alcătuită prin scanarea imaginilor unor boabe de grâu din trei specii diferite. Cuprinde 209 intrări și 7 atribute obținute prin cuantificarea parametrilor imaginilor boabelor de grâu.Parametri:- Area- Perimeter- Compactness- Length- Width- Asymmetry- Groove- Class  (imaginile au fost preluate din articolul sursă citat mai sus.)<jupyter_code>import numpy as np
import pandas as pd
from IPython.display import display, HTML
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import cross_validate,cross_val_score, train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score
from sklearn.model_selection import StratifiedKFold<jupyter_output><empty_output><jupyter_text>Split the data<jupyter_code>header = ['Area', 'Perimeter', 'Compactness', 'Length', 'Width', 'Asymmetry', 'Groove', 'Class']
data_seeds = pd.read_csv("./Datasets/seeds_dataset.txt", names=header,sep='\t')
display(HTML("<h3><b>Seeds Dataset"))
X = data_seeds.values[:, :7]
y = data_seeds.values[:, -1]
display(data_seeds)

if np.any(np.isnan(data_seeds))==False:print("Setul de date NU contine valori lipsa")
else: print("Setul de date contine valori lipsa")
    
if np.any(np.isfinite(data_seeds)==True):print("Setul de date NU contine valori infinite")
else: print("Setul de date contine valori infinite")<jupyter_output><empty_output><jupyter_text>The print function<jupyter_code>def print_function(data_set:dict):
    df_print = pd.DataFrame({"Test accuracy for each fold":data_set['test_accuracy'], 
                    "Train accuracy for each fold": data_set['train_accuracy'], 
                    "Average test accuracy %": round(data_set['test_accuracy'].mean() * 100, 4),
                    "Average train accuracy %": round(data_set['train_accuracy'].mean() * 100, 4),
                    "Test F1 score for each fold": data_set['test_f1_macro'],
                    "Train F1 score for each fold": data_set['train_f1_macro'],
                    "Average test F1 score %": round(data_set['test_f1_macro'].mean() * 100, 4),
                    "Average train F1 score %":round(data_set['train_f1_macro'].mean() * 100, 4)
                   })
    display(HTML(df_print.to_html()))<jupyter_output><empty_output><jupyter_text>Executare algoritmi cu hiperparametri hardcodați  K-Nearest Neighbors Classifier<jupyter_code># hiperparametri
knn_neighbors = 5
knn_minkowski_p = 3

# scalare date
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(X)

# implementare KNN
model = KNeighborsClassifier(n_neighbors=knn_neighbors, p=knn_minkowski_p)
model_cv_stats = cross_validate(model, X_scaled, y, cv=5, scoring=('accuracy', 'f1_macro'), return_train_score=True) 

# statistici
display(HTML(f"<h4>5-fold cross validation for {knn_neighbors}-nearest neighbors classification:</h4>"))
print_function(model_cv_stats)<jupyter_output><empty_output><jupyter_text>Decision Tree Classifier<jupyter_code># hiperparametri
dt_criterion = 'gini'# default value
dt_splitter = 'best' # intrebarea care reduce cel mai mult incertitudinea
# gini impurity-cantitatea de incertitudine pe un singur nod, cat de amestecate sunt clasificarile din frunze dupa intrebarea din nod
 
# implementare Decision Tree
model = DecisionTreeClassifier(criterion=dt_criterion, splitter=dt_splitter)
model_dc_stats = cross_validate(model, X, y, cv=5, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
# afisez cateva medii, sa vad daca sunt diferente in functie de clasa

print(data_seeds.groupby('Class')['Area'].mean()),
print(data_seeds.groupby('Class')['Perimeter'].mean()),
print(data_seeds.groupby('Class')['Asymmetry'].mean())

display(HTML(f"<h4>5-fold cross validation for Decision Trees classification:</h4>"))
print_function(model_dc_stats)<jupyter_output>Class
1    14.334429
2    18.334286
3    11.873857
Name: Area, dtype: float64
Class
1    14.294286
2    16.135714
3    13.247857
Name: Perimeter, dtype: float64
Class
1    2.667403
2    3.644800
3    4.788400
Name: Asymmetry, dtype: float64<jupyter_text>Random Forest Classifier<jupyter_code># hiperparametri
rfc_n_estimators = 150
rfc_criterion = 'gini'

# implementare Random Forest
model = RandomForestClassifier(n_estimators=rfc_n_estimators, criterion=rfc_criterion)
model_rfc_stats = cross_validate(model, X, y, cv=5, scoring=('accuracy', 'f1_macro'), return_train_score=True)

display(HTML(f"<h4>5-fold cross validation for Random Forest classification</h4>"))
print_function(model_rfc_stats)<jupyter_output><empty_output><jupyter_text>Multilayer Perceptron Classifier<jupyter_code># hiperparametri
mlp_solver = 'adam'
mlp_activation = 'logistic'
mlp_alpha=1e-3
mlp_hidden_layer_sizes = (50,50)
max_iter=10000

# implementare MLP
model = MLPClassifier(solver=mlp_solver, activation=mlp_activation, alpha=mlp_alpha, hidden_layer_sizes=mlp_hidden_layer_sizes, max_iter=max_iter, random_state=0)
model_mpc_stats = cross_validate(model, X, y, cv=5, scoring=('accuracy', 'f1_macro'), return_train_score=True)
print_function(model_mpc_stats)<jupyter_output><empty_output><jupyter_text>Gaussian Naive Bayes Classifier<jupyter_code># implementare GNB
model = GaussianNB()
model_gnb_stats = cross_validate(model, X, y, cv=5, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>5-fold cross validation for Gaussian NB classification</h4>"))
print_function(model_gnb_stats)<jupyter_output><empty_output><jupyter_text>Optimizarea hiperparametrilor  Split the data<jupyter_code>header = ['Area', 'Perimeter', 'Compactness', 'Length', 'Width', 'Asymmetry', 'Groove', 'Class']
data_seeds = pd.read_csv("./Datasets/seeds_dataset.txt", names=header,sep='\t')
X = data_seeds.values[:, :7]
y = data_seeds.values[:, -1]<jupyter_output><empty_output><jupyter_text>K-Nearest Neighbors Classifier<jupyter_code>pipe = Pipeline([('scaler', MinMaxScaler()), ('knn', KNeighborsClassifier())])
parameter_grid = {'knn__n_neighbors': list(range(1, 10)), 'knn__p': list(range(1, 5))}
strat_k_fold = StratifiedKFold(n_splits=5, shuffle=True)
grid_search =GridSearchCV(pipe, param_grid=parameter_grid, scoring='accuracy', cv=4, return_train_score=True)

scores = cross_val_score(grid_search, X, y, cv=strat_k_fold )
print("Scorurile rezultate in urma 5-fold cross validation:",scores)
print("Media scorurilor:",scores.mean())


grid_search.fit(X, y)
print("Cel mai bun set de parametri:",grid_search.best_params_)

grid_search = pd.DataFrame(grid_search.cv_results_)
display(HTML(grid_search.to_html()))<jupyter_output>Scorurile rezultate in urma 5-fold cross validation: [0.9047619  0.92857143 0.88095238 0.92857143 1.        ]
Media scorurilor: 0.9285714285714286
Cel mai bun set de parametri: {'knn__n_neighbors': 9, 'knn__p': 3}<jupyter_text>Decision Tree Classifier<jupyter_code>pipe = Pipeline([('dtc', DecisionTreeClassifier())])

parameter_grid = { 'dtc__criterion': ['gini', 'entropy'],'dtc__splitter': ['best', 'random'] }
grid_search =GridSearchCV(pipe, param_grid=parameter_grid, scoring='accuracy', cv=4)
strat_k_fold = StratifiedKFold(n_splits=5, shuffle=True)
scores = cross_val_score(grid_search, X, y, cv=strat_k_fold )

print("Scorurile rezultate in urma 5-fold cross validation:",scores)
print("Media scorurilor:",scores.mean())
grid_search.fit(X, y)
print("Cel mai bun set de parametri:",grid_search.best_params_)
grid_search = pd.DataFrame(grid_search.cv_results_)
display(HTML(grid_search.to_html()))<jupyter_output>Scorurile rezultate in urma 5-fold cross validation: [0.83333333 0.85714286 0.9047619  0.88095238 0.95238095]
Media scorurilor: 0.8857142857142858
Cel mai bun set de parametri: {'dtc__criterion': 'entropy', 'dtc__splitter': 'best'}<jupyter_text>Random Forest Classifier<jupyter_code>pipe = Pipeline([('rfc', RandomForestClassifier())])

parameter_grid = { 'rfc__criterion': ['gini', 'entropy'],'rfc__n_estimators': np.linspace(start=1, stop=150, num=25, dtype=int)}
grid_search =GridSearchCV(pipe, param_grid=parameter_grid, scoring='accuracy', cv=4)
strat_k_fold = StratifiedKFold(n_splits=5, shuffle=True)
scores = cross_val_score(grid_search, X, y, cv=strat_k_fold )

print("Scorurile rezultate in urma 5-fold cross validation:",scores)
print("Media scorurilor:",scores.mean())
grid_search.fit(X, y)
print("Cel mai bun set de parametri:",grid_search.best_params_)
grid_search = pd.DataFrame(grid_search.cv_results_)
display(HTML(grid_search.to_html()))<jupyter_output>Scorurile rezultate in urma 5-fold cross validation: [0.92857143 0.92857143 0.85714286 0.88095238 0.97619048]
Media scorurilor: 0.9142857142857143
Cel mai bun set de parametri: {'rfc__criterion': 'gini', 'rfc__n_estimators': 32}<jupyter_text>Gaussian Naive Bayes Classifier<jupyter_code>pipe = Pipeline([('gnb', GaussianNB())])

parameter_grid = { 'gnb__var_smoothing': np.linspace(start=1e-9, stop=1e-2, num=100)}
grid_search =GridSearchCV(pipe, param_grid=parameter_grid, scoring='accuracy', cv=4)
strat_k_fold = StratifiedKFold(n_splits=5, shuffle=True)
scores = cross_val_score(grid_search, X, y, cv=strat_k_fold )

print("Scorurile rezultate in urma 5-fold cross validation:",scores)
print("Media scorurilor:",scores.mean())
grid_search.fit(X, y)
print("Cel mai bun set de parametri:",grid_search.best_params_)
grid_search = pd.DataFrame(grid_search.cv_results_)
display(HTML(grid_search.to_html()))<jupyter_output>Scorurile rezultate in urma 5-fold cross validation: [0.92857143 0.88095238 0.97619048 0.95238095 0.78571429]
Media scorurilor: 0.9047619047619048
Cel mai bun set de parametri: {'gnb__var_smoothing': 0.0028282835454545457}<jupyter_text>Multilayer Perceptron Classifier<jupyter_code>pipe = Pipeline([('mlp', MLPClassifier(max_iter= 10000))])

parameter_grid = { 'mlp__alpha': np.linspace(start=0, stop=1e-1, num=50),
                  'mlp__activation': ['identity', 'logistic', 'tanh', 'relu']
                 }
grid_search =GridSearchCV(pipe, param_grid=parameter_grid, scoring='accuracy', cv=4)
strat_k_fold = StratifiedKFold(n_splits=5, shuffle=True)
scores = cross_val_score(grid_search, X, y, cv=strat_k_fold )

print("Scorurile rezultate in urma 5-fold cross validation:",scores)
print("Media scorurilor:",scores.mean())
grid_search.fit(X, y)
print("Cel mai bun set de parametri:",grid_search.best_params_)
grid_search = pd.DataFrame(grid_search.cv_results_)
display(HTML(grid_search.to_html()))<jupyter_output>Scorurile rezultate in urma 5-fold cross validation: [0.88095238 0.92857143 0.95238095 0.95238095 0.88095238]
Media scorurilor: 0.919047619047619
Cel mai bun set de parametri: {'mlp__activation': 'identity', 'mlp__alpha': 0.030612244897959186}<jupyter_text>4. DermatologyTătaru Dragoș-Cătălin, _grupa 10LF383__Sursă dataset_: http://archive.ics.uci.edu/ml/datasets/Dermatology_Articol relevant_: <NAME>, <NAME>, and <NAME>, "Learning Differential Diagnosis of Eryhemato-Squamous Diseases using Voting Feature Intervals", Artificial Intelligence in Medicine_Scurtă descriere_: Baza de date cuprinde informații privitoare la condițiile necesare pentru diagnosticarea diferențiată pentru bolile de piele. Cuprinde 366 intrări și 33 atribute ce pot lua valori între 0 și 3, fiecare dintre acestea indicând severitatea unei condiții medicale. Sunt incluse 6 clase codificate cu cifre între 1 și 6 după cum urmează:_Cuprinde clasele_:- psoriasis- seboreic dermatitis- lichen planus- pityriasis rosea- cronic dermatitis- pityriasis rubra pilaris<jupyter_code>import numpy as np
import pandas as pd
from IPython.display import display, HTML
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import cross_validate,cross_val_score, train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score
from sklearn.model_selection import StratifiedKFold
from sklearn.impute import SimpleImputer<jupyter_output><empty_output><jupyter_text>Load the data<jupyter_code>header =['erythema','scaling','definite borders','itching','koebner phenomenon','polygonal papules',
'follicular papules','oral mucosal involvement','knee and elbow involvement','scalp involvement','family history','melanin incontinence',
'eosinophils in the infiltrate','PNL infiltrate','fibrosis of the papillary dermis','exocytosis','acanthosis','hyperkeratosis',
'parakeratosis','clubbing of the rete ridges','elongation of the rete ridges','thinning of the suprapapillary epidermis','spongiform pustule','munro microabcess','focal hypergranulosis',
'disappearance of the granular layer','vacuolisation and damage of basal layer','spongiosis','saw-tooth appearance of retes','follicular horn plug','perifollicular parakeratosis','inflammatory monoluclear inflitrate',
'band-like infiltrate','Age','Class']

print(len(header))
# Alcatuim o lista de stringuri ce ar putea reprezenta missing data in data setul nostru
missingValues = ["n/a", "na", "--"," ","?"]
data_spam = pd.read_csv("./Datasets/dermatology.data",names=header,sep=',',na_values=missingValues)<jupyter_output>35<jupyter_text>Se observa existenta unor date lipsa, fiecare coloana ar trebui sa aibe 4601 intrari, acest lucru reiese din informatiile despre setul de date:<jupyter_code>display(data_spam.info())
# Afisam numarul de missind data in functie de coloana
print (data_spam.isnull().sum())
# Afisam numarul de missind data in total
print (data_spam.isnull().sum().sum())
display(HTML("<h3><b>Dermatology Dataset"))
display(HTML(data_spam[5:27].to_html()))<jupyter_output><empty_output><jupyter_text>Inlocuim datele lipsa:<jupyter_code>imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
print(imputer)
imputer.fit(data_spam)
data_spam = pd.DataFrame(data=imputer.transform(data_spam), columns=header)

display(HTML("<i>Dataset extract with values imputed:</i>"))
display(HTML(data_spam[5:20].to_html()))

X = data_spam.values[:, :34]
y = data_spam.values[:, -1]

display(X)
display(np.unique(y))<jupyter_output>SimpleImputer(add_indicator=False, copy=True, fill_value=None,
              missing_values=nan, strategy='mean', verbose=0)<jupyter_text>The print Function<jupyter_code>def print_function(data_set:dict):
    df_print = pd.DataFrame({"Test accuracy for each fold":data_set['test_accuracy'], 
                    "Train accuracy for each fold": data_set['train_accuracy'], 
                    "Average test accuracy %": round(data_set['test_accuracy'].mean() * 100, 4),
                    "Average train accuracy %": round(data_set['train_accuracy'].mean() * 100, 4),
                    "Test F1 score for each fold": data_set['test_f1_macro'],
                    "Train F1 score for each fold": data_set['train_f1_macro'],
                    "Average test F1 score %": round(data_set['test_f1_macro'].mean() * 100, 4),
                    "Average train F1 score %":round(data_set['train_f1_macro'].mean() * 100, 4)
                   })
    display(HTML(df_print.to_html()))<jupyter_output><empty_output><jupyter_text>Executare algoritmi cu hiperparametri hardcodați:  K-Nearest Neighbors Classifier<jupyter_code># hiperparametri
knn_neighbors = 5
knn_minkowski_p = 3

# scalare date
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(X)

# implementare KNN
model = KNeighborsClassifier(n_neighbors=knn_neighbors, p=knn_minkowski_p)
model_cv_stats = cross_validate(model, X_scaled, y, cv=5, scoring=('accuracy', 'f1_macro'), return_train_score=True) 

# statistici
display(HTML(f"<h4>5-fold cross validation for {knn_neighbors}-nearest neighbors classification:</h4>"))
print_function(model_cv_stats)<jupyter_output><empty_output><jupyter_text>Decision Tree Classifier<jupyter_code># hiperparametri
dt_criterion = 'gini'#default value
dt_splitter = 'best' #intrebarea care reduce cel mai mult incertitudinea
#gini imputity-cantitatea de incertitudine pe un singur nod, cat de amestecate sunt clasificarile din frunze dupa intrebarea din nod
 
# implementare Decision Tree
model = DecisionTreeClassifier(criterion=dt_criterion, splitter=dt_splitter)
model_dc_stats = cross_validate(model, X, y, cv=5, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>5-fold cross validation for Decision Trees classification:</h4>"))
print_function(model_dc_stats)<jupyter_output><empty_output><jupyter_text>Random Forest Classifier<jupyter_code># hiperparametri
rfc_n_estimators = 150
rfc_criterion = 'gini'

# implementare Random Forest
model = RandomForestClassifier(n_estimators=rfc_n_estimators, criterion=rfc_criterion)
model_rfc_stats = cross_validate(model, X, y, cv=5, scoring=('accuracy', 'f1_macro'), return_train_score=True)

display(HTML(f"<h4>5-fold cross validation for Random Forest classification</h4>"))
print_function(model_rfc_stats)<jupyter_output><empty_output><jupyter_text>Multilayer Perceptron Classifier<jupyter_code># hiperparametri
mlp_solver = 'adam'
mlp_activation = 'logistic'
mlp_alpha=1e-3
mlp_hidden_layer_sizes = (50,50)
max_iter=1000

# implementare MLP
model = MLPClassifier(solver=mlp_solver, activation=mlp_activation, alpha=mlp_alpha, hidden_layer_sizes=mlp_hidden_layer_sizes, max_iter=max_iter, random_state=0)
model_mpc_stats = cross_validate(model, X, y, cv=5, scoring=('accuracy', 'f1_macro'), return_train_score=True)
print_function(model_mpc_stats)<jupyter_output><empty_output><jupyter_text>Gaussian Naive Bayes Classifier<jupyter_code># implementare GNB
model = GaussianNB()
model_gnb_stats = cross_validate(model, X, y, cv=5, scoring=('accuracy', 'f1_macro'), return_train_score=True)

# statistici
display(HTML(f"<h4>5-fold cross validation for Gaussian NB classification</h4>"))
print_function(model_gnb_stats)<jupyter_output><empty_output><jupyter_text>Optimizarea hiperparametrilor  K-Nearest Neighbors Classifier<jupyter_code>pipe = Pipeline([('scaler', MinMaxScaler()), ('knn', KNeighborsClassifier())])
parameter_grid = {'knn__n_neighbors': list(range(1, 10)), 'knn__p': list(range(1, 5))}
strat_k_fold = StratifiedKFold(n_splits=5, shuffle=True)
grid_search =GridSearchCV(pipe, param_grid=parameter_grid, scoring='accuracy', cv=4, return_train_score=True)

scores = cross_val_score(grid_search, X, y, cv=strat_k_fold)
print("Scorurile rezultate in urma 5-fold cross validation:",scores)
print("Media scorurilor:",scores.mean())

grid_search.fit(X, y)
print("Cel mai bun set de parametri:",grid_search.best_params_)

grid_search = pd.DataFrame(grid_search.cv_results_)
display(HTML(grid_search.to_html()))<jupyter_output>Scorurile rezultate in urma 5-fold cross validation: [0.98648649 0.95890411 0.95890411 1.         0.95890411]
Media scorurilor: 0.972639763050722
Cel mai bun set de parametri: {'knn__n_neighbors': 6, 'knn__p': 1}<jupyter_text>Decision Tree Classifier<jupyter_code>pipe = Pipeline([('dtc', DecisionTreeClassifier())])

parameter_grid = { 'dtc__criterion': ['gini', 'entropy'],'dtc__splitter': ['best', 'random'] }
grid_search =GridSearchCV(pipe, param_grid=parameter_grid, scoring='accuracy', cv=4)
strat_k_fold = StratifiedKFold(n_splits=5, shuffle=True)
scores = cross_val_score(grid_search, X, y, cv=strat_k_fold)

print("Scorurile rezultate in urma 5-fold cross validation:",scores)
print("Media scorurilor:",scores.mean())
grid_search.fit(X, y)
print("Cel mai bun set de parametrii:",grid_search.best_params_)
grid_search = pd.DataFrame(grid_search.cv_results_)
display(HTML(grid_search.to_html()))<jupyter_output>Scorurile rezultate in urma 5-fold cross validation: [0.97297297 0.93150685 0.94520548 0.95890411 0.93150685]
Media scorurilor: 0.9480192521288411
Cel mai bun set de parametrii: {'dtc__criterion': 'entropy', 'dtc__splitter': 'random'}<jupyter_text>Random Forest Classifier<jupyter_code>pipe = Pipeline([('rfc', RandomForestClassifier())])

parameter_grid = { 'rfc__criterion': ['gini', 'entropy'],'rfc__n_estimators': np.linspace(start=1, stop=150, num=25, dtype=int)}
grid_search =GridSearchCV(pipe, param_grid=parameter_grid, scoring='accuracy', cv=4)
strat_k_fold = StratifiedKFold(n_splits=5, shuffle=True)
scores = cross_val_score(grid_search, X, y, cv=strat_k_fold)

print("Scorurile rezultate in urma 5-fold cross validation:",scores)
print("Media scorurilor:",scores.mean())
grid_search.fit(X, y)
print("Cel mai bun set de parametri:",grid_search.best_params_)
grid_search = pd.DataFrame(grid_search.cv_results_)
display(HTML(grid_search.to_html()))<jupyter_output>Scorurile rezultate in urma 5-fold cross validation: [0.98648649 0.98630137 0.95890411 0.94520548 0.98630137]
Media scorurilor: 0.9726397630507219
Cel mai bun set de parametri: {'rfc__criterion': 'gini', 'rfc__n_estimators': 69}<jupyter_text>Gaussian Naive Bayes Classifier<jupyter_code>pipe = Pipeline([('gnb', GaussianNB())])

parameter_grid = { 'gnb__var_smoothing': np.linspace(start=1e-9, stop=1e-2, num=100)}
grid_search =GridSearchCV(pipe, param_grid=parameter_grid, scoring='accuracy', cv=4)
strat_k_fold = StratifiedKFold(n_splits=5, shuffle=True)
scores = cross_val_score(grid_search, X, y, cv=strat_k_fold)

print("Scorurile rezultate in urma 5-fold cross validation:",scores)
print("Media scorurilor:",scores.mean())
grid_search.fit(X, y)
print("Cel mai bun set de parametri:",grid_search.best_params_)
grid_search = pd.DataFrame(grid_search.cv_results_)
display(HTML(grid_search.to_html()))<jupyter_output>Scorurile rezultate in urma 5-fold cross validation: [0.95945946 0.93150685 0.98630137 0.94520548 1.        ]
Media scorurilor: 0.9644946316179193
Cel mai bun set de parametri: {'gnb__var_smoothing': 0.0004040413636363637}<jupyter_text>Multilayer Perceptron Classifier<jupyter_code>pipe = Pipeline([('mlp', MLPClassifier(max_iter= 1000))])

parameter_grid = { 'mlp__alpha': np.linspace(start=0, stop=1e-1, num=50),
                  'mlp__activation': ['identity', 'logistic', 'tanh', 'relu']
                 }

grid_search =GridSearchCV(pipe, param_grid=parameter_grid, scoring='accuracy', cv=4)
strat_k_fold = StratifiedKFold(n_splits=5, shuffle=True)
scores = cross_val_score(grid_search, X, y, cv=strat_k_fold )

print("Scorurile rezultate in urma 5-fold cross validation:",scores)
print("Media scorurilor:",scores.mean())
grid_search.fit(X, y)
print("Cel mai bun set de parametri:",grid_search.best_params_)
grid_search = pd.DataFrame(grid_search.cv_results_)
display(HTML(grid_search.to_html()))<jupyter_output>Scorurile rezultate in urma 5-fold cross validation: [0.98648649 0.98630137 0.97260274 0.95890411 0.95890411]
Media scorurilor: 0.9726397630507219
Cel mai bun set de parametri: {'mlp__activation': 'identity', 'mlp__alpha': 0.010204081632653062}
{'fraction_non_alphanumeric': 0.11219504951965195, 'fraction_numerical': 0.036007543023445734, 'mean_word_length': 6.71454753722795, 'pattern_counts': {'":': 16, '<': 283, '<?xml version=': 0, '>': 288, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 9, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '56', 'dir': 'jupyter-structured-clean-dedup', 'id': '386172', 'license': 'MIT', 'n_tokens_mistral': 25257, 'n_tokens_neox': 23191, 'n_words': 5755, 'path': 'Laborator6/IDS_Sirbu_Tataru.ipynb', 'repo_name': 'MateiSirbu/IDS_Sirbu_Tataru', 'size': '67450'}