<jupyter_start><jupyter_text>JEST internal workshop &8594; Intro to ML (Crash Course)  This is an short version (without explanations) of the workshop, to access the full version of the document click [here](https://github.com/tamagusko/workshop-intro-ml/blob/main/introML.ipynb). Topics1. Data Collection2. Exploratory data analysis      2.1 Data visualization  3. Preprocessing      3.1 Missing Data, Encoding, Standardizing, Feature Engineering      3.2 Split data in train/test4. Model      4.1  Decision Tree      4.2. Random Forest      4.3. KNN (k-nearest neighbors)      4.4. XGBoost (eXtreme Gradient Boosting)      4.5. Neural Network (Multilayer perceptron)   5. Comparison of models (Metrics)  6. Improving the model7. Saving and loading models8. How to identify overfitting?9. Automating tasks (Choosing the best model)--- --- Practical example: Predicting Pulsar StarMore details in: [Predicting Pulsar Star](https://www.kaggle.com/colearninglounge/predicting-pulsar-starintermediate)  Proposed problemClassify stars between Pulsar (1) and non-Pulsar (0).Note: The data were collected and previously treated.  1. Data Collection Dataset - [pulsar_stars.csv](data/pulsar_stars.csv)**rows**: 16,259 spurious examples (RFI/noise), and 1,639 real pulsar examples. Total of 17,898 rows.  **columns**: 9 features, namely:    1. Mean of the integrated profile.  2. Standard deviation of the integrated profile.  3. Excess kurtosis of the integrated profile.  4. Skewness of the integrated profile.  5. Mean of the DM-SNR curve.  6. Standard deviation of the DM-SNR curve.  7. Excess kurtosis of the DM-SNR curve.  8. Skewness of the DM-SNR curve.  9. Class (1=Pulsar, 0=non-Pulsar)So, the features are columns 1 to 8, and the label is the column 9.<jupyter_code># Data manipulation
import pandas as pd 
# Data visualization
import seaborn as sns 
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>More details in [A Complete Pandas Guide](https://towardsdatascience.com/a-complete-pandas-guide-2dc53c77a002)<jupyter_code>url = 'https://raw.githubusercontent.com/spingegod/ProjetoTI_part1/master/ProjetoTI/pulsar_stars.csv'
df = pd.read_csv(url)
df<jupyter_output><empty_output><jupyter_text>2. Exploratory data analysis 2.1 Data visualization<jupyter_code>df.shape
df.tail()<jupyter_output><empty_output><jupyter_text>View data type<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 17898 entries, 0 to 17897
Data columns (total 9 columns):
 #   Column                                         Non-Null Count  Dtype  
---  ------                                         --------------  -----  
 0    Mean of the integrated profile                17898 non-null  float64
 1    Standard deviation of the integrated profile  17898 non-null  float64
 2    Excess kurtosis of the integrated profile     17898 non-null  float64
 3    Skewness of the integrated profile            17898 non-null  float64
 4    Mean of the DM-SNR curve                      17898 non-null  float64
 5    Standard deviation of the DM-SNR curve        17898 non-null  float64
 6    Excess kurtosis of the DM-SNR curve           17898 non-null  float64
 7    Skewness of the DM-SNR curve                  17898 non-null  float64
 8   target_class                                   17898 non-null  int64  
dtypes: float64(8), int64(1)
memory usage: 1.2 MB<jupyter_text>Data overview<jupyter_code>df.describe().T<jupyter_output><empty_output><jupyter_text>NA values?<jupyter_code>df.isna().any()<jupyter_output><empty_output><jupyter_text>Duplicate rows?<jupyter_code>print("Duplicate rows: ", df.duplicated().sum())<jupyter_output>Duplicate rows:  0<jupyter_text>Target (label) class<jupyter_code>df.target_class.value_counts()
df.target_class.value_counts().plot.pie(autopct='%.2f%%')<jupyter_output><empty_output><jupyter_text>Correlation matrix<jupyter_code>correlation = df.corr()
correlation.style.background_gradient(cmap='coolwarm').set_precision(2)<jupyter_output><empty_output><jupyter_text>Data visualization ()<jupyter_code>%%javascript
// Avoid scroll bar on very large images
IPython.OutputArea.prototype._should_scroll = function(lines) {
    return false;
}
# Plots the histogram of all columns (excluding target_class).
for column in df.columns[:-1]:
    plt.figure()
    sns.histplot(x=df[column], hue=df.target_class)<jupyter_output><empty_output><jupyter_text>3 Preprocessing 3.1 Missing Data, Encoding, Standardizing, Feature EngineeringLook in: [freecodecamp: Preprocessing](https://www.youtube.com/watch?v=0B5eIE_1vpU&t=2076s) (not necessary in this dataset)  3.2 Split data in train/test<jupyter_code>X = df.drop(['target_class'], axis=1)
y = df.target_class
# Prepare data to train models
from sklearn.model_selection import train_test_split

# 50/50 (common: 70/30 or 80/20)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, train_size=0.5, random_state=42, stratify=y)

X_train.shape, X_test.shape, y_train.shape, y_test.shape<jupyter_output><empty_output><jupyter_text>4. Models  How to use Scikit-Learn?By [<NAME>](https://www.linkedin.com/in/andre-ye-501746150/), in KDnuggets:[The Ultimate Scikit-Learn Machine Learning Cheatsheet](https://www.kdnuggets.com/2021/01/ultimate-scikit-learn-machine-learning-cheatsheet.html).```python Importfrom sklearn.BRANCH import MODEL_NAMEfrom sklearn.metrics import METRIC_NAME BuildMODEL = MODEL_NAME() TrainMODEL.fit(X_train, y_train) Predicty_pred_MODEL = MODEL.predict(X_test) Eval METRIC = METRIC_NAME((y_test, y_pred_MODEL)) ```<jupyter_code># Metrics
from sklearn.metrics import accuracy_score
from sklearn.metrics import precision_score
from sklearn.metrics import matthews_corrcoef
from sklearn.metrics import confusion_matrix
from sklearn.metrics import roc_curve
from sklearn.metrics import auc
from sklearn.metrics import confusion_matrix<jupyter_output><empty_output><jupyter_text>*Confusion Matrix:*\begin{pmatrix} True Negative & False Positive\\False Negative & True Positive\end{pmatrix}  \begin{equation}Accuracy = \frac{TP+TN}{(TP+TN+FP+FN)} \\Precision = \frac{TP}{(TP+FP)} \\MCC = \frac{TP*TN-FP*FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}} \\\end{equation}*where:**TP = True Positive  TN = True Negative  FP = False Positive  FN = False Negative*<jupyter_code># Simple functions to run metrics and print results

def evaluationModels(y_test, y_pred):
    accuracy = accuracy_score(y_test, y_pred)
    precision = precision_score(y_test, y_pred)
    # Best metric for binary (0/1) classification 
    mcc = matthews_corrcoef(y_test, y_pred)
    fp = confusion_matrix(y_test, y_pred)[0][1]
    return accuracy, precision, mcc, fp


def printMetrics(model, accuracy, precision, mcc, fp):
    modelName = model.__class__.__name__
    bar = "-" * 25
    print(f"{bar}\n {modelName}\n{bar}")
    print(f'- Accuracy: {accuracy:0.3f}')
    print(f'- Precision: {precision:0.3f}')
    print(f'- MCC: {mcc:0.3f}')
    print(f'- False Positives: {fp}')<jupyter_output><empty_output><jupyter_text>4.1 Decision TreeSee the parameters in: [sklearn.tree.DecisionTreeClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html)More details in: [Decision Trees](https://scikit-learn.org/stable/modules/tree.html)How to tune the parameters: [InDepth: Parameter tuning for Decision Tree](https://medium.com/@mohtedibf/indepth-parameter-tuning-for-decision-tree-6753118a03c3)<jupyter_code>from sklearn.tree import DecisionTreeClassifier

DT = DecisionTreeClassifier(max_depth=3, random_state=42)
# Training 
DT.fit(X_train, y_train)
# Training parameters
DT.get_params()
# Predict
y_pred_DT = DT.predict(X_test)
# Evaluation
accuracy, precision, mcc, fp = metricsDT = evaluationModels(y_test, y_pred_DT)
printMetrics(DT, accuracy, precision, mcc, fp)<jupyter_output>-------------------------
 DecisionTreeClassifier
-------------------------
- Accuracy: 0.978
- Precision: 0.893
- MCC: 0.868
- False Positives: 85<jupyter_text>MCC varies beetwen -1 (perfect negative correlation) and 1 (perfect positive correlation).OBS: 0 (random flip)<jupyter_code>confusion_matrix(y_test, y_pred_DT) # (TN FP \\ FN & TP)<jupyter_output><empty_output><jupyter_text>Trick (worst possible threshold)<jupyter_code># Bad practice (don't do it)
import warnings
warnings.filterwarnings('ignore')
threshold = [0] * len(X_test)  # All results equal to 0

accuracy, precision, mcc, fp = metricsThreshold = evaluationModels(y_test, threshold)
printMetrics(threshold, accuracy, precision, mcc, fp)  # The name is not defined
# Feature Importance

featureImportanceDT = pd.DataFrame({'Feature': list(X_train.columns),
                                  'Importance': DT.feature_importances_}).\
    sort_values('Importance', ascending=False)

featureImportanceDT<jupyter_output><empty_output><jupyter_text>4.2 Random ForestSee the parameters in: [sklearn.ensemble.RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)More details in: [Random Forests](https://scikit-learn.org/stable/modules/ensemble.htmlrandom-forests)How to tune the parameters: [Random Forest: Hyperparameters and how to fine-tune them](https://towardsdatascience.com/random-forest-hyperparameters-and-how-to-fine-tune-them-17aee785ee0d)<jupyter_code>from sklearn.ensemble import RandomForestClassifier

# Building (need to tune for better results)
RF = RandomForestClassifier(max_depth=4, random_state=42)

# Training 
RF.fit(X_train, y_train)
# Training parameters
RF.get_params()
# Predict
y_pred_RF = RF.predict(X_test)
# Evaluating
accuracy, precision, mcc, fp = metricsRF = evaluationModels(y_test, y_pred_RF)
printMetrics(RF, accuracy, precision, mcc, fp)
# Feature Importance
featureImportanceRF = pd.DataFrame({'feature': list(X_train.columns),
                                  'importance': RF.feature_importances_}).\
    sort_values('importance', ascending=False)

featureImportanceRF<jupyter_output><empty_output><jupyter_text>4.3 K Nearest Neighbors (KNN)See the parameters in: [sklearn.neighbors.KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)More details in: [Nearest Neighbors](https://scikit-learn.org/stable/modules/neighbors.html)<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

# Building
KNN = KNeighborsClassifier(metric='euclidean')

# Training 
KNN.fit(X_train, y_train)
KNN.get_params()
# Predict
y_pred_KNN = KNN.predict(X_test)
# Evaluating
accuracy, precision, mcc, fp = metricsKNN = evaluationModels(y_test, y_pred_KNN)
printMetrics(KNN, accuracy, precision, mcc, fp)<jupyter_output>-------------------------
 KNeighborsClassifier
-------------------------
- Accuracy: 0.972
- Precision: 0.892
- MCC: 0.827
- False Positives: 79<jupyter_text>4.4 XGBoost (Ensemble model)See the parameters in: [XGBoost Parameters](https://xgboost.readthedocs.io/en/latest/parameter.html)More details in: [Getting Started with XGBoost in scikit-learn](https://towardsdatascience.com/getting-started-with-xgboost-in-scikit-learn-f69f5f470a97)* XGBoost is regularized (avoid overfit).* XGBoost Have hyperparameters to handle imbalanced data and null values.* Combine some of the best practices and algorithms in a single model.* Fast and easy to use.* It is my go-to algorithm for dealing with structured data.<jupyter_code>import xgboost as xgb

# Building (need to tune for better results)
XGB = xgb.XGBClassifier(max_depth=4, random_state=42)

# Training 
XGB.fit(X_train, y_train)
# Training parameters
XGB.get_params()
# Predict
y_pred_XGB = XGB.predict(X_test)
# Evaluating
accuracy, precision, mcc, fp = metricsXGB = evaluationModels(y_test, y_pred_XGB)
printMetrics(XGB, accuracy, precision, mcc, fp)
# Feature Importance
featureImportanceXGB = pd.DataFrame({'feature': list(X_train.columns),
                                  'importance': XGB.feature_importances_}).\
    sort_values('importance', ascending=False)

featureImportanceXGB<jupyter_output><empty_output><jupyter_text>4.5 Neural Network: Multi-Layer Perceptron (MLP) - Most simple NN modelSee the parameters in: [sklearn.neural_network.MLPClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html)More details in: [Multi-layer Perceptron (MLP)](https://scikit-learn.org/stable/modules/neural_networks_supervised.html)<jupyter_code>from sklearn.neural_network import MLPClassifier

MLP = MLPClassifier(alpha=1, max_iter=1000)

# Training 
MLP.fit(X_train, y_train)
# Training parameters
MLP.get_params()
# Predict
y_pred_MLP = MLP.predict(X_test)
# Evaluating
accuracy, precision, mcc, fp = metricsMLP = evaluationModels(y_test, y_pred_MLP)
printMetrics(MLP, accuracy, precision, mcc, fp)<jupyter_output>-------------------------
 MLPClassifier
-------------------------
- Accuracy: 0.977
- Precision: 0.937
- MCC: 0.855
- False Positives: 44<jupyter_text>5. Comparison of models<jupyter_code>results_accuracy = {
    'Threshold': metricsThreshold[0],
    'DT': metricsDT[0],
    'RF': metricsRF[0],
    'KNN': metricsKNN[0],
    'MLP': metricsMLP[0],
    'XGB': metricsXGB[0],
    
}
results_precision = {
    'Threshold': metricsThreshold[1],
    'DT': metricsDT[1],
    'RF': metricsRF[1],
    'KNN': metricsKNN[1],
    'MLP': metricsMLP[1],
    'XGB': metricsXGB[1],
}
results_mcc = {
    'Threshold': metricsThreshold[2],
    'DT': metricsDT[2],
    'RF': metricsRF[2],
    'KNN': metricsKNN[2],
    'MLP': metricsMLP[2],
    'XGB': metricsXGB[2],
}

results_fp = {
    'Threshold': metricsThreshold[3],
    'DT': metricsDT[3],
    'RF': metricsRF[3],
    'KNN': metricsKNN[3],
    'MLP': metricsMLP[3],
    'XGB': metricsXGB[3],
}

acc_df = pd.DataFrame.from_dict(results_accuracy, orient='index', columns=['Accuracy'])
precision_df = pd.DataFrame.from_dict(results_precision, orient='index', columns=['Precision'])
mcc_df = pd.DataFrame.from_dict(results_mcc, orient='index', columns=['MCC'])
fp_df = pd.DataFrame.from_dict(results_fp, orient='index', columns=['FP'])
results = pd.concat([acc_df, precision_df, mcc_df, fp_df], axis=1)
results<jupyter_output><empty_output><jupyter_text>6. Improving the modelWhat can we do better? - Be careful in this improvement phase that we often make the model worse.<jupyter_code>from sklearn.model_selection import GridSearchCV
# Try possible combinations of the parameters and find the best ones.

from sklearn.model_selection import RandomizedSearchCV
# Random search for the best parameters (slow and computationally intensive).

from sklearn.ensemble import StackingClassifier
# Combine algorithms and make a hybrid model.<jupyter_output><empty_output><jupyter_text>More details in [Machine Learning: GridSearchCV & RandomizedSearchCV](https://towardsdatascience.com/machine-learning-gridsearchcv-randomizedsearchcv-d36b89231b10) and [Stacking made easy with Sklearn](https://towardsdatascience.com/stacking-made-easy-with-sklearn-e27a0793c92b).  7. Save Model (doesn't work in the colab env) ```pythonimport pickleimport datetimecurrent_date: str = datetime.date.today().strftime("%Y-%m-%d")     save modelsfilename_DT = f'models/{current_date}_Pulsar_DT.mlmodel'pickle.dump(DT, open(filename_DT, 'wb'))filename_RF = f'models/{current_date}_Pulsar_RF.mlmodel'pickle.dump(RF, open(filename_RF, 'wb'))filename_KNN = f'models/{current_date}_Pulsar_KNN.mlmodel'pickle.dump(KNN, open(filename_KNN, 'wb'))filename_MLP = f'models/{current_date}_Pulsar_MLP.mlmodel'pickle.dump(MLP, open(filename_MLP, 'wb'))filename_XGB = f'models/{current_date}_Pulsar_XGB.mlmodel'pickle.dump(XGB, open(filename_XGB, 'wb'))```  7.1 Load model (doesn't work in the colab env) ```pythonloaded_model = pickle.load(open(filename_RF, 'rb')) In real life you will use data from the production pipeline.result = loaded_model.score(X_test, y_test)print(f'Random Forest: {result:0.3f}')``` Random Forest: 0.979  8. How to identify overfitting?<jupyter_code>maxDepths = 15

depths = [i for i in range(1, maxDepths+1)]  # +1 to include run 20
train_results, test_results = list(), list()

for i in depths:
    MODEL = RandomForestClassifier(max_depth=i, random_state=42)
    MODEL.fit(X_train, y_train)
    method = MODEL.__class__.__name__
    # train
    y_pred_train = MODEL.predict(X_train)
    train_accuracy = accuracy_score(y_train, y_pred_train)
    train_results.append(train_accuracy)
    # test
    y_pred_test = MODEL.predict(X_test)
    test_accuracy = accuracy_score(y_test, y_pred_test)
    test_results.append(test_accuracy)
    
# Results train vs test
plt.figure(figsize=(14, 5))
plt.plot(depths, train_results, '-o', label='Train')
plt.plot(depths, test_results, '-o', label='Test')
plt.legend(loc="best")
plt.title(f"{method} accuracy for training and testing a maximum of {maxDepths} depths")
plt.show()<jupyter_output><empty_output><jupyter_text>9. Automating boring tasks (Choosing the best model)<jupyter_code># importing more models

from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier
from sklearn.naive_bayes import GaussianNB
results_cols = ["Classifier", "Accuracy", "Precision", "MCC", "False Positives"]
results = pd.DataFrame(columns = results_cols)

# Models
classifiers = [
    DecisionTreeClassifier(max_depth=3, random_state=42),
    RandomForestClassifier(max_depth=4, random_state=42),
    KNeighborsClassifier(metric='euclidean'),
    xgb.XGBClassifier(max_depth=4, random_state=42),
    MLPClassifier(alpha=1, max_iter=1000),
    AdaBoostClassifier(random_state=42),
    GradientBoostingClassifier(max_depth=4, random_state=42),
    GaussianNB()
]
for clf in classifiers:
    clf.fit(X_train, y_train)
    method = clf.__class__.__name__
    y_pred = clf.predict(X_test)
    accuracy, precision, mcc, fp = evaluationModels(y_test, y_pred)
    printMetrics(clf, accuracy, precision, mcc, fp)
    
    temp = pd.DataFrame([[method, accuracy * 100, precision * 100, mcc * 100, fp]], columns = results_cols)
    results = results.append(temp)
for metrics in results_cols[1:]:
    plt.figure(figsize = (10,6))
    sns.barplot(x = metrics, y = 'Classifier', data = results)
    plt.xlabel(f'{metrics} %')
    plt.title(f'{metrics} of models')
    plt.show()
results.sort_values('MCC', ascending=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12443958447238929, 'fraction_numerical': 0.022963367960634227, 'mean_word_length': 4.983316977428852, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 94, 'https://': 21, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '594206', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 6540, 'n_tokens_neox': 6129, 'n_words': 1369, 'path': 'introMLcolab.ipynb', 'repo_name': 'tamagusko/workshop-intro-ml', 'size': '18292'}