<jupyter_start><jupyter_text>Exploratory Data AnalysisThis Notebook is dedicated to performing the data analysis of the variables that describe the dataset. This notebook will perform data exploration, as well feature engineering using the known physics related to the problem at hand.<jupyter_code>#For development
#Reload the library when a change is detected in one of the imported libraries
%load_ext autoreload
%autoreload 2
import pandas as pd
import numpy as np

from sklearn.model_selection import StratifiedKFold
from sklearn.utils import shuffle

import matplotlib.pyplot as plt
#Custom utils file
from src import feature_engineering as fe<jupyter_output><empty_output><jupyter_text>df_b = pd.read_csv('data/Data.csv')df_b['Author'] = df_b["Author"].str.slice(stop=8)di = {'A': 4, 'DB': 1, 'I': 5, 'SW': 3, 'SS': 2, 'B': 7}inv_di = {i:k for k,i in di.items()}df_b.replace({"Flow_label" : inv_di}, inplace=True)df_b.to_csv("data/Dataset.csv", index=False)<jupyter_code>#Load the RAW dataset
df_b = pd.read_csv('data/Dataset.csv')
df_b.head()
df_b = shuffle(df_b, random_state=0)
df_b.reset_index(drop=True, inplace=True)
df_b.head()<jupyter_output><empty_output><jupyter_text>Define features and target columns<jupyter_code>target = 'Flow_label'
features_b = df_b.columns.drop(target) #Bronze features<jupyter_output><empty_output><jupyter_text>Train Test SplittingThe first and most important part of the analysis, to avoid information leakage (and hence optimistic expectations) is to split the data between training and testing. Since we are dealing with a classification problem, stratified k fold is applied to the labels in order to preserve the original distribution in both training and testing<jupyter_code>skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) #SKF
train_idx, test_idx = next(skf.split(df_b[features_b].values, df_b[target].values.ravel())) #Get the split
df_b, df_b_test = df_b.loc[train_idx].reset_index(drop=True), df_b.loc[test_idx] #Train, Test<jupyter_output><empty_output><jupyter_text>We now save the bronze (unprocessed) data in the right folder. No reasonable infomration leakage can now happen between training and testing now  Null Values and Variables DistributionWe can now properly analyze the data, fist by checking for missing values, and then by exploring variables distributions<jupyter_code>Null_values = df_b.isnull().mean()[df_b.isnull().mean()!=0]
Null_values<jupyter_output><empty_output><jupyter_text>It is clear that most of the data related to the lenght of the pipe is missing. While some argument could be made about treating this variable as a categorical and mean encode it, to keep the model relaviely simple, it will be removed in the following step (in the tranformation of the data from bronze to silver)  Target DistributionWe see that the data is far from being evenly distribuited. The next step is to assess weather the target is balanced or not<jupyter_code>fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 4))   

out_norm = df_b[target].value_counts(sort=False, normalize=True).mul(1)
out_norm.plot.bar(rot=0)
axes.set_ylabel("Freq")
axes.set_xlabel("Flow Regime")
axes.grid()

plt.show()
fig.savefig(f'plots/others/flow_patterns.png', dpi=fig.dpi, bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>There is a strong imbalance between the labels, the intermittent flow regime cover more than half the data. During training, oversampling technqiues will be required to improve the detection of the minority classes  Feature EngineeringOnce we have analyzed the various distribution, we see that feature engineering will be required to better separate the various classes. This is fundamental in tree based models like RandomForest and LightGBM. Additionally, to improve generability (sacrificng performance on this specific dataset) dimensionless quantities will replace dimensional quantities previously analyzed<jupyter_code>df_g = fe.bronze_to_gold(df_b) #Gold data (Ready to be processed)
features_g = df_g.columns.drop(target) #Gold features
df_g.head()<jupyter_output><empty_output><jupyter_text>Specific BoxPlotsIt can be interesting to display the boxplots of some promising features, to check if they are of use to separate the various classes. Here the three most important features found in the analysis are displayed, and they are clearly able to differentiate the flow regimes by some degree. Note that for display purposes, the logarithm of the velocities is taken (the models used for the analysis are however unaffected by the scale of the features used)<jupyter_code>log_q = [q for q in df_g[features_g].columns if df_g[q].min()>=0]
df_log = pd.DataFrame(df_b[target], columns=[target])
for q in log_q:
    df_log[f'log({q})'] = np.log10(df_g[q])
df_ALL = pd.concat([df_log, df_g, df_b], axis=1)
df_ALL = df_ALL.loc[:,~df_ALL.columns.duplicated()]

to_plot_features = ["log(FrG)", "log(FrL)", "Ang"]

fig, axes = plt.subplots(nrows=1, ncols=len(to_plot_features), figsize=(14, 6)) 
for quantity, i in zip(to_plot_features, range(len(to_plot_features))):
    df_ALL[[quantity, target]].boxplot(by=target, 
                                           showfliers=False, 
                                           ax=axes[i])
    axes[i].set_xlabel("Flow Pattern")
    fig.suptitle('')

axes[0].set_title(r"$log(Fr_{G})$")
axes[1].set_title(r"$log(Fr_{L})$")
axes[2].set_title(r"$\theta$")

fig.savefig(f'plots/boxplots/specific_features.png', dpi=fig.dpi, bbox_inches='tight')
plt.show()<jupyter_output><empty_output><jupyter_text>Feature CorrelationWe can check the quality of the generated features by mean of correlation. It is evident that the proposed features are mostly uncorrelated, and offer different types of signals for our model.<jupyter_code>df_corr = df_g.drop(columns=target)
df_corr.columns = [r'$\theta$', r'$Re_{L}$', r'$Re_{G}$', 
                   r'$Fr_{L}$', r'$Fr_{G}$', r'$N_{L}$', 
                   r'$X_{LM}$', r'$X_{LM}^2$', r'$Y$', 
                   r'$We$', r'$Eo$', r'$\alpha_{L}$', 
                   r'$K_{G}$', r'$T_{TB}$']

fig = plt.figure(figsize=(10, 8), dpi=100)
plt.matshow((df_corr).corr(), fignum=fig.number, cmap='seismic')

plt.xticks(range(df_corr.shape[1]), df_corr.columns, fontsize=14, rotation=45)
plt.yticks(range(df_corr.shape[1]), df_corr.columns, fontsize=14)

cb = plt.colorbar()
cb.ax.tick_params(labelsize=14)

fig.savefig(f'plots/others/correlation_matrix.png', dpi=fig.dpi, bbox_inches='tight')
plt.show()
df_b.to_csv('data/Train_bronze.csv', index=False)
df_b_test.to_csv('data/Test_bronze.csv', index=False)
df_g.to_csv('data/Train_gold.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1125599520383693, 'fraction_numerical': 0.007044364508393285, 'mean_word_length': 5.706532663316583, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '111528', 'license': 'Apache-2.0', 'n_tokens_mistral': 2232, 'n_tokens_neox': 2120, 'n_words': 647, 'path': '1_ExploratoryAnalysis.ipynb', 'repo_name': 'Benetti-Hub/Multiphase-Flow-Regimes', 'size': '6672'}