<jupyter_start><jupyter_text>Machine Learning - Personal Project Description du projet : This work assesses the potential for applying machine learning techniques and algorithms to suspended sediment concentration in a proglacial river to predict sediment concentration as a function of discharge and to observe whether these two variables are correlated.  Part I : Connection to my Drive and downloading of data Here, the goal is to connect this notebook to my personal Drive and download my data file<jupyter_code>!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
from google.colab import drive
drive.mount('/content/drive')
import pandas as pd
df_1 =pd.read_excel('/content/LGS1_data.xlsx')
df_2 =pd.read_excel('/content/LGS2_data.xlsx')
df_1
df_2<jupyter_output><empty_output><jupyter_text>Information about the diffferent variables :*   Time  = time, in days since 1.1.21*   C     = suspended load concentration*   SD_C  = standard deviation of C*   Q     = flow rate*   SD_Q  = standard deviation of Q*   QC    = suspended load tranport, kg/s (QC)*   SD_QC = standard deviation of Q*   NoUse = column with variables not useful for this exerciseGS1 : The station is located downstream of the alluvial plainGS2 : The station is located at the exit of the glacier, upstream of the alluvial plain  Part II : Loading of packaging useful for the entire project Here, the goal is to compile and load all the packages used to complete the project<jupyter_code>import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
from sklearn import preprocessing
label_encoder = preprocessing.LabelEncoder()<jupyter_output><empty_output><jupyter_text>Part III : Data processing The purpose of this section is to remove unnecessary collums from the dataframe as well as to remove NaN values present in the data<jupyter_code>df_1.info()
df_2.info()
# Removing the two "NoUse" collums
LGS1 = df_1[['Time','C','SD_C','Q','SD_Q','QC','SD_QC']].copy()
LGS2 = df_2[['Time','C','SD_C','Q','SD_Q','QC','SD_QC']].copy()
LGS1.info()
LGS2.info()
# check that no raw has NaN values
LGS1.isna().values.any()
LGS2.isna().values.any()
# if true -> deleting all raw containing NaN
LGS1_noNAN = LGS1.dropna(axis=0)
LGS2_noNAN = LGS2.dropna(axis=0)
# check again that no raw has NaN values
LGS1_noNAN.isna().values.any()
LGS2_noNAN.isna().values.any()
LGS1_noNAN.info()
LGS2_noNAN.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1964 entries, 0 to 1963
Data columns (total 7 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   Time    1964 non-null   float64
 1   C       1964 non-null   float64
 2   SD_C    1964 non-null   float64
 3   Q       1964 non-null   float64
 4   SD_Q    1964 non-null   float64
 5   QC      1964 non-null   float64
 6   SD_QC   1964 non-null   float64
dtypes: float64(7)
memory usage: 122.8 KB<jupyter_text>Part IV : Apply a Randomforest classifier on the LGS2 dataset The aim of this section is to apply the RandomForest algorithm to the dataset from the measuring station located at outlet of the glacier (GS2)<jupyter_code>Y = LGS2[['C']].copy()
X = LGS2[['Q']].copy()
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=2200, train_size=3000)
X_test, X_valid, y_test, y_valid = train_test_split(X_train, y_train, test_size=1500, train_size=1500)

print(len(X_train), len(X_test), len(X_valid))
label_encoder = preprocessing.LabelEncoder()
y_train = label_encoder.fit_transform(y_train)
y_test = label_encoder.fit_transform(y_test)
y_valid = label_encoder.fit_transform(y_valid)
rfc = RandomForestClassifier()
rfc.fit(X_train, y_train)
rfc_valid = rfc.predict(X_valid)
print(accuracy_score(y_valid, rfc_valid))
print('\nMatrix confusion for RandomForestClassifier :')
print(confusion_matrix(y_valid, rfc_valid))

acc_rfc = accuracy_score(y_valid, rfc_valid)
print(f'\nThe accuracy of the model RandomForestClassifier is {acc_rfc:.1%}')<jupyter_output>Matrix confusion for RandomForestClassifier :
[[939   0   0 ...   0   0   0]
 [  0   1   0 ...   0   0   0]
 [  0   0   1 ...   0   0   0]
 ...
 [  0   0   0 ...   0   0   0]
 [  0   0   0 ...   0   0   0]
 [  0   0   0 ...   0   0   0]]

The accuracy of the model RandomForestClassifier is 62.9%<jupyter_text>Part V : Apply a Randomforest classifier on the LGS1 dataset The aim of this section is to apply the RandomForest algorithm to the dataset from the measuring station located downstream of the alluvial plaine (GS1)<jupyter_code>Y = LGS1[['C']].copy()
X = LGS1[['Q']].copy()
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=2200, train_size=3000)
X_test, X_valid, y_test, y_valid = train_test_split(X_train, y_train, test_size=1500, train_size=1500)
print(len(X_train), len(X_test), len(X_valid))
label_encoder = preprocessing.LabelEncoder()
y_train = label_encoder.fit_transform(y_train)
y_test = label_encoder.fit_transform(y_test)
y_valid = label_encoder.fit_transform(y_valid)
rfc = RandomForestClassifier()
rfc.fit(X_train, y_train)
rfc_valid = rfc.predict(X_valid)
print(accuracy_score(y_valid, rfc_valid))
print('\nMatrix confusion for RandomForestClassifier :')
print(confusion_matrix(y_valid, rfc_valid))

acc_rfc = accuracy_score(y_valid, rfc_valid)
print(f'\nThe accuracy of the model RandomForestClassifier is {acc_rfc:.1%}')<jupyter_output>Matrix confusion for RandomForestClassifier :
[[19  0  0 ...  0  0  0]
 [ 0  1  0 ...  0  0  0]
 [ 0  0  1 ...  0  0  0]
 ...
 [ 0  0  0 ...  0  0  0]
 [ 0  0  0 ...  0  0  0]
 [ 0  0  0 ...  0  0  0]]

The accuracy of the model RandomForestClassifier is 1.4%
{'fraction_non_alphanumeric': 0.09979564032697548, 'fraction_numerical': 0.03797683923705722, 'mean_word_length': 4.378205128205129, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '457296', 'license': 'MIT', 'n_tokens_mistral': 2136, 'n_tokens_neox': 2006, 'n_words': 633, 'path': 'Personal_Project_Kdiba.ipynb', 'repo_name': 'KeyvanDiba/2022_ML_Earth_Env_Sci', 'size': '5872'}