<jupyter_start><jupyter_text>Prediction of the Loop Aggregation Rate  ContextModflow, a hydrological simulator, is used to predict the underground watertable mouvements in order to assess the vulnerability of a geographical site (i.e. watershed). However, a simulation can last more than 20 hours and we need to run numerous simulations (i.e., across a lot of different sites and/or chronicles). We developped an approach named `Loop Aggregation` which reduces the execution time of the simulation while preserving acceptable results.The loop aggregation technique reduces the execution time by reducing the number of computations done inside the iterative loop of the simulator. For instance, instead of computing the position of the watertable every day over a 50 year period, only the computations corresponding to one every `p` days will be done. `p` can take the value inside the following list : [2, 7, 15, 21, 30, 45, 50, 60, 75, 90, 100, 125,150,182,200,250,300,330,350,550,640,750,1000,1500,2000,2250,3000,3182,3652]. `p` is named the `aggregation rate` (it is named `Rate` in the data). The higher `p` is, the less computations is done and the faster the simulation is.To know to what extend we can "approximate", meaning to which value of `p`, we can reduce the execution time, we need to check that the results produced by the approximate simulation are still scientifically acceptable. To do so, an `acceptability indicator` (i.e. named in the data `HError`) has been elaborated and is computed for every approximate simulation (for every simulation using a different value of `p`).When the value of the `acceptability indicator` is below the threshold of `0.1`, that means that the approximate simulation is `valid` and we can use the `loop aggregation` approach on the simulator with the corresponding aggregation rate.When HError > 0.1, the approximate simulation is invalid. We can use this rate for applying the `loop aggregation` technique. Goal:Using the data we have already generated, we would like to predict the maximal value of the aggregation rate (i.e. pmax) leading to the fastest approximate simulation being valid, meaning the rate for which HError is below 0.1 and the closest to 0.1, in the context of a new chronicle or a new site. The goal is to use some Machine Learning techniques to do so.  1.Prediction with Sub-Catchment Areas For Each Site  Imports<jupyter_code>import pandas as pd
import prediction_of_H_indicator_with_subCatchmentData as prediction
import ipywidgets as widgets
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score<jupyter_output><empty_output><jupyter_text>Variables<jupyter_code>INPUT_FOLDER = "data"
site_number = 0<jupyter_output><empty_output><jupyter_text>Import Data and Clean them<jupyter_code>input_data = prediction.import_input_data()
input_data
input_data_cleaned = prediction.clean_data(input_data)
input_data_cleaned<jupyter_output><empty_output><jupyter_text>Choosing the new site to make predictions for<jupyter_code># INSTRUCTIONS : choose the site you want to make prediction for & click on the 'valid' button. Then go to the 'Prepare datasets for training the prediction model'

# CAUTION : if running from jupyterlba => jupyter labextension install @jupyter-widgets/jupyterlab-manager
## IF this cell is not running => SKIP THIS CELL and GO TO NEXT CELL :D


from IPython.display import display, Javascript

drop = widgets.Dropdown(
    options=[('Breville-Sur-Mer', 1), ('Agon-Coutainville', 2), ('Saint-Germain-Sur-Ay', 3), ('Jullouville', 4), ('Etréham', 5), ('Genêts', 6), ('Saint-Pair-sur-Mer', 7), ('Arromanches-les-Bains', 8), ('Port-en-Bessin', 9), ('Courtils', 10), ('Lessay', 11), ('Doville', 12), ('Graye-sur-Mer', 13), ('Saint-Potan', 14), ('Saint-Vaast-la-Hougue', 15), ('La_Pernelle', 16), ('Lestre', 17), ('Banville', 18), ('Isigny-sur-Mer', 19), ('Saint-Malo', 20), ('Pierreville', 21), ('La_Feuillie', 22), ('Tourlaville', 23), ('Octeville', 24), ('Saint-Briac-sur-Mer', 25), ('Granville', 26), ('Blainville-sur-Mer', 27), ('Hauteville-sur-Mer', 28), ('Sainte-Marie-du-Mont', 29), ('Vierville-sur-Mer', 30), ('Cherrueix', 31), ('Chef-du-Pont', 32), ('Saint-Lô', 33), ('Couvains', 34), ('Rocheville', 35), ('Lison', 36), ('Marigny', 37), ('Carville', 38), ('Percy', 39), ('Saint-Armand', 40)],
    value=7,
    description='Site:',
)
button = widgets.Button(description="Valid")
output = widgets.Output()

display(drop, button, output)

def on_button_clicked(b):
    global site_number
    site_number = drop.value

button.on_click(on_button_clicked)
# Manually change the site to predict the H values for ;p

site_number = 7<jupyter_output><empty_output><jupyter_text>Prepare datasets for training the prediction model<jupyter_code>features, variable_to_predict = prediction.split_dataset_into_features_and_variable_to_predict(input_data_cleaned)
features_train, features_test, variable_train, variable_test = prediction.get_train_and_test_data(features, variable_to_predict, site_number)<jupyter_output><empty_output><jupyter_text>Training prediction model<jupyter_code>def train_random_forest_model(X_train, y_train):
    forest = RandomForestRegressor(
        n_estimators=1000, criterion="mse", random_state=1, n_jobs=-1
    )
    forest.fit(X_train, y_train.values.ravel())
    return forest

prediction_model = train_random_forest_model(features_train, variable_train)<jupyter_output><empty_output><jupyter_text>Prediction<jupyter_code>def predict_with_trained_model(forest, X_train, X_test):
    # Predicting results
    y_train_pred = forest.predict(X_train)
    y_test_pred = forest.predict(X_test)
    return y_train_pred, y_test_pred

variable_train_pred, variable_test_pred = predict_with_trained_model(prediction_model, features_train, features_test)<jupyter_output><empty_output><jupyter_text>Quality metrics for H values (MSE & NSE/R² scores)<jupyter_code>subCatchment_numbers = prediction.get_subcatchment_numbers_for_a_site(site_number, variable_to_predict)
liste_variable_test_HError = prediction.get_list_variable_test_Hind_Values(variable_test)
liste_variable_test_pred_HError = variable_test_pred
mse_test, r2_test = prediction.get_standard_quality_metrics(subCatchment_numbers, liste_variable_test_HError, liste_variable_test_pred_HError)<jupyter_output><empty_output><jupyter_text>Get Pmax real and Pmax predicted<jupyter_code>rates = prediction.get_rates_for_a_site(site_number, features)
pmax_test, pmax_pred = prediction.get_real_and_pred_pmax(subCatchment_numbers, rates, liste_variable_test_HError, variable_test_pred)<jupyter_output>Real value of p:  {1: 3652.0, 2: 3652.0, 3: 3652.0, 4: 3652.0, 5: 3652.0, 6: 3652.0, 7: 3652.0, 8: 3652.0, 9: 3652.0, 10: 3652.0, 11: 3652.0, 12: 3652.0, 13: 3652.0, 14: 3652.0, 15: 3652.0, 16: 3652.0, 17: 3652.0, 18: 3652.0, 19: 3652.0, 20: 3652.0}
Predicted value of p:  {1: 3652.0, 2: 3652.0, 3: 330.0, 4: 250.0, 5: 150.0, 6: 1000.0, 7: 365.0, 8: 125.0, 9: 640.0, 10: 150.0, 11: 150.0, 12: 1000.0, 13: 330.0, 14: 200.0, 15: 125.0, 16: 640.0, 17: 125.0, 18: 125.0, 19: 125.0, 20: 90.0}<jupyter_text>Save results into files<jupyter_code>prediction.save_Hind_results_into_file(site_number, subCatchment_numbers, rates, liste_variable_test_HError, variable_test_pred, approx=0, chronicle=0, permeability=27.32)
prediction.save_Pmax_results_into_file(site_number, pmax_test, pmax_pred, mse_test, r2_test, approx=0, chronicle=0, permeability=27.32)<jupyter_output>The directory did not exist. It has been created here: output/Approx0/Chronicle0/SiteTest25
File created here:  output/Approx0/Chronicle0/SiteTest25/Prediction_HErrorValues_SubCatch_Chronicle0_Approx0_K27.32_Slope_Elevation_LC_SAR_Area_CV_HV.csv
File created here: output/Approx0/Chronicle0/SiteTest25/Prediction_PMax_SubCatch_Chronicle0_Approx0_K27.32_Slope_Elevation_LC_SAR_Area_CV_HV.csv
{'fraction_non_alphanumeric': 0.10467762326169405, 'fraction_numerical': 0.05802781289506953, 'mean_word_length': 6.527117031398668, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '185905', 'license': 'MIT', 'n_tokens_mistral': 3025, 'n_tokens_neox': 2607, 'n_words': 782, 'path': 'PredictionWithSubCatch.ipynb', 'repo_name': 'Jnsll/DemoNotebookLAPrediction', 'size': '7910'}