<jupyter_start><jupyter_text>Metrics for prediction model based on 1h resolution<jupyter_code>import datetime
import calendar
import time
import json
import numpy as np
import pandas as pd
from sklearn import tree
import matplotlib.pyplot as plt
from matplotlib import rcParams
rcParams['figure.figsize'] = 12, 4<jupyter_output><empty_output><jupyter_text>Load projectLoad rainfall and flow data from the files and clean it by:  * Resampling to 5 minutes  * Slice to the common range  * Fill NaNs<jupyter_code>PROJECT_FOLDER = '../../datasets/radon-medium/'

flow = pd.read_csv(PROJECT_FOLDER + 'flow1.csv', parse_dates=['time'])
flow = flow.set_index('time').flow
flow = flow.resample('1H').mean()

rainfall = pd.read_csv(PROJECT_FOLDER + 'rainfall1.csv', parse_dates=['time'])
rainfall = rainfall.set_index('time').rainfall
rainfall = rainfall.resample('1H').mean()

data_frame = pd.concat([flow, rainfall], axis=1).dropna()
data_frame['day'] = data_frame.index.map(lambda x: x.date())
data_frame['hour'] = data_frame.index.map(lambda x: x.time().hour)
data_frame = data_frame['2015-01-01':'2018-07-05']
print(data_frame.head())
print(data_frame.tail())<jupyter_output>flow  rainfall         day  hour
time                                                      
2015-01-01 00:00:00  76.796188       0.0  2015-01-01     0
2015-01-01 01:00:00  71.892892       0.0  2015-01-01     1
2015-01-01 02:00:00  63.906876       0.0  2015-01-01     2
2015-01-01 03:00:00  60.286973       0.0  2015-01-01     3
2015-01-01 04:00:00  57.049687       0.0  2015-01-01     4
                          flow  rainfall         day  hour
time                                                      
2018-04-13 04:00:00  43.278284       0.0  2018-04-13     4
2018-04-13 05:00:00  41.779852       0.0  2018-04-13     5
2018-04-13 06:00:00  49.345211       0.0  2018-04-13     6
2018-04-13 07:00:00  67.298808       0.0  2018-04-13     7
2018-04-13 08:00:00  75.363243       0.0  2018-04-13     8<jupyter_text>Prepare functions for calculating model scoreThe basic prediction model uses daily pattern as a prediction<jupyter_code>def loss(y_hat, y):
    """
    https://en.wikipedia.org/wiki/Mean_absolute_percentage_error
    """
    return 100.0 * np.sum(np.abs((y-y_hat) / y)) / y.shape[0]


err = loss(np.array([1,1,1]), np.array([2,2,2]))
print("loss 50.0 == {}".format(err))<jupyter_output>loss 50.0 == 50.0<jupyter_text>Evaluate datasetUse all data before 2018-01-01 as training set and later data as test set<jupyter_code>train_data = data_frame[data_frame.day < datetime.date(2018,1,1)]
test_data = data_frame[data_frame.day >= datetime.date(2018,1,1)]
print(train_data.tail())
print(test_data.head())
x_train = train_data[['hour']]
y_train = train_data['flow']
x_test = test_data[['hour']]
y_test = test_data['flow']<jupyter_output>flow  rainfall         day  hour
time                                                      
2017-12-03 19:00:00  76.851403       0.0  2017-12-03    19
2017-12-03 20:00:00  81.333705       0.0  2017-12-03    20
2017-12-03 21:00:00  84.180263       0.0  2017-12-03    21
2017-12-03 22:00:00  82.342646       0.0  2017-12-03    22
2017-12-03 23:00:00  72.624863       0.0  2017-12-03    23
                          flow  rainfall         day  hour
time                                                      
2018-03-01 00:00:00  63.987647       0.0  2018-03-01     0
2018-03-01 01:00:00  56.025623       0.0  2018-03-01     1
2018-03-01 02:00:00  50.192542       0.0  2018-03-01     2
2018-03-01 03:00:00  46.678541       0.0  2018-03-01     3
2018-03-01 04:00:00  43.955056       0.0  2018-03-01     4<jupyter_text>Test mean model<jupyter_code>y_test.plot()
plt.show()
data_frame['2015-10-01':].flow.plot()
plt.show()
mu = y_train.mean()
y_hat = np.ones(len(y_test)) * mu
loss(y_hat, y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1301173402868318, 'fraction_numerical': 0.1926988265971317, 'mean_word_length': 2.4872727272727273, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '127747', 'license': 'Apache-2.0', 'n_tokens_mistral': 1868, 'n_tokens_neox': 1437, 'n_words': 303, 'path': 'notebooks/klangner/radon-sandbox.ipynb', 'repo_name': 'carldata/argo', 'size': '3835'}