<jupyter_start><jupyter_text>Load data from csv-filesThe used data can be found [here](https://github.com/rc9000/wiewarm-data) and is free to use according to Creatvie Commons Attribution-ShareAlike 3.0.<jupyter_code>os.chdir("./data/raw")
extension = 'csv'
filenames = [i for i in glob.glob('*.{}'.format(extension))]
filenames
df = pd.concat([pd.read_csv(f, sep=';') for f in filenames])<jupyter_output><empty_output><jupyter_text>Data explorationFollowing steps are done during this exploration and manipulation:* Dropping unneseccary columns* Manual outlier detection (could be done with ML)* Grouping the temperatures per day* Creating a time series and some analysing plots<jupyter_code>df.head()
df.shape
# gapminder_2002 = gapminder[gapminder['year']==2002]
df = df[df['beckenid'] == 52]
df.head()
df.shape
unused_cols = ['badid', 'bad', 'adresse1', 'adresse2', 'ort', 'plz', 'kanton', 'beckenid', 'becken', 'typ']
df =  df.drop(unused_cols, axis=1)
df['datum'] = pd.to_datetime(df['datum'])
df['datum'] = df['datum'].dt.date
# Filter unrealistic temperatures (Manual outliers detection)
df.drop(df[(df.temperatur > 30) | (df.temperatur < 5)].index, inplace=True)
df.head()
ts = df.groupby('datum')['temperatur'].mean().rename('Aare-temperature')
ts.plot()
window = 365

rolling_mean = ts.rolling(window=window).mean()
rolling_std = ts.rolling(window=window).std()
plt.figure(figsize=(15,5))
plt.title(ts.name)
plt.plot(ts[window:], label='Actual values', color='black')
plt.plot(rolling_mean, 'g', label='Moving Avg '+str(window), color='red')
lower_bound = rolling_mean - (2 * rolling_std)
upper_bound = rolling_mean + (2 * rolling_std)
plt.fill_between(x=ts.index, y1=lower_bound, y2=upper_bound, color='lightskyblue', alpha=0.4)
plt.legend(loc='best')
plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>Modeling Preparing dataThe predictions of the value at a certain date are made based on n-time steps before the certain date. For this window implementation the method create_dataset is used. To prevent our model from overfitting we split our data in 80% train and 20% test data. The sequence of the data is static and should not randomized.<jupyter_code>df = df.groupby('datum').mean()
train_size = int(len(df) * 0.8)
test_size = len(df) - train_size
train, test = df.iloc[0:train_size], df.iloc[train_size:len(df)]

print(len(train), len(test))
def create_dataset(X, y, time_steps=1):
    Xs, ys = [], []
    for i in range(len(X) - time_steps):
        v = X.iloc[i:(i + time_steps)].values
        Xs.append(v)
        ys.append(y.iloc[i + time_steps])
    return np.array(Xs), np.array(ys)
time_steps = 10

X_train, y_train = create_dataset(train, train, time_steps)
X_test, y_test = create_dataset(test, test, time_steps)

print(X_train.shape, y_train.shape)<jupyter_output>(2742, 10, 1) (2742, 1)<jupyter_text>Creating the model<jupyter_code>model = keras.Sequential()
model.add(keras.layers.LSTM(128, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(keras.layers.Dense(1))
model.compile(loss='mean_squared_error', optimizer=keras.optimizers.Adam(0.001))
learner = model.fit(
    X_train, y_train, 
    epochs=30, 
    batch_size=16, 
    validation_split=0.1, 
    verbose=1, 
    shuffle=False
)
plt.plot(learner.history['loss'], label='train')
plt.plot(learner.history['val_loss'], label='test')
plt.legend();
y_pred = model.predict(X_test)
plt.figure(figsize=(15,5))
plt.plot(np.arange(0, len(y_train)), y_train, color='green', label="history")
plt.plot(np.arange(len(y_train), len(y_train) + len(y_test)), y_test, marker='.', label="true")
plt.plot(np.arange(len(y_train), len(y_train) + len(y_test)), y_pred, color='red', label="prediction")
plt.ylabel('Value')
plt.xlabel('Time Step')
plt.legend()
plt.show();
plt.figure(figsize=(15,5))
plt.plot(y_test[:100], marker='.', label="true")
plt.plot(y_pred[:100], color='red', label="prediction")
plt.ylabel('Value')
plt.xlabel('Time Step')
plt.legend()
plt.show();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16829207301825455, 'fraction_numerical': 0.021505376344086023, 'mean_word_length': 6.339449541284404, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '138461', 'license': 'MIT', 'n_tokens_mistral': 1568, 'n_tokens_neox': 1459, 'n_words': 313, 'path': 'aare-temperature-time-series-model.ipynb', 'repo_name': 'wauchi/aare-temp-forecast', 'size': '3999'}