<jupyter_start><jupyter_text>This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**This notebook was generated for TensorFlow 2.6.  Deep learning for timeseries  Different kinds of timeseries tasks  A temperature-forecasting example<jupyter_code># !wget https://s3.amazonaws.com/keras-datasets/jena_climate_2009_2016.csv.zip
# !unzip jena_climate_2009_2016.csv.zip<jupyter_output><empty_output><jupyter_text>**Inspecting the data of the Jena weather dataset**<jupyter_code>import os
fname = os.path.join("data/jena_climate_2009_2016.csv")

with open(fname) as f:
    data = f.read()

lines = data.split("\n")
header = lines[0].split(",")
lines = lines[1:]
print(header)
print(len(lines))<jupyter_output>['"Date Time"', '"p (mbar)"', '"T (degC)"', '"Tpot (K)"', '"Tdew (degC)"', '"rh (%)"', '"VPmax (mbar)"', '"VPact (mbar)"', '"VPdef (mbar)"', '"sh (g/kg)"', '"H2OC (mmol/mol)"', '"rho (g/m**3)"', '"wv (m/s)"', '"max. wv (m/s)"', '"wd (deg)"']
420451<jupyter_text>**Parsing the data**<jupyter_code>import numpy as np
temperature = np.zeros((len(lines),))
raw_data = np.zeros((len(lines), len(header) - 1))
for i, line in enumerate(lines):
    values = [float(x) for x in line.split(",")[1:]] # discard the 'Date Time' column
    temperature[i] = values[1]
    raw_data[i, :] = values[:]<jupyter_output><empty_output><jupyter_text>**Plotting the temperature timeseries**<jupyter_code>from matplotlib import pyplot as plt
plt.plot(range(len(temperature)), temperature)<jupyter_output><empty_output><jupyter_text>**Plotting the first 10 days of the temperature timeseries**<jupyter_code>plt.plot(range(1440), temperature[:1440])<jupyter_output><empty_output><jupyter_text>**Computing the number of samples we'll use for each data split**<jupyter_code>num_train_samples = int(0.5 * len(raw_data))
num_val_samples = int(0.25 * len(raw_data))
num_test_samples = len(raw_data) - num_train_samples - num_val_samples
print("num_train_samples:", num_train_samples)
print("num_val_samples:", num_val_samples)
print("num_test_samples:", num_test_samples)<jupyter_output>num_train_samples: 210225
num_val_samples: 105112
num_test_samples: 105114<jupyter_text>Preparing the data **Normalizing the data**<jupyter_code>mean = raw_data[:num_train_samples].mean(axis=0)
raw_data -= mean
std = raw_data[:num_train_samples].std(axis=0)
raw_data /= std
mean
import numpy as np
from tensorflow import keras
int_sequence = np.arange(10)

dummy_dataset = keras.utils.timeseries_dataset_from_array(
    data=int_sequence[:-3],
    targets=int_sequence[3:],
    sequence_length=3,
    batch_size=2,
    shuffle=True
)

for inputs, targets in dummy_dataset:
    for i in range(inputs.shape[0]):
        print([int(x) for x in inputs[i]], int(targets[i]))
    break<jupyter_output>[3, 4, 5] 6
[4, 5, 6] 7<jupyter_text>**Instantiating datasets for training, validation, and testing**<jupyter_code>sampling_rate = 6
sequence_length = 120
delay = sampling_rate * (sequence_length + 24 - 1)
batch_size = 256

train_dataset = keras.utils.timeseries_dataset_from_array(
    raw_data[:-delay],
    targets=temperature[delay:],
    sampling_rate=sampling_rate,
    sequence_length=sequence_length,
    shuffle=True,
    batch_size=batch_size,
    start_index=0,
    end_index=num_train_samples)

val_dataset = keras.utils.timeseries_dataset_from_array(
    raw_data[:-delay],
    targets=temperature[delay:],
    sampling_rate=sampling_rate,
    sequence_length=sequence_length,
    shuffle=True,
    batch_size=batch_size,
    start_index=num_train_samples,
    end_index=num_train_samples + num_val_samples)

test_dataset = keras.utils.timeseries_dataset_from_array(
    raw_data[:-delay],
    targets=temperature[delay:],
    sampling_rate=sampling_rate,
    sequence_length=sequence_length,
    shuffle=True,
    batch_size=batch_size,
    start_index=num_train_samples + num_val_samples)<jupyter_output><empty_output><jupyter_text>**Inspecting the output of one of our datasets**<jupyter_code>for samples, targets in train_dataset:
    print("samples shape:", samples.shape)
    print("targets shape:", targets.shape)
    break
raw_data.shape<jupyter_output><empty_output><jupyter_text>A common-sense, non-machine-learning baseline **Computing the common-sense baseline MAE**<jupyter_code>def evaluate_naive_method(dataset):
    total_abs_err = 0.
    samples_seen = 0
    for samples, targets in dataset:
        preds = samples[:, -1, 1] * std[1] + mean[1] # normalized temp is in column 1
        total_abs_err += np.sum(np.abs(preds - targets))
        samples_seen += samples.shape[0]
    return total_abs_err / samples_seen

print(f"Validation MAE: {evaluate_naive_method(val_dataset):.2f}")
print(f"Test MAE: {evaluate_naive_method(test_dataset):.2f}")<jupyter_output><empty_output><jupyter_text>Let's try a basic machine-learning model **Training and evaluating a densely connected model**<jupyter_code>from tensorflow import keras
from tensorflow.keras import layers

inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1])) 
x = layers.Flatten()(inputs)
x = layers.Dense(16, activation="relu")(x)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)

callbacks = [
    keras.callbacks.ModelCheckpoint("jena_dense.keras",
                                    save_best_only=True)
]
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])
history = model.fit(train_dataset,
                    epochs=10,
                    validation_data=val_dataset,
                    callbacks=callbacks)

model = keras.models.load_model("jena_dense.keras")
print(f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}")<jupyter_output><empty_output><jupyter_text>**Plotting results**<jupyter_code>import matplotlib.pyplot as plt
loss = history.history["mae"]
val_loss = history.history["val_mae"]
epochs = range(1, len(loss) + 1)
plt.figure()
plt.plot(epochs, loss, "bo", label="Training MAE")
plt.plot(epochs, val_loss, "b", label="Validation MAE")
plt.title("Training and validation MAE")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Let's try a 1D convolutional model<jupyter_code>inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))
x = layers.Conv1D(8, 24, activation="relu")(inputs)
x = layers.MaxPooling1D(2)(x)
x = layers.Conv1D(8, 12, activation="relu")(x)
x = layers.MaxPooling1D(2)(x)
x = layers.Conv1D(8, 6, activation="relu")(x)
x = layers.GlobalAveragePooling1D()(x)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)

callbacks = [
    keras.callbacks.ModelCheckpoint("jena_conv.keras",
                                    save_best_only=True)
]
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])
history = model.fit(train_dataset,
                    epochs=10,
                    validation_data=val_dataset,
                    callbacks=callbacks)

model = keras.models.load_model("jena_conv.keras")
print(f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}")
model.summary()<jupyter_output><empty_output><jupyter_text>A first recurrent baseline **A simple LSTM-based model**<jupyter_code>inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))
x = layers.LSTM(16)(inputs)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)

callbacks = [
    keras.callbacks.ModelCheckpoint("jena_lstm.keras",
                                    save_best_only=True)
]
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])
history = model.fit(train_dataset,
                    epochs=10,
                    validation_data=val_dataset,
                    callbacks=callbacks)

model = keras.models.load_model("jena_lstm.keras")
print(f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}")
model.summary()<jupyter_output><empty_output><jupyter_text>Understanding recurrent neural networks **NumPy implementation of a simple RNN**<jupyter_code>import numpy as np
timesteps = 100
input_features = 32
output_features = 64
inputs = np.random.random((timesteps, input_features))
state_t = np.zeros((output_features,))
W = np.random.random((output_features, input_features))
U = np.random.random((output_features, output_features))
b = np.random.random((output_features,))
successive_outputs = []
for input_t in inputs:
    output_t = np.tanh(np.dot(W, input_t) + np.dot(U, state_t) + b)
    successive_outputs.append(output_t)
    state_t = output_t
final_output_sequence = np.stack(successive_outputs, axis=0)<jupyter_output><empty_output><jupyter_text>A recurrent layer in Keras **An RNN layer that can process sequences of any length**<jupyter_code>num_features = 14
inputs = keras.Input(shape=(None, num_features))
outputs = layers.SimpleRNN(16)(inputs)<jupyter_output><empty_output><jupyter_text>**An RNN layer that returns only its last output step**<jupyter_code>num_features = 14
steps = 120
inputs = keras.Input(shape=(steps, num_features))
outputs = layers.SimpleRNN(16, return_sequences=False)(inputs)
print(outputs.shape)<jupyter_output><empty_output><jupyter_text>**An RNN layer that returns its full output sequence**<jupyter_code>num_features = 14
steps = 120
inputs = keras.Input(shape=(steps, num_features))
outputs = layers.SimpleRNN(16, return_sequences=True)(inputs)
print(outputs.shape)<jupyter_output><empty_output><jupyter_text>**Stacking RNN layers**<jupyter_code>inputs = keras.Input(shape=(steps, num_features))
x = layers.SimpleRNN(16, return_sequences=True)(inputs)
x = layers.SimpleRNN(16, return_sequences=True)(x)
outputs = layers.SimpleRNN(16)(x)<jupyter_output><empty_output><jupyter_text>Advanced use of recurrent neural networks  Using recurrent dropout to fight overfitting **Training and evaluating a dropout-regularized LSTM**<jupyter_code>from tensorflow import keras
from tensorflow.keras import layers

inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))
x = layers.LSTM(32, recurrent_dropout=0.25)(inputs)
#x = layers.LSTM(32, recurrent_dropout=0.25, unroll=True)(inputs)  # unrolling 
x = layers.Dropout(0.5)(x)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)

callbacks = [
    keras.callbacks.ModelCheckpoint("jena_lstm_dropout.keras",
                                    save_best_only=True)
]
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])
history = model.fit(train_dataset,
                    epochs=50,
                    validation_data=val_dataset,
                    callbacks=callbacks)
inputs = keras.Input(shape=(sequence_length, num_features))
x = layers.LSTM(32, recurrent_dropout=0.2, unroll=True)(inputs)<jupyter_output><empty_output><jupyter_text>Stacking recurrent layers **Training and evaluating a dropout-regularized, stacked GRU model**<jupyter_code>inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))
#x = layers.GRU(32, recurrent_dropout=0.5, return_sequences=True)(inputs)
#x = layers.GRU(32, recurrent_dropout=0.5)(x)
x = layers.GRU(32, return_sequences=True)(inputs)
x = layers.GRU(32)(x)
x = layers.Dropout(0.5)(x)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)

callbacks = [
    keras.callbacks.ModelCheckpoint("jena_stacked_gru_dropout.keras",
                                    save_best_only=True),
    keras.callbacks.EarlyStopping(monitor='val_mae', patience=2)
]
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])
history = model.fit(train_dataset,
                    epochs=50,
                    validation_data=val_dataset,
                    callbacks=callbacks)
model = keras.models.load_model("jena_stacked_gru_dropout.keras")
print(f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}")<jupyter_output>Epoch 1/50
819/819 [==============================] - 16s 17ms/step - loss: 24.3549 - mae: 3.6321 - val_loss: 9.2588 - val_mae: 2.3519
Epoch 2/50
819/819 [==============================] - 13s 16ms/step - loss: 13.4352 - mae: 2.8327 - val_loss: 8.9393 - val_mae: 2.3061
Epoch 3/50
819/819 [==============================] - 13s 16ms/step - loss: 12.4185 - mae: 2.7302 - val_loss: 9.3118 - val_mae: 2.3702
Epoch 4/50
819/819 [==============================] - 13s 16ms/step - loss: 11.7054 - mae: 2.6527 - val_loss: 9.1069 - val_mae: 2.3576
405/405 [==============================] - 3s 7ms/step - loss: 10.1342 - mae: 2.4978
Test MAE: 2.50<jupyter_text>Using bidirectional RNNs **Training and evaluating a bidirectional LSTM**<jupyter_code>inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))
x = layers.Bidirectional(layers.LSTM(16))(inputs)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)

model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])
history = model.fit(train_dataset,
                    epochs=10,
                    validation_data=val_dataset)<jupyter_output>Epoch 1/10
819/819 [==============================] - 16s 16ms/step - loss: 27.0224 - mae: 3.7360 - val_loss: 10.5874 - val_mae: 2.5203
Epoch 2/10
819/819 [==============================] - 13s 16ms/step - loss: 9.4919 - mae: 2.4056 - val_loss: 9.7927 - val_mae: 2.4195
Epoch 3/10
819/819 [==============================] - 13s 16ms/step - loss: 8.5497 - mae: 2.2798 - val_loss: 9.7709 - val_mae: 2.4144
Epoch 4/10
819/819 [==============================] - 13s 16ms/step - loss: 8.0597 - mae: 2.2164 - val_loss: 10.2177 - val_mae: 2.4553
Epoch 5/10
819/819 [==============================] - 13s 16ms/step - loss: 7.7531 - mae: 2.1733 - val_loss: 10.1270 - val_mae: 2.4479
Epoch 6/10
819/819 [==============================] - 13s 16ms/step - loss: 7.4701 - mae: 2.1315 - val_loss: 10.5394 - val_mae: 2.4675
Epoch 7/10
819/819 [==============================] - 13s 16ms/step - loss: 7.1969 - mae: 2.0897 - val_loss: 11.2156 - val_mae: 2.5353
Epoch 8/10
819/819 [==============================] - 13[...]
{'fraction_non_alphanumeric': 0.16130403968816442, 'fraction_numerical': 0.045499645641389086, 'mean_word_length': 5.071858864027539, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 84, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '366307', 'license': 'MIT', 'n_tokens_mistral': 5420, 'n_tokens_neox': 4924, 'n_words': 942, 'path': 'chapter10_dl-for-timeseries.ipynb', 'repo_name': 'ChengUVa/dlp2', 'size': '14110'}