<jupyter_start><jupyter_text>This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=ml-ninja&a_cid=11111111&chan=c2). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**This notebook was generated for TensorFlow 2.6.  Getting started with neural networks: Classification and regression  Classifying movie reviews: A binary classification example  The IMDB dataset **Loading the IMDB dataset**<jupyter_code>from tensorflow.keras.datasets import imdb
(train_data, train_labels), (test_data, test_labels) = imdb.load_data(
    num_words=10000)
train_data[0]
train_labels[0]
max([max(sequence) for sequence in train_data])<jupyter_output><empty_output><jupyter_text>**Decoding reviews back to text**<jupyter_code>word_index = imdb.get_word_index()
reverse_word_index = dict(
    [(value, key) for (key, value) in word_index.items()])
decoded_review = " ".join(
    [reverse_word_index.get(i - 3, "?") for i in train_data[0]])<jupyter_output><empty_output><jupyter_text>Preparing the data **Encoding the integer sequences via multi-hot encoding**<jupyter_code>import numpy as np
def vectorize_sequences(sequences, dimension=10000):
    results = np.zeros((len(sequences), dimension))
    for i, sequence in enumerate(sequences):
        for j in sequence:
            results[i, j] = 1.
    return results
x_train = vectorize_sequences(train_data)
x_test = vectorize_sequences(test_data)
x_train[0]
y_train = np.asarray(train_labels).astype("float32")
y_test = np.asarray(test_labels).astype("float32")<jupyter_output><empty_output><jupyter_text>Building your model **Model definition**<jupyter_code>from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Dense(16, activation="relu"),
    layers.Dense(16, activation="relu"),
    layers.Dense(1, activation="sigmoid")
])<jupyter_output><empty_output><jupyter_text>**Compiling the model**<jupyter_code>model.compile(optimizer="rmsprop",
              loss="binary_crossentropy",
              metrics=["accuracy"])<jupyter_output><empty_output><jupyter_text>Validating your approach **Setting aside a validation set**<jupyter_code>x_val = x_train[:10000]
partial_x_train = x_train[10000:]
y_val = y_train[:10000]
partial_y_train = y_train[10000:]<jupyter_output><empty_output><jupyter_text>**Training your model**<jupyter_code>history = model.fit(partial_x_train,
                    partial_y_train,
                    epochs=20,
                    batch_size=512,
                    validation_data=(x_val, y_val))
history_dict = history.history
history_dict.keys()<jupyter_output><empty_output><jupyter_text>**Plotting the training and validation loss**<jupyter_code>import matplotlib.pyplot as plt
history_dict = history.history
loss_values = history_dict["loss"]
val_loss_values = history_dict["val_loss"]
epochs = range(1, len(loss_values) + 1)
plt.plot(epochs, loss_values, "bo", label="Training loss")
plt.plot(epochs, val_loss_values, "b", label="Validation loss")
plt.title("Training and validation loss")
plt.xlabel("Epochs")
plt.ylabel("Loss")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Plotting the training and validation accuracy**<jupyter_code>plt.clf()
acc = history_dict["accuracy"]
val_acc = history_dict["val_accuracy"]
plt.plot(epochs, acc, "bo", label="Training acc")
plt.plot(epochs, val_acc, "b", label="Validation acc")
plt.title("Training and validation accuracy")
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Retraining a model from scratch**<jupyter_code>model = keras.Sequential([
    layers.Dense(16, activation="relu"),
    layers.Dense(16, activation="relu"),
    layers.Dense(1, activation="sigmoid")
])
model.compile(optimizer="rmsprop",
              loss="binary_crossentropy",
              metrics=["accuracy"])
model.fit(x_train, y_train, epochs=4, batch_size=512)
results = model.evaluate(x_test, y_test)
results<jupyter_output><empty_output><jupyter_text>Using a trained model to generate predictions on new data<jupyter_code>model.predict(x_test)<jupyter_output><empty_output><jupyter_text>Further experiments  Wrapping up  Classifying newswires: A multiclass classification example  The Reuters dataset **Loading the Reuters dataset**<jupyter_code>from tensorflow.keras.datasets import reuters
(train_data, train_labels), (test_data, test_labels) = reuters.load_data(
    num_words=10000)
len(train_data)
len(test_data)
train_data[10]<jupyter_output><empty_output><jupyter_text>**Decoding newswires back to text**<jupyter_code>word_index = reuters.get_word_index()
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
decoded_newswire = " ".join([reverse_word_index.get(i - 3, "?") for i in
    train_data[0]])
train_labels[10]<jupyter_output><empty_output><jupyter_text>Preparing the data **Encoding the input data**<jupyter_code>x_train = vectorize_sequences(train_data)
x_test = vectorize_sequences(test_data)<jupyter_output><empty_output><jupyter_text>**Encoding the labels**<jupyter_code>def to_one_hot(labels, dimension=46):
    results = np.zeros((len(labels), dimension))
    for i, label in enumerate(labels):
        results[i, label] = 1.
    return results
y_train = to_one_hot(train_labels)
y_test = to_one_hot(test_labels)
from tensorflow.keras.utils import to_categorical
y_train = to_categorical(train_labels)
y_test = to_categorical(test_labels)<jupyter_output><empty_output><jupyter_text>Building your model **Model definition**<jupyter_code>model = keras.Sequential([
    layers.Dense(64, activation="relu"),
    layers.Dense(64, activation="relu"),
    layers.Dense(46, activation="softmax")
])<jupyter_output><empty_output><jupyter_text>**Compiling the model**<jupyter_code>model.compile(optimizer="rmsprop",
              loss="categorical_crossentropy",
              metrics=["accuracy"])<jupyter_output><empty_output><jupyter_text>Validating your approach **Setting aside a validation set**<jupyter_code>x_val = x_train[:1000]
partial_x_train = x_train[1000:]
y_val = y_train[:1000]
partial_y_train = y_train[1000:]<jupyter_output><empty_output><jupyter_text>**Training the model**<jupyter_code>history = model.fit(partial_x_train,
                    partial_y_train,
                    epochs=20,
                    batch_size=512,
                    validation_data=(x_val, y_val))<jupyter_output><empty_output><jupyter_text>**Plotting the training and validation loss**<jupyter_code>loss = history.history["loss"]
val_loss = history.history["val_loss"]
epochs = range(1, len(loss) + 1)
plt.plot(epochs, loss, "bo", label="Training loss")
plt.plot(epochs, val_loss, "b", label="Validation loss")
plt.title("Training and validation loss")
plt.xlabel("Epochs")
plt.ylabel("Loss")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Plotting the training and validation accuracy**<jupyter_code>plt.clf()
acc = history.history["accuracy"]
val_acc = history.history["val_accuracy"]
plt.plot(epochs, acc, "bo", label="Training accuracy")
plt.plot(epochs, val_acc, "b", label="Validation accuracy")
plt.title("Training and validation accuracy")
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Retraining a model from scratch**<jupyter_code>model = keras.Sequential([
  layers.Dense(64, activation="relu"),
  layers.Dense(64, activation="relu"),
  layers.Dense(46, activation="softmax")
])
model.compile(optimizer="rmsprop",
              loss="categorical_crossentropy",
              metrics=["accuracy"])
model.fit(x_train,
          y_train,
          epochs=9,
          batch_size=512)
results = model.evaluate(x_test, y_test)
results
import copy
test_labels_copy = copy.copy(test_labels)
np.random.shuffle(test_labels_copy)
hits_array = np.array(test_labels) == np.array(test_labels_copy)
hits_array.mean()<jupyter_output><empty_output><jupyter_text>Generating predictions on new data<jupyter_code>predictions = model.predict(x_test)
predictions[0].shape
np.sum(predictions[0])
np.argmax(predictions[0])<jupyter_output><empty_output><jupyter_text>A different way to handle the labels and the loss<jupyter_code>y_train = np.array(train_labels)
y_test = np.array(test_labels)
model.compile(optimizer="rmsprop",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])<jupyter_output><empty_output><jupyter_text>The importance of having sufficiently large intermediate layers **A model with an information bottleneck**<jupyter_code>model = keras.Sequential([
    layers.Dense(64, activation="relu"),
    layers.Dense(4, activation="relu"),
    layers.Dense(46, activation="softmax")
])
model.compile(optimizer="rmsprop",
              loss="categorical_crossentropy",
              metrics=["accuracy"])
model.fit(partial_x_train,
          partial_y_train,
          epochs=20,
          batch_size=128,
          validation_data=(x_val, y_val))<jupyter_output><empty_output><jupyter_text>Further experiments  Wrapping up  Predicting house prices: A regression example  The Boston Housing Price dataset **Loading the Boston housing dataset**<jupyter_code>from tensorflow.keras.datasets import boston_housing
(train_data, train_targets), (test_data, test_targets) = boston_housing.load_data()
train_data.shape
test_data.shape
train_targets<jupyter_output><empty_output><jupyter_text>Preparing the data **Normalizing the data**<jupyter_code>mean = train_data.mean(axis=0)
train_data -= mean
std = train_data.std(axis=0)
train_data /= std
test_data -= mean
test_data /= std<jupyter_output><empty_output><jupyter_text>Building your model **Model definition**<jupyter_code>def build_model():
    model = keras.Sequential([
        layers.Dense(64, activation="relu"),
        layers.Dense(64, activation="relu"),
        layers.Dense(1)
    ])
    model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])
    return model<jupyter_output><empty_output><jupyter_text>Validating your approach using K-fold validation **K-fold validation**<jupyter_code>k = 4
num_val_samples = len(train_data) // k
num_epochs = 100
all_scores = []
for i in range(k):
    print(f"Processing fold #{i}")
    val_data = train_data[i * num_val_samples: (i + 1) * num_val_samples]
    val_targets = train_targets[i * num_val_samples: (i + 1) * num_val_samples]
    partial_train_data = np.concatenate(
        [train_data[:i * num_val_samples],
         train_data[(i + 1) * num_val_samples:]],
        axis=0)
    partial_train_targets = np.concatenate(
        [train_targets[:i * num_val_samples],
         train_targets[(i + 1) * num_val_samples:]],
        axis=0)
    model = build_model()
    model.fit(partial_train_data, partial_train_targets,
              epochs=num_epochs, batch_size=16, verbose=0)
    val_mse, val_mae = model.evaluate(val_data, val_targets, verbose=0)
    all_scores.append(val_mae)
all_scores
np.mean(all_scores)<jupyter_output><empty_output><jupyter_text>**Saving the validation logs at each fold**<jupyter_code>num_epochs = 500
all_mae_histories = []
for i in range(k):
    print(f"Processing fold #{i}")
    val_data = train_data[i * num_val_samples: (i + 1) * num_val_samples]
    val_targets = train_targets[i * num_val_samples: (i + 1) * num_val_samples]
    partial_train_data = np.concatenate(
        [train_data[:i * num_val_samples],
         train_data[(i + 1) * num_val_samples:]],
        axis=0)
    partial_train_targets = np.concatenate(
        [train_targets[:i * num_val_samples],
         train_targets[(i + 1) * num_val_samples:]],
        axis=0)
    model = build_model()
    history = model.fit(partial_train_data, partial_train_targets,
                        validation_data=(val_data, val_targets),
                        epochs=num_epochs, batch_size=16, verbose=0)
    mae_history = history.history["val_mae"]
    all_mae_histories.append(mae_history)<jupyter_output><empty_output><jupyter_text>**Building the history of successive mean K-fold validation scores**<jupyter_code>average_mae_history = [
    np.mean([x[i] for x in all_mae_histories]) for i in range(num_epochs)]<jupyter_output><empty_output><jupyter_text>**Plotting validation scores**<jupyter_code>plt.plot(range(1, len(average_mae_history) + 1), average_mae_history)
plt.xlabel("Epochs")
plt.ylabel("Validation MAE")
plt.show()<jupyter_output><empty_output><jupyter_text>**Plotting validation scores, excluding the first 10 data points**<jupyter_code>truncated_mae_history = average_mae_history[10:]
plt.plot(range(1, len(truncated_mae_history) + 1), truncated_mae_history)
plt.xlabel("Epochs")
plt.ylabel("Validation MAE")
plt.show()<jupyter_output><empty_output><jupyter_text>**Training the final model**<jupyter_code>model = build_model()
model.fit(train_data, train_targets,
          epochs=130, batch_size=16, verbose=0)
test_mse_score, test_mae_score = model.evaluate(test_data, test_targets)
test_mae_score<jupyter_output><empty_output><jupyter_text>Generating predictions on new data<jupyter_code>predictions = model.predict(test_data)
predictions[0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12411637770667461, 'fraction_numerical': 0.013840315499665154, 'mean_word_length': 5.899383983572895, 'pattern_counts': {'":': 0, '<': 141, '<?xml version=': 0, '>': 141, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '35077', 'license': 'MIT', 'n_tokens_mistral': 4800, 'n_tokens_neox': 4560, 'n_words': 822, 'path': 'chapter04_getting-started-with-neural-networks.ipynb', 'repo_name': 'ml-ninja/deep-learning-with-python-notebooks', 'size': '13439'}