<jupyter_start><jupyter_text>End-to-End NLP: News Headline Classifier (Local Version)_**Train a Keras-based model to classify news headlines between four domains**_This notebook works well with the `Python 3 (TensorFlow 2.3 Python 3.7 CPU Optimized)` kernel on SageMaker Studio, or `conda_tensorflow2_p37` on classic SageMaker Notebook Instances.---In this version, the model is trained and evaluated here on the notebook instance itself. We'll show in the follow-on notebook how to take advantage of Amazon SageMaker to separate these infrastructure needs.Note that you can safely ignore the WARNING about the pip version.<jupyter_code># First install some libraries which might not be available across all kernels (e.g. in Studio):
!pip install ipywidgets<jupyter_output><empty_output><jupyter_text>Download News Aggregator DatasetWe will download **FastAI AG News** dataset from the [Registry of Open Data on AWS](https://registry.opendata.aws/fast-ai-nlp/) public repository. This dataset contains a table of news headlines and their corresponding classes.<jupyter_code>local_dir = "data"
# Download the AG News data from the Registry of Open Data on AWS.
!rm -rf {local_dir}
!mkdir -p {local_dir}
!aws s3 cp s3://fast-ai-nlp/ag_news_csv.tgz {local_dir} --no-sign-request

# Un-tar the AG News data.
!tar zxf {local_dir}/ag_news_csv.tgz -C {local_dir}/ --strip-components=1 --no-same-owner<jupyter_output><empty_output><jupyter_text>Let's visualize the datasetWe will load the ag_news_csv/train.csv file to a Pandas dataframe for our data processing work.<jupyter_code>%load_ext autoreload
%autoreload 2

import os
import re
import numpy as np
import pandas as pd

import util.preprocessing
column_names = ["CATEGORY", "TITLE", "CONTENT"]
# we use the train.csv only
df = pd.read_csv(f"{local_dir}/train.csv", names=column_names, header=None, delimiter=",")
# shuffle the DataFrame rows
df = df.sample(frac=1, random_state=1337)
# make the category classes more readable
mapping = {1: 'World', 2: 'Sports', 3: 'Business', 4: 'Sci/Tech'}
df = df.replace({'CATEGORY': mapping})
df.head()<jupyter_output><empty_output><jupyter_text>For this exercise we'll **only use**:- The **title** (Headline) of the news story, as our input- The **category**, as our target variable<jupyter_code>df["CATEGORY"].value_counts()<jupyter_output><empty_output><jupyter_text>The dataset has **four article categories** with equal weighting:- Business- Sci/Tech- Sports- World  Natural Language Pre-ProcessingWe'll do some basic processing of the text data to convert it into numerical form that the algorithm will be able to consume to create a model.We will do typical pre processing for NLP workloads such as: dummy encoding the labels, tokenizing the documents and set fixed sequence lengths for input feature dimension, padding documents to have fixed length input vectors.  Dummy Encode the Labels<jupyter_code>encoded_y, labels = util.preprocessing.dummy_encode_labels(df, "CATEGORY")
print(labels)

df["CATEGORY"].iloc[0]
encoded_y[0]<jupyter_output><empty_output><jupyter_text>Tokenize and Set Fixed Sequence LengthsWe want to describe our inputs at the more meaningful word level (rather than individual characters), and ensure a fixed length of the input feature dimension.<jupyter_code>padded_docs, tokenizer = util.preprocessing.tokenize_and_pad_docs(df, "TITLE")

df["TITLE"].iloc[0]
padded_docs[0]<jupyter_output><empty_output><jupyter_text>Import Word EmbeddingsTo represent our words in numeric form, we'll use pre-trained vector representations for each word in the vocabulary: In this case we'll be using [pre-trained word embeddings from FastText](https://fasttext.cc/docs/en/crawl-vectors.html), which are also available for a broad range of languages other than English.You could also explore training custom, domain-specific word embeddings using SageMaker's built-in [BlazingText algorithm](https://docs.aws.amazon.com/sagemaker/latest/dg/blazingtext.html). See the official [blazingtext_word2vec_text8 sample](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/introduction_to_amazon_algorithms/blazingtext_word2vec_text8) for an example notebook showing how.<jupyter_code>%%time
embedding_matrix = util.preprocessing.get_word_embeddings(tokenizer, f"{local_dir}/embeddings")

np.save(
    file=f"{local_dir}/embeddings/docs-embedding-matrix",
    arr=embedding_matrix,
    allow_pickle=False,
)
vocab_size=embedding_matrix.shape[0]
print(embedding_matrix.shape)<jupyter_output><empty_output><jupyter_text>Split Train and Test SetsFinally we need to divide our data into model training and evaluation sets:<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    padded_docs,
    encoded_y,
    test_size=0.2,
    random_state=42
)

# Do you always remember to save your datasets for traceability when experimenting locally? ;-)
os.makedirs(f"{local_dir}/train", exist_ok=True)
np.save(f"{local_dir}/train/train_X.npy", X_train)
np.save(f"{local_dir}/train/train_Y.npy", y_train)
os.makedirs(f"{local_dir}/test", exist_ok=True)
np.save(f"{local_dir}/test/test_X.npy", X_test)
np.save(f"{local_dir}/test/test_Y.npy", y_test)<jupyter_output><empty_output><jupyter_text>Define the Model<jupyter_code>import tensorflow as tf
from tensorflow.keras.layers import Conv1D, Dense, Dropout, Embedding, Flatten, MaxPooling1D
from tensorflow.keras.models import Sequential

seed = 42
np.random.seed(seed)
num_classes=len(labels)

model = Sequential()
model.add(Embedding(
    embedding_matrix.shape[0],  # Final vocabulary size
    embedding_matrix.shape[1],  # Word vector dimensions
    weights=[embedding_matrix],
    input_length=40,
    trainable=False,
    name="embed"
))
model.add(Conv1D(filters=128, kernel_size=3, activation="relu", name="conv_1"))
model.add(MaxPooling1D(pool_size=5, name="maxpool_1"))
model.add(Flatten(name="flat_1"))
model.add(Dropout(0.3, name="dropout_1"))
model.add(Dense(128, activation="relu", name="dense_1"))
model.add(Dense(num_classes, activation="softmax", name="out_1"))

# Compile the model
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)
model.compile(optimizer=optimizer, loss="binary_crossentropy", metrics=["acc"])

model.summary()<jupyter_output><empty_output><jupyter_text>Fit (Train) and Evaluate the Model<jupyter_code>%%time
# fit the model here in the notebook:
print("Training model")
model.fit(X_train, y_train, batch_size=16, epochs=5, verbose=1)
print("Evaluating model")
# TODO: Better differentiate train vs val loss in logs
scores = model.evaluate(X_test, y_test, verbose=2)
print(
    "Validation results: "
    + "; ".join(map(
        lambda i: f"{model.metrics_names[i]}={scores[i]:.5f}", range(len(model.metrics_names))
    ))
)<jupyter_output><empty_output><jupyter_text>Use the Model (Locally)Let's evaluate our model with some example headlines...If you struggle with the widget, you can always simply call the `classify()` function from Python. You can be creative with your headlines!<jupyter_code>from IPython import display
import ipywidgets as widgets
from tensorflow.keras.preprocessing.sequence import pad_sequences

def classify(text):
    """Classify a headline and print the results"""
    encoded_example = tokenizer.texts_to_sequences([text])
    # Pad documents to a max length of 40 words
    max_length = 40
    padded_example = pad_sequences(encoded_example, maxlen=max_length, padding="post")
    result = model.predict(padded_example)
    print(result)
    ix = np.argmax(result)
    print(f"Predicted class: '{labels[ix]}' with confidence {result[0][ix]:.2%}")

interaction = widgets.interact_manual(
    classify,
    text=widgets.Text(
        value="The markets were bullish after news of the merger",
        placeholder="Type a news headline...",
        description="Headline:",
        layout=widgets.Layout(width="99%"),
    )
)
interaction.widget.children[1].description = "Classify!"<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10751343917989749, 'fraction_numerical': 0.009876234529316165, 'mean_word_length': 6.073386383731211, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '597342', 'license': 'MIT', 'n_tokens_mistral': 2619, 'n_tokens_neox': 2510, 'n_words': 720, 'path': '03_customer_tensorflow_keras_nlp/Headline Classifier Local.ipynb', 'repo_name': 'tom5610/amazon_sagemaker_intermediate_workshop', 'size': '7999'}