<jupyter_start><jupyter_text>Sentiment Prediction in IMDB Reviews using an LSTM<jupyter_code>import tempfile
import os
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as fn
import fastestimator as fe
from fastestimator.dataset.data import imdb_review
from fastestimator.op.numpyop.univariate.reshape import Reshape
from fastestimator.op.tensorop.loss import CrossEntropy
from fastestimator.op.tensorop.model import ModelOp, UpdateOp
from fastestimator.trace.io import BestModelSaver
from fastestimator.trace.metric import Accuracy
from fastestimator.backend import load_model
MAX_WORDS = 10000
MAX_LEN = 500
batch_size = 64
epochs = 10
max_train_steps_per_epoch = None
max_eval_steps_per_epoch = None<jupyter_output><empty_output><jupyter_text>Building components  Step 1: Prepare training & evaluation data and define a `Pipeline` We are loading the dataset from tf.keras.datasets.imdb which contains movie reviews and sentiment scores. All the words have been replaced with the integers that specifies the popularity of the word in corpus. To ensure all the sequences are of same length we need to pad the input sequences before defining the `Pipeline`.<jupyter_code>train_data, eval_data = imdb_review.load_data(MAX_LEN, MAX_WORDS)
pipeline = fe.Pipeline(train_data=train_data,
                       eval_data=eval_data,
                       batch_size=batch_size,
                       ops=Reshape(1, inputs="y", outputs="y"))<jupyter_output><empty_output><jupyter_text>Step 2: Create a `model` and FastEstimator `Network` First, we have to define the neural network architecture, and then pass the definition, associated model name, and optimizer into fe.build:<jupyter_code>class ReviewSentiment(nn.Module):
    def __init__(self, embedding_size=64, hidden_units=64):
        super().__init__()
        self.embedding = nn.Embedding(MAX_WORDS, embedding_size)
        self.conv1d = nn.Conv1d(in_channels=64, out_channels=32, kernel_size=3, padding=1)
        self.maxpool1d = nn.MaxPool1d(kernel_size=4)
        self.lstm = nn.LSTM(input_size=125, hidden_size=hidden_units, num_layers=1)
        self.fc1 = nn.Linear(in_features=hidden_units, out_features=250)
        self.fc2 = nn.Linear(in_features=250, out_features=1)

    def forward(self, x):
        x = self.embedding(x)
        x = x.permute((0, 2, 1))
        x = self.conv1d(x)
        x = fn.relu(x)
        x = self.maxpool1d(x)
        output, _ = self.lstm(x)
        x = output[:, -1]  # sequence output of only last timestamp
        x = fn.tanh(x)
        x = self.fc1(x)
        x = fn.relu(x)
        x = self.fc2(x)
        x = fn.sigmoid(x)
        return x<jupyter_output><empty_output><jupyter_text>`Network` is the object that defines the whole training graph, including models, loss functions, optimizers etc. A `Network` can have several different models and loss functions (ex. GANs). `fe.Network` takes a series of operators, in this case just the basic `ModelOp`, loss op, and `UpdateOp` will suffice. It should be noted that "y_pred" is the key in the data dictionary which will store the predictions.<jupyter_code>model = fe.build(model_fn=lambda: ReviewSentiment(), optimizer_fn="adam")
network = fe.Network(ops=[
    ModelOp(model=model, inputs="x", outputs="y_pred"),
    CrossEntropy(inputs=("y_pred", "y"), outputs="loss"),
    UpdateOp(model=model, loss_name="loss")
])<jupyter_output><empty_output><jupyter_text>Step 3: Prepare `Estimator` and configure the training loop `Estimator` is the API that wraps the `Pipeline`, `Network` and other training metadata together. `Estimator` also contains `Traces`, which are similar to the callbacks of Keras. In the training loop, we want to measure the validation loss and save the model that has the minimum loss. `BestModelSaver` is a convenient `Trace` to achieve this. Let's also measure accuracy over time using another `Trace`:<jupyter_code>model_dir = tempfile.mkdtemp()
traces = [Accuracy(true_key="y", pred_key="y_pred"), BestModelSaver(model=model, save_dir=model_dir)]
estimator = fe.Estimator(network=network,
                         pipeline=pipeline,
                         epochs=epochs,
                         traces=traces,
                         max_train_steps_per_epoch=max_train_steps_per_epoch,
                         max_eval_steps_per_epoch=max_eval_steps_per_epoch)<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>estimator.fit()<jupyter_output>______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Start: step: 1; model_lr: 0.001;<jupyter_text>Inferencing For inferencing, first we have to load the trained model weights. We previously saved model weights corresponding to our minimum loss, and now we will load the weights using `load_model()`:<jupyter_code>model_name = 'model_best_loss.pt'
model_path = os.path.join(model_dir, model_name)
load_model(model, model_path)<jupyter_output>Loaded model weights from /tmp/tmp69qyfzvm/model_best_loss.pt<jupyter_text>Let's get some random sequence and compare the prediction with the ground truth:<jupyter_code>selected_idx = np.random.randint(10000)
print("Ground truth is: ",eval_data[selected_idx]['y'])<jupyter_output>Ground truth is:  0<jupyter_text>Create data dictionary for the inference. The `Transform()` function in Pipeline and Network applies all the operations on the given data:<jupyter_code>infer_data = {"x":eval_data[selected_idx]['x'], "y":eval_data[selected_idx]['y']}
data = pipeline.transform(infer_data, mode="infer")
data = network.transform(data, mode="infer")<jupyter_output><empty_output><jupyter_text>Finally, print the inferencing results.<jupyter_code>print("Prediction for the input sequence: ", np.array(data["y_pred"])[0][0])<jupyter_output>Prediction for the input sequence:  0.30634004
{'fraction_non_alphanumeric': 0.10164835164835165, 'fraction_numerical': 0.012281835811247576, 'mean_word_length': 3.7865429234338746, 'pattern_counts': {'":': 2, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '344786', 'license': 'Apache-2.0', 'n_tokens_mistral': 2018, 'n_tokens_neox': 1937, 'n_words': 504, 'path': 'apphub/NLP/imdb/imdb.ipynb', 'repo_name': 'purujitb/fastestimator', 'size': '6188'}