<jupyter_start><jupyter_text>This code example trains an encoder-decoder network to generate textual descriptions of images. The architecture resembles the language translation network in c14e1_seq2seq2_translate, but the encoder is a convolutional network instead of a recurrent network. Additionally, the decoder makes use of attention. The implementation relies on precomputed feature vectors from c16e1_create_vectors. More context for this code example can be found in the section "Programming Example: Attention-Based Image Captioning" in Chapter 16 in the book Learning Deep Learning by <NAME> (ISBN: 9780137470358).We start with import statements in the code snippet below.<jupyter_code>import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import Embedding
from tensorflow.keras.layers import LSTM
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Attention
from tensorflow.keras.layers import Concatenate
from tensorflow.keras.layers import GlobalAveragePooling2D
from tensorflow.keras.layers import Reshape
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.text import \
    text_to_word_sequence
from tensorflow.keras.applications import VGG19
from tensorflow.keras.applications.vgg19 import \
    preprocess_input
from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.utils import Sequence
from tensorflow.keras.preprocessing.sequence import \
    pad_sequences
import pickle
import gzip
import logging
tf.get_logger().setLevel(logging.ERROR)<jupyter_output><empty_output><jupyter_text>Initialization statements for our program are found below. They are similar to what we used in the language translation example, but some of the lines deserve further attention. The variable READ_IMAGES can be used to limit the number of images that we use for training. We set it to 90,000, which is more than the total number of images we have. You can decrease it if necessary (e.g., if you run into memory limits of your machine). We also provide the paths to four files that we will use as test images. You can replace those to point to images of your own choice when you run this experiment.<jupyter_code>EPOCHS = 20
BATCH_SIZE = 128
MAX_WORDS = 10000
READ_IMAGES = 90000
LAYER_SIZE = 256
EMBEDDING_WIDTH = 128
OOV_WORD = 'UNK'
PAD_INDEX = 0
OOV_INDEX = 1
START_INDEX = MAX_WORDS - 2
STOP_INDEX = MAX_WORDS - 1
MAX_LENGTH = 60
TRAINING_FILE_DIR = 'tf_data/feature_vectors/'
TEST_FILE_DIR = '../data/test_images/'
TEST_IMAGES = ['boat.jpg',
               'cat.jpg',
               'table.jpg',
               'bird.jpg']<jupyter_output><empty_output><jupyter_text>The next code snippet shows the functions we use to read the image captions. The function to read the captions reads the pickled directory file that we previously prepared. From this, we create a list image_paths with the filenames for the feature vectors and one list, dest_word_sequences, which contains the first image caption for each image. To keep things simple, we simply discard the alternative captions for each image.The list dest_word_sequences is equivalent to the destination language sentence in the language translation example. This function does not load all the feature vectors but just the paths to them. The reason for this is that the feature vectors for all the images consume a fair amount of space, so for many machines, it would be impractical to hold the entire dataset in memory during training. Instead, we read the feature vectors on the fly when they are needed. This is a common technique when working with large datasets.<jupyter_code># Function to read file.
def read_training_file(file_name, max_len):
    pickle_file = gzip.open(file_name, 'rb')
    image_dict = pickle.load(pickle_file)
    pickle_file.close()
    image_paths = []
    dest_word_sequences = []
    for i, key in enumerate(image_dict):
        if i == READ_IMAGES:
            break
        image_item = image_dict[key]
        image_paths.append(image_item[0])
        caption = image_item[1]
        word_sequence = text_to_word_sequence(caption)
        dest_word_sequence = word_sequence[0:max_len]
        dest_word_sequences.append(dest_word_sequence)
    return image_paths, dest_word_sequences<jupyter_output><empty_output><jupyter_text>The next code snippet contains functions to tokenize and untokenize the sentences. These are similar, if not identical, to what we used in c14e1_seq2seq_translate. We finally call the functions to read and tokenize the image captions.<jupyter_code># Functions to tokenize and un-tokenize sequences.
def tokenize(sequences):
    tokenizer = Tokenizer(num_words=MAX_WORDS-2,
                          oov_token=OOV_WORD)
    tokenizer.fit_on_texts(sequences)
    token_sequences = tokenizer.texts_to_sequences(sequences)
    return tokenizer, token_sequences

def tokens_to_words(tokenizer, seq):
    word_seq = []
    for index in seq:
        if index == PAD_INDEX:
            word_seq.append('PAD')
        elif index == OOV_INDEX:
            word_seq.append(OOV_WORD)
        elif index == START_INDEX:
            word_seq.append('START')
        elif index == STOP_INDEX:
            word_seq.append('STOP')
        else:
            word_seq.append(tokenizer.sequences_to_texts(
                [[index]])[0])
    print(word_seq)

# Read files.
image_paths, dest_seq = read_training_file(TRAINING_FILE_DIR \
    + 'caption_file.pickle.gz', MAX_LENGTH)
dest_tokenizer, dest_token_seq = tokenize(dest_seq)<jupyter_output><empty_output><jupyter_text>As previously mentioned, we cannot afford to keep the entire dataset in memory during training but need to create our training batches on the fly. We handle this task by creating a class that inherits from the Keras Sequence class in the code snippet below. In the constructor, we supply the paths to the feature vectors, as well as the tokenized captions, and the batch size. Just as for the language translation example, the recurrent network in the decoder will need the tokenized data both as input and output but shifted by one location and with a START token on the input side. This explains why we provide two variables dest_input_data and dest_target_data to the constructor. We also need to provide the batch size.The __len__() method is expected to provide the number of batches that our dataset provides, which is simply the number of images divided by the batch size.The main functionality in the class is the __getitem__() method, which is expected to return the training data for the batch number indicated by the argument idx. The output format of this method depends on what our network requires as input. For a single training example, our network needs a set of feature vectors as input from the encoder side and a shifted version of the target sentence as input to the decoder recurrent network. It also needs the original version of the target sentence as the desired output for the network. Thus, the output from this method should be a list with two elements representing the two inputs and a single element representing the output. The details become clearer when we later build our training network. There is one more thing to consider, though. The __getitem__() method is expected to return a batch instead of a single training example, so each of the three items we described will be an array where the number of elements is determined by the batch size.<jupyter_code># Sequence class to create batches on the fly.
class ImageCaptionSequence(Sequence):
    def __init__(self, image_paths, dest_input_data,
                 dest_target_data, batch_size):
        self.image_paths = image_paths
        self.dest_input_data = dest_input_data
        self.dest_target_data = dest_target_data
        self.batch_size = batch_size

    def __len__(self):
        return int(np.ceil(len(self.dest_input_data) /
            float(self.batch_size)))

    def __getitem__(self, idx):
        batch_x0 = self.image_paths[
            idx * self.batch_size:(idx + 1) * self.batch_size]
        batch_x1 = self.dest_input_data[
            idx * self.batch_size:(idx + 1) * self.batch_size]
        batch_y = self.dest_target_data[
            idx * self.batch_size:(idx + 1) * self.batch_size]
        image_features = []
        for image_id in batch_x0:
            file_name = TRAINING_FILE_DIR \
                + image_id + '.pickle.gzip'
            pickle_file = gzip.open(file_name, 'rb')
            feature_vector = pickle.load(pickle_file)
            pickle_file.close()
            image_features.append(feature_vector)
        return [np.array(image_features),
                np.array(batch_x1)], np.array(batch_y)<jupyter_output><empty_output><jupyter_text>The constructor for the ImageCaptionSequence class above assumes that we already have created three arrays with appropriate input data. Two of these arrays (for the recurrent network in the decoder) directly correspond to what we created in the language translation example c14e1_seq2seq_translate. This is shown in the code snippet below, where we also call the constructor for ImageCaptionSequence.<jupyter_code># Prepare training data.
dest_target_token_seq = [x + [STOP_INDEX] for x in dest_token_seq]
dest_input_token_seq = [[START_INDEX] + x for x in
                        dest_target_token_seq]
dest_input_data = pad_sequences(dest_input_token_seq,
                                padding='post')
dest_target_data = pad_sequences(
    dest_target_token_seq, padding='post',
    maxlen=len(dest_input_data[0]))
image_sequence = ImageCaptionSequence(
    image_paths, dest_input_data, dest_target_data, BATCH_SIZE)<jupyter_output><empty_output><jupyter_text>We are now ready to define the encoder and decoder models and connect them. For a detailed description, including figures, please consult the description in the book. The architecture is a typical encoder-decoder architecture, although most of the encoding has already been done offline. The code snippet below shows the implementation of the encoder model.<jupyter_code># Build encoder model.
# Input is feature vector.
feature_vector_input = Input(shape=(14, 14, 512))

# Create the encoder layers.
enc_mean_layer = GlobalAveragePooling2D()
enc_layer_h = Dense(LAYER_SIZE)
enc_layer_c = Dense(LAYER_SIZE)

# Connect the encoding layers.
enc_mean_layer_output = enc_mean_layer(feature_vector_input)
enc_layer_h_outputs = enc_layer_h(enc_mean_layer_output)
enc_layer_c_outputs = enc_layer_c(enc_mean_layer_output)

# Organize the output state for encoder layers.
enc_layer_outputs = [enc_layer_h_outputs, enc_layer_c_outputs]

# Build the model.
enc_model_top = Model(feature_vector_input, enc_layer_outputs)
enc_model_top.summary()<jupyter_output><empty_output><jupyter_text>The next code snippet implements the decoder model.<jupyter_code># Build decoder model.
# Input to the network is feature_vector, image caption
# sequence, and intermediate state.
dec_feature_vector_input = Input(shape=(14, 14, 512))
dec_embedding_input = Input(shape=(None, ))
dec_layer1_state_input_h = Input(shape=(LAYER_SIZE,))
dec_layer1_state_input_c = Input(shape=(LAYER_SIZE,))

# Create the decoder layers.
dec_reshape_layer = Reshape((196, 512),
                            input_shape=(14, 14, 512,))
dec_attention_layer = Attention()
dec_query_layer = Dense(512)
dec_embedding_layer = Embedding(output_dim=EMBEDDING_WIDTH,
                                input_dim=MAX_WORDS,
                                mask_zero=False)
dec_layer1 = LSTM(LAYER_SIZE, return_state=True,
                  return_sequences=True)
dec_concat_layer = Concatenate()
dec_layer2 = Dense(MAX_WORDS, activation='softmax')

# Connect the decoder layers.
dec_embedding_layer_outputs = dec_embedding_layer(
    dec_embedding_input)
dec_reshape_layer_outputs = dec_reshape_layer(
    dec_feature_vector_input)
dec_layer1_outputs, dec_layer1_state_h, dec_layer1_state_c = \
    dec_layer1(dec_embedding_layer_outputs, initial_state=[
        dec_layer1_state_input_h, dec_layer1_state_input_c])
dec_query_layer_outputs = dec_query_layer(dec_layer1_outputs)
dec_attention_layer_outputs = dec_attention_layer(
    [dec_query_layer_outputs, dec_reshape_layer_outputs])
dec_layer2_inputs = dec_concat_layer(
    [dec_layer1_outputs, dec_attention_layer_outputs])
dec_layer2_outputs = dec_layer2(dec_layer2_inputs)

# Build the model.
dec_model = Model([dec_feature_vector_input,
                   dec_embedding_input,
                   dec_layer1_state_input_h,
                   dec_layer1_state_input_c],
                  [dec_layer2_outputs, dec_layer1_state_h,
                   dec_layer1_state_c])
dec_model.summary()<jupyter_output><empty_output><jupyter_text>Finally, we create a joint model from the encoder and decoder in the code snippet below. This model will be used for training. Just as in the text translation example, we discard the state outputs from the decoder in this joint model. There is no need for explicit state management for this joint model because TensorFlow does it for us during training.<jupyter_code># Build and compile full training model.
# We do not use the state output when training.
train_feature_vector_input = Input(shape=(14, 14, 512))
train_dec_embedding_input = Input(shape=(None, ))
intermediate_state = enc_model_top(train_feature_vector_input)
train_dec_output, _, _ = dec_model([train_feature_vector_input,
                                    train_dec_embedding_input] +
                                    intermediate_state)
training_model = Model([train_feature_vector_input,
                        train_dec_embedding_input],
                        [train_dec_output])
training_model.compile(loss='sparse_categorical_crossentropy',
                       optimizer='adam', metrics =['accuracy'])
training_model.summary()<jupyter_output><empty_output><jupyter_text>Just as for the language translation example, we use the encoder and decoder separately during inference. However, in this image-captioning example, the encoder also needs to include the VGG19 layers, as we will not do inference on precomputed feature vectors. We therefore create yet another model in the code snippet below, which consists of the VGG19 network (except for the top layers) followed by our decoder model.<jupyter_code># Build full encoder model for inference.
conv_model = VGG19(weights='imagenet')
conv_model_outputs = conv_model.get_layer('block5_conv4').output
intermediate_state = enc_model_top(conv_model_outputs)
inference_enc_model = Model([conv_model.input],
                            intermediate_state
                            + [conv_model_outputs])
inference_enc_model.summary()<jupyter_output><empty_output><jupyter_text>We are finally ready to train and evaluate our model, and the code is found below. One key difference compared to past code examples is that instead of providing the training set, we provide the image_sequence object as argument to the fit() function. The image_sequence object will provide the training data batch by batch as the feature vectors are read from disk.After each training epoch, we run through our four test images. The process for this is similar to what we did in the language translation example but with one difference. Instead of running an input sentence through the encoder model that was based on a recurrent network, we read an image from disk, preprocess it, and run it through our encoder model that is based on the convolutional VGG19 network.<jupyter_code>for i in range(EPOCHS): # Train and evaluate model
    print('step: ' , i)
    history = training_model.fit(image_sequence, epochs=1)
    for filename in TEST_IMAGES:
        # Determine dimensions.
        image = load_img(TEST_FILE_DIR + filename)
        width = image.size[0]
        height = image.size[1]

        # Resize so shortest side is 256 pixels.
        if height > width:
            image = load_img(
                TEST_FILE_DIR + filename,
                target_size=(int(height/width*256), 256))
        else:
            image = load_img(
                TEST_FILE_DIR + filename,
                target_size=(256, int(width/height*256)))
        width = image.size[0]
        height = image.size[1]
        image_np = img_to_array(image)

        # Crop to center 224x224 region.
        h_start = int((height-224)/2)
        w_start = int((width-224)/2)
        image_np = image_np[h_start:h_start+224,
                            w_start:w_start+224]

        # Run image through encoder.
        image_np = np.expand_dims(image_np, axis=0)
        x = preprocess_input(image_np)
        dec_layer1_state_h, dec_layer1_state_c, feature_vector = \
            inference_enc_model.predict(x, verbose=0)

        # Predict sentence word for word.
        prev_word_index = START_INDEX
        produced_string = ''
        pred_seq = []
        for j in range(MAX_LENGTH):
            x = np.reshape(np.array(prev_word_index), (1, 1))
            preds, dec_layer1_state_h, dec_layer1_state_c = \
                dec_model.predict(
                    [feature_vector, x, dec_layer1_state_h,
                     dec_layer1_state_c], verbose=0)
            prev_word_index = np.asarray(preds[0][0]).argmax()
            pred_seq.append(prev_word_index)
            if prev_word_index == STOP_INDEX:
                break
        tokens_to_words(dest_tokenizer, pred_seq)
        print('\n\n')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06093776448682503, 'fraction_numerical': 0.011454042769282852, 'mean_word_length': 3.8163043478260867, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '244643', 'license': 'MIT', 'n_tokens_mistral': 5141, 'n_tokens_neox': 4884, 'n_words': 1712, 'path': 'tf_framework/c16e2_image_captioning.ipynb', 'repo_name': 'praneetheddu/LDL', 'size': '17729'}