<jupyter_start><jupyter_text>Artificial Intelligence Nanodegree Machine Translation Project Introduction- **Preprocess** - Convert text to sequence of integers.- **Models** Create models which accepts a sequence of integers as input and returns a probability distribution over possible translations. After learning about the basic types of neural networks that are often used for machine translation, you will engage in your own investigations, to design your own model!- **Prediction** Run the model on spanish text. DatasetDatasets for machine translation from [WMT](http://www.statmt.org/).<jupyter_code>import collections
import numpy as np

from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import Model
from keras.layers import GRU, LSTM, Input, Dense, TimeDistributed, Activation, RepeatVector, Bidirectional
from keras.layers.embeddings import Embedding
from keras.optimizers import Adam
from keras.models import Sequential
from keras.losses import sparse_categorical_crossentropy
# Load spanish data
spanish_sentences = [line.rstrip('\n') for line in open('data/eng.txt')]
# Load english data
english_sentences = [line.rstrip('\n') for line in open('data/span.txt')]
print('Dataset Loaded')

# Filter - running out of memory
spanish_sentences = spanish_sentences[10000:20000]
english_sentences = english_sentences[10000:20000]<jupyter_output>Dataset Loaded<jupyter_text>FilesEach line contains an spanish sentence with the respective translation in english.  View the first two lines from each file.<jupyter_code>for sample_i in range(10):
    print('small_vocab_en Line {}:  {}'.format(sample_i + 1, spanish_sentences[sample_i]))
    print('small_vocab_fr Line {}:  {}'.format(sample_i + 1, english_sentences[sample_i]))
import collections

spanish_words_counter = collections.Counter([word for sentence in spanish_sentences for word in sentence.split()])
english_words_counter = collections.Counter([word for sentence in english_sentences for word in sentence.split()])

print('{} spanish words.'.format(len([word for sentence in spanish_sentences for word in sentence.split()])))
print('{} unique spanish words.'.format(len(spanish_words_counter)))
print('10 Most common words in the spanish dataset:')
print('"' + '" "'.join(list(zip(*spanish_words_counter.most_common(10)))[0]) + '"')
print()
print('{} english words.'.format(len([word for sentence in english_sentences for word in sentence.split()])))
print('{} unique english words.'.format(len(english_words_counter)))
print('10 Most common words in the english dataset:')
print('"' + '" "'.join(list(zip(*english_words_counter.most_common(10)))[0]) + '"')<jupyter_output>38005 spanish words.
4501 unique spanish words.
10 Most common words in the spanish dataset:
"I" "Tom" "is" "a" "you" "He" "to" "the" "You" "I'm"

36647 english words.
7446 unique english words.
10 Most common words in the english dataset:
"es" "Tom" "a" "No" "un" "de" "Él" "la" "está" "el"<jupyter_text>For comparison, _Alice's Adventures in Wonderland_ contains 2,766 unique words of a total of 15,500 words. PreprocessFor this project, you won't use text data as input to your model. Instead, you'll convert the text into sequences of integers using the following preprocess methods:1. Tokenize the words into ids2. Add padding to make all the sequences the same length.Time to start preprocessing the data... Tokenize (IMPLEMENTATION)For a neural network to predict on text data, it first has to be turned into data it can understand. Text data like "dog" is a sequence of ASCII character encodings.  Since a neural network is a series of multiplication and addition operations, the input data needs to be number(s).We can turn each character into a number or each word into a number.  These are called character and word ids, respectively.  Character ids are used for character level models that generate text predictions for each character.  A word level model uses word ids that generate text predictions for each word.  Word level models tend to learn better, since they are lower in complexity, so we'll use those.Turn each sentence into a sequence of words ids using Keras's [`Tokenizer`](https://keras.io/preprocessing/text/tokenizer) function. Use this function to tokenize `spanish_sentences` and `english_sentences` in the cell below.Running the cell will run `tokenize` on sample data and show output for debugging.<jupyter_code>import keras
def tokenize(x):
    """
    Tokenize x
    :param x: List of sentences/strings to be tokenized
    :return: Tuple of (tokenized x data, tokenizer used to tokenize x)
    """
    x_tk = Tokenizer(char_level=False,
                     filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~',
                     lower=True)
    x_tk.fit_on_texts(x)

    return x_tk.texts_to_sequences(x), x_tk

# Tokenize Example output
text_sentences = [
    'The quick brown fox jumps over the lazy dog .',
    'By Jove , my quick study of lexicography won a prize .',
    'This is a short sentence .']
text_tokenized, text_tokenizer = tokenize(text_sentences)
print(text_tokenizer.word_index)
print()
for sample_i, (sent, token_sent) in enumerate(zip(text_sentences, text_tokenized)):
    print('Sequence {} in x'.format(sample_i + 1))
    print('  Input:  {}'.format(sent))
    print('  Output: {}'.format(token_sent))<jupyter_output>{'short': 20, 'of': 14, 'lexicography': 15, 'dog': 9, 'over': 7, 'quick': 2, 'my': 12, 'prize': 17, 'lazy': 8, 'sentence': 21, 'is': 19, 'jumps': 6, 'fox': 5, 'a': 3, 'the': 1, 'won': 16, 'by': 10, 'jove': 11, 'brown': 4, 'study': 13, 'this': 18}

Sequence 1 in x
  Input:  The quick brown fox jumps over the lazy dog .
  Output: [1, 2, 4, 5, 6, 7, 1, 8, 9]
Sequence 2 in x
  Input:  By Jove , my quick study of lexicography won a prize .
  Output: [10, 11, 12, 2, 13, 14, 15, 16, 3, 17]
Sequence 3 in x
  Input:  This is a short sentence .
  Output: [18, 19, 3, 20, 21]<jupyter_text>Padding When batching the sequence of word ids together, each sequence needs to be the same length.  Since sentences are dynamic in length, we can add padding to the end of the sequences to make them the same length.<jupyter_code>def pad(x, length=None):
    """
    Pad x
    :param x: List of sequences.
    :param length: Length to pad the sequence to.  If None, use length of longest sequence in x.
    :return: Padded numpy array of sequences
    """
    seq = pad_sequences(x, maxlen=length,padding='post',value=0.0)
    return seq

# Pad Tokenized output
test_pad = pad(text_tokenized)
for sample_i, (token_sent, pad_sent) in enumerate(zip(text_tokenized, test_pad)):
    print('Sequence {} in x'.format(sample_i + 1))
    print('  Input:  {}'.format(np.array(token_sent)))
    print('  Output: {}'.format(pad_sent))<jupyter_output>Sequence 1 in x
  Input:  [1 2 4 5 6 7 1 8 9]
  Output: [1 2 4 5 6 7 1 8 9 0]
Sequence 2 in x
  Input:  [10 11 12  2 13 14 15 16  3 17]
  Output: [10 11 12  2 13 14 15 16  3 17]
Sequence 3 in x
  Input:  [18 19  3 20 21]
  Output: [18 19  3 20 21  0  0  0  0  0]<jupyter_text>Preprocess Pipeline<jupyter_code>preprocess_y, y_tk = tokenize(english_sentences)
preprocess_y = pad(preprocess_y)
preprocess_y.shape
def preprocess(x, y):
    """
    Preprocess x and y
    :param x: Feature List of sentences
    :param y: Label List of sentences
    :return: Tuple of (Preprocessed x, Preprocessed y, x tokenizer, y tokenizer)
    """
    preprocess_x, x_tk = tokenize(x)
    preprocess_y, y_tk = tokenize(y)

    preprocess_x = pad(preprocess_x)
    preprocess_y = pad(preprocess_y)

    # Keras's sparse_categorical_crossentropy function requires the labels to be in 3 dimensions
    preprocess_y = preprocess_y.reshape((10000, 12, 1))

    return preprocess_x, preprocess_y, x_tk, y_tk

preproc_spanish_sentences, preproc_english_sentences, spanish_tokenizer, english_tokenizer =\
    preprocess(spanish_sentences, english_sentences)
    
max_spanish_sequence_length = preproc_spanish_sentences.shape[1]
max_english_sequence_length = preproc_english_sentences.shape[1]
spanish_vocab_size = len(spanish_tokenizer.word_index)
english_vocab_size = len(english_tokenizer.word_index) 

print('Data Preprocessed')
print("Max spanish sentence length:", max_spanish_sequence_length)
print("Max english sentence length:", max_english_sequence_length)
print("spanish vocabulary size:", spanish_vocab_size)
print("english vocabulary size:", english_vocab_size)<jupyter_output>Data Preprocessed
Max spanish sentence length: 6
Max english sentence length: 12
spanish vocabulary size: 3041
english vocabulary size: 5459<jupyter_text>ModelsIn this section, you will experiment with various neural network architectures.You will begin by training four relatively simple architectures.- Model 1 is a simple RNN- Model 2 is a RNN with Embedding- Model 3 is a Bidirectional RNN- Model 4 is an optional Encoder-Decoder RNNAfter experimenting with the four simple architectures, you will construct a deeper architecture that is designed to outperform all four models. Ids Back to TextThe neural network will be translating the input to words ids, which isn't the final form we want.  We want the english translation.  The function `logits_to_text` will bridge the gab between the logits from the neural network to the english translation.  You'll be using this function to better understand the output of the neural network.<jupyter_code>def logits_to_text(logits, tokenizer):
    """
    Turn logits from a neural network into text using the tokenizer
    :param logits: Logits from a neural network
    :param tokenizer: Keras Tokenizer fit on the labels
    :return: String that represents the text of the logits
    """
    index_to_words = {id: word for word, id in tokenizer.word_index.items()}
    index_to_words[0] = '<PAD>'

    return ' '.join([index_to_words[prediction] for prediction in np.argmax(logits, 1)])

print('`logits_to_text` function loaded.')<jupyter_output>`logits_to_text` function loaded.<jupyter_text>Model 1: RNN (IMPLEMENTATION)A basic RNN model is a good baseline for sequence data.<jupyter_code>from keras.layers import GRU, Input, Dense, TimeDistributed, Dropout
from keras.models import Model, Sequential
from keras.layers import Activation
from keras.optimizers import Adam
from keras.losses import sparse_categorical_crossentropy

def simple_model(input_shape, output_sequence_length, spanish_vocab_size, english_vocab_size):
    """
    Build and train a basic RNN on x and y
    :param input_shape: Tuple of input shape
    :param output_sequence_length: Length of output sequence
    :param spanish_vocab_size: Number of unique spanish words in the dataset
    :param english_vocab_size: Number of unique english words in the dataset
    :return: Keras model built, but not trained
    """
    
    learning_rate = 1e-3
    inputs = Input(shape=input_shape[1:])
    model = LSTM(50, return_sequences=True)(inputs)
    model = LSTM(50, return_sequences=True)(model)
    model = TimeDistributed(Dense(english_vocab_size))(model)
    model = Model(inputs, Activation('softmax')(model))

    model.compile(loss=sparse_categorical_crossentropy,
                  optimizer=Adam(learning_rate),
                  metrics=['accuracy'])   

    return model


# Reshaping the input to work with a basic RNN
tmp_x = pad(preproc_spanish_sentences, max_english_sequence_length)
tmp_x = tmp_x.reshape((-1, preproc_english_sentences.shape[-2], 1))

# Train the neural network
simple_rnn_model = simple_model(
    tmp_x.shape,
    max_english_sequence_length,
    spanish_vocab_size,
    english_vocab_size)
simple_rnn_model.fit(tmp_x, preproc_english_sentences,
                     batch_size=1024, epochs=10, validation_split=0.2)

# Print prediction(s)<jupyter_output>Train on 8000 samples, validate on 2000 samples
Epoch 1/10
8000/8000 [==============================] - 6s - loss: 8.5369 - acc: 0.4158 - val_loss: nan - val_acc: 0.6808
Epoch 2/10
8000/8000 [==============================] - 2s - loss: 8.2672 - acc: 0.6980 - val_loss: nan - val_acc: 0.6808
Epoch 3/10
8000/8000 [==============================] - 2s - loss: 7.7143 - acc: 0.6980 - val_loss: nan - val_acc: 0.6808
Epoch 4/10
8000/8000 [==============================] - 2s - loss: 6.8745 - acc: 0.6980 - val_loss: nan - val_acc: 0.6808
Epoch 5/10
8000/8000 [==============================] - 2s - loss: 5.8116 - acc: 0.6980 - val_loss: nan - val_acc: 0.6808
Epoch 6/10
8000/8000 [==============================] - 2s - loss: 4.9332 - acc: 0.6980 - val_loss: nan - val_acc: 0.6808
Epoch 7/10
8000/8000 [==============================] - 2s - loss: 4.1732 - acc: 0.6980 - val_loss: nan - val_acc: 0.6808
Epoch 8/10
8000/8000 [==============================] - 2s - loss: 3.5452 - acc: 0.6980 - val_loss[...]<jupyter_text>Model 2: EmbeddingAn embedding is a vector representation of the word that is close to similar words in n-dimensional space, where the n represents the size of the embedding vectors.RNN model using embedding.<jupyter_code>from keras.layers.embeddings import Embedding


def embed_model(input_shape, output_sequence_length, spanish_vocab_size, english_vocab_size):
    """
    Build and train a RNN model using word embedding on x and y
    :param input_shape: Tuple of input shape
    :param output_sequence_length: Length of output sequence
    :param spanish_vocab_size: Number of unique spanish words in the dataset
    :param english_vocab_size: Number of unique english words in the dataset
    :return: Keras model built, but not trained
    """
    # TODO: Implement
    model = Sequential()
    # output dim of embedding matrix tunable hyperparameter
    model.add(Embedding(spanish_vocab_size, 128, input_length=input_shape[1],
                         input_shape=input_shape[1:]))
    model.add(GRU(128, return_sequences=True))
    model.add(TimeDistributed(Dense(512, activation='tanh')))
    model.add(Dropout(0.5))
    model.add(TimeDistributed(Dense(english_vocab_size, activation='softmax')))
    learning_rate = 0.001
    model.compile(loss=sparse_categorical_crossentropy,
                  optimizer=Adam(learning_rate),
                  metrics=['accuracy'])
    print('Input Shape {}, Output Shape {}'.format(model.input_shape, model.output_shape))
    return model

# TODO: Reshape the input
tmp_x = pad(preproc_spanish_sentences, preproc_english_sentences.shape[1])
tmp_x = tmp_x.reshape((-1, preproc_english_sentences.shape[-2]))
# TODO: Train the neural network
# increased passed index length by 1 to avoid index error
embed_rnn_model = embed_model(
    tmp_x.shape,
    preproc_english_sentences.shape[1],
    len(spanish_tokenizer.word_index)+1,
    len(english_tokenizer.word_index)+1)
print(embed_rnn_model.summary())
# reduced batch size
embed_rnn_model.fit(tmp_x, preproc_english_sentences, batch_size=100, epochs=10, validation_split=0.2)

# TODO: Print prediction(s)
print(logits_to_text(embed_rnn_model.predict(tmp_x[:1])[0], english_tokenizer))
print("Original text and translation:")
print(spanish_sentences[:1])
print(english_sentences[:1])<jupyter_output>Input Shape (None, 12), Output Shape (None, 12, 5460)
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_4 (Embedding)      (None, 12, 128)           389376    
_________________________________________________________________
gru_9 (GRU)                  (None, 12, 128)           98688     
_________________________________________________________________
time_distributed_24 (TimeDis (None, 12, 512)           66048     
_________________________________________________________________
dropout_6 (Dropout)          (None, 12, 512)           0         
_________________________________________________________________
time_distributed_25 (TimeDis (None, 12, 5460)          2800980   
=================================================================
Total params: 3,355,092
Trainable params: 3,355,092
Non-trainable params: 0
____________[...]<jupyter_text>Model 3: Bidirectional RNNs (IMPLEMENTATION)One restriction of a RNN is that it can't see the future input, only the past.  This is where bidirectional recurrent neural networks come in.  They are able to see the future data.<jupyter_code>from keras.layers import Bidirectional

def bd_model(input_shape, output_sequence_length, spanish_vocab_size, english_vocab_size):
    """
    Build and train a bidirectional RNN model on x and y
    :param input_shape: Tuple of input shape
    :param output_sequence_length: Length of output sequence
    :param spanish_vocab_size: Number of unique spanish words in the dataset
    :param english_vocab_size: Number of unique english words in the dataset
    :return: Keras model built, but not trained
    """
    # TODO: Implement
    model = Sequential()
    model.add(Bidirectional(GRU(128, return_sequences=True), input_shape=input_shape[1:]))
    model.add(TimeDistributed(Dense(512, activation='tanh')))
    model.add(Dropout(0.5))
    model.add(TimeDistributed(Dense(english_vocab_size, activation='softmax')))
    learning_rate = 0.001   
    model.compile(loss=sparse_categorical_crossentropy,
                  optimizer=Adam(learning_rate),
                  metrics=['accuracy'])
    return model

# TODO: Train and Print prediction(s)
# Reshaping the input
tmp_x = pad(preproc_spanish_sentences, preproc_english_sentences.shape[1])
tmp_x = tmp_x.reshape((-1, preproc_english_sentences.shape[-2], 1))

# Train the neural network
# increased passed index length by 1 to avoid index error
bd_rnn_model = bd_model(
    tmp_x.shape,
    preproc_english_sentences.shape[1],
    len(spanish_tokenizer.word_index)+1,
    len(english_tokenizer.word_index)+1)
print(bd_rnn_model.summary())
# reduced batch size
bd_rnn_model.fit(tmp_x, preproc_english_sentences, batch_size=100, epochs=10, validation_split=0.2)

# Print prediction(s)
print(logits_to_text(bd_rnn_model.predict(tmp_x[:1])[0], english_tokenizer))
print("Original text and translation:")
print(spanish_sentences[:1])
print(english_sentences[:1])<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
bidirectional_6 (Bidirection (None, 12, 256)           99840     
_________________________________________________________________
time_distributed_26 (TimeDis (None, 12, 512)           131584    
_________________________________________________________________
dropout_7 (Dropout)          (None, 12, 512)           0         
_________________________________________________________________
time_distributed_27 (TimeDis (None, 12, 5460)          2800980   
=================================================================
Total params: 3,032,404
Trainable params: 3,032,404
Non-trainable params: 0
_________________________________________________________________
None
Train on 8000 samples, validate on 2000 samples
Epoch 1/10
8000/8000 [==============================] - 8s - loss: 3.0646 - acc[...]<jupyter_text>Model 4: Encoder-Decoder This model is made up of an encoder and decoder. The encoder creates a matrix representation of the sentence.  The decoder takes this matrix as input and predicts the translation as output.Create an encoder-decoder model in the cell below.<jupyter_code>from keras.layers import RepeatVector


def encdec_model(input_shape, output_sequence_length, spanish_vocab_size, english_vocab_size):
    """
    Build and train an encoder-decoder model on x and y
    :param input_shape: Tuple of input shape
    :param output_sequence_length: Length of output sequence
    :param spanish_vocab_size: Number of unique spanish words in the dataset
    :param english_vocab_size: Number of unique english words in the dataset
    :return: Keras model built, but not trained
    """
    # OPTIONAL: Implement
    model = Sequential()
    #Encoder
    # reverse input sequence order for improved accuracy 
    model.add(GRU(128, input_shape=input_shape[1:], go_backwards=True))
    #Adapter to fit 2D output to required 3D input shape [samples, time steps, features]
    model.add(RepeatVector(output_sequence_length))
    #Decoder
    model.add(GRU(128, return_sequences=True))
    model.add(TimeDistributed(Dense(512, activation='tanh')))
    model.add(Dropout(0.5))
    model.add(TimeDistributed(Dense(english_vocab_size, activation='softmax')))
    learning_rate = 0.001   
    model.compile(loss=sparse_categorical_crossentropy,
                  optimizer=Adam(learning_rate),
                  metrics=['accuracy'])
    return model

# OPTIONAL: Train and Print prediction(s)
# Reshaping the input to work with a basic RNN
tmp_x = pad(preproc_spanish_sentences, preproc_english_sentences.shape[1])
tmp_x = tmp_x.reshape((-1, preproc_english_sentences.shape[-2], 1))

# Train the neural network
# increased passed index length by 1 to avoid index error
encdec_rnn_model = encdec_model(
    tmp_x.shape,
    preproc_english_sentences.shape[1],
    len(spanish_tokenizer.word_index)+1,
    len(english_tokenizer.word_index)+1)
print(encdec_rnn_model.summary())
# reduced batch size
encdec_rnn_model.fit(tmp_x, preproc_english_sentences, batch_size=100, epochs=10, validation_split=0.2)

# Print prediction(s)
print(logits_to_text(encdec_rnn_model.predict(tmp_x[:1])[0], english_tokenizer))
print("Original text and translation:")
print(spanish_sentences[:1])
print(english_sentences[:1])<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
gru_11 (GRU)                 (None, 128)               49920     
_________________________________________________________________
repeat_vector_4 (RepeatVecto (None, 12, 128)           0         
_________________________________________________________________
gru_12 (GRU)                 (None, 12, 128)           98688     
_________________________________________________________________
time_distributed_28 (TimeDis (None, 12, 512)           66048     
_________________________________________________________________
dropout_8 (Dropout)          (None, 12, 512)           0         
_________________________________________________________________
time_distributed_29 (TimeDis (None, 12, 5460)          2800980   
=================================================================
Total para[...]<jupyter_text>Model 5: Custom (IMPLEMENTATION)Use everything you learned from the previous models to create a model that incorporates embedding and a bidirectional rnn into one model.<jupyter_code>from keras.layers import GRU, Input, Dense, TimeDistributed, Dropout
from keras.models import Model, Sequential
from keras.layers import Activation, RepeatVector, Bidirectional
from keras.optimizers import Adam
from keras.losses import sparse_categorical_crossentropy
from keras.layers.embeddings import Embedding

def model_final(input_shape, output_sequence_length, spanish_vocab_size, english_vocab_size):
    """
    Build and train a model that incorporates embedding, encoder-decoder, and bidirectional RNN on x and y
    :param input_shape: Tuple of input shape
    :param output_sequence_length: Length of output sequence
    :param spanish_vocab_size: Number of unique spanish words in the dataset
    :param english_vocab_size: Number of unique english words in the dataset
    :return: Keras model built, but not trained
    """
    # TODO: Implement
    model = Sequential()
    #Embedding
    model.add(Embedding(spanish_vocab_size, 128, input_length=input_shape[1],
                         input_shape=input_shape[1:]))
    #Bidirectional Encoder
    model.add(Bidirectional(GRU(128)))
    #Adapter to fit 2D output to required GRU 3D input shape [samples, time steps, features]
    model.add(RepeatVector(output_sequence_length))
    #Decoder
    model.add(Bidirectional(GRU(128, return_sequences=True)))
    model.add(TimeDistributed(Dense(512, activation='tanh')))
    model.add(Dropout(0.5))
    model.add(TimeDistributed(Dense(english_vocab_size, activation='softmax')))
    learning_rate = 0.001
    model.compile(loss=sparse_categorical_crossentropy,
                  optimizer=Adam(learning_rate),
                  metrics=['accuracy'])
    return model

print('Final Model Loaded')<jupyter_output>Final Model Loaded<jupyter_text>Prediction (IMPLEMENTATION)<jupyter_code>import numpy as np
from keras.preprocessing.sequence import pad_sequences


def final_predictions(x, y, x_tk, y_tk):
    """
    Gets predictions using the final model
    :param x: Preprocessed spanish data
    :param y: Preprocessed english data
    :param x_tk: spanish tokenizer
    :param y_tk: english tokenizer
    """
    # increased passed index length by 1 to avoid index error
    model = model_final(x.shape,y.shape[1],
                        len(x_tk.word_index)+1,
                        len(y_tk.word_index)+1)
    print(model.summary())
    # reduced batch size to 100 and train for 20 epochs
    model.fit(x, y, batch_size=100, epochs=10, validation_split=0.2)

    y_id_to_word = {value: key for key, value in y_tk.word_index.items()}
    y_id_to_word[0] = '<PAD>'

    sentence = 'él tiene un diario'
    sentence = [x_tk.word_index[word] for word in sentence.split()]
    sentence = pad_sequences([sentence], maxlen=x.shape[-1], padding='post')
    sentences = np.array([sentence[0], x[0]])
    predictions = model.predict(sentences, len(sentences))

final_predictions(preproc_spanish_sentences, preproc_english_sentences, spanish_tokenizer, english_tokenizer)
# validation_split argument in model.fit holds out the last 20% of the training data set
# validation accuracy should be verified over different data sets
# try stratfied Kfold cross validation as described in:
    # https://machinelearningmastery.com/evaluate-performance-deep-learning-models-keras/

from sklearn.model_selection import StratifiedKFold
import numpy
# fixed random seed for reproducibility
seed = 7
numpy.random.seed(seed)
# define 3-fold cross validation test harness
kfold = StratifiedKFold(n_splits=3, shuffle=True, random_state=seed)
cvscores=[]
for train, test in kfold.split(preproc_spanish_sentences,preproc_english_sentences[:,0,0]):
    # compile model
    model = model_final(preproc_spanish_sentences[train].shape,
                        preproc_english_sentences[train].shape[1],
                        len(spanish_tokenizer.word_index)+1,
                        len(english_tokenizer.word_index)+1)
    # fit the model - 10 epochs
    model.fit(preproc_spanish_sentences[train], preproc_english_sentences[train],
              batch_size=100, epochs=10, verbose=0)
    # evaluate the model
    scores = model.evaluate(preproc_spanish_sentences[test], preproc_english_sentences[test], verbose=0)
    print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
    cvscores.append(scores[1]*100)
print("%.2f%% (+/- %.2f%%)" % (numpy.mean(cvscores), numpy.std(cvscores)))<jupyter_output>/usr/local/lib64/python3.4/site-packages/sklearn/model_selection/_split.py:652: Warning: The least populated class in y has only 1 members, which is too few. The minimum number of members in any class cannot be less than n_splits=3.
  % (min_groups, self.n_splits)), Warning)
{'fraction_non_alphanumeric': 0.11293843690874691, 'fraction_numerical': 0.03307378838597002, 'mean_word_length': 4.610328638497653, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '646395', 'license': 'MIT', 'n_tokens_mistral': 8976, 'n_tokens_neox': 8110, 'n_words': 2509, 'path': 'machine_translation.ipynb', 'repo_name': 'rsanchezavalos/RNN-Language-Translator', 'size': '27484'}