<jupyter_start><jupyter_text>NMT-Keras tutorial---This notebook describes, step by step, how to build a neural machine translation model with NMT-Keras. The tutorial is organized in different sections:1. Create a Dataset instance, in order to properly manage the data. 2. Create and train the Neural Translation Model in the training data.3. Apply the trained model on new (unseen) data.All these steps are automatically run by the toolkit. But, to learn and understand the full process, it is didactic to follow this tutorial.So, let's start installing the toolkit.<jupyter_code>!pip install update pip
!pip uninstall -y keras  # Avoid crashes with pre-installed packages
!git clone https://github.com/lvapeab/nmt-keras
import os
os.chdir('nmt-keras')
!pip install -e .<jupyter_output>Collecting update
  Downloading https://files.pythonhosted.org/packages/9f/c4/dfe8a392edd35cc635c35cd3b20df6a746aacdeb39b685d1668b56bf819b/update-0.0.1-py2.py3-none-any.whl
Requirement already satisfied: pip in /usr/local/lib/python3.6/dist-packages (19.3.1)
Collecting style==1.1.0
  Downloading https://files.pythonhosted.org/packages/4c/0b/6be2071e20c621e7beb01b86e8474c2ec344a9750ba5315886f24d6e7386/style-1.1.0-py2.py3-none-any.whl
Installing collected packages: style, update
Successfully installed style-1.1.0 update-0.0.1
Uninstalling Keras-2.3.1:
  Successfully uninstalled Keras-2.3.1
Cloning into 'nmt-keras'...
remote: Enumerating objects: 127, done.[K
remote: Counting objects: 100% (127/127), done.[K
remote: Compressing objects: 100% (66/66), done.[K
remote: Total 4667 (delta 82), reused 87 (delta 60), pack-reused 4540[K
Receiving objects: 100% (4667/4667), 5.70 MiB | 8.93 MiB/s, done.
Resolving deltas: 100% (3152/3152), done.
Obtaining file:///content/nmt-keras
Requirement al[...]<jupyter_text>1. Building a Dataset modelFirst, we are creating a [Dataset](https://github.com/MarcBS/multimodal_keras_wrapper/keras_wrapper/dataset.py) object (from the [Multimodal Keras Wrapper](https://github.com/MarcBS/multimodal_keras_wrapper) library). This object will be the interface between our data (text files) and the model:<jupyter_code>from keras_wrapper.dataset import Dataset, saveDataset
from data_engine.prepare_data import keep_n_captions
ds = Dataset('tutorial_dataset', 'tutorial', silence=False)<jupyter_output><empty_output><jupyter_text>Now that we have the empty dataset, we must indicate its inputs and outputs. In our case, we'll have two different inputs and one single output:1. Outputs:**target_text**: Sentences in our target language.2. Inputs:**source_text**: Sentences in the source language.**state_below**: Sentences in the target language, but shifted one position to the right (for teacher-forcing training of the model).For setting up the outputs, we use the setOutputs function, with the appropriate parameters. Note that, when we are building the dataset for the training split, we build the vocabulary (up to 30000 words).<jupyter_code>ds.setOutput('examples/EuTrans/training.en',
             'train',
             type='text',
             id='target_text',
             tokenization='tokenize_none',
             build_vocabulary=True,
             pad_on_batch=True,
             sample_weights=True,
             max_text_len=30,
             max_words=30000,
             min_occ=0)

ds.setOutput('examples/EuTrans/dev.en',
             'val',
             type='text',
             id='target_text',
             pad_on_batch=True,
             tokenization='tokenize_none',
             sample_weights=True,
             max_text_len=30,
             max_words=0)<jupyter_output>[14/04/2020 11:18:59] 	Applying tokenization function: "tokenize_none".
[14/04/2020 11:18:59] Creating vocabulary for data with data_id 'target_text'.
[14/04/2020 11:18:59] 	 Total: 513 unique words in 9900 sentences with a total of 98304 words.
[14/04/2020 11:18:59] Creating dictionary of 30000 most common words, covering 100.0% of the text.
[14/04/2020 11:18:59] Loaded "train" set outputs of data_type "text" with data_id "target_text" and length 9900.
[14/04/2020 11:18:59] 	Applying tokenization function: "tokenize_none".
[14/04/2020 11:18:59] Loaded "val" set outputs of data_type "text" with data_id "target_text" and length 100.<jupyter_text>Similarly, we introduce the source text data, with the setInputs function. Again, when building the training split, we must construct the vocabulary.<jupyter_code>ds.setInput('examples/EuTrans/training.es',
            'train',
            type='text',
            id='source_text',
            pad_on_batch=True,
            tokenization='tokenize_none',
            build_vocabulary=True,
            fill='end',
            max_text_len=30,
            max_words=30000,
            min_occ=0)
ds.setInput('examples/EuTrans/dev.es',
            'val',
            type='text',
            id='source_text',
            pad_on_batch=True,
            tokenization='tokenize_none',
            fill='end',
            max_text_len=30,
            min_occ=0)<jupyter_output>[14/04/2020 11:18:59] 	Applying tokenization function: "tokenize_none".
[14/04/2020 11:18:59] Creating vocabulary for data with data_id 'source_text'.
[14/04/2020 11:18:59] 	 Total: 686 unique words in 9900 sentences with a total of 96172 words.
[14/04/2020 11:18:59] Creating dictionary of 30000 most common words, covering 100.0% of the text.
[14/04/2020 11:18:59] Loaded "train" set inputs of data_type "text" with data_id "source_text" and length 9900.
[14/04/2020 11:18:59] 	Applying tokenization function: "tokenize_none".
[14/04/2020 11:18:59] Loaded "val" set inputs of data_type "text" with data_id "source_text" and length 100.<jupyter_text>...and for the 'state_below' data. Note that: 1) The offset flat is set to 1, which means that the text will be shifted to the right 1 position. 2) During sampling time, we won't have this input. Hence, we 'hack' the dataset model by inserting an artificial input, of type 'ghost' for the validation split.<jupyter_code>ds.setInput('examples/EuTrans/training.en',
            'train',
            type='text',
            id='state_below',
            required=False,
            tokenization='tokenize_none',
            pad_on_batch=True,
            build_vocabulary='target_text',
            offset=1,
            fill='end',
            max_text_len=30,
            max_words=30000)
ds.setInput(None,
            'val',
            type='ghost',
            id='state_below',
            required=False)<jupyter_output>[14/04/2020 11:18:59] 	Applying tokenization function: "tokenize_none".
[14/04/2020 11:18:59] 	Reusing vocabulary named "target_text" for data with data_id "state_below".
[14/04/2020 11:18:59] Loaded "train" set inputs of data_type "text" with data_id "state_below" and length 9900.
[14/04/2020 11:18:59] Loaded "val" set inputs of data_type "ghost" with data_id "state_below" and length 100.<jupyter_text>We can also keep the literal source words (for replacing unknown words).<jupyter_code>for split, input_text_filename in zip(['train', 'val'], ['examples/EuTrans/training.es', 'examples/EuTrans/dev.es']):
    ds.setRawInput(input_text_filename,
                  split,
                  type='file-name',
                  id='raw_source_text',
                  overwrite_split=True)<jupyter_output>[14/04/2020 11:18:59] Loaded "train" set inputs of type "file-name" with id "raw_source_text".
[14/04/2020 11:18:59] Loaded "val" set inputs of type "file-name" with id "raw_source_text".<jupyter_text>We also need to match the references with the inputs. Since we only have one reference per input sample, we set `repeat=1`.<jupyter_code>keep_n_captions(ds, repeat=1, n=1, set_names=['val'])<jupyter_output>[14/04/2020 11:18:59] Keeping 1 captions per input on the val set.
[14/04/2020 11:18:59] Samples reduced to 100 in val set.<jupyter_text>Finally, we can save our dataset instance for using in other experiments:<jupyter_code>saveDataset(ds, 'datasets')<jupyter_output>[14/04/2020 11:18:59] <<< creating directory datasets ... >>>
[14/04/2020 11:18:59] <<< Saving Dataset instance to datasets/Dataset_tutorial_dataset.pkl ... >>>
[14/04/2020 11:18:59] <<< Dataset instance saved >>><jupyter_text>2. Creating and training a Neural Translation ModelNow, we'll create and train a Neural Machine Translation (NMT) model. Since there is a significant number of hyperparameters, we'll use the default ones, specified in the `config.py` file. Note that almost every hardcoded parameter is automatically set from config if we run  `main.py `.We'll create an `'AttentionRNNEncoderDecoder'` (a LSTM encoder-decoder with attention mechanism). Refer to the [`model_zoo.py`](https://github.com/lvapeab/nmt-keras/blob/master/nmt_keras/model_zoo.py) file for other models (e.g. Transformer). So first, let's import the model and the hyperparameters. We'll also load the dataset we stored in the previous section (not necessary as it is in memory, but as a demonstration):<jupyter_code>from config import load_parameters
from nmt_keras.model_zoo import TranslationModel
from keras_wrapper.cnn_model import loadModel
from keras_wrapper.dataset import loadDataset
from keras_wrapper.extra.callbacks import PrintPerformanceMetricOnEpochEndOrEachNUpdates
params = load_parameters()
dataset = loadDataset('datasets/Dataset_tutorial_dataset.pkl')<jupyter_output>Using TensorFlow backend.<jupyter_text>Since the number of words in the dataset may be unknown beforehand, we must update the params information according to the dataset instance:<jupyter_code>params['INPUT_VOCABULARY_SIZE'] = dataset.vocabulary_len['source_text']
params['OUTPUT_VOCABULARY_SIZE'] = dataset.vocabulary_len['target_text']<jupyter_output><empty_output><jupyter_text>Now, we create a `TranslationModel` instance:<jupyter_code>params['MODEL_TYPE'] = 'AttentionRNNEncoderDecoder' #  Supported models: 'AttentionRNNEncoderDecoder' and 'Transformer'.
nmt_model = TranslationModel(params,
                             model_type=params['MODEL_TYPE'], 
                             model_name='tutorial_model',
                             vocabularies=dataset.vocabulary,
                             store_path='trained_models/tutorial_model/',
                             verbose=True)<jupyter_output>[14/04/2020 11:19:02] <<< Building AttentionRNNEncoderDecoder Translation_Model >>>
[14/04/2020 11:19:02] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:650: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

[14/04/2020 11:19:02] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4786: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

[14/04/2020 11:19:02] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:157: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

[14/04/2020 11:19:03] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3561: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where<jupyter_text>Next, we must define the inputs and outputs mapping from our Dataset instance to our model:<jupyter_code>inputMapping = dict()
for i, id_in in enumerate(params['INPUTS_IDS_DATASET']):
    pos_source = dataset.ids_inputs.index(id_in)
    id_dest = nmt_model.ids_inputs[i]
    inputMapping[id_dest] = pos_source
nmt_model.setInputsMapping(inputMapping)

outputMapping = dict()
for i, id_out in enumerate(params['OUTPUTS_IDS_DATASET']):
    pos_target = dataset.ids_outputs.index(id_out)
    id_dest = nmt_model.ids_outputs[i]
    outputMapping[id_dest] = pos_target
nmt_model.setOutputsMapping(outputMapping)<jupyter_output><empty_output><jupyter_text>We can add some callbacks for controlling the training (e.g. Sampling each N updates, early stop, learning rate annealing...). For instance, let's build a sampling callback. After each epoch, it will compute the BLEU scores on the development set using the sacreBLEU package. We need to pass some configuration variables to the callback (in the extra_vars dictionary):<jupyter_code>is_transformer = params.get('ATTEND_ON_OUTPUT', 'transformer' in params['MODEL_TYPE'].lower())
search_params = {
    'language': 'en',
    'tokenize_f': eval('dataset.' + 'tokenize_none'),
    'beam_size': 12,
    'optimized_search': True,
    'model_inputs': params['INPUTS_IDS_MODEL'],
    'model_outputs': params['OUTPUTS_IDS_MODEL'],
    'dataset_inputs':  params['INPUTS_IDS_DATASET'],
    'dataset_outputs':  params['OUTPUTS_IDS_DATASET'],
    'n_parallel_loaders': 1,
    'maxlen': 50,
    'normalize_probs': True,
    'pos_unk': True and not is_transformer,  # Pos_unk is unimplemented for transformer models
    'heuristic': 0,
    'state_below_maxlen': -1,
    'attend_on_output': is_transformer,
    'val': {'references': dataset.extra_variables['val']['target_text']}
  }

vocab = dataset.vocabulary['target_text']['idx2words']
callbacks = []
input_text_id = params['INPUTS_IDS_DATASET'][0]

callbacks.append(PrintPerformanceMetricOnEpochEndOrEachNUpdates(nmt_model,
                                                                dataset,
                                                                gt_id='target_text',
                                                                metric_name=['sacrebleu'],
                                                                set_name=['val'],
                                                                batch_size=50,
                                                                each_n_epochs=1,
                                                                extra_vars=search_params,
                                                                reload_epoch=0,
                                                                is_text=True,
                                                                input_text_id=input_text_id,
                                                                index2word_y=vocab,
                                                                sampling_type='max_likelihood',
                                                                beam_search=True,
                                                                save_path=nmt_model.model_path,
                                                                start_eval_on_epoch=0,
                                                                write_samples=True,
                                                                write_type='list',
                                                                verbose=True))<jupyter_output><empty_output><jupyter_text>Now we are ready to train. Let's set up some training parameters...<jupyter_code>training_params = {'n_epochs': 4,
                   'batch_size': 50,
                   'maxlen': 30,
                   'epochs_for_save': 1,
                   'verbose': 1,
                   'eval_on_sets': [], 
                   'n_parallel_loaders': 1,
                   'extra_callbacks': callbacks,
                   'reload_epoch': 0,
                   'epoch_offset': 0}<jupyter_output><empty_output><jupyter_text>And train!<jupyter_code>nmt_model.trainNet(dataset, training_params)<jupyter_output>[14/04/2020 11:19:03] <<< Training model >>>
[14/04/2020 11:19:03] Training parameters: { 
	batch_size: 50
	class_weights: None
	da_enhance_list: []
	da_patch_type: resize_and_rndcrop
	data_augmentation: False
	each_n_epochs: 1
	epoch_offset: 0
	epochs_for_save: 1
	eval_on_epochs: True
	eval_on_sets: []
	extra_callbacks: [<keras_wrapper.extra.callbacks.EvalPerformance object at 0x7fafbd6275f8>]
	homogeneous_batches: False
	initial_lr: 1.0
	joint_batches: 4
	lr_decay: None
	lr_gamma: 0.1
	lr_half_life: 50000
	lr_reducer_exp_base: 0.5
	lr_reducer_type: linear
	lr_warmup_exp: -1.5
	maxlen: 30
	mean_substraction: False
	metric_check: None
	min_delta: 0.0
	min_lr: 1e-09
	n_epochs: 4
	n_gpus: 1
	n_parallel_loaders: 1
	normalization_type: None
	normalize: False
	num_iterations_val: None
	patience: 0
	patience_check_split: val
	reduce_each_epochs: True
	reload_epoch: 0
	shuffle: True
	start_eval_on_epoch: 0
	start_reduction_on_epoch: 0
	tensorboard: False
	tensorboard_params: {'log_dir': 'tens[...]<jupyter_text>3. Decoding with a trained Neural Machine Translation ModelNow, we'll load from disk the model we just trained and we'll apply it for translating new text. In this case, we want to translate the 'test' split from our dataset.Since we want to translate a new data split ('test') we must add it to the dataset instance, just as we did before (at the first tutorial). In case we also had the refences of the test split and we wanted to evaluate it, we can add it to the dataset. Note that this is not mandatory and we could just predict without evaluating.<jupyter_code>dataset.setInput('examples/EuTrans/test.es',
            'test',
            type='text',
            id='source_text',
            pad_on_batch=True,
            tokenization='tokenize_none',
            fill='end',
            max_text_len=30,
            min_occ=0)

dataset.setInput(None,
            'test',
            type='ghost',
            id='state_below',
            required=False)

dataset.setRawInput('examples/EuTrans/test.es',
              'test',
              type='file-name',
              id='raw_source_text',
              overwrite_split=True)<jupyter_output>[14/04/2020 11:20:56] 	Applying tokenization function: "tokenize_none".
[14/04/2020 11:20:56] Loaded "test" set inputs of data_type "text" with data_id "source_text" and length 2996.
[14/04/2020 11:20:56] Loaded "test" set inputs of data_type "ghost" with data_id "state_below" and length 2996.
[14/04/2020 11:20:56] Loaded "test" set inputs of type "file-name" with id "raw_source_text".<jupyter_text>Now, let's load the translation model. Suppose we want to load the model saved at the end of the epoch 4:<jupyter_code>params['INPUT_VOCABULARY_SIZE'] = dataset.vocabulary_len[params['INPUTS_IDS_DATASET'][0]]
params['OUTPUT_VOCABULARY_SIZE'] = dataset.vocabulary_len[params['OUTPUTS_IDS_DATASET'][0]]

# Load model
nmt_model = loadModel('trained_models/tutorial_model', 4)<jupyter_output>[14/04/2020 11:20:56] <<< Loading model from trained_models/tutorial_model/epoch_4_Model_Wrapper.pkl ... >>>
[14/04/2020 11:20:56] <<< Loading model from trained_models/tutorial_model/epoch_4.h5 ... >>>
[14/04/2020 11:20:58] <<< Loading optimized model... >>>
[14/04/2020 11:21:02] <<< Optimized model loaded. >>>
[14/04/2020 11:21:02] <<< Model loaded in 5.9873 seconds. >>><jupyter_text>Once we loaded the model, we just have to invoke the sampling method (in this case, the Beam Search algorithm) for the 'test' split:<jupyter_code>is_transformer = params.get('ATTEND_ON_OUTPUT', 'transformer' in params['MODEL_TYPE'].lower())

params_prediction = {
    'language': 'en',
    'tokenize_f': eval('dataset.' + 'tokenize_none'),
    'beam_size': 12,
    'optimized_search': True,
    'model_inputs': params['INPUTS_IDS_MODEL'],
    'model_outputs': params['OUTPUTS_IDS_MODEL'],
    'dataset_inputs':  params['INPUTS_IDS_DATASET'],
    'dataset_outputs':  params['OUTPUTS_IDS_DATASET'],
    'n_parallel_loaders': 1,
    'maxlen': 50,
    'normalize_probs': True,
    'pos_unk': True and not is_transformer,
    'heuristic': 0,
    'state_below_maxlen': -1,
    'predict_on_sets': ['test'],
    'verbose': 0,
    'attend_on_output': is_transformer
  }
predictions = nmt_model.predictBeamSearchNet(dataset, params_prediction)['test']<jupyter_output>[14/04/2020 11:21:02] <<< Predicting outputs of test set >>><jupyter_text>Up to now, in the variable 'predictions', we have the indices of the words of the hypotheses. We must decode them into words. For doing this, we'll use the dictionary stored in the dataset object:<jupyter_code>from keras_wrapper.utils import decode_predictions_beam_search
vocab = dataset.vocabulary['target_text']['idx2words']
samples = predictions['samples'] # Get word indices from the samples.

predictions = decode_predictions_beam_search(samples,  
                                             vocab,
                                             verbose=params['VERBOSE'])<jupyter_output>[14/04/2020 11:23:59] Decoding beam search prediction ...<jupyter_text>Finally, we store the hypotheses:<jupyter_code>filepath = 'test.pred'
from keras_wrapper.extra.read_write import list2file
list2file(filepath, predictions)
!head -n 4 test.pred<jupyter_output>I would like to book a room until tomorrow , please .
please wake us up tomorrow at a quarter past one .
I am leaving today in the afternoon .
would you mind taxi for me , please ?<jupyter_text>If we have the references of this split, we can also evaluate the performance of our system on it. First, we must add them to the dataset object:<jupyter_code>dataset.setOutput('examples/EuTrans/test.en',
             'test',
             type='text',
             id='target_text',
             pad_on_batch=True,
             tokenization='tokenize_none',
             sample_weights=True,
             max_text_len=30,
             max_words=0)
keep_n_captions(dataset, repeat=1, n=1, set_names=['test'])<jupyter_output>[14/04/2020 11:24:01] 	Applying tokenization function: "tokenize_none".
[14/04/2020 11:24:01] Loaded "test" set outputs of data_type "text" with data_id "target_text" and length 2996.
[14/04/2020 11:24:01] Keeping 1 captions per input on the test set.
[14/04/2020 11:24:01] Samples reduced to 2996 in test set.<jupyter_text>Next, we call the evaluation system: the sacreBLEU package:<jupyter_code>from keras_wrapper.extra.evaluation import select
metric = 'sacrebleu'
# Apply sampling
extra_vars = dict()
extra_vars['tokenize_f'] = eval('dataset.' + 'tokenize_none')
extra_vars['language'] = params['TRG_LAN']
extra_vars['test'] = dict()
extra_vars['test']['references'] = dataset.extra_variables['test']['target_text']
metrics = select[metric](pred_list=predictions,
                                          verbose=1,
                                          extra_vars=extra_vars,
                                          split='test')<jupyter_output>[14/04/2020 11:24:02] Computing SacreBleu scores on the test split...
[14/04/2020 11:24:02] Bleu_4: 60.97881069640092
{'fraction_non_alphanumeric': 0.10194001957817923, 'fraction_numerical': 0.05028032392987452, 'mean_word_length': 3.1744056463595838, 'pattern_counts': {'":': 0, '<': 106, '<?xml version=': 0, '>': 106, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 48, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '58299', 'license': 'MIT', 'n_tokens_mistral': 7677, 'n_tokens_neox': 6833, 'n_words': 1825, 'path': 'examples/tutorial.ipynb', 'repo_name': 'kamalbhavyam/nmt-keras', 'size': '22474'}