<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# NMT-Keras tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This set of notebooks describes how to build a neural machine translation model with Keras-NMT. It's assumed that you properly set up all the required dependencies (Theano, Keras, Staged Keras Wrapper, COCO Caption...). First, we'll create a Dataset instance, in order to properly manage the data. Next, we'll create and train a Neural Translation Model. Finally, we'll apply the trained model on new (unseen) data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Building a Dataset model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we are creating a [Dataset](https://github.com/MarcBS/staged_keras_wrapper/keras_wrapper/dataset.py) object (from the [Staged Keras Wrapper](https://github.com/MarcBS/staged_keras_wrapper) library). Let's make some imports and create an empty Dataset instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras_wrapper.dataset import Dataset\n",
    "from data_engine.prepare_data import keep_n_captions\n",
    "ds = Dataset('tutorial_dataset', 'tutorial', silence=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the empty dataset, we must indicate its inputs and outputs. In our case, we'll have two different inputs and one single output:\n",
    "1) Outputs:\n",
    " \t**target_text**: Sentences in our target language.\n",
    "2) Inputs: \n",
    " \t**source_text**: Sentences in the source language.\n",
    "\t**state_below**: Sentences  in the target language, but shifted one position to the right (for teacher-forcing training of the model)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For setting up the outputs, we use the setOutputs function, with the appropriate parameters. Note that, when we are building the dataset for the training split, we build the vocabulary (up to 30000 words)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:24] Creating vocabulary for data with id 'target_text'.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:25] \t Total: 513 unique words in 9900 sentences with a total of 98304 words.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:25] Creating dictionary of 30000 most common words, covering 100.0% of the text.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:25] Loaded \"train\" set outputs of type \"text\" with id \"target_text\" and length 9900.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:25] Loaded \"val\" set outputs of type \"text\" with id \"target_text\" and length 100.\n"
     ]
    }
   ],
   "source": [
    "ds.setOutput('examples/EuTrans/DATA/training.en',\n",
    "             'train',\n",
    "             type='text',\n",
    "             id='target_text',\n",
    "             tokenization='tokenize_none',\n",
    "             build_vocabulary=True,\n",
    "             pad_on_batch=True,\n",
    "             sample_weights=True,\n",
    "             max_text_len=30,\n",
    "             max_words=30000,\n",
    "             min_occ=0)\n",
    "\n",
    "ds.setOutput('examples/EuTrans/DATA//dev.en',\n",
    "             'val',\n",
    "             type='text',\n",
    "             id='target_text',\n",
    "             pad_on_batch=True,\n",
    "             tokenization='tokenize_none',\n",
    "             sample_weights=True,\n",
    "             max_text_len=30,\n",
    "             max_words=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we introduce the source text data, with the setInputs function. Again, when building the training split, we must construct the vocabulary.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:27] Creating vocabulary for data with id 'source_text'.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:27] \t Total: 686 unique words in 9900 sentences with a total of 96172 words.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:27] Creating dictionary of 30000 most common words, covering 100.0% of the text.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:27] Loaded \"train\" set inputs of type \"text\" with id \"source_text\" and length 9900.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:27] Loaded \"val\" set inputs of type \"text\" with id \"source_text\" and length 100.\n"
     ]
    }
   ],
   "source": [
    "ds.setInput('examples/EuTrans/DATA//training.es',\n",
    "            'train',\n",
    "            type='text',\n",
    "            id='source_text',\n",
    "            pad_on_batch=True,\n",
    "            tokenization='tokenize_none',\n",
    "            build_vocabulary=True,\n",
    "            fill='end',\n",
    "            max_text_len=30,\n",
    "            max_words=30000,\n",
    "            min_occ=0)\n",
    "ds.setInput('examples/EuTrans/DATA//dev.es',\n",
    "            'val',\n",
    "            type='text',\n",
    "            id='source_text',\n",
    "            pad_on_batch=True,\n",
    "            tokenization='tokenize_none',\n",
    "            fill='end',\n",
    "            max_text_len=30,\n",
    "            min_occ=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and for the 'state_below' data. Note that: 1) The offset flat is set to 1, which means that the text will be shifted to the right 1 position. 2) During sampling time, we won't have this input. Hence, we 'hack' the dataset model by inserting an artificial input, of type 'ghost' for the validation split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:29] \tReusing vocabulary named \"target_text\" for data with id \"state_below\".\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:29] Loaded \"train\" set inputs of type \"text\" with id \"state_below\" and length 9900.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:29] Loaded \"val\" set inputs of type \"ghost\" with id \"state_below\" and length 100.\n"
     ]
    }
   ],
   "source": [
    "ds.setInput('examples/EuTrans/DATA//training.en',\n",
    "            'train',\n",
    "            type='text',\n",
    "            id='state_below',\n",
    "            required=False,\n",
    "            tokenization='tokenize_none',\n",
    "            pad_on_batch=True,\n",
    "            build_vocabulary='target_text',\n",
    "            offset=1,\n",
    "            fill='end',\n",
    "            max_text_len=30,\n",
    "            max_words=30000)\n",
    "ds.setInput(None,\n",
    "            'val',\n",
    "            type='ghost',\n",
    "            id='state_below',\n",
    "            required=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We must match the references with inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:31] Keeping 1 captions per input on the val set.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 15:56:31] Samples reduced to 100 in val set.\n"
     ]
    }
   ],
   "source": [
    "# If we had multiple references per sentence\n",
    "keep_n_captions(ds, repeat=1, n=1, set_names=['val'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can save our dataset instance for using in other experiments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "saveDataset(ds, 'datasets')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    ""
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },<commit_msg>Tutorial Dataset<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# NMT-Keras tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This set of notebooks describes how to build a neural machine translation model with Keras-NMT. It's assumed that you properly set up all the required dependencies (Theano, Keras, Staged Keras Wrapper, COCO Caption...). First, we'll create a Dataset instance, in order to properly manage the data. Next, we'll create and train a Neural Translation Model. Finally, we'll apply the trained model on new (unseen) data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Building a Dataset model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we are creating a [Dataset](https://github.com/MarcBS/staged_keras_wrapper/keras_wrapper/dataset.py) object (from the [Staged Keras Wrapper](https://github.com/MarcBS/staged_keras_wrapper) library). Let's make some imports and create an empty Dataset instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras_wrapper.dataset import Dataset, saveDataset\n",
    "from data_engine.prepare_data import keep_n_captions\n",
    "ds = Dataset('tutorial_dataset', 'tutorial', silence=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the empty dataset, we must indicate its inputs and outputs. In our case, we'll have two different inputs and one single output:\n",
    "1) Outputs:\n",
    " \t**target_text**: Sentences in our target language.\n",
    "2) Inputs: \n",
    " \t**source_text**: Sentences in the source language.\n",
    "\t**state_below**: Sentences  in the target language, but shifted one position to the right (for teacher-forcing training of the model)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For setting up the outputs, we use the setOutputs function, with the appropriate parameters. Note that, when we are building the dataset for the training split, we build the vocabulary (up to 30000 words)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:03:58] Creating vocabulary for data with id 'target_text'.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:03:58] \t Total: 513 unique words in 9900 sentences with a total of 98304 words.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:03:58] Creating dictionary of 30000 most common words, covering 100.0% of the text.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:03:58] Loaded \"train\" set outputs of type \"text\" with id \"target_text\" and length 9900.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:03:58] Loaded \"val\" set outputs of type \"text\" with id \"target_text\" and length 100.\n"
     ]
    }
   ],
   "source": [
    "ds.setOutput('examples/EuTrans/DATA/training.en',\n",
    "             'train',\n",
    "             type='text',\n",
    "             id='target_text',\n",
    "             tokenization='tokenize_none',\n",
    "             build_vocabulary=True,\n",
    "             pad_on_batch=True,\n",
    "             sample_weights=True,\n",
    "             max_text_len=30,\n",
    "             max_words=30000,\n",
    "             min_occ=0)\n",
    "\n",
    "ds.setOutput('examples/EuTrans/DATA//dev.en',\n",
    "             'val',\n",
    "             type='text',\n",
    "             id='target_text',\n",
    "             pad_on_batch=True,\n",
    "             tokenization='tokenize_none',\n",
    "             sample_weights=True,\n",
    "             max_text_len=30,\n",
    "             max_words=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we introduce the source text data, with the setInputs function. Again, when building the training split, we must construct the vocabulary.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:00] Creating vocabulary for data with id 'source_text'.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:00] \t Total: 686 unique words in 9900 sentences with a total of 96172 words.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:00] Creating dictionary of 30000 most common words, covering 100.0% of the text.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:00] Loaded \"train\" set inputs of type \"text\" with id \"source_text\" and length 9900.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:00] Loaded \"val\" set inputs of type \"text\" with id \"source_text\" and length 100.\n"
     ]
    }
   ],
   "source": [
    "ds.setInput('examples/EuTrans/DATA//training.es',\n",
    "            'train',\n",
    "            type='text',\n",
    "            id='source_text',\n",
    "            pad_on_batch=True,\n",
    "            tokenization='tokenize_none',\n",
    "            build_vocabulary=True,\n",
    "            fill='end',\n",
    "            max_text_len=30,\n",
    "            max_words=30000,\n",
    "            min_occ=0)\n",
    "ds.setInput('examples/EuTrans/DATA//dev.es',\n",
    "            'val',\n",
    "            type='text',\n",
    "            id='source_text',\n",
    "            pad_on_batch=True,\n",
    "            tokenization='tokenize_none',\n",
    "            fill='end',\n",
    "            max_text_len=30,\n",
    "            min_occ=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and for the 'state_below' data. Note that: 1) The offset flat is set to 1, which means that the text will be shifted to the right 1 position. 2) During sampling time, we won't have this input. Hence, we 'hack' the dataset model by inserting an artificial input, of type 'ghost' for the validation split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:02] \tReusing vocabulary named \"target_text\" for data with id \"state_below\".\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:02] Loaded \"train\" set inputs of type \"text\" with id \"state_below\" and length 9900.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:02] Loaded \"val\" set inputs of type \"ghost\" with id \"state_below\" and length 100.\n"
     ]
    }
   ],
   "source": [
    "ds.setInput('examples/EuTrans/DATA//training.en',\n",
    "            'train',\n",
    "            type='text',\n",
    "            id='state_below',\n",
    "            required=False,\n",
    "            tokenization='tokenize_none',\n",
    "            pad_on_batch=True,\n",
    "            build_vocabulary='target_text',\n",
    "            offset=1,\n",
    "            fill='end',\n",
    "            max_text_len=30,\n",
    "            max_words=30000)\n",
    "ds.setInput(None,\n",
    "            'val',\n",
    "            type='ghost',\n",
    "            id='state_below',\n",
    "            required=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We must match the references with inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:06] Keeping 1 captions per input on the val set.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:06] Samples reduced to 100 in val set.\n"
     ]
    }
   ],
   "source": [
    "# If we had multiple references per sentence\n",
    "keep_n_captions(ds, repeat=1, n=1, set_names=['val'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can save our dataset instance for using in other experiments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:09] <<< Saving Dataset instance to datasets/Dataset_tutorial_dataset.pkl ... >>>\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[30/11/2016 16:04:09] <<< Dataset instance saved >>>\n"
     ]
    }
   ],
   "source": [
    "saveDataset(ds, 'datasets')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    ""
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
{'fraction_non_alphanumeric': 0.17682514938945182, 'fraction_numerical': 0.03611327617563003, 'mean_word_length': 2.116256476683938, 'pattern_counts': {'":': 251, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1987459', 'n_tokens_mistral': 7133, 'n_tokens_neox': 6469, 'n_words': 1541}