<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TF-Slim Walkthrough\n",
    "\n",
    "This notebook will walk you through the basics of using TF-Slim to define, train and evaluate neural networks on various tasks. It assumes a basic knowledge of neural networks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of contents\n",
    "\n",
    "<a href=\"#Install\">Installation and setup</a><br>\n",
    "<a href='#MLP'>Creating your first neural network with TF-Slim</a><br>\n",
    "<a href='#ReadingTFSlimDatasets'>Reading Data with TF-Slim</a><br>\n",
    "<a href='#CNN'>Training a convolutional neural network (CNN)</a><br>\n",
    "<a href='#Pretained'>Using pre-trained models</a><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation and setup\n",
    "<a id='Install'></a>\n",
    "\n",
    "As of 8/28/16, the latest stable release of TF is r0.10, which does not contain the latest version of slim.\n",
    "To obtain the latest version of TF-Slim, please install the most recent nightly build of TF\n",
    "as explained [here](https://github.com/tensorflow/models/tree/master/slim#installing-latest-version-of-tf-slim).\n",
    "\n",
    "To use TF-Slim for image classification (as we do in this notebook), you also have to install the TF-Slim image models library from [here](https://github.com/tensorflow/models/tree/master/slim). Let's suppose you install this into a directory called TF_MODELS. Then you should change directory to  TF_MODELS/slim **before** running this notebook, so that these files are in your python path.\n",
    "\n",
    "To check you've got these two steps to work, just execute the cell below. If it complains about unknown modules, restart the notebook after moving to the TF-Slim models directory.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import time\n",
    "\n",
    "from datasets import dataset_utils\n",
    "\n",
    "# Main slim library\n",
    "slim = tf.contrib.slim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating your first neural network with TF-Slim\n",
    "<a id='MLP'></a>\n",
    "\n",
    "Below we give some code to create a simple multilayer perceptron (MLP)  which can be used\n",
    "for regression problems. The model has 2 hidden layers.\n",
    "The output is a single node. \n",
    "When this function is called, it will create various nodes, and silently add them to whichever global TF graph is currently in scope. When a node which corresponds to a layer with adjustable parameters (eg., a fully connected layer) is created, additional parameter variable nodes are silently created, and added to the graph. (We will discuss how to train the parameters later.)\n",
    "\n",
    "We use variable scope to put all the nodes under a common name,\n",
    "so that the graph has some hierarchical structure.\n",
    "This is useful when we want to visualize the TF graph in tensorboard, or if we want to query related\n",
    "variables. \n",
    "The fully connected layers all use the same L2 weight decay and ReLu activations, as specified by **arg_scope**. (However, the final layer overrides these defaults, and uses an identity activation function.)\n",
    "\n",
    "We also illustrate how to add a dropout layer after the first fully connected layer (FC1). Note that at test time, \n",
    "we do not drop out nodes, but instead use the average activations; hence we need to know whether the model is being\n",
    "constructed for training or testing, since the computational graph will be different in the two cases\n",
    "(although the variables, storing the model parameters, will be shared, since they have the same name/scope)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def regression_model(inputs, is_training=True, scope=\"deep_regression\"):\n",
    "    \"\"\"Creates the regression model.\n",
    "\n",
    "    Args:\n",
    "        inputs: A node that yields a `Tensor` of size [batch_size, dimensions].\n",
    "        is_training: Whether or not we're currently training the model.\n",
    "        scope: An optional variable_op scope for the model.\n",
    "\n",
    "    Returns:\n",
    "        predictions: 1-D `Tensor` of shape [batch_size] of responses.\n",
    "        end_points: A dict of end points representing the hidden layers.\n",
    "    \"\"\"\n",
    "    with tf.variable_scope(scope, 'deep_regression', [inputs]):\n",
    "        end_points = {}\n",
    "        # Set the default weight _regularizer and acvitation for each fully_connected layer.\n",
    "        with slim.arg_scope([slim.fully_connected],\n",
    "                            activation_fn=tf.nn.relu,\n",
    "                            weights_regularizer=slim.l2_regularizer(0.01)):\n",
    "\n",
    "            # Creates a fully connected layer from the inputs with 32 hidden units.\n",
    "            net = slim.fully_connected(inputs, 32, scope='fc1')\n",
    "            end_points['fc1'] = net\n",
    "\n",
    "            # Adds a dropout layer to prevent over-fitting.\n",
    "            net = slim.dropout(net, 0.8, is_training=is_training)\n",
    "\n",
    "            # Adds another fully connected layer with 16 hidden units.\n",
    "            net = slim.fully_connected(net, 16, scope='fc2')\n",
    "            end_points['fc2'] = net\n",
    "\n",
    "            # Creates a fully-connected layer with a single hidden unit. Note that the\n",
    "            # layer is made linear by setting activation_fn=None.\n",
    "            predictions = slim.fully_connected(net, 1, activation_fn=None, scope='prediction')\n",
    "            end_points['out'] = predictions\n",
    "\n",
    "            return predictions, end_points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's create the model and examine its structure.\n",
    "\n",
    "We create a TF graph and call regression_model(), which adds nodes (tensors) to the graph. We then examine their shape, and print the names of all the model variables which have been implicitly created inside of each layer. We see that the names of the variables follow the scopes that we specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with tf.Graph().as_default():\n",
    "    # Dummy placeholders for arbitrary number of 1d inputs and outputs\n",
    "    inputs = tf.placeholder(tf.float32, shape=(None, 1))\n",
    "    outputs = tf.placeholder(tf.float32, shape=(None, 1))\n",
    "\n",
    "    # Build model\n",
    "    predictions, end_points = regression_model(inputs)\n",
    "\n",
    "    # Print name and shape of each tensor.\n",
    "    print \"Layers\"\n",
    "    for k, v in end_points.iteritems():\n",
    "        print 'name = {}, shape = {}'.format(v.name, v.get_shape())\n",
    "\n",
    "    # Print name and shape of parameter nodes  (values not yet initialized)\n",
    "    print \"\\n\"\n",
    "    print \"Parameters\"\n",
    "    for v in slim.get_model_variables():\n",
    "        print 'name = {}, shape = {}'.format(v.name, v.get_shape())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's create some 1d regression data .\n",
    "\n",
    "We will train and test the model on some noisy observations of a nonlinear function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def produce_batch(batch_size, noise=0.3):\n",
    "    xs = np.random.random(size=[batch_size, 1]) * 10\n",
    "    ys = np.sin(xs) + 5 + np.random.normal(size=[batch_size, 1], scale=noise)\n",
    "    return [xs.astype(np.float32), ys.astype(np.float32)]\n",
    "\n",
    "x_train, y_train = produce_batch(200)\n",
    "x_test, y_test = produce_batch(200)\n",
    "plt.scatter(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's fit the model to the data\n",
    "\n",
    "The user has to specify the loss function and the optimizer, and slim does the rest.\n",
    "In particular,  the slim.learning.train function does the following:\n",
    "\n",
    "- For each iteration, evaluate the train_op, which updates the parameters using the optimizer applied to the current minibatch. Also, update the global_step.\n",
    "- Occasionally store the model checkpoint in the specified directory. This is useful in case your machine crashes  - then you can simply restart from the specified checkpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convert_data_to_tensors(x, y):\n",
    "    inputs = tf.constant(x)\n",
    "    inputs.set_shape([None, 1])\n",
    "    \n",
    "    outputs = tf.constant(y)\n",
    "    outputs.set_shape([None, 1])\n",
    "    return inputs, outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The following snippet trains the regression model using a sum_of_squares loss.\n",
    "ckpt_dir = '/tmp/regression_model/'\n",
    "\n",
    "with tf.Graph().as_default():\n",
    "    tf.logging.set_verbosity(tf.logging.INFO)\n",
    "    \n",
    "    inputs, targets = convert_data_to_tensors(x_train, y_train)\n",
    "\n",
    "    # Make the model.\n",
    "    predictions, nodes = regression_model(inputs, is_training=True)\n",
    "\n",
    "    # Add the loss function to the graph.\n",
    "    loss = slim.losses.sum_of_squares(predictions, targets)\n",
    "    \n",
    "    # The total loss is the uers's loss plus any regularization losses.\n",
    "    total_loss = slim.losses.get_total_loss()\n",
    "\n",
    "    # Specify the optimizer and create the train op:\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=0.005)\n",
    "    train_op = slim.learning.create_train_op(total_loss, optimizer) \n",
    "\n",
    "    # Run the training inside a session.\n",
    "    final_loss = slim.learning.train(\n",
    "        train_op,\n",
    "        logdir=ckpt_dir,\n",
    "        number_of_steps=5000,\n",
    "        save_summaries_secs=5,\n",
    "        log_every_n_steps=500)\n",
    "  \n",
    "print(\"Finished training. Last batch loss:\", final_loss)\n",
    "print(\"Checkpoint saved in %s\" % ckpt_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training with multiple loss functions.\n",
    "\n",
    "Sometimes we have multiple objectives we want to simultaneously optimize.\n",
    "In slim, it is easy to add more losses, as we show below. (We do not optimize the total loss in this example,\n",
    "but we show how to compute it.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with tf.Graph().as_default():\n",
    "    inputs, targets = convert_data_to_tensors(x_train, y_train)\n",
    "    predictions, end_points = regression_model(inputs, is_training=True)\n",
    "\n",
    "    # Add multiple loss nodes.\n",
    "    sum_of_squares_loss = slim.losses.sum_of_squares(predictions, targets)\n",
    "    absolute_difference_loss = slim.losses.absolute_difference(predictions, targets)\n",
    "\n",
    "    # The following two ways to compute the total loss are equivalent\n",
    "    regularization_loss = tf.add_n(slim.losses.get_regularization_losses())\n",
    "    total_loss1 = sum_of_squares_loss + absolute_difference_loss + regularization_loss\n",
    "\n",
    "    # Regularization Loss is included in the total loss by default.\n",
    "    # This is good for training, but not for testing.\n",
    "    total_loss2 = slim.losses.get_total_loss(add_regularization_losses=True)\n",
    "    \n",
    "    init_op = tf.initialize_all_variables()\n",
    "    \n",
    "    with tf.Session() as sess:\n",
    "        sess.run(init_op) # Will initialize the parameters with random weights.\n",<commit_msg>Update deprecated sum_of_squares (TF 0.10) to mean_squared_error (TF 0.12)<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TF-Slim Walkthrough\n",
    "\n",
    "This notebook will walk you through the basics of using TF-Slim to define, train and evaluate neural networks on various tasks. It assumes a basic knowledge of neural networks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of contents\n",
    "\n",
    "<a href=\"#Install\">Installation and setup</a><br>\n",
    "<a href='#MLP'>Creating your first neural network with TF-Slim</a><br>\n",
    "<a href='#ReadingTFSlimDatasets'>Reading Data with TF-Slim</a><br>\n",
    "<a href='#CNN'>Training a convolutional neural network (CNN)</a><br>\n",
    "<a href='#Pretained'>Using pre-trained models</a><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation and setup\n",
    "<a id='Install'></a>\n",
    "\n",
    "As of 8/28/16, the latest stable release of TF is r0.10, which does not contain the latest version of slim.\n",
    "To obtain the latest version of TF-Slim, please install the most recent nightly build of TF\n",
    "as explained [here](https://github.com/tensorflow/models/tree/master/slim#installing-latest-version-of-tf-slim).\n",
    "\n",
    "To use TF-Slim for image classification (as we do in this notebook), you also have to install the TF-Slim image models library from [here](https://github.com/tensorflow/models/tree/master/slim). Let's suppose you install this into a directory called TF_MODELS. Then you should change directory to  TF_MODELS/slim **before** running this notebook, so that these files are in your python path.\n",
    "\n",
    "To check you've got these two steps to work, just execute the cell below. If it complains about unknown modules, restart the notebook after moving to the TF-Slim models directory.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import time\n",
    "\n",
    "from datasets import dataset_utils\n",
    "\n",
    "# Main slim library\n",
    "slim = tf.contrib.slim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating your first neural network with TF-Slim\n",
    "<a id='MLP'></a>\n",
    "\n",
    "Below we give some code to create a simple multilayer perceptron (MLP)  which can be used\n",
    "for regression problems. The model has 2 hidden layers.\n",
    "The output is a single node. \n",
    "When this function is called, it will create various nodes, and silently add them to whichever global TF graph is currently in scope. When a node which corresponds to a layer with adjustable parameters (eg., a fully connected layer) is created, additional parameter variable nodes are silently created, and added to the graph. (We will discuss how to train the parameters later.)\n",
    "\n",
    "We use variable scope to put all the nodes under a common name,\n",
    "so that the graph has some hierarchical structure.\n",
    "This is useful when we want to visualize the TF graph in tensorboard, or if we want to query related\n",
    "variables. \n",
    "The fully connected layers all use the same L2 weight decay and ReLu activations, as specified by **arg_scope**. (However, the final layer overrides these defaults, and uses an identity activation function.)\n",
    "\n",
    "We also illustrate how to add a dropout layer after the first fully connected layer (FC1). Note that at test time, \n",
    "we do not drop out nodes, but instead use the average activations; hence we need to know whether the model is being\n",
    "constructed for training or testing, since the computational graph will be different in the two cases\n",
    "(although the variables, storing the model parameters, will be shared, since they have the same name/scope)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def regression_model(inputs, is_training=True, scope=\"deep_regression\"):\n",
    "    \"\"\"Creates the regression model.\n",
    "\n",
    "    Args:\n",
    "        inputs: A node that yields a `Tensor` of size [batch_size, dimensions].\n",
    "        is_training: Whether or not we're currently training the model.\n",
    "        scope: An optional variable_op scope for the model.\n",
    "\n",
    "    Returns:\n",
    "        predictions: 1-D `Tensor` of shape [batch_size] of responses.\n",
    "        end_points: A dict of end points representing the hidden layers.\n",
    "    \"\"\"\n",
    "    with tf.variable_scope(scope, 'deep_regression', [inputs]):\n",
    "        end_points = {}\n",
    "        # Set the default weight _regularizer and acvitation for each fully_connected layer.\n",
    "        with slim.arg_scope([slim.fully_connected],\n",
    "                            activation_fn=tf.nn.relu,\n",
    "                            weights_regularizer=slim.l2_regularizer(0.01)):\n",
    "\n",
    "            # Creates a fully connected layer from the inputs with 32 hidden units.\n",
    "            net = slim.fully_connected(inputs, 32, scope='fc1')\n",
    "            end_points['fc1'] = net\n",
    "\n",
    "            # Adds a dropout layer to prevent over-fitting.\n",
    "            net = slim.dropout(net, 0.8, is_training=is_training)\n",
    "\n",
    "            # Adds another fully connected layer with 16 hidden units.\n",
    "            net = slim.fully_connected(net, 16, scope='fc2')\n",
    "            end_points['fc2'] = net\n",
    "\n",
    "            # Creates a fully-connected layer with a single hidden unit. Note that the\n",
    "            # layer is made linear by setting activation_fn=None.\n",
    "            predictions = slim.fully_connected(net, 1, activation_fn=None, scope='prediction')\n",
    "            end_points['out'] = predictions\n",
    "\n",
    "            return predictions, end_points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's create the model and examine its structure.\n",
    "\n",
    "We create a TF graph and call regression_model(), which adds nodes (tensors) to the graph. We then examine their shape, and print the names of all the model variables which have been implicitly created inside of each layer. We see that the names of the variables follow the scopes that we specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with tf.Graph().as_default():\n",
    "    # Dummy placeholders for arbitrary number of 1d inputs and outputs\n",
    "    inputs = tf.placeholder(tf.float32, shape=(None, 1))\n",
    "    outputs = tf.placeholder(tf.float32, shape=(None, 1))\n",
    "\n",
    "    # Build model\n",
    "    predictions, end_points = regression_model(inputs)\n",
    "\n",
    "    # Print name and shape of each tensor.\n",
    "    print \"Layers\"\n",
    "    for k, v in end_points.iteritems():\n",
    "        print 'name = {}, shape = {}'.format(v.name, v.get_shape())\n",
    "\n",
    "    # Print name and shape of parameter nodes  (values not yet initialized)\n",
    "    print \"\\n\"\n",
    "    print \"Parameters\"\n",
    "    for v in slim.get_model_variables():\n",
    "        print 'name = {}, shape = {}'.format(v.name, v.get_shape())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's create some 1d regression data .\n",
    "\n",
    "We will train and test the model on some noisy observations of a nonlinear function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def produce_batch(batch_size, noise=0.3):\n",
    "    xs = np.random.random(size=[batch_size, 1]) * 10\n",
    "    ys = np.sin(xs) + 5 + np.random.normal(size=[batch_size, 1], scale=noise)\n",
    "    return [xs.astype(np.float32), ys.astype(np.float32)]\n",
    "\n",
    "x_train, y_train = produce_batch(200)\n",
    "x_test, y_test = produce_batch(200)\n",
    "plt.scatter(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's fit the model to the data\n",
    "\n",
    "The user has to specify the loss function and the optimizer, and slim does the rest.\n",
    "In particular,  the slim.learning.train function does the following:\n",
    "\n",
    "- For each iteration, evaluate the train_op, which updates the parameters using the optimizer applied to the current minibatch. Also, update the global_step.\n",
    "- Occasionally store the model checkpoint in the specified directory. This is useful in case your machine crashes  - then you can simply restart from the specified checkpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convert_data_to_tensors(x, y):\n",
    "    inputs = tf.constant(x)\n",
    "    inputs.set_shape([None, 1])\n",
    "    \n",
    "    outputs = tf.constant(y)\n",
    "    outputs.set_shape([None, 1])\n",
    "    return inputs, outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The following snippet trains the regression model using a mean_squared_error loss.\n",
    "ckpt_dir = '/tmp/regression_model/'\n",
    "\n",
    "with tf.Graph().as_default():\n",
    "    tf.logging.set_verbosity(tf.logging.INFO)\n",
    "    \n",
    "    inputs, targets = convert_data_to_tensors(x_train, y_train)\n",
    "\n",
    "    # Make the model.\n",
    "    predictions, nodes = regression_model(inputs, is_training=True)\n",
    "\n",
    "    # Add the loss function to the graph.\n",
    "    loss = slim.losses.mean_squared_error(predictions, targets)\n",
    "    \n",
    "    # The total loss is the uers's loss plus any regularization losses.\n",
    "    total_loss = slim.losses.get_total_loss()\n",
    "\n",
    "    # Specify the optimizer and create the train op:\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=0.005)\n",
    "    train_op = slim.learning.create_train_op(total_loss, optimizer) \n",
    "\n",
    "    # Run the training inside a session.\n",
    "    final_loss = slim.learning.train(\n",
    "        train_op,\n",
    "        logdir=ckpt_dir,\n",
    "        number_of_steps=5000,\n",
    "        save_summaries_secs=5,\n",
    "        log_every_n_steps=500)\n",
    "  \n",
    "print(\"Finished training. Last batch loss:\", final_loss)\n",
    "print(\"Checkpoint saved in %s\" % ckpt_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training with multiple loss functions.\n",
    "\n",
    "Sometimes we have multiple objectives we want to simultaneously optimize.\n",
    "In slim, it is easy to add more losses, as we show below. (We do not optimize the total loss in this example,\n",
    "but we show how to compute it.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with tf.Graph().as_default():\n",
    "    inputs, targets = convert_data_to_tensors(x_train, y_train)\n",
    "    predictions, end_points = regression_model(inputs, is_training=True)\n",
    "\n",
    "    # Add multiple loss nodes.\n",
    "    mean_squared_error_loss = slim.losses.mean_squared_error(predictions, targets)\n",
    "    absolute_difference_loss = slim.losses.absolute_difference(predictions, targets)\n",
    "\n",
    "    # The following two ways to compute the total loss are equivalent\n",
    "    regularization_loss = tf.add_n(slim.losses.get_regularization_losses())\n",
    "    total_loss1 = mean_squared_error_loss + absolute_difference_loss + regularization_loss\n",
    "\n",
    "    # Regularization Loss is included in the total loss by default.\n",
    "    # This is good for training, but not for testing.\n",
    "    total_loss2 = slim.losses.get_total_loss(add_regularization_losses=True)\n",
    "    \n",
    "    init_op = tf.initialize_all_variables()\n",
    "    \n",
    "    with tf.Session() as sess:\n",
    "        sess.run(init_op) # Will initialize the parameters with random weights.\n",
{'fraction_non_alphanumeric': 0.14655746698538344, 'fraction_numerical': 0.0058779732748148435, 'mean_word_length': 2.7055321620444315, 'pattern_counts': {'":': 134, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4708779', 'n_tokens_mistral': 8330, 'n_tokens_neox': 8077, 'n_words': 2471}