<commit_before># Scalable Tensorflow

## Set Up

Make sure you have installed gcloud, and created an output bucket (hereafter referred to as `$BUCKET_NAME`), which you have authorized the Google Cloud ML API to access.

## Data I/O

TFRecords are a scalable way to provide data to TensorFlow training processes. They allow you to read data from files (local or Google Cloud Storage) as part of TensorFlow graph execution.

### Writing TFRecords - Exercise 1

```
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("data",
                                  one_hot=False,
                                  validation_size=0)
```

Gives you a `Dataset` object with two members `mnist.train` and `mnist.test`, each of these datasets has `images` A numpy array where each row is an images pixels (flattened), and `labels` A numpy vector`.

Write a Python CLI which writes each of `mnist.train` and `mnist.test` to a TFRecords file, containing `tf.train.Examples` with two features:

* `'labels'`: An `tf.train.Int64List` containing the single label value for the example
* `'images'`: A `tf.train.FloatList` containing the flattened array of pixels for the example

The solution can be fonud in `prepare_data.py` and run with

```
python prepare_data.py data/
```

After you have successfully completed this, run

```
gsutil -m cp -z -r data/* gs://$BUCKET_NAME/
```

### Reading TFRecords - Exercise 2

Next we need to provide an `input_fn` to our model. When called `input_fn` will add TFRecord reading operations to our graph, and return two Tensors: `features` and `targets` which represent the training data and labels of our model.

For readability we break this down into two functions you must implement:

* `parse_examples` should take a string Tensor of TFRecord Examples, and return two tensors, representing `'images'` and `'labels'` from part one

* `make_input_fn` should have the signature:

```
def make_input_fn(files, example_parser, batch_size, num_epochs=None):
```

Which takes a list of file names, our `parse_examples` function, an integer batch size, and an optional number of times to read through the data, and returns a valid `input_fn`, which when called with no arguments, returns two tensors representing a *batch* of `'images'` and `'labels'` respectively.

To validate that your functions are working, you can `mv trainer/util.py trainer/util_solution.py` and in your own `trainer/util.py` first implement `parse_examples`, and `from util_solution import * `

Then run:

```<commit_msg>Specify file ext to gzip<commit_after># Scalable Tensorflow

## Set Up

Make sure you have installed gcloud, and created an output bucket (hereafter referred to as `$BUCKET_NAME`), which you have authorized the Google Cloud ML API to access.

## Data I/O

TFRecords are a scalable way to provide data to TensorFlow training processes. They allow you to read data from files (local or Google Cloud Storage) as part of TensorFlow graph execution.

### Writing TFRecords - Exercise 1

```
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("data",
                                  one_hot=False,
                                  validation_size=0)
```

Gives you a `Dataset` object with two members `mnist.train` and `mnist.test`, each of these datasets has `images` A numpy array where each row is an images pixels (flattened), and `labels` A numpy vector`.

Write a Python CLI which writes each of `mnist.train` and `mnist.test` to a TFRecords file, containing `tf.train.Examples` with two features:

* `'labels'`: An `tf.train.Int64List` containing the single label value for the example
* `'images'`: A `tf.train.FloatList` containing the flattened array of pixels for the example

The solution can be fonud in `prepare_data.py` and run with

```
python prepare_data.py data/
```

After you have successfully completed this, run

```
gsutil -m cp -z pb2 -r data/* gs://$BUCKET_NAME/
```

### Reading TFRecords - Exercise 2

Next we need to provide an `input_fn` to our model. When called `input_fn` will add TFRecord reading operations to our graph, and return two Tensors: `features` and `targets` which represent the training data and labels of our model.

For readability we break this down into two functions you must implement:

* `parse_examples` should take a string Tensor of TFRecord Examples, and return two tensors, representing `'images'` and `'labels'` from part one

* `make_input_fn` should have the signature:

```
def make_input_fn(files, example_parser, batch_size, num_epochs=None):
```

Which takes a list of file names, our `parse_examples` function, an integer batch size, and an optional number of times to read through the data, and returns a valid `input_fn`, which when called with no arguments, returns two tensors representing a *batch* of `'images'` and `'labels'` respectively.

To validate that your functions are working, you can `mv trainer/util.py trainer/util_solution.py` and in your own `trainer/util.py` first implement `parse_examples`, and `from util_solution import * `

Then run:

```
{'fraction_non_alphanumeric': 0.08230934269553344, 'fraction_numerical': 0.0021455041934854693, 'mean_word_length': 4.211382113821138, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '819046', 'n_tokens_mistral': 1457, 'n_tokens_neox': 1399, 'n_words': 679}