<jupyter_start><jupyter_text>Loading large datasets**Learning Objectives**  - Understand difference between loading data entirely in-memory and loading in batches from disk  - Practice loading a `.csv` file from disk in batches using the `tf.data` module  IntroductionIn the previous notebook, we read the the whole taxifare .csv files into memory, specifically a Pandas dataframe, before invoking `tf.data.from_tensor_slices` from the tf.data API. We could get away with this because it was a small sample of the dataset, but on the full taxifare dataset this wouldn't be feasible.In this notebook we demonstrate how to read .csv files directly from disk, one batch at a time, using `tf.data.TextLineDataset` Run the following cell and restart the kernel if needed<jupyter_code># Ensure that we have Tensorflow 1.13.1 installed.
!pip3 freeze | grep tensorflow==1.13.1 || pip3 install tensorflow==1.13.1
import tensorflow as tf
import shutil
print(tf.__version__)
tf.enable_eager_execution()<jupyter_output><empty_output><jupyter_text>Input function reading from CSVWe define `read_dataset()` which given a csv file path returns a `tf.data.Dataset` in which each row represents a (features,label) in the Estimator API required format - features: A python dictionary. Each key is a feature column name and its value is the tensor containing the data for that feature- label: A Tensor containing the labelsWe then invoke `read_dataset()` function from within the `train_input_fn()` and `eval_input_fn()`. The remaining code is as before.  **Exercise 1**In the next cell, implement a `parse_row` function that takes as input a csv row (as a string) and returns a tuple (features, labels) as described above.First, use the [tf.decode_csv function](https://www.tensorflow.org/api_docs/python/tf/io/decode_csv) to read in the features from a csv file. Next, once `fields` has been read from the `.csv` file, create a dictionary of features and values. Lastly, define the label and remove it from the `features` dict you created. This can be done in one step with pythons pop operation.The column names and the default values you'll need for these operations are given by global variables `CSV_COLUMN_NAMES`and `CSV_DEFAULTS`. The labels are stored in the first column.<jupyter_code>CSV_COLUMN_NAMES = ["fare_amount","dayofweek","hourofday","pickuplon","pickuplat","dropofflon","dropofflat"]
CSV_DEFAULTS = [[0.0],[1],[0],[-74.0], [40.0], [-74.0], [40.7]]
cols_with_defaults = dict(zip(CSV_COLUMN_NAMES, CSV_DEFAULTS))
label = cols_with_defaults.pop("fare_amount")
cols_with_defaults
CSV_COLUMN_NAMES[1]
def parse_row(row):
    cols = tf.decode_csv(row, record_defaults = CSV_DEFAULTS)
    features = dict(zip(CSV_COLUMN_NAMES, cols))
    labels = features.pop("fare_amount") 
    return features, labels<jupyter_output><empty_output><jupyter_text>Run the following test to make sure your implementation is correct<jupyter_code>a_row = "0.0,1,0,-74.0,40.0,-74.0,40.7"
features, labels = parse_row(a_row)

assert labels.numpy() == 0.0
assert features["pickuplon"].numpy() == -74.0
print("You rock!")<jupyter_output>You rock!<jupyter_text>**Exercise 2**Use the function `parse_row` you implemented in the previous exercise to implement a `read_dataset` function that- takes as input the path to a csv file- returns a `tf.data.Dataset` object containing the features, labelsAssume that the .csv file has a header, and that your `read_dataset` will skip it. Have a look at the [tf.data.TextLineDataset documentation](https://www.tensorflow.org/api_docs/python/tf/data/TextLineDataset) to see what variables to pass when initializing the dataset pipeline. Then use the `parse_row` operation we created above to read the values from the .csv file<jupyter_code>def parse_row_with_feateng(row):
    cols = tf.decode_csv(row, record_defaults = CSV_DEFAULTS)
    features = dict(zip(CSV_COLUMN_NAMES, cols))
    features["mandist"] = abs(features["pickuplat"] - features["dropofflat"]) + abs(features["pickuplon"] - features["dropofflon"])
    labels = features.pop("fare_amount") 
    return features, labels
def read_dataset(csv_path):  
    dataset = tf.data.TextLineDataset(csv_path).skip(1)
    #dataset = dataset.map(parse_row)
    dataset = dataset.map(parse_row_with_feateng)
    return dataset<jupyter_output><empty_output><jupyter_text>TestsLet's create a test dataset to test our function.<jupyter_code>%%writefile test.csv
fare_amount,dayofweek,hourofday,pickuplon,pickuplat,dropofflon,dropofflat
28,1,0,-73.0,41.0,-74.0,20.7
12.3,1,0,-72.0,44.0,-75.0,40.6
10,1,0,-71.0,41.0,-71.0,42.9<jupyter_output>Overwriting test.csv<jupyter_text>You should be able to iterate over what's returned by `read_dataset`. We'll print the `dropofflat` and `fare_amount` for each entry in `./test.csv`<jupyter_code>for feature, label in read_dataset("./test.csv"):
    print("dropofflat:", feature["dropofflat"].numpy())
    print("fare_amount:", label.numpy())
    print("mandist:", feature["mandist"].numpy())<jupyter_output>dropofflat: 20.7
fare_amount: 28.0
mandist: 21.3
dropofflat: 40.6
fare_amount: 12.3
mandist: 6.4000015
dropofflat: 42.9
fare_amount: 10.0
mandist: 1.9000015<jupyter_text>Run the following test cell to make sure you function works properly:<jupyter_code>dataset= read_dataset("./test.csv")
dataset_iterator = dataset.make_one_shot_iterator()
features, labels = dataset_iterator.get_next()

assert features['dayofweek'].numpy() == 1
assert labels.numpy() == 28
print("You rock!")<jupyter_output>You rock!<jupyter_text>**Exercise 3**In the code cell below, implement a `train_input_fn` function that- takes as input a path to a csv file along with a batch_size- returns a dataset object that shuffle the rows and returns them in batches of `batch_size`**Hint:** Reuse the `read_dataset` function you implemented above. Once you've initialized the `dataset`, be sure to add a step to `shuffle`, `repeat` and `batch` to your pipeline.<jupyter_code>def train_input_fn(csv_path, batch_size = 128):
    dataset = read_dataset(csv_path)
    dataset = dataset.shuffle(1000).repeat(None).batch(batch_size)
    return dataset<jupyter_output><empty_output><jupyter_text>**Exercise 4**Next, implement as `eval_input_fn` similar to the `train_input_fn` you implemented above. Remember, the only difference is that this function does not need to shuffle the rows.<jupyter_code>def eval_input_fn(csv_path, batch_size = 128):
    dataset = read_dataset(csv_path)
    dataset = dataset.batch(batch_size)
    return dataset<jupyter_output><empty_output><jupyter_text>Create feature columns The features of our models are the following:<jupyter_code>list(features.keys())
FEATURE_NAMES = list(features.keys())
print(FEATURE_NAMES)<jupyter_output>['mandist', 'hourofday', 'dropofflat', 'pickuplon', 'dayofweek', 'pickuplat', 'dropofflon']<jupyter_text>**Exercise 5**In the cell below, create a variable called `feature_cols` which contains a list of the appropriate `tf.feature_column` to be passed to a `tf.estimator`.<jupyter_code>cat_cols = ['dayofweek']
num_cols = [col for col in FEATURE_NAMES if col not in cat_cols]
vocab = {}
vocab['dayofweek'] = [0, 1, 2, 3, 4, 5, 6]
## how do I avoid hardcoding the vocabulary? 

num_feature_cols = [tf.feature_column.numeric_column(col) for col in num_cols]
cat_feature_cols = [tf.feature_column.categorical_column_with_vocabulary_list(col, vocab[col]) for col in cat_cols]
indic_cols = [tf.feature_column.indicator_column(col) for col in cat_feature_cols]
feature_cols = num_feature_cols + indic_cols
print(feature_cols)<jupyter_output>[NumericColumn(key='mandist', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='hourofday', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='dropofflat', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='pickuplon', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='pickuplat', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='dropofflon', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='dayofweek', vocabulary_list=(0, 1, 2, 3, 4, 5, 6), dtype=tf.int64, default_value=-1, num_oov_buckets=0))]<jupyter_text>Choose Estimator   **Exercise 6**In the cell below, create an instance of a `tf.estimator.DNNRegressor` such that- it has two layers of 10 units each- it uses the features defined in the previous exercise- it saves the trained model into the directory `./taxi_trained`- it has a random seed set to 1 for replicability and debuggingHave a look at [the documentation for Tensorflow's DNNRegressor](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNRegressor) to remind you of the implementation.**Hint:** Remember, the random seed is set by passing a `tf.estimator.RunConfig` object  to the `config` parameter of the `tf.estimator`.<jupyter_code>OUTDIR = "taxi_trained"

model = tf.estimator.DNNRegressor(
    hidden_units = [10,10], # specify neural architecture
    feature_columns = feature_cols, 
    model_dir = OUTDIR,
    config = tf.estimator.RunConfig(tf_random_seed = 1)
)<jupyter_output>INFO:tensorflow:Using config: {'_service': None, '_keep_checkpoint_max': 5, '_tf_random_seed': 1, '_task_type': 'worker', '_train_distribute': None, '_save_checkpoints_secs': 600, '_protocol': None, '_save_checkpoints_steps': None, '_task_id': 0, '_model_dir': 'taxi_trained', '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_is_chief': True, '_log_step_count_steps': 100, '_master': '', '_eval_distribute': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe87b452e10>, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_global_id_in_cluster': 0, '_save_summary_steps': 100, '_keep_checkpoint_every_n_hours': 10000, '_device_fn': None, '_evaluation_master': '', '_experimental_distribute': None}<jupyter_text>TrainNext, we'll train the model.  **Exercise 7**Complete the code in the cell below to train the `DNNRegressor` model you instantiated above on our data. Have a look at [the documentation for the `train` method of the `DNNRegressor`](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNRegressortrain) to see what variables you should pass. You'll use the `train_input_function` you created above and the `./taxi-train.csv` dataset. If you train your model for 500 steps. How many epochs of the dataset does this represent?<jupyter_code>%%time
tf.logging.set_verbosity(tf.logging.INFO) # so loss is printed during training
shutil.rmtree(path = OUTDIR, ignore_errors = True) # start fresh each time

model.train(
    input_fn = lambda: train_input_fn(csv_path = "./taxi-train.csv"),
    steps = 500
)<jupyter_output>INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:From /home/jupyter/.local/lib/python3.5/site-packages/tensorflow/python/feature_column/feature_column_v2.py:3727: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/jupyter/.local/lib/python3.5/site-packages/tensorflow/python/feature_column/feature_column_v2.py:4266: IndicatorColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /home/jupyter/.local/lib/python3.5/site-packages/tensorflow/python/feature_column/feature_column_v2.py:4321: VocabularyListCategoricalColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-[...]<jupyter_text>EvaluateLastly, we'll evaluate our model.  **Exercise 8**In the cell below, evaluate the model using its `.evaluate` method and the `eval_input_fn` function you implemented above on the `./taxi-valid.csv` dataset. Capture the result of running evaluation on the evaluation set in a variable called `metrics`. Then, extract the `average_loss` for the dictionary returned by `model.evaluate` and contained in `metrics`. This is the RMSE.<jupyter_code>metrics = model.evaluate(input_fn = lambda: eval_input_fn("./taxi-valid.csv"))
print("RMSE on dataset = {}".format(metrics["average_loss"]**.5))<jupyter_output>INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-07-17T16:59:50Z
INFO:tensorflow:Graph was finalized.
WARNING:tensorflow:From /home/jupyter/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-500
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-07-17-16:59:53
INFO:tensorflow:Saving dict for global step 500: average_loss = 85.584335, global_step = 500, label/mean = 11.229713, loss = 10936.922, prediction/mean = 11.801599
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 500: taxi_trained/model.ckpt-500
RMSE on dataset = 9.25118[...]<jupyter_text>Challenge exerciseCreate a neural network that is capable of finding the volume of a cylinder given the radius of its base (r) and its height (h). Assume that the radius and height of the cylinder are both in the range 0.5 to 2.0. Unlike in the challenge exercise for c_estimator.ipynb, assume that your measurements of r, h and V are all rounded off to the nearest 0.1. Simulate the necessary training dataset. This time, you will need a lot more data to get a good predictor.Hint (highlight to see):Create random values for r and h and compute V. Then, round off r, h and V (i.e., the volume is computed from the true value of r and h; it's only your measurement that is rounded off). Your dataset will consist of the round values of r, h and V. Do this for both the training and evaluation datasets.Now modify the "noise" so that instead of just rounding off the value, there is up to a 10% error (uniformly distributed) in the measurement followed by rounding off.<jupyter_code>import numpy as np
import math
import pandas as pd
min_dim = 0.5
max_dim = 2.0
n = 100000

radius = np.array([round(r, 1) for r in np.random.uniform(min_dim, max_dim, n)])
height = np.array([round (h, 1) for h in np.random.uniform(min_dim, max_dim, n)])
volume = np.array([round (v, 1) for v in math.pi * height * (radius**2)])

cylinder = pd.DataFrame()
cylinder['radius'] = radius
cylinder['height'] = height
cylinder['volume'] = volume
cylinder.head()
cyl_features = ['radius', 'height']
cyl_target = 'volume'
cyl_feature_columns = [tf.feature_column.numeric_column(col) for col in cyl_features]
def train_input_fn_cyl(df, features, target, batch_size = 128):
    #1. Convert dataframe into correct (features, label) format for Estimator API
    dataset = tf.data.Dataset.from_tensor_slices((dict(df[features]), df[target]))
    
    # Note:
    # If we returned now, the Dataset would iterate over the data once  
    # in a fixed order, and only produce a single element at a time.
    
    #2. Shuffle, repeat, and batch the examples.
    dataset = dataset.shuffle(len(df), seed=42).repeat().batch(batch_size)
   
    return dataset

def eval_input_fn_cyl(df, features, target, batch_size = 128):
    #1. Convert dataframe into correct (features, label) format for Estimator API
    dataset = tf.data.Dataset.from_tensor_slices((dict(df[features]), df[target]))
    
    # Note:
    # If we returned now, the Dataset would iterate over the data once  
    # in a fixed order, and only produce a single element at a time.
    
    #2. Shuffle, repeat, and batch the examples.
    dataset = dataset.batch(batch_size)
   
    return dataset
def print_rmse_cyl(model, df, features, target):
    metrics = model.evaluate(
        input_fn = lambda: eval_input_fn_cyl(df, features, target)
    )
    print("RMSE on dataset = {}".format(metrics["average_loss"]**.5))
cylinder_train = cylinder[:85000]
cylinder_valid = cylinder[85000:]
OUTDIR = "./cyl_trained"

tf.logging.set_verbosity(tf.logging.INFO)
shutil.rmtree(path = OUTDIR, ignore_errors = True)

cyl_model = tf.estimator.DNNRegressor(
    hidden_units = [10,10], # specify neural architecture
    feature_columns = cyl_feature_columns, 
    model_dir = OUTDIR,
    config = tf.estimator.RunConfig(tf_random_seed = 1)
)
cyl_model.train(
    input_fn = lambda: train_input_fn_cyl(cylinder_train, cyl_features, cyl_target), 
    steps = 500)
print_rmse_cyl(cyl_model, cylinder_valid, cyl_features, cyl_target)<jupyter_output>INFO:tensorflow:Using config: {'_protocol': None, '_task_type': 'worker', '_log_step_count_steps': 100, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_eval_distribute': None, '_experimental_distribute': None, '_keep_checkpoint_max': 5, '_save_summary_steps': 100, '_num_worker_replicas': 1, '_task_id': 0, '_num_ps_replicas': 0, '_keep_checkpoint_every_n_hours': 10000, '_save_checkpoints_steps': None, '_master': '', '_device_fn': None, '_evaluation_master': '', '_train_distribute': None, '_model_dir': './cyl_trained', '_is_chief': True, '_service': None, '_save_checkpoints_secs': 600, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fef08235a20>, '_tf_random_seed': 1, '_global_id_in_cluster': 0}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_i[...]
{'fraction_non_alphanumeric': 0.10711909514304724, 'fraction_numerical': 0.024839210467952982, 'mean_word_length': 5.9266513056835635, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '417845', 'license': 'Apache-2.0', 'n_tokens_mistral': 6053, 'n_tokens_neox': 5678, 'n_words': 1736, 'path': 'courses/machine_learning/deepdive/02_tensorflow/labs/d_csv_input_fn.ipynb', 'repo_name': 'aleistra/training-data-analyst', 'size': '18036'}