<jupyter_start><jupyter_text>13. Loading and Preprocessing Data with TensorFlow Introducing the **Data API**, a useful tool in ingesting and preprocessing large datasets.   The Data APIThe whole API revolves around the concept of **dataset**. Here a simple example:<jupyter_code>import tensorflow as tf

X = tf.range(10) # any data tensor
dataset = tf.data.Dataset.from_tensor_slices(X)
dataset<jupyter_output><empty_output><jupyter_text>Chaining Transformations We can apply all sorts of transformation to our datasets, and these will create new datasets. We can also chain them:<jupyter_code>dataset = dataset.repeat(3).batch(7)
for item in dataset:
    print(item)<jupyter_output>tf.Tensor([0 1 2 3 4 5 6], shape=(7,), dtype=int32)
tf.Tensor([7 8 9 0 1 2 3], shape=(7,), dtype=int32)
tf.Tensor([4 5 6 7 8 9 0], shape=(7,), dtype=int32)
tf.Tensor([1 2 3 4 5 6 7], shape=(7,), dtype=int32)
tf.Tensor([8 9], shape=(2,), dtype=int32)<jupyter_text>We can transform it using `map()`:<jupyter_code>dataset = dataset.map(lambda x: x * 2) # items: [0,2,4,6,8,10,12]<jupyter_output><empty_output><jupyter_text>Shuffling the Data Example:<jupyter_code>dataset = tf.data.Dataset.range(10).repeat(3) # 0 to 9, three times
dataset = dataset.shuffle(buffer_size=5, seed=42).batch(7)
for item in dataset:
    print(item)<jupyter_output>tf.Tensor([0 2 3 6 7 9 4], shape=(7,), dtype=int64)
tf.Tensor([5 0 1 1 8 6 5], shape=(7,), dtype=int64)
tf.Tensor([4 8 7 1 2 3 0], shape=(7,), dtype=int64)
tf.Tensor([5 4 2 7 8 9 9], shape=(7,), dtype=int64)
tf.Tensor([3 6], shape=(2,), dtype=int64)<jupyter_text>Another method to shuffle the data is **interleaving lines from multiple files**. The idea is simple: take the dataset and split it into a training set, a validation set, and a test set. Then split each set into many files. Now let's suppose we have a list `train_filepaths` that contains the list of training file paths. Next, we create a dataset containing filepaths:<jupyter_code>filepath_dataset = tf.data.Dataset.list_files(train_filepaths, seed=42)<jupyter_output><empty_output><jupyter_text>And finally`interleave` to read one line at the time from all of them (sequentially):<jupyter_code>n_readers = 5
dataset = filepath_dataset.interleave(
    lambda filepath: tf.data.TextLineDataset(filepath).skip(1),
    cycle_length=n_readers)<jupyter_output><empty_output><jupyter_text>TFRecord FormatUseful when bottleneck is loading and parsing the data. We can create a TFRecord file easily:<jupyter_code>with tf.io.TFRecordWriter("my_data.tfrecord") as f:
    f.write(b"This is the first record")
    f.write(b"And this is the second record")<jupyter_output><empty_output><jupyter_text>Then we use `tf.data.TFRecordDataset` to read it:<jupyter_code>filepaths = ["my_data.tfrecord"]
dataset = tf.data.TFRecordDataset(filepaths)
for item in dataset:
    print(item)<jupyter_output>tf.Tensor(b'This is the first record', shape=(), dtype=string)
tf.Tensor(b'And this is the second record', shape=(), dtype=string)<jupyter_text>Preprocessing the Input FeaturesAka converting all features into numerical features, generally normalizing them, etc. We have several options, some already covered:1. Preparing data files in advance using another package (e.g. pandas)2. Preprocessing using Data API3. Preprocessing layer directly embedded in our modelLet's cover `3`. Let's implement a standardization layer using a `Lambda` layer. For each feature, it subtracts the mean and divides by its standard deviation (plus a tiny smoothing term to avoid division by zero):<jupyter_code>import numpy as np

means = np.mean(X_train, axis=0, keepdims=True)
stds = np.std(X_train, axis=0, keepdims=True)
eps = keras.backend.epsilon()
model = keras.models.Sequential([
    keras.layers.Lambda(lambda inputs: (inputs - means) / (stds + eps)),
    [...] # other layers
])<jupyter_output><empty_output><jupyter_text>However, we may prefer to use a nice self-contained custom layer instead of having global variables like `means` and `stds`:<jupyter_code>from tensorflow import keras 

class Standardization(keras.layers.Layer):
    def adapt(self, data_sample):
        self.means_ = np.mean(data_sample, axis=0, keepdims=True)
        self.stds_ = np.std(data_sample, axis=0, keepdims=True)
    def call(self, inputs):
        return (inputs - self.means_) / (self.stds_ +
keras.backend.epsilon())<jupyter_output><empty_output><jupyter_text>Encoding Categorical Features Using One-Hot Vectors Encoding categorical features is an essential step in any preprocesssing pipeline. Let's use how this can be done:<jupyter_code># list of all possible categories
vocab = ["<1H OCEAN", "INLAND", "NEAR OCEAN", "NEAR BAY", "ISLAND"]
# tensor with indices (0-4)
indices = tf.range(len(vocab), dtype=tf.int64)
# initializer for the lookup table
table_init = tf.lookup.KeyValueTensorInitializer(vocab, indices)
# out of vocab buckets
num_oov_buckets = 2
# lookup for oov buckets
table = tf.lookup.StaticVocabularyTable(table_init, num_oov_buckets)<jupyter_output><empty_output><jupyter_text>Now let’s use the lookup table to encode a small batch of categorical features to one-hot vectors:<jupyter_code>categories = tf.constant(["NEAR BAY", "DESERT", "INLAND", "INLAND"])
cat_indices = table.lookup(categories)
cat_one_hot = tf.one_hot(cat_indices, depth=len(vocab) + num_oov_buckets)
cat_one_hot<jupyter_output><empty_output><jupyter_text>**Tip**: This works well with relatively few (50) vocabs, it may be better to switch to **embeddings**.  Encoding Categorical Features Using Embeddings An embedding is a trainable dense vector that represents a category. In some cases (e.g. word embeddings) they can be reused. In addition to important concept of **representation learning** embeddings can also encode concepts (e.g. King - Man + Woman $\approx$ Queen).Here how it works in a simplified 2D embedding example:<jupyter_code>embedding_dim = 2 #typically 10-300
embed_init = tf.random.uniform([len(vocab) + num_oov_buckets,
embedding_dim])
embedding_matrix = tf.Variable(embed_init)
embedding_matrix<jupyter_output><empty_output><jupyter_text>Encoding:<jupyter_code>categories = tf.constant(["NEAR BAY", "DESERT", "INLAND", "INLAND"])
cat_indices = table.lookup(categories)
cat_indices
tf.nn.embedding_lookup(embedding_matrix, cat_indices)<jupyter_output><empty_output><jupyter_text>Putting everything together, we can create a model that learns categorical features through embeddings.<jupyter_code>regular_inputs = keras.layers.Input(shape=[8])
categories = keras.layers.Input(shape=[], dtype=tf.string)
# look up each category
cat_indices = keras.layers.Lambda(lambda cats: table.lookup(cats))
(categories)
# look up embeddings
cat_embed = keras.layers.Embedding(input_dim=6, output_dim=2)
(cat_indices)
encoded_inputs = keras.layers.concatenate([regular_inputs, cat_embed])
outputs = keras.layers.Dense(1)(encoded_inputs)
model = keras.models.Model(inputs=[regular_inputs, categories],
                            outputs=[outputs])<jupyter_output><empty_output><jupyter_text>TF TransformIt may happen that we want to deploy our production model, and having to modify preprocessing for every platform can definitely be a problem in the long run. TF Transform takes care of making sure that there is only one preprocessing step.<jupyter_code>import tensorflow_transform as tft

def preprocess(inputs): # inputs = a batch of input features
    median_age = inputs["housing_median_age"]
    ocean_proximity = inputs["ocean_proximity"]
    standardized_age = tft.scale_to_z_score(median_age)
    ocean_proximity_id = tft.compute_and_apply_vocabulary(ocean_proximity)
    return {
        "standardized_median_age": standardized_age,
        "ocean_proximity_id": ocean_proximity_id
    }<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10615028306742151, 'fraction_numerical': 0.01878538342768914, 'mean_word_length': 6.047144152311876, 'pattern_counts': {'":': 2, '<': 63, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '508862', 'license': 'MIT', 'n_tokens_mistral': 2620, 'n_tokens_neox': 2478, 'n_words': 756, 'path': '.ipynb_checkpoints/13. Loading_and_Preprocessing_Data_with_TensorFlow-checkpoint.ipynb', 'repo_name': 'GiuseppeBaldini/ML-Scikit-Keras-TensorFlow', 'size': '7772'}