<jupyter_start><jupyter_text>Copyright 2021 The TensorFlow IO Authors.<jupyter_code>#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.<jupyter_output><empty_output><jupyter_text>Tensorflow datasets from MongoDB collections        View on TensorFlow.org        Run in Google Colab        View source on GitHub            Download notebook    OverviewThis tutorial focuses on preparing `tf.data.Dataset`s by reading data from mongoDB collections and using it for training a `tf.keras` model.**NOTE:** A basic understanding of [mongodb storage](https://docs.mongodb.com/guides/) will help you in following the tutorial with ease.  Setup packagesThis tutorial uses `pymongo` as a helper package to create a new mongodb database and collection to store the data.   Install the required tensorflow-io and mongodb (helper) packages<jupyter_code>!pip install tensorflow-io-nightly
!pip install pymongo<jupyter_output><empty_output><jupyter_text>Import packages<jupyter_code>import os
import time
from pprint import pprint
from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras.layers.experimental import preprocessing
import tensorflow_io as tfio
from pymongo import MongoClient<jupyter_output><empty_output><jupyter_text>Validate tf and tfio imports<jupyter_code>print("tensorflow-io version: {}".format(tfio.__version__))
print("tensorflow version: {}".format(tf.__version__))<jupyter_output><empty_output><jupyter_text>Download and setup the MongoDB instanceFor demo purposes, the open-source version of mongodb is used.<jupyter_code>%%bash

sudo apt install -y mongodb >log
service mongodb start
# Sleep for few seconds to let the instance start.
time.sleep(5)<jupyter_output><empty_output><jupyter_text>Once the instance has been started, grep for `mongo` in the processes list to confirm the availability.<jupyter_code>%%bash

ps -ef | grep mongo<jupyter_output><empty_output><jupyter_text>query the base endpoint to retrieve information about the cluster.<jupyter_code>client = MongoClient()
client.list_database_names() # ['admin', 'local']<jupyter_output><empty_output><jupyter_text>Explore the datasetFor the purpose of this tutorial, lets download the [PetFinder](https://www.kaggle.com/c/petfinder-adoption-prediction) dataset and feed the data into mongodb manually. The goal of this classification problem is predict if the pet will be adopted or not.<jupyter_code>dataset_url = 'http://storage.googleapis.com/download.tensorflow.org/data/petfinder-mini.zip'
csv_file = 'datasets/petfinder-mini/petfinder-mini.csv'
tf.keras.utils.get_file('petfinder_mini.zip', dataset_url,
                        extract=True, cache_dir='.')
pf_df = pd.read_csv(csv_file)
pf_df.head()<jupyter_output><empty_output><jupyter_text>For the purpose of the tutorial, modifications are made to the label column.0 will indicate the pet was not adopted, and 1 will indicate that it was.<jupyter_code># In the original dataset "4" indicates the pet was not adopted.
pf_df['target'] = np.where(pf_df['AdoptionSpeed']==4, 0, 1)

# Drop un-used columns.
pf_df = pf_df.drop(columns=['AdoptionSpeed', 'Description'])

# Number of datapoints and columns
len(pf_df), len(pf_df.columns)<jupyter_output><empty_output><jupyter_text>Split the dataset<jupyter_code>train_df, test_df = train_test_split(pf_df, test_size=0.3, shuffle=True)
print("Number of training samples: ",len(train_df))
print("Number of testing sample: ",len(test_df))<jupyter_output><empty_output><jupyter_text>Store the train and test data in mongo collections<jupyter_code>URI = "mongodb://localhost:27017"
DATABASE = "tfiodb"
TRAIN_COLLECTION = "train"
TEST_COLLECTION = "test"
db = client[DATABASE]
if "train" not in db.list_collection_names():
  db.create_collection(TRAIN_COLLECTION)
if "test" not in db.list_collection_names():
  db.create_collection(TEST_COLLECTION)
def store_records(collection, records):
  writer = tfio.experimental.mongodb.MongoDBWriter(
      uri=URI, database=DATABASE, collection=collection
  )
  for record in records:
      writer.write(record)
store_records(collection="train", records=train_df.to_dict("records"))
time.sleep(2)
store_records(collection="test", records=test_df.to_dict("records"))<jupyter_output><empty_output><jupyter_text>Prepare tfio datasetsOnce the data is available in the cluster, the `mongodb.MongoDBIODataset` class is utilized for this purpose. The class inherits from `tf.data.Dataset` and thus exposes all the useful functionalities of `tf.data.Dataset` out of the box.  Training dataset<jupyter_code>train_ds = tfio.experimental.mongodb.MongoDBIODataset(
        uri=URI, database=DATABASE, collection=TRAIN_COLLECTION
    )

train_ds<jupyter_output><empty_output><jupyter_text>Each item in `train_ds` is a string which needs to be decoded into a json. To do so, you can select only a subset of the columns by specifying the `TensorSpec`<jupyter_code># Numeric features.
numerical_cols = ['PhotoAmt', 'Fee'] 

SPECS = {
    "target": tf.TensorSpec(tf.TensorShape([]), tf.int64, name="target"),
}
for col in numerical_cols:
  SPECS[col] = tf.TensorSpec(tf.TensorShape([]), tf.int32, name=col)
pprint(SPECS)
BATCH_SIZE=32
train_ds = train_ds.map(
        lambda x: tfio.experimental.serialization.decode_json(x, specs=SPECS)
    )

# Prepare a tuple of (features, label)
train_ds = train_ds.map(lambda v: (v, v.pop("target")))
train_ds = train_ds.batch(BATCH_SIZE)

train_ds<jupyter_output><empty_output><jupyter_text>Testing dataset<jupyter_code>test_ds = tfio.experimental.mongodb.MongoDBIODataset(
        uri=URI, database=DATABASE, collection=TEST_COLLECTION
    )
test_ds = test_ds.map(
        lambda x: tfio.experimental.serialization.decode_json(x, specs=SPECS)
    )
# Prepare a tuple of (features, label)
test_ds = test_ds.map(lambda v: (v, v.pop("target")))
test_ds = test_ds.batch(BATCH_SIZE)

test_ds<jupyter_output><empty_output><jupyter_text>Define the keras preprocessing layersAs per the [structured data tutorial](https://www.tensorflow.org/tutorials/structured_data/preprocessing_layers), it is recommended to use the [Keras Preprocessing Layers](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing) as they are more intuitive, and can be easily integrated with the models. However, the standard [feature_columns](https://www.tensorflow.org/api_docs/python/tf/feature_column) can also be used.For a better understanding of the `preprocessing_layers` in classifying structured data, please refer to the [structured data tutorial](https://www.tensorflow.org/tutorials/structured_data/preprocessing_layers)<jupyter_code>def get_normalization_layer(name, dataset):
  # Create a Normalization layer for our feature.
  normalizer = preprocessing.Normalization(axis=None)

  # Prepare a Dataset that only yields our feature.
  feature_ds = dataset.map(lambda x, y: x[name])

  # Learn the statistics of the data.
  normalizer.adapt(feature_ds)

  return normalizer

all_inputs = []
encoded_features = []

for header in numerical_cols:
  numeric_col = tf.keras.Input(shape=(1,), name=header)
  normalization_layer = get_normalization_layer(header, train_ds)
  encoded_numeric_col = normalization_layer(numeric_col)
  all_inputs.append(numeric_col)
  encoded_features.append(encoded_numeric_col)<jupyter_output><empty_output><jupyter_text>Build, compile and train the model<jupyter_code># Set the parameters

OPTIMIZER="adam"
LOSS=tf.keras.losses.BinaryCrossentropy(from_logits=True)
METRICS=['accuracy']
EPOCHS=10

# Convert the feature columns into a tf.keras layer
all_features = tf.keras.layers.concatenate(encoded_features)

# design/build the model
x = tf.keras.layers.Dense(32, activation="relu")(all_features)
x = tf.keras.layers.Dropout(0.5)(x)
x = tf.keras.layers.Dense(64, activation="relu")(x)
x = tf.keras.layers.Dropout(0.5)(x)
output = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(all_inputs, output)
# compile the model
model.compile(optimizer=OPTIMIZER, loss=LOSS, metrics=METRICS)
# fit the model
model.fit(train_ds, epochs=EPOCHS)<jupyter_output><empty_output><jupyter_text>Infer on the test data<jupyter_code>res = model.evaluate(test_ds)
print("test loss, test acc:", res)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09963603275705187, 'fraction_numerical': 0.004663330300272975, 'mean_word_length': 6.231085526315789, 'pattern_counts': {'":': 1, '<': 69, '<?xml version=': 0, '>': 70, 'https://': 7, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '18628', 'license': 'Apache-2.0', 'n_tokens_mistral': 2862, 'n_tokens_neox': 2774, 'n_words': 750, 'path': 'docs/tutorials/mongodb.ipynb', 'repo_name': 'ZWael/io', 'size': '8792'}