<jupyter_start><jupyter_text>E2E ML on GCP: MLOps stage 1 : data management: get started with Vertex datasets            Run in Colab                  View on GitHub                         Open in Vertex AI Workbench           OverviewThis tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 1 : data management: get started with Vertex datasets.  ObjectiveIn this tutorial, you learn how to use `Vertex AI Dataset` for training with `Vertex AI`.This tutorial uses the following Google Cloud ML services:- `Vertex AI Datasets`- `BigQuery Datasets`The steps performed include:- Create a Vertex AI `Dataset` resource for:    - image data    - text data    - video data    - tabular data    - forecasting data- Search `Dataset` resources using a filter.- Read a sample of a `BigQuery` dataset into a dataframe.- Generate statistics and data schema using TensorFlow Data Validation from the samples in the dataframe.- Detect anomalies in new data using TensorFlow Data Validation.- Generate a TFRecord feature specification using TensorFlow Transform from the data schema.- Export a dataset and convert to TFRecords.  RecommendationsWhen doing E2E MLOps on Google Cloud, the following best practices with Vertex AI Datasets:- Use CSV index file format for image data- Use CSV index file format for text data:    - For short text strings, embed the text string in the CSV file.    - For long text strings, place text in referenced text file.- Use JSON index file format for video data- For tabular data:    - For small datasets use CSV index file format.    - For large datasets use BigQuery table.- Use `filter` and `order_by` parameters in the `list()` methods to find the latest versions of datasets.- When custom training with a `Vertex AI Dataset`:    - tabular data :        - Use the CSV index file or BigQuery table reference.        - Create a tf.data.Dataset generator from the CSV index file/BigQuery table.    - image/video data:        - Export the data to a JSONL index file.        - Using the index file, convert the images/videos and labels to TFRecords.        - Create a tf.data.Dataset generator from the TFRercords.    - text data:        - If text strings are embedded:            - Convert to CSV file.            - Create a tf.data.Dataset generator from the CSV index file.        - If text strings are in text files:            - Using the JSON index file, convert the text files and labels to TFRecords.            - Create a tf.data.Dataset from the TFRecords.     CostsThis tutorial uses billable components of Google Cloud:- Vertex AI- Cloud Storage- BigQueryLearn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.  InstallationsInstall the packages required for executing this notebook.<jupyter_code>import os

# The Vertex AI Workbench Notebook product has specific requirements
IS_WORKBENCH_NOTEBOOK = os.getenv("DL_ANACONDA_HOME")
IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(
    "/opt/deeplearning/metadata/env_version"
)

# Vertex AI Notebook requires dependencies to be installed with '--user'
USER_FLAG = ""
if IS_WORKBENCH_NOTEBOOK:
    USER_FLAG = "--user"

! pip3 install -U tensorflow $USER_FLAG -q
! pip3 install -U tensorflow-data-validation $USER_FLAG -q
! pip3 install -U tensorflow-transform $USER_FLAG -q
! pip3 install -U tensorflow-io $USER_FLAG -q
! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG -q
! pip3 install --upgrade google-cloud-bigquery $USER_FLAG -q
! pip3 install -U tensorflow-io==0.18 $USER_FLAG -q
! pip3 install --upgrade future $USER_FLAG -q<jupyter_output><empty_output><jupyter_text>Restart the kernelOnce you've installed the additional packages, you need to restart the notebook kernel so it can find the packages.<jupyter_code>import os

if not os.getenv("IS_TESTING"):
    # Automatically restart kernel after installs
    import IPython

    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)<jupyter_output><empty_output><jupyter_text>Set up your Google Cloud project**The following steps are required, regardless of your notebook environment.**1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). 1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).1. Enter your project ID in the cell below. Then run the cell to make sure theCloud SDK uses the right project for all the commands in this notebook.**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.  Set your project ID**If you don't know your project ID**, you may be able to get your project ID using `gcloud`.<jupyter_code>import os

PROJECT_ID = ""

if not os.getenv("IS_TESTING"):
    # Get your Google Cloud project ID from gcloud
    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT_ID = shell_output[0]
    print("Project ID: ", PROJECT_ID)<jupyter_output><empty_output><jupyter_text>Otherwise, set your project ID here.<jupyter_code>if PROJECT_ID == "" or PROJECT_ID is None:
    PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
! gcloud config set project $PROJECT_ID<jupyter_output><empty_output><jupyter_text>RegionYou can also change the `REGION` variable, which is used for operationsthroughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.- Americas: `us-central1`- Europe: `europe-west4`- Asia Pacific: `asia-east1`You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations).<jupyter_code>REGION = "[your-region]"  # @param {type: "string"}

if REGION == "[your-region]":
    REGION = "us-central1"<jupyter_output><empty_output><jupyter_text>TimestampIf you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial.<jupyter_code>from datetime import datetime

TIMESTAMP = datetime.now().strftime("%Y%m%d%H%M%S")<jupyter_output><empty_output><jupyter_text>Authenticate your Google Cloud account**If you are using Vertex AI Workbench Notebooks**, your environment is already authenticated. Skip this step.**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.**Otherwise**, follow these steps:In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.**Click Create service account**.In the **Service account name** field, enter a name, and click **Create**.In the **Grant this service account access to project** section, click the Role drop-down list. Type "Vertex" into the filter box, and select **Vertex Administrator**. Type "Storage Object Admin" into the filter box, and select **Storage Object Admin**.Click Create. A JSON file that contains your key downloads to your local environment.Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.<jupyter_code># If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

import os
import sys

# If on Vertex AI Workbench, then don't execute this code
IS_COLAB = False
if not os.path.exists("/opt/deeplearning/metadata/env_version") and not os.getenv(
    "DL_ANACONDA_HOME"
):
    if "google.colab" in sys.modules:
        IS_COLAB = True
        from google.colab import auth as google_auth

        google_auth.authenticate_user()

    # If you are running this notebook locally, replace the string below with the
    # path to your service account key and run this cell to authenticate your GCP
    # account.
    elif not os.getenv("IS_TESTING"):
        %env GOOGLE_APPLICATION_CREDENTIALS ''<jupyter_output><empty_output><jupyter_text>Create a Cloud Storage bucket**The following steps are required, regardless of your notebook environment.**When you submit a custom training job using the Vertex SDK, you upload a Python packagecontaining your training code to a Cloud Storage bucket. Vertex AI runsthe code from this package. In this tutorial, Vertex AI also saves thetrained model that results from your job in the same bucket. You can thencreate an `Endpoint` resource based on this output in order to serveonline predictions.Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.<jupyter_code>BUCKET_URI = "gs://[your-bucket-name]"  # @param {type:"string"}
if BUCKET_URI == "" or BUCKET_URI is None or BUCKET_URI == "gs://[your-bucket-name]":
    BUCKET_URI = "gs://" + PROJECT_ID + "aip-" + TIMESTAMP<jupyter_output><empty_output><jupyter_text>**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket.<jupyter_code>! gsutil mb -l $REGION $BUCKET_URI<jupyter_output><empty_output><jupyter_text>Finally, validate access to your Cloud Storage bucket by examining its contents:<jupyter_code>! gsutil ls -al $BUCKET_URI<jupyter_output><empty_output><jupyter_text>Set up variablesNext, set up some variables used throughout the tutorial. Import libraries and define constantsImport the BigQuery package, TensorFlow Data Validation (TFDV) package and TensorFlow Data Validation package into your Python environment. Import TensorFlow Transform (TFT) package and pandas into your Python environment.<jupyter_code>import google.cloud.aiplatform as aip
import pandas as pd
import tensorflow_data_validation as tfdv
import tensorflow_transform as tft
from google.cloud import bigquery<jupyter_output><empty_output><jupyter_text>Initialize Vertex AI SDK for PythonInitialize the Vertex AI SDK for Python for your project and corresponding bucket.<jupyter_code>aip.init(project=PROJECT_ID, location=REGION)<jupyter_output><empty_output><jupyter_text>Create BigQuery clientCreate the BigQuery client.<jupyter_code>bqclient = bigquery.Client()<jupyter_output><empty_output><jupyter_text>Vertex AI DatasetsVertex AI `Datasets` are the means for managing your datasets within Vertex AI services. Vertex AI Datasets are also referred to as `Dataset` resources. There are four types of `Dataset` resources, specific to the data type:- `ImageDataset`: image data- `TabularDataset`: tabular (structured) data- `TextDataset`: text (natural language) data- `VideoDataset`: video data- `TimeSeriesDataset`: forecasting dataA Vertex AI `Dataset` provides the following capabilities:- A unique internal identifier for automatic (programatic) processes.- A user specificed (display name) identifier for interactive processes.- Compatible with AutoML training.- Exporting dataset for custom training.- Dataset search capability.- Creation/update timestamps.- StatisticsLearn more about [All dataset documentation](https://cloud.google.com/vertex-ai/docs/datasets/datasets)<jupyter_code>IMPORT_FILE = (
    "gs://cloud-samples-data/vision/automl_classification/flowers/all_data_v2.csv"
)<jupyter_output><empty_output><jupyter_text>Create the DatasetNext, create the `Dataset` resource using the `create` method for the `ImageDataset` class, which takes the following parameters:- `display_name`: The human readable name for the `Dataset` resource.- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.- `import_schema_uri`: The data labeling schema for the data items:  - `single_label`: Binary and multi-class classification  - `multi_label`: Multi-label multi-class classification  - `bounding_box`: Object detection  - `image_segmentation`: SegmentationLearn more about [ImageDataset](https://cloud.google.com/vertex-ai/docs/datasets/prepare-image).<jupyter_code>dataset = aip.ImageDataset.create(
    display_name="example" + "_" + TIMESTAMP,
    gcs_source=[IMPORT_FILE],
    import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,
)

print(dataset.resource_name)
IMPORT_FILE = "gs://automl-video-demo-data/hmdb_split1_5classes_train_inf.csv"<jupyter_output><empty_output><jupyter_text>Create the DatasetNext, create the `Dataset` resource using the `create` method for the `VideoDataset` class, which takes the following parameters:- `display_name`: The human readable name for the `Dataset` resource.- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.- `import_schema_uri`: The data labeling schema for the data items.  - `classification`: Binary and multi-class classification  - `object_tracking`: Object tracking  - `action_recognition`: Action recognitionLearn more about [VideoDataset](https://cloud.google.com/vertex-ai/docs/datasets/prepare-video).<jupyter_code>dataset = aip.VideoDataset.create(
    display_name="example" + "_" + TIMESTAMP,
    gcs_source=[IMPORT_FILE],
    import_schema_uri=aip.schema.dataset.ioformat.video.classification,
)

print(dataset.resource_name)
IMPORT_FILE = "gs://cloud-ml-data/NL-classification/happiness.csv"<jupyter_output><empty_output><jupyter_text>Create the DatasetNext, create the `Dataset` resource using the `create` method for the `TextDataset` class, which takes the following parameters:- `display_name`: The human readable name for the `Dataset` resource.- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.- `import_schema_uri`: The data labeling schema for the data items.  - `single_label`: Binary and multi-class classification  - `multi_label`: Multi-label multi-class classification  - `sentiment`: Sentiment analysis  - `extraction`: Entity extractionLearn more about [TextDataset](https://cloud.google.com/vertex-ai/docs/datasets/prepare-text).<jupyter_code>dataset = aip.TextDataset.create(
    display_name="example" + "_" + TIMESTAMP,
    gcs_source=[IMPORT_FILE],
    import_schema_uri=aip.schema.dataset.ioformat.text.single_label_classification,
)

print(dataset.resource_name)
IMPORT_FILE = "bq://bigquery-public-data.samples.gsod"
BQ_TABLE = "bigquery-public-data.samples.gsod"<jupyter_output><empty_output><jupyter_text>Create the Dataset CSV input dataNext, create the `Dataset` resource using the `create` method for the `TabularDataset` class, which takes the following parameters:- `display_name`: The human readable name for the `Dataset` resource.- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.- `labels`: User defined metadata. In this example, you store the location of the Cloud Storage bucket containing the user defined data.Learn more about [TabularDataset from CSV files](https://cloud.google.com/vertex-ai/docs/datasets/create-dataset-apiaiplatform_create_dataset_tabular_gcs_sample-python)<jupyter_code>dataset = aip.TabularDataset.create(
    display_name="example" + "_" + TIMESTAMP, bq_source=[IMPORT_FILE]
)

print(dataset.resource_name)
IMPORT_FILE = "gs://cloud-samples-data/ai-platform/covid/bigquery-public-covid-nyt-us-counties-train.csv"<jupyter_output><empty_output><jupyter_text>Create the DatasetNext, create the `Dataset` resource using the `create` method for the `TimeSeriesDataset` class, which takes the following parameters:- `display_name`: The human readable name for the `Dataset` resource.- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.- `bq_source`: Alternatively, import data items from a BigQuery table into the `Dataset` resource.Learn more about [TimeSeriesDataset](https://cloud.google.com/vertex-ai/docs/datasets/prepare-tabular).<jupyter_code>dataset = aip.TimeSeriesDataset.create(
    display_name="example" + "_" + TIMESTAMP,
    gcs_source=[IMPORT_FILE],
)

print(dataset.resource_name)<jupyter_output><empty_output><jupyter_text>Vertex AI Dataset properties and methodsThe following are the `Dataset` methods:- `list()`: List instances of a `Dataset` resource.- `import_data()`: Import additional data into a `Dataset` resource.- `export_data()`: Export dataset index file for custom training.- `delete()`: Delete the dataset.- `update()`: Not implemented yet.Get more information on each method, by executing in Python: help(method_name)The following are the `Dataset` properties:- `name`: The internal unique identifier.- `resource_name`: The fully qualified internal unique identifier.- `display_name`: The human assigned identifier.- `create_time`: The timestamp when the dataset was created.- `update_time`: The timestamp when the dataset was last updated.- `metadata_schema_uri`: The data labeling schema.  List datasetsThe `list()` method returns all datasets, as a list, of the corresponding data type -- e.g., Tabular.<jupyter_code>datasets = aip.TabularDataset.list()
for _dataset in datasets:
    print(_dataset.name)<jupyter_output><empty_output><jupyter_text>List datasets matching a filterThe `list()` method supports returning only datasets that match a `filter`. For example, all datasets where the `display_name` matches the specified display name:```    list(filter='display_name=my_display_name')```When the search matches multiple datasets, one can sort the list based on the dataset properties. For eample, sort by creation time, where the first dataset is the latest:```    list(filter='display_name=my_display_name,order_by=create_time')```<jupyter_code>datasets = aip.TabularDataset.list(
    filter=f'display_name="example_{TIMESTAMP}"', order_by="create_time"
)
latest_dataset = datasets[0]
print(latest_dataset)<jupyter_output><empty_output><jupyter_text>TensorFlow Data ValidationThe TensorFlow Data Validation (TFDV) package is used in conjunction with Vertex AI and BigQuery datasets for:- Generating dataset statistics.- Generating data schema for data validation.- Detecting anomalies in new data using the data schema.- Generating feature specifications for data conversion to TFRecords.Learn more about [TensorFlow Data Validation: Get Started](https://www.tensorflow.org/tfx/data_validation/get_started)  Read the BigQuery dataset into a pandas dataframeNext, you read a sample of the dataset into a pandas dataframe using BigQuery `list_rows()` and `to_dataframe()` method, as follows:- `list_rows()`: Performs a query on the specified table and returns a row iterator to the query results. Optionally specify: - `selected_fields`: Subset of fields (columns) to return. - `max_results`: The maximum number of rows to return. Same as SQL LIMIT command.- `rows.to_dataframe()`: Invokes the row iterator and reads in the data into a pandas dataframe.Learn more about [Loading BigQuery table into a dataframe](https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas)<jupyter_code># Download a table.
table = bigquery.TableReference.from_string("bigquery-public-data.samples.gsod")

rows = bqclient.list_rows(
    table,
    max_results=500,
    selected_fields=[
        bigquery.SchemaField("station_number", "STRING"),
        bigquery.SchemaField("year", "INTEGER"),
        bigquery.SchemaField("month", "INTEGER"),
        bigquery.SchemaField("day", "INTEGER"),
        bigquery.SchemaField("mean_temp", "FLOAT"),
    ],
)

dataframe = rows.to_dataframe()
print(dataframe.head())<jupyter_output><empty_output><jupyter_text>Generate dataset statistics Dataframe input dataGenerate statistics on the dataset with the TensorFlow Data Validation (TFDV) package. Use the `generate_statistics_from_dataframe()` method, with the following parameters:- `dataframe`: The dataset in an in-memory pandas dataframe.- `stats_options`: The selected statistics options:  - `label_feature`: The column which is the label to predict.  - `sample_rate`: The sampling rate. If specified, statistics is computed over the sample.  - `num_top_values`: number of most frequent feature values to keep for string features.Learn about [TensorFlow Data Validation (TFDV)](https://www.tensorflow.org/tfx/data_validation/get_started).<jupyter_code>stats = tfdv.generate_statistics_from_dataframe(
    dataframe=dataframe,
    stats_options=tfdv.StatsOptions(
        label_feature="mean_temp", sample_rate=1, num_top_values=50
    ),
)

print(stats)<jupyter_output><empty_output><jupyter_text>Visualize dataset statisticsA visualization of the dataset statistics can be displayed using the TFDV `visualize_statistics()` method.<jupyter_code>tfdv.visualize_statistics(stats)<jupyter_output><empty_output><jupyter_text>Generate the raw data schemaGenerate the data schema on the dataset with the TensorFlow Data Validation (TFDV) package. Use the `infer_schema()` method, with the following parameters:- `statistics`: The statistics generated by TFDV.<jupyter_code>schema = tfdv.infer_schema(statistics=stats)
print(schema)<jupyter_output><empty_output><jupyter_text>Detect Anomalizes in additional dataWhen additional data is available for a dataset, you can check for anomalies between the new and previous data using the TFDV `validate_statistics` method.The accompanying code example mimics new data by getting the next slice of data (2nd 500 rows) and including an additional field into a dataframe and generating statistics for the new data.The `validate_statistics()` method is called with the following parameters:- `statistics`: The statistics for the new data.- `schema`: The data schema for the previous data.<jupyter_code># Download a table.
table = bigquery.TableReference.from_string("bigquery-public-data.samples.gsod")

rows = bqclient.list_rows(
    table,
    max_results=500,
    start_index=500,
    selected_fields=[
        bigquery.SchemaField("station_number", "STRING"),
        bigquery.SchemaField("year", "INTEGER"),
        bigquery.SchemaField("month", "INTEGER"),
        bigquery.SchemaField("day", "INTEGER"),
        bigquery.SchemaField("mean_temp", "FLOAT"),
        bigquery.SchemaField("num_mean_temp_samples", "INTEGER"),
    ],
)

dataframe = rows.to_dataframe()

new_stats = tfdv.generate_statistics_from_dataframe(
    dataframe=dataframe,
    stats_options=tfdv.StatsOptions(
        label_feature="mean_temp", sample_rate=1, num_top_values=50
    ),
)

anomalies = tfdv.validate_statistics(statistics=new_stats, schema=schema)
print(anomalies.anomaly_info)<jupyter_output><empty_output><jupyter_text>Generate the feature specificationGenerate the feature specification, compatible with TFRecords, on the dataset with the TensorFlow Transform (TFT) package. Use the `schema_as_feature_spec()` method, with the following parameters:- `schema`: The data schema generated by TFDV.<jupyter_code>feature_spec = tft.tf_metadata.schema_utils.schema_as_feature_spec(schema).feature_spec

print(feature_spec)<jupyter_output><empty_output><jupyter_text>Copy the dataset to Cloud StorageNext, you make a copy of the BigQuery dataset, as a CSV file, to Cloud Storage using the BigQuery extract command.Learn more about [BigQuery command line interface](https://cloud.google.com/bigquery/docs/reference/bq-cli-reference).<jupyter_code>comps = BQ_TABLE.split(".")
BQ_PROJECT_DATASET_TABLE = comps[0] + ":" + comps[1] + "." + comps[2]

! bq --location=us extract --destination_format CSV $BQ_PROJECT_DATASET_TABLE $BUCKET_URI/mydata*.csv

IMPORT_FILES = ! gsutil ls $BUCKET_URI/mydata*.csv

print(IMPORT_FILES)

EXAMPLE_FILE = IMPORT_FILES[0]

! gsutil cat $EXAMPLE_FILE | head<jupyter_output><empty_output><jupyter_text>Generate dataset statistics CSV input dataGenerate statistics on the dataset with the TensorFlow Data Validation (TFDV) package. Use the `generate_statistics_from_csv()` method, with the following parameters:- `data_location`: The dataset Cloud Storage file location.- `stats_options`: The selected statistics options:  - `label_feature`: The column which is the label to predict.  - `num_top_values`: number of most frequent feature values to keep for string features.Learn about [TensorFlow Data Validation (TFDV)](https://www.tensorflow.org/tfx/data_validation/get_started).<jupyter_code>stats = tfdv.generate_statistics_from_csv(
    data_location=EXAMPLE_FILE,
    stats_options=tfdv.StatsOptions(label_feature="mean_temp", num_top_values=50),
)

print(stats)<jupyter_output><empty_output><jupyter_text>Export `Text Dataset` to pandas dataframe.The property `gca_resource.metadata['inputConfig']['gcsSource']['uri']` contains the list of the one or more imported CSV files.To create a dataframe from multiple CSV sources, you read each CSV file and concatenate the dataframes together. If you are running this notebook on Colab, run the following cell to install packages fsspec and gcsfs.<jupyter_code># If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

import os
import sys

# If on Google Cloud Notebook, then don't execute this code
if not os.path.exists("/opt/deeplearning/metadata/env_version"):
    if "google.colab" in sys.modules:
        ! pip3 install fsspec
        ! pip3 install gcsfs
all_files = dataset.gca_resource.metadata["inputConfig"]["gcsSource"]["uri"]
df = pd.concat(pd.read_csv(f) for f in all_files)

print(df.head)<jupyter_output><empty_output><jupyter_text>Export dataset indexNext, you export the dataset index to a JSONL file which will then be used by your custom training job to get the data and corresponding labels for training your model. The JSONL index file is generated by calling the export_data() method, with the following parameters: - `output_dir`: The Cloud Storage bucket to write the JSONL dataset index file to.<jupyter_code>datasets = aip.ImageDataset.list(
    filter=f'display_name="example_{TIMESTAMP}"', order_by="create_time"
)
latest_dataset = datasets[0]
print(latest_dataset)
dataset = latest_dataset
EXPORTED_DIR = f"{BUCKET_URI}/exported"
exported_files = dataset.export_data(output_dir=EXPORTED_DIR)

! gsutil ls $EXPORTED_DIR<jupyter_output><empty_output><jupyter_text>Quick peak at your exported dataset index fileTake a quick peak at the contents of the exported dataset index file. When the `export_data()` completed, the method returned a list of the paths to the exported dataset index files.You get the path to the exported dataset index file (`exported_files[0]`) and then display the first ten JSON objects in the file -- i.e., data items.The JSONL format for each data item is:    { "imageGcsUri": path_to_the_image, "classificationAnnotation": { "displayName": label } }<jupyter_code>jsonl_index = exported_files[0]

! gsutil cat $jsonl_index | head<jupyter_output><empty_output><jupyter_text>Reading the index fileYou will need to add code to your custom training Python script to read the exported dataset index, so that you can generate training batches for custom training your model.Below is an example of how you might each the exported dataset index file:1. Use Tensorflow's Cloud Storage file methods to open the file (`tf.io.gfile.GFile()`) and read all the lines (`f.readlines()`), where each line is a data item represented as a JSONL object.2. For each line in the file, convert the line to a JSON object (`json.loads()`).3. Extract the path to the image (`['imageGcsUri']`) and label (`['classificationAnnotation']['displayName']`).<jupyter_code>import json

import tensorflow as tf

with tf.io.gfile.GFile(jsonl_index, "r") as f:
    export_data_items = f.readlines()

for _ in range(10):
    j = json.loads(export_data_items[_])
    print(j["imageGcsUri"], j["classificationAnnotation"]["displayName"])<jupyter_output><empty_output><jupyter_text>Create TFRecordsNext, we needs to create a feeder mechanism to feed data to the model you will train from the dataset index file. There are lots of choices for how to construct a feeder. We will cover a two options here, both using TFRecords:    1. Storing the image data as raw uncompressed image data (1 byte per pixel).    2. Storing the image data as preprocessed data -- machine learning ready --  (4 bytes per pixel).These two methods demonstrate a trade-off between disk storage and compute time. In both cases, we do a prepass over the image data to cache the data into a form that will accelerate the training time for the model. But, by caching we are both using disk space and increasing I/O traffic from the disk to the compute device -- e.g., CPU, GPU, TPU.In the raw uncompressed format, you are minimizing the size on disk and I/O traffic for the cache data, but have the overhead that on each epoch, the preprocessing of the image data has to be repeated. In the preprocessed format, you are minimizing the compute time by preprocessing once and caching the preprocessed data -- i.e., machine learning ready. The amount of data on disk will be four times the size when training as Float32, and you are increasing by the same amount disk space and I/O traffic from the disk to the compute engine.The helper functions `TFExampleImageUncompressed` and `TFExampleImagePreprocessed` both take the parameters:- `path`: The Cloud Storage path to the image file.- `label`: The corresponding label for the image file.- `shape`: The (H,W) input shape to resize the image. If `None`, no resizing occurs.The helper function `TFExampleImagePreprocessed` has an additional parameter:- `dtype`: The floating point representation after the pixel data has been normalized. By default, it is set to 32-bit float (np.float32). If you are using NVIDIA GPUs or TPUs you can alternatively train in 16-bit float, by setting `dtype = np.float16`. There are two benefits to training with 16-bit float, when it does not effect the accuracy or number of epochs:    1. Each matrix multiply operation is 4 times faster than the 32-bit equivalent -- albeit the model weights need to be stored as 16-bit as well.    2. The disk space and I/O bandwidth is reduced by 1/2.Let's look at bit deeper into the functions for creating `TFRecord` training data. First, `TFRecord` is a serialized binary encoding of the training data. As an encoding, one needs to specify a schema for how the fields are encoded, which is then used later to decode during when feeding training data to your model.The schema is defined as an instance of `tf.train.Example` per data item in the training data. Each instance of `tf.train.Example` consists of a sequence fields, each defined as a key/value pair. In our helper function, the key entries are:- `image`: The encoded raw image data.- `label`: The label assigned to the image.- `shape`: The shape of the image when decoded.The value for each key/value pair is an instance of `tf.train.Feature`, where:- `bytes_list`: the data to encode is a byte string.- `int64_list`: the data to encode is an array of one or more integer values.- `float_list`: the data to encode is an array of one or more floating point values.<jupyter_code>import numpy as np


def TFExampleImageUncompressed(path, label, shape=None):
    """The uncompressed version of the image"""

    # read in (and uncompress) the image
    with tf.io.gfile.GFile(path, "rb") as f:
        data = f.read()
    image = tf.io.decode_image(data)

    if shape:
        image = tf.image.resize(image, shape)
    image = image.numpy()
    shape = image.shape

    # make the record
    return tf.train.Example(
        features=tf.train.Features(
            feature={
                "image": tf.train.Feature(
                    bytes_list=tf.train.BytesList(value=[image.tostring()])
                ),
                "label": tf.train.Feature(int64_list=tf.train.Int64List(value=[label])),
                "shape": tf.train.Feature(
                    int64_list=tf.train.Int64List(value=[shape[0], shape[1], shape[2]])
                ),
            }
        )
    )


def TFExampleImagePreprocessed(path, label, shape=None, dtype=np.float32):
    """The normalized version of the image"""

    # read in (uncompress) the image and normalize the pixel data
    image = (cv2.imread(path) / 255.0).astype(dtype)

    if shape:
        image = tf.image.resize(image, shape)
    image = image.numpy()
    shape = image.shape

    # make the record
    return tf.train.Example(
        features=tf.train.Features(
            feature={
                "image": tf.train.Feature(
                    bytes_list=tf.train.BytesList(value=[image.tostring()])
                ),
                "label": tf.train.Feature(int64_list=tf.train.Int64List(value=[label])),
                "shape": tf.train.Feature(
                    int64_list=tf.train.Int64List(value=[shape[0], shape[1], shape[2]])
                ),
            }
        )
    )<jupyter_output><empty_output><jupyter_text>Write training data to TFRecord fileNext, you will create a single TFRecord for all the training data specified in the exported dataset index:- Specify the cache method by setting the variable `CACHE` to either `TFExampleImageUncompressed` or `TFExampleImagePreprocessed`.- Convert class names from the dataset to integer labels, using `cls2label`.- Read in the data item list from the exported dataset index file -- `tf.io.gfile.GFile(jsonl_index, 'r')`.- Set the Cloud Storage location to store the cached TFRecord file -- `GCS_TFRECORD_URI`.- Generate the cached data using `tf.io.TFRecordWriter(gcs_tfrecord_uri)` for each data item in the exported dataset index. - Extract the Cloud Storage path and class name - `json.loads(data_item)` - Convert class name to integer label - `label = cls2label[cls]` - Encode the data item - `example = CACHE(image, label)` - Write the encoded data item to the TFRecord file - `writer.write(example.SerializeToString())`This may take about 20 minutes.<jupyter_code># Select TFRecord method of encoding
CACHE = TFExampleImageUncompressed  # [ TFExampleImageUncompressed, TFExampleImagePreprocessed]

# Map labels to class names
cls2label = {"daisy": 0, "dandelion": 1, "roses": 2, "sunflowers": 3, "tulips": 4}

# Read in each example from exported dataset index
with tf.io.gfile.GFile(jsonl_index, "r") as f:
    data = f.readlines()

# The path to the TFRecord cached file.
GCS_TFRECORD_URI = BUCKET_URI + "/flowers.tfrecord"

# Create the TFRecord cached file
with tf.io.TFRecordWriter(GCS_TFRECORD_URI) as writer:
    n = 0
    for data_item in data:
        j = json.loads(data_item)
        image = j["imageGcsUri"]
        cls = j["classificationAnnotation"]["displayName"]
        label = cls2label[cls]
        example = CACHE(image, label, shape=(128, 128))
        writer.write(example.SerializeToString())
        n += 1
        if n % 10 == 0:
            print(n, image)

listing = ! gsutil ls -la $GCS_TFRECORD_URI
print("TFRecord File", listing)<jupyter_output><empty_output><jupyter_text>Cleaning upTo clean up all Google Cloud resources used in this project, you can [delete the Google Cloudproject](https://cloud.google.com/resource-manager/docs/creating-managing-projectsshutting_down_projects) you used for the tutorial.Otherwise, you can delete the individual resources you created in this tutorial:- Dataset- Bucket<jupyter_code># Delete the dataset using the Vertex dataset object
dataset.delete()

# Delete the bucket
if os.getenv("IS_TESTING"):
    ! gsutil rm -r $BUCKET_URI<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0868620167836016, 'fraction_numerical': 0.004044572843582336, 'mean_word_length': 4.817221510883483, 'pattern_counts': {'":': 17, '<': 145, '<?xml version=': 0, '>': 146, 'https://': 22, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '201492', 'license': 'Apache-2.0', 'n_tokens_mistral': 10319, 'n_tokens_neox': 9978, 'n_words': 3854, 'path': 'notebooks/community/ml_ops/stage1/get_started_vertex_datasets.ipynb', 'repo_name': 'nayaknishant/vertex-ai-samples', 'size': '36345'}