<jupyter_start><jupyter_text>Tutorial 2: Dealing with large datasets with FastEstimator_____In Tutorial 1, we introduced our 3 main APIs and general workflow of a deep learning task using FastEstimator:  `Pipeline` -> `Network` -> `Estimator`. Then we used in-memory data for training.  *__But what if the dataset size was too big to fit in memory? Imagine data is the size of ImageNet?__*The short answer is: __the user will use one more API to use in disk data for training: `RecordWriter`__, such that the overall workflow becomes:   `RecordWriter` -> `Pipeline` -> `Network` -> `Estimator`. In this tutorial, we are going to show you how to do in-disk data training in FastEstimator.  Before we start:Two things are required regarding in-disk data : * Data files, obviously :)* A csv file that describes the data (prepare two csv files if you have a separate validation set)In the csv file, the rows of csv represent different examples and columns represent different features within example. For example, for a classification task, a csv may look like this:| image  | label  ||---|---||/data/image1.png   | 0  ||/data/image2.png   |  1 ||/data/image3.png | 0  ||... | .  |The csv of a multi-mask segmentation task may look like this:| img  | msk1  | msk2  ||---|---|---||/data/image1.png   | /maska/mask1.png  |/maskb/mask1.png||/data/image2.png   |  /maska/mask2.png |/maskb/mask2.png||/data/image3.png | /maska/mask3.png  |/maskb/mask3.png||... | ...  |...|Please keep in mind that, there is no restriction on the data folder structures, number of features or name of features.  Now, let's generate some in-disk data for this tutorial!  Step 0: Get the paths to the csv filesFor this example, we will use the function `load_data` from MNIST dataset to create images and csv files in disk, and get the paths to the above-mentioned csv files for training, evaluation.<jupyter_code>from fastestimator.dataset.mnist import load_data

train_csv, eval_csv, folder_path = load_data()
print("training csv path is {}".format(train_csv))
print("evaluation csv path is {}".format(eval_csv))
print("mnist image path is {}".format(folder_path))<jupyter_output><empty_output><jupyter_text>Let's take a look at the csv file and image file.<jupyter_code>import pandas as pd
df_train = pd.read_csv(train_csv)
df_train.head()
import matplotlib.pyplot as plt
import os
img = plt.imread(os.path.join(folder_path, df_train["x"][1]))
plt.imshow(img)
print("ground truth of image is {}".format(df_train["y"][1]))<jupyter_output>ground truth of image is 0<jupyter_text>Step 1: RecordWriterIn FastEstimator, we convert user's in-disk data to TFRecord for the best training speed. The RecordWriter API takes care of it. You will have to specify : - `save_dir` : the path to write the record.          - `train_data` : can either be a csv path or a dictionary like the one used in tutorial 1.    - `validation_data` (optional) : it can take all input formats of `train_data`. In addition, `validation_data` can also take a floating point number between 0 to 1 which indicates the validation split ratio, then validation data will be randomly sampled from training data.    - `ops`: before converting data to TFRecord, users can apply a series of propcoessing tasks to the data in `ops` argument, we will talk about them in detail in tutorial 3.<jupyter_code>from fastestimator.util import RecordWriter
from fastestimator.op.numpyop import ImageReader
import fastestimator as fe
import tempfile

# We simply create a RecordWriter will all required arguments. 
writer = RecordWriter(save_dir=os.path.join(folder_path, "tfrecords"),
                         train_data=train_csv,
                         validation_data=eval_csv,
                         ops=ImageReader(inputs="x", parent_path=folder_path, outputs="x", grey_scale=True))<jupyter_output><empty_output><jupyter_text>Step 2: Pipeline -> Network -> Estimator (see tutorial 1 for details)<jupyter_code>from fastestimator.op.tensorop import Minmax
from fastestimator.architecture import LeNet
from fastestimator.op.tensorop.model import ModelOp
from fastestimator.op.tensorop.loss import SparseCategoricalCrossentropy

# Pipeline creation
pipeline = fe.Pipeline(batch_size=32, data=writer, ops=Minmax(inputs="x", outputs="x"))

# Model and network definition
model = fe.build(model_def=LeNet, model_name="lenet", optimizer="adam", loss_name="loss")
network = fe.Network(ops=[ModelOp(inputs="x", model=model, outputs="y_pred"), 
                          SparseCategoricalCrossentropy(y_pred="y_pred", y_true="y", outputs="loss")])

# Estimator definition
estimator = fe.Estimator(network=network, pipeline=pipeline, epochs=2)<jupyter_output><empty_output><jupyter_text>Step 3: Start training<jupyter_code># Launch the training!
estimator.fit()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09869104508622481, 'fraction_numerical': 0.007271971743195512, 'mean_word_length': 4.6304093567251465, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '198313', 'license': 'Apache-2.0', 'n_tokens_mistral': 1456, 'n_tokens_neox': 1394, 'n_words': 530, 'path': 'docs/tutorial/t02_using_data_in_disk.ipynb', 'repo_name': 'chm123/fastestimator-misc', 'size': '4813'}