<jupyter_start><jupyter_text>Tutorial 1: Basic Usage of FastEstimator In FastEstimator, there are 3 APIs that you need to know:* Pipeline: takes care of loading data and preprocessing data, it usually happens on CPU.* Network: responsible for trainable & differentiable model. It usually happens on GPU.* Estimator: manages training loop.Any deep learning implementation will follow the `Pipeline` -> `Network` -> `Estimator` process as illustrated below:  Step 1: PipelineFor in-memory data, `Pipeline` accepts a nested dictionary like {"train": {"x": x_train, "y": y_train}, "eval": {"x": x_eval, "y": y_eval}}. If valdiation data is not available, then `eval` key is not needed. We will leave the explanation of `ops` argument to tutorial 3.<jupyter_code>import tensorflow as tf
import numpy as np
import fastestimator as fe
from fastestimator.pipeline.processing import Minmax

(x_train, y_train), (x_eval, y_eval) = tf.keras.datasets.mnist.load_data()
x_train = np.expand_dims(x_train, -1) #adding channel dimension for convolution later
x_eval = np.expand_dims(x_eval, -1) #adding channel dimension for convolution later
data = {"train": {"x": x_train, "y": y_train}, "eval": {"x": x_eval, "y": y_eval}}
pipeline = fe.Pipeline(batch_size=32, data=data, ops=Minmax(inputs="x", outputs="x"))<jupyter_output><empty_output><jupyter_text>Step 2: NetworkFor Network architecture, users can choose one of the following:* Define a custom network architecture using  `tf.keras.Model` `tf.keras.Sequential`.  * Use existing architecture provided by `tf.keras.applications` or `fe.architecture`. In this tutorial, we are going to import a pre-defined LeNet architecture in [fastestimator.architecture.lenet](https://github.com/fastestimator/fastestimator/blob/master/fastestimator/architecture/lenet.py).After defining the architecture, we can associate model definition with its own optimizer and expected loss name (default to be 'loss') to `FEModel`.<jupyter_code>from fastestimator.architecture import LeNet
from fastestimator.network.model import FEModel
from fastestimator.network.model import ModelOp
from fastestimator.network.loss import SparseCategoricalCrossentropy

model = FEModel(model_def=LeNet, model_name="lenet", optimizer="adam", loss_name="loss")
network = fe.Network(ops=[ModelOp(inputs="x", model=model, outputs="y_pred"), 
                          SparseCategoricalCrossentropy(y_pred="y_pred", y_true="y", outputs="loss")])<jupyter_output><empty_output><jupyter_text>Step 3: Estimator`Estimator` takes both `pipeline` and `network` and combines them in training loop. Here's the basic usage of `Estimator`.<jupyter_code>estimator = fe.Estimator(network=network, pipeline=pipeline, epochs=2)<jupyter_output><empty_output><jupyter_text>Start trainingSince the data is already loaded in memory, the training is happening without any disk reading.<jupyter_code>estimator.fit()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1054072553045859, 'fraction_numerical': 0.0037645448323066393, 'mean_word_length': 6.533505154639175, 'pattern_counts': {'":': 12, '<': 17, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '543558', 'license': 'Apache-2.0', 'n_tokens_mistral': 885, 'n_tokens_neox': 862, 'n_words': 285, 'path': 'tutorial/t01_basic_usage.ipynb', 'repo_name': 'AriChow/fastestimator', 'size': '2922'}