<commit_before># Building Standard TensorFlow Model Server

This tutorial shows you how to use TensorFlow Serving components to build the
standard TensorFlow model server that dynamically discovers and serves new
versions of a trained TensorFlow model. If you just want to use the standard
server to serve your models, see
[TensorFlow Serving basic tutorial](serving_basic.md).

This tutorial uses the simple Softmax Regression model introduced in the
TensorFlow tutorial for handwritten image (MNIST data) classification. If you
don't know what TensorFlow or MNIST is, see the
[MNIST For ML Beginners](http://www.tensorflow.org/tutorials/mnist/beginners/index.html#mnist-for-ml-beginners)
tutorial.

The code for this tutorial consists of two parts:

  * A Python file
  [mnist_export.py](https://github.com/tensorflow/serving/tree/master/tensorflow_serving/example/mnist_export.py)
  that trains and exports multiple versions of the model.

  * A C++ file
  [main.cc](https://github.com/tensorflow/serving/tree/master/tensorflow_serving/model_servers/main.cc)
  which is the standard TensorFlow model server that discovers new exported
  models and runs a [gRPC](http://www.grpc.io) service for serving them.

This tutorial steps through the following tasks:

  1. Train and export a TensorFlow model.
  2. Manage model versioning with TensorFlow Serving `ServerCore`.
  3. Configure batching using `SessionBundleSourceAdapterConfig`.
  4. Serve request with TensorFlow Serving `ServerCore`.
  5. Run and test the service.

Before getting started, please complete the [prerequisites](setup.md#prerequisites).

## Train And Export TensorFlow Model

Clear the export directory if it already exists:

~~~shell
$>rm -rf /tmp/mnist_model
~~~

Train (with 100 iterations) and export the first version of model:

~~~shell
$>bazel build //tensorflow_serving/example:mnist_export
$>bazel-bin/tensorflow_serving/example/mnist_export --training_iteration=100 --export_version=1 /tmp/mnist_model
~~~

Train (with 2000 iterations) and export the second version of model:

~~~shell
$>bazel-bin/tensorflow_serving/example/mnist_export --training_iteration=2000 --export_version=2 /tmp/mnist_model
~~~

As you can see in `mnist_export.py`, the training and exporting is done the same
way it is in the [TensorFlow Serving basic tutorial](serving_basic.md). For
demonstration purposes, you're intentionally dialing down the training
iterations for the first run and exporting it as v1, while training it normally
for the second run and exporting it as v2 to the same parent directory -- as we
expect the latter to achieve better classification accuracy due to more
intensive training. You should see training data for each training run in your
`mnist_model` directory:

~~~shell
$>ls /tmp/mnist_model
00000001  00000002
~~~

## ServerCore

Now imagine v1 and v2 of the model are dynamically generated at runtime, as new
algorithms are being experimented with, or as the model is trained with a new
data set. In a production environment, you may want to build a server that can
support gradual rollout, in which v2 can be discovered, loaded, experimented,
monitored, or reverted while serving v1. Alternatively, you may want to tear
down v1 before bringing up v2. TensorFlow Serving supports both options -- while
one is good for maintaining availability during the transition, the other is
good for minimizing resource usage (e.g. RAM).

TensorFlow Serving `Manager` does exactly that. It handles the full lifecycle of
TensorFlow models including loading, serving and unloading them as well as
version transitions. In this tutorial, you will build your server on top of a
TensorFlow Serving `ServerCore`, which internally wraps an
`AspiredVersionsManager`.

~~~c++
int main(int argc, char** argv) {
  ...

  std::unique_ptr<ServerCore> core;
  TF_CHECK_OK(ServerCore::Create(
      config, std::bind(CreateSourceAdapter, source_adapter_config,
                        std::placeholders::_1, std::placeholders::_2),
      &CreateServableStateMonitor, &LoadDynamicModelConfig, &core));
  RunServer(port, std::move(core));

  return 0;
}
~~~

`ServerCore::Create()` takes four parameters:

  * `ModelServerConfig` that specifies models to be loaded. Models are declared
  either through `model_config_list`, which declares a static list of models, or
  through `dynamic_model_config`, which declares a dynamic list of models that
  may get updated at runtime.
  * `SourceAdapterCreator` that creates the `SourceAdapter`, which adapts
  `StoragePath` (the path where a model version is discovered) to model
  `Loader` (loads the model version from storage path and provides state
  transition interfaces to the `Manager`). In this case, `CreateSourceAdapter`
  creates `SessionBundleSourceAdapter`, which we will explain later.
  * `ServableStateMonitorCreator` that creates `ServableStateMonitor`, which
  keeps track for `Servable` (model version) state transition and provides a
  query interface to the user. In this case, `CreateServableStateMonitor`
  creates the base `ServableStateMonitor`, which keeps track of servable states
  in memory. You can extend it to add state tracking capabilities (e.g. persists
  state change to disk, remote server, etc.)
  * `DynamicModelConfigLoader` that loads models from `dynamic_model_config`.
  The standard TensorFlow model server supports only `model_config_list` for
  now and therefore `LoadDynamicModelConfig` CHECK-fails when called. You can
  extend it to add dynamic model discovery/loading capabilities (e.g. through
  RPC, external service, etc.)

`SessionBundle` is a key component of TensorFlow Serving. It represents a
TensorFlow model loaded from a given path and provides the same `Session::Run`
interface as TensorFlow to run inference.
`SessionBundleSourceAdapter` adapts storage path to `Loader<SessionBundle>`
so that model lifetime can be managed by `Manager`.

With all these, `ServerCore` internally does the following:

  * Instantiates a `FileSystemStoragePathSource` that monitors model export
  paths declared in `model_config_list`.
  * Instantiates a `SourceAdapter` using the `SourceAdapterCreator` with the
  model type declared in `model_config_list` and connects the
  `FileSystemStoragePathSource` to it. This way, whenever a new model version is
  discovered under the export path, the `SessionBundleSourceAdapter` adapts it
  to a `Loader<SessionBundle>`.
  * Instantiates a specific implementation of `Manager` called
  `AspiredVersionsManager` that manages all such `Loader` instances created by
  the `SessionBundleSourceAdapter`.

Whenever a new version is available, this `AspiredVersionsManager` loads the new
version, and under its default behavior unloads the old one. If you want to
start customizing, you are encouraged to understand the components that it
creates internally, and how to configure them.

It is worth mentioning that TensorFlow Serving is designed from scratch to be
very flexible and extensible. You can build various plugins to customize system
behavior, while taking advantage of generic core components like
`AspiredVersionsManager`. For example, you could build a data source plugin that
monitors cloud storage instead of local storage, or you could build a version
policy plugin that does version transition in a different way -- in fact, you
could even build a custom model plugin that serves non-TensorFlow models. These
topics are out of scope for this tutorial. However, you can refer to the
[custom source](custom_source.md) and [custom servable](custom_servable.md)
tutorials for more information.

## Batching

Another typical server feature we want in a production environment is batching.
Modern hardware accelerators (GPUs, etc.) used to do machine learning inference
usually achieve best computation efficiency when inference requests are run in
large batches.

Batching can be turned on by providing proper `SessionBundleSourceAdapterConfig`
when creating the `SessionBundleSourceAdapter`. In this case we set the
`BatchingParameters` with pretty much default values. Batching can be fine-tuned
by setting custom timeout, batch_size, etc. values. For details, please refer
to `BatchingParameters`.

~~~c++
SessionBundleSourceAdapterConfig source_adapter_config;
// Batching config
if (enable_batching) {
  BatchingParameters* batching_parameters =
      source_adapter_config.mutable_config()->mutable_batching_parameters();
  batching_parameters->mutable_thread_pool_name()->set_value(
      "model_server_batch_threads");
}
~~~<commit_msg>Update documentation for ServerCore. Change: 139287084<commit_after># Building Standard TensorFlow Model Server

This tutorial shows you how to use TensorFlow Serving components to build the
standard TensorFlow model server that dynamically discovers and serves new
versions of a trained TensorFlow model. If you just want to use the standard
server to serve your models, see
[TensorFlow Serving basic tutorial](serving_basic.md).

This tutorial uses the simple Softmax Regression model introduced in the
TensorFlow tutorial for handwritten image (MNIST data) classification. If you
don't know what TensorFlow or MNIST is, see the
[MNIST For ML Beginners](http://www.tensorflow.org/tutorials/mnist/beginners/index.html#mnist-for-ml-beginners)
tutorial.

The code for this tutorial consists of two parts:

  * A Python file
  [mnist_export.py](https://github.com/tensorflow/serving/tree/master/tensorflow_serving/example/mnist_export.py)
  that trains and exports multiple versions of the model.

  * A C++ file
  [main.cc](https://github.com/tensorflow/serving/tree/master/tensorflow_serving/model_servers/main.cc)
  which is the standard TensorFlow model server that discovers new exported
  models and runs a [gRPC](http://www.grpc.io) service for serving them.

This tutorial steps through the following tasks:

  1. Train and export a TensorFlow model.
  2. Manage model versioning with TensorFlow Serving `ServerCore`.
  3. Configure batching using `SessionBundleSourceAdapterConfig`.
  4. Serve request with TensorFlow Serving `ServerCore`.
  5. Run and test the service.

Before getting started, please complete the [prerequisites](setup.md#prerequisites).

## Train And Export TensorFlow Model

Clear the export directory if it already exists:

~~~shell
$>rm -rf /tmp/mnist_model
~~~

Train (with 100 iterations) and export the first version of model:

~~~shell
$>bazel build //tensorflow_serving/example:mnist_export
$>bazel-bin/tensorflow_serving/example/mnist_export --training_iteration=100 --export_version=1 /tmp/mnist_model
~~~

Train (with 2000 iterations) and export the second version of model:

~~~shell
$>bazel-bin/tensorflow_serving/example/mnist_export --training_iteration=2000 --export_version=2 /tmp/mnist_model
~~~

As you can see in `mnist_export.py`, the training and exporting is done the same
way it is in the [TensorFlow Serving basic tutorial](serving_basic.md). For
demonstration purposes, you're intentionally dialing down the training
iterations for the first run and exporting it as v1, while training it normally
for the second run and exporting it as v2 to the same parent directory -- as we
expect the latter to achieve better classification accuracy due to more
intensive training. You should see training data for each training run in your
`mnist_model` directory:

~~~shell
$>ls /tmp/mnist_model
00000001  00000002
~~~

## ServerCore

Now imagine v1 and v2 of the model are dynamically generated at runtime, as new
algorithms are being experimented with, or as the model is trained with a new
data set. In a production environment, you may want to build a server that can
support gradual rollout, in which v2 can be discovered, loaded, experimented,
monitored, or reverted while serving v1. Alternatively, you may want to tear
down v1 before bringing up v2. TensorFlow Serving supports both options -- while
one is good for maintaining availability during the transition, the other is
good for minimizing resource usage (e.g. RAM).

TensorFlow Serving `Manager` does exactly that. It handles the full lifecycle of
TensorFlow models including loading, serving and unloading them as well as
version transitions. In this tutorial, you will build your server on top of a
TensorFlow Serving `ServerCore`, which internally wraps an
`AspiredVersionsManager`.

~~~c++
int main(int argc, char** argv) {
  ...

  ServerCore::Options options;
  options.model_server_config = model_server_config;
  options.source_adaptor_creator = [source_adapter_config](
      const string& platform_type,
      std::unique_ptr<ModelServerSourceAdapter>* adapter) {
    return CreateSourceAdapter(source_adapter_config, platform_type, adapter);
  };
  options.servable_state_monitor_creator = &CreateServableStateMonitor;
  options.custom_model_config_loader = &LoadCustomModelConfig;
  std::unique_ptr<ServerCore> core;
  TF_CHECK_OK(ServerCore::Create(options, &core));
  RunServer(port, std::move(core));

  return 0;
}
~~~

`ServerCore::Create()` takes a ServerCore::Options parameter. Here are a few
commonly used options:

  * `ModelServerConfig` that specifies models to be loaded. Models are declared
  either through `model_config_list`, which declares a static list of models, or
  through `dynamic_model_config`, which declares a dynamic list of models that
  may get updated at runtime.
  * `SourceAdapterCreator` that creates the `SourceAdapter`, which adapts
  `StoragePath` (the path where a model version is discovered) to model
  `Loader` (loads the model version from storage path and provides state
  transition interfaces to the `Manager`). In this case, `CreateSourceAdapter`
  creates `SessionBundleSourceAdapter`, which we will explain later.
  * `ServableStateMonitorCreator` that creates `ServableStateMonitor`, which
  keeps track for `Servable` (model version) state transition and provides a
  query interface to the user. In this case, `CreateServableStateMonitor`
  creates the base `ServableStateMonitor`, which keeps track of servable states
  in memory. You can extend it to add state tracking capabilities (e.g. persists
  state change to disk, remote server, etc.)
  * `CustomModelConfigLoader` that instantiates and connects the necessary
  custom sources and source adapters to the manager based on a passed in config
  (any).

`SessionBundle` is a key component of TensorFlow Serving. It represents a
TensorFlow model loaded from a given path and provides the same `Session::Run`
interface as TensorFlow to run inference.
`SessionBundleSourceAdapter` adapts storage path to `Loader<SessionBundle>`
so that model lifetime can be managed by `Manager`.

With all these, `ServerCore` internally does the following:

  * Instantiates a `FileSystemStoragePathSource` that monitors model export
  paths declared in `model_config_list`.
  * Instantiates a `SourceAdapter` using the `SourceAdapterCreator` with the
  model type declared in `model_config_list` and connects the
  `FileSystemStoragePathSource` to it. This way, whenever a new model version is
  discovered under the export path, the `SessionBundleSourceAdapter` adapts it
  to a `Loader<SessionBundle>`.
  * Instantiates a specific implementation of `Manager` called
  `AspiredVersionsManager` that manages all such `Loader` instances created by
  the `SessionBundleSourceAdapter`. `ServerCore` exports `Manager` interface by
  forwarding the calls to `AspiredVersionsManager`.

Whenever a new version is available, this `AspiredVersionsManager` loads the new
version, and under its default behavior unloads the old one. If you want to
start customizing, you are encouraged to understand the components that it
creates internally, and how to configure them.

It is worth mentioning that TensorFlow Serving is designed from scratch to be
very flexible and extensible. You can build various plugins to customize system
behavior, while taking advantage of generic core components like `ServerCore`
and `AspiredVersionsManager`. For example, you could build a data source plugin
that monitors cloud storage instead of local storage, or you could build a
version policy plugin that does version transition in a different way -- in
fact, you could even build a custom model plugin that serves non-TensorFlow
models. These topics are out of scope for this tutorial. However, you can refer
to the [custom source](custom_source.md) and
[custom servable](custom_servable.md) tutorials for more information.

## Batching

Another typical server feature we want in a production environment is batching.
Modern hardware accelerators (GPUs, etc.) used to do machine learning inference
usually achieve best computation efficiency when inference requests are run in
large batches.

Batching can be turned on by providing proper `SessionBundleSourceAdapterConfig`
when creating the `SessionBundleSourceAdapter`. In this case we set the
`BatchingParameters` with pretty much default values. Batching can be fine-tuned
by setting custom timeout, batch_size, etc. values. For details, please refer
to `BatchingParameters`.

~~~c++
SessionBundleSourceAdapterConfig source_adapter_config;
// Batching config
if (enable_batching) {
  BatchingParameters* batching_parameters =
      source_adapter_config.mutable_config()->mutable_batching_parameters();
  batching_parameters->mutable_thread_pool_name()->set_value(
      "model_server_batch_threads");
}
~~~
{'fraction_non_alphanumeric': 0.06284859985049739, 'fraction_numerical': 0.005922603645563797, 'mean_word_length': 5.1959387246170285, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 26, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3391079', 'n_tokens_mistral': 4654, 'n_tokens_neox': 4418, 'n_words': 2079}