chore: init

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+None-ResNet-None-CTC.pth filter=lfs diff=lfs merge=lfs -text
+frozen_east_text_detection.pb filter=lfs diff=lfs merge=lfs -text
+frozen_east_text_detection.tar.gz filter=lfs diff=lfs merge=lfs -text
+img1.jpg filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text

None-ResNet-None-CTC.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5846635e760137ddb9d6d1485641d802ddaaa987232b1de79f2be78031cf98dc
+size 177272573

README.md

@@ -1,3 +1,387 @@
----
-license: mit
----
+<!--
+# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#  * Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#  * Neither the name of NVIDIA CORPORATION nor the names of its
+#    contributors may be used to endorse or promote products derived
+#    from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+-->
+
+# Deploy models using Triton
+
+| Navigate to | [Part 2: Improving Resource Utilization](../Part_2-improving_resource_utilization/) | [Documentation: Model Repository](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_repository.md) | [Documentation: Model Configuration](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md) |
+| ------------ | --------------- | --------------- | --------------- |
+
+Any deep learning inference serving solution needs to tackle two fundamental challenges:
+
+* Managing multiple models.
+* Versioning, loading, and unloading models.
+
+## Before we begin
+
+The conceptual guide aims to educate developers about the challenges faced whilst building inference infrastructure for deploying deep learning pipelines. `Part 1 - Part 5` of this guide build towards solving a simple problem: deploying a performant and scalable pipeline for transcribing text from images. This pipeline includes 5 steps:
+
+1. Pre-process the raw image
+1. Detect which parts of the image contain text (Text Detection Model)
+1. Crop image to regions with text
+1. Find text probabilities (Text Recognition Model)
+1. Convert probabilities to actual text
+
+In `Part 1`, we start by deploying both models on Triton with the pre/post processing steps done on the client.
+
+## Deploying multiple models
+
+The key challenge around managing multiple models is to build an infrastructure that can cater to the different requirements of different models. For instance, users may need to deploy a PyTorch model and TensorFlow model on the same server, and they have different loads for both the models, need to run them on different hardware devices, and need to independently manage the serving configurations (model queues, versions, caching, acceleration, and more). The Triton Inference Server caters to all of the above and more.
+
+![multiple models](./img/multiple_models.PNG)
+
+The first step in deploying models using the Triton Inference Server is building a repository that houses the models which will be served and the configuration schema. For the purposes of this demonstration, we will be making use of an [EAST](https://arxiv.org/pdf/1704.03155v2.pdf) model to detect text and a text recognition model. This workflow is largely an adaptation of [OpenCV's Text Detection](https://docs.opencv.org/4.x/db/da4/samples_2dnn_2text_detection_8cpp-example.html) samples.
+
+To begin, let's clone the repository and navigate to this folder.
+
+```bash
+cd Conceptual_Guide/Part_1-model_deployment
+```
+
+Next, we'll be downloading the necessary models and making sure they are in a format that triton can deploy.
+
+### Model 1: Text Detection
+
+Download and unzip OpenCV's EAST model.
+
+```bash
+wget https://www.dropbox.com/s/r2ingd0l3zt8hxs/frozen_east_text_detection.tar.gz
+tar -xvf frozen_east_text_detection.tar.gz
+```
+
+Export to ONNX.
+>Note: The following step requires you to have the TensorFlow library installed. We recommend executing the following step within the NGC TensorFlow container environment, which you can launch with `docker run -it --gpus all -v ${PWD}:/workspace nvcr.io/nvidia/tensorflow:<yy.mm>-tf2-py3`
+
+```bash
+pip install -U tf2onnx
+python -m tf2onnx.convert --input frozen_east_text_detection.pb --inputs "input_images:0" --outputs "feature_fusion/Conv_7/Sigmoid:0","feature_fusion/concat_3:0" --output detection.onnx
+```
+
+### Model 2: Text Recognition
+
+Download the Text Recognition model weights.
+
+```bash
+wget https://www.dropbox.com/sh/j3xmli4di1zuv3s/AABzCC1KGbIRe2wRwa3diWKwa/None-ResNet-None-CTC.pth
+```
+
+Export the models as `.onnx` using the file in the model definition file in the `utils` folder. This file is adapted from [Baek et. al. 2019](https://github.com/clovaai/deep-text-recognition-benchmark).
+
+>Note:  The following python script requires you to have the PyTorch library installed. We recommend executing the following step within the NGC PyTorch container environment, which you can launch with `docker run -it --gpus all -v ${PWD}:/workspace nvcr.io/nvidia/pytorch:<yy.mm>-py3`
+
+```python
+import torch
+from utils.model import STRModel
+
+# Create PyTorch Model Object
+model = STRModel(input_channels=1, output_channels=512, num_classes=37)
+
+# Load model weights from external file
+state = torch.load("None-ResNet-None-CTC.pth")
+state = {key.replace("module.", ""): value for key, value in state.items()}
+model.load_state_dict(state)
+
+# Create ONNX file by tracing model
+trace_input = torch.randn(1, 1, 32, 100)
+torch.onnx.export(model, trace_input, "str.onnx", verbose=True)
+```
+
+### Setting up the model repository
+
+A [model repository](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_repository.html) is Triton's way of reading your models and any associated metadata with each model (configurations, version files, etc.). These model repositories can live in a local or network attatched filesystem, or in a cloud object store like AWS S3, Azure Blob Storage or Google Cloud Storage. For more details on model repository location, refer to [the documentation](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_repository.html#model-repository-locations). Servers can use also multiple different model repositories. For simplicity, this explanation only uses a single repository stored in the [local filesystem](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_repository.html#local-file-system), in the following format:
+
+```bash
+# Example repository structure
+<model-repository>/
+  <model-name>/
+    [config.pbtxt]
+    [<output-labels-file> ...]
+    <version>/
+      <model-definition-file>
+    <version>/
+      <model-definition-file>
+    ...
+  <model-name>/
+    [config.pbtxt]
+    [<output-labels-file> ...]
+    <version>/
+      <model-definition-file>
+    <version>/
+      <model-definition-file>
+    ...
+  ...
+```
+
+There are three important components to be discussed from the above structure:
+
+* `model-name`: The identifying name for the model.
+* `config.pbtxt`: For each model, users can define a model configuration. This configuration, at minimum, needs to define: the backend, name, shape, and datatype of model inputs and outputs. For most of the popular backends, this configuration file is autogenerated with defaults. The full specification of the configuration file can be found in the [`model_config` protobuf definition](https://github.com/triton-inference-server/common/blob/main/protobuf/model_config.proto).
+* `version`: versioning makes multiple versions of the same model available for use depending on the policy selected. [More Information about versioning.](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_repository.html#model-versions)
+
+For this example you can set up the model repository structure in the following manner:
+
+```bash
+mkdir -p model_repository/text_detection/1
+mv detection.onnx model_repository/text_detection/1/model.onnx
+
+mkdir -p model_repository/text_recognition/1
+mv str.onnx model_repository/text_recognition/1/model.onnx
+```
+
+These commands should give you a repository that looks this:
+
+```bash
+# Expected folder layout
+model_repository/
+├── text_detection
+│   ├── 1
+│   │   └── model.onnx
+│   └── config.pbtxt
+└── text_recognition
+    ├── 1
+    │   └── model.onnx
+    └── config.pbtxt
+```
+
+Note that, for this example, we've already created the `config.pbtxt` files and placed them in the necessary location. In the next section, we'll discuss the contents of these files.
+
+### Model configuration
+
+With the models and the file structure ready, the next things we need to look at are the `config.pbtxt` model configuration files. Let's first look at the model configuration for the `EAST text detection` model that's been provided for you at `/model_repository/text_detection/config.pbtxt`. This shows that `text_detection` is an ONNX model that has one `input` and two `output` tensors.
+
+``` text proto
+name: "text_detection"
+backend: "onnxruntime"
+max_batch_size : 256
+input [
+  {
+    name: "input_images:0"
+    data_type: TYPE_FP32
+    dims: [ -1, -1, -1, 3 ]
+  }
+]
+output [
+  {
+    name: "feature_fusion/Conv_7/Sigmoid:0"
+    data_type: TYPE_FP32
+    dims: [ -1, -1, -1, 1 ]
+  }
+]
+output [
+  {
+    name: "feature_fusion/concat_3:0"
+    data_type: TYPE_FP32
+    dims: [ -1, -1, -1, 5 ]
+  }
+]
+```
+
+* `name`: "name" is an optional field, the value of which should match the name of the directory of the model.
+* `backend`: This field indicates which backend is being used to run the model. Triton supports a wide variety of backends like TensorFlow, PyTorch, Python, ONNX and more. For a complete list of field selection refer to [these comments](https://github.com/triton-inference-server/backend#backends).
+* `max_batch_size`: As the name implies, this field defines the maximum batch size that the model can support.
+* `input` and `output`: The input and output sections specify the name, shape, datatype, and more, while providing operations like [reshaping](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#reshape) and support for [ragged batches](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/ragged_batching.md#ragged-batching).
+
+In [most cases](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_configuration.html#auto-generated-model-configuration), it's possible to leave out the `input` and `output` sections and let Triton extract that information from the model files directly. Here, we've included them for clarity and because we'll need to know the names of our output tensors in the client application later on.
+
+For details of all supported fields and their values, refer to the [model config protobuf definition file](https://github.com/triton-inference-server/common/blob/main/protobuf/model_config.proto).
+
+### Launching the server
+
+With our repository created and our models configured, we're ready to launch the server. While the Triton Inference Server can be [built from source](https://github.com/triton-inference-server/server/blob/main/docs/customization_guide/build.md#building-triton), the use of [pre-built Docker containers](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver) freely available from NGC is highly recommended for this example.
+
+```bash
+# Replace the yy.mm in the image name with the release year and month
+# of the Triton version needed, eg. 22.08
+
+docker run --gpus=all -it --shm-size=256m --rm -p8000:8000 -p8001:8001 -p8002:8002 -v $(pwd)/model_repository:/models nvcr.io/nvidia/tritonserver:<yy.mm>-py3
+```
+
+Once Triton Inference Server has been built or once inside the container, it can be launched with the command:
+
+```bash
+tritonserver --model-repository=/models
+```
+
+This will spin up the server and model instances will be ready for inference.
+
+```text
+I0712 16:37:18.246487 128 server.cc:626]
++------------------+---------+--------+
+| Model            | Version | Status |
++------------------+---------+--------+
+| text_detection   | 1       | READY  |
+| text_recognition | 1       | READY  |
++------------------+---------+--------+
+
+I0712 16:37:18.267625 128 metrics.cc:650] Collecting metrics for GPU 0: NVIDIA GeForce RTX 3090
+I0712 16:37:18.268041 128 tritonserver.cc:2159]
++----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Option                           | Value                                                                                                                                                                                        |
++----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| server_id                        | triton                                                                                                                                                                                       |
+| server_version                   | 2.23.0                                                                                                                                                                                       |
+| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |
+| model_repository_path[0]         | /models                                                                                                                                                                                      |
+| model_control_mode               | MODE_NONE                                                                                                                                                                                    |
+| strict_model_config              | 1                                                                                                                                                                                            |
+| rate_limit                       | OFF                                                                                                                                                                                          |
+| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |
+| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |
+| response_cache_byte_size         | 0                                                                                                                                                                                            |
+| min_supported_compute_capability | 6.0                                                                                                                                                                                          |
+| strict_readiness                 | 1                                                                                                                                                                                            |
+| exit_timeout                     | 30                                                                                                                                                                                           |
++----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+I0712 16:37:18.269464 128 grpc_server.cc:4587] Started GRPCInferenceService at 0.0.0.0:8001
+I0712 16:37:18.269956 128 http_server.cc:3303] Started HTTPService at 0.0.0.0:8000
+I0712 16:37:18.311686 128 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002
+```
+
+## Building a client application
+
+Now that our Triton server has been launched, we can start sending messages to it. There are three ways to interact with the Triton Inference Server:
+
+* HTTP(S) API
+* gRPC API
+* Native C API
+
+There are also pre-built [client libraries](https://github.com/triton-inference-server/client#client-library-apis) in [C++](https://github.com/triton-inference-server/client/tree/main/src/c%2B%2B), [Python](https://github.com/triton-inference-server/client/tree/main/src/python), and [Java](https://github.com/triton-inference-server/client/tree/main/src/java) that wrap over the HTTP and gRPC APIs. This example contains a Python client script in `client.py` which uses the `tritonclient` python library to communicate with Triton over the HTTP API.
+
+Let's examine the contents of this file:
+
+* First, we import our HTTP client from the `tritonclient` library, as well as a few other libraries we'll use for processing our images:
+
+  ```python
+  import math
+  import numpy as np
+  import cv2
+  import tritonclient.http as httpclient
+  ```
+
+* Next, we'll define a few helper functions for taking care of the pre and post processing steps for our pipeline. The details are omitted here for brevity, but you can check the `client.py` file for more details
+
+  ```python
+  def detection_preprocessing(image: cv2.Mat) -> np.ndarray:
+    ...
+
+  def detection_postprocessing(scores: np.ndarray, geometry: np.ndarray, preprocessed_image: np.ndarray) -> np.ndarray:
+    ...
+
+  def recognition_postprocessing(scores: np.ndarray) -> str:
+    ...
+  ```
+
+* Then, we create a client object, and initialize a connection with the Triton Inference Server.
+
+  ```python
+  client = httpclient.InferenceServerClient(url="localhost:8000")
+  ```
+
+* Now, we'll create the `InferInput` that we'll be sending to Triton from our data.
+
+  ```python
+  raw_image = cv2.imread("./img2.jpg")
+  preprocessed_image = detection_preprocessing(raw_image)
+
+  detection_input = httpclient.InferInput("input_images:0", preprocessed_image.shape, datatype="FP32")
+  detection_input.set_data_from_numpy(preprocessed_image, binary_data=True)
+  ```
+
+* Finally, we're ready to send an inference request to the Triton Inference Server and retrieve the response
+
+  ```python
+  detection_response = client.infer(model_name="text_detection", inputs=[detection_input])
+  ```
+
+* After that, we'll repeat the process with the text recognition model, performing our next processing step, creating the input object, querying the server and finally performing postprocessing and printing the result.
+
+  ```python
+  # Process responses from detection model
+  scores = detection_response.as_numpy('feature_fusion/Conv_7/Sigmoid:0')
+  geometry = detection_response.as_numpy('feature_fusion/concat_3:0')
+  cropped_images = detection_postprocessing(scores, geometry, preprocessed_image)
+
+  # Create input object for recognition model
+  recognition_input = httpclient.InferInput("input.1", cropped_images.shape, datatype="FP32")
+  recognition_input.set_data_from_numpy(cropped_images, binary_data=True)
+
+  # Query the server
+  recognition_response = client.infer(model_name="text_recognition", inputs=[recognition_input])
+
+  # Process response from recognition model
+  text = recognition_postprocessing(recognition_response.as_numpy('308'))
+
+  print(text)
+  ```
+
+Let's try it out!
+
+```bash
+pip install tritonclient[http] opencv-python-headless
+python client.py
+```
+
+You might have noticed that it's a bit redundant to retrieve the results of the first model only to do some processing and send them right back to Triton. In [Part 5](../Part_5-Model_Ensembles/) of this tutorial we explore how you can move more processing steps to the server and execute multiple models in a single network call.
+
+## Model Versioning
+
+The ability to deploy different versions of a model is essential to building an MLOps pipeline. The need arises from use cases like conducting A/B tests, easy model version rollbacks and more. Triton users can add a folder and the new model in the same repository:
+
+```text
+model_repository/
+├── text_detection
+│   ├── 1
+│   │   └── model.onnx
+│   ├── 2
+│   │   └── model.onnx
+│   └── config.pbtxt
+└── text_recognition
+    ├── 1
+    │   └── model.onnx
+    └── config.pbtxt
+```
+
+By default Triton serves the "latest" model, but the policy to serve different versions of the model is customizable. For more information, [refer this guide](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#version-policy).
+
+## Loading & Unloading Models
+
+Triton has model management API that can be used to control the model loading unloading policies. This API is extremely useful in cases where one or more models need to be loaded or unloaded without interrupting inference for other models being served on the same server. Users can select from one of three control modes:
+
+* NONE
+* EXPLICIT
+* POLL
+
+```bash
+tritonserver --model-repository=/models --model-control-mode=poll
+```
+
+The policies can also be set via command line arguments whilst launching the server. For more information, refer [this section](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_management.md#model-management) of the documentation.
+
+# What's next?
+
+In this tutorial, we covered the very basics of setting up and querying a Triton Inference Server. This is Part 1 of a 6 part tutorial series that covers the challenges faced in deploying Deep Learning models to production. [Part 2](../Part_2-improving_resource_utilization/) covers `Concurrent Model Execution and Dynamic Batching`. Depending on your workload and experience you might want to jump to [Part 5](../Part_5-Model_Ensembles/) which covers `Building an Ensemble Pipeline with multiple models, pre and post processing steps, and adding business logic`.

client.py

@@ -0,0 +1,212 @@
+# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#  * Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#  * Neither the name of NVIDIA CORPORATION nor the names of its
+#    contributors may be used to endorse or promote products derived
+#    from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import math
+import numpy as np
+import cv2
+import tritonclient.http as httpclient
+
+SAVE_INTERMEDIATE_IMAGES = False
+
+
+def detection_preprocessing(image: cv2.Mat) -> np.ndarray:
+
+    inpWidth = 640
+    inpHeight = 480
+
+    # pre-process image
+    blob = cv2.dnn.blobFromImage(
+        image, 1.0, (inpWidth, inpHeight), (123.68, 116.78, 103.94), True, False
+    )
+    blob = np.transpose(blob, (0, 2, 3, 1))
+    return blob
+
+
+def detection_postprocessing(scores, geometry, preprocessed_image):
+    def fourPointsTransform(frame, vertices):
+        vertices = np.asarray(vertices)
+        outputSize = (100, 32)
+        targetVertices = np.array(
+            [
+                [0, outputSize[1] - 1],
+                [0, 0],
+                [outputSize[0] - 1, 0],
+                [outputSize[0] - 1, outputSize[1] - 1],
+            ],
+            dtype="float32",
+        )
+
+        rotationMatrix = cv2.getPerspectiveTransform(vertices, targetVertices)
+        result = cv2.warpPerspective(frame, rotationMatrix, outputSize)
+        return result
+
+    def decodeBoundingBoxes(scores, geometry, scoreThresh=0.5):
+        detections = []
+        confidences = []
+
+        ############ CHECK DIMENSIONS AND SHAPES OF geometry AND scores ########
+        assert len(scores.shape) == 4, "Incorrect dimensions of scores"
+        assert len(geometry.shape) == 4, "Incorrect dimensions of geometry"
+        assert scores.shape[0] == 1, "Invalid dimensions of scores"
+        assert geometry.shape[0] == 1, "Invalid dimensions of geometry"
+        assert scores.shape[1] == 1, "Invalid dimensions of scores"
+        assert geometry.shape[1] == 5, "Invalid dimensions of geometry"
+        assert (
+            scores.shape[2] == geometry.shape[2]
+        ), "Invalid dimensions of scores and geometry"
+        assert (
+            scores.shape[3] == geometry.shape[3]
+        ), "Invalid dimensions of scores and geometry"
+        height = scores.shape[2]
+        width = scores.shape[3]
+        for y in range(0, height):
+            # Extract data from scores
+            scoresData = scores[0][0][y]
+            x0_data = geometry[0][0][y]
+            x1_data = geometry[0][1][y]
+            x2_data = geometry[0][2][y]
+            x3_data = geometry[0][3][y]
+            anglesData = geometry[0][4][y]
+            for x in range(0, width):
+                score = scoresData[x]
+
+                # If score is lower than threshold score, move to next x
+                if score < scoreThresh:
+                    continue
+
+                # Calculate offset
+                offsetX = x * 4.0
+                offsetY = y * 4.0
+                angle = anglesData[x]
+
+                # Calculate cos and sin of angle
+                cosA = math.cos(angle)
+                sinA = math.sin(angle)
+                h = x0_data[x] + x2_data[x]
+                w = x1_data[x] + x3_data[x]
+
+                # Calculate offset
+                offset = [
+                    offsetX + cosA * x1_data[x] + sinA * x2_data[x],
+                    offsetY - sinA * x1_data[x] + cosA * x2_data[x],
+                ]
+
+                # Find points for rectangle
+                p1 = (-sinA * h + offset[0], -cosA * h + offset[1])
+                p3 = (-cosA * w + offset[0], sinA * w + offset[1])
+                center = (0.5 * (p1[0] + p3[0]), 0.5 * (p1[1] + p3[1]))
+                detections.append((center, (w, h), -1 * angle * 180.0 / math.pi))
+                confidences.append(float(score))
+
+        # Return detections and confidences
+        return [detections, confidences]
+
+    scores = scores.transpose(0, 3, 1, 2)
+    geometry = geometry.transpose(0, 3, 1, 2)
+    frame = np.squeeze(preprocessed_image, axis=0)
+    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+    [boxes, confidences] = decodeBoundingBoxes(scores, geometry)
+    indices = cv2.dnn.NMSBoxesRotated(boxes, confidences, 0.5, 0.4)
+
+    cropped_list = []
+    cv2.imwrite("frame.png", frame)
+    count = 0
+    for i in indices:
+        # get 4 corners of the rotated rect
+        count += 1
+        vertices = cv2.boxPoints(boxes[i])
+        cropped = fourPointsTransform(frame, vertices)
+        cv2.imwrite(str(count) + ".png", cropped)
+        cropped = np.expand_dims(cv2.cvtColor(cropped, cv2.COLOR_BGR2GRAY), axis=0)
+
+        cropped_list.append(((cropped / 255.0) - 0.5) * 2)
+    cropped_arr = np.stack(cropped_list, axis=0)
+
+    # Only keep the first image, since the models don't currently allow batching.
+    # See part 2 for enabling batch sizes > 0
+    return cropped_arr[None, 0]
+
+
+def recognition_postprocessing(scores: np.ndarray) -> str:
+    text = ""
+    alphabet = "0123456789abcdefghijklmnopqrstuvwxyz"
+
+    scores = np.transpose(scores, (1,0,2))
+
+    for i in range(scores.shape[0]):
+        c = np.argmax(scores[i][0])
+        if c != 0:
+            text += alphabet[c - 1]
+        else:
+            text += "-"
+    # adjacent same letters as well as background text must be removed
+    # to get the final output
+    char_list = []
+    for i, char in enumerate(text):
+        if char != "-" and (not (i > 0 and char == text[i - 1])):
+            char_list.append(char)
+    return "".join(char_list)
+
+
+if __name__ == "__main__":
+
+    # Setting up client
+    client = httpclient.InferenceServerClient(url="localhost:8000")
+
+    # Read image and create input object
+    raw_image = cv2.imread("./img1.jpg")
+    preprocessed_image = detection_preprocessing(raw_image)
+
+    detection_input = httpclient.InferInput(
+        "input_images:0", preprocessed_image.shape, datatype="FP32"
+    )
+    detection_input.set_data_from_numpy(preprocessed_image, binary_data=True)
+
+    # Query the server
+    detection_response = client.infer(
+        model_name="text_detection", inputs=[detection_input]
+    )
+
+    # Process responses from detection model
+    scores = detection_response.as_numpy("feature_fusion/Conv_7/Sigmoid:0")
+    geometry = detection_response.as_numpy("feature_fusion/concat_3:0")
+    cropped_images = detection_postprocessing(scores, geometry, preprocessed_image)
+
+    # Create input object for recognition model
+    recognition_input = httpclient.InferInput(
+        "input.1", cropped_images.shape, datatype="FP32"
+    )
+    recognition_input.set_data_from_numpy(cropped_images, binary_data=True)
+
+    # Query the server
+    recognition_response = client.infer(
+        model_name="text_recognition", inputs=[recognition_input]
+    )
+
+    # Process response from recognition model
+    final_text = recognition_postprocessing(recognition_response.as_numpy("308"))
+
+    print(final_text)

convert_to_str.py

@@ -0,0 +1,14 @@
+import torch
+from utils.model import STRModel
+
+# Create PyTorch Model Object
+model = STRModel(input_channels=1, output_channels=512, num_classes=37)
+
+# Load model weights from external file
+state = torch.load("None-ResNet-None-CTC.pth", map_location=torch.device('cpu'))
+state = {key.replace("module.", ""): value for key, value in state.items()}
+model.load_state_dict(state)
+
+# Create ONNX file by tracing model
+trace_input = torch.randn(1, 1, 32, 100)
+torch.onnx.export(model, trace_input, "str.onnx", verbose=True)

frozen_east_text_detection.pb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b486f3c3eee77b4c8cc91a83892c37026cca7d29b79bf3b93772ccd2db58454
+size 96662756

frozen_east_text_detection.tar.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba5e7c7e9080c3014ce5f253a397a72c7be0d83b51d0a6f9fc6b0ad80db7fe9d
+size 89824490

model_repository/text_detection/1/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92c96959955dcf92ef5a9bba0c5f1979e05f8556ae68f35f0accbe7a68b0e43c
+size 96224417

model_repository/text_detection/config.pbtxt

@@ -0,0 +1,50 @@
+# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#  * Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#  * Neither the name of NVIDIA CORPORATION nor the names of its
+#    contributors may be used to endorse or promote products derived
+#    from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+name: "text_detection"
+backend: "onnxruntime"
+max_batch_size : 0
+input [
+  {
+    name: "input_images:0"
+    data_type: TYPE_FP32
+    dims: [ -1, -1, -1, 3 ]
+  }
+]
+output [
+  {
+    name: "feature_fusion/Conv_7/Sigmoid:0"
+    data_type: TYPE_FP32
+    dims: [ -1, -1, -1, 1 ]
+  }
+]
+output [
+  {
+    name: "feature_fusion/concat_3:0"
+    data_type: TYPE_FP32
+    dims: [ -1, -1, -1, 5 ]
+  }
+]

model_repository/text_recognition/1/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73e18fde8ed993862ecc7636fed1f70ff810bee1345bef25f79e549a17fcb347
+size 177304847

model_repository/text_recognition/config.pbtxt

@@ -0,0 +1,43 @@
+# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#  * Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#  * Neither the name of NVIDIA CORPORATION nor the names of its
+#    contributors may be used to endorse or promote products derived
+#    from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+name: "text_recognition"
+backend: "onnxruntime"
+max_batch_size : 0
+input [
+  {
+    name: "input.1"
+    data_type: TYPE_FP32
+    dims: [ 1, 1, 32, 100 ]
+  }
+]
+output [
+  {
+    name: "308"
+    data_type: TYPE_FP32
+    dims: [ 1, 26, 37 ]
+  }
+]

utils/model.py

@@ -0,0 +1,248 @@
+"""
+Copyright (c) 2019-present NAVER Corp.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import torch.nn as nn
+
+# from modules.feature_extraction import ResNet_FeatureExtractor
+
+
+class ResNet_FeatureExtractor(nn.Module):
+    """FeatureExtractor of FAN (http://openaccess.thecvf.com/content_ICCV_2017/papers/Cheng_Focusing_Attention_Towards_ICCV_2017_paper.pdf)"""
+
+    def __init__(self, input_channel, output_channel=512):
+        super(ResNet_FeatureExtractor, self).__init__()
+        self.ConvNet = ResNet(input_channel, output_channel, BasicBlock, [1, 2, 5, 3])
+
+    def forward(self, input):
+        return self.ConvNet(input)
+
+
+class BasicBlock(nn.Module):
+    expansion = 1
+
+    def __init__(self, inplanes, planes, stride=1, downsample=None):
+        super(BasicBlock, self).__init__()
+        self.conv1 = self._conv3x3(inplanes, planes)
+        self.bn1 = nn.BatchNorm2d(planes)
+        self.conv2 = self._conv3x3(planes, planes)
+        self.bn2 = nn.BatchNorm2d(planes)
+        self.relu = nn.ReLU(inplace=True)
+        self.downsample = downsample
+        self.stride = stride
+
+    def _conv3x3(self, in_planes, out_planes, stride=1):
+        "3x3 convolution with padding"
+        return nn.Conv2d(
+            in_planes, out_planes, kernel_size=3, stride=stride, padding=1, bias=False
+        )
+
+    def forward(self, x):
+        residual = x
+
+        out = self.conv1(x)
+        out = self.bn1(out)
+        out = self.relu(out)
+
+        out = self.conv2(out)
+        out = self.bn2(out)
+
+        if self.downsample is not None:
+            residual = self.downsample(x)
+        out += residual
+        out = self.relu(out)
+
+        return out
+
+
+class ResNet(nn.Module):
+    def __init__(self, input_channel, output_channel, block, layers):
+        super(ResNet, self).__init__()
+
+        self.output_channel_block = [
+            int(output_channel / 4),
+            int(output_channel / 2),
+            output_channel,
+            output_channel,
+        ]
+
+        self.inplanes = int(output_channel / 8)
+        self.conv0_1 = nn.Conv2d(
+            input_channel,
+            int(output_channel / 16),
+            kernel_size=3,
+            stride=1,
+            padding=1,
+            bias=False,
+        )
+        self.bn0_1 = nn.BatchNorm2d(int(output_channel / 16))
+        self.conv0_2 = nn.Conv2d(
+            int(output_channel / 16),
+            self.inplanes,
+            kernel_size=3,
+            stride=1,
+            padding=1,
+            bias=False,
+        )
+        self.bn0_2 = nn.BatchNorm2d(self.inplanes)
+        self.relu = nn.ReLU(inplace=True)
+
+        self.maxpool1 = nn.MaxPool2d(kernel_size=2, stride=2, padding=0)
+        self.layer1 = self._make_layer(block, self.output_channel_block[0], layers[0])
+        self.conv1 = nn.Conv2d(
+            self.output_channel_block[0],
+            self.output_channel_block[0],
+            kernel_size=3,
+            stride=1,
+            padding=1,
+            bias=False,
+        )
+        self.bn1 = nn.BatchNorm2d(self.output_channel_block[0])
+
+        self.maxpool2 = nn.MaxPool2d(kernel_size=2, stride=2, padding=0)
+        self.layer2 = self._make_layer(
+            block, self.output_channel_block[1], layers[1], stride=1
+        )
+        self.conv2 = nn.Conv2d(
+            self.output_channel_block[1],
+            self.output_channel_block[1],
+            kernel_size=3,
+            stride=1,
+            padding=1,
+            bias=False,
+        )
+        self.bn2 = nn.BatchNorm2d(self.output_channel_block[1])
+
+        self.maxpool3 = nn.MaxPool2d(kernel_size=2, stride=(2, 1), padding=(0, 1))
+        self.layer3 = self._make_layer(
+            block, self.output_channel_block[2], layers[2], stride=1
+        )
+        self.conv3 = nn.Conv2d(
+            self.output_channel_block[2],
+            self.output_channel_block[2],
+            kernel_size=3,
+            stride=1,
+            padding=1,
+            bias=False,
+        )
+        self.bn3 = nn.BatchNorm2d(self.output_channel_block[2])
+
+        self.layer4 = self._make_layer(
+            block, self.output_channel_block[3], layers[3], stride=1
+        )
+        self.conv4_1 = nn.Conv2d(
+            self.output_channel_block[3],
+            self.output_channel_block[3],
+            kernel_size=2,
+            stride=(2, 1),
+            padding=(0, 1),
+            bias=False,
+        )
+        self.bn4_1 = nn.BatchNorm2d(self.output_channel_block[3])
+        self.conv4_2 = nn.Conv2d(
+            self.output_channel_block[3],
+            self.output_channel_block[3],
+            kernel_size=2,
+            stride=1,
+            padding=0,
+            bias=False,
+        )
+        self.bn4_2 = nn.BatchNorm2d(self.output_channel_block[3])
+
+    def _make_layer(self, block, planes, blocks, stride=1):
+        downsample = None
+        if stride != 1 or self.inplanes != planes * block.expansion:
+            downsample = nn.Sequential(
+                nn.Conv2d(
+                    self.inplanes,
+                    planes * block.expansion,
+                    kernel_size=1,
+                    stride=stride,
+                    bias=False,
+                ),
+                nn.BatchNorm2d(planes * block.expansion),
+            )
+
+        layers = []
+        layers.append(block(self.inplanes, planes, stride, downsample))
+        self.inplanes = planes * block.expansion
+        for i in range(1, blocks):
+            layers.append(block(self.inplanes, planes))
+
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+        x = self.conv0_1(x)
+        x = self.bn0_1(x)
+        x = self.relu(x)
+        x = self.conv0_2(x)
+        x = self.bn0_2(x)
+        x = self.relu(x)
+
+        x = self.maxpool1(x)
+        x = self.layer1(x)
+        x = self.conv1(x)
+        x = self.bn1(x)
+        x = self.relu(x)
+
+        x = self.maxpool2(x)
+        x = self.layer2(x)
+        x = self.conv2(x)
+        x = self.bn2(x)
+        x = self.relu(x)
+
+        x = self.maxpool3(x)
+        x = self.layer3(x)
+        x = self.conv3(x)
+        x = self.bn3(x)
+        x = self.relu(x)
+
+        x = self.layer4(x)
+        x = self.conv4_1(x)
+        x = self.bn4_1(x)
+        x = self.relu(x)
+        x = self.conv4_2(x)
+        x = self.bn4_2(x)
+        x = self.relu(x)
+
+        return x
+
+
+class STRModel(nn.Module):
+    def __init__(self, input_channels, output_channels, num_classes):
+        super(STRModel, self).__init__()
+        self.FeatureExtraction = ResNet_FeatureExtractor(
+            input_channels, output_channels
+        )
+        self.FeatureExtraction_output = output_channels  # int(imgH/16-1) * 512
+        self.AdaptiveAvgPool = nn.AdaptiveAvgPool2d(
+            (self.FeatureExtraction_output, 1)
+        )  # Transform final (imgH/16-1) -> 1
+        self.SequenceModeling_output = self.FeatureExtraction_output
+        self.Prediction = nn.Linear(self.SequenceModeling_output, num_classes)
+
+    def forward(self, input):
+
+        """Feature extraction stage"""
+        visual_feature = self.FeatureExtraction(input)
+        visual_feature = self.AdaptiveAvgPool(
+            visual_feature.permute(0, 3, 1, 2)
+        )  # [b, c, h, w] -> [b, w, c, h]
+        visual_feature = visual_feature.squeeze(3)
+
+        """ Prediction stage """
+        prediction = self.Prediction(visual_feature.contiguous())
+
+        return prediction