<jupyter_start><jupyter_text>Train Triplet Model Using Table API  Table of Contents* [Overview](Overview)* [Set the Environment](Set-the-environment)* [Prepare the Data](Prepare-the-data)* [Build a Triplet Model](Build-a-triplet-model)* [Train the Model](Train-the-model)* [Assess the Model](Assess-the-model)  Overview This example shows how to train a Triplet Network by invoking the ImageEmbeddingTable API to create train/test data sets. We then show how to train and assess the Triplet Network.The data used for this example consists of a subset of images obtained from [Google's Open Image V4](https://arxiv.org/abs/1811.00982). To recreate this data set: Our data is stored in a path accessible by the CAS server. The data is structured in a directory as follows:../data   |--german_shepherd   |  |--image1   |  |--image2   |  |--etc..   |--golden_retriever   |  |--image1   |  |--image2   |  |--etc..   |--labrador_retriever   |  |--image1   |  |--image2   |  |--etc..   |--etc..   Note: Image data can only be loaded from the server. Client-side loads of image data are unsupported.  Set the Environment First, set notebook environment by doing the following:1. Import relevant modules2. Use matplotlib inline magic so plots render correctly3. Start a CAS session**Remember:** Change **`host_name`** and **`port_number`** to match your CAS server. You should also have an authinfo (authentication) file set up as well.<jupyter_code>from swat import *
from pprint import pprint

%matplotlib inline
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import ImageGrid
import pandas as pd
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.max_colwidth', 500)

from dlpy.layers import * 
from dlpy.applications import *
from dlpy import Model, Sequential
from dlpy.utils import *
from dlpy.splitting import two_way_split
from dlpy.lr_scheduler import *
from dlpy.images import *
from dlpy.image_embedding import *
from dlpy.embedding_model import EmbeddingModel
s = CAS('cloud.example.com', port_number)
s.sessionprop.setsessopt(caslib='CASUSER', timeout=31535000)<jupyter_output>NOTE: 'CASUSER(userid)' is now the active caslib.<jupyter_text>Prepare the Data Now, we will create an ImageEmbeddingTable for a Triplet Network. Essentially, this is an ImageTable that is composed of a trio of images and some data about each image. The images consist of an anchor image, a positive image (one of the same class), and a negative image (one of a different class).For arguments to the **`ImageEmbeddingTable.load_files`** method, we specify the current CAS session, the server-side path where images reside, the model type as a Triplet Network, and label level. Since the path to any of our images is of the form ../data/category/image_name, we want to choose **`label_level_ =-2`** so that 'category' is used for that image's label.  **Remember:** Change _server_side_data_path_ to the path where your images reside.<jupyter_code>my_table= ImageEmbeddingTable.load_files(s, path='server_side_data_path', 
                                         embedding_model_type='triplet', label_level=-2)
my_table.label_freq[3]<jupyter_output><empty_output><jupyter_text>We can uncomment the following line to show image triplets from the dog breed image table.<jupyter_code>#my_table.show()
my_table.numrows()<jupyter_output><empty_output><jupyter_text>Build a Triplet Model A Triplet Network uses three copies of a neural network to learn discriminative features. We use ResNet18 as a base network for these three copies. Likewise, ResNet18 will serve as the base of our embedding, but in order to customize the embedding output, we add an embedding layer to ResNet18 that is also copied. We define our embedding layer to be a dense 4-neuron layer with an identity activation function. If this is not given, the last layer before the output layer will be used as the embedding layer.The output features from each network are fed into an embedding loss layer to compute the triplet loss function. The triplet loss function is designed to minimize the Euclidean distance between the anchor image and the positive image and increase the distance between the anchor image and the negative image. Margin is used to tune how discriminative the features should be.The resulting Triplet Network structure is stored in memory as the table "test_tr".<jupyter_code>resnet18_model = ResNet18_Caffe(s,
                                width=224,
                                height=224,
                                random_crop='RESIZETHENCROP',
                                random_mutation='random'
                                )
resnet18_model.plot_network()
embedding_layer = Dense(n=4, act='identity')
model_tr = EmbeddingModel.build_embedding_model(resnet18_model, model_table='test_tr',
                                              embedding_model_type='triplet', margin=-3.0,
                                              embedding_layer=embedding_layer)
model_tr.plot_network()<jupyter_output><empty_output><jupyter_text>Note that, not only do we have three copies of ResNet18, but we have the 4-neuron EmbeddingLayer and an EmbeddingLossLayer to aggregate and compare the output.  Train the model First, we import utilities needed in order to train the model on GPU. Then, we set some of the optimization parameters.We train the model using **`fit_embedding_model()`**, which automatically binds the input and output information from the image embedding table to the model.<jupyter_code>from dlpy.model import Gpu
gpu = Gpu(devices=1)
from dlpy.model import Optimizer, MomentumSolver, AdamSolver
solver = AdamSolver(lr_scheduler=StepLR(learning_rate=0.0001, step_size=20), clip_grad_max = 100, clip_grad_min = -100)
optimizer = Optimizer(algorithm=solver, mini_batch_size=8, log_level=2, max_epochs=40, reg_l2=0.0001)
model_tr.fit_embedding_model(data=my_table, optimizer=optimizer, n_threads=4, gpu=gpu, seed=1234,record_seed=23435)<jupyter_output>NOTE: Training from scratch.
WARNING: Using dataSpecs settings. Additional input or target option settings will be ignored.
NOTE: Using cloud.example.com: 1 out of 2 available GPU devices.
NOTE:  Synchronous mode is enabled.
NOTE:  The total number of parameters is 33548364.
NOTE:  The approximate memory cost is 885.00 MB.
NOTE:  Loading weights cost       0.00 (s).
NOTE:  Initializing each layer cost       4.23 (s).
NOTE:  The total number of threads on each worker is 4.
NOTE:  The total mini-batch size per thread on each worker is 8.
NOTE:  The maximum mini-batch size across all workers for the synchronous mode is 32.
NOTE:  Epoch Learning Rate        Loss  Fit Error   Time(s)
NOTE:  0        0.0001          0.2692      2.247     8.54
NOTE:  1        0.0001         0.04495      2.475     3.71
NOTE:  2        0.0001        -0.03341      2.747     3.70
NOTE:  3        0.0001         -0.1182      3.064     3.72
NOTE:  4        0.0001         -0.1294      2.832     3.93
NOTE:  5        0[...]<jupyter_text>Assess the model Having trained the Triplet Network, we need to test it. Like the training data, the test data needs three images and their corresponding image data. Again, we use the ImageEmbeddingTable API to create data in this format. We form 64 image triplets from images in the given path.Note, however, that the Triplet Network is three copies of a network, joined at an embedding loss layer. Because it has three networks, we have to reconsider what scoring the model means. In the case of the Triplet Network, scoring an observation produces computed variables: \_DL_Pred0\_ and \_DL_Pred1\_. \_DL_Pred0\_ denotes the Euclidean distance between the anchor image and the positive image. For a well-trained embedding, this distance should be small. \_DL_Pred1\_ denotes the distance between the anchor image and the negative image. This distance should be large.We consider an observation "correctly classified" if \_DL_Pred0\_ > \_DL_Pred1\_. The means of these distances can provide an idea of the interclass and intraclass distance of the embedding.**Remember:** Change _server_side_data_path_ to the path where your images reside.<jupyter_code>my_table_test= ImageEmbeddingTable.load_files(s, path='server_side_data_path', 
                                              embedding_model_type='triplet', label_level=-2, 
                                              n_samples=64)
from dlpy.model import Gpu
gpu = Gpu(devices=1)
model_tr.predict(data=my_table_test, gpu=gpu, random_crop='RESIZETHENCROP')
s.summary(table=model_tr.valid_res_tbl, inputs=['_DL_Pred0_', '_DL_Pred1_'])<jupyter_output><empty_output><jupyter_text>Here, we calculate the number of correct rows where "correct" means the anchor is closer to the positive image than the negative image. We also calculate the total number of rows and use the two statistics to determine the proportion of correctly embedded observations.<jupyter_code>tot_obs = s.numrows(table=dict(name=model_tr.valid_res_tbl.name))
correctly_ranked_obs = s.numrows(table=dict(name=model_tr.valid_res_tbl.name, where='_DL_Pred0_ < _DL_Pred1_'))
correctly_ranked_obs['numrows']/tot_obs['numrows']<jupyter_output><empty_output><jupyter_text>ConclusionThough there's room for improvement, our model does classify most dogs correctly.<jupyter_code>s.endsession()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07303848607271954, 'fraction_numerical': 0.025090367850308312, 'mean_word_length': 3.930293501048218, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '470149', 'license': 'Apache-2.0', 'n_tokens_mistral': 2820, 'n_tokens_neox': 2606, 'n_words': 1057, 'path': 'examples/image_embedding/EmbeddingModel_train_API_example_for_triplet_model.ipynb', 'repo_name': 'ChipRobie/python-dlpy', 'size': '9406'}