<jupyter_start><jupyter_text>Creating New MMARs for your New Project with Clara Train SDKClara Train SDK consists of different modules as shown below ![side_bar](screenShots/TrainBlock.png)By the end of this notebook you will:1. Understand components of [Medical Model ARchive (MMAR)](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/mmar.html)2. Know how to configure train config json to train a CNN3. Train a CNN with single and muiltple GPUS4. Fine tune a model5. Export a model 6. Perform inference on testing dataset   Prerequisites- Nvidia GPU with 8Gb of memory     ResourcesYou could watch the free GTC 2020 talks covering Clara Train SDK - [S22563](https://developer.nvidia.com/gtc/2020/video/S22563)Clara train Getting started: cover basics, BYOC, AIAA, AutoML     Lets get startedBefore we get started lets check that we have an NVIDIA GPU available in the docker by running the cell below<jupyter_code># following command should show all gpus available 
!nvidia-smi<jupyter_output><empty_output><jupyter_text>Next cell defines functions that we will use throughout the notebook<jupyter_code>MMAR_ROOT="/claraDevDay/GettingStarted/"
print ("setting MMAR_ROOT=",MMAR_ROOT)
%ls $MMAR_ROOT

!chmod 777 $MMAR_ROOT/commands/*
def printFile(filePath,lnSt,lnOffset):
    print ("showing ",str(lnOffset)," lines from file ",filePath, "starting at line",str(lnSt))
    lnOffset=lnSt+lnOffset
    !< $filePath head -n "$lnOffset" | tail -n +"$lnSt"<jupyter_output><empty_output><jupyter_text>Medical Model ARchive (MMAR)Clara Train SDK uses the [Medical Model ARchive (MMAR)](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/mmar.html). The MMAR defines a standard structure for organizing all artifacts produced during the model development life cycle. Clara Train SDK simple basic idea is to train using config file as shown below![side_bar](screenShots/MMAR.png) You can download sample models for different problems from [NGC](https://ngc.nvidia.com/catalog/models?orderBy=modifiedDESC&pageNumber=0&query=clara&quickFilter=&filters=)  All MMAR follow the structure provided in this Notebook. if you navigate to the parent folder structure it should contain the following subdirectories```./GettingStarted ├── commands├── config├── docs├── eval├── models└── resources```* `commands` contains a number of ready-to-run scripts for:    - training    - training with multiple GPUS    - validation    - inference (testing)    - exporting models in TensorRT Inference Server format* `config` contains configuration files (in JSON format) for eac training, validation, and deployment for [AI-assisted annotation](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/aiaa/index.html) (_Note:_ these configuration files are used in the scripts under the `commands` folder)* `docs` contains local documentation for the model, but for a more complete view it is recommended you visit the NGC model page* `eval` is used as the output directory for model evaluation (by default)* `models` is where the tensorflow checkpoint-formatted model is stored (`.index`, `.meta`, `.data-xxxxx-of-xxxxx`), and the corresponding graph definition files (`fzn.pb` for frozen models, and `trt.pb` for TRT models)* `resources` currently contains the logger configuration in `log.config` fileSome of the most important files you will need to understand to configure and use Clara Train SDK are1. `environment.json` which has important common parameters to set the path for     * `DATA_ROOT` is the root folder where the data with which we would like to train, validate, or test resides in    * `DATASET_JSON` expects the path to a JSON-formatted file     * `MMAR_CKPT_DIR` the path to the where the tensorflow checkpoint files reside    * `MMAR_EVAL_OUTPUT_PATH` the path to output evaluation metrics for the neural network during training, validation, and inference    * `PROCESSING_TASK` the type of processing task the neural net is intended to perform (currently limited to `annotation`, `segmentation`, `classification`)    * `PRETRAIN_WEIGHTS_FILE` (_optional_) 	determines the location of the pre-trained weights file; if the file does not exist and is needed,     the training program will download it from predefined URL from the web<jupyter_code>printFile(MMAR_ROOT+"/config/environment.json",0,30)<jupyter_output><empty_output><jupyter_text>2. `train.sh` and `train_finetune.sh` run the command to train the network based on the `config_train.json` configuration; this shell script can be also used to override parameters in `config_train.json` using the `--set` argument (see `train_finetune.sh`)_Note_: The main difference between the two is that `train_finetune.sh` specifies a `ckpt` file, while `train.sh` does not since it is training from scratch.Let's take a look at `train.sh` by executing the following cell.<jupyter_code>printFile(MMAR_ROOT+"/commands/train_W_Config.sh",30,30)
printFile(MMAR_ROOT+"/commands/train_finetune.sh",0,30)<jupyter_output><empty_output><jupyter_text>Config.json Main Concepts  `config_train.json` contains all the parameters necessary to define the neural net, how is it trained (training hyper-parameters, loss, etc.), pre- and post-transformation functions necessary to modify and/or augment the data before input to the neural net, etc. The complete documentation on the training configuration is laid out [here](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/appendix/configuration.htmltraining-configuration).Configuration file defines all training related configurations. This is were most the researcher would spent most of his time.![s](screenShots/MMARParts.png) Lets take some time to examine each part of it.   1. Global configurations<jupyter_code>confFile=MMAR_ROOT+"/config/trn_base.json"
printFile(confFile,0,10)<jupyter_output>"model": {
      "name": "Unet",
      "args": {
        "num_classes": 6,
        "nf_enc":"32,64,64,64",
        "nf_dec":"64,64,64,64,64,32,32"
      }
    },
    "pre_transforms": [
      {
        "name": "LoadNifti",<jupyter_text>2. Training config which includes:    1. Loss functions:    [Dice](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.losses.html?highlight=dicemodule-ai4med.components.losses.dice)    , [CrossEntropy](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.losses.html?highlight=crossentropyai4med.components.losses.cross_entropy.CrossEntropy)    , [Focal](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.losses.html?highlight=focalmodule-ai4med.components.losses.focal)    , [FocalDice](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.losses.html?highlight=focaldiceai4med.components.losses.focal_dice.FocalDice)     , [CrossEntropyDice](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.losses.html?highlight=crossentropydiceai4med.components.losses.cross_entropy_dice.CrossEntropyDice)     , [BinaryClassificationLoss](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.losses.html?highlight=binaryclassificationlossai4med.components.losses.classification_loss.BinaryClassificationLoss)    , [MulticlassClassificationLoss](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.losses.html?highlight=multiclassclassificationlossai4med.components.losses.classification_loss.MulticlassClassificationLoss)    , [WeightedMulticlassClassificationLoss](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.losses.html?highlight=weightedmulticlassclassificationlossai4med.components.losses.classification_loss.WeightedMulticlassClassificationLoss)    2. Optimizer    [Momentum]()    , [Adam]()    , [NovaGrad]()    3. Network architecture    [SegAhnet]()    , [SegResnet]()    , [Unet]()    , [UnetParallel]()    , [DenseNet121]()    , [Alexnet]()    4. Learing rate Policy     [ReducePoly]()    , [DecayOnStep]()    , [ReduceCosine]()    , [ReduceOnPlateau]()    5. Image pipeline        1. Classification         , [ClassificationImagePipeline]()        , [ClassificationImagePipelineWithCache]()        , [ClassificationKerasImagePipeline]()        , [ClassificationKerasImagePipelineWithCache]()        2. Segmenatation         , [SegmentationImagePipeline]()        , [SegmentationImagePipelineWithCache]()        , [SegmentationKerasImagePipeline]()        , [SegmentationKerasImagePipelineWithCache]()        4. Pretransforms        1. Loading transformations:            [LoadNifti](https://docs.nvidia.com/clara/tlt-m[i/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=loadniftiai4med.components.transforms.load_nifti.LoadNifti)            , [LoadPng](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=loadpngai4med.components.transforms.load_png.LoadPng)            , [ConvertToChannelsFirst](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=converttochannelsfirstai4med.components.transforms.convert_to_channels_first.ConvertToChannelsFirst)            , [LoadImageMasksFromNumpy](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=loadimagemasksfromnumpyai4med.components.transforms.load_image_masks_from_numpy.LoadImageMasksFromNumpy)            , [LoadJpg](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=loadjpgai4med.components.transforms.load_jpg.LoadJpg)        2. Resample Transformation            [RepeatChannel](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=repeatchannelai4med.components.transforms.repeat_channel.RepeatChannel)            , [ScaleByFactor](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=scalebyfactorai4med.components.transforms.scale_by_factor.ScaleByFactor)            , [ScaleByResolution](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=scalebyresolutionai4med.components.transforms.scale_by_resolution.ScaleByResolution)            , [ScaleBySpacing](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=scalebyspacingai4med.components.transforms.scale_by_spacing.ScaleBySpacing)            , [ScaleToShape](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=scaletoshapeai4med.components.transforms.scale_to_shape.ScaleToShape)            , [RestoreOriginalShape](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=restoreoriginalshapeai4med.components.transforms.restore_original_shape.RestoreOriginalShape)            , [LoadImageMasksFromNumpy](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=loadimagemasksfromnumpyai4med.components.transforms.load_image_masks_from_numpy.LoadImageMasksFromNumpy)        3. Cropping transformations            [CropForegroundObject](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=cropforegroundobjectai4med.components.transforms.crop_foreground_object.CropForegroundObject)            , [FastPosNegRatioCropROI](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=fastposnegratiocroproiai4med.components.transforms.fast_pos_neg_ratio_crop_roi.FastPosNegRatioCropROI)            , [CropByPosNegRatio](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=cropbyposnegratioai4med.components.transforms.crop_by_pos_neg_ratio.CropByPosNegRatio)            , [SymmetricPadderDiv](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=symmetricpadderdivai4med.components.transforms.symmetric_padder_div.SymmetricPadderDiv)            , [FastCropByPosNegRatio](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=fastcropbyposnegratioai4med.components.transforms.fast_crop_by_pos_neg_ratio.FastCropByPosNegRatio)            , [CropByPosNegRatioLabelOnly](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=cropbyposnegratiolabelonlyai4med.components.transforms.crop_by_pos_neg_ratio_label_only.CropByPosNegRatioLabelOnly)            , [CropForegroundObject](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=cropforegroundobjectai4med.components.transforms.crop_foreground_object.CropForegroundObject)            , [CropSubVolumeCenter](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=cropsubvolumecenterai4med.components.transforms.crop_sub_volume_center.CropSubVolumeCenter)            , [CropRandomSizeWithDisplacement](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=croprandomsizewithdisplacementai4med.components.transforms.crop_random_size_w_displacement.CropRandomSizeWithDisplacement)            , [CropFixedSizeRandomCenter](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=cropfixedsizerandomcenterai4med.components.transforms.crop_fixed_size_random_center.CropFixedSizeRandomCenter)        4. Deformable transformations            [FastPosNegRatioCropROI](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=fastposnegratiocroproiai4med.components.transforms.fast_pos_neg_ratio_crop_roi.FastPosNegRatioCropROI)        5. Intensity Transforms            [ScaleIntensityRange](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=scaleintensityrangeai4med.components.transforms.scale_intensity_range.ScaleIntensityRange)            , [ScaleIntensityOscillation](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=scaleintensityoscillationai4med.components.transforms.scale_intensity_oscillation.ScaleIntensityOscillation)            , [AddGaussianNoise](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=addgaussiannoiseai4med.components.transforms.add_gaussian_noise.AddGaussianNoise)            , [NormalizeNonzeroIntensities](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=normalizenonzerointensitiesai4med.components.transforms.normalize_nonzero_intensities.NormalizeNonzeroIntensities)            , [CenterData](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=centerdataai4med.components.transforms.center_data.CenterData)            , [AdjustContrast](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=adjustcontrastai4med.components.transforms.adjust_contrast.AdjustContrast)            , [RandomGaussianSmooth]()            , [RandomMRBiasField]()        6. Augmentation Transforms            [RandomZoom](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=randomzoomai4med.components.transforms.random_zoom.RandomZoom)            , [RandomAxisFlip](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=randomaxisflipai4med.components.transforms.random_axis_flip.RandomAxisFlip)            , [RandomSpatialFlip](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=randomspatialflipai4med.components.transforms.random_spatial_flip.RandomSpatialFlip)            , [RandomRotate2D](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=randomrotate2dai4med.components.transforms.random_rotate_2d.RandomRotate2D)            , [RandomRotate3D](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=randomrotate3dai4med.components.transforms.random_rotate_3d.RandomRotate3D)        7. Special transforms             [AddExtremePointsChannel](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=addextremepointschannelai4med.components.transforms.add_extreme_points_channel.AddExtremePointsChannel)            , [SplitAcrossChannels](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=splitacrosschannelsai4med.components.transforms.split_across_channels.SplitAcrossChannels)            , [SplitBasedOnLabel](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=splitbasedonlabelai4med.components.transforms.split_based_on_label.SplitBasedOnLabel)            , [ThresholdValues](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=thresholdvaluesai4med.components.transforms.apply_threshold.ThresholdValues)            , [SplitBasedOnBratsClasses](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=splitbasedonbratsclassesai4med.components.transforms.split_based_on_brats_classes.SplitBasedOnBratsClasses)            , [ConvertToMultiChannelBasedOnLabel](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=converttomultichannelbasedonlabelai4med.components.transforms.convert_to_multi_channel_based_on_label.ConvertToMultiChannelBasedOnLabel)            , [KeepLargestCC](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=keeplargestccai4med.components.transforms.keep_largest_connected_component.KeepLargestCC)            , [CopyProperties](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=copypropertiesai4med.components.transforms.copy_properties.CopyProperties)            , [ConvertToMultiChannelBasedOnBratsClasses](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=converttomultichannelbasedonbratsclassesai4med.components.transforms.convert_to_multi_channel_based_on_brats_classes.ConvertToMultiChannelBasedOnBratsClasses)            , [ArgmaxAcrossChannels](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/apidocs/ai4med/ai4med.components.transforms.html?highlight=argmaxacrosschannelsai4med.components.transforms.argmax_across_channels.ArgmaxAcrossChannels)<jupyter_code>confFile=MMAR_ROOT+"/config/trn_base.json"
printFile(confFile,9,8)
printFile(confFile,16,8)
printFile(confFile,25,20)
printFile(confFile,108,15)<jupyter_output><empty_output><jupyter_text>3. Validation config which includes:    1. Metric     2. pre-transforms. Since these transforms are usually a subset from teh pre-transforms in the training section,     we can use the alias to point to these transforms by name as ` "ref": "LoadNifti"`.     In case we use 2 transforms with the same name as `ScaleByResolution`     we can give each an alias to refer to as `"name": "ScaleByResolutionScaleImg"`     then refer to it in the validation section as `ScaleImg`     3. Image pipeline    4. Inference<jupyter_code>confFile=MMAR_ROOT+"/config/trn_base.json"
printFile(confFile,120,13)
printFile(confFile,135,16)
printFile(confFile,152,12)
printFile(confFile,164,10)<jupyter_output><empty_output><jupyter_text>Start TensorBoard Before we start training or while the network is training, you can monitor its accuracy using tensorboard in side jupyter lab as shown below ![tb](screenShots/TensorBoard.png)   Lets start trainingNow that we have our training configuration, to start training simply run `train.sh` as below. Please keep in mind that we have setup a dummy data with one file to train a dummy network fast (we only train for 2 epochs). Please see exercises on how to easily switch data and train a real segmentation network.**_Note:_** We have renamed `train.sh` to `train_W_Config` as we modified it to accept parameters with the config to use<jupyter_code>! $MMAR_ROOT/commands/train_W_Config.sh trn_base.json<jupyter_output><empty_output><jupyter_text>Now lets see the `models` directory, which would includes out models and the tensorboard files<jupyter_code>! ls -la $MMAR_ROOT/models/trn_base<jupyter_output><empty_output><jupyter_text>Export ModelTo export the model we simply run `export.sh` which will: - Removes back propagation information from checkpoint files- Generates two frozen graphs in the models folderThis optimized model will be used by TRTIS server in Clara Deploy.<jupyter_code>! $MMAR_ROOT/commands/export.sh<jupyter_output><empty_output><jupyter_text>lets check out what was created in the folder. after running cell below you should see:1. Frozen File Generated: /claraDevDay/GettingStarted/commands/../models/trn_base/model.fzn.pb2. TRT File Generated: /claraDevDay/GettingStarted/commands/../models/trn_base/model.trt.pb<jupyter_code>!ls -la $MMAR_ROOT/models/trn_base/*.pb<jupyter_output><empty_output><jupyter_text>Evaluate and Prediction Now that we have trained our model we would like to run evaluation to get some statistics and also do inference to see the resulted output  1. Evaluate To run evaluation on your validation dataset you should run `validate.sh`. This will run evaluation on the validation dataset and place it in the `MMAR_EVAL_OUTPUT_PATH` as configured in the [environment.json](config/environment.json) file (default is eval folder). This evaluation would give min, max, mean of the metric as specified in the config_validation file<jupyter_code>! $MMAR_ROOT/commands/validate.sh<jupyter_output><empty_output><jupyter_text>Now lets see results in the folder by running cells below. You should see statistics and dice per file in the validation dataset<jupyter_code>! ls -la $MMAR_ROOT/eval/
# statistic summary
!cat $MMAR_ROOT/eval/mean_dice_class1_summary_results.txt
!cat $MMAR_ROOT/eval/mean_dice_class1_raw_results.txt<jupyter_output><empty_output><jupyter_text>2. PredictTo run inference on validation dataset or test dataset you should run `infer.sh`. This will run prediction on the validation dataset and place it in the `MMAR_EVAL_OUTPUT_PATH` as configured in the [environment.json](config/environment.json) file (default is eval folder)<jupyter_code>! $MMAR_ROOT/commands/infer.sh<jupyter_output><empty_output><jupyter_text>Now lets see results in the folder<jupyter_code>! ls -la $MMAR_ROOT/eval/
! ls -la $MMAR_ROOT/eval/spleen_8<jupyter_output><empty_output><jupyter_text>Multi-GPU TrainingClara train aims to simplify scaling and utilizing all available gpus. Using the same config we already used for train we can simply invoke `train_2gpu.sh` to train on multiple gpus. We are using MPI and Horovod to speed up training and passing weights between GPUs as shown below![tb](screenShots/MultiGPU.png)Lets examine `train_2gpu.sh` script by running cell below. You can see we are changing the learning rate as the batch size has doubled.<jupyter_code>printFile(MMAR_ROOT+"/commands/train_2gpu.sh",0,50)<jupyter_output><empty_output><jupyter_text>Lets give it a try and run cell below to train on 2 gpus<jupyter_code>! $MMAR_ROOT/commands/train_2gpu.sh<jupyter_output><empty_output><jupyter_text>Exercise:Now that you are familiar with clara train, you can try to: 1. Explore different options of clara train by changing / creating a new config file and running training:     1. Model architecture: Ahnet, Unet, Segresnet     2. Losses    3. Transformation 2. Train on real spleen data for this you should:    1. Download spleen dataset by running the [download](download) Notebook    2. Switch the dataset file in the [environment.json](config/environment.json)    3. rerun the `train.sh`3. Experiment with multi-GPU training by changing number of gpus to train on from 2 to 3 or 4. You should edit [train_2gpu.sh](commands/train_2gpu.sh) then rerun the script  4. Use DLprof tool for debugging<jupyter_code>!$MMAR_ROOT/commands/debug_dlprof.sh trn_base.json<jupyter_output><empty_output><jupyter_text>You then need to run tensor board manually (Not through jupyterlab) using ```cd /claraDevDay/GettingStarted/models/trn_base_debugtensorboard --logdir ./dlprof --port 80```remember we startdocker.sh mounted port 80 to 5000 by default for AIAA, we simply are using that mapping here for simplicity now if you navigate to `` you should see DlProf tool as below. This analysis shows you the GPUs you have along improvements that you can do to train faster. For example this run shows multiple operations that would be accelerated from AMP.To test this you can run cell below with AMP enabled in the configuration ![dlprof](screenShots/Dlprof.png)<jupyter_code>!$MMAR_ROOT/commands/debug_dlprof.sh trn_base_AMP.json<jupyter_output><empty_output><jupyter_text>Advanced Exercise for K8s:In case your system has K8s cluster and you would like to run clara train jobs, you should try the following:1. Run a K8s job using 1 gpu by running `kubectl apply -f ./config/k8config.yaml`2. Run a K8s job using 2 (or N gpus) by running `kubectl apply -f ./config/k8config2gpu.yaml`you may need to delete the pod with the `clarak8` name to re-run using `kubectl delete pod clarak8`.  Advanced Exercise Connect train to deployYou can export model to be ready to be consumed by clara deploy. You can use [export2Triton.sh](commands/export2Triton.sh) script to generate the model as triton expects it (used by clara deploy)<jupyter_code>!$MMAR_ROOT/commands/export2Triton.sh<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10387108723837747, 'fraction_numerical': 0.01696610483422856, 'mean_word_length': 5.512907117008444, 'pattern_counts': {'":': 10, '<': 81, '<?xml version=': 0, '>': 80, 'https://': 58, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '262333', 'license': 'Apache-2.0', 'n_tokens_mistral': 9272, 'n_tokens_neox': 8766, 'n_words': 1596, 'path': 'NoteBooks/GettingStarted/GettingStarted.ipynb', 'repo_name': 'ZQyou/clara-train-examples', 'size': '26995'}