<jupyter_start><jupyter_text>NVIDIA Nsight Systems[NVIDIA Nsight™ Systems](https://developer.nvidia.com/nsight-systems) is a system-wide performance analysis tool designed to visualize an application’s algorithms, help to identify the largest opportunities to optimize, and tune to scale efficiently across any quantity or size of CPUs and GPUs.  NVIDIA Tools Extension (NVTX)The [NVIDIA® Tools Extension Library (NVTX)](https://github.com/NVIDIA/NVTX) is a powerful mechanism that allows users to manually instrument their application. With a C-based and a python-based Application Programming Interface (API) for annotating events, code ranges, and resources in your applications. Applications which integrate NVTX can use NVIDIA Nsight, Tegra System Profiler, and Visual Profiler to capture and visualize these events and ranges. In general, the NVTX can bring valuable insignt into the application while incurring almost no overhead.   MONAI Training Pipeline and NVTX[MONAI](https://github.com/Project-MONAI/MONAI) is a high level framework for deep learning in healthcare imaging. For performance profiling, we mainly focus on two fronts: data loading/transforms, and training/validation iterations.[Transforms](https://github.com/Project-MONAI/MONAI/tree/dev/monai/transforms) is one core concept of data handling in MONAI, similar to [TorchVision Transfoms] (https://pytorch.org/vision/stable/transforms.html). Several of these transfoms are usualy chained together, using a [Compose](https://github.com/Project-MONAI/MONAI/blob/2f1c7a5d1b47c8dd21681dbe1b67213aa3278cd7/monai/transforms/compose.pyL35) class, to create a preprocessing or postprocessing pipeline that performs manipulation of the input data and make it suitable for training a deep learning model or inference. To dig into the cost from each individual transform, we enable the insertion of NVTX annotations via [MONAI NVTX Tranforms]().For training and validation steps, they are easier to track by setting NVTX annotations within the loop.  Profiling Spleen Segmentation Pipeline Environment SetupInstall packages and download dataset.<jupyter_code>!pip install git+https://github.com/Project-MONAI/MONAI
!pip install pytorch-ignite
!pip install nibabel
!pip install tqdm
!pip install -q "nvidia-pyindex"
!pip install -q "nvidia-dlprof[pytorch]"
!pip install -q "nvtx"<jupyter_output><empty_output><jupyter_text>Run Nsight Profiling We run DLprof (v1.4.0 / r21.08) on the trainer under basice settings for 6 epochs, with validation every 2 epochs.<jupyter_code>!dlprof --mode pytorch \
        --reports=summary \
        --formats json \
        --output_path ./outputs_base \
        python3 train_base_nvtx.py<jupyter_output><empty_output><jupyter_text>Identify Potential Performance Improvements  Profile ResultsAfter profiling, DLProf provides summary regarding the training process. Also, the computing details can be visulized via Nsight System GUI. (The version of Nsight used in the tutorial is 2021.3.1.54-ee9c30a OSX)<jupyter_code>from IPython.display import display, Image
display(Image(filename="Figure/nsight_base.png", width=800))<jupyter_output><empty_output><jupyter_text>ObservationsAs can be observed from the Nsight GUI, clear patterns can be found:- 6 epochs' training:  - Epochs 1/3/5 withour validation;  - Epoch 2/4/6 with validation;- Clear data loading/IO gaps between epochs.It can be noticed from the profiling result that:- CacheDataset would load all training images into CPU RAM (the initial ~40 sec.);- Between epochs of training/validation, there are considerable amount of time cost for data loading (red);- Between steps/iterations of training/validation, the time of data loading is much smaller (red);- The GPU utilization (CUDA HW) is decent during each training/validation step (green/blue/yellow);Upon further analysis of convergence, it appears that the convergence is relatively slow. Therefore, there are two directions of performance improvement:- Per-epoch time reduction: by utilizing pre-indexing and thread-based dataloader;- Faster convergence: by utilizing different optimizers and loss functions.One optimized solution can be found here (https://github.com/Project-MONAI/tutorials/blob/master/acceleration/fast_training_tutorial.ipynb). Then we add the corresponding tensorboard figure. Based on the valiation accuracy curves, we can observe that the optimized solution has much better convergence. And the GPU utilization rate is above 90% on average.<jupyter_code>display(Image(filename="Figure/tensorboard.png", width=800))<jupyter_output><empty_output><jupyter_text>Analyzing Performance Improvement Profile ResultsWe again use DLProf to further analyze the optimized training script.<jupyter_code>!dlprof --mode pytorch \
        --reports=summary \
        --formats json \
        --output_path ./outputs_fast \
        python3 train_fast_nvtx.py 
display(Image(filename="Figure/nsight_fast.png", width=800))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07200321802091714, 'fraction_numerical': 0.013676588897827836, 'mean_word_length': 5.335031847133758, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '325569', 'license': 'Apache-2.0', 'n_tokens_mistral': 1411, 'n_tokens_neox': 1312, 'n_words': 535, 'path': 'performance_profiling/profiling_train_base_nvtx.ipynb', 'repo_name': 'holgerroth/tutorials', 'size': '4972'}