<jupyter_start><jupyter_text>Note book for Digital Pathology usecase   DescriptionA pre-trained model for automated detection of metastases in whole-slide histopathology images. The prediction map is generated in a sliding-window manner by classifying local 224x224x3 RGB patches as either tumor or normal. Disclamer This note book will use a single file to train. This is **ONLY** intended to show the user how to easily get started. For the best model please download the final model from NGC.  Prerequisites- Before running any code, please install "openslide-python" and OpenSlide libraries.  (Temporarily till GA)We noticed some torch version issues that would be fixed in GA release. For you cell below would fix the issue<jupyter_code>!pip install --no-deps torchvision==0.8.0<jupyter_output><empty_output><jupyter_text>1. Download DataAll the data used to train, validate, and test this model is from [Camelyon-16 Challenge](https://camelyon16.grand-challenge.org/).First lets setup directories for the data<jupyter_code>import os
DataDirRoot="/claraDevDay/Data/DP_CAMELYON16/"

DataDirJson=DataDirRoot+"jsons/train/"
DataDirCoord=DataDirRoot+"coords/"
os.makedirs(DataDirCoord, exist_ok=True)
os.makedirs(DataDirJson, exist_ok=True)
os.makedirs(DataDirRoot+"tif/", exist_ok=True)
os.makedirs(DataDirRoot+"LocLabel/", exist_ok=True)<jupyter_output><empty_output><jupyter_text>1.1 Download tiff manually You can download all the images for "CAMELYON16 data set" from various sources listed [here](https://camelyon17.grand-challenge.org/Data/).For simplicity you only need the smallest file tumor_091.tif (500Mb) **Please note: This download would take 15+ minutes**<jupyter_code>! pip install progressbar2
from progressbar import ProgressBar, Percentage, Bar, ETA, FileTransferSpeed
def download_file_with_progressbar(data):
    f.write(data) 
    global bar
    bar += len(data)

import ftplib
import os
def download_camelyon16_image(filename):
    filename = filename.lower()
    if os.path.exists(filename):
        print(f"The image [{filename}] already exist locally.")
    else:
        print(f"Downloading '{filename}'...")
        prefix = filename.split("_")[0].lower()
        if prefix == "test":
            folder_name = "testing/images"
        elif prefix in ["normal", "tumor"]:
            folder_name = f"training/{prefix}"
        else:
            raise ValueError(
                f"'{filename}' not found on the server."
                " File name should be like 'test_001.tif', 'tumor_001.tif', or 'normal_001.tif'"
            )
        path = f"gigadb/pub/10.5524/100001_101000/100439/CAMELYON16/{folder_name}/"
        ftp = ftplib.FTP("parrot.genomics.cn")
        ftp.login("anonymous", "")
        filepath=path+filename
        print("Downloading ",filepath)
        size = ftp.size(filepath)
        global bar
        bar = ProgressBar(widgets=['Downloading: ', Percentage(), ' ',
                        Bar(marker='#',left='[',right=']'),
                        ' ', ETA(), ' ', FileTransferSpeed()], maxval=size)
        bar.start()    
        global f
        f = open(filename, 'wb')  
        #ftp.cwd(path)
        #ftp.retrbinary("RETR " + filename, open(filename, "wb").write)
        ftp.retrbinary("RETR " + filepath, download_file_with_progressbar)

        ftp.quit()
%cd $DataDirRoot/tif
download_camelyon16_image("tumor_091.tif")<jupyter_output><empty_output><jupyter_text>Check that file was downloaded in tif folder<jupyter_code>!ls $DataDirRoot/tif<jupyter_output><empty_output><jupyter_text>1.2. Download Json filesLocation information for training/validation patches are adopted from [NCRF/coords](https://github.com/baidu-research/NCRF/tree/master/coords).Cell below will download the needed files<jupyter_code>DataDirJson=DataDirRoot+"jsons/train/"
blobURL="https://raw.githubusercontent.com/baidu-research/NCRF/master/jsons/train/"

FileName="Tumor_091.json"
wget_URL=blobURL+FileName
!wget $wget_URL -P $DataDirJson<jupyter_output><empty_output><jupyter_text>1.3. Download coordsAnotation information are adopted from [NCRF/jsons](https://github.com/baidu-research/NCRF/tree/master/jsons).Cell below will download the needed files<jupyter_code>coordsURL="https://raw.githubusercontent.com/baidu-research/NCRF/master/coords/"
FileName="tumor_train.txt"
wget_URL=coordsURL+FileName
!wget $wget_URL -P $DataDirRoot<jupyter_output><empty_output><jupyter_text>let's only keep locations for tumors we downloaded<jupyter_code>cmd="grep Tumor_091 "+DataDirRoot+FileName+" > "+DataDirCoord+FileName
! $cmd<jupyter_output><empty_output><jupyter_text>Lets Get Started<jupyter_code>MMAR_ROOT="/claraDevDay/DomainExamples/DP_detection/"
print ("setting MMAR_ROOT=",MMAR_ROOT)
%ls $MMAR_ROOT
!chmod 777 $MMAR_ROOT/commands/*<jupyter_output><empty_output><jupyter_text>2. Data Preparation Input and output formatsInput for the training pipeline includes: 1. folder containing all WSIs2. txt files listing the location and label information for training patches.Output of the network itself is the probability of a 224x224x3 patch.- For training / validation: `prepare_train_data.sh` is used to generate the LocLabel files needed for training and validation from /coords and /jsons listed above. It will append the labels after each filename + coordinate pairs. Together with training images, they will be passed to training/validation pipeline.- For inference: `prepare_inference_data.sh` is used to generate foreground masks that will be used to reduce computation burden during inference. The input is the test images, and output is the foreground masks.- For FROC: refer to "Annotation" section of [Camelyon challenge](https://camelyon17.grand-challenge.org/Data/) to prepare ground truth images, which are needed for FROC computation.<jupyter_code>! $MMAR_ROOT/commands/prepare_train_data.sh<jupyter_output><empty_output><jupyter_text>3. Training  Model OverviewThe model is based on ResNet18 with the option of replacing last fully connected layer by a 1x1 convolution layer.  3.1 Normal TrainingLets start training with basic configuration<jupyter_code>! $MMAR_ROOT/commands/train.sh<jupyter_output><empty_output><jupyter_text>3.2 Training using smart cacheNow lets take advantage of smart cache data pipeline<jupyter_code>! $MMAR_ROOT/commands/train_smartcache.sh<jupyter_output><empty_output><jupyter_text>Scores and ResultsExample shown here uses single tumor image for simplisity. Therefore, the model is a dummy model that is not useful. You can either download all the data and retrain or use our model form NGC.Our trained model on NGC achieve the ~0.92 accuracy on validation patches, and FROC of ~0.72 on the 48 Camelyon testing data that have ground truth annotations available.  Exercise  1. Train using open slide You can compare performance against openslide.For this you would to install openslide packages by running cell below<jupyter_code># Install openslide packages  
!apt-get -y install openslide-tools
!apt-get -y install python-openslide
!python3 -m pip install --upgrade pip
!pip install openslide-python<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09432105707056508, 'fraction_numerical': 0.016727579420860277, 'mean_word_length': 4.8559670781893, 'pattern_counts': {'":': 1, '<': 49, '<?xml version=': 0, '>': 50, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '186540', 'license': 'Apache-2.0', 'n_tokens_mistral': 2243, 'n_tokens_neox': 2095, 'n_words': 640, 'path': 'PyTorch-Early-Access/NoteBooks/DomainExamples/DP_detection/DP.ipynb', 'repo_name': 'ZQyou/clara-train-examples', 'size': '7114'}