Fix inference print logger and froc

README.md

@@ -70,7 +70,8 @@ Inference is performed on WSI in a sliding window manner with specified stride.
 # Model Performance
 
 FROC score is used for evaluating the performance of the model. After inference is done, `evaluate_froc.sh` needs to be run to evaluate FROC score based on predicted probability map (output of inference) and the ground truth tumor masks.
-This model achieve the ~0.92 accuracy on validation patches, and FROC of ~0.72 on the 48 Camelyon testing data that have ground truth annotations available.
+This model achieve the ~0.91 accuracy on validation patches, and FROC of 0.685 on the 48 Camelyon testing data that have ground truth annotations available.
+![model performance](<https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_tumor_detection_train_and_val_metrics.png>)
 
 # Commands example
 
@@ -89,16 +90,16 @@ torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training
 Please note that the distributed training related options depend on the actual running environment, thus you may need to remove `--standalone`, modify `--nnodes` or do some other necessary changes according to the machine you used.
 Please refer to [pytorch's official tutorial](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) for more details.
 
-Override the `train` config to execute evaluation with the trained model:
+Execute inference:
 
 ```
-python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf
+CUDA_LAUNCH_BLOCKING=1 python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf
 ```
 
-Execute inference:
+Evaluate FROC metric:
 
 ```
-python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf
+cd scripts && source evaluate_froc.sh
 ```
 
 Export checkpoint to TorchScript file:

configs/inference.json

@@ -106,7 +106,7 @@
         {
             "_target_": "StatsHandler",
             "tag_name": "progress",
-            "iteration_print_logger": "$lambda engine: print(f'image: \"{engine.state.batch[\"metadata\"][\"name\"][0]}\", iter: {engine.state.iteration}/{engine.state.epoch_length}') if engine.state.iteration % 100 == 0 else None",
+            "iteration_print_logger": "$lambda engine: print(f'image: \"{engine.state.batch[\"image\"].meta[\"name\"][0]}\", iter: {engine.state.iteration}/{engine.state.epoch_length}') if engine.state.iteration % 100 == 0 else None",
             "output_transform": "$lambda x: None"
         },
         {

configs/metadata.json

@@ -1,7 +1,8 @@
 {
     "schema": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_20220324.json",
-    "version": "0.4.0",
+    "version": "0.4.1",
     "changelog": {
+        "0.4.1": "Fix inference print logger and froc",
         "0.4.0": "add lesion FROC calculation and wsi_reader",
         "0.3.3": "update to use monai 1.0.1",
         "0.3.2": "enhance readme on commands example",

docs/README.md

@@ -63,7 +63,8 @@ Inference is performed on WSI in a sliding window manner with specified stride.
 # Model Performance
 
 FROC score is used for evaluating the performance of the model. After inference is done, `evaluate_froc.sh` needs to be run to evaluate FROC score based on predicted probability map (output of inference) and the ground truth tumor masks.
-This model achieve the ~0.92 accuracy on validation patches, and FROC of ~0.72 on the 48 Camelyon testing data that have ground truth annotations available.
+This model achieve the ~0.91 accuracy on validation patches, and FROC of 0.685 on the 48 Camelyon testing data that have ground truth annotations available.
+![model performance](<https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_tumor_detection_train_and_val_metrics.png>)
 
 # Commands example
 
@@ -82,16 +83,16 @@ torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training
 Please note that the distributed training related options depend on the actual running environment, thus you may need to remove `--standalone`, modify `--nnodes` or do some other necessary changes according to the machine you used.
 Please refer to [pytorch's official tutorial](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) for more details.
 
-Override the `train` config to execute evaluation with the trained model:
+Execute inference:
 
 ```
-python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf
+CUDA_LAUNCH_BLOCKING=1 python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf
 ```
 
-Execute inference:
+Evaluate FROC metric:
 
 ```
-python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf
+cd scripts && source evaluate_froc.sh
 ```
 
 Export checkpoint to TorchScript file:

models/model.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:90c8ae76f342c6fa5d89b9df9f5af014c9fd547067db5bae00a8e9a94e242e7a
-size 44790201
+oid sha256:9b2684dad87e6c1d25e765c6408a6a0f387c1887eb318a6b5b367e3d963231ac
+size 44780565

scripts/evaluate_froc.sh

@@ -4,7 +4,7 @@ LEVEL=6
 SPACING=0.243
 READER=openslide
 EVAL_DIR=../eval
-GROUND_TRUTH_DIR=/workspace/data/medical/pathology/ground_truths
+GROUND_TRUTH_DIR=/workspace/data/medical/pathology/testing/ground_truths
 
 echo "=> Level= ${LEVEL}"
 echo "=> Spacing = ${SPACING}"

scripts/lesion_froc.py

@@ -17,7 +17,7 @@ def load_data(ground_truth_dir: str, eval_dir: str, level: int, spacing: float):
     for prob_name in prob_files:
         if prob_name.endswith(".npy"):
             sample = {
-                "tumor_mask": full_path(ground_truth_dir, prob_name.replace("npy", "tif")),
+                "tumor_mask": full_path(ground_truth_dir, prob_name[:-4]),
                 "prob_map": full_path(eval_dir, prob_name),
                 "level": level,
                 "pixel_spacing": spacing,