Weakly-supervised High-resolution Segmentation of Mammography Images for Breast Cancer Diagnosis

In the last few years, deep learning classifiers have shown promising results in image-based medical diagnosis. However, interpreting the outputs of these models remains a challenge. In cancer diagnosis, interpretability can be achieved by localizing the region of the input image responsible for the output, i.e. the location of a lesion. Alternatively, segmentation or detection models can be trained with pixel-wise annotations indicating the locations of malignant lesions. Unfortunately, acquiring such labels is labor-intensive and requires medical expertise. To overcome this difficulty, weakly-supervised localization can be utilized. These methods allow neural network classifiers to output saliency maps highlighting the regions of the input most relevant to the classification task (e.g. malignant lesions in mammograms) using only image-level labels (e.g. whether the patient has cancer or not) during training. When applied to high-resolution images, existing methods produce low-resolution saliency maps. This is problematic in applications in which suspicious lesions are small in relation to the image size. In this work, we introduce a novel neural network architecture to perform weakly-supervised segmentation of high-resolution images. The proposed model selects regions of interest via coarse-level localization, and then performs fine-grained segmentation of those regions. We apply this model to breast cancer diagnosis with screening mammography, and validate it on a large clinically-realistic dataset. Measured by Dice similarity score, our approach outperforms existing methods by a large margin in terms of localization performance of benign and malignant lesions, relatively improving the performance by 39.6% and 20.0%, respectively. Code and the weights of some of the models are available at https://github.com/nyukat/GLAM


Introduction
Convolutional neural networks (CNNs) have revolutionized medical image analysis (Bekkers et al., 2018;Ching et al., 2018;Topol, 2019;Prevedello et al., 2019;Lutnick et al., 2019;Karimi et al., 2020;Gaonkar et al., 2021). These networks achieve impressive results, but their outputs are often difficult to interpret, which is problematic for clinical decision making (Yao et al., 2018). Designing Figure 1: Comparison of saliency maps generated by CAM (Zhou et al., 2016), GMIC (Shen et al., 2021), and the proposed method on a mammography image containing a malignant lesion (first row, red) and a benign lesion (second row, green). Both CAM and GMIC produce coarse saliency maps that fail to localize the lesion accurately. The proposed method generates a high-resolution saliency map that precisely localizes the lesions.
In this work, we propose GLAM (Global-Local Activation Maps), a novel framework to generate fine-grained segmentation using only image-level labels. The proposed model processes high resolution medical images in a memory-efficient way. The main idea behind GLAM is to select informative regions (patches) that may contain ROIs via coarse-level localization and then to perform segmentation on selected patches rather than the entire image in a weakly supervised manner. We train and evaluate GLAM on a dataset containing more than one million mammography images. We demonstrate that the model outperforms existing baselines in segmentation of both benign and malignant lesions, improving the Dice similarity score relatively by 39.6% and 20%, respectively, while preserving classification accuracy. To achieve that, GLAM produces fine-grained saliency maps with a 300 times higher resolution than previous works (Shen et al., 2021) (736 × 480 pixels for 2944 × 1920 pixels input images). In Figure 1, we illustrate how the saliency maps generated by GLAM enable high-resolution segmentation of lesions relevant to breast cancer diagnosis.

Background
WSL is the task of learning to locate ROIs (i.e. the objects) in an image when only image-level labels are available during training. WSL methods are usually based on CNNs that produce saliency maps encoding the location of ROIs. To train the whole system using only image-level labels, the saliency maps are collapsed to predictions indicating the presence of each class using a pooling function. Once the CNN is trained, the saliency map can be used for localization (Oquab et al., 2015;Zhou et al., 2016). WSL has been applied in a wide range of medical-imaging applications, including the detection of lung disease in chest X-ray images (Wang et al., 2017;Yao et al., 2018;Tang et al., 2018;Ma et al., 2019;Liu et al., 2019;Guan et al., 2018), diagnosis of injuries from pelvic X-ray images (Wang et al., 2019), brain lesion segmentation (Wu et al., 2019a), breast MRI analysis (Luo et al., 2019), cancer detection in lung CT images (Feng et al., 2017;Schlemper et al., 2018), and scan-plane detection in ultrasound (Schlemper et al., 2018;Baumgartner et al., 2017).
The majority of these works focus on images that have relatively low resolution, that is, 512 × 512 pixels or less. Only a few works have considered higher resolution images, which are standard in some imaging procedures such as screening mammography (Shen et al., 2019(Shen et al., , 2021. In this work we focus on the diagnosis of breast cancer from screening mammography images. Breast cancer is the second leading cause of cancer-related deaths among women (Bray et al., 2018) and screening mammography is the main tool for its early detection (Marmot et al., 2013). CNN classifiers have shown promise in diagnosis from mammograms (Zhu et al., 2017;Kim et al., 2018;Ribli et al., 2018;Wu et al., 2019b;Geras et al., 2019;McKinney et al., 2020;Shen et al., 2019Shen et al., , 2021. Accurate localization of suspicious lesions is crucial to aid clinicians in interpreting model outputs, and can provide guidance for future diagnostic procedures. However, existing methods that explain their predictions, e.g. Shen et al. (2019Shen et al. ( , 2021, offer only coarse localization. GLAM is inspired by recent works (Yao et al., 2018;Shen et al., 2019Shen et al., , 2021Shamout et al., 2020), which improve classification accuracy by processing image patches selected from coarse saliency maps. The main innovation of GLAM with respect to these works is that it generates a a high-resolution saliency map from the selected patches, which significantly improves lesion segmentation accuracy.

Proposed Approach
Our goal is to generate fine-grained saliency maps that localize objects of interest in high-resolution images using only image-level labels during training. We start this section by describing the inference pipeline of our approach in Section 3.1. We then describe each module in detail in Section 3.2. Finally, we explain the training strategy in Section 3.3.

Inference pipeline
As illustrated in Figure 2, during inference, our system processes an input x ∈ R H,W as follows: 1. The image x is fed into the global module, a memory-efficient CNN denoted by f g , to produce an image-level coarse saliency map S g and an image-level class predictionŷ g . Figure 2: Inference pipeline of GLAM. 1) The global network f g is applied to the whole image x to obtain a coarse image-level segmentation map S g . 2) Based on this coarse-level segmentation, several patches are extracted from the input image.
3) The local network f l processes these patches to generate a high-resolution saliency map S l .
2. We select M patches from x based on S g . To do that, we greedily select the patches for which the sum of the entries in S g is the largest (see Algorithm 1 for a detailed description).
3. We feed the selected patchesx 1 , . . . ,x M to the local module f l , another CNN which produces a fine-grained saliency map associated with each patch. We then remap the patch-level saliency maps back to their location in the original input image. We denote the saliency map obtained through this procedure by S l . GLAM produces an image-level saliency map S g and a fine-grained multi-patch saliency map S l . These maps are aggregated through averaging to produce the final saliency map S c = (S g + S l )/2. In addition, GLAM generates a classification output, which is produced by the global module (as this yields the best classification accuracy).

Module parameterization
Global module The architecture of f g is based on the design of Shen et al. (2019Shen et al. ( , 2021, which is similar to ResNet (He et al., 2016) with a reduced number of channels for memory efficiency. The main difference between f g and the global module of Shen et al. (2019Shen et al. ( , 2021 is that we combine saliency maps at different scales to generate the global saliency map, inspired by Sedai et al. (2018) who observed that using convolutional feature maps extracted only from the last layer of a CNN may be suboptimal in localization of small objects. We generate saliency maps at different scales (S 0 , S 1 and S 2 ) using a pyramidal hierarchy of feature maps (see Figure 6 in Appendix B). Our image-level saliency map S g is obtained by averaging them. For each S n (n ∈ {0, 1, 2}), we obtain a classification predictionỹ n associated with S n using top t% pooling (see Appendix B), where t is a hyperparameter. The image-level classification predictionŷ g is calculated by averagingỹ 0 ,ỹ 1 , andỹ 2 . Additionally, we output a representation vector z g to feed it into a fusion module (described below) to enable joint training with the local module. See Appendix B for more details.
Local module Our local module is based on ResNet-34 with a reduced stride in the residual blocks to maintain a higher resolution. We replace the global average pooling and the last fully connected layer by a 1 × 1 convolution layer followed by a sigmoid non-linearity. The local module is applied to each of the selected patchesx k (k ∈ 1, . . . , K) to extract a patch-level saliency map A k . As we only have image-level labels, we need to train the patch-level network without patchlevel labels. To address this challenge, we combine insights from weakly-supervised localization and multiple-instance learning to train the patch-level saliency maps hierarchically. In multiple  instance learning (Maron and Lozano-Pérez, 1998) the labels are associated with bags of instances (the label is negative if all instances in a bag are negative, and positive otherwise). In our case each instance is a patch, and the patches from an image form a bag. We use a patch aggregation function f p to combine the information across all patches and form an image-level predictionŷ l . Formally, we haveŷ l = f p (A 1 , . . . , A k ). We propose two different patch aggregation functions.
• Attention-based aggregation: top t% pooling is applied to A k to obtain a patch-level prediction y k . Additionally, we output a representation vector z k for each patch, which will be aggregated to z l and fed into the fusion module. We use the Gated Attention Mechanism ((Ilse et al., 2018)) to combine the prediction and the representation vectors using attention weights α i ∈ [0, 1]. The prediction is given byŷ l = K i=1 α iŷi , and the representation vector by z l = K i=1 α i z i . Section 3.3 explains when we use each aggregation method. Refer to Appendix C for more details.
Fusion module In order to jointly optimize the parameters of the global and local modules, we incorporate a fusion module consisting of one fully connected layer that combines the representation vectors from the global (z g ) and local modules (z l ) to produce a fusion predictionŷ f . Formally, , where w f is a vector of learnable parameters.

Training strategy
The training strategy that achieves the best experimental performance for GLAM is sequential. We first train the global module, then we train the local module and finally we train both together. This makes sense because in order to train the local module f l effectively, we need to select meaningful input patches. Since the selection relies on the global saliency map, this requires pretraining the global module. Our training strategy is as follows (see also Figure 3).
1. Train the global module with the loss function L g = n∈{0,1,2} (BCE(y,ỹ n ) + λ (i,j) |S n (i, j)|) using the whole training set. Here, BCE(y,ỹ n ) is the binary cross-entropy loss and (i,j) |S n (i, j)| is an 1 regularization enforcing sparsity of the saliency map. The hyperparameter λ balances the two terms. The model is selected based on segmentation performance on the validation set.
2. Create training data for the local module. To produce negative examples, we select K patches randomly from images with negative labels. The positive examples are generated by selecting K patches from images with positive labels using Algorithm 1 applied to the saliency maps produced by the pretrained global module.
3. Train the local module minimizing the cost function L l = BCE(y,ŷ l ) + λ (i,j) |S l (i, j)|. Here we apply the concatenation-based aggregation described in Section 3.2 because it yields the best results experimentally and speeds up training.
4. Train the global and local modules jointly with the fusion module. Here we use the attentionbased aggregation in the local module. The fusion module outputs a fusion predictionŷ f . The loss function for joint training is Note that the number of patches selected as input to the local module during training (K) and inference (M ) do not need to be the same. In fact, we find empirically that it is beneficial to use K as large as possible (within the constraints imposed by GPU memory), which is consistent with Shen et al. (2021). However, choosing a large M increases the false positive rate. We used K = 6 and M = 1, based on experiments reported in Appendix G.4 and G.5.

Dataset and evaluation metrics
We trained and evaluated the proposed model on the NYU Breast Cancer Screening Dataset v1.0 (Wu et al., 2019c) that includes 229,426 exams (1,001,093 images) from 141,472 patients. Each exam contains at least four images with a resolution of 2944×1920 pixels, corresponding to the four standard views used in screening mammography: R-CC (right craniocaudal), L-CC (left craniocaudal), R-MLO (right mediolateral oblique) and L-MLO (left mediolateral oblique). The dataset is divided into disjoint training (186,816), validation (28,462) and test (14,148) sets, ensuring that each patient only belongs to one of the sets. Each breast has two binary labels indicating whether malignant or benign lesions are present. A subset of the images with lesions have pixel-level annotations provided by radiologists, which indicate the position of the lesions. Note that the dataset (458,852 breasts) contains more exams without lesions (452,311 compared to 5,556 with benign lesions, and 985 with malignant lesions). To account for this imbalance when training our models, at each epoch we use all exams with lesions and an equal number of randomly-sampled exams without lesions.
To measure classification performance, we report the area under the ROC curve (AUC) for identifying breasts with both malignant and benign lesions. To evaluate localization ability, we use the Dice similarity coefficient and pixel average precision (PxAP) (Choe et al., 2020). PxAP is the average of the area under the precision-recall curve for each pixel. The threshold to compute the precision and recall is either chosen for each image (image-level PxAP) or fixed for the whole test set (dataset-level PxAP). These metrics are described in more detail in Appendix E.

Comparison to baselines
We compare GLAM to two baselines: GMIC (Shen et al., 2021), a WSOL method specifically designed for high-resolution medical images, and CAM (Zhou et al., 2016), one of the most popular WSOL methods for natural images. We use the same backbone architecture for the two baselines and the global module of GLAM. The same hyperparameter tuning is applied to all models to ensure a fair comparison, as described in Appendix F. The models are selected based on segmentation performance, which is not necessarily equivalent to selection based on classification performance (see Appendix D for further discussion). We also compare to a U-Net (Ronneberger et al., 2015) trained with strong supervision using the pixel-level annotations. This provides an upper bound for the segmentation performance of the WSL methods. In Table 1, we report the performance of GLAM and the baselines. GLAM outperforms both GMIC and CAM in all segmentation evaluation metrics by a large margin, while achieving very similar classification accuracy. We observe a performance gap between GLAM and the strongly supervised U-Net model trained with ground-truth segmentation annotations, which indicates that there is room for further improvement.

Ablation study
We analyze different design choices in GLAM through an extensive ablation study. Due to space constraints, here we only discuss the advantages of training the global and local modules jointly, and the segmentation properties of the global and local saliency maps. We defer results on the following choices to the appendices: design of the global module (Appendix G.1), selection of training data for the local module (Appendix G.2), design of the local module (Appendix G.3), number of input patches used by the local module during training (Appendix G.4) and inference (Appendix G.5), the fusion module (Appendix G.6), and hyper-parameter selection: (Appendix G.7, G.8 and G.9).
Joint training of local and global modules. Here we compare the GLAM saliency map S c−joint obtained via joint training as described in Section 3.3 with (1) the global saliency map S g from the global module pretrained in isolation, (2) the local saliency map S l for the local module trained using patches selected using a frozen global module, (3) a saliency map S c−sep obtained by averaging S g and S l . The results reported in Table 2 show that S c−sep is superior to S g and S l , but joint training achieves better performance across all metrics.
Segmentation properties of the global and local saliency maps In Figure 4, we plot the Dice scores of the global (S g ) and local (S l ) saliency maps generated by GLAM for 400 randomly selected examples from the validation set. Each saliency map has different strengths and weaknesses. S l fails completely for a subset of examples, this is because the patch-selection procedure did not select the correct patches (lower row in Figure 5). On the remaining examples, S l tends to outperform S g because it has a much higher resolution. However, in some cases it underperforms S g , often in cases where the ground-truth segmentation is larger than the size of the patch (upper row in Figure 5). Averaging S l and S g (our strategy of choice in GLAM) achieves high-resolution segmentation, while hedging against the failure of the local module.

Conclusion
In this work, we propose a novel framework to perform weakly-supervised segmentation of images with very high resolution, which outperforms existing methods. Our results suggest that the general principle underlying GLAM (hierarchical selection of saliency maps) could be effective in other applications involving high-resolution images and videos. Another interesting question for future research is how to extend this principle to settings where high-resolution segmentation is desired, but (in contrast to breast cancer) the regions of interest do not tend to be small. 3. Additional details on the local module in Appendix C.
4. A discussion on the relationship between the localization and classification performance in Appendix D.

A detailed description of the evaluation metrics in Appendix E
6. A description of the hyperparameter tuning procedure in Appendix F.

A description of additional ablation studies in Appendix G.
Appendix A. Patch selection algorithm Algorithm 1 Patch selection algorithm: x denotes the input image, S denotes the saliency map that the patch selection method is based on (in our case of interest, this is the saliency map S g from the global module).x k denotes the selected patches cropped from the original image. K is the predefined number of selected patches.
Require: We perform patch selection using the same greedy algorithm (Algorithm 1) as in GMIC (Shen et al., 2021). In each iteration, we select the rectangular region in the coarse saliency map generated by the global module, which has the largest average intensity (see line 7 of Algorithm 1). Then the region is interpolated so that it maps to the corresponding location on the input image. Line 12 ensures that the extracted ROI patches do not significantly overlap with each other. It is worth noting that we do not assume that all patches extracted from a positive example contain tumors: only some of the K patches extracted by the global module will. This is accounted for by the patch aggregation function in the local module, which produces a single classification output for the K aggregated patches. A problem arises if the patch-selection algorithm fails to select any patches containing lesions for a positive example. This is why it is beneficial to use a larger number of patches during training (see Appendix H.4). It is worth noting, however, that this situation does not arise often in our experiments. In order to verify this, we used a set of images with lesions from the validation set, and checked the three first patches extracted by the global module. These patches did not contain any lesions only in 40 out of 300 images with benign lesions (13%) and in 6 out of 75 images with malignant lesions (8%).

Appendix B. Global module
Top t% pooling  Figure 6: Architecture of the global module. Both high-level (h 2 ∈ R 46×30×256 ) and low-level feature maps (h 0 ∈ R 184×120×64 , h 1 ∈ R 92×60×128 ) are utilized to obtain multi-scale saliency maps (S 0 ∈ R 184×120×2 , S 1 ∈ R 92×60×2 and S 2 ∈ R 46×30×2 ). For each scale n ∈ {0, 1, 2}, we use top t% pooling to transform saliency maps S n into class predictionsỹ n . A combined cross-entropy loss from the three scales is used for training. We compute the global saliency map S g by combining the individual saliency maps from three scales (S 0 , S 1 and S 2 ). We apply maximum pooling on the spatial dimension of h 2 ∈ R 46×30×256 to obtain the representation vector z g ∈ R 256 , which is fed to the fusion module during joint training.
As shown in Figure 6, the global module provides a multi-scale pyramidal hierarchy of feature maps (h 0 , h 1 , h 2 ) when processing an input image. The feature map corresponding to the deepest layer (h 2 ) has low resolution due to downsampling and therefore has limited localization ability. However, it is useful for classification, because it aggregates information from the whole image. Sedai et al. (2018) found that using only the deepest feature maps can negatively influence the localization of small objects. This motivates combining feature maps at several layers (in our case h 0 , h 1 , h 2 ) to produce saliency maps (Feng et al., 2017;Sedai et al., 2018;Yao et al., 2018). To do this, we use a training loss that includes classification estimates obtained from the three feature maps (see Eq. (1) below).
As shown in Figure 6, given a gray-scale input image of x ∈ R 2944×1920 pixels, we have feature maps in three scales (h 0 ∈ R 184×120×64 , h 1 ∈ R 92×60×128 and h 2 ∈ R 46×30×256 ). For each feature map h n of scale n, we use 1 × 1 convolution followed by a sigmoid function to transform h n to a saliency map S n ∈ [0, 1] h×w×|C| , where C = {malignant, benign}. Each element of S n , S n c (i, j), represents the contribution of a spatial location (i, j) towards classifying the input as class c ∈ C. We apply top t% pooling (Shen et al., 2021) as an aggregation function f agg (S n c ) : R h,w → [0, 1] to transform S n c to the image-level class prediction for class c. Formally, y c n = f agg (S n c ) = 1 where H + denotes the set containing the locations of the top t% values in S n c , where t is a hyperparameter. During training we minimize a sum of the cross-entropy losses between the image-level label y andỹ n for the three scales (n ∈ {0, 1, 2}). The training loss function of the global module is the following: (1) Here, BCE(y,ỹ n ) is the cross-entropy loss, (i,j) |S n (i, j)| is a 1 regularization term enforcing sparsity of the saliency map (Shen et al., 2021) and λ is a hyperparameter to balance the two loss terms.
We obtain the global module predictionŷ g by taking the average of the class prediction across the levelsŷ g = (ỹ 0 +ỹ 1 +ỹ 2 )/3. We obtain the global module saliency map S g by combining S 0 , S 1 and S 2 . Note that the resolutions of S 0 , S 1 and S 2 are different (S 0 ∈ R 184×120×2 , S 1 ∈ R 92×60×2 and S 2 ∈ R 46×30×2 ). In order to combine them, we upsample S 0 and S 1 to match the resolution of the S 2 using nearest-neighbor interpolation. Then, we compute the global module saliency map as S g = γ 0 S 0 + γ 1 S 1 + γ 2 S 2 . Here γ 0 , γ 1 and γ 2 are hyperparameters that can be tuned on the validation set. They should satisfy the conditions: γ 0 +γ 1 +γ 2 = 1 and γ 0 ≥ 0, γ 1 ≥ 0, γ 2 ≥ 0. We set γ 0 = 0.2, γ 1 = 0.6, γ 3 = 0.2 according to the Dice score on the validation set. We apply maximum pooling on the spatial dimension of h 2 ∈ R 46×30×256 to obtain a representation vector for the global module z g ∈ R 256 , which is fed into the fusion module during joint training of the global and local modules.
Utilizing multi-scale feature maps improves the segmentation performance of S g . As a result, we obtain a better feature map S c . This is demonstrated empirically in the ablation study reported in Appendix G.1.

Appendix C. Local module
We design the local module in order to preserve as much spatial resolution as possible. The resolution of the global saliency map is only 184 × 120 pixels. Therefore, the global module backbone is not suitable as a local module backbone. Typical classification networks also do not suit our needs. In our case, the size of the patches that are processed by the local module is 512 × 512. For such input resolution, the size of the saliency map produced by ResNet-34 is only 28 × 28 pixels. In order to preserve spatial resolution, we build the backbone for the local module by reducing the stride in all ResNet blocks of the ResNet-34 network to 1. When operating on an input patch with size 512 × 512 pixels, the resulting network outputs a saliency map with a resolution of 127 × 127 pixels. Note that we use ResNet-34 rather than ResNet-18 because it has a larger receptive field, which compensates for the reduced receptive field due to the reduced stride. We provide an ablation study of different choices for the local-module backbone architecture in Appendix G.3.
Architecture of the backbone of the local module is described in the left panel of Figure 7. In the right panel, we show two patch-aggregation strategies. We report an ablation study on the value  Figure 7: Architecture of the local module. The backbone network (left) is applied to each of the selected patchesx k (k ∈ 1, . . . , K) to extract a patch-level saliency map A k and a feature map h k . The patch-level saliency maps can be combined using two aggregation strategies: (1) Concatenation-based aggregation (top right), where we concatenate the saliency maps spatially and apply top t% pooling.
(2) Attention-based aggregation (bottom right), where top t% pooling is used to obtain patch-level predictionsŷ k from each patch, which are then combined using attention weights α i ∈ [0, 1] computed by the gated attention mechanism (Ilse et al., 2018). That is, the classification prediction is computed asŷ l = K i=1 α iŷi , and the representation vector as of t in the top t% pooling step in Appendix G.7. The selected value for both aggregation strategies is 20.

Appendix D. Localization and classification
A few works (Choe et al., 2020;Choe and Shim, 2019;Singh and Lee, 2017;Yao et al., 2018) observed that localization and classification accuracy of weakly-supervised models are not perfectly correlated. An empirical study by Choe et al. (2020) suggested that the best localization performance is usually achieved at early epochs of training, when the classifier is not fully trained. Furthermore, it has been observed that classification performance does not necessarily correlate with the localization performance across different architectures (Choe et al., 2020). We made similar observations in our work. Our local module reduces the stride of the convolution kernel to preserve more spatial resolution, which results in a smaller receptive field and has some adversarial effects for classification performance. Therefore, only the global module classification prediction is used as the classification result for our system. We provide the segmentation and classification results of different local module architectures in Appendix G.3. In this ablation study, we investigate the influence of the design of the local module on its segmentation and classification performance. We compare Resnet-34, a ResNet-18 with reduced stride in the residual blocks (ResNet-18-HR) and the proposed local module, which consists of a Resnet-34 with reduced stride in the residual blocks (Resnet-34-HR). Recall that the motivation to reduce the stride is to produce a saliency map with higher resolution, as explained in Appendix C. The results are shown in the following table. We see that carefully selecting the backbone architecture is crucial to ensure a good segmentation performance of the local module. It is worth mentioning that reducing the stride decreases the classification performance of the local module (which is why we do not use its output to perform classification). In this ablation study we use the model trained with six patches and evaluate its performance when using different numbers of patches during inference. The scatter plot in Figure 8 compares the segmentation performance when using one patch and when using three patches for 400 example images from the validation set. Using one patch leads to better performance for most of the images, but completely fails for some. Failure may occur when the patch selected by the global module for the positive example does not contain lesions. For our dataset, using one patch produces better segmentation performance overall, but this may not be the case for other applications (e.g. if there are a large number of lesions in each image).

G.6. Fusion module
In this ablation study, we compare the proposed model to two different models without fusion modules. In Model 1 the local module is trained using concatenation-based aggregation, in Model 2 it is trained with attention-based aggregation (see Section 3.2). To ensure a fair comparison, the models are obtained by jointly training the same pretrained global and local modules. We report the results in the following  Dice score (benign) Figure 8: Dice score of the local module when using one or three patches as its input during inference. The results correspond to 400 examples from the validation set. Using one patch leads to better performance for most of the images, but completely fails for a some. Failure may occur when the patch selected by the global module for the positive example does not contain lesions.

G.7. Impact of top t% pooling
As already shown by Shen et al. (2021), the choice of pooling hyperparameter has a significant impact on localization performance. In order to study how this hyperparameter influences the localization ability of the local module, we experimented with five choices of top t% pooling: t ∈ {10, 15, 20, 30, 60}. For all of them, we use the same trained global module, and train the local module without joint training. In the following table, we report the segmentation score achieved by each value of t on the test set. The results indicate that the pooling hyperparameter has a significant impact on the localization performance of the patch-level network (i.e. the local module), and the optimal t may be different for different classes. To investigate the influence of the λ hyperparameter associated with 1 regularization, we conduct the following experiments for the global module. We fixed all other hyper-parameter to the same values as used in the optimal model. To keep the comparison scenario simple, the saliency map of the global module is computed as S g = S 0 +S 1 +S 2

3
. λ is randomly selected from 10 [−5.5,−3.5] on a logarithmic scale during hyper-parameter searching for training, we take the following samples of λ for ablation study by randomly selection within the searching space: 10 −3.00 , 10 −3.53 , 10 −3.81 , 10 −4.24 , 10 −4.73 and report their Dice score on the test set. From the following table, we see that the performance of the saliency map is stable within the range of our hyper-parameter search space. For simplicity, in GLAM we set S c = (S g + S l )/2. Here, we investigate applying a weighted average S c = γ c S g + (1 − γ c )S l , with an additional hyperparameter γ c ∈ [0, 1]. The results in the following table show that this may slightly improve performance for some metrics.