Mine yOur owN Anatomy: Revisiting Medical Image Segmentation with Extremely Limited Labels

Recent studies on contrastive learning have achieved remarkable performance solely by leveraging few labels in the context of medical image segmentation. Existing methods mainly focus on instance discrimination and invariant mapping. However, they face three common pitfalls: (1) tailness: medical image data usually follows an implicit long-tail class distribution. Blindly leveraging all pixels in training hence can lead to the data imbalance issues, and cause deteriorated performance; (2) consistency: it remains unclear whether a segmentation model has learned meaningful and yet consistent anatomical features due to the intra-class variations between different anatomical features; and (3) diversity: the intra-slice correlations within the entire dataset have received significantly less attention. This motivates us to seek a principled approach for strategically making use of the dataset itself to discover similar yet distinct samples from different anatomical views. In this paper, we introduce a novel semi-supervised 2D medical image segmentation framework termed Mine yOur owN Anatomy (MONA), and make three contributions. First, prior work argues that every pixel equally matters to the model training; we observe empirically that this alone is unlikely to define meaningful anatomical features, mainly due to lacking the supervision signal. We show two simple solutions towards learning invariances - through the use of stronger data augmentations and nearest neighbors. Second, we construct a set of objectives that encourage the model to be capable of decomposing medical images into a collection of anatomical features in an unsupervised manner. Lastly, our extensive results on three benchmark datasets with different labeled settings validate the effectiveness of our proposed MONA which achieves new state-of-the-art under different labeled settings.


Introduction
With the advent of deep learning, medical image segmentation has drawn great attention and substantial research efforts in recent years. Traditional supervised training schemes coupled with large-scale annotated data can engender remarkable performance. However, training with massive high-quality annotated data is infeasible in clinical practice since a large amount of expert-annotated medical data often incurs considerable clinical expertise and time. Under such a setting, this poses the question of how models benefit from a large amount of unlabelled data during training. Recently emerged methods based on contrastive learning (CL) significantly reduce the training cost by learning strong visual representations in an unsupervised manner [12,18,19,35,36,38,40,41,62,66,91]. A popular way of formulating this idea is through imposing feature consistency to differently augmented views of the same imagewhich treats each view as an individual instance.
Despite great promise, the main technical challenges remain: (1) How far is CL from becoming a principled framework for medical image segmentation? (2) Is there any better way to implicitly learn some intrinsic properties from the original data (i.e., the inter-instance relationships and intrainstance invariance)? (3) What will happen if models can only access a few labels in training?
To address the above challenges, we outline three principles below: (1) tailness: existing approaches inevitably suffer from class collapse problems -wherein similar pairs from the same latent class are assumed to have the same representation [2,21,50]. This assumption, however, rarely holds for real-world clinical data. We observe that the long- tail distribution problem has received increasing attention in the computer vision community [23,44,46,97,113]. In contrast, there have been a few prior long-tail works for medical image segmentation. For example, as illustrated in Figure 1, most medical image images follow a Zipf long-tail distribution where various anatomical features share very different class frequencies, which can result in worse performance; (2) consistency: considering the scarcity of medical data in practice, augmentations are a widely adopted pre-text task to learn meaningful representations. Intuitively, the anatomical features should be semantically consistent across different transformations and deformations. Thus, it is important to assess whether the model is robust to diverse views of anatomy; (3) diversity: recent work [3,84,110] pointed out that going beyond simple augmentations to create more diverse views can learn more discriminative anatomical features. At the same time, this is particularly challenging to both introduce sufficient diversity and preserve the anatomy of the original data, especially in data-scarce clinical scenarios. To deploy into the wild, we need to quantify and address three research gaps from different anatomical views.
In this paper, we present Mine yOur owN Anatomy (MONA), a novel contrastive semi-supervised 2D medical segmentation framework, based on different anatomical views. The workflow of MONA is illustrated in Figure 2.
The key innovation in MONA is to seek diverse views (i.e., augmented/mined views) of different samples whose anatomical features are homogeneous within the same class type, while distinctive for different class types in imbalanced and unlabeled clinical scenarios. We make the following contributions. First, we consider the problem of tailness. An issue is that label classes within medical images typically exhibit a long-tail distribution. Another one, technically more challenging, is the fact that there is only a few labeled data and large quantities of unlabeled ones during training. Intuitively we would like to sample more pixel-level representations from tail classes. Thus, we go beyond the naïve setting of instance discrimination in CL [18,35,38] by decomposing images into diverse and yet consistent anatomical features, each belonging to different classes. In particular, we propose to use pseudo labeling and knowledge distillation to learn better pixel-level representations within multiple semantic classes in a training minibatch. Considering performing pixel-level CL with medi-cal images is impractical for both memory cost and training time, we then adopt active sampling strategies [56] such as in-batch hard negative pixels, to better discriminate the representations at a larger scale.
We further address the two other challenges: consistency and diversity. The success of the common CL theme is mainly attributed to invariant mapping [36] and instance discrimination [18,91]. Starting from these two key aspects, we try to further improve the segmentation quality. More specifically, we suggest that consistency to transformation (equivariance) is an effective strategy to establish the invariances (i.e., anatomical features and shape variance) to various image transformations. Furthermore, we investigate two ways to include diversity-promoting views in sample generation. First, we incorporate a memory buffer to alleviate the demand for large batch size, enabling much more efficient training without inhibiting segmentation quality. Second, we leverage stronger augmentations and nearest neighbors to mine views as positive views for more semantic similar contexts.
Extensive experiments are conducted on a variety of datasets and the latest CL frameworks (i.e., Mo-Cov2 [19], kNN-MoCo [84], SimCLR [18], BYOL [35], and ISD [81]), which consistently demonstrate the effectiveness of our proposed MONA. For example, our MONA establishes the new state-of-the-art performance, compared to both the state-of-the-art semi-supervised and fullysupervised approaches with 10% label ratio. Moreover, we present a systematic evaluation for analyzing why our approach performs so well and how different factors contribute to the final performance (See Section 4.2). We also study whether these principle can effectively complement each CL frameworks (See Appendix C). We hope our findings will provide useful insights on medical image segmentation to other researchers.
To summarise, our contributions are as follows: ❶ we carefully examine the problem of semi-supervised 2D medical image segmentation with extremely limited labels, and identify the three principles to address such challenging tasks; ❷ we construct a set of objectives, which significantly improves the segmentation quality, both long-tail class distribution and anatomical features; ❸ we analyse several critical components of our method and conduct thorough ablation studies to validate their necessity; ❹ with the combina-  tion of different components, we establish state-of-the-art, for all the challenging three benchmarks.

Related work
Medical Image Segmentation. Medical image segmentation aims to assign a class label to each pixel in an image, and plays a major role in real-world applications, such as assisting the radiologists for better disease diagnosis and reduced cost. With sufficient annotated training data, significant progress has been achieved with the introduction of Fully convolutional networks (FCN) [57] and UNet [72]. Follow-up works can be categorized into two main directions. One direction is to improve modern segmentation network design. Many CNN-based [39,78] and Transformerlike [26,85] model variants [1, 10, 15-17, 25, 37, 43, 61, 65, 82, 89, 90, 95, 98-101] have been proposed since then. For example, some works [16,17,24] proposed to use dilated/atrous/deformable convolutions with larger receptive fields for more dense anatomical features. Other works [10,15,37,82,95,101] include Transformer blocks to capture more long-range information, achieving the impressive performance. A parallel direction is to select proper optimization strategies, by designing loss functions to learn meaningful representations [55,77,96]. However, those methods assume access to a large, labeled dataset. This restrictive assumption makes it challenging to deploy in most real-world clinical practices. In contrast, our MONA is more robust as it leverages only a few labeled data and large quantities of unlabeled one in the learning stage. Semi-Supervised Learning (SSL). The goal in robust SSL is to improve the medical segmentation performance by taking advantage of large amounts of unlabelled data during training. It can be roughly categorized into three groups: (1) self-training by generating unreliable pseudolabels for performance gains, such as pseudo-label estimation [4,20,27,49], model uncertainty [9,11,33,45,60,63,104,106], confidence estimation [7,31,48], and noisy student [93]; (2) consistency regularization [8,22,28,29,111] by integrating consistency corresponding to different transformation, such as pi-model [74], co-training [70,112], and mean-teacher [53,71,80]; (3) other training strategies such as adversarial training [51,64,83,[107][108][109] and entropy minimization [34]. In contrast to these works, we do not explore more advanced pseudo-labelling strategy to learn spatially structured representations. In this work, we are the first to explore a novel direction for discovering distinctive and semantically consistent anatomical features without image-level or region-level labels. Further, we expect that our findings can be relevant for other medical image segmentation frameworks.
Contrastive Learning. CL has recently emerged as a promising paradigm for medical image segmentation via exploiting abundant unlabeled data, leading to state-of-theart results [13,14,42,92,94,102,103,105]. The high-level idea of CL is to pull closer the different augmented views of the same instance but pushes apart all the other instances away. Intuitively, differently augmented views of the same image are considered positives, while all the other images serve as negatives. The major difference between different CL-based frameworks lies in the augmentation strategies to obtain positives and negatives. A few very recent studies [44,46] confirm the superiority of CL of addressing imbalance issues in image classification. Moreover, existing CL frameworks [13,42,102] mainly focus on the in-stance level discrimination (i.e., different augmented views of the same instance should have similar anatomical features or clustered around the class weights). However, we argue that not all negative samples equally matter, and the above issues have not been explored from the perspective of medical image segmentation, considering the class distributions in the medical image are perspectives diverse and always exhibit long tails [32,73]. Inspired by the aforementioned, we address these two issues in medical image segmentation -two appealing perspectives that still remain under-explored.

.1. Framework
Overview. We introduce our contrastive learning framework (See Figure 2), which includes (1) relational semisupervised pre-training, and (2) anatomical contrastive reconstruction fine-tuning. The key idea is to seek diverse yet semantically consistent views whose anatomical features are homogeneous within the same class type, while distinctive for different class types in imbalanced and unlabeled clinical scenarios. In this paper, our pre-training stage is built upon ISD [81] -a competitive framework for image classification. The main differences between ISD and MONA are: ❶ MONA is more tailored to medical image segmentation, i.e., considering the dense nature of this problem both in global and local manners, and can generalize well to those long-tail scenarios; ❷ we directly work at the pixel-level local representations by allowing for the correspondence between anatomical features; ❸ we contrast the query image with other random images instead of only with different augmented views of the same query images. We believe our method can be formulated as the relaxed CL in both global and local manners. Such global and local relaxations enable the model to prioritize the most informative demonstrations among the entire dataset; ❹ our principles are expected to apply to other CL frameworks (i.e., Mo-Cov2 [19], kNN-MoCo [84], SimCLR [18], BYOL [35], and ISD [81]). More detailed analysis can be found in the Appendix C. Pre-training preliminary. Let (X, Y ) be our dataset, including training images x ∈ X and their corresponding Cclass segmentation labels y ∈ Y , where X is composed of N labeled and M unlabeled slices. Note that, for brevity, y can be either sampled from Y or pseudo-labels. The student and teacher networks F, parameterized by weights θ and ξ, each consist of a encoder E and a decoder D (i.e., UNet [72]). Concretely, given a sample s from our unlabeled dataset, we have two ways to generate views: (1) we formulate augmented views (i.e., x, x ′ ) through two different augmentation chains; and (2) we create d mined views (i.e., x r,i ) by randomly selecting from the unlabeled dataset followed by additional augmentation. 1 We then fed the augmented views to both F θ and F ξ , and the mined views to F ξ . In contrast to [81], we propose to perform instance discrimination in both global latent and local output feature spaces. 2 Specifically, the encoders generate global fea- , and z r,g = E ξ (x r ), which are then fed into the nonlinear projection heads to obtain v g = h θ (z g ), v ′ g = h ξ (z ′ g ), and w g = h ξ (z r,g ). The augmented embeddings from the student network are further projected into secondary space, i.e., u g = h ′ θ (v g ). We calculate similarities across mined views and augmented views from the student and teacher in both global and local manners. Then a softmax function is applied to process the calculated similarities, which models the relationship distributions: where τ θ and τ ξ are different temperature parameters, and sim(·, ·) denotes cosine similarity. The unsupervised instance discrimination loss (i.e., Kullback-Leibler divergence KL) can be defined as: The parameters ξ of F ξ is updated as: ξ = tξ + (1 − t)θ with t = 0.99 as a momentum hyperparameter. In our pretraining stage, the total loss is the sum of global and local instance discrimination loss L inst (on pseudo-labels), and supervised segmentation loss L sup (i.e., equal combination of dice loss and cross-entropy loss on ground-truth labels): L global inst + L local inst + L sup . Principles. As shown in Figure 2, the principles behind MONA (i.e., the second anatomical contrastive reconstruction stage) aim to ensure tailness, consistency, and diversity. Concretely, tailness is for actively sampling more tail class hard pixels; consistency ensures the feature invariances; and diversity further encourages to discover more anatomical features in different images.

Anatomical Contrastive Reconstruction
Tailness. Motivated by the observations (Figure 1), our primary cue is that medical images naturally exhibit an imbalanced or long-tailed class distribution, wherein many class labels are associated with only a few pixels. To generalize well on such imbalanced setting, we propose to use anatomical contrastive formulation (ACF) (See Figure 3).  Here we additionally attach the representation heads to fuse the multi-scale features with the feature pyramid network (FPN) [54] structure and generate the m-dimensional representations with consecutive convolutional layers. The high-level idea is that the features should be very similar among the same class type, while very dissimilar across different class types. Particularly for long-tail medical data, a naïve application of this idea would require substantially computational resources proportional to the square of the number of pixels within the dataset, and naturally overemphasize the anatomy-rich head classes and leaves the tail classes under-learned in learning invariances, both of which suffer performance drops.
To this end, we address this issue by actively sampling a set of pixel-level anchor representations r q ∈ R c q (queries), pulling them closer to the class-averaged mean of representations r c,+ k within this class c (positive keys), and pushing away from representations r − k ∈ R c k from other classes (negative keys). Formally, the contrastive loss is defined as: 3) where C denotes a set of all available classes for each minibatch, and τ is a temperature hyperparameter. Suppose A is a collection including all pixel coordinates within x, these representations are: We then note that CL might benefit more, where the instance discrimination task is achieved by incorporating  more positive and negative pairs. However, naively unrolling CL to this setting is impractical since it requires extra memory overheads that grow proportionally with the amount of instance discrimination tasks. To this end, we adopt a random set (i.e., the mini-batch) of other images. Intuitively, we would like to maximize the anatomical similarity between all the representations from the query class, and analogously minimize all other class representations. We then create a graph G to compute the pair-wise class relationship: G[p, q] = r p,+ k · r q,+ k , ∀p, q ∈ C, and p ̸ = q, where G ∈ R |C|×|C| . Here finding the accurate decision boundary can be formulated mathematically by normalizing the pair-wise relationships among all negative class representations via the softmax operator. To address the challenge in imbalanced medical image data, we define the pseudo-label (i.e., easy and hard queries) based on a defined threshold as follows: where y q is the c th -class pseudo-label corresponding to r q , and δ θ is the user-defined threshold. For further improvement in long-tail scenarios, we construct a class-aware memory bank [38] to store a fixed number of negative samples per class c. Consistency. The proposed ACF is designed to address imbalanced issues, but anatomical consistency remains to be weak in the long-tail medical image setting since medical segmentation should be robust to different tissue types which show different anatomical variations. As shown in Figure 4, we hence construct a random image transformation T and define the equivariance loss on both labeled and unlabeled data by measuring the feature consistency distance between each original segmentation map and the segmentation map generated from the transformed image:    ).
Here we define T on both the input image x and F θ (x), via the random transformations (i.e., affine, intensity, and photo-metric augmentations), since the model should learn to be robust and invariant to these transformations. Diversity. Oversampling too many images from the random set would create extra memory overhead, and more importantly, our finding also uncovers that a large number of random images might not necessarily help impose additional invariances between neighboring samples since redundant images might introduce additional noise during training (see the Appendix D). Therefore, we formulate our insight as an auxiliary loss that regularizes the representations -keeping the anatomical contrastive reconstruction task as the main force. In contrast to sampling from latent space [84], we perform in the output space (see Appendix D for more analysis). In practice, we first search for Knearest neighbors from the first-in-first-out (FIFO) memory bank [38], and then use the nearest neighbor loss L nn in a way that maximizing cosine similarity, to exploit the interinstance relationship.
Setup. The total loss L total is the sum of contrastive loss L contrast (on both ground-truth labels and pseudo-labels), equivariance loss L eqv (on both ground-truth labels and pseudo-labels), nearest neighbors loss L nn (on both groundtruth labels and pseudo-labels), unsupervised cross-entropy loss L unsup (on pseudo-labels) and supervised segmentation loss L sup (on ground-truth labels): L sup + λ 1 L contrast + λ 2 L eqv + λ 3 L unsup + λ 4 L nn . See the Appendix D for an ablation study of hyperparameters.

Experiments
In this section, we evaluate our proposed MONA on three popular medical image segmentation datasets under varying labeled ratio settings: the ACDC dataset [5], the LiTS dataset [6], and the MMWHS dataset [114] (See Ap-

Main Results
We show the effectiveness of our method under three different label ratios (i.e., 1%, 5%, 10%). We also compare MONA with various state-of-the-art SSL and fullysupervised methods on three datasets: ACDC [5], LiTS [6], MMWHS [114]. We choose 2D UNet [72] as backbone, and compare against SSL methods including UNet trained with full/limited supervisions (UNet-F/UNet-L), EM [87], CCT [67], DAN [107], URPC [59], DCT [70], ICT [86], MT [80], UAMT [104], CPS [20], SCS [42], GCL [13], and PLC [14]. We report quantitative comparisons on ACDC and LiTS in Table 1, and average all our results over three runs. (More results on MMWHS in Appendix B). ACDC. We benchmark performances on ACDC with respect to different amounts of labeled ratios (i.e., 1%, 5%, 10%). The following observations can be drawn: First, our proposed MONA significantly outperforms all other SSL methods under three different label ratios. Especially, with only extremely limited labeled data available (e.g., 1%), our method obtains massive gains of 82.6% and 2.03 in Dice and ASD (i.e., dramatically improving the performance from 59.4% to 82.6%). Second, our method achieves consistently improved performance, and performs better or on par with the fully-supervised approach under all three different label ratios. In particular, MONA with limited labeled training data available (e.g., 10%) -for the first time -surpasses the fully supervised counterparts. For example, 3 The PyTorch-like pseudo-code of our method can be found in the supplementary material. Codes will be available upon the publication. the best Dice score on ACDC rises from 89.9% to 90.7%. Third, as shown in Figure 5, we can see the clear advantage of MONA, where the anatomical boundaries of different tissues are clearly more pronounced such as RV and Myo regions. As seen, our method is capable of producing consistently sharp and accurate object boundaries across various challenge scenarios. LiTS. We then evaluate MONA on LiTS, using 1%, 5%, 10% labeled ratios. The results are summarized in Table 1 and Figure 6. The conclusions we can draw are highly consistent with the above ACDC case: First, at the different label ratios (i.e., 1%, 5%, 10%), MONA consistently outperforms all the other SSL methods, which again demonstrates the effectiveness of learning representations for the inter-class correlations and intra-class invariances under imbalanced class-distribution scenarios. In particular, our MONA, trained on a 1% labeled ratio (i.e., extremely limited labels), dramatically improves the previous best averaged Dice score from 59.3% to 64.1% by a large margin, and even performs on par with previous SSL methods using 10% labeled ratio. Second, the most impressive results come from MONA at 10% label ratio. To the best of our knowledge, this is the first time in the literature that SSL schemes trained at 10% label ratio outperform the fullysupervised model by 1.1% improvements in Dice (i.e., from 68.2% to 69.3%). Third, as shown in Figure 6, we observe that MONA is able to produce more accurate results compared to the previous best schemes.
Overall, we conclude that MONA provides robust performance on all the medical datasets we evaluated, exceeding that of the fully-supervised baseline, and outperforming all other SSL methods.

Ablation Study
In this subsection, we conduct comprehensive analyses to understand the inner workings of MONA on ACDC under 5% labeled ratio. Note that for reproducibility, we report the average performance of three independent runs with  (3) diversity. The results are in Table  2. As is shown, each key component makes a clear difference and leveraging all of them contributes to the remarkable performance improvements. This suggests the importance of learning meaningful representations for the interclass correlations and intra-class invariances within the entire dataset. The intuitions behind each concept are as follows: (1) Only tailness: many anatomy-rich head classes would be sampled; (2) Only consistency: it would lead to object collapsing due to the different anatomical variations; (3) Only diversity: oversampling too many negative samples often comes at the cost of performance degradation. By combining tailness, consistency, and diversity, our method confers a significant advantage at representation learning in imbalanced feature similarity, semantic consistency and anatomical diversity, which further highlights the superiority of our proposed MONA (More results in Appendix C). Effects of Different Augmentations. In addition to further improving the quality and stability in anatomical representation learning, we claim that MONA also gains robustness using augmentation strategies. For augmentation strategies, previous works [79,81,110] show that composing the weak augmentation strategy for the "pivot-to-target" model (i.e., trained with limited labeled data and a large number of unlabeled data) is helpful for anatomical representation learning since the standard contrastive strategy is too aggressive, intuitively leading to a "hard" task (i.e., introducing too many disturbances and yielding model collapses). Here we examine whether and how applying different data augmentations helps MONA. In this work, we implement the weak augmentation to the teacher's input as random rotation, random cropping, horizontal flipping, and strong augmentation to the student's input as random rotation, random cropping, horizontal flipping, random contrast, CutMix [30], brightness changes [69], morphological changes (diffeomorphic deformations). We summarize the results in Table 3, and list the following observations: (1) weak augmentations benefits more: composing the weak augmentation for the teacher model and strong augmentation for the student model significantly boosts the performance across two benchmark datasets. (2) same augmentation pairs do not make more gains: interestingly, applying same type of augmentation pairs does not lead to the best performance compared to different types of augmentation pairs. We postulate that composing different augmentations can be considered as a harder albeit more useful strategy for anatomical representation learning, making feature more generalizable. Extra Study. More investigations about (1) generalization across CL frameworks; (2) ablation on different training settings are in Appendix C and D.

Conclusion
In this paper, we have presented MONA, a semisupervised contrastive learning method for 2D medical image segmentation. We start from the observations that medical image data always exhibit a long-tail class distribution, and the same anatomical objects (i.e., liver regions for two people) are more similar to each other than different objects (e.g.liver and tumor regions). We further expand upon this idea by introducing anatomical contrastive formulation, as well as equivariance and invariances constraints. An extensive empirical study shows that we can formulate a generic set of perspectives that allows us to learn meaningful representations across different anatomical features, which can dramatically improve the segmentation quality and alleviate the training memory bottleneck. Extensive experiments on three datasets demonstrate the state-of-the-art performance of our proposed framework in the long-tailed medical data regimes with extremely limited labels. We believe our results contribute to a better understanding of medical image segmentation and point to new avenues for long-tailed medical image data in realistic clinical applications. The ACDC dataset was hosted in MICCAI 2017 ACDC challenge [5], which includes 200 3D cardiac cine MRI scans with expert annotations for three classes (i.e., left ventricle (LV), myocardium (Myo), and right ventricle (RV)). Following [58,88], we use 140, 20 and 40 3D scans for training, validation, and testing. Among the training scans including 1312 training slices, we follow the similar setting in [58,88] to define different label ratios, i.e., 1% (13 slices), 5% (65 slices), 10% (131 slices). For pre-processing, we adopt the similar setting in [13] by normalizing the intensity of each 3D scan (i.e., using min-max normalization) into [0, 1], and re-sampling all 2D scans and the corresponding segmentation maps into a fixed spatial resolution of 256 × 256 pixels.
The LiTS dataset was hosted in MICCAI 2017 Liver Tumor Segmentation Challenge [6], which includes 131 contrast-enhanced 3D abdominal CT volumes with expert annotations for two classes (i.e., liver and tumor). Following [75,76], we use the first 100 and the rest 31 3D scans for training, and testing. Among the training scans including 44102 training slices, we follow the similar setting in [75,76] to define different label ratios, i.e., 1% (441 slices), 5% (2205 slices), 10% (4410 slices). For pre-processing, we adopt the similar setting in [52] by truncating the intensity of each 3D scan into [−200, 250] HU for removing irrelevant and redundant details, normalizing each 3D scan into [0, 1], and re-sampling all 2D scans and the corresponding segmentation maps into a fixed spatial resolution of 256 × 256 pixels.

Representation Head
Ground-truth Labels Generated Labels G GT Figure 7. Representation head architecture.

A.2. Implementation Details.
We implement all the evaluated models using PyTorch library [68]. All the models are trained using Stochastic Gradient Descent (SGD) (i.e., initial learning rate = 0.01, momentum = 0.9, weight decay = 0.0001) with batch size of 6, and the initial learning rate is divided by 10 every 2500 iterations. All of our experiments are conducted on NVIDIA GeForce RTX 3090 GPUs. We first train our model with 100 epochs during the pre-training, and then retrain the model for 200 epochs during the fine-tuning. We set the temperature τ ξ , τ θ , τ as 0.01, 0.1, 0.5. The size of the memory bank is 36. During the pre-training, we follow the settings of ISD, including global projection head setting, and predictors with the 512-dimensional output embedding, and adopt the setting of local projection head in [42]. More specifically, given the predicted logits y ∈ R C×H×W , we create 36 different views (i.e., random crops at the same location) of y and y ′ with the fixed size 64 × 64, and then project all pixels into 512-dimensional output embedding space, and the output feature dimension of h ′ θ is also 512. An illustration of our representation head is presented in Figure 7. We then actively sample 256 query embeddings and 512 key embeddings for each mini-batch, and the con-  fidence threshold δ θ is set to 0.97. When fine-tuning we use an equally sized pool of candidates K = 5, as well as λ 1 = 0.01, λ 2 = 1.0, λ 3 = 1.0, and λ 4 = 1.0. For different augmentation strategies, we implement the weak augmentation to the teacher's input as random rotation, random cropping, horizontal flipping, and strong augmentation to the student's input as random rotation, random cropping, horizontal flipping, random contrast, CutMix [30], brightness changes [69], morphological changes (diffeomorphic deformations). We adopt two popular evaluation metrics: Dice coefficient (DSC) and Average Symmetric Surface Distance (ASD) for 3D segmentation results. Of note, the projection heads, the predictor, and the representation head are only used in training, and will be discarded during inference.

B. More Experiments Results -MMWHS
Lastly, we validate MONA on MMWHS, under 1%, 5%, 10% labeled ratios. The results are provided in Table 4 and Figure 8. Again, we found that MONA consistently outperforms all other SSL methods with a significant performance margin, and achieves the highest accuracy among all the SSL and fully supervised approaches under three labeled ratios. As is shown, MONA trained at the 1% labeled ratio significantly outperforms all other methods trained at the 1% labeled ratio, even over the 5% labeled ratio. Concretely, MONA trained at only 1% labeled ratio outperforms Table 5. Ablation study of different contrastive learning frameworks on ACDC under three labeled ratio settings (1%, 5%, 10%). We compare two settings: with or without fine-tuning on the segmentation performance (DSC[%]/ASD[mm]). We denote 'without fine-tuning" to only pretaining. On all three labeled settings, our methods (i.e., GLCon and MONA) significantly outperform all the state-of-the-art methods by a significant margin. All the experiments are run with three different random seeds. The best results are in bold. the second-best method (i.e., GLCon) both at the 1% and 5% labeled, yielding 12.3% and 0.4% gains in Dice. We also observe the similar patterns that, MONA performs better or on par with all the other methods at 10% labeled. Particularly, MONA trained at both 5% and 10% labeled ratio surpasses the fully-supervised scheme by 0.6% and 1.8% improvements in Dice, which again demonstrates the superiority of MONA in extremely limited labeled data regimes.

C. Generalization Study of Contrastive Learning Pre-training
As discussed in Section 3.1, our motivation comes from the observation that there are only very limited labeled data and a large amount of unlabeled data in real-world clinical practice. As the fully-supervised methods generally outperform all other SSL methods by clear margins, we postulate that leveraging massive unlabeled data usually introduces additional noise during training, leading to degraded segmentation quality. To address this challenge, "contrastive learning" is a straightforward way to leverage existing unlabeled data in the learning procedure. As supported in Section 4 and Appendix B, our findings have shown that MONA generalizes well across different benchmark datasets (i.e., ACDC, LiTS, MMWHS) with diverse labeled settings (i.e., 1%, 5%, 10%). In the following subsection, we further demonstrate that our proposed principles (i.e., tailness, consistency, diversity) are beneficial to various state-of-the-art CL-based frameworks (i.e., MoCov2 [19], kNN-MoCo [84], SimCLR [18], BYOL [35], and ISD [81]) with different label settings. More details about these three principles can be found in Section 3.2. Of note, to the best of our knowledge, MONA is the first SSL training scheme that consistently outperforms the fully-supervised method on diverse benchmark datasets with only 10% labeled ratio.
Training Details of Competing CL Methods. We identically follow the default setting in each CL framework [18,19,35,81,84] except the epochs number. We train each model in the semi-supervised setting. For labeled data, we follow the same training strategy in Section 3.1. As for unlabeled data, we strictly follow the default settings in each baseline. Specifically, for fair comparisons, we pre-train each CL baseline and our proposed CL pre-trained method (i.e., GLCon) for 100 epochs in all our experiments. Then we fine-tune each CL model with our proposed principles with the same setting, as provided in Appendix A. For kNN-MoCo [84], given the following ablation study we set the number of neighbors k as 5, and further compare different settings of k in kNN-MoCo [84] in the following subsection. All the experiments are run with three different random seeds.
Comparisons with CL-based Frameworks. Table 5 presents the comparisons between our proposed methods (i.e., GLCon and MONA) and various CL baselines. After analyzing these extensive results, we can draw several consistent observations. First, we can observe that our proposed GLCon achieves performance gains under all the labeled ratios, which not only demonstrates the effectiveness of our method, but also further verifies this argument using "global-local" strategy [13]. The average improvement in Dice obtained by GLCon could reach up to 2.53%, compared to the second best scores at different labeled ratios. Second, we can find that incorporating our proposed three principles significantly outperforms the CL baselines without fine-tuning, across all frameworks and different labeled ratios. These experimental findings suggest that our pro- Table 6. Ablation study of different principles across different contrastive learning frameworks under various labeled ratio settings (1%, 5%, 10%). Experiments are conducted on ACDC using UNet [72] as the backbone with three independent runs. Here we report the segmentation performance in terms of DSC[%] and ASD [mm]. On all three labeled settings, incorporating our methods (i.e., tailness, consistency, and diversity) consistently achieve superior model robustness gains across different state-of-the-art CL frameworks.  (c) Figure 9. Effects of k-nearest neighbour in global feature space, mined view-set size, and mined view patch size. We report Dice and ASD of GLCon on the ACDC dataset at the 5% labeled ratio. All the experiments are run with three different random seeds.
formance of MoCov2, kNN-MoCo, SimCLR, BYOL, ISD, and our GLCon by 39.1%, 38.5%, 40.9%, 41.2%, 34.3%, 33.3%, respectively. The ACDC dataset is a popular multiclass medical image segmentation dataset, with massive imbalanced or long-tailed class distribution cases. The imbalanced or long-tailed class distribution gap could result in the vanilla models overfitting to the head class, and generalizing very poorly to the tail class. With the addition of under-sampling the head classes, the principle -tailness -can be deemed as the prominent strategy to yield better generalization and segmentation performance of the models across different labeled ratios. Similar results are found under 5% and 10% labeled ratios. Third, over a wide range of labeled ratios, MONA can establish the new state-of-theart performance bar for semi-supervised 2D medical image segmentation. Particularly, MONA -for the first timeboots the segmentation performance with 10% labeled ratio over the fully-supervised method while significantly outperforming all the other semi-supervised methods by a large margin. In summary, our proposed methods (i.e., GLCon and MONA) obtain remarkable performance on all labeled settings. The results verify the superiority of our proposed three principles (i.e., tailness, consistency, diversity) jointly, which makes the model well generalize to different labeled settings, and can be easily and seamlessly plugged into all other CL frameworks [18,19,35,81,84] adopting the twobranch design, demonstrating that these concepts consis-tently help the model yield extra performance boosts for them all.

Generalization Across CL Frameworks
As demonstrated in Table 6, incorporating tailness, consistency, and diversity have obviously superior performance boosts, which is aligned with consistent observations with Section 4.2 can be drawn. This suggests that these three principles can serve as desirable properties for medical image segmentation in both supervised and unsupervised settings.
Does k-nearest neighbour in global feature space help? Prior work suggests that the use of stronger augmentations and nearest neighbour can be the very effective tools in learning additional invariances [84]. That is, both the specific number of nearest neighbours and specific augmentation strategies are necessary to achieve superior performance. In this subsection, we study the relationship of k-nearest neighbour in global feature space and the behavior of our GLCon for the downstream medical image segmentation. Here we first follow the same augmentation strategies in [84] (More analysis on data augmentation can be found in Section 4.2), and then conduct ablation studies on how the choices of k-nearest neighbour can influence the performance of GLCon. Specifically, we run GLCon on the ACDC dataset at the 5% labeled ratio with a range of k ∈ {3, 5, 7, 10, 12}. Figure 9(a) shows the ablation study on k-nearest neighbour in global feature on the segmentation performance. As is shown, we find that GLCon Figure 10. Effects of hyperparameters λ1, λ2, λ3, λ4. We report Dice and ASD of MONA on the ACDC dataset at the 5% labeled ratio.
All the experiments are run with three different random seeds.
at k = 5, 7, 10 have almost identical performance (k = 5 has slightly better performance compared to other two settings), and all have superior performance compared to all others. In contrast, GLCon -through the use of randomly selected samples -is capable of finding diverse yet semantically consistent anatomical features from the entire dataset, which at the same time gives better segmentation performance. Ablation Study of Mined View-Set Size.
We then conduct ablation studies on how the mined view-set size in GLCon can influence the segmentation performance. We run GLCon on the ACDC dataset at 5% labeled ratio with a range of the mined view-set size ∈ {12, 18, 24, 30, 36, 42, 48}. The results are summarized in Figure 9(b). As is shown, we find that GLCon trained with view-set size 36 and 42 have similar or superior performance compared to all other settings, and our model with view-set size of 36 achieves the highest performance. Ablation Study of Mined View Size. Lastly, we study the influence of mined view size on the segmentation performance. Specifically, we run GLCon on the ACDC dataset at the 5% labeled ratio with a range of the mined view size ∈ {8, 16, 32, 64, 128}. Figure 9(c) shows the ablation study of mined view size on the segmentation performance. As is shown, we observe that GLCon trained with mined view size of 32 and 64 have similar segmentation abilities, and both achieve superior performance compared to other settings. Here the mined view size of 64 works the best for GLCon to yield the superior segmentation performance. Conclusion. Given the above ablation study, we set k, mined view-set size, patch size as 5, 36, 64 × 64 in our experiments, respectively. This can contribute to satisfactory segmentation performance.

D. Ablation Study of Anatomical Contrastive Reconstruction
In this section, we give a detailed analysis on the choice of the parameters in the anatomical contrastive reconstruction fine-tuning, and take a deeper look and understand how they contribute to the final segmentation performance. All the hyperparameters in training are the same across three benchmark datasets. All the experiments are run with three different random seeds. Ablation Study of Total Loss L total . Proper choices of hyperparameters in total loss L total (See Section 3.2) play a significant role in improving overall segmentation quality. We hence conduct the fine-grained analysis of the hyperparameters in L total . In practice, we fine-tune the models with three independent runs, and grid search to select multiple hyperparameters. Specifically, we run MONA on the ACDC dataset at the 5% labeled ratio with a range of different hyperparameters λ 1 ∈ {0.005, 0.001, 0.05, 0.01, 0.05, 0.1}, and λ 2 , λ 3 , λ 4 ∈ {0.1, 0.2, 0.5, 1.0, 2.0, 10.0}. We summarize the results in Figure 10, and take the best setting λ 1 = 0.01, λ 2 = 1.0, λ 3 = 1.0, λ 4 = 1.0.  Figure 11. Effects of confidence threshold δ θ , K-nearest neighbour constraint, and output embedding dimension. We report Dice and ASD of MONA on the ACDC dataset at the 5% labeled ratio. All the experiments are run with three different random seeds.
Ablation Study of Confidence Threshold δ θ . We then assess the influence of δ θ on the segmentation performance. Specifically, we run MONA on the ACDC dataset at the 5% labeled ratio with a range of the confidence threshold δ θ ∈ {0.85, 0.88, 0.91, 0.94, 0.97, 1.0}. Figure 11(a) shows the ablation study of δ θ on the segmentation performance. As we can see, MONA on δ θ = 0.97 has superior performance compared to other settings.
Ablation Study of K-Nearest Neighbour Constraint.
Next, we conduct ablation studies on how the choices of K in K-nearest neighbour constraint can influence the segmentation performance. Specifically, we run MONA on the ACDC dataset at the 5% labeled ratio with a range of the choices K ∈ {3, 5, 7, 10, 12}. Figure 11(b) shows the ablation study of K choices on the segmentation performance. As we can see, MONA on K = 5 achieves the best performance compared to other settings.
Ablation Study of Output Embedding Dimension. Finally, we study the influence of the output embedding dimension on the segmentation performance of MONA. Specifically, we run MONA on the ACDC dataset at the 5% labeled ratio with a range of output embedding dimension ∈ {64, 128, 256, 512, 768}. Figure 11(c) shows the ablation study of output embedding dimension on the segmentation performance. As we can see, MONA with output embedding dimension of 512, can be trained to outperform other settings. Conclusion. Given the above ablation study, we select λ 1 = 0.01, λ 2 = 1.0, λ 3 = 1.0, λ 4 = 1.0, δ θ = 0.97, K = 5, output embedding dimension = 512 in our experiments. This can provide the optimal segmentation performance across different labeled ratios.