Adapting Off-the-Shelf Source Segmenter for Target Medical Image Segmentation

Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to an unlabeled and unseen target domain, which is usually trained on data from both domains. Access to the source domain data at the adaptation stage, however, is often limited, due to data storage or privacy issues. To alleviate this, in this work, we target source free UDA for segmentation, and propose to adapt an ``off-the-shelf"segmentation model pre-trained in the source domain to the target domain, with an adaptive batch-wise normalization statistics adaptation framework. Specifically, the domain-specific low-order batch statistics, i.e., mean and variance, are gradually adapted with an exponential momentum decay scheme, while the consistency of domain shareable high-order batch statistics, i.e., scaling and shifting parameters, is explicitly enforced by our optimization objective. The transferability of each channel is adaptively measured first from which to balance the contribution of each channel. Moreover, the proposed source free UDA framework is orthogonal to unsupervised learning methods, e.g., self-entropy minimization, which can thus be simply added on top of our framework. Extensive experiments on the BraTS 2018 database show that our source free UDA framework outperformed existing source-relaxed UDA methods for the cross-subtype UDA segmentation task and yielded comparable results for the cross-modality UDA segmentation task, compared with a supervised UDA methods with the source data.


Introduction
Accurate tumor segmentation is a critical step for early tumor detection and intervention, and has been significantly improved with advanced deep neural networks (DNN) [25,18,10,9,17]. A segmentation model trained in a source domain, however, usually cannot generalize well in a target domain, e.g., data acquired from a new scanner or different clinical center, in implementation. Besides, annotating data in the new target domain is costly and even infeasible [11]. To address this, unsupervised domain adaptation (UDA) was proposed to transfer knowledge from a labeled source domain to unlabeled target domains [13].
The typical UDA solutions can be classified into three categories: statistic moment matching, feature/pixel-level adversarial learning [15,14,12], and selftraining [33,16]. These UDA methods assume that the source domain data are available and usually trained together with target data. The source data, however, are often inaccessible, due to data storage or privacy issues, for cross-clinical center implementation [1]. Therefore, it is of great importance to apply an "offthe-shelf" source domain model, without access to the source data. For sourcefree classification UDA, Liang et al. [8] proposed to enforce the diverse predictions, while the diversity of neighboring pixels is not suited for the segmentation purpose. In addition, the class prototype [13] and variational inference methods [11] are not scalable for pixel-wise classification based segmentation. More importantly, without distribution alignment, these methods relied on unreliable noisy pseudo labeling.
Recently, the source relaxed UDA [1] was presented to pre-train an additional class ratio predictor in the source domain, by assuming that the class ratio, i.e., pixel proportion in segmentation, is invariant between source and target domains. At the adaptation stage, the class ratio was used as the only transferable knowledge. However, that work [1] has two limitations. First, the class ratio can be different between the two domains, due to label shift [11,13]. For example, a disease incident rate could vary between different countries, and tumor size could vary between different subtypes and populations. Second, the pre-trained class ratio predictor used in [1] is not typical for medical image segmentation, thereby requiring an additional training step using the data in the source domain.
In this work, to address the aforementioned limitations, we propose a practical UDA framework aimed at the source-free UDA for segmentation, without an additional network trained in the source domain or the unrealistic assumption of class ratio consistency between source and target domains. More specifically, our framework hinges on the batch-wise normalization statistics, which are easy to access and compute. Batch Normalization (BN) [6] has been a default setting in the most of modern DNNs, e.g., ResNet [5] and U-Net [30], for faster and more stable training. Notably, the BN statistics of the source domain are stored in the model itself. The low-order batch statistics, e.g., mean and variance, are domain-specific, due to the discrepancy of input data. To gradually adapt the low-order batch statistics from the source domain to the target domain, we develop a momentum-based progression scheme, where the momentum follows an exponential decay w.r.t. the adaptation iteration. For the domain shareable highorder batch statistics, e.g., scaling and shifting parameters, a high-order batch statistics consistent loss is applied to explicitly enforce the discrepancy minimization. The transferability of each channel is adaptively measured first, from which to balance the contribution of each channel. Moreover, the proposed unsupervised self-entropy minimization can be simply added on top of our framework to boost the performance further.
Our contributions are summarized as follows: • To our knowledge, this is the first source relaxed or source free UDA framework for segmentation. We do not need an additional source domain network, or the Fig. 1: Comparison of (a) conventional UDA [28] and (b) our source-relaxed OS-UDA segmentation framework based on the pre-trained "off-the-shelf" model with BN. We minimize the domain discrepancy based on the adaptively computed batch-wise statistics in each channel. The model consists of a feature encoder (Enc) and a segmentor (Seg) akin to [3,32].
unrealistic assumption of the class ratio consistency [1]. Our method only relies on an "off-the-shelf" pre-trained segmentation model with BN in the source domain.
• The domain-specific and shareable batch-wise statistics are explored via the low-order statistics progression with an exponential momentum decay scheme and transferability adaptive high-order statistics consistency loss, respectively.
• Comprehensive evaluations on both cross-subtype (i.e., HGG to LGG) and cross-modality (i.e., T2 to T1/T1ce/FLAIR) UDA tasks using the BraTS 2018 database demonstrate the validity of our proposed framework and its superiority to conventional source-relaxed/source-based UDA methods.

Methodology
We assume that a segmentation model with BN is pre-trained with source domain data, and the batch statistics are inherently stored in the model itself. At the adaptation stage, we fine-tune the model based on the batch-wise statistics and the self-entropy (SE) of target data prediction. The overview of the different setups of conventional UDA and our "off-the-shelf (OS)" UDA is shown in Fig. 1. Below, we briefly revisit the BN in Subsec. 2.1 first and then introduce our OSUDA in Subsec. 2.2. The added unsupervised SE minimization and the overall training protocol are detailed in Subsec. 2.3.

Preliminaries on Batch Normalization
As a default setting in the most of modern DNNs, e.g., ResNet [5] and U-Net [30], Batch Normalization (BN) [6] normalizes the input feature in the lth layer f l ∈ R B×H l ×W l ×C l within a batch in a channel-wise manner to have zero mean and unit variance. B denotes the number of images in a batch, and H l , W l , and C l are the height, width, and channels of layer l. We have samples in a batch, with index b ∈ {1, · · · , B}, spatial index n ∈ {1, · · · , H l × W l }, and channel index c ∈ {1, · · · , C l }. BN calculates the mean of each channel Then, the input feature is normalized as where ∈ R + is a small scalar for numerical stability. γ l,c and β l,c are learnable scaling and shifting parameters, respectively. In testing, the input is usually a single sample rather than a batch with B samples. Therefore, BN stores the exponentially weighted average of the batch statistics at the training stage and used it in testing. Specifically, the mean and variance over the training are tracked progressively, i.e., where η ∈ [0, 1] is a momentum parameter. After K training iterations, µ K l,c , {σ 2 } K l,c , γ K l,c , and β K l,c are stored and used for testing normalization [6].

Adaptive source-relaxed batch-wise statistics adaptation
Early attempts of BN for UDA simply added BN in the target domain, without the interaction with the source domain [7]. Recent studies [2,20,26,19] indicated that the low-order batch statistics, i.e., mean µ l,c and variance {σ 2 } l,c , are domain-specific, because of the divergence of cross-domain representation distributions. Therefore, brute-forcing the same mean and variance across domains can lead to a loss of expressiveness [29]. In contrast, after the low-order batch statistics discrepancy is partially reduced, with domain-specific mean and variance normalization, the high-order batch statistics, i.e., scaling and shifting parameters γ l,c and β l,c , are shareable across domains [20,26]. However, all of the aforementioned methods [2,20,29,26,19] require the source data at the adaptation stage. To address this, in this work, we propose to mitigate the domain shift via the adaptive low-order batch statistics progression with momentum, and explicitly enforce the consistency of the high-order statistics in a source-relaxed manner. Low-order statistics progression with an exponential momentum decay scheme. In order to gradually learn the target domain-specific mean and variance, we propose an exponential low-order batch statistics decay scheme. We initialize the mean and variance in the target domain with the tracked µ K l,c and {σ 2 } K l,c in the source domain, which is similar to applying a model with BN in testing [6]. Then, we progressively update the mean and variance in the t-th adaptation iteration in the target domain as where η t = η 0 exp(−t) is a target adaptation momentum parameter with an exponential decay w.r.t. the iteration t. µ t l,c and {σ 2 } t l,c are the mean and variance of the current target batch. Therefore, the weight of µ K l,c and {σ 2 } K l,c are smoothly decreased along with the target domain adaptation, while µ t l,c and {σ 2 } t l,c gradually represent the batch-wise low-order statistics of the target data. Transferability adaptive high-order statistics consistency. For the highorder batch statistics, i.e., the learned scaling and shifting parameters, we explicitly encourage its consistency between the two domains with the following high-order batch statistics (HBS) loss: where γ K l,c and β K l,c are the learned scaling and shifting parameters in the last iteration of pre-training in the source domain. γ t l,c and β t l,c are the learned scaling and shifting parameters in the t-th adaptation iteration. α l,c is an adaptive parameter to balance between the channels.
We note that the domain divergence can be different among different layers and channels, and the channels with smaller divergence can be more transferable [22]. Accordingly, we would expect that the channels with higher transferability contribute more to the adaptation. In order to quantify the domain discrepancy in each channel, a possible solution is to measure the difference between batch statistics. In the source-relaxed UDA setting, we define the channel-wise sourcetarget distance in the t-th adaptation iteration as Then, the transferability of each channel can be measured by α l,c = L×C×(1+d l,c ) −1 l c (1+d l,c ) −1 . Therefore, the more transferable channels will be assigned with higher importance, i.e., with larger weight (1 + α l,c ) in L l,c .

Self-entropy minimization and overall training protocol
The training in the unlabeled target domain can also be guided by an unsupervised learning framework. The SE minimization is a widely used objective in modern DNNs to encourage the confident prediction, i.e., the maximum softmax value can be high [4,8,24,1]. SE for pixel segmentation is calculated by the averaged entropy of the classifier's softmax prediction given by where H 0 and W 0 are the height and width of the input, and δ b,n is the histogram distribution of the softmax output of the n-th pixel of the b-th image in a batch. Minimizing L SE leads to the output close to a one-hot distribution. At the source-domain pre-training stage, we follow the standard segmentation network training protocol. At the target domain adaptation stage, the overall training objective can be formulated as L = L HBS + λL SE , where λ is used to balance between the BN statistics matching and SE minimization. We note that a trivial solution of SE minimization is that all unlabeled target data could have the same one-hot encoding [4]. Thus, to stabilize the training, we linearly change the hyper-parameter λ from 10 to 0 in training.
For evaluation, we adopted the widely used Dice similarity coefficient and Hausdorff distance metrics as in [32]. The Dice similarity coefficient (the higher, the better) measures the overlapping part between our prediction results and the ground truth. The Hausdorff distance (the lower, the better) is defined between two sets of points in the metric space.

Cross-subtype HGG to LGG UDA
HGG and LGG have different size and position distributions for tumor regions [23]. Following the standard protocol, we used the HGG training set (source Fig. 2: The comparison with the other UDA methods, and an ablation study of adaptive channel-wise weighting and SE minimization for HGG to LGG UDA. domain) to pre-train the segmentation model and adapted it with the LGG training set (target domain) [23]. The evaluation was implemented in the LGG testing set. We adopted the same 2D U-Net backbone in [23], sliced 3D volumes into 2D axial slices with the size of 128×128, and concatenated all four MRI modalities to get a 4-channel input.
The quantitative evaluation results are shown in Table 1. Since the pixel proportion of each class is different between HGG and LGG domains, the class ratio-based CRUDA [1] only achieved marginal improvements with its unsupervised learning objective. We note that the Dice score of the core tumor was worse than the pre-trained source-only model, which can be the case of negative transfer [27]. Our proposed OSUDA achieved the state-of-the-art performance for source-relaxed UDA segmentation, approaching the performance of SEAT [23] with the source data, which can be seen as an "upper-bound. " We used OSUDA-AC and OSUDA-SE to indicate the OSUDA without the adaptive channel-wise weighting and self-entropy minimization, respectively. The better performance of OSUDA over OSUDA-AC and OSUDA-SE demonstrates the effectiveness of adaptive channel-wise weighting and self-entropy minimization. The illustration of the segmentation results is given in Fig. 2. We can see that the predictions of our proposed OSUDA are better than the no adaptation model. In addition, CRUDA [1] had a tendency to predict a larger area for the tumor; and the tumor core is often predicted for the slices without the core.
3.2 Cross-modality T2 to T1/T1ce/FLAIR UDA Because of large appearance discrepancies between different MRI modalities, we further applied our framework to the cross-modality UDA task. Since clinical annotation of the whole tumor is typically performed on T2-weighted MRI, the typical cross-modality UDA setting is to use T2-weighted MRI as the labeled source domain, and T1/T1ce/FLAIR MRI as the unlabeled target domains [32]. We followed the UDA training (80% subjects) and testing (20% subjects) split as in [32], and adopted the same single-channel input backbone. We note that the data were used in an unpaired manner [32]. The quantitative evaluation results are provided in Table 2. Our proposed OSUDA outperformed CRUDA [1] consistently. In addition, in CRUDA, the additional class ratio prediction model was required to be trained with the source data, which is prohibitive in many real-world cases. Furthermore, our OSUDA outperformed several UDA methods trained with the source data, e.g., Cycle-GAN [31] and SIFA [3], for the two metrics. The visual segmentation results of three target modalities are shown in Fig. 3, showing the superior performance of our framework, compared with the comparison methods.

Discussion and Conclusion
This work presented a practical UDA framework for the tumor segmentation task in the absence of the source domain data, only relying on the "off-the-shelf" pre-trained segmentation model with BN in the source domain. We proposed a low-order statistics progression with an exponential momentum decay scheme to gradually learn the target domain-specific mean and variance. The domain shareable high-order statistics consistency is enforced with our HBS loss, which is adaptively weighted based on the channel-wise transferability. The performance was further boosted with the unsupervised learning objective via self-entropy minimization. Our experimental results on the cross-subtype and cross-modality UDA tasks demonstrated that the proposed framework outperformed the comparison methods, and was robust to the class ratio shift.