Deformable image registration with deep network priors: a study on longitudinal PET images

Longitudinal image registration is challenging and has not yet benefited from major performance improvements thanks to deep-learning. Inspired by Deep Image Prior, this paper introduces a different use of deep architectures as regularizers to tackle the image registration question. We propose a subject-specific deformable registration method called MIRRBA, relying on a deep pyramidal architecture to be the prior parametric model constraining the deformation field. Diverging from the supervised learning paradigm, MIRRBA does not require a learning database, but only the pair of images to be registered to optimize the network's parameters and provide a deformation field. We demonstrate the regularizing power of deep architectures and present new elements to understand the role of the architecture in deep learning methods for registration. Hence, to study the impact of the network parameters, we ran our method with different architectural configurations on a private dataset of 110 metastatic breast cancer full-body PET images with manual segmentations of the brain, bladder and metastatic lesions. We compared it against conventional iterative registration approaches and supervised deep learning-based models. Global and local registration accuracies were evaluated using the detection rate and the Dice score respectively, while registration realism was evaluated using the Jacobian's determinant. Moreover, we computed the ability of the different methods to shrink vanishing lesions with the disappearing rate. MIRRBA significantly improves the organ and lesion Dice scores of supervised models. Regarding the disappearing rate, MIRRBA more than doubles the best performing conventional approach SyNCC score. Our work therefore proposes an alternative way to bridge the performance gap between conventional and deep learning-based methods and demonstrates the regularizing power of deep architectures.


Introduction
Medical image registration is the precise overlaying of a fixed and moving image. It is notably used to create patient models (inter-patient monomodal registration), exploit information from different modalities acquired for a single patient (intra-patient multimodal registration), and to monitor tumor evolution (longitudinal registration).
However, the accuracy of image registration is a long-standing fundamental problem in medical image analysis ( [32,16]). In the past 30 years, registration methods have considered registration as an optimization problem: the goal is to minimize a dissimilarity term between moving and fixed images. It is usually measured with mean square error (MSE), mutual information (MI) or normalized cross-correlation (NCC). In addition, a term enforcing smooth and realistic transformations is often used as a regularizing term.
Most conventional methods manage registration by parameterizing the transformation between the fixed and moving image. They use for instance discrete cosine transforms ( [15]), 3D Fourier series ( [9]), cubic B-splines ( [26]) or optimized velocity fields ( [1]). The parameters of the model are typically optimized though efficient second-order minimization ( [44]) or stochastic gradient descent (SGD) ( [25]), within a single-or multi-level optimization strategy. The minimized cost function combines dissimilarity and regularization terms.
Recent deep learning (DL) supervised registration methods ( [16]) also rely on dissimilarity measurements and regularization terms that can be optimized with variants of SGD. However, there are two key differences compared to conventional approaches. DL methods require a training stage with a training database that gives the model prior knowledge regarding deformations. They also use a different type of transformation model, with convolutional neural networks (CNN) used as an over-parameterized but structured model of the deformation field.
Despite recent developments in DL-based registration ( [8]), conventional methods still perform better and obtain more accurate results in most applications ([21]). In addition, even though trained networks are faster than conventional methods, training requires large databases which often restricts their use to a specific therapeutic area.
With this work, we propose a new registration method formalized as a conventional registration approach, with a deformation field modeled by an untrained deep pyramidal network. We named our approach MIRRBA for Medical Image Registration method Regularized By Architecture.
We applied MIRRBA on longitudinal PET images acquired for the evaluation of treatment response in patients with metastatic breast cancer and compared the results to registrations obtained with standard approaches. We also studied the impact of different architecture configurations (14 configurations) on the deformation field.
The contributions of this paper are i) the proposition of a novel registration method regularized by architecture MIRRBA, ii) an extensive comparative study of the effects of different network components on the deformation field, and iii) a solution to register whole body PET images both at a global and local level without the need for prior registration to facilitate the simultaneous monitoring of multiple lesions.

Related work
Automatic longitudinal lesion monitoring Automatic approaches for lesion segmentation and/or image registration have been developed over the years to help with cancer monitoring and treatment response assessments. [36] used affine registration to align baseline and follow-up PET images, before subtracting them to identify tumor voxels showing significant changes between the two scans. However, since affine registration is a global method, it mainly performs well for single localized tumors. In a situation where there were multiple lesions to assess, [22] first segmented all lesions on liver-centered CT images, then used a longitudinal correspondence module to find matching pairs of lesions from the segmentation maps, and finally computed the lesions' evolution. Even if promising, multi-staged methods suffer from error propagation between stages. [7] removed the need for a prior segmentation step and relied on a conventional registration algorithm between longitudinal CT pairs to obtain the deformation fields and their Jacobian determinants. This information was then used to train a classifier network assessing systemic sclerosis interstitial lung disease.
Image registration can be a key step in lesion monitoring. As our long-term goal is to use registration results to develop tools for the monitoring of metastatic breast cancer, this work focuses on registration and on how to improve its accuracy. We present hereafter the most used registration approaches and the good practices which contribute to make them successful.
Established registration tools There is a large body of literature on conventional registration methods ( [27,41]). Some of today's most used registration tools include the Demons method ( [45]) using non-parametric diffeomorphic displacement fields, the Elastix toolbox ( [26]) based on cubic B-splines, and the Advanced Normalization Tools (ANTs) ( [1]) parametrizing the velocity field and relying on bi-directional diffeomorphisms. We compared our proposed approach to the last two methods, as they perform well with different datasets ( [27]) and their pyramidal coarse-to-fine optimization has inspired recent works on DL-based registration which is presented hereafter.
DL methods In recent years, different types of DL-based registration approaches have been proposed ( [8]). With the monitoring of metastatic breast cancer, deformation fields are not available and are difficult to obtain. We therefore focused on unsupervised registration methods. We categorize as unsupervised methods which do not require ground truth deformation fields, segmentations, or landmarks. Typically, they learn by minimizing a dissimilarity term between the fixed and the wrapped moving image. Among the unsupervised registration methods, various CNN architectures have been proposed. The widely used U-Net ( [38]) inspired the reference Voxelmorph network first trained on brain MRIs ( [4]). [42] added dilated convolutions to the encoder path of Voxelmorph to capture a wider field of view from lung MRI images. Later, [46] proposed a U-Net-based cascade network applied to cardiac cine MRI and chest CT data to perform affine and deformable registrations in stages, at the cost of an increase in complexity. On 3D PET images, [29] also proposed an iterative DL-based registration method to reconstruct motion compensated PET images. While still using a U-Net-shaped architecture, [14] proposed a coarse-to-fine training mimicking the best performing conventional approaches. With a similar training strategy, [33] won the Learn2Reg 2021 MICCAI challenge 1 using a pyramidal network named LapIRN. Contrary to the above cited methods, [33] did not only impose a smoothness constraint on the deformation field through its gradient, but also enforced diffeomorphic transformations using stationary vector fields under the Log-Euclidean framework, as in [12].
Despite recent efforts, when using DL-based registration techniques no major gain in registration performance was reported. Conventional iterative optimization methods still yield better accuracy in many tasks such as inter-patient alignment or intra-patient lung motion registration ( [21]). Large databases would be needed to learn the network parameters and produce accurate deformation fields. Moreover, we can argue that the generalization ability of the trained network is questionable when the deformation patterns do not repeat consistently across the dataset.
Deep Image Prior registration To alleviate the dependency to a dataset and the need for repeated deformation patterns, we adapted the Deep Image Prior (DIP) framework ( [43]) to propose a learning-free method for deformable medical image registration. Contrary to standard DL-based approaches, DIP does not learn from a database but relies on a single image. It uses a deep architecture not to summarize the information across samples but as a prior. Effectively, the architecture plays the role of a parametric model in an optimization problem restricting the solution space. While DIP was initially designed for denoising and inpainting tasks ( [43]), we adapted it here to medical image registration. [28] suggested image registration as a potential application of DIP, with preliminary results in the context of 2D brain MR images. DIP has also been used on medical data for the reconstruction of CT and PET images ( [17,3]). To perform these reconstructions, an untrained deep network was used as a denoiser and iteratively optimized by a conventional algorithm. Apart from the use of an initial reconstruction and classical regularization, images from other modalities were also given as input to condition the network output.
With this work, we establish a link between conventional, deep learning and DIP-based approaches. In particular, we focus on the role of the untrained network in parameterizing the displacement field, showing that each architecture induces an implicit regularization when used within a conventional optimization scheme. Similar observations have been made in the context of DL for inverse problems [31] and [13]. As a result, well-structured architectures (e.g. [33]) provide better over-parameterizations for the deformation fields, both in the supervised and in the untrained cases. We also investigate the role of the input (random, moving or moving and fixed images) and potential interactions of the untrained network with conventional supervised approaches.

Image registration
Given a pair of fixed and moving (F , M ) images, deformable image registration aims to estimate a dense deformation field Φ such that the wrapped image W = M (x + Φ(x)) is aligned with F for each voxel x. Given a metric S(., .) measuring the dissimilarity between two images, conventional registration methods choose a parametrization of transformation and optimize it through a cost function of the form: arg min However, the problem stated Eq. 1 is ill-posed: it belongs to the nonlinear transformation class ( [35]). By explicitly constraining the transformation, parametric approaches can become well-posed ( [40]), at the cost of limiting the range of admissible transformations. Constraining an over-parameterized or complex deformable transformation is effectively done by adding a regularization term R smooth (Φ) to the objective function Eq. 1. Regularization terms incorporate priors, which if correctly chosen, guide the optimization towards better estimates of the deformation fields. Weighted by λ smooth , the most common regularization term enforces smoothness onto the displacement field by penalizing the spatial derivatives of Φ.
DL-based registration approaches ( [16]) adapt the cost function in Eq. 2 by modeling Φ to be the output of a CNN trained on a dataset (See Eq. 3). The inputs of this network are the (F , M ) pair, and the warping operation to get W is performed using a spatial transformer layer ( [23]), handling the transformation, the sampling and the interpolation. Regarding the regularization, DLregistration based methods continue relying on a term enforcing the smoothness of the transformation R smooth , as done in conventional approaches. Smoothness is often completed by R dif f eo , which enforces the diffeomorphism of the transformation by penalizing the determinant of the Jacobian negative values. Regularization terms are weighted respectively by λ smooth and λ dif f eo . Moreover, we state that two additional regularizing priors are also implicitly added in the DL setup and influence the predicted deformation field Φ: the first one R dataset is induced by training on a domain-specific dataset, while the second one R archi is entailed by the network architecture choice (see Fig. 1.B). Due to the dataset dependency, one limitation of DL-based registration methods is the difficulty to generalize across organs or modalities.
arg min Next, we recall the deep image prior concept, which removes the dataset dependency in Section 3.2, and present our method for registration based on an untrained network in Section 3.3.

Deep Image Prior
The DIP method proposed in [43] uses a deep architecture to denoise images using a network, but without any prior learning. Supposing X 0 a distorted image and X the network output, the fitting process is characterized by Eq. 4, with R archi the implicit prior captured by the network architecture and S DIP a reconstruction function on a single image.
arg min DIP reconstructs a noisy image (e.g. with JPEG compression noise) from a white noise image by training a generator architecture to fit the noisy image. A denoised image is obtained by stopping the training before completely fitting the noise. We propose to adapt this idea to the registration of a pair of images, where we modify the moving image to match the fixed one. Contrary to the original DIP, we are interested in fitting all the way up to the finest deformations.

MIRRBA (Medical Image Registration method Regularized by Architecture)
In this paper, we argue that deep architectures, as parametric models with high capacity, are powerful representations for deformation fields. They can thus be exploited as implicit priors for image registration in iterative optimization schemes without a training stage ( [19,13,31]). Although DIP priors have been explored in the context of image reconstruction, there is no prior in depth study of such architecture priors in the context of image registration.
Since there is no learning step, the training dataset no longer influences the transformation. Thus, we transform Eq. 3 into Eq. 5 to directly optimize a patient-specific CNN for the pair of interest (F , M ), as traditionally done with iterative optimization methods (see Fig. 1 By optimizing Eq. 5, we find the best warped image allowed by the overparametrization of the architecture. Carefully designing the architecture to be used, it is possible to integrate several of the commonly used tricks in conventional approaches. In this paper, we rely on the LapIRN architecture ( [33]), which incorporates filtering through convolutions, pyramidal coarse-to-fine refinement, and interpolation steps with down-and transpose convolutions. Eq. 5 is optimized with an SGD for an input pair of images. Our idea applies to other architectures and optimization algorithms.

Dataset description
We ran our experiments on images from a private dataset, acquired in the context of the ongoing prospective multicentric EPICURE seinmeta study (NCT03958136) for metastatic breast cancer monitoring([10]) 2 . Patients underwent between two and three PET/CT acquisitions, corresponding to pre-, early-(after a month) and mid-treatment time points. A total number of 110 pairs of images were obtained, a pair being composed of a pre-treatment and either an early-or a mid-treatment image (58 and 52 images respectively). Images were acquired at two different centers. The 54 pairs of images from center A were obtained using a Philips Vereos or a GE Discovery PET/CT imaging systems, while the 56 pairs of images from center B were acquired on two different dual-slice Siemens Biograph PET/CT scanners.
Since we are interested in lesion monitoring in the context of metastatic breast cancer, we worked only on PET images, as shown useful in previous studies [6] and [2]. Expert physicians manually segmented all lesions. The brain and the bladder were also delineated, since they can be useful to mask irrelevant regions for patient response assessment. All PET images were normalized by the standardized uptake value (SUV) ( [24]) and resampled to obtain an isotropic resolution of 1 × 1 × 1 mm 3 with 200 pixels for each side. Besides, no prior registration of any kind was performed.

Architectural implementation details
Our MIRRBA method relies on the LapIRN network architecture proposed in [33]. LapIRN is a pyramidal network with N = 3 depth levels, each level being composed of a feature encoder, a set of residual blocks and a feature decoder, as shown in Fig. 1. For each level L i∈{1,2,3} , input images are downsampled by a factor 0.5 (N −1) using a trilinear interpolation. Hence, for the coarsest level L 1 the image resolution is divided by 4, while for the finest L 3 it remains identical. Moreover, a scaling and squaring module ( [12]) enforces diffeomorphic deformations.
Network levels were trained in a coarse-to-fine manner, meaning the coarsest level L 1 is first trained alone, and then higher levels are progressively trained to refine the registration. To avoid unstable starts when training levels L i>1 , lower levels weights were frozen for a fixed number of epochs. Regarding the optimization process, the learning rate was set to 10e−4, and the Adam optimizer was used for 1000 iterations on the two lower levels and 2000 iterations on the finest. As a dissimilarity metric, we used the NCC, regularized by smooth and diffeomorphic terms, for which λ smooth and λ dif f eo were determined with a grid search to 0.1 and 1.0 respectively to minimize the overall loss function over a test set (see Eq. 5), as described below for DL-based methods training. Indeed, since we aim at using registration-based features to monitor metastatic breast cancer in future work, we need to reach precise global and local registration. More details can be found in [33].
Starting from the architecture in Fig. 1, we performed an ablation study to measure the influence of each component. Hence, we i) changed the depth of the network (network with 1, 2, 3 or 4 depth levels), ii) computed the results after training each level during the coarse-to-fine registration i.e. the coarsest, the intermediate and the finest, iii) deleted the residual connections of residual blocks to transform them into simple convolutional blocks, iv) replaced the downand up-convolutions to respectively max-pooling and upsampling operations, v) used deformable convolutions ( [11]) in the finest level, and vi) used a Gaussian noise image N (0, 0.001) (as in [28]) or concatenated fixed and moving images as network inputs. Moreover, we also vii) set both λ smooth and λ dif f eo to 0 to remove the registration-specific regularization terms (see Fig. 1.C). All these implementations are detailed in Table 1.

Reference methods implementation details
Regarding the methods used as reference, we used a grid search on a test set (as described below for DL-based methods training) to find hyperparameters yielding best performance running the ANTs pipeline ([1]), i.e. with a three resolutions coarse-to-fine optimization, a gradient step of 0.2 and a symmetry transformation penalty. We used both MI and NCC as similarity metrics, leading to SyN and SyNCC methods respectively. We also ran the Elastix pipeline ( [26]) to perform successive rigid, affine and deformable image registration with four resolutions optimized with an adaptive SGD minimizing the NCC similarity term for 1000 iterations (Elastix). As for our method, we used a penalty on the bending energy as regularization to enforce smooth deformations.
To highlight the regularizing power of the architecture in MIRRBA, we ran the pipeline without the network, optimizing directly the deformation field (initialized as a Gaussian noise N (0, 0.001)) with the Adam optimizer, MIRRBA wo Archi (see Table 1 and Fig 1.D).
Regarding deep learning-based methods, DL LapIRN ( [33]) was ran with a similar regularization balance as MIRRBA, i.e. with λ smooth = 0.1, while all other recommended settings were used. For DL Voxelmorph, we used the latest diffeomorphic version to date ( [12]), with the NCC loss and recommended settings. For methods relying on a training stage, we split our dataset into five folds, paying attention to balance data from different acquisition centers among folds. For each fold, we refer to D train and D test as the train and test dataset, respectively. We trained the DL-based approaches on D train before testing them on D test for the five folds.
All architectures were implemented with PyTorch ( [37]) and trained from scratch on a Nvidia V100 32GB SXM2 GPU.

Evaluation metrics
The first criterion we used to evaluate the registration accuracy is the detection rate, defined for an individual as the percentage of lesions presenting an overlap greater than 50% between the wrapped and fixed segmentations ( [34]). A higher percentage refers to a better detection.
To evaluate the registration more locally, we computed Dice scores between fixed and wrapped i) brain and bladder and ii) lesions segmentation masks. A Dice score close to 1 means a precise local registration, while a Dice close to 0 is unsatisfactory. Since the detection rate represents the percentage of lesions correctly detected, it is positively correlated with the Dice score of the lesions. However, since some lesions (155) are cured over time and disappear on PET images, we removed them from the Dice score and detection rate computations to avoid erroneous null values. Instead, we evaluated the capacity of a method to effectively make lesions disappear by computing its disappearing rate, or percentage of volume reduction of a lesion induced by the registration, where a complete disappearance would mean a rate of 100%.
Registration smoothness was evaluated by measuring, for every deformation field, the standard deviation of its Jacobian determinant SDJDet. Null values indicate an identity transformation and high ones disorganized and incoherent displacements. Although an optimal value is difficult to define, we sought to obtain small positive values, characterizing smooth deformations ( [33]).
Finally, we evaluated the approximate running time of each approach, using a CPU for SyN, SyNCC and Elastix, and a GPU for all other methods.

Statistical analysis
To evaluate the statistical significance of our results, we first studied their distribution. According to the Shapiro-Wilk test (testing the null hypothesis that a sample comes from a normal distribution), we cannot reject the null hypothesis. Hence, we ran a paired sample t-test on our results and considered them of statistical significance if p < 0.05.

Regularization terms
In the first experiment, we looked at the influence of the different regularization terms from Eq. 3, i.e. R smooth , R dif f eo , R dataset and R archi . To this end, we compared the results to our MIRRBA approach optimized directly (without learning) on the D test pairs, using R smooth , R dif f eo and R archi terms (see Table 1, Fig. 1.A and Eq. 5). Dataset regularization We trained the Voxelmorph (DL Voxelmorph) ( [12]) and LapIRN (DL LapIRN) ( [33]) with the four regularization terms from Eq. 3 (see Table 1 and Fig. 1.B). Predictions were made on D test images. Even if both DLbased methods reached similar numerical results (see Table 2) and coherent visual ones (see Fig. 2 and supplementary material), MIRRBA performed better than both of them globally and locally. Indeed, from Table 2, we note that running a patient-specific optimization, i.e. removing R dataset and optimizing an untrained network MIRRBA compared to DL Voxelmorph and DL LapIRN, helps improve the results especially for the lesions. The organs' Dice score improved by 6% and 5% while the Dice of the lesions improved by 52% and 65% for DL Voxelmorph and DL LapIRN respectively. Moreover, MIRRBA presented lower SDJDet values, hence produced smoother deformations than training-based approaches. On the other hand, DL Voxelmorph and DL LapIRN had a higher disappearing rate.
Registration-specific regularization To understand the influence of the registrationspecific regularization terms R smooth and R dif f eo , we removed them from Eq. 5 in MIRRBA wo Regu (see Table 1) and Fig. 1.C). As quantitatively show in Table  3, MIRRBA presented Dice scores for the organs and lesions which are respectively 6% and 4% higher compared to MIRRBA wo Regu. Yet, the detection and disappearing rates were increased for MIRRBA wo Regu over MIRRBA.
Architecture regularization Finally, to also understand the impact of the regularization power of the network architecture on the registration, we looked at the results of MIRRBA wo Archi, which directly optimizes the deformation field without including any specific architecture, hence not including R archi (see Table 1 and Fig. 1.D). According to Table 3, MIRRBA wo Archi was outperformed by MIRRBA for both organ and lesion segmentations by 22% and 78% respectively. Although MIRRBA wo Archi disappearing rate was higher than MIRRBA's, the method without architectural regularization presented a low detection rate confirming the structural bias of convolutional generators.

Architectural choices
To study the regularization effect of a network architecture on registration, we compared the results of various architectural choices built from the pyramidal network presented in [33] (see Table 1). Quantitative results are presented in Table 3, while qualitative ones are visible Fig. 2

and in supplementary materials.
Depth of the pyramidal network First, we modified the architecture to optimize a simple U-Net-shaped network (MIRRBA Depth 1), and pyramidal ones with two (MIRRBA Depth 2), three (MIRRBA), and four (MIRRBA Depth 4) resolutions. While MIRRBA Depth 1 was optimized on full resolution images, all other architectures were trained using a coarse-to-fine strategy.
Results show that increasing the network depth improves the Dice results, as well as the detection and disappearing rates. Indeed, MIRRBA Depth 4 presented the second highest Dice scores for both the organs and the lesions among all MIRRBA setups. Regarding the SDJDet values, they were higher when using a depth 3 or 4, than when the network was trained on less resolutions.

Trained network level
To understand the amount of information brought by each network level during the coarse-to-fine training strategy, we computed the registration after optimizing only the lower level (MIRRBA Level 1) on coarse resolution images, both lower levels (MIRRBA Level 2) on coarse and medium resolution images, and the whole network (MIRRBA) with the complete coarseto-fine training strategy.
According to Table 3, training on all three levels of the network improved the registration accuracy over training only on low resolution images. Moreover, the disappearing rate was null when the training only occurred on low resolution images.
Max-pooling and upsampling operations For each network level, we replaced the down-convolution (convolution with stride 2) by a max-pooling operation in MIRRBA Max, and the transpose convolution by an upsampling in MIRRBA Up. As presented Table 3, using these learning-free operations improved the registration accuracy globally and locally. The disappearing rate was however slightly reduced.
Residual blocks We removed the residual connections of the residual blocks (MIRRBA wo RB) to understand their influence in our architecture. Without them, we obtained null Dice scores and detection rate, as well as very high SDJDet value. Besides, wrapped images did not look realistic (see Fig. 2 in the supplementary materials).
Deformable convolutions To adapt the receptive field of the convolutions to the local scale of objects to be registered, we replaced those of the highest resolution level (i.e. level 3) by deformable convolutions (MIRRBA DefConv). While global registration results were improved over MIRRBA, local ones were not. Moreover, the SDJDet was low.
Input images Regarding the network inputs, instead of conditioning the network with the moving image, we fed it with a Gaussian noise, as in [28]. In Table 3, MIRRBA NoiseImg shows worse results than MIRRBA, except for the SDJDet. We also provided more information to the network by concatenating the fixed to the moving image in MIRRBA FixImg, which significantly improved all MIRRBA results, especially the disappearing rate.
Combining best practices Finally, we combined the best architectural variations presented above to perform registration with four resolutions, both fixed and moving images as input, max-pooling and upsampling operations, as well as residual blocks: MIRRBA Best.

Comparison to conventional methods
Conventional registration According to Table 4, even after rigid and affine preregistration, locally precise deformable registration is challenging on whole-body images for the conventional Elastix ( [26]) pipeline (see Fig. 1

in supplementary material).
On the other hand, both SyN and SyNCC ([1]) statistically performed better than MIRRBA (which used the NCC metric) on organ segmentation, while the NCC similarity metric allowed SyNCC to also reach a better accuracy on lesion segmentation. Regarding the disappearing rate, MIRRBA performed better than both SyN-based methods, whereas their SDJDet was lower than ours.
Combination of DIP and conventional registration Finally, we pushed the analysis by combining our best MIRRBA-based method with the best performing conventional SyNCC approach. To do so, we optimized MIRRBA Best, using as input the deformation fields and already registered images obtained by SyNCC.
Our assumption was that the conventional pre-registration would be improved by our method.
Results of this Combined pipeline are shown in Table 4. In terms of global and local accuracy, the combined approach outperformed both SyNCC and MIRRBA Best. In addition, the disappearing rate was significantly improved for both methods.

Impact of training on a database
When comparing MIRRBA to the DL-based approaches (DL LapIRN and DL Voxelmorph), we show that not learning registration patterns from a dataset helps to obtain precise segmentations, especially at a lesion level. Indeed, while the size of organs and their locations are relatively consistent across patients, this is not the case with the lesions. Hence, there are fewer deformation patterns that can be learned from a database. Since performing locally precise registration with a DL-based method is very challenging in this situation, our method adapts to each subject. Moreover, according to its lower SDJDet value, MIRRBA produces smoother deformations than the DL-based approaches.

Architectural choices
Regularization terms As presented in Section 3.1, removing the registrationspecific regularization terms is equivalent to solving an ill-posed problem in conventional registration methods. In our case, since we use a network to perform registration, R archi is present when running MIRRBA wo Regu. For similar Dice scores and detection rate, MIRRBA wo Regu has higher SDJDet than MIRRBA, indicating R smooth and R dif f eo help smooth the deformation field, even though they have less impact than the regularization of the architecture.
On the opposite configuration MIRRBA wo Archi, where R archi is not used but R smooth and R dif f eo are, quantitative results show that the lack of architecture negatively impacts the registration smoothness, as well as its local and global precision, making questionable the convergence of the method without architecture. Hence, we confirm that the architecture has a regularization effect on the registration, which helps to find an admissible solution.
Coarse to fine strategies and image resolution Considering the overall network, [13] affirms that an architecture ran in a DIP pipeline acts as a low-pass filter in the beginning of the optimization, allowing higher frequencies to pass only after lower ones. We observe similar results over the coarse-to-fine training strategy (see Fig. 2), where low frequencies are registered first (global registration), followed later by higher frequencies (local registration). In the same way, looking at the intermediate results during the 4-depth pyramidal optimization of Elastix (see Fig. 1 in supplementary materials), we observe a tendency to register global features before local ones.
Modifying the number of coarse resolution levels, i.e. the depth of the pyramidal architecture, we studied the impact of global structural choices to optimize the whole network. The higher SDJDet values obtained when more resolutions are used could be explained by the generation of more local transformations, hence a globally less regular deformation field. Moreover, with four resolutions, the receptive field of the coarsest level of MIRRBA Depth 4 captures the whole image (i.e. 200 × 200 × 200), explaining the high Dice scores, as well as detection and disappearing rates. Indeed, successful conventional pipelines as Elastix ( [26]) or ANTs ( [1]) also uses this type of pyramidal strategy.
Network conditioning with input images Contrary to [43] and [28], who used Gaussian noise as input for their architectures, [17] and [3] respectively improved CT and PET DIP-based image reconstructions by providing acquisitions from other modalities to their networks. We made similar observations, as MIRRBA NoiseImg reached less accurate results than when we conditioned the model with PET images, as in MIRRBA. Moreover, feeding the network with more patient information, using the fixed image as additional input to the moving image in MIRRBA FixImg, improves the results by increasing the network conditioning on a single patient.
Other architectural choices Moving inside the architecture, residual blocks can be related to diffeomorphic registration according to [39]. Indeed, stacking residual blocks in ResNets ( [18]) aims to incrementally map the embedding space to a new unknown space, each block being defined as y = F (x) + x, with x and y the respective input and output of the residual blocks, and F the residual mapping to be learned. Similarly, diffeomorphic registration models ( [5,41]) address the registration issue by piling up incremental diffeomorphic mappings. Making the link between ResNets and registration, the function F can be seen as a parametrization of an elementary deformation flow, and training a series of residual blocks as learning continuous and integral diffeomorphic operator. With our deep architecture, MIRRBA wo RB results indicate that the registration without residual blocks fails to converge. Indeed, as explained above, residual blocks allow incremental diffeomorphic mappings, and removing them leads to gradient vanishing for our patient-specific method.
Regarding the convolutions, [20] suggested deformable convolutions ( [11]) to capture larger deformations. These convolutions add 2D or 3D offsets to the regular grid sampling of standard operations. If these offsets are set to zero, deformable convolutions become standard convolutions, otherwise they modify the receptive field. Since these offsets are learnable, deformable convolutions are trained to adapt their receptive fields, in order to focus on objects of interest in classification problems for instance. According to the high detection rate, this is performed globally by MIRRBA DefConv. Yet, the resulting low Dice scores show that MIRRBA DefConv did not achieve precise local registration, which can be explained by the low SDJDet value, showing that only smooth and regular deformations occurred instead of locally irregular ones. Indeed, deformable convolutions might need a more complex integration in the architecture ( [30]). Moreover, the additional learnable parameters and adaptive receptive field of deformable convolutions are to some extent redundant with the pyramidal structure, and make it harder to train.
Considering the amount of parameters to optimize in a network, we looked at the results of MIRRBA Max and MIRRBA Up compared to MIRRBA. Even if the learnable down-and up-convolutions are now common to respectively increase or decrease image dimensions within a network, max-pooling and upsampling operations were originally used. As in conventional registration methods, these operations are not learnable. Hence, their results do not depend on optimization parameters, and they help to control the overfitting in a traditional trainable setup. Therefore, the good results obtained using max-pooling and upsampling operations in MIRRBA Max, MIRRBA Up and MIRRBA Depth 4 Max Up may be explained by the fewer number of parameters to fit (see Fig. 2 in supplementary materials).

Lesion segmentation
Regarding lesion registration, Fig. 3 shows different lesion evolution scenarios. We can see that the moving masks adapt to the fixed ones when the lesion does not disappear. Although the disappearing lesion from the first column was not deleted, it was reduced by the registration algorithm. Indeed, lesion disappearance implies deformations which are not smooth nor diffeomorphic, hence the good disappearing rate of MIRRBA wo Regu and the probable need to adapt our registration strategy to the specific problem of lesion disappearance in future work.

Conclusion
In this paper, we propose an alternative method to perform image registration using a neural network without the typical learning stage on a database. We formalize the registration problem by following the conventional approaches relying on image-based similarities and regularization terms, but also explicitly consider the dataset and architecture bias. Indeed, our study was motivated by recent work on DIP, implying that neural networks create an inductive bias when learning from a database, but also an intrinsic structural bias induced by the architecture [19]. These biases are shown sufficient to solve certain image processing tasks. Our proposition also resonates with [13], who suggested that deep convolutional neural networks process low-frequency information first, to later focus on the finer deformations, both desirable properties for a registration algorithm. We integrate the LapIRN network from [33], who developed an effective pyramidal architecture, tested in the DL-based set-up. Here, we further demonstrate that beyond any prior coming from the dataset learning step, the architecture design has an important effect on the registration results, acting as an implicit regularizer. Our study also shows the impact of some of the architecture components, particularly the residual blocks, and we justify this behavior by making a link with findings from [39]. Moreover, we found that for our application, a pyramidal architecture capturing the whole image with a limited amount of parameters to optimize, as conventional registration methods, provides precise registration results.
The architectural prior seems to be a better option than learning from data in cases where there are no positional consistency, which is the case with metastatic breast cancer lesions, which arbitrarily vary in position, size and number. Indeed, finding a set of network parameters allowing precise registration for a whole dataset is a challenging task. Our approach enables to correctly register active organs such as the brain and the bladder, which could be used to automatically propagate annotations masking regions irrelevant for patient response evaluation. Although the Dice scores are relatively low for the lesions, we obtain good detection values and improve the disappearance rate.
Our approach makes a step in bridging conventional and DL-based methods for image registration, and provides several suitable approaches for the challenging 3D full-body longitudinal registration problem. We demonstrated the possibility to perform both global and local registration on whole body medical images using a network but without suffering from dataset bias. In future work, we would study the feasibility of extracting registration-based feature from our method to monitor lesion evolution without depending on manually performed segmentations.  Grayscale color indicates good overlapping. We can see that the higher the depth, the more precise the registration. A simple U-Net-shaped architecture as MIRRBA Depth 1 produces transformations of very low accuracy, while the more resolutions are used, the more precise the registration. The all green MIRRBA wo RB image is due to the non-convergence of the registration algorithm and a wrapped image not registered to the fixed one. All four other approaches produce realistic looking and coherent transformations, even if MIRRBA DefConv lacks a bit of precision around the bladder. MIRRBA NoiseImg does not reach local precise registration. Best viewed in color. Table 1. Loss terms and structural setups of the methods presented in the paper. "Pyr. net." stands for "Pyraminal network", "Sym. diffeo. transfo." for "Symmetric diffeomorphic transformation", "Max." for "Maxpooling", "Up." for "Upsampling", and "Def. conv." for "Deformable convolutions". An absence of information in the Other column indicates the use of setups described in Sections 4.2 and 4.3. Table 2. Comparison of MIRRBA to DL-based methods (Section 5.1) -Detection rate, Dice score of the organs and lesions (Dice organs and Dice lesions resp.), SDJDet, disappearing rate, and approximate computational time. Both training and inference computational times are indicated for DL-based methods. All pipelines were computed on GPU. Statistically significant improvement of our MIRRBA method over the others with p < 0.05 is indicated with *. Best results are marked in bold, and second best ones in bold-italic, except for SDJDet since no ideal value is defined.