In our search, we only optimize the data inference path inside the merged model and keep parameters in the models intact. In this setting, a layer may face an input whose distribution is different from what it is used to (from its original model), leading to unexpected outputs. For example, our preliminary studies show that swapping a pair of neighboring layers in a language model makes its performance drop. Although more theoretical studies are needed to model the distribution shift, empirically we find that appropriately scaling an input that wishes to go from layer \(i\) to \(j\) by \(W_{ij}\) help alleviate the problem. Here, \(W\in\mathcal{R}^{M\times M}\) is a matrix that is also optimized by the evolutionary search together with the indicator array \(\mathcal{I}\).