In our DFS merging experiments, \(M=64,r=3\), and consequently, \(T=M\times r=192\). We kept the last 200 examples in the training data as our validation set and optimize on the rest of the data with a batch size of 200. We report the performance of the snapshot that achieved the highest accuracy in the validation set, and the test set is strictly isolated from the optimization process. We adopted CMA-ES in EvoJAX [46], it optimized \(\mathcal{I}\) and \(W\) for a total of 100 generations with a popluation size of 128, and we used the default hyper-parameters. We limited our DFS merging to two models \(A\) and \(B\) to ensure that the final model remains modest in size and can be run on a single GPU, but in principle, the methodology can scale to merging multiple models. During the merging, model \(A\)'stokenizer, and input/output embeddings are utilized. Furthermore, to maintain compatibility with the embedding layers, we mandate that the initial and final transformer layers of model \(A\) define the start and the end of the inference path. We initialized the indicator array \(\mathcal{I}\) so that all layers in model \(A\) are more likely to be included as initial hops in the inference path to shorten the search time.