We reproduce the performance of the selected single-step models using a 10-fold cross-validation with USPTO-50k. By averaging across the folds, we reproduce the results reported in Chemformer [1]}, MHNreact [2]}, and LocalRetro [3]} (Fig. REF  and Appendix REF ). For all models the data split has no discernable effect on the accuracy, shown by the small standard deviation across all folds. Additionally, we calculate the performance of the baseline model implemented in AZF [4]} as a benchmark to compare the single-step models.
For top-1 accuracy, Chemformer outperforms the other models, with an average accuracy of 54.7% (± 1.1%). LocalRetro and MHNreact follow this with 52.5% (± 0.7%) and 49.8% (± 0.8%) accuracy, respectively, and AZF performs notably worse with an accuracy of 43.3% (± 1.0%). This pattern, however, is not maintained across the top-k measures. Accuracy noticeably ascends within the top-3 for all models, LocalRetro seeing a +24.1% increase in accuracy to 76.6% (± 0.6%), similar to MHNreact with a +23.0% increase to 72.8% (± 1.0%). AZF has a +16.8% increase in accuracy to 60.0% (± 1.0%), and Chemformer has the smallest gain in accuracy with an +11.2% increase to 65.9% (± 1.0%). Within the top-50 predictions, LocalRetro shows 96.6% (± 0.3%) accuracy, followed by MHNreact with 93.3% (± 0.4%) accuracy, both showing similar profiles across the top-k. AZF notably increases its performance across top-3 to top-10, giving 78.1% (± 0.7%) accuracy at top-50. Surprisingly, Chemformer delivers the lowest accuracy in the top-50 of the models tested, with an accuracy of 73.3% (± 0.3%). Though Chemformer outperforms other models in top-1, it is less able to find the ground-truth reactants for the remaining products despite additional explored alternatives with higher top-k.
<FIGURE><FIGURE>