When replicating results, what should I do when the author didn't use the method the paper said they did? I am replicating a model to report on how well it does on a new evaluation metric I am proposing. The method/hyperparameters of the model are given in a paper about it, where it breaks all benchmarks on the tasks on which it is being evaluated.

After that paper came out, several people tried to replicate the results, and found that they could not. Their questions of “Am I doing this right?” can be found on a online message board for the project. They got several responses from the second author of the paper (very well-known and respected in the field), saying that he didn’t carry out the experiments and that they would have to wait for a response from the first author.

Eventually the first author responded with just a few lines saying: “The hyperparameters to use are ....” People ran with those parameters and replicated the results.

However, the parameters he gave disagree with those reported in the paper.
Quite likely this is through no malicious intent: possibly the model was rerun and the changes were not re-incorporated into the draft.

Now, I would like my model when I am evaluating to be as similar as possible to the one in their paper. That way, people can compare my new proposed metric to the metric that that paper uses and see that this model is good at X, but less good at Y, without having to correct for the fact that the models are different.



I have three options:


Use what I think are good hyperparameters for their model (this is what I am currently doing).
Use the hyperparameters given in their paper.
Use the hyperparameters given in the online message board post, which actually achieve the results given in the paper.


Which of these three options is best?
Given that you don't have space enough to present all three sets of results, I would suggest that you choose the parameters that best illustrate the scientific point that you are aiming to make.

When you state the parameters, however, you should add a parenthetical note or footnote that says something along the lines of:


  Note that we have chosen these parameters because [REASON].  Alternative parameters given [in the original paper / by the author] produce results that are [better / worse].


This balances the openness of making sure the reader is aware of the choice you have made and its consequences, while at the same time restricting your presentation space budget to only the most relevant set of results.