Guidelines for communication of machine learning methods in empirical research papers It seems to me that machine learning methods are being adopted more and more often in empirical academic research and modelling. Whereas scientists are often quite familiar with traditional statistical methods to report on their findings, they may by comparison often be somewhat less knowledgeable about the machine learning algorithms they employ. 

I wonder whether there are any articles on the (in)correct communication of machine learning algorithms in empirical research. I'm interested in how scientists ought to present the results found by means of such methods, to what extent they should describe the algorithms they've used, and other related matters they should take into consideration.
Probably, precise communication guidelines will (and should) vary between different fields, depending in part on how well machine learning methods have been described, utilized and benchmarked in prior works within the field. However, in my opinion, every empirical field should already be able to take note of lessons learned within computer science, and also implement sanity checks on results based on both general and field-specific knowledge. Indeed, it's not all that different from applying statistical methods to a given study.

The main new challenge is guaranteeing reproducibility while using ML methods. See e.g. M. Hutson: Artificial intelligence faces reproducibility crisis, Science (2018). (Non-paywall link here.) The article stresses the need of providing code, test data, and details on training procedures, but does not provide very specific guidelines.

However, Joelle Pineau, professor of computer science at McGill, has been pushing for higher reproducibility standards in machine learning-related research. Although not published in the traditional sense, the Reproducibility Checklist on her website is the closest to a standard guideline I know of. For example, it was used for submissions to the 2019 NeurIPS conference. (A report on how this worked out can be found here.) It gets routinely updated, so I recommend checking the source. However, I will reproduce some of the points that are relevant for empirical science:


  For all datasets used, check if you include:
  
  
  The relevant statistics, such as number of examples.
  The details of train / validation / test splits.
  An explanation of any data that were excluded, and all pre-processing step.
  A link to a downloadable version of the dataset or simulation environment.
  For new data collected, a complete description of the data collection process, such as
  instructions to annotators and methods for quality control.
  
  
  For all reported experimental results, check if you include:
  
  
  The range of hyper-parameters considered, method to select the best hyper-parameter
  configuration, and specification of all hyper-parameters used to generate results.
  The exact number of training and evaluation runs.
  A clear definition of the specific measure or statistics used to report results
  A description of results with central tendency (e.g. mean) &amp; variation (e.g. error bars).
  The average runtime for each result, or estimated energy cost.
  A description of the computing infrastructure used.
  


This was based on version 2.0 of the checklist, dated April 7, 2020. Note that experimental above does not mean experiment in the sense of empirical science, but in the sense of running a "numerical experiment" on some set of data.



Finally, note that this is a developing field. I fully expect more guidelines to be written over the next few years, as people realize the need for it. For a sign that this coming, see e.g. this 2019 DOE report, which declared scientific reproducibility in applications of scientific machine learning a priority research direction.