How to review a software-tool paper? I often get asked to review papers on new software tools. This always makes me wonder how one should review such a paper, or even whether software papers should undergo a typical review process at all?

My private checklist typically consists of:


Is it available for download for others?
Does it run?
Did the authors provide benchmark problems? Do they give expected results?
Is documentation sufficient?
Go through numerical methods used in the software (briefly, no meticulous code analysis) 


If everything runs fine and produces expected results, I honestly have nothing else to “demand” from the authors. My referee reports are positive and (embarrassingly) brief in those cases. I’m of the opinion that the user community should “do the rest”; evaluate the software and decide whether it’s useful in their workflows.
One perspective is from the Journal of Open Source Software, which explicitly establishes reviewer guidelines that cover these questions (http://joss.theoj.org/about#reviewer_guidelines).

A (slightly edited) quote of the demands:


  Software license
  There should be an OSI approved license included in the repository.[...]
  
  Documentation
  There should be sufficient documentation for you, the reviewer to understand the core functionality of the software under review.
  
  A statement of need
  The authors should clearly state what problems the software is designed to solve and who the target audience is.
  
  Installation instructions
  There should be a clearly-stated list of dependencies. Ideally these should be  handled with an automated package management solution.
  
  
  Good: A package management file such as a Gemfile or package.json or > equivalent
     OK: A list of dependencies to install
     Bad (not acceptable): Reliance on other software not listed by the authors
  
  
  Example usage
  The authors should include examples of how to use the software (ideally to solve real-world analysis problems).
  
  API documentation
  Reviewers should check that the software API is documented to a suitable level. This decision is left largely to the discretion of the reviewer and their experience of evaluating the software. [...]
  
  Tests
  Authors are strongly encouraged to include an automated test suite covering the core functionality of their software.
  
  
     Good: An automated test suite hooked up to an external service such as Travis-CI or similar
     OK: Documented manual steps that can be followed to check the expected functionality of the software (e.g. a sample input file to assert behaviour)
     Bad (not acceptable): No way for you the reviewer to check whether the software works
  
  
  Community guidelines
  There should be clear guidelines for third-parties wishing to: Contribute to the software, report issues or problems with the software, or seek support


This is a valuable way of thinking about reviewing software, and I think we should move closer to this. But it is not yet accepted! In my field, most "methods" papers don't even include software - and those that do would almost certainly fail these tests. (I'd never heard of continuous integration until recently, for instance.)