Open Source Software and Reproducibility in Computer Science As a PhD researcher in computer science, my scientific results often take the form of algorithms, which in turn come with implementations. I am convinced that releasing these implementations under an open-source model has various benefits especially for scientific codes, one of which is reproducibility of results. I also believe that "Release Often, Release Early" applies to science, as it opens up the possibility of feedback that ultimately leads to better software and better research.

However, my advisor is concerned with the danger of plagiarism in case the code is released before the respective paper is published, and therefore disagrees with that philosophy. To me it seems like a very theoretical threat since I have never heard of a comparable case of plagiarism. However, we just got a paper rejected in which one major criticism of the reviewer was that it was impossible to reproduce results since the code was not available. My advisor thinks that this is unfair criticism because the conference has not explicitly asked for a release of the source code. 

What do you think? Is it sensible to keep implementations secret until the paper is through, or is open-source actually a requirement for reproducible computer science?
Computer science is a particularly friendly environment for releasing material in advance of publication.  In fact, there are a number of methods for doing so that provide a clear archival time-stamp on your work, including university technical reports, arXiv, and big repository sites like github and bitbucket.  Moreover, unlike many other disciplines, such pre-release almost never impairs your ability to build conference and journal publications on top of your released material.  In fact, you can typically release not just your code, but also your draft paper through the same mechanisms.

If you do such pre-release, you are essentially immune to plagiarism, because anybody who pirates your results will look like a fool with date-stamps marking their guilt for all to see.  It may not stop bottom-feeders from trying, but it will mean that nobody you care about (including reviewers) should take such an attempt seriously.

The only likely problem that you can run into is double-blind reviewing, as discussed in this question.  If that's not a major concern for you, or if you can use one of the mechanisms suggested in that question, then I see no likely downside in computer science publication.

On a side note, however: just because the code is available doesn't mean it's reproducible, as anybody who's spent significant time having a dependency-fight with poorly-document open source projects can attest.