Are datasets really random for Bioinformatics tools For my masters thesis, I needed a to run a tool to filter out RNA sequences and then research on the filtered data. I researched online reading papers to find an appropriate tool and I found one that really fitted my needs. The tool was published by a renowned group in a good journal (IF around 7). The tool claimed to have been run on a 30 random datasets and claimed to have around 90% specificity and sensitivity. I was very happy. 

When I ran it on my dataset, the tool produced so many false positives and false negetives. Its accuracy was less than 10%. I ran on other datasets too and never got an accuracy of more than 20% in any one of them. To my surprise, the tool only worked well on the dataset that was published with the tool, with a good accuracy and no other input. I cannot believe that the dataset was randomly chosen. 

I spoke to my PI about this and he said that he understands. He allowed me to choose another topic for my thesis and helped me a lot to complete. In the end I could successfully defend my thesis.

My questions are:


Is this common for groups (esp in Bioinformatics) to select a biased dataset as input and claim it to be random?
Does reviewers really run the tools (or read the source code) that are accompanied by the paper in order to test it and see whether it really does what it claims or they only read the paper?
Can an end user later write to the editor and let him know that the tool published in their journal isn't what it was suppose to be?
What you are describing here is an in silico form of Cherry Picking Fallacy. Before answering your questions specifically, I would like to point out that before assuming bad intent or foul play, consider the biological variability in the datasets, and difficulties writing generalised solutions to biological problems. Take it from a 3rd year grad student and a bioinformatician in training, it's not straight-forward to develop tools for biological datasets. There are both biological and technical challenges in writing "foolproof" software in bioinformatics. 

Consider for instance, that there are multiple ways to estimate error and false discovery rate (FDR). There are also varying levels of thresholds used in different labs. Any heuristic value, if hardcoded, might alter the results. 

Similarly the biological diversity, as well as technical variability introduced by wetlab benchwork will likely have serious influence on the datasets. Once everything boils down to a large table of numbers, all of that variability is implicit and thus often forgotten.

That said, my answers to your question(s):


No it's not common to pick datasets just to show that your model/software works, at least not publicly. That would go against scientific rigour and ethics. At the same time, you start from what you have available. I typically start with datasets we have in the lab, when I start developing a new tool. But in order to publish your method, you should typically show that it works on data from independent sources, or alternative data types etc. All in all, your results might not be representative of all datasets out there.
It is impossible to answer this question, factually. I am sure there are reviewers that take a look at raw data and/or source code, but it's unlikely that any reviewer actually digs into the source code, line by line, to see if everything checks out. But I am also certain that there are some who just see if you have actually made the source code or test datasets available or not, without actually look at them. 
You can always do that, end-user or not. But beware, before you go blowing the whistle on someone else's handiwork (which probably took them a significant amount of time and money) you better get your facts straight. Claiming that they are selling false results or simply misinterpreted their results is a very serious accusation, especially if the authors are renown and respected in the field. 


Prior to writing a letter to the editor (who AFAIK have a duty of publishing scrutinising critique on something that has been published in that journal) you should check, and double check, your results. Make sure you are not misunderstanding the statements made in the original work. Make sure you are not misunderstanding your own results. Make sure what you have observed is not a side-effect of your datasets or your own use of the software, try to talk to other people who have used the software. And make sure, your supervisor (and others in the group) are behind you, get your own results investigated by your superiors and colleagues. 

If everything checks out you can, and should, inform the journal that the results in the original article might not reflect a general reality, and that you could not replicate the results on independent datasets, then show your results.