Do “researcher degrees of freedom” invalidate my analysis? I tested a hypothesis using some observational data, but did not find support for the hypothesis (the difference was not significant at the p &lt; 0.05 level). I subsequently realised that the inclusion of a certain subset of individuals was dubious, since there was some doubt over their measurements. I filtered out those individuals (on objective criteria), and lo and behold, I now find evidence to support my original hypothesis. 

I am acutely aware that, had the original hypothesis test returned a significant result, I probably would have found justification to support the inclusion of those doubtful measurements (or perhaps never even stopped to think about removing them). Although this wasn't my deliberate intention, what I have done seems a lot like I have employed my “researcher degrees of freedom” to find a version of the analysis that supports my original hypothesis. 

I realise that what I should have done was to plan out my analysis more carefully in advance, and decide whether or not to include the doubtful measurements before carrying out any analysis. But I can’t change what has already happened, so my question is, what do I do with my data/analysis now? I can see a number of options that vary in their sensibleness, but none of which seem ideal:


Continue to use my updated analysis and present a clear argument for why those individuals should be excluded (i.e. ignore the RDF issue).
Continue to use my updated analysis but, in any write-up/publication be fully open about the less-than-ideal path that I took to reach it.
Conduct some kind of multiple-comparisons adjustment to take my multiple different analyses into account (I don’t know if this is even a valid approach in this context).
Sigh and throw the analysis in the bin.


How (if at all) can I make good use of my analysis while still adhering to good research practice?

My field is Ecology, in case that makes any difference. 

Edit, in response to close vote: I did consider submitting the question to Stats.SE instead, but I felt that this is more an issue of research philosophy than a detailed statistical question. The answers that I am interested in (and indeed have already received) are connected with a general research strategy, and how to present results, rather than details regarding particular statistical methods, and so I felt that it was appropriate for this site. Having received very useful answers and comments already, it won't make too much difference to me personally if it ends up closed (and I can understand the argument for doing so), but I think the material could be useful to others in a similar situation to my own.
Clearly report what you did and both results.  Readers will make up their own minds about the strength of the findings.

The situation that you have encountered is very common.  While testing a prior hypothesis with predefined methods is the right thing to do, in practice it is unavoidable that you will sometimes find problems that you need to correct after running an initial analysis.

Also, if your analysis produces an unexpected result, it is quite logical that this will lead to extra checking of what you have done, and further analyses to confirm it.  

There's nothing wrong with doing extra things to refine the analysis. The problem would be refining the analysis and then failing to report on earlier versions.

A multiple testing correction would not typically be used for only two analyses.   Furthermore, it is a judgement call whether this counts as two tests or one.  If the second analysis was really the "correction" of something flawed, that is not quite the same as repeating two different but valid approaches.  And there is not a clear line as to when that becomes true.

In summary, there is no problem with what you have done, as long as you report it, but your concern with doing this the right way is a very good thing--please keep it up!  Inadequate reporting of this sort of thing is quite common, and it undermines the quality of scientific evidence.