Bad data - how to move on successfully I work in an interdisciplinary field. My input is not generated by myself, but by talented people that I trust and that trust me to analyse their data and generate fascinating insights.

But here I am, once more stuck with a project where the input is bad. There is no use for blame and finding a scape goat, we are in this together. And people learn. But I have been stuck in projects with big promises and bad/insufficient input since the start of my PhD. I moved to another place for a PostDoc now, but this situation seemingly haunts me wherever I go.

There is not much I can achieve when most of my projects stop after the input quality control. But if I want to stay in academia, I desperately need to step up my game in actual output, not just in my ability to troubleshoot, right?

How do I move on from this? What chances do I have if I can just never land a "prestigious project" that results in a valuable publication? Is there a chance to still build up a good scientific reputation without those? Should I try myself at writing a review? Take on projects until one finally works out? (But how long will I succeed in getting another job if they don't?)
I could work with published data for a bit, but it is often not comparable between studies and severely lacking metadata. 

The question is, do I have to accept that success (as in, basically being able to stay) in academia is to a large degree based on luck and I am not one of the lucky ones or is there anything major I can do? I really love the work I do, I would like to keep going.

So far I have taken on additional projects, tried to do my own 'side projects' on at least roughly usable parts of the data in the hope I will find a better dataset eventually where this could come in handy and kept in touch with collaborators in an effort to troubleshoot and eventually produce better input.

EDIT:
To address some questions:
I produce analysis pipelines, partially based on my own methods. Without "real data" application it's hard to publish those in my field.
Yes, it is "real world data". I do not expect perfect data at all. But I do expect technically correct, usable data. If the input is random/to few features to be statistically relevant there is nothing I can do, though.
Imagine trying to do a statistical test on the similarity of blog posts based on word usage written by different groups of people but many "groups" only are represented by two authors, the text is sometimes just one sentence long and quite a few of the posts looking like they are produced by a random letter generator, not having any actual words in them. While I was promised at least 5 authors per group, minimum 5000 words per text and of course the post actually written by the author assigned to it.
Summary:


Real research life with real world data is messy*, and there will hardly ever be sufficient samples (my very personal prediction).  
There are huge opportunities (and needs) in working about small and messy data. Maybe that could become your area of research?  
Good data analysis work requires close collaboration. Actually already when planning the experiments, but for sure during data analysis.
Close collaboration will allow you to make them aware of your needs and that data analysis cannot work miracles. It is also necessary for your because otherwise you may be employing inadequate analysis methods.

* When I say messy I don't mean bad curation (although I do see opportunities here as well - though maybe more business than research) but reality sneaking in with lots of influencing factors creating structure in your data where many (most?) data analysis approaches assume nicely independent data. I think this is a field that not only deserves more research but also that this has high practical importance.    




I feel your pain. Been (almost) there as well. Actually, I still am (just my PhD is long done): so far with ≈ 15 years of professional experience in chemometrics, all the real world data I've encountered so far has one thing in common: too small sample size (even if it may look nice at first glance).


One consequence I drew for me is that I started to do research on such less-than-ideal situations I encounter in practice, e.g. 


small sample size situations: knowing that I have far too few cases (some orders of magnitude below rule of thumb recommendations), how to diagnose when things break down, how to stabilize models, what will break down, are there hard limits, etc. 
in terms of messy data in the sense above (with lots of influencing factors): 
I had situations where it turned out that biology doesn't really obey the disease classification medical doctors use (that was developed for totally different purposes, as I learnt later on) how to adapt data analysis methodology to these situations (that were somewhere in between classification and regression)
how to adapt validation/verification procedures in such situations
(I work very much along the lines that you can to whatever you think may work for modeling as long as you do a honest verification and validation of that model)
I see tons of similarly important questions that are unanswered.
To the extent that if you need such research ideas, I'd happily supply you with questions ;-) 






In my field, I think, it is going to stay like this: well-characterized samples are expensive.
In some respect you may even say that basic restearch isn't meant to have comfortable sample sizes. It's meant to find basic knowledge and point out promising possibility but the leg work of obtaining (and paying for) large sample sizes to make a method robust for routine use is something applied research/industry is supposed to do (and pay for). That point of view would say that taxpayer money should not wasted on work that industry could and should do. 
On the other hand, I often see unnecessarily too small sample sizes in academic research: too small here meaning that given the sample size even without any experimental data it is (or would have been if one had bothered to check) clear that no knowledge is gained because the study is too underpowered. This is clearly just bad science, and a total waste of experimental and data analysis effort.
If that's what you refer to in your question, it's going to be hard work to improve this - but don't give up! Science needs people like you pointing this out. 

My experience with that is that as PhD student or fresh postdoc how much you can actually do to improve the data may depend very much on how much weight what you say has with your supervisor (or even top level director).
What you can (and should) always do is to clearly discuss the limitations in terms of possible interpretation of the results of your study - including in the manuscripts you write.    
To be fair, there are practical limitations. If we study a rare disease where the big university hospital gets maybe one sample per year, I tend to think that working with very few cases is necessary (but again: spell out the limitations). After all, one has to start somewhere.
Whereas, if we're talking easily accessible measurements of no particular ethical concern of a disease where the hospital sees 10s of cases per week then of course a thesis on 5 cases looks somehow lazy (not necessarily on the PhD student's side, though: The PhD student may not have been able to change pre-existing sample plans)  
One consequence for my PhD thesis was: as I did not only data analysis but also sample preparation and measurements for my thesis, I put in substantial effort to have more samples (fortunately I had access to a comparatively large data bank, but in the end also that approach was limited by availability of the more rare conditions).
I'd recommend to at least take a decided interest how the data are generated (get a lab tour, get the collaboration partners explain how things work and what the data mean).





   If the input is random/to few features to be statistically relevant there is nothing I can do, though. 


Yes. Again, this needs to be clearly communicated: I do have the experience that applied groups may expect miracles from data analysis (and you may even have a particularly uphill fight here if this group in the past got data analyses that were badly overfit and thus looking overoptimistic and noone realized this).
In addition, you'll have to document that it isn't your "fault" that no nice results come out of this data. It's doable though (and again in my experience something that is needed in everyday data analysis work as well: I'm just having such a situation on my desk right now again).


  [...] "groups" only are represented by two authors, the text is sometimes just one sentence long and quite a  few of the posts looking like they are produced by a random letter generator, not having any actual words in them. While I was promised at least 5 authors per group, minimum 5000 words per text and of course the post actually written by the author assigned to it.


A few thoughts about this. I "smell" some communication/collaboration issues here. Again, they are typical everyday issues in my research and data analysis work:


I've met similar things due to fundamental communication issues (e.g. between statisticians talking about necessary sample sizes in the 1000s to answer a particular question and experimental scientists "translating" this to "many" ≈ 7.  
"quite a  few of the posts looking like they are produced by a random letter generator"
Your experimental collaboration partners may have no clue what you don't know about their techiques (again communication issue): unless you have a background in these techniques you have no chance to recognize what is going in those noisy measurements and how to deal with them.
They could be anything from artifacts that should be deleted because the underlying mechanism that causes them is well known and can be disregarded over "the signal is hidden under this noise and you data analyst surely have some magic to get it of there" (not going to work, but typical expectation) to "your outlier is my most interesting case" - without the help of the experimental/data supplying folks you won't be able to adequately deal with this.
Seeing all this makes me think whether you do have sufficient information about the background of the study to even decide which data analysis approach is suitable?