We have three sets of test suites:A small suite, taking only a couple of hours to runA medium suite that takes multiple hours, usually ran every night (nightly)A large suite that takes a week+ to runWe also have a bunch of shorter test suites, but I'm not focusing on them here.The current methodology is to run the small suite before each commit to the trunk. Then, the medium suite runs every night, and if in the morning it turned out it failed, we try to isolate which of yesterday's commits was to blame, rollback that commit and retry the tests. A similar process, only at a weekly instead of nightly frequency, is done for the large suite.Unfortunately, the medium suite does fail pretty frequently. That means that the trunk is often unstable, which is extremely annoying when you want to make modifications and test them. It's annoying because when I check out from the trunk, I cannot know for certain it's stable, and if a test fails I cannot know for certain if it's my fault or not.My question is, is there some known methodology for handling these kinds of situations in a way which will leave the trunk always in top shape? e.g. commit into a special precommit branch which will then periodically update the trunk every time the nightly passes.And does it matter if it's a centralized source control system like SVN or a distributed one like git?By the way I am a junior developer with a limited ability to change things, I'm just trying to understand if there's a way to handle this pain I am experiencing.
How to keep the trunk stable when tests take a long time?
testing;version control;development methodologies;branching
The only way to fix the root cause of the instability is to decouple the code so changes are more isolated, as other answers have suggested.However, as an individual developer, if you want a more stable build for you personally to work on, that's relatively easy to solve.  Instead of working off of the tip, you only pull the last build that passed the overnight test suite into your working tree.  If you can create feature branches for each change, then branch off of the last stable build.Yes, your tree will be a few days behind, but most of the time that doesn't matter.  Do your work against the stable build, so you know your changes are the ones that broke any tests, then before you check in, update to the latest and do your normal integration.  Then after you check in, back up to the last stable build again.You still have to do the messy integration work, but what I like about this method is it isolates the integration work to a time more convenient for me, and gives me a stable code base for development when it isn't convenient.  I have a much better idea when it's my changes that likely broke the build versus someone else's.