I think the short answer to this question is probably yes, but I thought I'd ask anyway!We are in the process of taking a number of existing similar-but-different applications and, as part of what is for us possibly the largest development project ever, attempting to replace them and any subsequent similar-but-different applications with one over-arching super configurable system to rule them all.  I fully expect Difficult Second System Syndrome to give us plenty of problems delivering the behemoth in the first place, but I don't want our choice of methodology to make it all worse.  We've dabbled in agile, from something mostly waterfall, but this would be our first substantial project using some new improved methodology.We have Big Plans for the new system, and while it doesn't all have to be delivered at once we will need to implement large chunks of it from the word go to allow anything to be given to the customer that they would even understand.  My concern is that, since we're dramatically changing so much at once, there is little scope for small chunks of work that lead to useful/demoable deliverables.  I'm worried that, even if we did manage to find a way to do it in small chunks, we'd need those chunks to fit well into a bigger picture, and so would find it hard to develop them in isolation for that reason too.  We're aiming for quite a decoupled SOA approach, which is something.Timescales are tight, and reputations have been staked on delivering on time.So I'd like to know if, in this case: it is advisable to do loads of up-front design, put together the overall shape of the app and get the crucial modules working in one big push (which might be sprintable in terms of keeping track of work, but not so much in terms of feedback and improving design and so on), before maybe going more agile from then on, or...try to be as agile as possible from the start - agree things like coding standards, maybe some API basics, general module splits, and then trust to independent coding while bearing flexibility in mind to let the other stuff gradually flesh out; maybe some stuff has to be reworked, but anything that can be better steered during development rather than up-front will benefit, or...maybe something else, and...any general tips?
Can agile be effectively used for initial development on a substantial project?
agile;methodology;enterprise architecture
Yes... If...I'll only say yes if you can drop features from your project to hit the deadline.Otherwise you will only be getting a portion of the benefits from an Agile Approach.Arbritrary Deadlines don't workTimescales are tight, and reputations have been staked on delivering on time.This worries me, this smacks of an arbritrary deadline created from optimistic best guess analysis.  Arbritrary deadlines do not work, and in fact have been proven to work in detriment to a project.Joel Spolsky wrote a great essay on Evidence Based Scheduling3) Simulate the futureRather than just adding up estimates to get a single ship date, which sounds right but >gives you a profoundly wrong result, youre going to use the Monte Carlo method to >simulate many possible futures. In a Monte Carlo simulation, you can create 100 possible >scenarios for the future. Each of these possible futures has 1% probability, so you can >make a chart of the probability that you will ship by any given date. Be prepared to drop features to save timePutting that aside having a fixed timeframe for an Agile Project is fine, however you need to be able to prioritize your most valuable features in your software product and then drop your not so valuable features to reach your time milestone.This should happen every single sprint, the only features that get merged into a release at the end of a sprint are features that are fully tested and functioning.  Features that are not fully tested and functioning get dropped for the next sprint or next release.  Release doesn't mean deliverable to the customer, but after every sprint you have a fully working (albeit in a limited fashion) software product that is fully tested.Use the best features of AgileThe idea of using agile is that your features between sprints is not set in stone.  In a normal agile project the customer has input at every sprint and may request a new feature be added as a higher priority than a feature requested 3 sprints ago.  Avoid a project race (if at all possible)In your case this probably isn't going to happen, you already have the feature set out in your existing projects, and in addition it also sounds like you are going to have a race condition whereby you are going to be racing your current maintenance cycle for your existing projects with your new project so that you can replace it.http://butunclebob.com/ArticleS.UncleBob.CleanCodeArgsIf you have been a programmer for more than two or three years, you  have probably been significantly slowed down by someone elses messy  code. If you have been a programmer for longer than two or three  years, you have probably been slowed down by your own messy code. The  degree of the slow-down can be significant. Over the span of a year or  two, teams that were moving very fast at the beginning of a project  can find themselves moving at a snails pace. Every change they make  to the code breaks two or three other parts of the code. No change is  trivial. Every addition or modification to the system requires that  the tangles, twists, and knots be understood so that more tangles,  twists, and knots can be added. Over time the mess becomes so big and  so deep and so tall, they can not clean it up. There is no way at all.As the mess builds, the productivity of the team continues to  decrease, asymptotically approaching zero. As productivity decreases,  management does the only thing they can; they add more staff to the  project in hopes of increasing productivity. But that new staff is not  versed in the design of the system. They dont know the difference  between a change that matches the design intent, and a change that  thwarts the design intent. Furthermore, they, and everyone else on the  team are under horrific pressure to increase productivity. So they all  make more and more messes, driving the productivity ever further  towards zero.Eventually the team rebels; and they inform management that they  cannot continue to develop in this horrific code base. They demand a  redesign. Management does not want to expend the resources on a whole  new redesign of the project, but they cannot deny that productivity is  terrible. Eventually they bend to the demands of the developers, and  authorize the grand redesign in the sky.A new tiger team is selected. Everyone wants to be on this team  because its a green-field project. They get to start over and create  something truly beautiful. But only the best and brightest are chosen  for the tiger team. Everyone else must continue to maintain the  current system.Now the two teams are in a race. The tiger team must build a new  system that does everything that the old system does. Not only that,  they have to keep up with the changes that are continuously being made  to the old system. Management will not replace the old system until  the new system can do everything that the old system does, on the day  of the change.This race can go on for a very long time. Ive seen it take 10 years.  And by the time its done, the original members of the tiger team are  long gone, and the current members are demanding that the new system  be redesigned because its such a mess.If you have experienced even one small part of the story I just told,  then you already know that spending time keeping your code clean is  not just cost effective; its a matter of professional survival.An Alternative - clean up your existing architectureRobert Martin as quoted above expresses an alternative to a project rewrite in his talk about software craftsmanship.  If you haven't seen this before its well worth watching all the way through.http://www.infoq.com/presentations/craftmanship-ethicsI also recommended some methods to perform gradual refactoring as opposed to a complete rewrite in my answer to a similar question.  What is the best/ethical way to estimate code clean-up?Getting StartedIn the past where I have started large greenfield projects (unsurprisingly it was also SOA).  I have found that the initial architecture prototyping and bedding down of the framework took a bit of time where I may redesign my approach, scrap my ideas half a dozen times over as I prototype and test out theories and best design practices.  Don't waste time writing specifications that will only be thrown awayUsing a big up front design in a new architecture will waste time and effort so taking an agile approach of functional requirements and specifications over a fully fleshed out specifications document is essential to not waste time and throw away bad ideas that looked great on paper but didn't work out in real life.Initial PrototypingThis initial prototyping stage I have found generally doesn't produce anything that is deliverable to the customer until its actually finished and working.  However what you should end up with at the end should be a well thought out architecture, with easily configurable modules and refactoring the existing applications into the new architecture should lend itself quite easily to an agile methodology with customer visible results.Followed by a ramp up in velocityOnce your prototype is finished and your architecture bedded down you will find that you will have a transition period where you will need to educate the rest of your team as to how best to migrate or refactor your existing projects to fit in with the new framework.So you can expect a slow ramp up speed followed by increased velocity nearing the end of the project as you get more developers online and implementing your design.  