I'm not against agile/TDD and use it under most circumstances. However, in some circumstances I feel it doesn't work well.  For example, the other day I was building a fairly complex data processing app. In an ideal world I'd layout the foundation, design the components, write unit tests and demo on each sprints as I finish features. In practice however, I don't think this works before the first version is released because:The foundation/component design changes frequently as I figure outhow to do things better, and simple test drivers are much more usefulthan richer unit testsIf you are building a MVC app., the foundationisn't going to change, but with an app. where you more or less buildthe framework yourself, and don't quite know how to do things bestbeforehand this isn't the case. I feel simple test drivers with noautomated verification are much more useful than richer unit tests atthis stage.Until the basic infrastructure is in place, it's hard to breakdown things into individual featuresIf you already have an MVCframework, this is easy. But imagine how the the first MVC frameworkever was developed. It must have been much less useful to thinkthings as individual features in the initial stages, because themost important thing was to work out the principle and philosophy of separatingModel, View and Control, and how to do that. Demos help very little until a sizable part of the app. is doneIn aUI, you can just put up a mocked up interfaces andget lots of valuable feedback. However, e.g. in a web crawler, you can'treally do a meaningful demo to non-technical people until quite a bitis done, and even if you did, they usually won't be able to give useful inputs anyway. Sure,you can demo to your developer colleagues, but it's hard to giveinput based on some log output. It's usually much more productive toexplain how you did things and how you plan to do things perhapsshowing illustrations and code, and get feedback this way.Or am I just doing it wrong?
Can you apply agile/TDD in all circumstances?
agile;tdd
You seem to be developing in a very agile way, actually. A non-agile way would be to spend weeks drawing the design on paper and sticking to it when you program it and (inevitably) find that the whole thing is a house of cards.That is speaking about the general principle. For the concrete meanings of the terms like test driver, demo, feature etc. they are specific to the situation.They are always specific to the situation. The common meaning is the one applicable to the most common kind of application, which is business information system. That kind of application has a generic foundation that you already have and on that are built individual forms, reports and workflows, that can be added one at a time. So you can add the few most basic things, demo it, than add some more, demo again etc.But there are many applications where that is not true. You can still use agile process, but kind of internally in the project. Your user stories become defined in terms of programming APIs implemented, your tests test just those raw APIs and such. It is agile process in the sense you implement the minimal functionality for each component and refactor it as you need more functionality from it, and it may be test driven in that you have unit tests for the components. You just don't have the eventual customer involved (yet), because you have nothing to show them. You eventually will and than you'll still be refactoring the foundation, because some fundamental problems can still appear than.