In a recent empirical study we found that evaluating abstractions of Model-Driven Engineering (MDE) is not as straight forward as it might seem. In this paper, we report on the challenges that we as researchers faced when we conducted the aforementioned field study. In our study we found that modeling happens within a complex ecosystem of different people working in different roles. An empirical evaluation should thus mind the ecosystem, that is, focus on both technical and human factors. In the following, we present and discuss five lessons learnt from our recent work.
In this paper, we investigate model-driven engineering, reporting on an exploratory case-study conducted at a large automotive company. The study consisted of interviews with 20 engineers and managers working in different roles. We found that, in the context of a large organization, contextual forces dominate the cognitive issues of using model-driven technology. The four forces we identified that are likely independent of the particular abstractions chosen as the basis of software development are the need for diffing in software product lines, the needs for problem-specific languages and types, the need for live modeling in exploratory activities, and the need for point-to-point traceability between artifacts. We also identified triggers of accidental complexity, which we refer to as points of friction introduced by languages and tools. Examples of the friction points identified are insufficient support for model diffing, point-to-point traceability, and model changes at runtime. While Model-Driven Development (MDD) is an increasingly popular software development approach, its impact on the development process in large-scale, industrial practice is not yet clear. For this study the application of MDD in a large-scale industrial software development project is analyzed over a period of two years. Applying a grounded theory approach we identified 14 factors which impact the architectural process. We found that scope creep is more likely to occur, late changes can imply more extensive rework and that business engineers need to be more aware of the technical impact of their decisions. In addition, the introduced Domain-Specific Language (DSL) provides a new common idiom that can be used by more team members and will ease communication among team members and with clients. Also, modelers need to be much more explicit and complete in their descriptions. Parallel development of a code generator and defining a proper meta-model require additional time investments. Lastly, the more central role of software architecture design documentation requires more structured, detailed and complete architectural information and consequently, more frequent reviews.