Definitions and approaches to model quality in model-based software development - A review of literature <s> Examples of Empirical Evidence <s> Use cases have become an important tool in software engineering. There has been much focus on the diagram notation but relatively little on use-case descriptions. As part of a welcome and important research project into the use of scenarios in requirements engineering, the CREWS (Co-operative Requirements Engineering With Scenarios, an EU funded ESPRIT project 21903) team has proposed a set of guidelines for writing use-case descriptions. This paper describes the replication of a CREWS project experiment that suggests CREWS use-case authoring guidelines improve the completeness of use-case descriptions. Our results show that the CREWS guidelines do not necessarily improve the use-case descriptions, only that the subjects implemented varying numbers of guidelines in their use-case descriptions. Subjects in the control group implemented a significant percentage of the guidelines by `chance.' To further justify our results, we also apply a different marking scheme to compare with the CREWS approach. The results from the alternative marking approach show that there was no significant difference between the qualities of the use-case descriptions across the various groups. <s> BIB001 </s> Definitions and approaches to model quality in model-based software development - A review of literature <s> Examples of Empirical Evidence <s> Use cases are the main requirements vehicle of the UML and are used widely to specify systems. Hence, the need to write clear and accurate use case descriptions has a significant impact for many practitioners. However, many have pointed to weaknesses in the support offered to those writing use cases, and a number of authors advocate the use of rules in the composition and structuring of use case descriptions. These rules constrain the user, by only allowing certain grammatical constructions, typically guiding the structure or the style of the description For example, the CREWS research project pioneered Use Case Authoring Guidelines, suggesting that the adoption of such guidelines improved resulting use case descriptions. Replication of CREWS studies appeared to confirm the view that use case descriptions were improved through the application of guideline sets, but also noted that learning such rules presented a significant overhead. Hence, a leaner set of guidelines (the CP rules) was developed. ::: ::: This paper describes empirical work to assess the utility of these two sets of writing guidelines (CREWS and CP). In particular, descriptions are assessed against a set of established criteria--a use case quality description checklist, which the authors described in a previous paper. ::: ::: Our findings suggest that the leaner set of guidelines performs at least as well in terms of their ability to produce clear and accurate (comprehensible) descriptions. Hence, that a tractable set of rules may prove applicable to the industrial context, which could lead to effective validation of use cases. <s> BIB002 </s> Definitions and approaches to model quality in model-based software development - A review of literature <s> Examples of Empirical Evidence <s> Model-Driven Engineering (MDE) has been promoted as a solution to handle the complexity of software development by raising the abstraction level and automating labor-intensive and error-prone tasks. However, few efforts have been made at collecting evidence to evaluate its benefits and limitations, which is the subject of this review. We searched several publication channels in the period 2000 to June 2007 for empirical studies on applying MDE in industry, which produced 25 papers for the review. Our findings include industry motivations for investigating MDE and the different domains it has been applied to. In most cases the maturity of third-party tool environments is still perceived as unsatisfactory for large-scale industrial adoption. We found reports of improvements in software quality and of both productivity gains and losses, but these reports were mainly from small-scale studies. There are a few reports on advantages of applying MDE in larger projects, however, more empirical studies and detailed data are needed to strengthen the evidence. We conclude that there is too little evidence to allow generalization of the results at this stage. <s> BIB003
Some of the proposed processes are either developed by industry or are evaluated in industrial cases:  Staron et al. performed a case in Volvo Information Technology and concluded that the project was successful [P34] .  Lange et al. evaluated the feedback from industry regarding their approach and tool as positive [P21] .  Berenbach reports that the proposed metrics for requirement modelling were applied in three different projects at Siemens and write that "as the team members gained experience with the measurement tools and increased ability with the UML, their productivity and confidence rose dramatically" [P4] . The improvement is however not quantified. The feedbacks are positive but we so far lack detailed case studies on the impact of development processes on the quality of models. Lange et al. have performed a controlled experiment with students as subjects on the effect of (UML) modelling conventions on modelling effort and defect density [P18] . The conventions they have included in the experiment yield correctness (e.g.; an abstract class should not be a leaf), comprehensibility (e.g.; the number of sequence diagrams per use case should indicate how well the functionality of a use case is documented or described) and design issues (e.g.; low coupling). The experiment results indicate that decreased defect density is attainable at the cost of increased effort when using modelling conventions, and moreover, that this trade-off is increased if tool-support is provided. Since we do not have the full list of conventions and the detailed results of experiments, it is difficult to say which types of errors are prevented. In another student experiment described in [P6] , DuBois et al. investigated whether using conventions has any effect on representational quality of a model; defined as the clarity, completeness and validity of the information the model is meant to represent. They did not observe any difference in representational quality and concluded that conventions should rather focus on identifying and consistently representing those types of information required for the model's future usage, e.g., in implementation and testing. Purchase et al. performed several student experiments on UML diagrams and the students' preference of one diagram over another, checking layout issues such as the number of crosses, the number of bends, use of colours or fonts and the width of diagrams, specially for class and collaboration diagrams [P28] . Also five UML experts performed the experiment. The comprehension task was that of matching a given textual specification against a set of diagrams, indicating whether each diagram correctly matches the specification or not. The set of diagrams included both correct and incorrect diagrams. Both the response time and the accuracy of responses were measured. Two examples of notations used in the experiment are shown in Figure VIII . When matching diagrams to the specifications, the (a) notations performed better, while when identifying errors in the diagrams, the (b) notations which are less intuitive and more ambiguous had better performance. It appears that subjects are less at ease with these notations and are more likely to detect errors in the diagrams. There have also been experiments by Cox and Phalp on applying conventions and styles to textual use cases BIB001 BIB002 . The results showed that their impact on quality is not always obvious and in an experiment comparing two guideline sets, the leaner one performed as well as the other. Thus there is little empirical evidence in the covered literature regarding the benefits of conventions and the results of few student experiments are not conclusive. The quality impact of conventions seems to depend on the task, the complexity of conventions and tool support as well and empirical studies should describe these factors better in order to help evaluating the usefulness and cost. Peleg and Dori write that two major open questions related to diagram multiplicity vs. singularity have been 1) whether or not a single diagram, rather than a combination of several diagrams, enables the synthesis of a better system specification and 2) which of the two alternative approaches yields a specification that is easier to comprehend [P27] . They have addressed these two questions through a double-blind controlled experiment with students as subjects. The two approaches for modelling used in the experiment are OPM and OMT extended with Timed Statecharts. The students were divided in two groups: The participants of the first group were asked to specify a system described textually in the problem statement using OMT, while the participants in the second group were asked to specify the same system using OPM. The quality of the resulting specifications was thoroughly analyzed. The results showed that OPM was significantly better than OMT in terms of overall specification quality; evaluated by the number of errors in the specifications such as missing events or missing feature. For comprehension experiment, the first group received specifications of a system in OPM and the second in OMT and they were requested to answer a questionnaire. The specification comprehension results show that there were significant differences between the two methods in specific issues. The study is inconclusive regarding overall specification comprehension. Retrieving information from a single OMT diagram was easier in some cases than retrieving the same information from the single OPM model, which is more complex. The relatively large number of issues for which no difference between the modelling languages exists underlines the advantage of focusing on a single diagram over spreading the specification information over several diagrams. Reinhartz-Berger and Dori have also performed an experiment with students comparing UML and OPM for modelling of web applications [P30] . The goal of the experiment was to compare OPM to UML with respect to two aspects: (1) comprehension, namely the level of comprehending a given model expressed in each language, and (2) construction, i.e., the quality and ease of modelling a given system in each language. In some questions when evaluating comprehension, UML scored higher but the general conclusion while OPM was easier to understand and apply by untrained users. Both experiments show that there are cases in which the existence of separate views can potentially help answering certain questions about a specific aspect of a system, which is expressed in only one type of diagram. But if answering a question needs combining information from multiple diagrams, the singlediagram approach scores higher. The single-diagram approach may also help identifying incompleteness and inconsistency defects. DSMLs tend to have fewer diagrams and thus solve the problems introduced by using multiple diagram modelling languages. Unfortunately, our search did not return any empirical evidence from industry on the benefits of the single-diagram approach. In our survey on MDE experiences form industry presented in BIB003 , we described a case from Motorola that refers to the benefits of formalism for catching defects by simulation. Otherwise, the studies related to formal models include only examples to demonstrate the practice. The only industrial case is mentioned in [P24] where details are left out. Empirical evidence reported in the studies cover student experiments on the benefits of using stereotypes and industry experience reports on the benefits and risks of using DSMLs. Kuzniarz et al. have performed a controlled experiment with students as subjects in order to evaluate the influence of UML stereotypes on the understanding of UML models [P16] . Two sets of models were used in the experiment; one stereotyped and one not. Understandability of the designs was measured by two dependent variables: 1) total score (NRESP) -the number of correct answers for each subject when asked questions about the design; 2) time (TSEC) -the time (in seconds) which was required to fill in a questionnaire on the design of the systems. The results of the experiment support the claim that stereotypes with graphical icons for their representation play a significant role in comprehension of models. The subjects understood the stereotyped models better (the improvement achieved in this category was 52%) and they were more consistent in their answers when the stereotypes were involved. On average the relative amount of time for a correct answer was shorter for the stereotyped model. Staron et al. describe two additional experiments in order to verify the credibility of the design of the above experiment before replicating it in industry [P35] . The improvements were not significant but stereotypes helped in particular when it comes to having correct answers. The industrial experiment was conducted at Volvo Information Technology, at their site in Gothenburg, Sweden with only four professionals. The number of correct answers was higher for the stereotyped models but the time spent varied. Staron et al. concluded that in general stereotypes improve the correctness of understanding of UML models. Improvements were achieved in all experiments-and half of the results were statistically significant. We have found several cases describing experience and benefits of using UML profiles or DSMLs in industry, but often without quantitative data. Two cases are mentioned here while other examples are to be found in BIB003 . The benefits are often related to improved generation of correct code by adding semantics in domain concepts, and improved understandability by using the language of the domain:  In a paper describing development at Phillips, Jonkers at al. write that using small DSMLs, as opposed to a universal modelling language such as UML, brings the modelling discipline much closer to the domain experts and at the same time enables simpler maintenance and evolution of such models, which contributes to the desired productivity increase as well as to the agility of the model-driven development [P13] .  In a paper from Matsushita Electric Works in Japan, Safa writes that practitioners considered UML's object-oriented notations too far apart from the C procedural world used for implementation [P33] . A corporate language has been evolved naturally over the years to express requirements, designs and implementations matters. It has notations, conventions and semantics that map precisely the problem domains, and it evolves incrementally when the problem domain changes. The approach has risks as well. Due to the metamodeling delay necessary to define visual languages, editors and compilers, the domain-specific modelling tool lags behind practice, so it is at risk of being perceived as constraining, especially for practitioners used to drawing with free-format whiteboards, pen and paper and general purpose diagram tools like Microsoft© PowerPoint. Studies covered in this review included only examples on applying the practice but no empirical evidence.