Abstract Bad requirements quality can cause expensive consequences during the software development lifecycle, especially if iterations are long and feedback comes late. We aim at a light-weight static requirements analysis approach that allows for rapid checks immediately when requirements are written down. We transfer the concept of code smells to requirements engineering as Requirements Smells . To evaluate the benefits and limitations, we define Requirements Smells, realize our concepts for a smell detection in a prototype called Smella and apply Smella in a series of cases provided by three industrial and a university context. The automatic detection yields an average precision of 59 at an average recall of 82 with high variation. The evaluation in practical environments indicates benefits such as an increase of the awareness of quality defects. Yet, some smells were not clearly distinguishable. Lightweight smell detection can uncover many practically relevant requirements defects in a reasonably precise way. Although some smells need to be defined more clearly, smell detection provides a helpful means to support quality assurance in requirements engineering, for instance, as a supplement to reviews.
Various authors have worked on QA for software requirements by applying manual techniques. Some put their focus on the classification of quality into characteristics @cite_22 , others develop comprehensive checklists @cite_59 @cite_19 @cite_43 @cite_70 @cite_50 . Regarding QA, some develop constructive QA approaches, such as creating new RE languages, e.g. @cite_46 , to prevent issues up front, others develop approaches to make analytic QA, such as reviews, more effective @cite_15 . In a recent empirical study on analytical QA, @cite_47 manually investigate the presence of defects in use cases. To sum it up, these works on manual QA provide analytical and constructive methods, as well as (varying) lists for defects. They strengthen our confidence that today's requirements artifacts are vulnerable to quality defects.
@cite_47 @cite_22 @cite_15 @cite_70 @cite_19 @cite_43 @cite_59 @cite_50 @cite_46
2108057276 1864331247 2057415299 2114647523 2054070929 2760783434 2495715420 2014418954 2037933327 1999785511 2607739056 841012168 1587460519 2099942748 1975988951 2157209130 2156723666 2148488947 2055647675 2951595269
Quality assurance (QA) tasks, such as testing, profiling, and performance evaluation, have historically been done in-house on developer-generated workloads and regression suites. Since this approach is inadequate for many systems, tools and processes are being developed to improve software quality by increasing user participation in the QA process. A limitation of these approaches is that they focus on isolated mechanisms, not on the coordination and control policies and tools needed to make the global QA process efficient, effective, and scalable. To address these issues, we have initiated the Skoll project, which is developing and validating novel software QA processes and tools that leverage the extensive computing resources of worldwide user communities in a distributed, continuous manner to significantly and rapidly improve software quality. This paper provides several contributions to the study of distributed continuous QA. First, it illustrates the structure and functionality of a generic around-the-world, around-the-clock QA process and describes several sophisticated tools that support this process. Second, it describes several QA scenarios built using these tools and process. Finally, it presents a feasibility study applying these scenarios to a 1MLOC+ software package called ACE+TAO. While much work remains to be done, the study suggests that the Skoll process and tools effectively manage and control distributed, continuous QA processes. Using Skoll we rapidly identified problems that had taken the ACE+TAO developers substantially longer to find and several of which had previously not been found. Moreover, automatic analysis of QA task results often provided developers information that quickly led them to the root cause of the problems. [Context and motivation] In market-driven software development it is crucial, but challenging, to find the right balance among competing quality requirements (QR). [Problem] In order to identify the unique challenges associated with the selection, trade-off, and management of quality requirements an interview study is performed. [Results] This paper describes how QR are handled in practice. Data is collected through interviews with five product managers and five project leaders from five software companies. [Contribution] The contribution of this study is threefold: Firstly, it includes an examination of the interdependencies among quality requirements perceived as most important by the practitioners. Secondly, it compares the perceptions and priorities of quality requirements by product management and project management respectively. Thirdly, it characterizes the selection and management of quality requirements in down-stream development activities. Question answering (QA) helps one go beyond traditional keywords-based querying and retrieve information in more precise form than given by a document or a list of documents. Several community-based QA (CQA) services have emerged allowing information seekers pose their information need as questions and receive answers from their fellow users. A question may receive multiple answers from multiple users and the asker or the community can choose the best answer. While the asker can thus indicate if he was satisfied with the information he received, there is no clear way of evaluating the quality of that information. We present a study to evaluate and predict the quality of an answer in a CQA setting. We chose Yahoo! Answers as such CQA service and selected a small set of questions, each with at least five answers. We asked Amazon Mechanical Turk workers to rate the quality of each answer for a given question based on 13 different criteria. Each answer was rated by five different workers. We then matched their assessments with the actual asker's rating of a given answer. We show that the quality criteria we used faithfully match with asker's perception of a quality answer. We furthered our investigation by extracting various features from questions, answers, and the users who posted them, and training a number of classifiers to select the best answer using those features. We demonstrate a high predictability of our trained models along with the relative merits of each of the features for such prediction. These models support our argument that in case of CQA, contextual information such as a user's profile, can be critical in evaluating and predicting content quality. There is a documented gap between academic and practitioner views on software testing. This paper tries to close the gap by investigating both views regarding the benefits and limits of test automation. The academic views are studied with a systematic literature review while the practitioners views are assessed with a survey, where we received responses from 115 software professionals. The results of the systematic literature review show that the source of evidence regarding benefits and limitations is quite shallow as only 25 papers provide the evidence. Furthermore, it was found that benefits often originated from stronger sources of evidence (experiments and case studies), while limitations often originated from experience reports. We believe that this is caused by publication bias of positive results. The survey showed that benefits of test automation were related to test reusability, repeatability, test coverage and effort saved in test executions. The limitations were high initial invests in automation setup, tool selection and training. Additionally, 45 of the respondents agreed that available tools in the market offer a poor fit for their needs. Finally, it was found that 80 of the practitioners disagreed with the vision that automated testing would fully replace manual testing. Objectives: In this survey, we reviewed the current state of the art in biomedical QA (Question Answering), within a broader framework of semantic knowledge-based QA approaches, and projected directions for the future research development in this critical area of intersection between Artificial Intelligence, Information Retrieval, and Biomedical Informatics. Materials and methods: We devised a conceptual framework within which to categorize current QA approaches. In particular, we used ''semantic knowledge-based QA'' as a category under which to subsume QA techniques and approaches, both corpus-based and knowledge base (KB)-based, that utilize semantic knowledge-informed techniques in the QA process, and we further classified those approaches into three subcategories: (1) semantics-based, (2) inference-based, and (3) logic-based. Based on the framework, we first conducted a survey of open-domain or non-biomedical-domain QA approaches that belong to each of the three subcategories. We then conducted an in-depth review of biomedical QA, by first noting the characteristics of, and resources available for, biomedical QA and then reviewing medical QA approaches and biological QA approaches, in turn. The research articles reviewed in this paper were found and selected through online searches. Results: Our review suggested the following tasks ahead for the future research development in this area: (1) Construction of domain-specific typology and taxonomy of questions (biological QA), (2) Development of more sophisticated techniques for natural language (NL) question analysis and classification, (3) Development of effective methods for answer generation from potentially conflicting evidences, (4) More extensive and integrated utilization of semantic knowledge throughout the QA process, and (5) Incorporation of logic and reasoning mechanisms for answer inference. Conclusion: Corresponding to the growth of biomedical information, there is a growing need for QA systems that can help users better utilize the ever-accumulating information. Continued research toward development of more sophisticated techniques for processing NL text, for utilizing semantic knowledge, and for incorporating logic and reasoning mechanisms, will lead to more useful QA systems. [Context and motivation] Research on eliciting requirements from a large number of online reviews using automated means has focused on functional aspects. Assuring the quality of an app is vital for its success. This is why user feedback concerning quality issues should be considered as well [Question problem] But to what extent do online reviews of apps address quality characteristics? And how much potential is there to extract such knowledge through automation? [Principal ideas results] By tagging online reviews, we found that users mainly write about "usability" and "reliability", but the majority of statements are on a subcharacteristic level, most notably regarding "operability", "adaptability", "fault tolerance", and "interoperability". A set of 16 language patterns regarding "usability" correctly identified 1,528 statements from a large dataset far more efficiently than our manual analysis of a small subset. [Contribution] We found that statements can especially be derived from online reviews about qualities by which users are directly affected, although with some ambiguity. Language patterns can identify statements about qualities with high precision, though the recall is modest at this time. Nevertheless, our results have shown that online reviews are an unused Big Data source for quality requirements. In engineering of safety critical systems, regulatory standards often put requirements on both traceable specification-based testing, and structural coverage on program units. Automated test generation techniques can be used to generate inputs to cover the structural aspects of a program. However, there is no conclusive evidence on how automated test generation compares to manual test design, or how testing based on the program implementation relates to specification-based testing. In this paper, we investigate specification -- and implementation-based testing of embedded software written in the IEC 61131-3 language, a programming standard used in many embedded safety critical software systems. Further, we measure the efficiency and effectiveness in terms of fault detection. For this purpose, a controlled experiment was conducted, comparing tests created by a total of twenty-three software engineering master students. The participants worked individually on manually designing and automatically generating tests for two IEC 61131-3 programs. Tests created by the participants in the experiment were collected and analyzed in terms of mutation score, decision coverage, number of tests, and testing duration. We found that, when compared to implementation-based testing, specification-based testing yields significantly more effective tests in terms of the number of faults detected. Specifically, specification-based tests more effectively detect comparison and value replacement type of faults, compared to implementation-based tests. On the other hand, implementation-based automated test generation leads to fewer tests (up to 85 improvement) created in shorter time than the ones manually created based on the specification. Developing high-quality requirements specifications often demands a thoughtful analysis and an adequate level of expertise from analysts. Although requirements modeling techniques provide mechanisms for abstraction and clarity, fostering the reuse of shared functionality (e.g., via UML relationships for use cases), they are seldom employed in practice. A particular quality problem of textual requirements, such as use cases, is that of having duplicate pieces of functionality scattered across the specifications. Duplicate functionality can sometimes improve readability for end users, but hinders development-related tasks such as effort estimation, feature prioritization, and maintenance, among others. Unfortunately, inspecting textual requirements by hand in order to deal with redundant functionality can be an arduous, time-consuming, and error-prone activity for analysts. In this context, we introduce a novel approach called ReqAligner that aids analysts to spot signs of duplication in use cases in an automated fashion. To do so, ReqAligner combines several text processing techniques, such as a use case-aware classifier and a customized algorithm for sequence alignment. Essentially, the classifier converts the use cases into an abstract representation that consists of sequences of semantic actions, and then these sequences are compared pairwise in order to identify action matches, which become possible duplications. We have applied our technique to five real-world specifications, achieving promising results and identifying many sources of duplication in the use cases. The purpose of this work is to identify potential evaluation criteria for interactive, analytical question-answering (QA) systems by analyzing evaluative comments made by users of such a system. Qualitative data collected from intelligence analysts during interviews and focus groups were analyzed to identify common themes related to performance, use, and usability. These data were collected as part of an intensive, three-day evaluation workshop of the High-Quality Interactive Question Answering (HITIQA) system. Inductive coding and memoing were used to identify and categorize these data. Results suggest potential evaluation criteria for interactive, analytical QA systems, which can be used to guide the development and design of future systems and evaluations. This work contributes to studies of QA systems, information seeking and use behaviors, and interactive searching. © 2007 Wiley Periodicals, Inc. Automated reliability assessment is essential for systems that entail dynamic adaptation based on runtime mission-specific requirements. One approach along this direction is to monitor and assess the system using machine learning-based software defect prediction techniques. Due to the dynamic nature of software data collected, Instance-based learning algorithms are proposed for the above purposes. To evaluate the accuracy of these methods, the paper presents an empirical analysis of four different real-time software defect data sets using different predictor models. The results show that a combination of 1R and Instance-based learning along with Consistency-based subset evaluation technique provides a relatively better consistency in achieving accurate predictions as compared with other models. No direct relationship is observed between the skewness present in the data sets and the prediction accuracy of these models. Principal Component Analysis (PCA) does not show a consistent advantage in improving the... This is a preprint, submitted on 2015-03-22. Question Answering as a sub-field of information retrieval and information extraction is recently enjoying renewed pop- ularity, triggered by the publicized success of IBM Watson in the Jeopardy! competition. But Question Answering re- search is now proceeding in several semi-independent tiers depending on the precise task formulation and constraints on the knowledge base, and new researchers entering the field can focus only on various restricted sub-tasks as no modern full-scale software system for QA has been openly available until recently. By our YodaQA system that we introduce here, we seek to re- unite and boost research efforts in Question Answering, pro- viding a modular, open source pipeline for this task — allow- ing integration of various knowledge base paradigms, an- swer production and analysis strategies and using a machine learned models to rank the answers. Within this pipeline, we also supply a baseline QA system inspired by DeepQA with solid performance and propose a reference experimen- tal setup for easy future performance comparisons. In this paper, we review the available open QA platforms, present the architecture of our pipeline, the components of the baseline QA system, and also analyze the system perfor- mance on the reference dataset. The field of automated software repair lacks a set of common benchmark problems. Although benchmark sets are used widely throughout computer science, existing benchmarks are not easily adapted to the problem of automatic defect repair, which has several special requirements. Most important of these is the need for benchmark programs with reproducible, important defects and a deterministic method for assessing if those defects have been repaired. This article details the need for a new set of benchmarks, outlines requirements, and then presents two datasets, ManyBugs and IntroClass , consisting between them of 1,183 defects in 15 C programs. Each dataset is designed to support the comparative evaluation of automatic repair algorithms asking a variety of experimental questions. The datasets have empirically defined guarantees of reproducibility and benchmark quality, and each study object is categorized to facilitate qualitative evaluation and comparisons by category of bug or program. The article presents baseline experimental results on both datasets for three existing repair methods, GenProg, AE, and TrpAutoRepair, to reduce the burden on researchers who adopt these datasets for their own comparative evaluations. Quality improvement such as increased reliability and maintainability are of utmost importance in software development. In this field, which was previously ad hoc and unpredictable rather than customerdoriented, increasing competition and focus on customer satisfaction have motivated management to put more emphasis on quality issues. This paper provides insight in techniques for dealing with what customers call generic quality attributes and what software engineers call nonfunctional requirements. Since requirements management more than many other disciplines in software engineering need practical insight, examples are provided for dealing with four nonfunctional requirements in large telecommunication systems, namely performance, usability, reliability, and maintainability. Guidelines are presented for specifying, constructing, analyzing, measuring and tracing nonfunctional requirements. Many examples from telecommunication system development show how to specifically and pragmatically deal with nonfunctional requirements. Extracts from three common standards (SEI CMM, ISO 9000–3, BELLCORE suite) are added in the form of a checklist for underlining the importance of proper requirements management from a customer viewpoint. In this paper we discuss our study of the problems 12 software companies experienced in software development. In total we present qualitative data collected from 45 focus groups that involved over 200 software staff. We look at how different practitioner groups respond to software process improvement problems. We show our classification and analysis of this data using correspondence analysis. Correspondence analysis is a graphical data representation method new to software development research. The aim of the work we present is to develop a more holistic understanding of the problems practitioners are experiencing in their attempts to improve their software processes. Our main finding is that there is an association between a company’s capability maturity and patterns of reported problems. Organizational problems are more associated with high maturity companies than with low maturity companies. Low maturity companies are closely linked to problems relating directly to projects such as documentation, timescales, tools and technology. Our findings also confirm differences in practitioner group problems. Senior managers cite problems with goals, culture and politics. Project managers are concerned with timescales, change management, budgets and estimates. Developers are experiencing problems with requirements, testing, documentation, communication, tools and technology. These associations are displayed graphically through correspondence analysis maps. Though very important in software engineering, linking artifacts of the same type (clone detection) or different types (traceability recovery) is extremely tedious, error-prone, and effort-intensive. Past research focused on supporting analysts with techniques based on Natural Language Processing (NLP) to identify candidate links. Because many NLP techniques exist and their performance varies according to context, it is crucial to define and use reliable evaluation procedures. The aim of this paper is to propose a set of seven principles for evaluating the performance of NLP techniques in identifying equivalent requirements. In this paper, we conjecture, and verify, that NLP techniques perform on a given dataset according to both ability and the odds of identifying equivalent requirements correctly. For instance, when the odds of identifying equivalent requirements are very high, then it is reasonable to expect that NLP techniques will result in good performance. Our key idea is to measure this random factor of the specific dataset(s) in use and then adjust the observed performance accordingly. To support the application of the principles we report their practical application to a case study that evaluates the performance of a large number of NLP techniques for identifying equivalent requirements in the context of an Italian company in the defense and aerospace domain. The current application context is the evaluation of NLP techniques to identify equivalent requirements. However, most of the proposed principles seem applicable to evaluating any estimation technique aimed at supporting a binary decision (e.g., equivalent nonequivalent), with the estimate in the range [0,1] (e.g., the similarity provided by the NLP), when the dataset(s) is used as a benchmark (i.e., testbed), independently of the type of estimator (i.e., requirements text) and of the estimation method (e.g., NLP). Under the studies of general core activities including software inspection, review and testing to achieve quality objectives in small-medium size enterprises (SMEs), the paper presents a contemporary view of such companies against quality measures. The results from a local empirical investigation of quality standards in the Turkish software industry are reported. Around 150 software companies have been approached from which 17 detailed feedback inform that in order to ensure software quality, standards including internationally recognized International Standards Organization (ISO) and Capability Maturity Model Integration (CMMI) are given credit. However the substantial workload and resources required to obtain them are also reported as serious; downscaled frameworks of such large models proposed in the literature are not well known by the SMEs either. The paper also discusses "work around" that bypasses such standards to ease delivery of products while keeping certificates as labels just to acquire new jobs for the business. Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http: defects4j.org. Question Answering (QA) is retrieving answers to natural language questions from a collection of documents rather than retrieving relevant documents containing the keywords of the query which is performed by search engines. What a user usually wants is often a precise answer to a question. For example, given the question “Who won the nobel prize in peace in 2006?” what a user really wants is the answer “Dr. Muhammad Yunus”, in stead of reading through lots of documents that contain the words “win”, “nobel”,“prize”, “peace” and “2006” etc. This means that question answering systems will possibly be integral to the next generation of search engines. The Text Retrieval Conference, TREC1 QA track is the major large-scale evaluation environment for open-domain question answering systems. The questions in the TREC-2007 QA track are clustered by target, which is the overall theme or topic of the questions. The track has three types of questions: 1. factoid that require only one correct response, 2. list that require a non redundant list of correct responses and 3. other questions that require a non redundant list of facts about the target that has not already been discovered by a previous answer. We took the approach of designing a question answering system that is based on document tagging and question classification. Question classification extracts useful information (i.e. answer type) from the question about how to answer the question. Document tagging extracts useful information from the documents, which will be used in finding the answer to the question. We used different available tools to tag the documents. Our system classifies the questions using manually developed rules. Recently, many formal methods, such as the SCR (Software Cost Reduction) requirements method, have been proposed for improving the quality of software specifications. Although improved specifications are valuable, the ultimate objective of software development is to produce software that satisfies its requirements. To evaluate the correctness of a software implementation, one can apply black-box testing to determine whether the implementation, given a sequence of system inputs, produces the correct system outputs. This paper describes a specification-based method for constructing a suite of test sequences , where a test sequence is a sequence of inputs and outputs for testing a software implementation. The test sequences are derived from a tabular SCR requirements specification containing diverse data types, i.e., integer, boolean, and enumerated types. From the functions defined in the SCR specification, the method forms a collection of predicates called branches , which “cover” all possible software behaviors described by the specification. Based on these predicates, the method then derives a suite of test sequences by using a model checker's ability to construct counterexamples. The paper presents the results of applying our method to four specifications, including a sizable component of a contractor specification of a real system. Due to the difficulty of repairing defect, many research efforts have been devoted into automatic defect repair. Given a buggy program that fails some test cases, a typical automatic repair technique tries to modify the program to make all tests pass. However, since the test suites in real world projects are usually insufficient, aiming at passing the test suites often leads to incorrect patches. In this paper we aim to produce precise patches, that is, any patch we produce has a relatively high probability to be correct. More concretely, we focus on condition synthesis, which was shown to be able to repair more than half of the defects in existing approaches. Our key insight is threefold. First, it is important to know what variables in a local context should be used in an "if" condition, and we propose a sorting method based on the dependency relations between variables. Second, we observe that the API document can be used to guide the repair process, and propose document analysis technique to further filter the variables. Third, it is important to know what predicates should be performed on the set of variables, and we propose to mine a set of frequently used predicates in similar contexts from existing projects. We develop a novel program repair system, ACS, that could generate precise conditions at faulty locations. Furthermore, given the generated conditions are very precise, we can perform a repair operation that is previously deemed to be too overfitting: directly returning the test oracle to repair the defect. Using our approach, we successfully repaired 18 defects on four projects of Defects4J, which is the largest number of fully automatically repaired defects reported on the dataset so far. More importantly, the precision of our approach in the evaluation is 78.3 , which is significantly higher than previous approaches, which are usually less than 40 .