One of the key concepts in testing is that of adequate test sets. A test selection criterion decides which test sets are adequate. In this paper, a language schema for specifying a large class of test selection criteria is developed; the schema is based on two operations for building complex criteria from simple ones. Basic algebraic properties of the two operations are derived. In the second part of the paper, a simple language-an instance of the general schema-is studied in detail, with the goal of generating small adequate test sets automatically. It is shown that one version of the problem is intractable, while another is solvable by an efficient algorithm. An implementation of the algorithm is described.
Gourlay @cite_11 presents a precise framework for the discussion of issues in testing. In his terminology, our test selection criteria are a special form of the test methods for the set-choice construction testing system. Gourlay reinterprets previously published discussions about the suitability of various test selection criteria. In our approach, we do not attempt to decide a priori which criteria are sufficient --- we leave that decision to the test designer. That is why we emphasize the importance of a language in which criteria are specified.
@cite_11
1715319291 1871418963 2949831835 2325933960
Selectional constraints specify, for a particular domain, the combinations of semantic classes acceptable in subject-verb-object relationships and other syntactic structures. These constraints are important in blocking incorrect analyses in natural language processing systems. However, these constraints are domain-specific and hence must be developed anew when a system is ported to a new domain. A discovery procedure for selectional constraints is therefore essential in enhancing the portability of such systems.This paper describes a semi-automated procedure for collecting the co-occurrence patterns from a sample of texts in a domain, and then using these patterns as the basis for selectional constraints in analyzing further texts. We discuss some of the difficulties in automating the collection process, and describe two experiments that measure the completeness of these patterns and their effectiveness compared with manually-prepared patterns. We then describe and evaluate a procedure for selectional constraint relaxation, intended to compensate for gaps in the set of patterns. Finally, we suggest how these procedures could be combined with a system that queries a domain expert, in order to produce a more efficient discovery procedure. Summary We consider a multiple-hypothesis testing setting where the hypotheses are ordered and one is only permitted to reject an initial contiguous block of hypotheses. A rejection rule in this setting amounts to a procedure for choosing the stopping point k. This setting is inspired by the sequential nature of many model selection problems, where choosing a stopping point or a model is equivalent to rejecting all hypotheses up to that point and none thereafter. We propose two new testing procedures and prove that they control the false discovery rate in the ordered testing setting. We also show how the methods can be applied to model selection by using recent results on p-values in sequential model selection settings. We consider a multiple hypothesis testing setting where the hypotheses are ordered and one is only permitted to reject an initial contiguous block, H_1, ,H_k, of hypotheses. A rejection rule in this setting amounts to a procedure for choosing the stopping point k. This setting is inspired by the sequential nature of many model selection problems, where choosing a stopping point or a model is equivalent to rejecting all hypotheses up to that point and none thereafter. We propose two new testing procedures, and prove that they control the false discovery rate in the ordered testing setting. We also show how the methods can be applied to model selection using recent results on p-values in sequential model selection settings. We present an instance-specific test-time dynamic feature selection algorithm. Our algorithm sequentially chooses features given previously selected features and their values. It stops the selection process to make a prediction according to a user-specified accuracy-cost trade-off. We cast the sequential decision-making problem as a Markov Decision Process and apply imitation learning techniques. We address the problem of learning and inference jointly in a simple multiclass classification setting. Experimental results on UCI datasets show that our approach achieves the same or higher accuracy using only a small fraction of features than static feature selection methods.