As advances in technology allow for the collection, storage, and analysis of vast amounts of data, the task of screening and assessing the significance of discovered patterns is becoming a major challenge in data mining applications. In this work, we address significance in the context of frequent itemset mining. Specifically, we develop a novel methodology to identify a meaningful support threshold s* for a dataset, such that the number of itemsets with support at least s* represents a substantial deviation from what would be expected in a random dataset with the same number of transactions and the same individual item frequencies. These itemsets can then be flagged as statistically significant with a small false discovery rate. We present extensive experimental results to substantiate the effectiveness of our methodology.
The problem of spurious discoveries in the mining of significant patterns is studied in @cite_24 . The paper is concerned with the discovery of significant pairs of items, where significance is measured through the @math -value, that is, the probability of occurrence of the observed support in a random dataset. Significant pairs are those whose @math -values are below a certain threshold that can be suitably chosen to bound the FWER, or to bound the FDR. The authors compare the relative power of the two metrics through experimental results, but do not provide methods to set a meaningful support threshold, which is the most prominent feature of our approach.
@cite_24
1547785593
The problem of spurious apparent patterns arising by chance is a fundamental one for pattern detection. Classical approaches, based on adjustments such as the Bonferroni procedure, are arguably not appropriate in a data mining context. Instead, methods based on the false discovery rate - the proportion of flagged patterns which do not represent an underlying reality - may be more relevant. We describe such procedures and illustrate their application on a marketing dataset.