There are several estimators of conditional probability from observed frequencies of features. In this paper, we propose using the lower limit of confidence interval on posterior distribution determined by the observed frequencies to ascertain conditional probability. In our experiments, this method outperformed other popular estimators.
Conditional probability is more popular in the database field than NLP. Apriori @cite_6 is the most practically used method for finding the relationship between features, when the true strength between features is known to be conditional probability. It uses MLE as the measure for the relationship. To overcome the problem of low frequency, Apriori ignores the rare features using a threshold value, which is called minimum supports. Although Apriori is efficient by ignoring low-frequency features, we sometimes need to compare a high frequency but low MLE value relationship with a low frequency but high MLE value relationship. Apriori simply ignores relation of the low frequency but high MLE value relationship.
@cite_6
1484413656
We consider the problem of discovering association rules between items in a large database of sales transactions. We present two new algorithms for solving thii problem that are fundamentally different from the known algorithms. Empirical evaluation shows that these algorithms outperform the known algorithms by factors ranging from three for small problems to more than an order of magnitude for large problems. We also show how the best features of the two proposed algorithms can be combined into a hybrid algorithm, called AprioriHybrid. Scale-up experiments show that AprioriHybrid scales linearly with the number of transactions. AprioriHybrid also has excellent scale-up properties with respect to the transaction size and the number of items in the database.