With the aim to propose a non parametric hypothesis test, this paper carries out a study on the Matching Error (ME), a comparison index of two partitions obtained from the same data set, using for example two clustering methods. This index is related to the misclassifica-tion error in supervised learning. Some properties of the ME and, especially, its distribution function for the case of two independent partitions are analyzed. Extensive simulations show the efficiency of the ME and we propose a hypothesis test based on it.
The entropy of a partition @math is defined by @math where @math is the estimate of the probability that an element is in cluster @math . The can be used to measure the independence of two partitions @math and @math . It is given by: @math , where @math is the estimate of the probability that an element belongs to cluster @math of @math and @math of @math . Mutual information is a metric over the space of all clusterings, but its value is not bounded which makes it difficult to interpret. As @math , other bounded indices have been proposed such as ( @cite_2 , @cite_5 ) where @math is divided either by the arithmetic or the geometric mean of the clustering entropies. Meila ( @cite_7 ) has also proposed an index based on Mutual information called .
@cite_5 @cite_7 @cite_2
2205170224 2002359780 1825700702 160196575
It was recently shown that estimating the Shannon entropy @math of a discrete @math -symbol distribution @math requires @math samples, a number that grows near-linearly in the support size. In many applications @math can be replaced by the more general R 'enyi entropy of order @math , @math . We determine the number of samples needed to estimate @math for all @math , showing that @math requires a near-linear @math samples, but, perhaps surprisingly, integer @math requires only @math samples. Furthermore, developing on a recently established connection between polynomial approximation and estimation of additive functions of the form @math , we reduce the sample complexity for noninteger values of @math by a factor of @math compared to the empirical estimator. The estimators achieving these bounds are simple and run in time linear in the number of samples. Our lower bounds provide explicit constructions of distributions with different R 'enyi entropies that are hard to distinguish. Given a metric space @math , @math , @math , and @math , a distribution over mappings @math is called a @math -sensitive hash family if any two points in @math at distance at most @math are mapped by @math to the same value with probability at least @math , and any two points at distance greater than @math are mapped by @math to the same value with probability at most @math . This notion was introduced by Indyk and Motwani in 1998 as the basis for an efficient approximate nearest neighbor search algorithm and has since been used extensively for this purpose. The performance of these algorithms is governed by the parameter @math , and constructing hash families with small @math automatically yields improved nearest neighbor algorithms. Here we show that for @math it is impossible to achieve @math . This almost matches the construction of Indyk and Motwani which achieves @math . The binary symmetric stochastic block model deals with a random graph of @math vertices partitioned into two equal-sized clusters, such that each pair of vertices is connected independently with probability @math within clusters and @math across clusters. In the asymptotic regime of @math and @math for fixed @math and @math , we show that the semidefinite programming relaxation of the maximum likelihood estimator achieves the optimal threshold for exactly recovering the partition from the graph with probability tending to one, resolving a conjecture of Abbe14 . Furthermore, we show that the semidefinite programming relaxation also achieves the optimal recovery threshold in the planted dense subgraph model containing a single cluster of size proportional to @math . We study theoretically and numerically the entanglement entropy of the @math -dimensional free fermions whose one body Hamiltonian is the Anderson model. Using basic facts of the exponential Anderson localization, we show first that the disorder averaged entanglement entropy @math of the @math dimension cube @math of side length @math admits the area law scaling @math even in the gapless case, thereby manifesting the area law in the mean for our model. For @math and @math we obtain then asymptotic bounds for the entanglement entropy of typical realizations of disorder and use them to show that the entanglement entropy is not selfaveraging, i.e., has non vanishing random fluctuations even if @math .