This paper investigates the problem of determining a binary-valued function through a sequence of strategically selected queries. The focus is an algorithm called Generalized Binary Search (GBS). GBS is a well-known greedy algorithm for determining a binary-valued function through a sequence of strategically selected queries. At each step, a query is selected that most evenly splits the hypotheses under consideration into two disjoint subsets, a natural generalization of the idea underlying classic binary search. This paper develops novel incoherence and geometric conditions under which GBS achieves the information-theoretically optimal query complexity; i.e., given a collection of N hypotheses, GBS terminates with the correct function after no more than a constant times log N queries. Furthermore, a noise-tolerant version of GBS is developed that also achieves the optimal query complexity. These results are applied to learning halfspaces, a problem arising routinely in image processing and machine learning.
Past work has provided a partial characterization of this problem. If the responses to queries are noiseless, then selecting the sequence of queries from @math is equivalent to determining a binary decision tree, where a sequence of queries defines a path from the root of the tree (corresponding to @math ) to a leaf (corresponding to a single element of @math ). In general the determination of the optimal (worst- or average-case) tree is NP-complete as shown by Hyafil and Rivest @cite_12 . However, there exists a greedy procedure that yields query sequences that are within a factor of @math of the optimal search tree depth; this result has been discovered independently by several researchers including Loveland @cite_41 , Garey and Graham @cite_41 , @cite_15 , and Dasgupta @cite_4 . The greedy procedure is referred to here as Generalized Binary Search (GBS) or the splitting algorithm , and it reduces to classic binary search, as discussed in .
@cite_41 @cite_15 @cite_4 @cite_12
1965313644 2135496842 1811083715 2593445670
This paper investigates the problem of determining a binary-valued function through a sequence of strategically selected queries. The focus is an algorithm called Generalized Binary Search (GBS). GBS is a well-known greedy algorithm for determining a binary-valued function through a sequence of strategically selected queries. At each step, a query is selected that most evenly splits the hypotheses under consideration into two disjoint subsets, a natural generalization of the idea underlying classic binary search. This paper develops novel incoherence and geometric conditions under which GBS achieves the information-theoretically optimal query complexity; i.e., given a collection of N hypotheses, GBS terminates with the correct function after no more than a constant times logN queries. Furthermore, a noise-tolerant version of GBS is developed that also achieves the optimal query complexity. These results are applied to learning halfspaces, a problem arising routinely in image processing and machine learning. This paper addresses the problem of noisy Generalized Binary Search (GBS). GBS is a well-known greedy algorithm for determining a binary-valued hypothesis through a sequence of strategically selected queries. At each step, a query is selected that most evenly splits the hypotheses under consideration into two disjoint subsets, a natural generalization of the idea underlying classic binary search. GBS is used in many applications, including fault testing, machine diagnostics, disease diagnosis, job scheduling, image processing, computer vision, and active learning. In most of these cases, the responses to queries can be noisy. Past work has provided a partial characterization of GBS, but existing noise-tolerant versions of GBS are suboptimal in terms of query complexity. This paper presents an optimal algorithm for noisy GBS and demonstrates its application to learning multidimensional threshold functions. We give a (ln n+ 1)-approximation for the decision tree (DT) problem. An instance of DT is a set of mbinary tests T= (T 1 , ..., T m ) and a set of nitems X= (X 1 , ..., X n ). The goal is to output a binary tree where each internal node is a test, each leaf is an item and the total external path length of the tree is minimized. Total external path length is the sum of the depths of all the leaves in the tree. DT has a long history in computer science with applications ranging from medical diagnosis to experiment design. It also generalizes the problem of finding optimal average-case search strategies in partially ordered sets which includes several alphabetic tree problems. Our work decreases the previous upper bound on the approximation ratio by a constant factor. We provide a new analysis of the greedy algorithm that uses a simple accounting scheme to spread the cost of a tree among pairs of items split at a particular node. We conclude by showing that our upper bound also holds for the DT problem with weighted tests. We consider the following generalization of the binary search problem. A search strategy is required to locate an unknown target node @math in a given tree @math . Upon querying a node @math of the tree, the strategy receives as a reply an indication of the connected component of @math containing the target @math . The cost of querying each node is given by a known non-negative weight function, and the considered objective is to minimize the total query cost for a worst-case choice of the target. Designing an optimal strategy for a weighted tree search instance is known to be strongly NP-hard, in contrast to the unweighted variant of the problem which can be solved optimally in linear time. Here, we show that weighted tree search admits a quasi-polynomial time approximation scheme: for any @math , there exists a @math -approximation strategy with a computation time of @math . Thus, the problem is not APX-hard, unless @math . By applying a generic reduction, we obtain as a corollary that the studied problem admits a polynomial-time @math -approximation. This improves previous @math -approximation approaches, where the @math -notation disregards @math -factors.