When humans perform inductive learning, they often enhance the process with background knowledge. With the increasing availability of well-formed collaborative knowledge bases, the performance of learning algorithms could be significantly enhanced if a way were found to exploit these knowledge bases. In this work, we present a novel algorithm for injecting external knowledge into induction algorithms using feature generation. Given a feature, the algorithm defines a new learning task over its set of values, and uses the knowledge base to solve the constructed learning task. The resulting classifier is then used as a new feature for the original problem. We have applied our algorithm to the domain of text classification using large semantic knowledge bases. We have shown that the generated features significantly improve the performance of existing learning algorithms.
This paper describes a "bootstrapping" approach to the engineering of appropriate training-data representations for inductive learning. The central idea is to begin with an initial set of human-created features and then generate additional features that have syntactic forms that are similar to the human-engineered features. More specifically, we describe a two-stage process for the engineering of good representations for learning: first, generating by hand (usually in consultation with domain experts) an initial set of features that seem to help learning, and second, "bootstrapping" off of these features by developing and applying operators that generate new features that look syntactically like the expert-based features. Our experiments in the domain of DNA sequence identification show that an initial successful human-engineered representation for data can be expanded in this fashion to yield dramatically improved results for learning. We present a method for learning higher-order polynomial functions from examples using linear regression and feature construction. Regression is used on a set of training instances to produce a weight vector for a linear function over the feature set. If this hypothesis is imperfect, a new feature is constructed by forming the product of the two features that most effectively predict the squared error of the current hypothesis. The algorithm is then repeated. In an extension to this method, the specific pair of features to combine is selected by measuring their joint ability to predict the hypothesis' error.