We propose a simple and efficient method for ranking features in multi-label classification. The method produces a ranking of features showing their relevance in predicting labels, which in turn allows to choose a final subset of features. The procedure is based on Markov Networks and allows to model the dependencies between labels and features in a direct way. In the first step we build a simple network using only labels and then we test how much adding a single feature affects the initial network. More specifically, in the first step we use the Ising model whereas the second step is based on the score statistic, which allows to test a significance of added features very quickly. The proposed approach does not require transformation of label space, gives interpretable results and allows for attractive visualization of dependency structure. We give a theoretical justification of the procedure by discussing some theoretical properties of the Ising model and the score statistic. We also discuss feature ranking procedure based on fitting Ising model using @math regularized logistic regressions. Numerical experiments show that the proposed methods outperform the conventional approaches on the considered artificial and real datasets.
Abstract Why and how the Partial Least Squares Regression (PLSR) was developed, is here described from the author's perspective. The paper outlines my frustrating experiences in the 70'ies with two conflicting and equally over-ambitious and over-simplified modelling cultures - in traditional chemistry and in traditional statistics. It describes my mental progress of first learning to combine them into least squares “unmixing” of known chemical mixtures, and later extending this into the “unscrambling” of partially unknown structures as well. The bi-linear regression framework is summarised in terms of the development from Principal Component Regression into the PLSR. Finally, the versatility of the PLSR is discussed in light of the urgent need for better eduacation in scientific data analysis.