I am training a normal and a balanced Linear SVM using imbalance data, and testing both using F1-score.(By balanced Linear SVM, I mean that each observation has a weight inverse to its frequency, so that it oversamples from the minority class and undersamples from the majority class.)Not surprisingly, using a multi-class dataset and solving it individually as one-vs-rest binary problems, balanced Linear SVM beats unbalanced Linear SVM on every class:                   normal vs balancedtarget 01 - scores: 0.272 vs 0.608target 02 - scores: 0.391 vs 0.587target 03 - scores: 0.433 vs 0.546target 04 - scores: 0.659 vs 0.655target 05 - scores: 0.000 vs 0.257target 06 - scores: 0.431 vs 0.475target 07 - scores: 0.000 vs 0.249target 08 - scores: 0.000 vs 0.053target 09 - scores: 0.576 vs 0.155target 10 - scores: 0.000 vs 0.550OneVsRest - scores: 0.565 vs 0.540But when it comes to using a OneVsRest classifier (last row), the unbalanced Linear SVM beats the Linear SVM, using the average of the F1 scores. This happens to me in a lot of datasets.Here is the code I have used:# -*- coding: utf-8 -*-from sklearn.svm import LinearSVCfrom sklearn.multiclass import OneVsRestClassifierfrom sklearn.cross_validation import StratifiedShuffleSplitfrom sklearn.metrics import f1_scoreimport numpy as npd = np.loadtxt('yeast.csv', delimiter=',')X = d[:, 0:-1]y = d[:, -1]print '                     none vs balanced'for target in np.unique(y):    scores = [0] * 2    k = 6    for tr, ts in StratifiedShuffleSplit(y, k, 0.2):        for i, w in enumerate([None, 'balanced']):            _y = (y == target).astype(int)            yp = LinearSVC(class_weight=w).fit(X[tr], _y[tr]).predict(X[ts])            scores[i] += f1_score(_y[ts], yp) / k    print 'target %02d - scores: %.3f vs %.3f' % (target, scores[0], scores[1])scores = [0] * 2for tr, ts in StratifiedShuffleSplit(y, k, 0.2):    for i, w in enumerate([None, 'balanced']):        m = OneVsRestClassifier(LinearSVC(class_weight=w))        yp = m.fit(X[tr], y[tr]).predict(X[ts])        scores[i] += f1_score(y[ts], yp, pos_label=None, average='weighted') / kprint 'OneVsRest - scores: %.3f vs %.3f' % (scores[0], scores[1])As the dataset, I have used here Yeast (UCI), for which a sklearn-ready version can be found here.Results:                     none vs balancedtarget 01 - scores: 0.241 vs 0.612target 02 - scores: 0.402 vs 0.604target 03 - scores: 0.444 vs 0.552target 04 - scores: 0.666 vs 0.650target 05 - scores: 0.000 vs 0.286target 06 - scores: 0.370 vs 0.500target 07 - scores: 0.000 vs 0.280target 08 - scores: 0.000 vs 0.082target 09 - scores: 0.563 vs 0.147target 10 - scores: 0.000 vs 0.511OneVsRest - scores: 0.567 vs 0.543Isn't this surprisingly? I know that OneVsRest has a lot of ties, and so scores will be used, as determined by the distance to the separation hyperplane. Still, why does balanced Linear SVM loses the war when it keeps wining every battle?EDIT: I think the fact that one model wins pretty much every battle, but not the war is related to the fact that One Vs Rest uses confidence scores, and not the actual classification, which makes sense since it avoids ties, and when score[k1] > score[k2] we can assume it prefers k1 > k2. See wikipedia.
Balanced Linear SVM wins every class except One vs All
classification;scikit learn;multiclass classification;unbalanced classes
I think the following synthetic example explains what is going on:from sklearn.base import clonefrom sklearn.svm import SVCimport numpy as npfrom sklearn.datasets import make_blobsX, y = make_blobs(1000, centers=[[1, 1], [-1, -1], [1, -1]], cluster_std=0.8)models = [    ('normal', SVC()),    ('balanced', SVC(class_weight='balanced')),]import matplotlib.pyplot as plotplot.ioff()for i, (name, m) in enumerate(models):    for k in np.unique(y):        m = clone(m).fit(X, (y == k).astype(int))        xx, yy = np.meshgrid(np.arange(-3, 3, 0.1), np.arange(-3, 3, 0.1))        z = m.predict(np.c_[xx.ravel(), yy.ravel()])        z = z.reshape(xx.shape)        colors = ['blue', 'green', 'red']        plot.contour(xx, yy, z, colors=colors[k])    plot.scatter(X[:, 0], X[:, 1], c=y)    plot.title(name)    plot.show() The balanced SVM is clearly superior when considering each model individually, at least as measured by the F1 score, which punishes false positives and false negatives equally.But One-vs-Rest uses maximum score, which in this case is given by the distance to the decision hyperplane. It is not yet clear why this is worse in some cases, but clearly it is possible for one model to beat another for individual classes, but to lose for One-vs-Rest: how well the relative distances are modeled is what counts in One-vs-Rest.