I am using a Random Forest classifier using two datasets, one for training and the other for testing and vice-versa, described below.First dataset $F$Continuous data, 28 features x 58 observations.Two classes: $A$ and $B$Original cardinalities: $A: 42~obs.$, $B: 16~obs.$Random subsampled to: $A: 16~obs.$, $B: 16~obs.$Second dataset $S$Continuous data, 28 features x 90 observations.Two classes: $A$ and $B$Cardinalities: $A: 76~obs.$, $B: 14~obs.$Random subsampled to: $A: 14~obs.$, $B: 14~obs.$Training methodologyAfter a procedure of feature selection with mRMR [1], selecting a number of feature no greater than $n/5$, where $n$ is the number of observations in the training dataset (as proposed by [2]), respectively 6 features for the $F$ dataset, 5 features for the $S$ dataset, I trained my classifier on the training dataset (in which I have downsized the majority class with a random subsample, in order to train each class with the same number of observations).ResultsIf I train my Random Forest classifier on $F$ and I test it on $S$ I get the following confusion matrix:$\begin{matrix}&B&A\\True~B&3&11\\True~A&3&73\\\end{matrix}$And normalised per-row:$\begin{matrix}&B&A\\True~B&0.214286&0.785714\\True~A&0.039474&0.960526\\\end{matrix}$Vice-versa, if I train my classifier on $S$ and I test it on $F$ I get the following confusion matrix:$\begin{matrix}&B&A\\True~B&16&0\\True~A&12&30\\\end{matrix}$And normalised per-row:$\begin{matrix}&B&A\\True~B&1&0\\True~A&0.285714&0.714286\\\end{matrix}$How could it be to have such asymmetry in classification?[1] Peng, Hanchuan, Fuhui Long, and Chris Ding. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy, IEEE Transactions on pattern analysis and machine intelligence 27.8 (2005): 1226-1238.[2] Johnstone, Iain M., and D. Michael Titterington. Statistical challenges of high-dimensional data. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 367.1906 (2009): 4237-4253.
Differences in classification performance switching training and test set
classification;training