Before presenting classification results, we first show the sparseness of S-DSN(sigm) and S-DSN(relu) compared to DSN. We use Hoyer's sparseness measure (HSM) [1]} to figure out how sparse representations learned by the S-DSN(sigm), S-DSN(relu) and DSN. This measure has good properties, which is in the interval \([0, 1]\)  and on a normalized scale. Its value more close to 1 means that there are more zero components in the vector. We perform comparisons on Extended YaleB and AR databases, and results are reported in Table 1. The sparseness results show that S-DSN(sigm) and S-DSN(relu) have higher sparseness and higher recognition accuracy. Table 1 compares the network HSM of the S-DSN(sigm) and the S-DSN(relu) to that of DSN. We observe that the average sparseness of two layers S-DSN(sigm) is about 0.105 (Extended YaleB) and the average sparseness of two layers S-DSN(relu) is about 0.291 (AR). In contract, the average sparseness of two layers DSN is on average below \(0.02\)  in the databases. It can be seen that the S-DSN can learn sparser representations. Due to space reasons, Figure 1 only visualizes the activation probabilities of first hidden layer, which are computed under the S-DSN(relu) and the DSN given an image from test set of AR.
<FIGURE>