Perturbation of Deep Autoencoder Weights for Model Compression and Classification of Tabular Data
Fully connected deep neural networks (DNN) often include redundant weights
leading to overfitting and high memory requirements. Additionally, the
performance of DNN is often challenged by traditional machine learning models
in tabular data classification. In this paper, we propose periodical
perturbations (prune and regrow) of DNN weights, especially at the
self-supervised pre-training stage of deep autoencoders. The proposed weight
perturbation strategy outperforms dropout learning in four out of six tabular
data sets in downstream classification tasks. The L1 or L2 regularization of
weights at the same pretraining stage results in inferior classification
performance compared to dropout or our weight perturbation routine. Unlike
dropout learning, the proposed weight perturbation routine additionally
achieves 15% to 40% sparsity across six tabular data sets for the compression
of deep pretrained models. Our experiments reveal that a pretrained deep
autoencoder with weight perturbation or dropout can outperform traditional
machine learning in tabular data classification when fully connected DNN fails
miserably. However, traditional machine learning models appear superior to any
deep models when a tabular data set contains uncorrelated variables. Therefore,
the success of deep models can be attributed to the inevitable presence of
correlated variables in real-world data sets.