Hyperparameter Sensitivity in Deep Outlier Detection: Analysis and a Scalable Hyper-Ensemble Solution
Outlier detection (OD) literature exhibits numerous algorithms as it applies
to diverse domains. However, given a new detection task, it is unclear how to
choose an algorithm to use, nor how to set its hyperparameter(s) (HPs) in
unsupervised settings. HP tuning is an ever-growing problem with the arrival of
many new detectors based on deep learning. While they have appealing properties
such as task- driven representation learning and end-to-end optimization, deep
models come with a long list of HPs. Surprisingly, the issue of model selection
in the outlier mining literature has been "the elephant in the room"; a
significant factor in unlocking the utmost potential of deep methods, yet
little said or done to systematically tackle the issue. In the first part of
this paper, we conduct the first large-scale analysis on the HP sensitivity of
deep OD methods, and through more than 35,000 trained models, quantitatively
demonstrate that model selection is inevitable. Next, we design a HP-robust and
scalable deep hyper-ensemble model called ROBOD that assembles models with
varying HP configurations, bypassing the choice paralysis. Importantly, we
introduce novel strategies to speed up ensemble training, such as parameter
sharing, batch/simultaneous training, and data subsampling, that allow us to
train fewer models with fewer parameters. Extensive experiments on both image
and tabular datasets show that ROBOD achieves and retains robust,
state-of-the-art detection performance as compared to its modern counterparts,
while taking only 2-10% of the time by the naive hyper-ensemble with
independent training.