In this paper, our goal is to design a framework such that its classification performance can  approach the upperbound performance without requiring a superclass oracle. Here, we propose a  two-stage Super-Sub framework (Fig. REF ) to reduce the performance gap. In details, in the first stage, we use a Superclass Network to decide the superclass of the test image. In the second stage, we choose the corresponding Subclass Network according to the superclass decision made in the first stage. This Subclass Network then outputs the final fine-grained subclass of the test image. More generally, our two-stage Super-Sub framework enables a neural network system to automatically adjust its behavior according to different inputs [1]}, here achieved through the aforementioned hierarchical two-step inference. From our first observation, the Superclass Network barely makes inter-superclass mistakes, so the role of the Superclass Network can be approximated as a superclass oracle, helping to choose the correct Subclass Network for subclass classification. As a result, our two-stage Super-Sub framework is 3.30% better than the lowerbound on the new Superclassing ImageNet dataset.
Nonetheless, our vanilla implementation of the framework incurs a large cost in both parameter storage and inference runtime memory on the GPU (a factor of \(N+1\)  for \(N\)  superclasses compared to a single network). To improve this, we propose a method that keeps the same inference runtime memory as only one network, and that reduces the parameter storage cost to \(0.2N+1\)  based on finetuning [2]}, delta and quantization aware training (QAT) [3]} techniques, all while preserving the same accuracy boost. Most importantly, our framework promises to be more scalable and generalizable than the common alternative of simply scaling up a vanilla network in size since very large networks suffer from overfitting and gradient vanishing. We demonstrate our framwork's capacity for scalability (see Results Sec. ) by increasing the representational capacity starting from an already very large network, i.e. Resnet-152.
