We pre-train a Faster-RCNN [1]} on the base dataset for bounding
box and feature extraction. The overall architecture is shown in fig:fasterrcnn.
To train the network, we use the original bounding
box labels within the base dataset to define the Region Proposal Network (RPN)
and second-stage losses of the Faster-RCNN. We adapt class-agnostic bounding box
regression model in the second-stage to get one bounding box per feature
proposal regardless of the number of base classes. Once trained, we
use the trained Faster-RCNN to extract \(P\)  bounding box proposals
\(B\in \mathbb {R}^{P\times 4}\)  and their corresponding d-dimensional features
\(F\in \mathbb {R}^{P\times d}\)  from each input image \(\mathbf {I}\) . We also apply an \(\ell _2\)  normalization layer to project all the features to the unit hypersphere. As discussed later, the normalization step is important as our model uses cosine similarity measure for better generalization.
