Similar to other categorical pose estimation work [1]}, we train a Mask R-CNN [2]} model on the same dataset used for pose estimation to obtain the object's bounding box \(\mathcal {B}\) , mask \(\mathcal {M}\)  and category label \(\mathcal {H}_c\) .
Patches of ray direction \(\mathcal {R}_{\mathcal {B}}\) , RGB \(\mathcal {I}_{\mathcal {B}}\)  and raw depth \(\mathcal {D}_{\mathcal {B}}\)  are extracted from the original data source following bounding box \(\mathcal {B}\) , before inputting to the first stage of TransNet.
