The semantic mapper is responsible for aggregating the semantic information from individual RGB-D observations from time 0 to \(t\)  into an allocentric semantic map \(m_t\) . We use the mapping procedure from a state-of-the-art semantic exploration method [1]}. The depth observations are used to compute point-clouds that are registered to an allocentric coordinate system using the agent's poses (\(p_0, \cdots , p_t\) ). Each point in the point-cloud is classified into one of \(N\)  object classes and 1 background class by segmenting the corresponding RGB image using state-of-the-art segmentation models [2]}, [3]}. The point-cloud is projected to the top-down map space by using differentiable geometric operations [4]} to obtain the \((N + 2)\times M \times M\)  semantic map \(m_t\) . Channels 1 and 2 correspond to obstacles and explored areas, and the rest contain the \(N\)  object categories.
