The architecture of QPIC [1]} consists of a backbone CNN, a transformer encoder, and a transformer decoder.
Given an image, a single-scale visual feature map is extracted by the backbone CNN, and then positional information is added to the feature map.
The transformer encoder takes the visual features and returns contextualized visual features with self-attention layers.
In the transformer decoder, HOI queries are first processed by the self-attention layer, and then the cross-attention layer associates the HOI queries with the contextualized visual features (given by the encoder) to capture relevant HOI representations.
Finally, predictions for HOI are computed from individual contextualized HOI query embeddings as mentioned above.
Note that both self-attention and cross-attention adopt multi-head attention.
