By leveraging our HOI-aware deformable attentions, the cross-attention layer in MSTR decoder extracts three different semantics (human, object, and contextual information) for each HOI query from the encoded image features.
For each decoder layer, we discovered that compositing the multiple semantics obtained from the previous cross-attention layer [1]} by summing the semantics after applying individual self-attention demonstrates the best performance (see Table REF  and Appendix).
The input for the \((k+1)\) -th layer of our HOI-aware deformable attention \(\bar{z}_q^{k+1}\)  is written as:
\(\bar{z}_q^{k+1}=\mbox{SA}(f^h_q(k))+\mbox{SA}(f^o_q(k))+\mbox{SA}(f^c_q(k)),\) 
