Spatial Interaction Encoder. Taking inspiration from recent approaches on modeling physical object dynamics [1]}, [2]}, we propose a Spatial Interaction Encoder Network to explicitly model object-object interactions. Specifically, given a sequence \(V^{\prime }_i\)  from \(D^{\prime }\) , we model each element \(I^{\prime }\)  as a graph, \(G = (O, R)\) , where \(O\)  is the set of objects \(\lbrace o_1, o_2, \dots , o_m\rbrace \) , \(m\)  is the total number of objects in \(I^{\prime }\) , and \(R\)  denotes the relationship between objects (i.e., whether two objects interact with each other). For simplicity, all objects are connected with all other objects in the graph such that \(R= \lbrace (i,j) \mid i \ne j \wedge i \le m \wedge j \le m \rbrace \) . We compose an object embedding for each of \(o_i \in O\)  by combining self and interactional representations as follows:
\(f_{o}(o_i) = \phi _{\text{agg}} (f_{\text{s}} + f_{\text{in}}) \quad \text{with}\quad f_s(o_i) = \phi _s(o)\,, \quad f_{\text{in}}(o_i) = \sum _{j=1}^{m} \phi _{\text{in}}((o_i, o_j)) \mid (i, j) \in \mathbb {R}\,,\) 
