We customize the transformer<cite> (Vaswani et al., 2017)</cite> featured by non-local operations (Wang et al., 2018) with two * The work was done when Yaoyiran was working at Living Analytics Research Centre, Singapore Management University who is now a PhD student at University of Cambridge. input branches and tailor the transformer's multihead attention mechanism to the needs of information exchange between these two parallel branches.