
Integration-Flow (IF): This mechanism uses contextualized embeddings as the propagated latent representation. FlowQA [1]} which also introduced the idea of FLOW, involves sequential processing along context tokens in parallel to the question turns followed by sequential processing in direction of the question turns (Flow), in parallel to context tokens as illustrated in Fig. REF . FlowQA employs multiple IF layers interleaved with self and cross attentions to reason over encoded embeddings (Fig. REF ). Recently released FlowDelta [2]} is an improvement on the IF approach that uses the similar FlowQA architecture and achieves better results. Instead of passing the latent representation directly, as in FlowQA, FlowDelta passes the information gain (the difference between the latent representation of previous 2 layers) with the intuition that information gain would allow the model to focus on more informative cues in context.

Integration-GraphFlow (IG): GraphFlow [3]} claims that the IF mechanism does not mimic human reasoning, as it first performs reasoning in parallel for each question, and then refines the reasoning results across different turns. Therefore, they use dynamically constructed, question-aware context graphs for each turn as the propagated latent representation. Processing through this flow (called GraphFlow) is facilitated by applying GNNs [4]} on the current context graph and previous context. To capture local interactions among consecutive words in context before feeding to a GNN, a BiLSTM is applied for contextual Integration. GraphFlow architecture alternates this mechanism with co-attention over the question and GNN output. This is illustrated in the figure provided in Appendix .

