In this paper, we propose Continuous Graph Flow, a generative continuous flow based method that aims to model complex distributions of graph-structured data.   Once learned, the model can be applied to an arbitrary graph, defining a probability density over the random variables represented by the graph. It is formulated as an ordinary differential equation system with shared and reusable functions that operate over the graphs.   This leads to a new type of neural graph message passing scheme that performs continuous message passing over time. This class of models offers several advantages: a flexible representation that can generalize to variable data dimensions; ability to model dependencies in complex data distributions; reversible and memory-efficient; and exact and efficient computation of the likelihood of the data. We demonstrate the effectiveness of our model on a diverse set of generation tasks across different domains: graph generation, image puzzle generation, and layout generation from scene graphs. Our proposed model achieves significantly better performance compared to  state-of-the-art models. Modeling and generating graph-structured data has important applications in various scientific fields such as building knowledge graphs (Lin et al., 2015; Bordes et al., 2011) , inventing new molecular structures (Gilmer et al., 2017) and generating diverse images from scene graphs (Johnson et al., 2018) . Being able to train expressive graph generative models is an integral part of AI research. Significant research effort has been devoted in this direction. Traditional graph generative methods (Erdős & Rényi, 1959; Leskovec et al., 2010; Albert & Barabási, 2002; Airoldi et al., 2008) are based on rigid structural assumptions and lack the capability to learn from observed data. Modern deep learning frameworks within the variational autoencoder (VAE) (Kingma & Welling, 2014) formalism offer promise of learning distributions from data. Specifially, for structured data, research efforts have focused on bestowing VAE based generative models with the ability to learn structured latent space models (Lin et al., 2018; He et al., 2018; Kipf & Welling, 2016) . Nevertheless, their capacity is still limited mainly because of the assumptions placed on the form of distributions. Another class of graph generative models are based on autoregressive methods (You et al., 2018; Kipf et al., 2018) . These models construct graph nodes sequentially wherein each iteration involves generation of edges connecting a generated node in that iteration with the previously generated set of nodes. Such autoregressive models have been proven to be the most successful so far. However, due to the sequential nature of the generation process, the generation suffers from the inability to maintain long-term dependencies in larger graphs. Therefore, existing methods for graph generation are yet to realize the full potential of their generative power, particularly, the ability to model complex distributions with the flexibility to address variable data dimensions. Alternatively, for modeling the relational structure in data, graph neural networks (GNNs) or message passing neural networks (MPNNs) (Scarselli et al., 2009; Gilmer et al., 2017; Duvenaud et al., 2015; Kipf & Welling, 2017; Santoro et al., 2017; Zhang et al., 2018) have been shown to be effective in learning generalizable representations over variable input data dimensions. These models operate on the underlying principle of iterative neural message passing wherein the node representations are updated iteratively for a fixed number of steps. Hereafter, we use the term message passing to refer to this neural message passing in GNNs. We leverage this representational ability towards graph generation. In this paper, we introduce a new class of models -Continuous Graph Flow (CGF): a graph generative model based on continuous normalizing flows Grathwohl et al., 2019 ) that Figure 1 : Illustration of evolution of message passing mechanisms from discrete updates (a) to our proposed continuous updates (b). Continuous Graph Flow leverages normalizing flows to transform simple distributions (e.g. Gaussian) at t 0 to the target distributions at t 1 . The distribution of only one graph node is shown here for visualization, but, all the node distributions transform over time. generalizes the message passing mechanism in GNNs to continuous time. Specifically, to model continuous time dynamics of the graph variables, we adopt a neural ordinary different equation (ODE) formulation. Our CGF model has both the flexibility to handle variable data dimensions (by using GNNs) and the ability to model arbitrarily complex data distributions due to free-form model architectures enabled by the neural ODE formulation. Inherently, the ODE formulation also imbues the model with following properties: reversibility and exact likelihood computation. Concurrent work on Graph Normalizing Flows (GNF) (Liu et al., 2019 ) also proposes a reversible graph neural network using normalizing flows. However, their model requires a fixed number of transformations. In contrast, while our proposed CGF is also reversible and memory efficient, the underlying flow model relies on continuous message passing scheme. Moreover, the message passing in GNF involves partitioning of data dimensions into two halves and employs coupling layers to couple them back. This leads to several constraints on function forms and model architectures that have a significant impact on performance (Kingma & Dhariwal, 2018) . In contrast, our CGF model has unconstrained (free-form) Jacobians, enabling it to learn more expressive transformations. Moreover, other similar work GraphNVP Madhawa et al. (2019) is also based on normalizing flows as compared to CGF that models continuous time dynamics. We demonstrate the effectiveness of our CGF-based models on three diverse tasks: graph generation, image puzzle generation, and layout generation based on scene graphs. Experimental results show that our proposed model achieves significantly better performance than state-of-the-art models. In this paper, we presented continuous graph flow, a generative model that generalizes the neural message passing in graphs to continuous time. We formulated the model as an neural ordinary differential equation system with shared and reusable functions that operate over the graph structure. We conducted evaluation for a diverse set of generation tasks across different domains: graph generation, image puzzle generation, and layout generation for scene graph. Experimental results showed that continuous graph flow achieves significant performance improvement over various of state-ofthe-art baselines. For future work, we will focus on generation tasks for large-scale graphs which is promising as our model is reversible and memory-efficient.
Graph generative models based on generalization of message passing to continuous time using ordinary differential equations 
{"src": "scitldr-AIC_train", "paper_id": "BkgZSCEtvr"}