Recurrent Brain Graph Mapper for Predicting Time-Dependent Brain Graph Evaluation Trajectory

Several brain disorders can be detected by observing alterations in the brain's structural and functional connectivities. Neurological findings suggest that early diagnosis of brain disorders, such as mild cognitive impairment (MCI), can prevent and even reverse its development into Alzheimer's disease (AD). In this context, recent studies aimed to predict the evolution of brain connectivities over time by proposing machine learning models that work on brain images. However, such an approach is costly and time-consuming. Here, we propose to use brain connectivities as a more efficient alternative for time-dependent brain disorder diagnosis by regarding the brain as instead a large interconnected graph characterizing the interconnectivity scheme between several brain regions. We term our proposed method Recurrent Brain Graph Mapper (RBGM), a novel efficient edge-based recurrent graph neural network that predicts the time-dependent evaluation trajectory of a brain graph from a single baseline. Our RBGM contains a set of recurrent neural network-inspired mappers for each time point, where each mapper aims to project the ground-truth brain graph onto its next time point. We leverage the teacher forcing method to boost training and improve the evolved brain graph quality. To maintain the topological consistency between the predicted brain graphs and their corresponding ground-truth brain graphs at each time point, we further integrate a topological loss. We also use l1 loss to capture time-dependency and minimize the distance between the brain graph at consecutive time points for regularization. Benchmarks against several variants of RBGM and state-of-the-art methods prove that we can achieve the same accuracy in predicting brain graph evolution more efficiently, paving the way for novel graph neural network architecture and a highly efficient training scheme.


Introduction
Latest neuroscience studies have emphasized the importance of personalized treatments for brain disorders that can significantly improve patient's recovery [1]. Brain disorders such as mild cognitive impairment (MCI) can be easily reversed if diagnosed at an early stage before evolving into irreversible Alzheimer's disease (AD) [2]. As such, recent landmark studies [3,4,5,6] proposed using the breadth of machine learning to predict brain connectome evolution trajectory. For instance, [3] has proposed a learning-based framework to predict the longitudinal development of cortical surface and white matter fibers. To do so, first, they used multiple atlases to generate a spatially heterogeneous atlas that mimics the cortical surface of the target subject. Second, they predicted spatio-temporal connectivity features from neonatal brains using low-rank tensor completion. Furthermore, [4] proposed a deep learning model that jointly classifies and predicts the evolution trajectory of the brain from a single acquisition point. At the baseline, they identified the landmarks, namely regions of interest (ROIs), and utilized supervised and unsupervised learning to predict the evolution trajectory at each landmark.
However, such studies were conducted only on brain images that solely consider local neighborhood connectivities undeniably overseeing the brain global connectivity pattern. To properly diagnose brain connectivity disorders, the brain must be viewed as a large interconnected graph [7] where each ROI represents a node, and each pairwise connectivity between two ROIs represents an edge. To address this limitation, we set out a more challenging problem, which is predicting the evolution of brain graphs from a baseline observation. In this context, recent works [5,6] leveraged generative adversarial networks (GANs) [8], where they proposed the first graph-based GAN specialized in AD evolution trajectory prediction. Namely, GANs [8] address the problem of unsupervised learning by training two neural networks: generator and discriminator. The generator takes randomly distributed data as input and generates synthetic data that mimics a real distribution. On the other hand, the discriminator inputs the generated data and predicts whether it is real or fake. Both generator and discriminator compete in an adversarial way. Hence, the overall quality of the generated data improves with training epochs. [5] used gGAN to learn how to normalize a brain graph with respect to a fixed connectional brain template (CBT). Their gGAN architecture is made up of a graph normalizer network that learns a high-order representation of each brain in a graph and produces a CBT-based normalized brain graph. Furthermore, [6] proposed EvoGraphNet that connects series of gGANs each specialized in how to construct a brain graph from the predicted brain graph of the previous gGANS in the time-dependent cascade.
Although gGANs were proven to be successful for predicting brain connectome evolution trajectory given a single observation, there is still a considerable amount of computational complexity. Since gGANs contain two graph neural networks (i.e., generator and discriminator), they require a significant amount of computational power and training time. Also, these architectures require a sequential framework where each gGAN has to generate a time point t i to get the following time point prediction t i+1 .
This challenge raises the following question: can we mimic the prediction power of the above-mentioned sequential gGANs without the need to use highly complex sequential networks?
According to [9], brain connectivities do not evolve randomly. Their evolution follows a temporal scheme that aims to satisfy the patient's needs for each given age and health condition. In that regard, recurrent neural networks (RNNs) [10] are known for their temporal pattern recognition. Therefore, in this paper, we propose to power our model with RNNs. We term our proposed model Recurrent Brain Graph Mapper (RBGM), the first framework to predict brain connectome evolution while efficiently reducing complexity and training time consumption. Our model uses a shallow graph convolutional neural network architecture to predict brain disease's evolution trajectory given a baseline graph. To do so, for a given time point, t i each mapper uses the ground-truth from the previous time point t i−1 to predict the brain connectivity scheme at the time point t i instead of starting from the initial time point. To do so, we apply the teacher forcing method [11], which is known for its quick and efficient training recurrent-based models.
We propose preserving the topological consistency between the predicted and ground-truth brain graphs at each time point. To do so, we integrate a topological loss measuring the topological discrepancy between the predicted brain graph and its corresponding ground-truth brain graph. Furthermore, we leverage a l1 loss to minimize the sparse distance between two serialized brain graphs to capture time-dependency between two consecutive observations. We also investigate the effect of Kullback-Leibler divergence (i.e., KL-divergence), where we enforce the preservation of node distribution between predicted and ground-truth brain graphs over time. We articulate the main contributions of our work as follows: 1. On a methodological level. Our proposed RGBM is the first RNN-based geometrical deep learning framework that predicts the time-dependent brain graph evolution trajectory from a single observation. 2. On a conceptual level. Our model reduces the complexity of the geometrical deep learning framework by speeding up the training while maintaining similar performances to the state-of-the-art methods. 3. On clinical level. Our RGBM can be used to prevent and reverse the onset of neurological diseases.

Proposed Method
This section introduces the key steps of our RBGM for predicting brain graph evolution from a single observation. Table 1 displays the mathematical notations that we use throughout our paper. We denote the matrices as boldface capital letters (e.g., X) and scalars as lowercase letters (e.g., m). The transpose operator is denoted as X T . The graph neural network takes the input brain graph X tr ti−1 ∈ R nr×nr for a given time point t i , where n r is the number of ROIs to learn node embedding V l = [v l 1 , v l 2 , . . . , v l nr ] T that captures the node-to-node relation and visualizes it in vector form. Then a transformation layer takes these node embeddings V l and computes the pairwise absolute difference to predict the brain graph at the time point t i given byX tr ti ∈ R nr×nr . First V l is repeated horizontally n r times to obtain R ∈ R nr×nr×nr . Next, we compute the absolute difference between R and its transpose. Finally, the resulting tensor is sum along z-axis to obtain the predicted brain graphX tr ti for the time point t i .
Overview of Recurrent Brain Graph Mapper for predicting brain graph evolution trajectory from a single baseline. Our proposed RBGM is composed of m mappers for m time points, as shown in Fig. 1. Each mapper can predict a brain graph for a given time point t i using its corresponding ground-truth brain graph at the time point t i−1 as an input. The recurrent graph convolution enables each mapper to capture temporal changes in the brain connectivity pattern between consecutive time points. Also, it increases the prediction power of each mapper, hence using fewer convolutional layers. Furthermore, Table 1: Mathematical definitions.

Mathematical notation
Definition m number of time points ns number of training subjects mr number of edges nr number of ROIs in brain S tr i node strength vector of ROI i in the ground-truth brain grapĥ S tr i node strength vector of ROI i in the predicted brain graph X tr t i training brain graph connectivity matrices ∈ R nr ×nr at tî X tr t i predicted brain graph connectivity matrices ∈ R nr ×nr at ti Mi the neighborhood containing all the adjacent nodes of node i F l edge-conditioned filter Θ l learnable edge-based parameter for dynamic graph convolution vi l node embedding of ROI i at layer l ∈ R d t W l weight parameter b l bias term R horizontally replicated brain connectivity matrix ∈ R nr ×nr ×nr R T transpose of horizontally replicated brain connectivity matrix ∈ R nr ×nr ×nr hi hidden state matrix at ti ∈ R mr ×mr W ih input to hidden weight for recurrent filter ∈ R 1×mr W hh hidden to hidden weight for recurrent filter ∈ R mr ×mr b h bias term for recurrent filter ∈ R mr ×mr we apply the teacher forcing method [11] to quickly and efficiently train our RGBM.
To enhance our method's robustness, we propose using l1 loss thanks to its resilience against outliers to enforce the connectivity consistency across time points. Thus, we express the l1 loss for each subject tr using the predicted brain graphX tr ti−1 from the mapper M i−1 , and its corresponding ground-truth brain graph at t i as follows: This acts as a regularizer over time and aligns with the sparse nature of brain connectivity evolution. In addition to the l1 loss, we propose a second loss to preserve the topological consistency between predicted brain graphs and their corresponding ground-truth at each time point. To frame the topological loss, we define a node strength vector measuring the topological strength for each node in a given graph. Since our brain graph is fully-connected (each node has a same number of edges), we chose node strength as a centrality measures. We compute the node strength vector by adding the weights of all edges connected to a node of interest. As such, S = [S , S , ..., S ] T represents the node strengths for all ROIs where n r is the number of ROIs. The following equation gives the topological loss: The full loss. We combine the previous losses to train our RGBM as follows: where λ 1 , and λ 2 are hyperparameters adjusting each corresponding loss.
The mapper network architecture. Each mapper m uses our proposed recurrent graph convolution (RGC) function. We leverage the teacher forcing method [11] to speed up training and increase the overall performance in training. Namely, the teacher forcing is a common method that speeds up training and improves the quality of recurrent-based models. It enforces the recurrent model to use ground-truth samples to predict the brain graph at the following time point. According to this method, for a given time point, t i the mapper takes the ground-truth X tr ti−1 from the time point t i−1 instead of taking the predicted brain graphX tr ti−1 from the preceding mapper M ti−1 to make prediction for the time point t i in training phase. Therefore, the need to start from the initial time point is eliminated during training.
Dynamic edge-filtered convolution. Each mapper in our RBGM uses the dynamic graph convolution with edge-conditioned filter proposed by [12]. Let G = (V, E) is a directed or undirected graph, where V is the set of n r ROIs and E ∈ V × V is a set of m r edges between each ROI. Let l be the layer index. For each layer l ∈ {1, 2, ..., L}, F l : R dm → R d l ×d l−1 represents a filter-generating network that generate edge weights for the message passing between ROIs i and j given features of e ij . d m and d l are dimensionality indexes. This operation is expressed as follows: where v l i is the node embedding for the ROI i at layer l. N (i) denotes the neighbors of ROI i. F l is the neural network that maps R dm to R d l ×d l−1 with weights W l . Θ l is the dynamically generated edge specific weights by F l . The b l ∈ R d l is the bias term. We note that F l can be any type of neural network.
We draw inspiration from the image-based recurrent neural network architecture, which shows outstanding performances on time-series data prediction [13,14]. This type of network can remember the former information and process new events accordingly thanks to its hidden state, which holds the former information (i.e., learned information in the previous layer). Each RNN cells takes two distinct inputs: (i) the input brain graphs from the current time point, and (ii) hidden state value from the brain graphs at the previous time point, then updates the hidden state, which holds the representation of the knowledge from the prior time point.
Proposed graph recurrent-filter. We propose the first edge-based recurrent graph neural network Fig.2 by re-designing the edge-conditioned filter [12] in the graph convolution layer as a graph recurrent-filter so that each mapper can capture temporal changes on brain connections over time, as shown in Fig.  2. Therefore, unlike [6], the need to start from the initial time point is eliminated during training. Our proposed graph recurrent filter can process past information when generating messages between each ROI to capture temporal changes of brain connectivity. To do so, it takes the set of edges e ∈ R mr×1 for a given time point t i and the hidden state matrix from the previous time point t i−1 given by h ti−1 ∈ R mr×mr , which acts as a memory and processes past information. Then it updates the hidden state matrix h ti ∈ R mr×mr for the current time point t i . In order to avoid the vanishing gradient problem which makes the network's gradients tend to zero (i.e., hard to learn parameters), we need a function which can bound the gradient and eliminate the risk of divergence during the training.
To do so, we use tanh [15] as an activation function in our graph-recurrent filter since it allows the state values to update by bounding in the range of [−1, 1] compared to other activation functions such as sigmoid. The equation for our recurrent edge-filtering function F l (e ti , h ti−1 ) is expressed as follows: where W ih ∈ R 1×mr and W hh ∈ R mr×mr are learnable parameters for inputto-hidden weight and hidden-to-hidden weight respectively. b h ∈ R mr×mr is bias term.
The transformation layer architecture. Let X tr ti−1 ∈ R nr×nr be the input brain connectivity matrix at a given time point, t i where n r is the number of ROIs. After obtaining the output node embeddings V l = [v l 1 , v l 2 , ..., v l n ] T of RGC layer from a given input X tr ti−1 , we construct predicted brain graphX tr ti at t i by computing pairwise absolute difference of learned embeddings [16]. To do so, first V l is replicated with respect to the horizontal axis n r times to obtain R ∈ R nr×nr×nr . Then, we compute the absolute difference between R and its transpose R T . Finally, the resulting tensor is the sum along z-axis to obtain the predicted brain graphX tr ti ∈ R nr×nr for the time point t i .

Results and Discussion
Evaluation dataset. We conducted experiments on OASIS-21 longitudinal dataset with 113 subjects [17]. This set contains longitudinal collection of 150 subjects aged between 60 to 96. Each subject's brain scans were acquired 3 times one year apart. For each subject, we construct a cortical morphological network derived from cortical thickness measure using structural T1-w MRI as proposed in [18]. Each cortical hemisphere is parcellated into 35 ROIs using the Desikan-Killiany cortical atlas. We construct our RBGM with PyTorch Geometric library [19]. To inherit the previous layer embedding V l−1 ∈ R nr×nr , we multiply V l−1 by dynamically generated edge specific weight Θ l . Finally, V l is computed by combining the previous layer embedding and the average message passing between ROIs. (B) Recurrent graph convolution. First, the graph recurrent-filter network creates the message between ROIs i and N (i) by taking hidden state value h ij in contradiction to the conventional edge-conditioned filter graph generation. To inherit previous layer embedding V l−1 ∈ R nr×nr , we multiply V l−1 by dynamically generated edge specific weight Θ l . Finally, V l is computed by combining the previous layer embedding and average message passing between ROIs.
Parameter setting. In Tab 2, we report the mean absolute error between ground-truth and synthesized brain graphs at follow-up time points t 1 and t 2 .
In Tab 3, we publish the required training time for each comparison method respectively. We set hyperparameters of each mapper as follows: λ 1 = 1, λ 2 = 10. We used AdamW [20] optimizer and set the learning rate at 0.0001 for each mapper. Finally, we trained our model by using 3-fold cross-validation for 200 epochs using an NVIDIA Tesla V100 GPU. Comparison Method and evaluation. Due to the lack of RNN-based comparison methods that consider time-dependency for brain graph prediction, we benchmarked our RBGM against some of its variants. We call the first benchmark method: RBGM (w/KL), where we replaced topological loss with KL-divergence loss between the predicted graph at t i and its corresponding ground-truth brain graph. The KL-divergence minimizes the discrepancy between ground-truth and predicted connectivity weight distribution at each time point t i . The second benchmarking method is against the current state-of-the-art EvoGraphNet [6] in order to assess the power of topological loss and the recurrent graph convolution. In Tab 2 we report the mean absolute error (MAE) between ground-truth and predicted brain graphs for consecutive time points t 1 and t 2 for each comparison method. Our proposed RBGM outperformed baseline methods at t 1 by achieving both the lowest mean MAE (averaged across the 3 folds) and the overall best MAE as shown in Tab 2. However, for t 2 , EvoGraphNet achieved both the best MAE and mean MAE results. Notably, results show that our RBGM closely matches the best results at time point t 2 by an error difference of 7 × 10 −3 in mean MAE yet outperformed EvoGraph-Net in time consumption by achieving 46% less training time. To the best of our knowledge, such time/complexity/error is a delicate compromise to make. Under such a compromisation paradigm, we can fairly judge the outperformance of our proposed RBGM is matching state-of-the-art results given less complexity and time consumption. Overall, our RBGM performs almost similar to the state-of-the-art method EvoGraphNet while speeding up the training and reducing the complexity Tab 3 since it consists of fewer convolutional layers. However, our RBGM has a few limitations. So far, we have worked on brain graphs where the single edge connects only two ROIs. We aim to generalize our RGBM to handle brain hypergraphs where multiple edges can link two ROIs in our future work. This will enable us to better model and capture the complexity of the brain as a highly interactive network with different topological properties.

Conclusion
In this paper, we proposed the first edge-based recurrent graph neural network RBGM that uses a novel recurrent graph convolution to predict the brain connectivity evolution trajectory from a single time point. Our architecture contains m number of mappers for m time points. We proposed a time-dependency loss between consecutive time points and a topological loss to preserve topological consistency between predicted and ground-truth brain graphs at the same time point. The results showed that our time-dependent RBGM achieved a similar prediction accuracy compared to the state-of-the-art EvoGraphNet while reducing the training time and complexity. The RBGM is generic and can be used to predict brain graphs for any given time point. In future studies, we aim to generalize our RBGM to using hypergraphs and account for brain hyperconnectivity.

Acknowledgements
This work was funded by generous grants from the European H2020 Marie Sklodowska-Curie action (grant no. 101003403, http://basira-lab.com/normnets/) to I.R. and the Scientific and Technological Research Council of Turkey to I.R. under the TUBITAK 2232 Fellowship for Outstanding Researchers (no. 118C288, http://basira-lab.com/reprime/). However, all scientific contributions made in this project are owned and approved solely by the authors.