reMap: Relabeling Multi-label Pathway Data with Bags to Enhance Predictive Performance

We present reMap (relabeling multi-label pathway data based on bag approach), a simple, and yet, generic framework, that performs relabeling examples to a different set of labels, characterized as bags. A bag is comprised of a subset of correlated pathways, and a pathway is allowed to be mixed over bags, constituting an overlapping pathway over a subset of bags. Bag based approach was followed to overcome low sensitivity scores of triUMPF for the pathway prediction task. The relabeling process in reMap is achieved by alternating between 1) assigning bags to each sample and 2) updating reMap’s parameters. reMap’s effectiveness was evaluated on metabolic pathway prediction where resulting performance metrics equaled or exceeded other prediction methods on organismal genomes with improved sensitivity score. Availability and implementation: The software package is published on github.com/reMap. Contact: shallam@mail.ubc.ca


Introduction
Recently, triUMPF [3] was proposed to reconstruct metabolic pathways from organismal and mutli-organismal genomes. This method uses meta-level interactions among pathways and enzymes within a network to improve the accuracy of pathway predictions in terms of communities represented by a cluster of nodes (pathways and enzymes). Despite triUMPF's predictive gains, its sensitivity scores on pathway datasets left extensive room for improvement. Here, we present an alternative approach called reMap that relabels each example with a new label set called "bag" which can employed by an appropriate pathway prediction algorithm to improve prediction performance. A bag is defined as a subset of correlated pathways (from a pathway label set) and a pathway may be inter-mixed across bags with different proportions, resulting in an overlapping pathway over a subset of bags, i.e., non-disjoint bags.

Methods
The reMap pipeline (the top panel in Fig. 1) iteratively alternates between the following two phases: Phase 1. Feed-Forward. During this stage, a minimal subset of bags is picked to tag each example in a three-step process: 1. Constructing Bag Pathway. In this step, pathways are partitioned into non-disjoint bags using any correlated models defined in Supp. Section 2. Note that this step is applied once during the initial round.
2. Building Bag Centroid. Having obtained a set of bags, reMap computes centroids for each bag to harness the relative association of each pathway to each bag's centroid. This ensure that pathways within a bag are semantically "close enough" to the center of that bag while also ensuring similar semantics among bags with overlapping pathways. To extract semantical information represented by pathways, the pathway2vec framework [1] is applied to automatically generate features for pathway inference. At the end of this step, reMap produces a maximum number of expected hypothetical bags for each example.
3. Re-assigning Pathway Labels to Bags. In this step, each example in a given dataset is re-annotated with a set of optimum multi-label bags using centroids and a history profile that stores bags probability is obtained from previous iterations. Figure 1: The top figure demonstrates the two-phased reMap workflow: feed-forward and feed-backward. The forward phase is consists of three components: (b) pathway bag construction to build correlated bags given multi-label data (a), (c) bag centroid determination that retrieves centroids of bags, and (d) pathway label re-assignment for each sample to bags. The feed-backward phase (e) optimizes reMap's parameters to maximize accuracy of mapping examples to bags. The process is repeated τ times. If the current iteration q reaches a desired number of rounds τ , the training is terminated while producing the multi-label bag data (f). The bag dataset then can be used to train a multi-label learning algorithm (e.g. leADS) (g). The bottom figure shows pairwise similarity scores (measured by cosine distance) among golden T1 data according to EC, pathway, and bag spaces for reMap+SOAP.

The reMap Method
In this section, we provide important notations and definitions that used throughout this paper and then state our research problem. All vectors are assumed to be column vectors and are represented by boldface lowercase letters (e.g., x) while matrices are encoded by boldface uppercase letters (e.g., X). The X i matrix indicates the i-th row of X and X i,j denotes the cell entry (i, j) of X. A subscript character to a vector, x i , denotes an i-th cell of x. Occasional superscript, X (i) , suggests an index to a sample or current epoch during the learning period. The sets are characterized by calligraphic letters (e.g., E) while we use the notation |.| to denote the cardinality of a given set. With these notations, we introduce the problem examined in this paper, starting with a multi-label dataset definition.  t ] ⊆ {−1, +1} t is a pathway label vector of size t that represents the total number of pathways, which themselves are derived from a set of universal metabolic pathway Y. The entry +1 (or −1) indicates presence (or absence) of a pathway corresponding the example i. The matrix form of x (i) and y (i) are symbolized as X and Y, respectively.
Both E and Y are extracted from reliable knowledge-bases (e.g. KEGG [18] and MetaCyc [10]). In this paper, we adopt MetaCyc. Moving on, We define the term bag, which is borrowed from multi-graph classification (MGC) technique [32]. In MGC, the goal is to learn a model from a set of labeled bags each containing several graphs. A bag is tagged positive, if at least one graph in the bag is positive, and negative otherwise. Here, we slightly abuse the term and reserve it to describe a composition of several correlated pathways. Bags are also assumed to be correlated, i.e, non-disjoint, and can be modeled by a Gaussian covariance matrix, denoted by Σ ∈ R b×b . Each entry s i,j in Σ characterizes the i-th bag association with j-th bag, where a larger score indicates both bags are highly correlated. This correlation can be discovered using either SOAP or SPREAT given pathway datasets, (Y) introduced in Def. 1.1, incorporating pathway abundance information which can be obtained by mapping enzyme with abundances onto pathways. Both models are a form of mixedmembership hierarchical Bayesian network, where each example is encoded as a vector of bag probabilities and each bag, in turn, is comprised of a set of correlated pathways. The pathways are permitted to be inter-mixed across bags with different proportion, resulting in overlapping pathways over bags.
These two models extend the functions of CTM (correlated topic model) [7] by incorporating dual sparseness and supplementary pathways in modeling bag proportions. That is, SOAP and SPREAT encodes missing probable pathways as "background" without compromising the original pathway labels, i.e., Y. The supplementary pathways are stored in a matrix M ∈ Z n×t ≥0 , where each entry is an integer value indicating the abundance of the pathway associated with a specific sample and n corresponds the number of samples. These background pathways can be modeled as in the case of SPREAT or employed directly as in SOAP. With regard to dual sparseness, it is applied to encourage selection of: i)-few focused bags for each individual example and ii)-few focused pathways for each bag. For thorough discussions about these models, see Section 2.
As a result of correlation, we define the following two terminologies: bag feature vector and bag's neighbor.   Fig. 2, where 2-dimensional feature vectors for bags encode presence or absence of two bags B 1 and B 2 , given a set of 6 pathways and bag-pathway association information, depicted as a cloud glyph. Definition 1.4. Bag's Neighbors. A bag B c ∈ B is said to be a neighbor to another bag B j ∈ B s.t. c = j, iff there exits an intersected pathway l in both bags, i.e., B c,l ∧ B j,l = 1.
With the above definitions, we formulate the problem in this work. Figure 2: An example of feature vectors for bags. The subfigure in the left represents the feature vector for six pathways corresponding to two instances. The right subfigure indicates two bags, B 1 and B 2 , and their features for the same two instances, where the first sample, D 1 , suggests that B 1 is positive because the corresponding pathways y 3 and y 4 are present, while the bag feature vector for the second example, D 2 , suggests that both bags are present.
Problem Statement 1. Given a set of bags B and a multi-label pathway dataset S, the goal is to learn an optimum relabeling function h bag : X → {+1, −1} b , such that leveraging bags to X incurs a high predictive score for the downstream multi-label pathway prediction task.
To better crystallize the idea of incorporating bags in the pathway prediction, consider the following example.
Example 1.4.1. Fig. 1 illustrates the benefit of incorporating bags for multi-label pathway classification (right panel). Here, a dataset consists of two bags, each of which groups a set of 4 correlated pathways. To determine positive pathways (y 2 , y 3 , and y 4 ) given X i , we first predict the relevant bag, indicated by +, then classify pathways within that bag. In contrast, the traditional multi-label classification approaches (left figure), mostly based on binary relevance technique, proceeds on predicting multiple pathway labels for X i .
Relabeling a multi-label dataset S will result in another dataset of S bag .
indicates the presence/absence of the associated bag that is inherited from a set B = {B 1 , B 2 , ..., B b }. Each bag B c ∈ {−1, +1} t is presumed to contain a subset of correlated pathways, i.e., Y c ⊆ Y. The presence or absence of a pathway in bag c is indicated by +1 or −1, respectively. The matrix representation of B is B ∈ {−1, +1} t×t . In addition, bags are correlated through overlapping pathways, i.e., for two correlated bags B c and B j s.t. c = j, there exits an intersected pathway l in both bags, i.e., B c,l ∧ B j,l = +1. The matrix form of bag label vector with n instances is denoted by D (∈ Z n×b ≥0 ). Now, we outline the reMap framework (depicted in Fig. 3), which iteratively alternates between the following two phases: i)-feed-forward, where it consists of three components: 1)-constructing pathway bag, 2)-building bag centroid, 3)-re-assigning labels to each example; and ii)-feed-backward to update reMap's parameters.

Feed-Forward Phase
During this phase, a minimal subset of bags is picked to tag each example according to the following three steps.

Constructing Pathway Bag
In this step, the pathways in S are partitioned into non-disjoint b bags using any correlated bag models (CTM, SOAP or SPREAT). In addition to bags, the covariance matrix (Σ ∈ R b×b ) can be obtained which is transformed to a correlation matrix ρ = C −1 ΣC −1 , where C = diag(Σ). The correlated bag models also provide us with pathway distribution over bags, denoted by Φ ∈ R b×t , where we trim the pathway distribution Φ ∈ R b×k (⊆ Φ) by keeping top k pathways for each bag, provided that aggregation of k pathways from all bags is equal to t.
Modeling pathway distribution and bag's correlations have two important implications: First, organisms encoding similar functions may share similar bags, thus, encouraging to have near-identical statistical strength (as in triUMPF); and Second, pathways that are observed to be frequently occurring together may suggest a similar relative contribution to a bag. The method consists of two phases: feed-forward and feed-backward. The forward phase is composed of three components: (b) pathway bag construction that aims to build correlated bags given multi-label data (a), (c) building bag centroid that retrieves centroids of bags, and (d) re-assigning pathway labels, for each sample, to bags. The feed-backward phase (e) optimizes reMap's parameters to maximize accuracy of mapping examples to bags. The process is repeated τ times. If the current iteration q reaches a desired number of rounds τ , the training is terminated while producing the final S bag data points (f). The bag dataset then can be used to train a multi-label learning algorithm (e.g. leADS ([3])) (g).

Building Bag Centroid
Inputs : 1 P: pathway features matrix (P ∈ R t×m ) 2 B: a set of b bags (B = {B 1 , ..., B b }) 3 α: a hyper-parameter for bags' centroids construction (α ∈ R >0 ) Outputs: 4 C: the centroids of bags (C ∈ R b×m ) Process : 5 C: 0 (∈ R b×m ); 6 for s ← 1 to b do 7 n s ← j I(B s,j = +1); 8 c s ← apply Eq. 1.1; 9 C s,: = c s ; 10 Return C Algorithm 1: BagCentroid(P, B, α) Having obtained a set of bags, we compute centroids of each bag to harness the relative association of each pathway to each bag's centroid. This procedure is described in Algorithm 1. Specifically, We assume that pathways in a bag are semantically "close enough" to the center of that bag and, hence, pathways in that bag should share similar low-level representation while also ensuring similar semantics among bags with overlapping pathways. Using this simple assumption, reMap computes the centroid c s of a bag s according to: where P ∈ R t×m is a matrix storing pathway representation obtained from pathway2vec framework [2], n s is the number of associated pathways to bag s, ||.|| is the length of a feature vector, and α is a hyper-parameter, which is determined by empirical evaluation (16 in this work). It is important to note that bags centroids enable us to obtain a maximum number of expected bags for a given example by applying a suitable metric, such as cosine similarity [23] as: where I(.) is an indicator function that results in either +1 or −1 depending on a user-defined threshold υ ∈ R >0 , c (i) s is the centroid for bag s, and D (i) is an aggregated hypothetical bags for an example i of size b after the results are vectorized into a series of +1 and −1 using vec operation. Algorithm 2 describes the pseudocode for Eq. 1.2.

Re-assigning Labels to Bags
The goal of this step is to map an example's input space onto a bag space using a decision function h bag which would produce an optimum multi-label bag set ( D opt ∈ Z n×b ≥0 ). Formally, let us denote a set of bags that are picked to relabel an example as B U }. Then, the process of re-annotation is achieved iteratively, mirroring sequential learning and prediction strategy [14], where for each example, a bag B j at round q is either: i)-added to L (i) , indicated by L or ii)-removed from the set of selected bags, represented by L More concretely, at each iteration q, we estimate the probability of an example i given the selected bags at q − 1, using threshold closeness (TC) metric [13]: is the average probability of classifying x (i) into the bag B j over values collected in H (i) q−1 that represents the history of prediction probability storing all p( D i,j |L (i) q−1 , x (i) ) before the current iteration q. The term ζ is a smoothness constant while Z is a normalization constant. Note that TC is a class conditional probability density function that encourages correct class probability to be close to the true unknown decision boundary.
To estimate p( D i,j |L (i) q−1 , x (i) ), we jointly compute the probability of bags and pathways that are associated with D i,j at round q − 1 as: where y (i) s = 1 if the pathway index s is found to be present in both bag j and in sample x (i) and 0 otherwise, and l s = 1 if the pathway index s is associated to bag j and 0 otherwise. z j,e is a normalized correlation between bags j and e, respectively, obtained from ρ (see Section 1.1.1) andc (i) j is presented in Eq 1.2. Θ bag j ∈ R m and Θ path s ∈ R r denote parameters for the bag j and the pathway s model's, respectively, and are learned during the feed-backward stage in Section 1.2.
Inputs : 1 n: number of samples (n ∈ N >2 ) 2 X: input space training set (X ∈ R n×r ) 3 Y: pathway space training set (Y ∈ Z n×t ≥0 ) 4 P: pathway features matrix (P ∈ R t×m ) 5 B: a set of b bags (B = {B 1 , ..., B b }) 6 C: the centroids of bags (C ∈ R b×m ) 7 v: a cutoff hyper-parameter (v ∈ R ≥0 ) Outputs: 8 D: the expected maximum number of bags ( D ∈ Z n×b ≥0 ) Process : 9 // an empty matrix that will contain maximum 10 // number of bags for each example 11 D ← 0; 12 // n is the number of samples 13 for i ← 1 to n do 14 // b is the number of bags 19 Return D Algorithm 2: MaxBags(n, X, Y, P, B, C, v) To reduce computational latency, instead of applying the above procedure to all bags for each example at every round, we randomly sub-sample bags of size γ. Also, the estimate is still in the probability realm, therefore, we utilize a cutoff decision threshold (β) to retrieve a subset of bags having less overlapping pathways. Afterwards, L (i) will be updated either by adding or removing bags from a previous iteration. Algorithm 3 presents the pseudocode for relabeling multi-label dataset with bags.

Feed-Backward Phase
Here, we set up a learning framework for computing reMap's bags and pathways parameters, jointly denoted as Θ = {Θ bag , Θ path }. From Eq. 1.3, it is clear that our algorithm has three learning components: i)-a hyper-plane in the bag feature space to absorb bag correlation, ii)-a hyper-plane in the pathway feature space to encode semantic information about pathways, and iii)-a joint learning between bags and pathways to exploit bag-pathway relationship. Let us define three empirical loss functions, corresponding the three components: bag : {0, 1} b → R ≥0 , path : {0, 1} t → R ≥0 , and bag-path : {0, 1} b → R ≥0 of margin dh bag (x), yh path (x), and dh bag-path (y), respectively, where h (.) are decision functions. The last two loss functions are based on the logistic loss while the first loss is a sum of the two other losses. Now, to compute Θ, we maximize the posterior probability of Eq. 1.4: Estimation of parameters in Eq. 1.5 is intractable due to the chain of probabilities H (i) q−1 and the two marginalizations over L q−1 and s. Hence, we propose the following two diagnoses: i)-conditional independence assumptions where the previous history values are independent given the most recent estimates and ii)-collapse the marginalization over L q−1 by choosing only the maximum correlation z, irrelevant to which bags were considered. These simplified treatments provide an efficient way to optimize the parameters, where we adopt the "one-vs-all" scheme learning for each bag and pathway [35].
In addition, we apply four constraints to retrieve a good set of parameters: i)-enforcing similarity between bags' and the associated pathway labels' parameters; ii)-weights of pathways, assembled in a bag, should be close to each other; iii) the input space and the label space should share similar local topological structure, which entails that if two instances in feature space exhibit strong association then they may share the same label set in the label space; and iv) the reMap' parameters should not be too large or too small.
Using the four constraints, the obtained an optimum bag set ( D opt ) in Section 1.1.3 and the given pathway label set (Y), the objective function is formulated according to: where ||.|| 2 2 represents the squared L 2 norm, ||.|| 2 2,1 is the sum of the Euclidean norms of columns of a matrix, v (i) is the weight of a sample x (i) to emphasize selection of informative samples, and λ [1,2,3,4,5] are hyper-parameters controlling the relative contributions of the associated constraint terms. Let us explain all the terms involved in Eq. 1.7. The function ||U Θ path k − Θ bag j || 2 2 reflects the first constraint, where it enforces similarities between pathways, associated to a bag j, and the bag j itself. U ∈ R r×m is the linear transformation matrix from r onto m dimensional space. For the second constraint, the term ||Θ path q − Θ path k || 2 2 considers the similarities among pathway labels, grouped under a specific bag. To adopt the third constraint, we used four terms: 2 maintains the integrity of both bag and pathway label vectors on example i, thus, encouraging bags to have similar contents as pathway labels, and W ∈ R b×t captures the correlation between bags and pathway labels. Both || d (q) − d (l) || 2 2 and ||y (q) − y (l) || 2 2 describes the resemblances between the two bag label vectors, d (q) and d (l) , and the two pathway label vectors, y (q) and y (l) , suggesting the similarity between input instances x (q) and x (k) . The similarity scores of the aforementioned instances are captured by S q,k ∈ R n×n ≥0 , where a high score, indicates both examples have near identical pathway labels and, hence, should have similar bags, and vice-versa hold as well.
The formula ||Θ path addresses the neighborhood relationship in the feature space between x (q) and x (k) , as characterized by S q,l score [33]. As discussed before, if two instances are close to each other then they may possess relevant labels, which leads to relabeling a dataset with a proper subset of bags, hence, mitigating from the negative influences of imperfectly labeling bags. The terms ||Θ bag j || 2,1 , ||Θ path j || 2,1 , and ||U|| 2,1 , constitute the fourth constraint in the form of L 2,1 -norm on all weights, which reduces the negative impact of label correlations in the model while at the same time shrink weights and perform feature selection. Finally, κ||S1 − 1|| 2 2 enforces equality constraint such that ∀q, k∈n S q,l = 1, where κ is a Lagrange multiplier, and 1 denotes a column vector with all of it's elements are equal to 1.
Taken together, the trainable parameters of reMap are: 1)-bag-projection weight matrix W, 2)-pathwayprojection weight matrix U, 3)-bag-specific weight matrix Θ bag , 4)-pathway-specific weight matrix Θ path , 5)instance-similarity specific weight matrix S, and 6)-bag-specific updating matrix D. The last parameter is a binary matrix indicating the presence/absence of bags in the training dataset, which is gradually updated based on the gradient score strategy.
Unfortunately, the objective function in Eq. 1.6 involves L 2,1 -norm that is non-smooth and difficult to be solved, instead we perform iterative gradient descent method for reMap which alternatively optimizes over one of six classes of variables (W, U, Θ bag , Θ path , S, and D) at a time while the others are held constant. The partial derivative of each term in Eq. 1.6 is a positive semi-definitive, hence, the whole term is jointly convex, which leads to the following independent optimization problems for all pathways and bags classifiers according to the multi-label 1-vs-All approach [35].
• Update W. The gradient of Eq. 1.6 w.r.t. W has the following formula: where, • Update U. The gradient of Eq. 1.6 w.r.t. U becomes: where, • Update Θ bag . The partial derivative for each bag, say Θ bag j , is: where U obtained from Eq. 1.9.
• Update Θ path . The partial derivative w.r.t one pathway k of Θ path with the new U and Θ bag updates has the following form: where L M − S is the graph Laplacian matrix and M is a diagonal matrix with M j,j = k=1 S j,k . Note that 1 2 q,l∈n S q,l ||Θ path Following the work of [27], it is important for practical purpose to normalize the graph Laplacian, to account for the fact that some examples are more similar than others [30]:L M −1/2 LM −1/2 = I − M −1/2 SM −1/2 . Adhering to this property, we consider the following formula: • Update S. Given the updated values of U and Θ path , we obtain the equivalent objective function of Eq. 1.6 with the terms only related to S as: For the inequality constraint, during iterative updates we force values of S to be within the range of [0, 1]. Then, the gradient update can be written as: As we have mentioned, the proposed similarity matrix S captures reliable and discriminative locality information in the projected feature space, and this information is utilized to optimize correlations in the predicted label space, which ensures feature-label space consistency. Consequently, a set of bags can be inferred with high fidelity, for each example, if features with labels correlation information is disseminated to features with bags correlations, thus, alleviating the effects of imperfectly detecting negative bags. That is what exactly S is expected to behave.
• Update D. We iteratively update bags, where a set of bags are added or removed at each round. In particular, a positive subset of bags is selected B (i) P ⊆ arg{ D i,j = +1 : ∀j}, for each example x (i) , and the remaining bags are considered to be negative to that example. While it is relatively easy to compile a set of positive bags for an example, however, bags not belonging to that example are too diverse to be considered as negative. Thus, it is better to consider the remaining bags as unassigned B (i) U . We use the gradient score strategy, where the values of D is updated based on the gradient score according to: After getting the gradient score, we ascertain the labels of examples based on: where D i,j = +1 (resp. −1 and 0) means the bag is selected to be positive (resp. negative and unknown) given a training example i. Having acquired a new selected set of bags for each instance, we update L (i) accordingly.

Correlated Models
In this section, we present the CBT (correlated bag-pathway) package that consists of three hierarchical Bayesian (parametric) mixture models: i)-CTM (correlated topic model), ii)-SOAP (sparse correlated bag pathway), and iii)-SPREAT (distributed sparse correlated bag pathway) that incorporate pathway abundance information to encode each example as a mixture distribution of bags, and each bag, in turn, is a mixture of pathways with different mixing proportions.
But first let us provide some background and notations. We note that each mathematical symbol is only related in the context of this section. Recovering latent distributions of P mirrors the concept modeling paradigm, which was introduced to reconstruct the thematic structure, called "topics", from a corpus [9]. Definition 2.2. Concept Modeling. Given a collection of n samples, a concept distribution for i-th example is a multinomial distribution vector, denoted by η (i) of size b concepts, i.e., {p(Φ a |η (i) )} a=b a=1 , where Φ j in a multinomial feature distribution over the concept j, i.e., {p(y k |Φ j )} k=t k=1 . The overall goal of concept modeling is to recover the b salient concepts of each example.
In this report, the term concept is referred to as "bag" which is coined from the multi-graph classification (MGC) technique [32], where each bag is comprised of several correlated pathways. For brevity purposes, the following terms: concept, topic, or bag, are used interchangeably. Also, features correspond to pathways.
The classical studies in concept modeling attempt to discover concepts from a collection of examples that are composed of features, as in the case of latent Dirichlet allocation (LDA) [9]. However, this approach neglects dependencies among concepts. We take advantage of the inherent thematic structure of examples and model the concept dependencies to extract the concept distributions of examples. Definition 2.3. Concept Correlation. Given P, the pairwise concept-correlation is defined by a Gaussian covariance matrix, denoted by Σ. Each entry s i,j in Σ characterizes the i-th bag association with the bag j, where a larger score indicates both concepts are highly correlated.
However, due to the inherent noise in curating P there are situations where a set of pathways may not included in P for a set of examples. An alternative way to incorporate missing pathways is to record them in a separate list while keeping the original pathway collection intact for further investigation. Lets us denote a matrix M ∈ Z n×t ≥0 which stores a set of missing pathways where each entry is an integer value indicating the abundance of the pathway associated with a specific sample. The matrix M is referred to as a "background" or "supplementary" set, analogous to studies in [36,17]. With these definitions, we describe the correlated models.

Correlated Topic Model
The correlated topic model (CTM) is a probabilistic graphical model that extends the generative story of LDA [9] to incorporate correlation among concepts. Fig. 4 (a) shows the Bayesian graphical model for CTM using plate notation. Like latent Dirichlet allocation [9], the CTM is comprised of a hierarchical Bayesian mixture model, where features (words as described in the original paper) are mixed to constitute concepts. And, the concepts, in contrast to LDA, are assumed to be correlated to each other, as reflected by a Gaussian covariance matrix. Algorithm 5: The generative process for CTM given a collection Formally, let n be the total number of a collection, where each example i consists of feature indices as y (i) . Then, the generative process for CTM is described as follows. First, we draw a multinomial feature distribution Φ a from a Dirichlet prior α > R >0 for each concept a ∈ {1, . . . , b}. Then, for each example i, a Gaussian random variable is drawn η (i) ∼ N (µ, Σ), where µ is a b dimensional mean and Σ ∈ R b×b is the covariance matrix. The random variable η (i) is projected onto the probability simplex to obtain the concept distributions θ (i) = softmax(η (i) ), corresponding the logistic-normal distribution, from which a concept indicator z (i) j ∈ {1, . . . , b} is sampled. Finally, each observed feature j ∈ {1, . . . , t i } is drawn from the associated feature distribution, as indicated by it's concept assignment, i.e., y The CTM's generative process is outlined in Algorithm 5, which can be observed that the process is identical to LDA except the concept distributions is sampled from the logistic normal rather than a Dirichlet prior.

Correlated Bag-Pathway Model
The correlated bag pathway is an extension to CTM and comes in two flavors, as depicted in Fig. 4 (b) and (c): i)-sparse correlated bag pathway (SOAP) and ii)-distributed sparse correlated bag pathway (SPREAT). Both models incorporate dual sparseness and supplementary pathways in modeling bag proportions. These important properties are not adopted in CTM. In contrast to SOAP, SPREAT is equipped to reconstruct the latent distribution of supplementary pathways. Let us formally explain both models in detail. For the purpose of coherence, we use features to define pathways as before. i. Sample a concept assignment z Algorithm 6: The generative process for SOAP and SPREAT Analogous to CTM, given n number of examples and a matrix encoding the missing features M, the generative process for SOAP and SPREAT can be described as follows. First, we draw a multinomial feature distribution Φ a from asymmetric Dirichlet prior α ∈ R >0 for each concept a ∈ {1, ..., b}, where b is assumed to be known and fixed in advance. The symmetric assumption is appropriate, in such a scenario, because our prior knowledge, associated with these features, is inaccessible. Similar to CTM, for each example i, a concept proportion is drawn θ (i) = softmax(η (i) ), where η (i) is a Gaussian random variable with mean and covariance are denoted by µ and Σ, respectability.
To sample a concept, it is reasonable to expect that each example is usually explained with a handful set of a mixed proportion of concepts, which may include interactions among them. Besides, a concept should cover only a few focused features, instead of absorbing all features. Thus, we borrow the idea from [22,1,26,6,15] to enforce dual sparsity to retain those relevant focused concepts and features by: i)-introducing an auxiliary Bernoulli variable Λ (i) of size b to determine whether a concept is selected for an example i or ignored, and ii)applying a cutoff threshold to keep only the top k t features for each concept. Instead of sampling each entry in Λ (i) directly from a Bernoulli coin toss, we assume that each entry is sampled from a Beta distribution β (i) , parameterized by two hyperparameters γ ∈ R >0 and κ ∈ R >0 . Applying this dual sparsity, we aim to enhance the interpretability of the learned concepts while minimizing the negative correlation among concepts on Σ.
Next, a concept indicator z , as indicated by it's concept assignment, using a smoothing prior ∈ R >0 .
The parameter Ω (i) ∈ R t , derived from M i , represents a normalized supplementary feature of size t, which is assumed to be drawn from a symmetric Dirichlet prior ξ ∈ R >0 . For SPREAT, this parameter encodes distribution, where each element of Ω (i) j corresponds to the example's probability of using feature y j ∈ M i . Here, the background feature is assumed to be drawn from a sparse binary vector prior ι ∈ R >0 that is included for completeness because each example's feature M i is already observed.
By representing SOAP and SPREAT as layer-wise mixing components supports the hierarchical modularity of metabolic pathway generation, where the components of one level (e.g., features) permit to contribute to other structures with different degrees of granularity (e.g., examples). The generative process of the proposed SOAP and SPREAT models is summarized in Algorithm 6. Note that by setting all entries in Ω, Λ, and to 1, SOAP and SPREAT are reduced to CTM ("collapse2ctm" or c2m), which is an additional benefit to the former models.

Evidence Lower Bound (ELBO) for SPREAT
Here, we discuss inference for SPREAT model. Similar expression is straightforward to derive for SOAP. Given P, the goal of inference is to compute the posterior distribution of the per-example concept proportions η (i) , the per-example concept selection parameters Λ (i) and the associated beta distributions β (i) , the per-example background feature distributions Ω (i) , the per-feature concept assignment z (i) j , and the per-concept distribution over features Φ a .
Looking at the topology of the Bayesian network, we can specify the complete-data likelihood, i.e., the joint distribution of all observed and latent variables given the hyperparameters and sparse supplementary feature matrix using the model's independence assumptions: By denoting all the parameters as Θ and variables as V while omitting the hyperparameters, we obtain the following posterior expression: Unfortunately, the exact posterior distribution of the latent variables is computationally intractable. The numerator is easy to compute for any configuration of the hidden variables and parameters. The problem is the denominator, which is the marginal probability of the data: Original parameter Φ µ Σ Λ Ω z Variational parameter φ ν ζ 2 λ ω ς Computing the marginal requires a complicated integral over n examples of |Θ| parameters and another integral over the |V| n configurations multiplied by the size of each variable in V. As such, we appeal to variational inference, which has been extensively employed in many complex probabilistic models. Examples include latent Dirichlet allocation (LDA) [9], sparse LDA [26], supervised topic models [24], mixed membership stochastic blockmodels [1], nested hierarchical Dirichlet process [28], and many others. The main intuition behind variational methods is to first posit a family of distributions over the hidden parameters and variables that are indexed by a set of free parameters, and then fitting the parameters to find the member of the family that is closest to the true posterior of interest in Eq. 2.2. The closeness is commonly measured using Kullback-Leibler (KL) divergence [21]. The resulting variational distribution is simpler than the true posterior so that the solution can be approximated.
However, directly minimizing the KL divergence is infeasible due to the same reason that the posterior is difficult to compute, but, we can optimize an objective function that is equal to the negative KL divergence up to a constant. This is known as the evidence lower bound (ELBO), a lower bound on the logarithm of the marginal probability in Eq. 2.3, i.e., log p(Y, M). This ELBO can be defined using Jensen's inequality on a variational distribution over the hidden variables q(Θ, V) as: The ELBO contains two terms. The first term, E q [log p(Y, M, Θ, V)], captures how well q(Θ, V) describes a distribution that is likely under the model, keeping both the priors and data in mind through the joint distribution. The second term is the entropy of the variational distribution, E q [− log q(Θ, V)], which protects the variational distribution from "overfitting" [8]. Both of these terms depend on q(Θ, V), the variational distribution of the hidden variables. The simplest variational family of distributions is the mean-field family where each hidden variable/parameter is fully-factorized and governed by its own parameter. This allows us to tractably optimize the parameters to find a local minimum of the KL divergence. For SPREAT, the mean-field variational distribution is expressed as: where φ, ν, ζ 2 , λ, ω and ς are variational free parameters. Table 1 shows the correspondence between variational and the original parameters.
Taking together, the first term in Eq. 2.4, E q [log p(Y, M, Θ, V )], can be decomposed into: (2.6) And, the second term H(q) in Eq. 2.4 can be defined as: (2.7)

Variational Lower Bound
Given Eq. 2.6, we derive expressions for each term: 1. For the concept distribution over features, which are Dirichlet-distributed, For the concepts probabilities for each example, which are Gaussian distributed, 3. For the focused concept distributions for each example, which are Bernoulli distributed, (2.10) 4. For selecting a set of focused concepts for each example, which are beta distributed, 5. For the hypothetical feature distributions for each example, which are Dirichlet distributed, (2.12) 6. For the feature assignments from both concept-feature and hypothetical feature distributions, (2.13) 7. For the concept assignments over features, the expectation of the log probability of the latent concepts is given by: The second term is hard to compute, hence, we appeal to the solution suggested by [7] in order to pertain the tightest lower bound on −E q log k=b k=1 exp(η k ) using a first-order Taylor expansion. Because the function − log is convex, a first-order Taylor expansion about the point , a variational parameter, produces the following inequality: Plugging back the results into Eq. 2.14, we obtain: Now, for the entropy H(q) in Eq. 2.7, we decompose their expectations as: 1. For the concept-feature distributions, which are Dirichlet distributed, (2.20) 5. For the feature assignments over examples, which are multinomially distributed, where the exceptions of all the above equations can be derived using: Not that Γ denotes the Gamma function while Ψ is the logarithmic derivative of the Gamma function.

Merging All the Expectations of the ELBO Terms
Now, by joining all the terms in Section 2.3.1, the full ELBO can be defined as:

Optimizing the ELBO Terms
In this section, we maximize the bound in Eqs 2.6 & 2.7 with respect to each variational parameters using coordinate ascent updates, which optimizes each variational parameter while holding the remaining variables fixed. Practically, a more convenient way is to apply the mini-batch gradient approach, as outlined in [16] that alternates between subsampling a batch of examples and updating each variational parameter, after being scaled by a learning rate. This structure of learning assists us to approximate the posterior with massive examples, making the complete problem computationally scalable.
1. Optimizing w.r.t. ς. Gathering only the terms in the bound that contain ς, we obtain: Taking derivatives w.r.t. ς (i) a,j , we obtain: The analytical expression of the variational concept assignment q(ς) for each feature y j and concept a is not amenable due to the non-conjugacy of logistic-normal with latent variables. Instead, we approximate the solution according to: where Ψ(.) is the digamma function. Observe how the variational parameter ω (i) * serves as the smoothing term in selecting concepts for each feature, either from M i or from P, when ω (i) a,j is updated based on φ a,j . 2. Optimizing w.r.t. ν. Collecting only the terms in the bound that contain ν gives, Taking derivatives w.r.t. ν a for each concept a, we obtain: where is another variational parameter, as in CTM [7]. However, the above equation in hard to optimize, instead, we use a conjugate gradient algorithm.
3. Optimizing w.r.t. ζ 2 . By symmetry, we gather all the terms that has ζ 2 from Eq. 2.22: Taking derivatives w.r.t. ζ 2 a for each concept a, we obtain: Again, there is no analytic solution. We use Newton's method for each coordinate, constrained such that ζ a ∈ R >0 .
4. Optimizing w.r.t. . Extracting the terms involving in the bound gives, Taking derivatives w.r.t. , we obtain: Equating the above formula to zero to obtain a maximum, we get: Optimizing w.r.t. ω. Isolating only the terms in the bound that contain variational background feature distributions q(ω), we obtain: Setting it's derivatives to zero does not lead to a closed-form solution, instead, we approximate ω (i) c for each sample i according to: 6. Optimizing w.r.t. λ. Collecting the terms that contain λ, we obtain: Taking derivatives w.r.t. λ (i) a , we obtain: Equating the above formula to zero to obtain a maximum, we get the canonical parameterisation of the Bernoulli distribution: Therefore, we get the following updates: Optimizing w.r.t. φ. Finally, the optimal solution of the variational concept feature distribution q(Φ a |φ a ) for each concept a is obtained by isolating terms involved in the bound Eq. 2.4: log Γ(φ a,j ) (2.40) 1 Initialize φ, ν, ζ 2 , λ, ω, ς, γ, κ, ξ, α, , ι, s = 0, l ≥ 0, g ∈ (0.5, 1] Update ν (i) with Eq. 2.27 using conjugate gradient algorithm; 9 Update ζ 2,(i) with Eq. 2.29 using Newton's method; 10 Update (i) with Eq. 2.32; 11 Update ω (i) with Eq. 2.35; 12 Update λ (i) with Eq. 2.39; 13 until local variational parameters converge; 14 Compute optimal values µ = ν |B| , Σ = diag( ζ 2 |B| ) + µµ ; 15 Compute global optimal values φ with Eq. 2.42; 16 Update the current estimate of the global variational paramters, Update the learning rate τ = (s + l) −g ; 18 until global convergence criterion is satisfied ; Algorithm 7: Stochastic variational inference for SPREAT After taking derivatives w.r.t. φ a,c , we obtain: Equating the above formula to zero to obtain a maximum, we get: The variational inference algorithm samples a mini-batch from a collection, and use it to compute the local latent parameters in Eqs 2.25, 2.27, 2.29, 2.32, 2.35, and 2.39 until the evidence lower bound (in Eq. 2.4) converges. Then, the global variational parameter φ is updated using the posteriors (β, Λ, η, z, Ω) collected from the previous step in Eq. 2.42, after being scaled according to the learning rate τ = (s + l) −g , where s is the current step, l ≥ 0 is the delay factor, and g ∈ (0.5, 1] is the forgetting rate. We summarize our variational inference in Algorithm 7.

Posterior Predictive Distribution for SPREAT
We apply posterior predictive distribution to evaluate model's fitness, which estimates the distribution of unobserved values given the observed values and parameters trained on a held-out training set [16]. This metric is useful in evaluating models as it avoids comparing bounds of those models. The predictive distribution is formulated as follows:

Experimental Setup
In this section, we describe the experimental settings and outline the materials used to evaluate the performance of reMap. The reMap and CBT were written in Python v3 and depends on third party libraries (e.g. Numpy [31]). Unless otherwise specified all tests were conducted on a Linux server using 10 cores of Intel Xeon CPU E5-2650.

Description of Datasets
We evaluated reMap's performance using a subset of datasets used in triUMPF [4]: i)-6 T1 golden data that are composed of six databases, retrieved from biocyc website: EcoCyc (v21), HumanCyc (v19.5), AraCyc (v18.5), YeastCyc (v19.5), LeishCyc (v19.5), and TrypanoCyc (v18.5), and are refined to include only those pathways that cross-intersect with the MetaCyc database (v21) [10]; ii)-Critical Assessment of Metagenome Interpretation (CAMI) initiative low complexity dataset consisting of 40 genomes [29] that can be obtained from edwards.sdsu.edu/research/cami-challenge-datasets/; and iii)-BioCyc (v20.5 T2 & 3) [11], which consists of 9255 PGDBs (Pathway/Genome Databases) with 1463 distinct pathway labels and is constructed using the Pathway Tools v21 [20]. We also employed Synset-2, a noisy corrupted training set, introduced in [5]. The detailed characteristics of the datasets are summarized in Table 2. For each dataset S, we use |S| and L(S) to represent the number of instances and pathway labels, respectively. In addition, we also present some characteristics of the multi-label datasets, which are denoted as: where I is an indicator function. It denotes the average number of pathways in S.

Label density (LDen(S) = LCard(S) L(S)
). This is simply obtained through normalizing LCard(S) by the number of total pathways in S.
3. Distinct label sets (DL(S)). This notation indicates the number of distinct pathways in S.

Parameter Settings
reMap depends on both correlated bag pathway models (Section 2) and pathway2vec [2], therefore, we applied the following default configurations for each component: • The parameters for reMap are configured as: the learning rate η = 0.0001, the batch size to 30, the number of epochs to τ = 10, the bag centroid hyperparameter α = 16, the cutoff threshold for cosine similarity v = 0.2, the cutoff decision threshold for bags β = 0.3, the number of bags b = 200, and the subsampled bag size γ = 50. For the regularized hyperparameters λ 1:5 and κ, we performed 10-fold cross-validation on a subsampled of (v20.5 T2 &3) collection and found the settings λ 1:5 = 0.01 and κ = 0.01 to be optimum on golden T1 datasets.
• The parameters for three correlated models are configured as: the pathway distribution over concepts Φ were initialized using gamma distribution (with shape and scale parameters were fixed to 100 and 1/100, respectively), the forgetting rate to g = 0.9, the delay rate to l = 1, the batch size to 100, the number of epochs to 3, the number of concepts b = 200, top k pathways to 100 (only for SOAP and SPREAT), the Dirichlet hyperparameters α and ξ to 0.0001, and the beta hyperparameters γ and κ to 2 and 3, respectively. The supplementary pathways M for BioCyc, CAMI, and golden T1 datasets were obtained using mlLGPR [5]. We train mlLGPR (elastic-net) using enzymatic reaction and pathway evidence features where hyperparameters were fixed to their default values. A schematic view of pathway frequency across datasets for BioCyc T2 & 3 and CAMI, along with their augmented pathways is depicted in Fig. 6.
• The parameters for pathway2vec are configured as: "crt" as the embedding method, the number of memorized domain is 3, the explore and the in-out hyperparameters are 0.55 and 0.84, respectively, the number of sampled path instances was 100, the walk length is 100, the embedding dimension size was m = 128, the neighborhood size was 5, the size of negative samples was 5, and the used configuration of MetaCyc was "uec", indicating links among ECs are being trimmed. Both reMap and correlated models were trained using BioCyc (v20.5 T2 &3) collection. After obtaining bags S bag , we train leADS [3] using built-in "factorization" option that enables training bags and pathways, simultaneously, for 10 epochs while the prediction strategy was set to "pref-vrank" with the ranking hyperparameter was fixed to 300. For all the remaining hyperparameters in pathway2vec, CBT, leADS, and mlLGPR [5], they are fixed to their default values.

Experimental Results and Discussion
We conducted four experimental studies: parameter sensitivity and visualization for correlated models, assessing prediction probability history during relabeling process of reMap, and metabolic pathway prediction effectiveness after obtaining S bag using leADS.

Sensitivity Analysis of Correlated Models
Experimental setup. A fundamental challenge for the reMap pipeline is to acquire a good distribution of bags and pathways from correlated models for the purpose of relabeling. Following the common practice, here we examined various hyperparameters associated with correlated models. First, we compared the sensitivity of SOAP and SPREAT against CTM by incorporating the background pathways M while varying the number of bags according to b ∈ {50, 100, 150, 200, 300}. Next, we examined the c2m option for SOAP and SPREAT to show that these two models exhibit similar performances as CTM. Finally, we conducted sparsity analysis of bag distribution by varying the cutoff threshold value according to k ∈ {50, 100, 150, 200, 300, 500}. For the comparative analysis, we used CAMI as a test data to report the log predictive distribution (Section 2.5), where a lower score entails higher generalization capability for the associated models. Experimental results. While the log predictive scores for SOAP and SPREAT in Fig. 5a appears to be horizontal across bag size, the CTM model projects a more realistic view where it's performances are seen to  1  510  510  1  510  510  2182  2182  1  1034  1034  0.2337  Arabidopsis  thaliana  EcoCyc  1  307  307  1  307  307  1134  1134  1  719  719  0.2707  Escherichia  coli  K-12  substr.MG1655  HumanCyc  1  279  279  1  279  279  1177  1177  1  693  693  0.2370  Homo sapiens  LeishCyc  1  87  87  1  87  87  363  363  1  292  292  0.2397  Leishmania  major  Friedlin  TrypanoCyc  1  175  175  1  175  175  743  743  1  512  512  0.2355  Trypanosoma  brucei  YeastCyc  1  229  229  1  229  229  966  966  1    be gaining by including more bags. For the former models, this phenomena is not a consequence of design flaw, instead, it is expected due to the effects of supplementary pathways. That is, both models are encouraged to learn more pathways from M because the average pathway size for an example in M is ∼ 500 whereas in BioCyc T2 & 3 is ∼ 195 while only retaining 100 pathways for each bag. By excluding M (through enabling c2m option), we observe that the log predictive distribution of SOAP and SPREAT are similar with that of CTM, as shown in Fig. 5b, which supports our previous discussion. We found that b = 200 gives a good set of overlapping pathways while having on average ∼ 15 distinct pathways for each bag from 2526 pathways. By fixing b = 200, we search for an optimum k value. As illustrated in Fig. 5c, both SOAP and SPREAT deteriorate their performances (< −0.6) when k > 100. Taken together, we suggest the settings b ∈ Z [150,300] and k ∈ Z [50,100] to recover good bag and pathway distributions.

Bag Visualization
Experimental setup. Here, we visualized the discovered bags using models in Section 4.1 with the goal to assess the quality of bags. First, we examined the influence of augmented pathways on bag correlation patterns, i.e., Σ, in SOAP and SPREAT and contrast the outputs with CTM, SOAP+c2m, and SPREAT+c2m. Then, we performed an in-depth comparison between the sparse models (SOAP+c2m and SPREAT+c2m) with CTM to ensure that our modeling assumptions are aligned with the observed data, where a bag containing more focused and fewer pathways is preferred. For all experiments here, we applied the settings described in Section 3.2. Experimental results. From Fig. 7, we notice two core findings. First, in contrast to CTM, SOAP+c2m, SPREAT, and SPREAT+c2m, bags in SOAP are densely connected (∼ 74 bags) where the width of edges indicates the strength of correlations. Second, leveraging background pathways in SOAP and SPREAT resulted in gathering more pathways for each bag. These observations demonstrate the influence of M (obtained from mlLGPR) to pathway distribution over bags. As an example, Fig. 8 shows samples from BioCyc T2 & 3 pathways and corresponding background pathways after projecting them onto 2D space using UMAP [25]. The colors encode samples from BioCyc T2 & 3 pathways that are clustered using the K-means algorithm [19] with 10 groups. While examples of the same color in BioCyc T2 & 3 pathways form a clear distinct group, the same examples are seen to be intermixed for M, possibly comprising of many false-positive pathways as depicted in Fig. 6 where many pathways (represented as column bars) are differentially distributed between BioCyc T2 & 3 pathways and M. For the collapsed models, they are observed to share similar behaviors as CTM (Fig. 5b). However, their bag distribution consists of fewer pathways than CTM (Fig. 7). For example, Fig. 9 shows 50  Figure 7: Visualizing 50 randomly picked bags for each model, trained with b = 200. The first term within the bracket, i.e., #bags, corresponds to the average number of correlated bags while the second term, i.e., #pathways, represents the average number of pathway size per bag. The circles represent bags, and their sizes reflect the correlation strength with other bags. Two clusters of bags can be seen for the last three models indicating the two clusters contain distinct pathways. randomly selected bags with associated 100 pathways, where CTM is shown to encapsulate more pathways per bag (encoded by gradient darker colors) while SOAP+c2m and SPREAT+c2m exhibit a sparse distribution. The results from these experiments link with our previous remarks, entailing that SOAP and SPREAT are equipped to reduce irrelevant pathways by applying dual sparseness. In particular, SPREAT is observed to generate fewer correlations than SOAP. With regard to incorporating supplementary pathways, SOAP and SPREAT are both sensitive to false-positive pathways, therefore, including accurate augmented pathways may recover better pathway distribution over bags.

Assessing the History Probability
Experimental setup. After analyzing the correlated models in Sections 4.1 and 4.2, here we investigated the accumulated history probability H on relabeling golden T1 datasets, during the feed-forward phase, using the settings in Section 3.2. Recall that reMap employs four constraints in the feed-backward phase to select bags with high probability to annotate data. The hope is to obtain an optimum h bag to remapping pathways to bags. For the demonstration purposes, we only focused on SOAP to collect bags although any models can be used to conduct this experiment. Experimental results. Fig. 10 shows H during the annotation process of the six datasets. In the beginning, reMap attempts to select a set of bags from D, corresponding to the maximum number of bags that may exist for each example. However, with progressive updates and calibration of parameters, reMap rectifies bags assignments where it picks fewer bags, containing more informative contents associated with each sample. For instance, at q = 1, the HumanCyc is tagged with multiple bags, represented by darker colors where higher Figure 9: Heatmap representing bag distribution of CTM, SOAP+c2m, SPREAT+c2m for randomly picked 50 bags with their associated 100 pathways. The entries is color-coded on a gradient scale ranging from light-gray to dark-gray, where higher intensity entails higher probability. Figure 10: Snapshot of the history probability H during the relabeling process of golden T1 data for 10 successive rounds. The x-axis shows 200 bags while the y-axis corresponds data. Darker colors indicate high probability to assigning bags to the corresponding data. Figure 11: The probability history H during annotation of T1 golden data after 10 successive rounds. The x-axis shows 200 bags while the y-axis corresponds the associated probability. Darker colors indicate high probability to assigning bags to the corresponding data.   intensity indicates a high probability of assigning the corresponding bag to HumanCyc, but at q = 10 less than 35 bags were retained out of 200 possible bags. Table 3 shows the 11 HumanCyc pathways corresponding the assigned bag index 16, which includes L-proline biosynthesis I, thyroid hormone metabolism II (via conjugation and/or degradation), and bile acid biosynthesis, neutral pathway. The pathway set in bag 16 for HumanCyc is an indication that reMap is robust to a certain degree and able to capture relevant pathways for a bag. Bags preferences by reMap can be further examined by computing similarity (measured in cosine distance metric) among datasets according to EC, pathway, and bag spaces, depicted in Fig. 12. As elucidated in Section 1.2, reMap exploits four constraints to strike balance among these spaces. This is particularly sighted for AraCyc and EcoCyc, where ECs (Fig. 12a) have similar density with the remaining golden data, however, the pathways are differentially represented (Fig. 12b). Hence, this information is down propagated to the bag annotation process depicted in Fig. 12c. Overall, reMap exhibits a steady, albeit slow, annotation progression while preserving consistency across different spaces.  Table 4: Predictive performance of each comparing algorithm on 6 benchmark datasets. For each performance metric, '↓' indicates the smaller score is better while '↑' indicates the higher score is better. Bold text suggests the best performance in each column.