Adaptive Divergence-based Non-negative Latent Factor Analysis

High-Dimensional and Incomplete (HDI) data are frequently found in various industrial applications with complex interactions among numerous nodes, which are commonly non-negative for representing the inherent non-negativity of node interactions. A Non-negative Latent Factor (NLF) model is able to extract intrinsic features from such data efficiently. However, existing NLF models all adopt a static divergence metric like Euclidean distance or {\alpha}-\b{eta} divergence to build its learning objective, which greatly restricts its scalability of accurately representing HDI data from different domains. Aiming at addressing this issue, this study presents an Adaptive Divergence-based Non-negative Latent Factor (ADNLF) model with three-fold ideas: a) generalizing the objective function with the {\alpha}-\b{eta}-divergence to expand its potential of representing various HDI data; b) adopting a non-negative bridging function to connect the optimization variables with output latent factors for fulfilling the non-negativity constraints constantly; and c) making the divergence parameters adaptive through particle swarm optimization, thereby facilitating adaptive divergence in the learning objective to achieve high scalability. Empirical studies are conducted on four HDI datasets from real applications, whose results demonstrate that in comparison with state-of-the-art NLF models, an ADNLF model achieves significantly higher estimation accuracy for missing data of an HDI dataset with high computational efficiency.


I. INTRODUCTION
HIGH-DIMENSIONAL AND INCOMPLETE (HDI) matrix [1-4, 7-9, 13-17, 36, 47] is commonly adopted to describe specific relationships among numerous nodes in big-data-related applications like user-item interactions in a recommender system [1-4, 11, 12] and user trustness in a social network services system [5][6][7][8]. In general, Owing to the impossibility of maintaining the full connections among involved nodes (e.g., it is impossible to make a user interact with all the others in a large-scale social network services system like Wechat [9,10]), an HDI matrix commonly has only a few known entries (describing the observed interactions) while the most others are unknown rather than zeroes (describing the unobserved ones).
Although an HDI matrix is highly incomplete, it has rich knowledge regarding desired patterns like users' potential favorites in a recommender system [18,19]. Hence, a data analytics model is highly needed to acquire such knowledge from an HDI matrix. According to prior research [3,14,[21][22][23], a latent factor (LF) model is highly efficient for extracting essential features from an HDI matrix. However, it mostly fails to fulfill the non-negativity constraints, which are desired in describing the inherent non-negativity of node interactions [24][25][26]39 A Non-negative Matrix Factorization (NMF) model [27][28][29][30][31][32][33]35] is ubiquitously adopted to represent a full matrix filled with non-negative data. However, they are either incompatible with an HDI matrix [14][15][16][17], or yield unnecessarily high cost in computation and storage to achieve such compatibility [34,35]. In order to perform non-negative latent factor analysis on an HDI matrix conveniently and efficiently, Luo et al. [41] propose a Single Latent Factor-dependent, Non-negative and Multiplicative Update (SLF-NMU) algorithm to build a Non-negative Latent Factor (NLF) model. Due to the data density-oriented characteristic of an SLF-NMU algorithm, an NLF model's computational and storage cost is nearly linear with the known entry count of an HDI matrix, which is significantly lower than an NMF model [27][28][29][30][31][32][33][34][35].
To improve the convergence rate of an NLF model, Luo et al. [40] further incorporates the principle of an alternating-direction-method of multipliers into it, thereby achieving an Alternating-direction-method-based Non-negative Latent Factor (ANLF) model. Moreover, to make an NLF model compatible with general and efficient learning algorithms such as stochastic gradient descent (SGD) [45], Luo et al. [43] further propose an Inherently Non-negative Latent Factor (INLF) model that adopts a non-negative bridging function to ensure the non-negativity of the output latent factors, thereby making its optimization process compatible with a general algorithm like SGD. the above NLF models' learning schemes must be designed with care for model non-negativity. To date, the above mentioned NLF models are widely adopted in various applications for HDI data representation learning [2,14,15].
In spite of their high efficiency in addressing an HDI matrix, existing NLF models [2,40,41,43] all adopt a static divergence metric, like the Euclidean distance [20] or α-β-divergence [2]. As discussed in [2,14], an NLF model's representation learning ability to an HDI matrix can be improved via tuning the divergence parameters that affects the expression of α-β-divergence. However, the divergence parameters are data-dependent and should be manual-tuned carefully to achieve such positive results. From this point of view, we have encountered the research question: RQ. Is it possible to achieve an adaptive divergence-based NLF model whose divergence metric is self-adaptive?
To answer this question, this paper proposes an Adaptive Divergence-based Non-negative Latent Factor (ADNLF) model with the following three-fold ideas: a) it adopts a generalized objective function with the α-β-divergence, thereby expanding its potential of representing various HDI data, b) it facilitates a non-negative bridging function to connect the optimization variables with output latent factors, thereby achieving a flexible training process compatible with a general learning algorithm as well as keeping model non-negativity constantly, and c) it makes its divergence parameters adaptive through particle swarm optimization, thereby establishing adaptive divergence in its learning objective to achieve high scalability on HDI data from various domains.
This paper mainly contributes in the following perspectives: a) it presents an ADNLF model, which can perform highly-accurate representation learning to HDI data from various domains, and b) Empirical studies on eight HDI matrices testify the excellent performance of the proposed ADNLF model in comparison with state-of-the-art models.
Section II states the preliminaries. Section III presents the methodology. Section IV conducts the empirical studies. Finally, Section V concludes this paper.

II. PRELIMINARIES
As our fundamental data source, an HDI matrix is defined as [14][15][16][17]: Definition 1: Given two large node sets U and I, let R |U|×|I| quantify a certain kind of interactions between them with each entry r u,i denoting the interaction between nodes u∈U and i∈I. Let Ʌ and Γ denote R's known and unknown node sets, R is an HDI matrix if |Λ|≪|Γ|. An NLF model tries to build a low-rank approximation to an HDI matrix, which is defined as follows: Definition 2. Given R and Ʌ, an NLF model seeks for a rank-f approximation R̃ to R based on Ʌ with R=PQ T as P |U|×f , Q |I|×f ≥0.
For such purposes, a Euclidean distance-based objective is: , where pu,  and qi,  denote the u-th and i-th row LF vectors of P and Q, denotes the estimate to ru,i∈Λ by the NLF model, εu,i denotes the instant loss on the training instance ru,i∈Λ, ||•||2 computes the L2 norm of an enclosed vector, and λ is regularization coefficient, respectively.

A. Generalized Objective Function
As discussed in [44], α-β-divergence is a generalized divergence parameterized by the divergence parameters α and β. By adjusting the divergence parameters, a wide range of divergences can be achieved. With it, (1) is extended as: , . . , 0; , : Note that when α=β=1 in (2), (1) can be achieved. Hence, the Euclidean distance-based objective (1) is actually a special case of the α-β-divergence-based objective (2). Moreover, the border conditions of α=0 and/or β=0 are not considered in (2) for preventing the divided-by-zero error.

B. Non-negative Bridging Function
Note that P and Q in (2) play the roles of output LF matrices and optimization parameters simultaneously. Therefore, the learning scheme should be designed with care to fulfill the non-negativity constraints like an SLF-NMU-based one. Nonetheless, such a specifically-designed learning scheme is not compatible with a general and efficient learning algorithm like SGD. This bottleneck can be overcome by incorporating a non-negative bridging function into the objective, thereby separating the constraints from the optimization variables: where g(•) denotes a non-negative bridging function, x u,. and y i,. denote the optimization variables corresponding to p u,. and q i,. , respectively. By substituting (3) to (2), we reformulate the objective as: , . . , 0; 2 In (4), the optimization is taken with the optimization variables X and Y, and the output LF matrices P and Q are guaranteed to be non-negative through g(•) as in (3). Thus, the optimization process is no long subject to the non-negativity constraints and becomes comparable with a general learning algorithm like SGD [24][25][26]46]. Note that the bridging function should be carefully chosen to ensure the performance of a resultant model. According to prior research [37], a sigmoid function can ensure a model's high representation learning ability to non-linear patterns in HDI data. However, a sigmoid function is constantly positive and unable to describe the soft-margins as discussed in [38]. Therefore, we further introduce a boundary threshold into the bridging function for representing such margins as follows: where ι denotes the threshold constant. With (5), when an output LF becomes small enough during the optimization process, it is then set at zero to describe the soft-margins between involved nodes, thereby boosting the representation learning ability of a resultant model. In general, we set ι at 5×10 -5 empirically.

D. Adaptive Divergence
We adopt the principle of Particle Swarm Optimization (PSO) [48] to make these four hyper-parameters, i.e., α j , β j , η j and λ j , adaptive. We start with building a swarm of q particles in a four-dimensional space, where the j-th particle maintains an individual group of hyper-parameters. Following the PSO principle, the evolution rule of the j-th particle at the t-th iteration is formulated as: where 1 t j pb  and 1 t gb  denote the best position of the j-th particle and the whole swarm in the (t-1)-th evolving iteration. Note that each particle's position and velocity should be bounded to prevent jumping out of the searching space [48]. ∀j∈{1,…,q}, (αj, βj, ηj, λj) is limited in the range of (  [48]. Note that in (8), each particle determines its next move based on pbj and gb, which are updated as: where F(•) denotes the fitness function defined as:  (11) where |•| abs calculates the absolute value of a given value, Ω represents the validation set, and r( j)u,i is the estimate by the j-th particle, respectively. With (8)(9)(10)(11), α, β, η and λ are made to be self-adaptive, thereby achieving adaptive divergence.

A. General Settings
Evaluation Protocol. For industrial applications [2,3,14,15], one major motivation to address an HDI matrix is missing data estimation. Note that the RMSE is commonly chosen as the evaluation metric for the estimation accuracy of a tested model: where |•| calculates the cardinality of an enclosed set, and Ф denotes testing dataset disjoint with Λ and Ω, respectively. Datasets. Four HDI matrices collected by real industrial applications are adopted in our experiments, whose details are summarized in Table I. Note that known entry set of each HDI matrix is randomly split into 10 disjoint and equally-sized subsets, where seven subsets are chosen as the training set, one as the validation set, and the remaining two as the testing set for 70%-10%-20% train-validation-test settings.
General Settings. For achieving the objective results, following general settings are applied to all involved models: a) the LF matrices of each model are initialized with the same randomly generated arrays in each single test on the same dataset to eliminate the initialization bias; b) the dimension of the LF space is set as f=20 [14,15]; c) for ADNLF, the searching space is [0.1, 1.5] for α and β, and [2 -8 , 2 -4 ] and [2 -7 , 2 -3 ] for η and λ, respectively; and d) in PSO, we choose the common parameter settings [48], i.e., w=0.729, q=10, c1=c2=2, r1 and r2∈(0,1) and is generated by a uniform distribution.

B. Parameter Sensitivity 1) Effects of α and β
In this part of experiments, we call this manually-tuned ADNLF model as the ADNLF-M. Figs. 1 depicts its RMSE as α and β vary. Table II records ADNLF-M's lowest RMSE with optimal α and β versus its RMSE as α=β=1. From them, we have the following findings: a) With appropriate α and β, ADNLF-M represents an HDI matrix more precisely than the most frequently adopted Euclidean distance. For instance, as shown in Fig. 1(a) and Table II, on D1, the lowest RMSE is 0.8208 with α=1.2 and β=0.1. However, its lowest RMSE is 0.8262 with α=β=1, i.e., with Euclidean distance. The accuracy gain with the appropriately-tuned α=β-divergence is 0.65%. Similar situations are also encountered on the other datasets. b) The optimal α and β are data-dependent. For instance, as shown in Table II, on D3 ADNLF-M achieves the lowest RMSE with α=1.2 and β=0.1. However, on D4, the optimal values are α=0.2 and β=1.5.  Fig.1. ADNLF-M's RMSE as α and β vary.

2) Effects of η and λ
We fix α=β=1 to test the effects of η and λ in this part of experiments. Fig. 2 depicts ADNLF-M's RMSE as they vary. As shown in Figs. 2, the optimal η and λ in ADNLF-M are also data-dependent, which should be chosen carefully to ensure ADNLF-M's performance. From this part of experiments, we evidently see that the adaptive divergence of ADNLF is vital for its practicability. Without this mechanism, an ADNLF-M model tends to suffer from unstable estimation accuracy for missing data of an HDI matrix. In addition, the divergence and learning parameters should be tuned with care, which results in high computational and time costs. Next, we compare an ADNLF-M model with the optimal hyper-parameters with an ADNLF model with adaptive divergence.

C. Self-adaptation of hyper-parameters
From Figs. 1 and 2, we clearly find that the hyper-parameters, i.e., α, β, η and λ, should be tuned with care to enable the best performance. Therefore, their self-adaptation is highly desired. In this part, we compare the performance of an ADNLF-M and a complete ADNLF models. Tables III compares their performance. These results evidently support the positive effects by the adaptive divergence. More specifically, we have the following findings: a) ADNLF's adaptive divergence does not impair its representation learning ability to HDI data. According to Table III, ADNLF achieves lower RMSE than Manual on D1, D2, D4 and D7. For instance, on D4, ADNLF achieves the lowest RMSE 1.3244 and the lowest RSME of Manual is 1.3272. Thus, self-adaptation of hyper-parameters yields an accuracy gain at 0.32%. This result is quite impressive, since ADNLF's RMSE is achieved with adaptive divergence where the hyper-parameters are all adaptive. In comparison, ADNLF-M's hyper-parameters are all tuned carefully to achieve its lowest RMSE. Considering D3, D5, D6 and D8, ADNLF's RMSE is slightly higher than that of ADNLF-M. However, according to Table III, the accuracy loss is low. For instance, on D8, ADNLF's RMSE is 1.2232 while ADNLF-M' RMSE is 1.2225, where the accuracy loss is 0.06%. b) ADNLF's adaptive divergence greatly improves its computational efficiency. As shown in Table III

D. Comparison against State-of-the-art Models
In this part of the experiments, we compare the proposed ADNLF with several state-of-the-art models on estimation accuracy and computational efficiency. Table IV summarizes the details of all the models. Note that involved LF models, i.e., M1-4, are run on a Tablet with a 2.6 GHz i7 CPU and 16 GB RAM in JAVA SE 7U60. Considering M5-7, due to their computational requirements, they are deployed on a GPU platform with two NVIDIA GeForce RTX 3090 GPU cards. Note that the batch size is set at 128 uniformly for neural network-based models.
Table V records their lowest RMSE and converging total time costs. From them, we have the following important findings: a) M1 significantly outperforms its peers in estimation accuracy for missing data of an HDI matrix. As shown in Table V, M1 can achieve the highest estimation accuracy on D1-8. For instance, on D1, the lowest RMSE of M1 is 0.8199, which is 1.38% lower than 0.8314 by M2, 1.56% lower than 0.8329 by M3, 7.19% lower than 0.8835 by M4, and 0.59% lower than 0.8248 by M5, 12.64% lower than 0.9386 by M6, and 9.39% lower than 0.9049 by M7. Hence, M1's ability to estimate an HDI matrix's missing data is impressive. b) M1's computational efficiency is competitive. Note that the neural network-based models, i.e., M5-7, they perform lots of matrix operations to implement backward propagation (BP)-based training. Therefore, they depend heavily on the GPU-based acceleration. However, as shown in Table V, we clearly see that M1 only consumes more time than M6 on D3. On other datasets, M1's computational efficiency is obviously higher than that of M5-7. Note that M1's computational efficiency is lower than that of M2-3. The main reason is that M1 frequently calls the function of f(z)=e -z as z be an arbitrary real number as discussed in Sections III-B-C. Compared with M4, M1 achieves higher computational efficiency on all datasets except on D3. However, note that when compared with the time costs M2-4, M1's time cost keeps in the same order of magnitude. In addition, M1 outperforms M2-4 significantly in estimation accuracy for missing data of D1-8. Hence, M1's computational efficiency is competitive.

V. CONCLUSIONS
This study proposes an ADNLF model for performing highly accurate representation on an HDI matrix. Compared with the state-of-the-art models, it enjoys a) high representation learning ability to HDI data, and b) high scalability to handle real problems owing to its adaptive divergence.