Sum-product networks: A survey <s> F. Online structural learning <s> Sum---product networks SPNs are deep architectures that can learn and infer at low computational costs. The structure of SPNs is especially important for their performance; however, structure learning for SPNs has until now been introduced only for batch-type dataset. In this study, we propose a new online incremental structure learning method for SPNs. We note that SPNs can be represented by mixtures of basis distributions. Online learning of SPNs can be formulated as an online clustering problem, in which a local assigning instance corresponds to modifying the tree-structure of the SPN incrementally. In the method, the number of hidden units and even layers are evolved dynamically on incoming data. The experimental results show that the proposed method outperforms the online version of the previous method. In addition, it achieves the performance of batch structure learning. <s> BIB001 </s> Sum-product networks: A survey <s> F. Online structural learning <s> Sum-product networks have recently emerged as an attractive representation due to their dual view as a special type of deep neural network with clear semantics and a special type of probabilistic graphical model for which inference is always tractable. Those properties follow from some conditions (i.e., completeness and decomposability) that must be respected by the structure of the network. As a result, it is not easy to specify a valid sum-product network by hand and therefore structure learning techniques are typically used in practice. This paper describes the first {\em online} structure learning technique for continuous SPNs with Gaussian leaves. We also introduce an accompanying new parameter learning technique. <s> BIB002
The algorithms presented so far need the complete dataset to produce a structure. However, sometimes the dataset is so big that the computer does not have enough memory to store it at once. In other situations, e.g., in recommender systems, the data arrive constantly. In these cases the learning algorithm must be able to update the structure instead of learning it from scratch every time new data arrives. In this context, Lee et al. BIB001 designed a version of LearnSPN where clustering (slicing) is replaced by online clustering, so that new sum children can be added when new data arrive, while product nodes are unmodified. Later Dennis and Ventura  extended their Search-SPN algorithm  to the online setting. This online version is as fast as the offline version that works only on the current batch and the quality of the resulting SPN is the same. Hsu et al. BIB002 created oSLRAU, an online structure learner for Gaussian leaves (oSLRAU) which begins with a completely uncorrelated SPN structure that is updated when the arriving data reveals a new correlation. The update consists in replacing a leaf with a multivariate Gaussian leaf or a mixture over its scope. Jaini et al.  proposed an algorithm, Prometheus, whose first concern is to avoid the parameter that decides when two subsets of variables are independent in order to perform a LearnSPN split. So instead of creating a product node, it creates a mixture of them, representing different subset partitions. The way the partitions are created allows them to share subsets, which is reflected in the structure by common children, thus overcoming the restriction to trees on the way. This is in some sense similar to bagging in sum nodes (cf. Sec. VI-E) and makes the algorithm robust in low data regimes. However, the complexity of the algorithm grows with the square of the number of variables. In order to extend it to highdimensional datasets, the authors created a version that samples in each step from the set of variables instead of using all of them. This algorithm can treat discrete, continuous, and mixed datasets. Their experiments showed that this algorithm surpasses both LearnSPN and ID-SPN in the three types of datasets. It is also robust in low data regimes, achieving the same performance as oSLRAU with only 30-40% of the data.