Sum-product networks: A survey <s> C. NLP and sequence data analysis <s> Sum-product networks (SPNs) are a recently proposed type of probabilistic graphical models allowing complex variable interactions while still granting efficient inference. In this paper we demonstrate the suitability of SPNs for modeling log-spectra of speech signals using the application of artificial bandwidth extension, i.e. artificially replacing the high-frequency content which is lost in telephone signals. We use SPNs as observation models in hidden Markov models (HMMs), which model the temporal evolution of log short-time spectra. Missing frequency bins are replaced by the SPNs using most-probable-explanation inference, where the state-dependent reconstructions are weighted with the HMM state posterior. According to subjective listening and objective evaluation, our system consistently and significantly improves the state of the art. <s> BIB001 </s> Sum-product networks: A survey <s> C. NLP and sequence data analysis <s> Sum product networks (SPNs) are a new class of deep probabilistic models. They can contain multiple hidden layers while keeping their inference and training times tractable. An SPN consists of interleaving layers of sum nodes and product nodes. A sum node can be interpreted as a hidden variable, and a product node can be viewed as a feature capturing rich interactions among an SPN’s inputs. We show that the ability of SPN to use hidden layers to model complex dependencies among words, and its tractable inference and learning times, make it a suitable framework for a language model. Even though SPNs have been applied to a variety of vision problems [1, 2], we are the first to use it for language modeling. Our empirical comparisons with six previous language models indicate that our SPN has superior performance. <s> BIB002 </s> Sum-product networks: A survey <s> C. NLP and sequence data analysis <s> We consider higher-order linear-chain conditional random fields (HO-LC-CRFs) for sequence modelling, and use sum-product networks (SPNs) for representing higher-order input- and output-dependent factors. SPNs are a recently introduced class of deep models for which exact and efficient inference can be performed. By combining HO-LC-CRFs with SPNs, expressive models over both the output labels and the hidden variables are instantiated while still enabling efficient exact inference. Furthermore, the use of higher-order factors allows us to capture relations of multiple input segments and multiple output labels as often present in real-world data. These relations can not be modelled by the commonly used first-order models and higher-order models with local factors including only a single output label. We demonstrate the effectiveness of our proposed models for sequence labeling. In extensive experiments, we outperform other state-of-the-art methods in optical character recognition and achieve competitive results in phone classification. <s> BIB003
Peharz et al. BIB001 applied SPNs to modeling speech by retrieving the lost frequencies of telephone communications (artificial bandwidth extension). In this problem tractable and quick (real-time) inference is essential. They used a hidden Markov model (HMM) to represent the temporal evolution of the log-spectrum, clustered the data using the Linde-Buzo-Gray algorithm and trained an SPN for each cluster. The SPNs model each cluster and can be used to retrieve the lost frequencies by MPE inference. This model has achieved better results than state-of-the-art algorithms both objectively, with a measure of log-spectral distortion, and subjectively, through listening tests. In language modeling, Cheng et al. BIB002 used a discriminative SPN  , whose leaf nodes represent vectors with information about previous words. This SPN was able to compute the probability of the next word more accurately than classic methods for language modeling, such as feedforward neural networks and recurrent neural networks. Later, Melibari et al.  used dynamic SPNs (DSPNs) to analyze different sequence datasets. Unlike dynamic Bayesian networks, for which inference is generally exponential in the number of variables per time slice, inference in DSPNs has linear complexity. In a comparative study with five other methods, including HMMs and neural networks with long short-term memory (LSTM), DSPNs were superior in 4 of the 5 datasets examined. Ratajczak et al. BIB003 replaced the local factors of higherorder linear-chain conditional random fields (HO-LC-CRFs) and maximum entropy Markov models (MEMMs) with SPNs. These outperformed other state-of-the-art methods in optical character recognition and achieved competitive results in phoneme classification and handwriting recognition.