Temporal point processes are the dominant paradigm for modeling sequences of events happening at irregular intervals. The standard way of learning in such models is by estimating the conditional intensity function.   However, parameterizing the intensity function usually incurs several trade-offs. We show how to overcome the limitations of intensity-based approaches by directly modeling the conditional distribution of inter-event times.   We draw on the literature on normalizing flows to design models that are flexible and efficient. We additionally propose a simple mixture model that matches the flexibility of flow-based models, but also permits sampling and computing moments in closed form.   The proposed models achieve state-of-the-art performance in standard prediction tasks and are suitable for novel applications, such as learning sequence embeddings and imputing missing data. Visits to hospitals, purchases in e-commerce systems, financial transactions, posts in social media -various forms of human activity can be represented as discrete events happening at irregular intervals. The framework of temporal point processes is a natural choice for modeling such data. By combining temporal point process models with deep learning, we can design algorithms able to learn complex behavior from real-world data. Designing such models, however, usually involves trade-offs along the following dimensions: flexibility (can the model approximate any distribution?), efficiency (can the likelihood function be evaluated in closed form?), and ease of use (is sampling and computing summary statistics easy?). Existing methods (Du et al., 2016; Mei & Eisner, 2017; Omi et al., 2019) that are defined in terms of the conditional intensity function typically fall short in at least one of these categories. Instead of modeling the intensity function, we suggest treating the problem of learning in temporal point processes as an instance of conditional density estimation. By using tools from neural density estimation (Bishop, 1994; Rezende & Mohamed, 2015) , we can develop methods that have all of the above properties. To summarize, our contributions are the following: • We connect the fields of temporal point processes and neural density estimation. We show how normalizing flows can be used to define flexible and theoretically sound models for learning in temporal point processes. • We propose a simple mixture model that performs on par with the state-of-the-art methods. Thanks to its simplicity, the model permits closed-form sampling and moment computation. • We show through a wide range of experiments how the proposed models can be used for prediction, conditional generation, sequence embedding and training with missing data. as a sequence of strictly positive inter-event times τ i = t i − t i−1 ∈ R + . Representations in terms of t i and τ i are isomorphic -we will use them interchangeably throughout the paper. The traditional way of specifying the dependency of the next arrival time t on the history H t = {t j ∈ T : t j < t} is using the conditional intensity function λ * (t) := λ(t|H t ). Here, the * symbol reminds us of dependence on H t . Given the conditional intensity function, we can obtain the conditional probability density function (PDF) of the time τ i until the next event by integration (Rasmussen, 2011) as p * (τ i ) := p(τ i |H ti ) = λ * (t i−1 + τ i ) exp − τi 0 λ * (t i−1 + s)ds . Learning temporal point processes. Conditional intensity functions provide a convenient way to specify point processes with a simple predefined behavior, such as self-exciting (Hawkes, 1971 ) and self-correcting (Isham & Westcott, 1979) processes. Intensity parametrization is also commonly used when learning a model from the data: Given a parametric intensity function λ * θ (t) and a sequence of observations T , the parameters θ can be estimated by maximizing the log-likelihood: θ * = arg max θ i log p * θ (τ i ) = arg max θ i log λ * The main challenge of such intensity-based approaches lies in choosing a good parametric form for λ Universal approximation. The SOSFlow and DSFlow models can approximate any probability density on R arbitrarily well (Jaini et al., 2019, Theorem 3), (Krueger et al., 2018, Theorem 4) . It turns out, a mixture model has the same universal approximation (UA) property. Theorem 1 (DasGupta, 2008, Theorem 33.2) . Let p(x) be a continuous density on R. If q(x) is any density on R and is also continuous, then, given ε > 0 and a compact set S ⊂ R, there exist number of components K ∈ N, mixture coefficients w ∈ ∆ K−1 , locations µ ∈ R K , and scales This results shows that, in principle, the mixture distribution is as expressive as the flow-based models. Since we are modeling the conditional density, we additionally need to assume for all of the above models that the RNN can encode all the relevant information into the history embedding h i . This can be accomplished by invoking the universal approximation theorems for RNNs (Siegelmann & Sontag, 1992; Schäfer & Zimmermann, 2006) . Note that this result, like other UA theorems of this kind (Cybenko, 1989; Daniels & Velikova, 2010) , does not provide any practical guarantees on the obtained approximation quality, and doesn't say how to learn the model parameters. Still, UA intuitively seems like a desirable property of a distribution. This intuition is supported by experimental results. In Section 5.1, we show that models with the UA property consistently outperform the less flexible ones. Interestingly, Theorem 1 does not make any assumptions about the form of the base density q(x). This means we could as well use a mixture of distribution other than log-normal. However, other popular distributions on R + have drawbacks: log-logistic does not always have defined moments and gamma distribution doesn't permit straightforward sampling with reparametrization. Intensity function. For both flow-based and mixture models, the conditional cumulative distribution function (CDF) F * (τ ) and the PDF p * (τ ) are readily available. This means we can easily compute the respective intensity functions (see Appendix A). However, we should still ask whether we lose anything by modeling p * (τ ) instead of λ * (t). The main arguments in favor of modeling the intensity function in traditional models (e.g. self-exciting process) are that it's intuitive, easy to specify and reusable (Upadhyay & Rodriguez, 2019) . "Intensity function is intuitive, while the conditional density is not." -While it's true that in simple models (e.g. in self-exciting or self-correcting processes) the dependence of λ * (t) on the history is intuitive and interpretable, modern RNN-based intensity functions (as in Du et al. (2016) ; Mei & Eisner (2017); Omi et al. (2019) ) cannot be easily understood by humans. In this sense, our proposed models are as intuitive and interpretable as other existing intensity-based neural network models. "λ * (t) is easy to specify, since it only has to be positive. On the other hand, p * (τ ) must integrate to one." -As we saw, by using either normalizing flows or a mixture distribution, we automatically enforce that the PDF integrates to one, without sacrificing the flexibility of our model. "Reusability: If we merge two independent point processes with intensitites λ * 1 (t) and λ * 2 (t), the merged process has intensity λ * (t) = λ * 1 (t) + λ * 2 (t)." -An equivalent result exists for the CDFs F * 1 (τ ) and F * 2 (τ ) of the two independent processes. The CDF of the merged process is obtained as 2 (τ ) (derivation in Appendix A). As we just showed, modeling p * (τ ) instead of λ * (t) does not impose any limitation on our approach. Moreover, a mixture distribution is flexible, easy to sample from and has well-defined moments, which favorably compares it to other intensity-based deep learning models. We use tools from neural density estimation to design new models for learning in TPPs. We show that a simple mixture model is competitive with state-of-the-art normalizing flows methods, as well as convincingly outperforms other existing approaches. By looking at learning in TPPs from a different perspective, we were able to address the shortcomings of existing intensity-based approaches, such as insufficient flexibility, lack of closed-form likelihoods and inability to generate samples analytically. We hope this alternative viewpoint will inspire new developments in the field of TPPs. Constant intensity model as exponential distribution. The conditional intensity function of the constant intensity model (Upadhyay et al., 2018 ) is defined as λ H is the history embedding produced by an RNN, and b ∈ R is a learnable parameter. By setting c = exp(v T h i + b), it's easy to see that the PDF of the constant intensity model p * (τ ) = c exp(−c) corresponds to an exponential distribution. Summary The main idea of the approach by Omi et al. (2019) is to model the integrated conditional intensity function using a feedforward neural network with non-negative weights are non-negative weight matrices, and (3) ∈ R are the remaining model parameters. FullyNN as a normalizing flow Let z ∼ Exponential(1), that is We can view f : R + → R + as a transformation that maps τ to z We can now use the change of variables formula to obtain the conditional CDF and PDF of τ . Alternatively, we can obtain the conditional intensity as and use the fact that p Both approaches lead to the same conclusion However, the first approach also provides intuition on how to draw samplesτ from the resulting distribution p * (τ ) -an approach known as the inverse method (Rasmussen, 2011) 1. Samplez ∼ Exponential(1) 2. Obtainτ by solving f (τ ) −z = 0 for τ (using e.g. bisection method) Similarly to other flow-based models, sampling from the FullyNN model cannot be done exactly and requires a numerical approximation.
Learn in temporal point processes by modeling the conditional density, not the conditional intensity.
{"src": "scitldr-AIC_train", "paper_id": "HygOjhEYDH"}