Existing sequence prediction methods are mostly concerned with time-independent sequences, in which the actual time span between events is irrelevant and the distance between events is simply the difference between their order positions in the sequence. While this time-independent view of sequences is applicable for data such as natural languages, e.g., dealing with words in a sentence, it is inappropriate and inefficient for many real world events that are observed and collected at unequally spaced points of time as they naturally arise, e.g., when a person goes to a grocery store or makes a phone call. The time span between events can carry important information about the sequence dependence of human behaviors. In this work, we propose a set of methods for using time in sequence prediction. Because neural sequence models such as RNN are more amenable for handling token-like input, we propose two methods for time-dependent event representation, based on the intuition on how time is tokenized in everyday life and previous work on embedding contextualization. We also introduce two methods for using next event duration as regularization for training a sequence prediction model. We discuss these methods based on recurrent neural nets. We evaluate these methods as well as baseline models on five datasets that resemble a variety of sequence prediction tasks. The experiments revealed that the proposed methods offer accuracy gain over baseline models in a range of settings. Event sequence prediction is a task to predict the next event 1 based on a sequence of previously occurred events. Event sequence prediction has a broad range of applications, e.g., next word prediction in language modeling BID10 , next place prediction based on the previously visited places, or next app to launch given the usage history. Depending on how the temporal information is modeled, event sequence prediction often decomposes into the following two categories: discrete-time event sequence prediction and continuous-time event sequence prediction.Discrete-time event sequence prediction primarily deals with sequences that consist of a series of tokens (events) where each token can be indexed by its order position in the sequence. Thus such a sequence evolves synchronously in natural unit-time steps. These sequences are either inherently time-independent, e.g, each word in a sentence, or resulted from sampling a sequential behavior at an equally-spaced point in time, e.g., busy or not busy for an hourly traffic update. In a discrete-time event sequence, the distance between events is measured as the difference of their order positions. As a consequence, for discrete-time event sequence modeling, the primary goal is to predict what event will happen next.Continuous-time event sequence prediction mainly attends to the sequences where the events occur asynchronously. For example, the time interval between consecutive clinical visits of a patient may potentially vary largely. The duration between consecutive log-in events into an online service can change from time to time. Therefore, one primary goal of continuous-time event sequence prediction is to predict when the next event will happen in the near future.Although these two tasks focus on different aspects of a future event, how to learn a proper representation for the temporal information in the past is crucial to both of them. More specifically, even though for a few discrete-time event sequence prediction tasks (e.g., neural machine translation), they do not involve an explicit temporal information for each event (token), a proper representation of the position in the sequence is still of great importance, not to mention the more general cases where each event is particularly associated with a timestamp. For example, the next destination people want to go to often depends on what other places they have gone to and how long they have stayed in each place in the past. When the next clinical visit BID3 will occur for a patient depends on the time of the most recent visits and the respective duration between them. Therefore, the temporal information of events and the interval between them are crucial to the event sequence prediction in general. However, how to effectively use and represent time in sequence prediction still largely remains under explored.A natural and straightforward solution is to bring time as an additional input into an existing sequence model (e.g., recurrent neural networks). However, it is notoriously challenging for recurrent neural networks to directly handle continuous input that has a wide value range, as what is shown in our experiments. Alternatively, we are inspired by the fact that humans are very good at characterizing time span as high-level concepts. For example, we would say "watching TV for a little while" instead of using the exact minutes and seconds to describe the duration. We also notice that these high-level descriptions about time are event dependent. For example, watching movies for 30 minutes might feel much shorter than waiting in the line for the same amount of time. Thus, it is desirable to learn and incorporate these time-dependent event representations in general. Our paper offers the following contributions:• We propose two methods for time-dependent event representation in a neural sequence prediction model: time masking of event embedding and event-time joint embedding. We use the time span associated with an event to better characterize the event by manipulating its embedding to give a recurrent model additional resolving power for sequence prediction.• We propose to use next event duration as a regularizer for training a recurrent sequence prediction model. Specifically , we define two flavors of duration-based regularization: one is based on the negative log likelihood of duration prediction error and the other measures the cross entropy loss of duration prediction in a projected categorical space.• We evaluated these proposed methods as well as several baseline methods on five datasets (four are public). These datasets span a diverse range of sequence behaviors, including mobile app usage, song listening pattern, and medical history. The baseline methods include vanilla RNN models and those found in the recent literature. These experiments offer valuable findings about how these methods improve prediction accuracy in a variety of settings. We proposed a set of methods for leveraging the temporal information for event sequence prediction. Based on our intuition about how humans tokenize time spans as well as previous work on contextual representation of words, we proposed two methods for time-dependent event representation. They transform a regular event embedding with learned time masking and form time-event joint embedding based on learned soft one-hot encoding. We also introduced two methods for using next duration as a way of regularization for training a sequence prediction model. Experiments on a diverse range of real data demonstrate consistent performance gain by blending time into the event representation before it is fed to a recurrent neural network.
Proposed methods for time-dependent event representation and regularization for sequence prediction; Evaluated these methods on five datasets that involve a range of sequence prediction tasks.
{"src": "scitldr-AIC_train", "paper_id": "SJDJNzWAZ"}