Analyzing human actions in videos has gained increased attention recently. While most works focus on classifying and labeling observed video frames or anticipating the very recent future, making long-term predictions over more than just a few seconds is a task with many practical applications that has not yet been addressed. In this paper, we propose two methods to predict a considerably large amount of future actions and their durations. Both, a CNN and an RNN are trained to learn future video labels based on previously seen content. We show that our methods generate accurate predictions of the future even for long videos with a huge amount of different actions and can even deal with noisy or erroneous input information.
Other works investigate future prediction in the sports domain. For example, @cite_13 use augmented hidden conditional random field to predict the location of future events in sports, like predicting shot location in tennis. Recently, a framework has been introduced in @cite_11 to predict the next move of players or the location of the ball in the immediate future from sports videos.
@cite_13 @cite_11
2402869773 2155853791 2949309567 1930032979
In highly dynamic and adversarial domains such as sports, short-term predictions are made by incorporating both local immediate as well global situational information. For forecasting complex events, higher-order models such as Hidden Conditional Random Field (HCRF) have been used to good effect as capture the long-term, high-level semantics of the signal. However, as the prediction is based solely on the hidden layer, fine-grained local information is not incorporated which reduces its predictive capability. In this paper, we propose an “augmented-Hidden Conditional Random Field” (a-HCRF) which incorporates the local observation within the HCRF which boosts it forecasting performance. Given an enormous amount of tracking data from vision-based systems, we show that our approach outperforms current state-of-the-art methods in forecasting short-term events in both soccer and tennis. Additionally, as the tracking data is long-term and continuous, we show our model can be adapted to recent data which improves performance. We propose a general framework for event detection and summary generation in broadcast sports video. Under this framework, important events in a class of sports are modeled by "plays", defined according to the semantics of the particular sport and the conventional broadcasting patterns. We propose both deterministic and probabilistic approaches for the detection of the plays. The detected plays are concatenated to generate a compact, time compressed summary of the original video. Such a summary is complete in the sense that it contains every meaningful action of the underlying game, and it also servers as a much better starting point for higher-level summarization and or analysis than the original video does. We provide experimental results on American football, baseball, and sumo wrestling. We introduce a Deep Stochastic IOC RNN Encoderdecoder framework, DESIRE, for the task of future predictions of multiple interacting agents in dynamic scenes. DESIRE effectively predicts future locations of objects in multiple scenes by 1) accounting for the multi-modal nature of the future prediction (i.e., given the same context, future may vary), 2) foreseeing the potential future outcomes and make a strategic prediction based on that, and 3) reasoning not only from the past motion history, but also from the scene context as well as the interactions among the agents. DESIRE achieves these in a single end-to-end trainable neural network model, while being computationally efficient. The model first obtains a diverse set of hypothetical future prediction samples employing a conditional variational autoencoder, which are ranked and refined by the following RNN scoring-regression module. Samples are scored by accounting for accumulated future rewards, which enables better long-term strategic decisions similar to IOC frameworks. An RNN scene context fusion module jointly captures past motion histories, the semantic scene context and interactions among multiple agents. A feedback mechanism iterates over the ranking and refinement to further boost the prediction accuracy. We evaluate our model on two publicly available datasets: KITTI and Stanford Drone Dataset. Our experiments show that the proposed model significantly improves the prediction accuracy compared to other baseline methods. This paper proposes the automatic annotation of sports video for content-based retrieval. Conventional methods using position information of objects such as locus, relative positions, their transitions, etc., as indices, have drawbacks that tracking errors of a certain object due to occlusions causes recognition failures, and that representation by position information essentially has a limited number of recognizable events in the retrieval. Our approach incorporates human behavior analysis and specific domain knowledge with conventional methods, to develop an integrated reasoning module for richer expressiveness of events and robust recognition. Based on the proposed method, we implemented a content-based retrieval system which can identify several actions on real tennis video. We select court and net lines, players' positions, ball positions, and players' actions, as indices. Court and net lines are extracted using a court model and Hough transforms. Players and ball positions are tracked by adaptive template matching and particular predictions against sudden changes of motion direction. Players' actions are analyzed by 2D appearance-based matching using the transition of players' silhouettes and a hidden Markov model. The results using two sets of tennis video is presented, demonstrating the performance and the validity of our approach.