Metrics on the space of sets of trajectories are important for scientists in the field of computer vision, machine learning, robotics, and general artificial intelligence. However, existing notions of closeness between sets of trajectories are either mathematically inconsistent or of limited practical use. In this paper, we outline the limitations in the current mathematically-consistent metrics, which are based on OSPA ( 2008); and the inconsistencies in the heuristic notions of closeness used in practice, whose main ideas are common to the CLEAR MOT measures (Keni and Rainer 2008) widely used in computer vision. In two steps, we then propose a new intuitive metric between sets of trajectories and address these limitations. First, we explain a solution that leads to a metric that is hard to compute. Then we modify this formulation to obtain a metric that is easy to compute while keeping the useful properties of the previous metric. Our notion of closeness is the first demonstrating the following three features: the metric 1) can be quickly computed, 2) incorporates confusion of trajectories' identity in an optimal way, and 3) is a metric in the mathematical sense.
The papers above are fairly recent and the search for similarity measures between sets of trajectories that are a metric is not older. However, researchers in the field of computer vision have been interested in defining similarly measures for sets of trajectories to evaluate the performance of tracking algorithms much prior to these works. It is impossible to review all work done in this area. Specially because the evaluation of the performance of trackers has many challenges other than the problem of defining a similarity measure. See @cite_35 @cite_10 for some examples of these other challenges. Nonetheless, we mention a few works and point out ideas in them that relate to our problem. We emphasize that none of the following works defines a metric mathematically.
@cite_35 @cite_10
1568589399 2131052232
Performance evaluation has become an increasingly important feature of video surveillance systems, as researchers attempt to assess the reliability and robustness of their operation. Although many algorithms and systems have been developed to address the problem of detecting and tracking moving objects in the image, few systems have been tested in anything other than fairly ideal conditions. In order to satisfy the requirements of a real video surveillance task, the algorithms will need to be assessed over a wide range of conditions. The aim of this paper is to examine some of the main requirements for effective performance analysis and to examine methods for characterising video datasets. Evaluating multi-target tracking based on ground truth data is a surprisingly challenging task. Erroneous or ambiguous ground truth annotations, numerous evaluation protocols, and the lack of standardized benchmarks make a direct quantitative comparison of different tracking approaches rather difficult. The goal of this paper is to raise awareness of common pitfalls related to objective ground truth evaluation. We investigate the influence of different annotations, evaluation software, and training procedures using several publicly available resources, and point out the limitations of current definitions of evaluation metrics. Finally, we argue that the development an extensive standardized benchmark for multi-target tracking is an essential step toward more objective comparison of tracking approaches.