Metrics on the space of sets of trajectories are important for scientists in the field of computer vision, machine learning, robotics, and general artificial intelligence. However, existing notions of closeness between sets of trajectories are either mathematically inconsistent or of limited practical use. In this paper, we outline the limitations in the current mathematically-consistent metrics, which are based on OSPA ( 2008); and the inconsistencies in the heuristic notions of closeness used in practice, whose main ideas are common to the CLEAR MOT measures (Keni and Rainer 2008) widely used in computer vision. In two steps, we then propose a new intuitive metric between sets of trajectories and address these limitations. First, we explain a solution that leads to a metric that is hard to compute. Then we modify this formulation to obtain a metric that is easy to compute while keeping the useful properties of the previous metric. Our notion of closeness is the first demonstrating the following three features: the metric 1) can be quickly computed, 2) incorporates confusion of trajectories' identity in an optimal way, and 3) is a metric in the mathematical sense.
This paper presents a new formalized procedure to assess the performance of automatic target tracking systems. The procedure involves comparing data output by a tracking system under test with track data in a truth file. The comparison of the data sources uses a total of 15 tracker performance metrics. These cover the categories of track initiation, track maintenance, track error and false tracks. The performance for each metric is expressed as a probability. An overall performance measure comes from using a weighted combination of these probabilities. The weighting relates to the metric’s importance. This approach is embodied in a Tracker Assessment Tool (TAT) and the processing steps in the TAT are reviewed with an example. A method to simplify the creation of truth data from recorded data is also described. This paper deals with the non-trivial problem of performance evaluation of motion tracking. We propose a rich set of metrics to assess different aspects of performance of motion tracking. We use six different video sequences that represent a variety of challenges to illustrate the practical value of the proposed metrics by evaluating and comparing two motion tracking algorithms. The contribution of our framework is that allows the identification of specific weaknesses of motion trackers, such as the performance of specific modules or failures under specific conditions. Scoring methods are described for evaluating the performance of a multiple target tracking (MTT) algorithm fairly, without undue bias towards any particular type. The methods were initially developed by individuals and further developed and adapted by the members of the SDI panels on tracking. Ambiguous track-to-target truth association is the fundamental difficulty in MTT performance evaluation. The methods use a global nearest neighbor assignment algorithm to uniquely associate tracks to targets. With the track- to-target associations, the methods employ measures of effectiveness for track purity, data association, state estimation accuracy, and credibility of filter calculated covariance.© (1991) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only. This paper presents a set of metrics and algorithms for performance evaluation of object tracking systems. Our emphasis is on wide-ranging, robust metrics which can be used for evaluation purposes without inducing any bias towards the evaluation results. The goal is to report a set of unbiased metrics and to leave the final evaluation of the evaluation process to the research community analyzing the results, keeping the human in the loop. We propose metrics from statistical detection and estimation theory tailored to object detection and tracking tasks using frame-based as well as object-based evaluation paradigms. Object correspondences between multiple ground truth objects to multiple tracker result objects are established from a correspondence matrix. The correspondence matrix is built using three different methods of distance computation between trajectories. Results on PETS 2001 data set are presented in terms of 1st and 2nd order statistical descriptors of these metrics.