1. A method for analyzing an object being tracked in a sequence of video frames, comprising: receiving a representation of the tracked object, as depicted by a current video frame, of the sequence of video frames; evaluating, by operation of one or more computer processors, the representation of the tracked object using at least a first classifier and a second classifier, wherein the first classifier is configured to determine a first classification score indicating whether the tracked object depicts an instance of a first classification type, and wherein the second classifier is configured to determine a second classification score indicating whether the tracked object depicts an instance of a second classification type; adding the first classification score to a first rolling average, wherein the first rolling average provides an average of the first classification score determined for the tracked object for each of a specified number of previous video frames, of the plurality of; adding the second classification score to a second rolling average, wherein the second rolling average provides an average of the second classification score determined for the tracked object for each of a specified number of previous video frames, of the plurality, wherein the final classification value is determined from the first rolling average and the second rolling average; determining a final classification value for the tracked object in the current video frame, based on the first and second rolling averages; and passing the final classification value for the tracked objects to a machine learning engine configured to identify patterns of behavior engaged in by the tracked object, based at least in part on the final classification value.