1. A video tracking system, comprising: a processor designed to execute layered processing comprising: a master peripheral tracker operative to interact with frames of image data, embodying a video scene, said master peripheral tracker including logic to monitor said scene and to detect an object within said frames of image data utilizing a windowed sequence of consecutive frame differences to track multiple objects using condensed object information and preserving spatial relationships of said multiple objects in said scene to coordinate task layers to understand object activities within overall dynamics of said scene utilizing several tunnel trackers; and a first tunnel tracker initiated by said master peripheral tracker, said first tunnel tracker including logic dedicated to track one said detected object, located within a first buffer area, utilizing a first portion of said image data frame-by-frame by building correspondences for said tracked object based on its attributes and detailed object information including a set of object features, said first buffer area being formed by said master peripheral tracker based on said first tracked object's current trajectory, size and location.