1. A surveillance system for tracking objects, comprising: one or more object-tracking modules that are configured to detect a plurality of events corresponding to appearances and disappearances of objects in images corresponding to one or more fields of view of one or more cameras, and a handoff controller, operably coupled to the one or more tracking modules, that is configured to associate a first event with a second event of the plurality of events, based on locations of the plurality of events in the one or more fields of view, wherein each location of each event of the plurality of events is associated with a defined zone within a field of view of a camera of the one or more cameras, the first event thereby being associated with a first zone, and the first zone has an associated set of linked zones, the set of linked zones being based on a feasibility of an object in the first zone traversing to or from each of the zones in the set of linked zones, the handoff controller is configured to select the second event from among events occurring in the set of linked zones associated with the first zone.