1. An object tracking device for tracking one or more objects, the object tracking device comprising: a tracking candidate storage unit which stores a plurality of candidates for the objects to be tracked; and a control unit which controls a likelihood of each of the plurality of candidates, wherein the tracking candidate storage unit stores, for each of the plurality of candidates, label information identifying which object the candidate corresponds to, position information indicating a position of the object, and a likelihood, and wherein the control unit controls the likelihood of a candidate, according to the label information of one or more other candidates around the candidate determined based on the position information of the candidate.