1. A method for detection and tracking of a moving object comprises the steps of: detecting at least two primitive classes for identifying the moving object within plural frames of video to generate plural primitive candidates for each class by detecting plural primitive candidates for each of the at least two primitive classes; extracting a primitive model of each of the at least two primitive classes; superimposing the primitive model on each primitive candidate; and determining which primitive candidate is a primitive; grouping each detected primitive candidates in each class separately according to a decision tree corresponding to each primitive class; training the decision tree corresponding to the primitive class; merging primitive groups for each class of primitives to identify the object being tracked; and tracking the moving object through the plural frames of video according to the merged primitive groups.