1. A system for tracking movement of an object's portion in a three-dimensional (3D) space, the system including: one or more processors coupled to memory loaded with computer instructions that, when executed on the processors, implement actions including: selecting five observation points on a sphere model fitted to a portion of a real world object, wherein one of the observation points is at a center of the sphere model and four perimeter observation points are along a perimeter of the sphere model; capturing at times t 0 and t 1 , projections of at least four perimeter observation points in at least a first image plane of at least one camera, wherein the object moved between t 0 and t 1 ; calculating a retraction of the four perimeter observation points at time t 1 to their positions at t 0 , including using the four perimeter observation points to: map the projections of at least four perimeter observation points captured at t 0 to a first image plane; and calculate orientation of a second image plane such that the projections of at least four perimeter observation points captured at t 1 to the second image plane are collinear with respective projections in the first image plane with respect to a retracted center observation point; and determining at least translation of the sphere model between the times t 0 and t 1 using the orientation of the second image plane calculated.