1. A method of manipulating a virtual object, the method including: detecting a hand in images of a three-dimensional (3D) sensory space captured by at least one sensor; generating from the images a predictive model of the hand, wherein the predictive model is used to track motion of the hand; assigning in the predictive model, calculation points assigned to portions of the predictive model corresponding to positions of fingers, thumb and palm of the hand; dynamically selecting at least one manipulation point describing a point of interaction of the hand with virtual objects based on the motion of the hand tracked by the predictive model by: weighting distances between positions of one or more of the calculation points and an anchor point defined for the predictive model and selected according to a type of interaction and/or a location of where an interaction is to occur; determining an average distance of the weighted distances between positions of the calculation points and the anchor point; and selecting the at least one manipulation point relative to the calculation points using the average distance; and manipulating the virtual object with the hand by interaction between at least some calculation points of the predictive model and the manipulation point.