1. A computer-implemented method, comprising: receiving a first set of points based on detection of an environment of an autonomous vehicle by one or more sensors on the autonomous vehicle during a first time period; selecting a plurality of points from the first set of points that form a first point cloud representing an object in the environment; receiving a second set of points based on detection of the environment by the one or more sensors on the autonomous vehicle during a second time period, wherein the second time period is after the first period; selecting a plurality of points from the second set of points that form a second point cloud representing the object in the environment; determining, by a control system of the autonomous vehicle, a transformation between the selected points from the first set of points that form the first point cloud representing the object in the environment and the selected points from the second set of points that form the second point cloud representing the object in the environment; using the transformation to determine a velocity of the object; and controlling the autonomous vehicle based at least in part on the velocity of the object.