1. A system for low latency stabilization of a head-worn display, the system comprising: a low latency pose tracker, including: at least one rolling-shutter camera that captures a 2D image by exposing each row of a frame at a later point in time than the previous row and outputs image data row by row; and a tracking module for receiving image data row by row from the at least one camera and generating a local appearance manifold, using the generated manifold to track camera movements, and producing a pose estimate.