1. A method comprising: moving a mobile device with a camera through space in a locally convex or locally concave motion; obtaining, from the camera throughout the movement of the mobile device, a plurality of frames having location information obtained from Inertial Measuring Unit (IMU) information and depth information, wherein the plurality of frames include a first frame and a second frame; moving a set of control points perpendicular to a trajectory between the first frame and the second frame, wherein the first frame includes a first image captured using the camera from a first location and the second frame includes a second image captured using the camera from a second location, wherein each control point is moved based on an associated depth of the control point, the depth of each of the control points being determined from the amount of frame-to-frame motion of the control points between the first frame and the second frame, wherein control points located at a further depth are moved less than control points located at a closer depth, wherein the set of control points is associated with a single layer among multiple layers; generating an artificially rendered image corresponding to a third location outside of the trajectory by extrapolating individual control points using the set of control points for the third location and extrapolating pixel locations using the individual control points, wherein generating the artificially rendered image occurs on the fly based on content-weighted keypoint tracks between the first image and the second image, the IMU information, and the depth information; and generating a surround view by fusing the plurality of frames along with the artificially rendered image, wherein the surround view is a three-dimensional multi-view interactive digital media representation (MIDMR) created from two-dimensional images in the plurality of frames and the artificially rendered image.