1. A computer implemented method of reducing a first optical distortion of a virtual reality or augmented reality head mounted display (VR or AR HMD) system that includes an image source and an optical element, the first optical distortion being introduced when light projected from the image source is reflected by a surface of the optical element to the eye of a viewer of the VR or AR HMD system; the method comprising: receiving, by a processing device, image data for display by the AR or VR HMD system; receiving, by the processing device, an inverse distortion map derived from the shape of the surface of the optical element; rendering, by the processing device, the image data to an image source display buffer according to coordinates indicated by the inverse distortion map; and displaying, by the image source, an image with a second optical distortion corresponding to the rendered image data, wherein the viewer of the VR or AR HMD system perceives a virtual image corresponding to the displayed image that is substantially free of the first optical distortion.