This application is a U.S. National Phase Application under 35 U.S.C. § 371 of International Application No. PCT/AU2019/050316, filed Apr. 10, 2019 (Published on Oct. 17, 2019 as WO 2019/195884); which claims priority to Australia Application No. 2018901183, filed Apr. 10, 2018, all of which are incorporated by reference herein in their entirety.

FIELD OF THE INVENTION

The present invention relates to the calibration of images projected onto surfaces and typically surfaces that are non-planar. In particular, but not exclusively, the present invention relates to methods of calibration of images projected onto curved surfaces from multiple projectors to create seamless images and to related systems and apparatus.

BACKGROUND TO THE INVENTION

In the field of projectors and image projection, a rectangular image is typically projected from the light source onto a display surface, such as a screen or a wall. Often, display surfaces are complex non-planar surfaces or not uniform in shape. For example, the display surface can be curved for applications such as flight simulators, or the surface can be non-uniform and comprise irregular shapes for applications such as building projections.

Conventional systems for projecting images onto complex surfaces include a projector or projectors, laser levels, measurement devices, such as a tape measure and/or a laser measure, theodolites, USB cameras, digital cameras, SLR cameras and/or IP cameras and fiducial markers.

However, some problems exist with known systems and methods for projecting images onto display surfaces. For example, for known categories of image correction, such as geometric correction, image warping and image blending, multiple projectors are used to create a seamless image across flat, cylindrical, spherical and non-uniform display surfaces. This can be achieved manually, or with digital cameras. However, these techniques are typically performed once in a time period from daily to yearly and often require human intervention. A separate maintenance or alignment sequence is generally required.

For the display of images where movement within structures are common, such as vibration, or for applications such as flight simulators comprising motion platforms, movement of the projectors can occur, resulting in alignment issues while the display is being used. This results in images that become misaligned during the general operation of the display system.

The reference to any prior art in this specification is not, and should not be taken as, an acknowledgement or any form of suggestion that the prior art forms part of the common general knowledge.

OBJECT OF THE INVENTION

It is a preferred object of the present invention to provide a system and/or a method and/or an apparatus that addresses or at least ameliorates one or more of the aforementioned problems and/or provides a useful commercial alternative.

SUMMARY OF THE INVENTION

Generally, the present invention relates to systems, methods and apparatus that allow complex display systems comprising one or more projection devices to be calibrated whilst the display system is in normal operation, rather than using a calibration sequence as a separate process outside normal operation of the display system. The methods of the present invention can be performed during, or outside normal operation of the display systems as a single process, repeatedly or continually.

The systems, methods and apparatus of the present invention use registration devices, such as cameras to align and blend overlapping images from one or more projection devices to the shape of a planar or non-planar screen.

The systems, methods and apparatus determine projected points or reference markers on the screen and calculate a mapping from the displayed image to provide a geometrically corrected output image that can be used to produce a seamless display.

The projected points or reference markers can be pre-defined or can be detected ad hoc, including features detected in arbitrary images. The systems, methods and apparatus enable repeated or continuous alignment at runtime, i.e. during display of the images. The systems, methods and apparatus enable alignment image processing to be embedded in the one or more projection devices, to be placed in the path between an image generator and the one or more projections devices, or to be applied by the image generator itself.

In one form, although not necessarily the broadest form, the invention resides in a method to continuously calibrate images projected on a screen or surface or intermediary viewing surface by during display of the images, the method comprising:

defining a plurality of fiducial markers on the screen;

detecting, by a registration device, such as a camera, the plurality of fiducial markers in an image of the screen;

adding a plurality of reference markers to an image projected on the screen;

detecting, by the registration device, the plurality of reference markers in the projected image;

determining a mapping between positions of the reference markers on the screen to positions of the reference markers in the projected images;

calculating an image warp based on the mapping and a layout of the images to be displayed; and

applying the image warp to images from an image generator to produce geometrically corrected images for display on the screen.

The method may include calculating a first transformation between positions of the fiducial markers on the screen and respective positions of the fiducial markers in the images detected by the registration device.

The method may include detecting a plurality of feature points in the images detected by the registration device, wherein the feature points are known positions in the images projected by the one or more projection devices.

Suitably, the method includes calculating a second transformation between the plurality of reference markers and feature points in the images displayed on the screen and respective positions of the plurality of reference markers and feature points in the images detected by the registration device.

Suitably, the method includes combining the first transformation and the second transformation to determine the mapping for positions on the screen to positions in the displayed images.

Suitably, the method includes defining a description of the screen, such as a 3D model of the screen, a geometric shape; a mesh of points; an interpolated point cloud. The description of the screen can be user-defined or computer generated using methods such as photogrammetry, reconstruction from LiDAR data or other measurements, or a depth-based method.

In some embodiments, the method includes defining a layout of the images to be displayed on the screen including any overlaps or blend zones of the images.

The method may include projecting and positioning the fiducial markers on the screen. Alternatively, the fiducial markers are physical markers on the screen.

Suitably, the feature points are calculated in known positions in the one or more images. Alternatively, the features points are extracted from a displayed arbitrary image.

The method may include applying the image warp to images from an image generator during runtime using hardware or software image warping techniques.

Suitably, adding the plurality of reference markers to the projected image comprises embedding the reference markers in an operational image with a higher and/or lower intensity, luminance or chrominance than the remainder of an operational image.

Suitably, adding the plurality of reference markers to the projected image comprises encoding a registration image into a structured pattern, such as a structured noise pattern, and combining the structured pattern with an operational image to produce the projected image.

The method may include extracting the plurality of reference markers from the projected image using a background subtraction technique to subtract the operational image from accumulated frames of the projected image.

The method may include tuning the sensitivity of the registration device based on an average luminance or chrominance of the operational image.

Preferably, the plurality of reference markers is imperceptible to a human.

The method may include chrominance shifting pixels in the projected image in a defined pattern, in particular, chrominance shifting a centre pixel in one direction and chrominance shifting the adjacent pixels in the opposite direction.

The method may include detecting one or more features in the operational image using computer vision analysis before projection of the operational image and searching for the same one or more features in the projected image detected by the registration device.

Suitably, the method includes one or more of the following: a) amplifying the reference markers embedded in the background of the operational image while reducing the reference markers embedded in the foreground of the operational image; b) determining shadow in the operational image and amplifying the reference markers in shadow.

Suitably, detecting reference markers in an overlap or blend zone of images from multiple projection devices comprises detecting the reference markers in an image from a single projection device at a time.

The method may include suppressing the same reference markers in the overlap or blend zone from the other projection devices until the reference markers are detected for the first projection device.

The method may include applying a weighted contribution of the pixels at the location of the reference markers that is being measured toward the selected projection device.

The method may include placing additional reference markers near the overlap or blend zone and extrapolating into the blend overlap or blend zone.

The method may include using a unique pattern of registration markers for each projection device.

Suitably, the method comprises projecting alternating image states on the screen or surface, such as alternating white and black images.

The method may comprise measuring latency between projection of the images and when the change in the image states is registered by the registration device.

The method may comprise measuring the sensitivity of the registration device, such as a white/black balance of the registration device.

The method may comprise receiving each frame of the operational image in a frame buffer of the image generator during display of the images.

Suitably, the method comprises one or more of the following: a) applying an image warping pattern to the frame buffer; b) calculating and adding the reference markers to the frame buffer; c) displaying the frame buffer through the projection device.

The method may comprise determining if the reference markers are outside a predetermined or specified tolerance, and if so, smoothly interpolating the image warping pattern over a period of time to a new image warping pattern.

In another form, although not necessarily the broadest form, the invention resides in a system to continuously calibrate images displayed on a screen or surface or intermediary viewing surface by one or more projection devices, the system comprising:

one or more image generators;

one or more projection devices to project images generated by the one or more image generators onto a screen;

one or more registration devices, such as cameras, to capture images of the screen; and

one or more processors configured to:

define a plurality of fiducial markers on the screen;

detect, by the registration device, the plurality of fiducial markers in an image of the screen;

add a plurality of reference markers to an image projected on the screen;

detect the plurality of reference markers in the projected image;

determine a mapping between positions of the reference markers on the screen to positions of the reference markers in the projected images;

calculate an image warp based on the mapping and a layout of the images to be displayed; and

apply the image warp to images from one or more image generators to generate geometrically corrected images for display on the screen.

In a further form, although not necessarily the broadest form, the invention resides in an apparatus to continuously calibrate images displayed on a screen or surface or intermediary viewing surface by one or more projection devices during display of the images, the apparatus comprising a non-transitory computer-readable medium have stored thereon programming instructions executable by a processor which, when executed, perform the following operations:

defining a plurality of fiducial markers on the screen;

detecting, by a registration device, such as a camera, the plurality of fiducial markers in an image of the screen;

adding a plurality of reference markers to an image projected on the screen;

detecting, by the registration device, the plurality of reference markers in the projected image;

determining a mapping between positions of the reference markers on the screen to positions of the reference markers in the projected images;

calculating an image warp based on the mapping and a layout of the images to be displayed; and

applying the image warp to images of an image generator to produce geometrically corrected images for display on the screen.

Further forms and/or features of the present invention will become apparent from the following detailed description.

