TECHNICAL FIELD

This disclosure relates generally to image or video capture devices, including processing of image frames from multiple image sensors by an image signal processor, such as for spatial alignment transform of the image frames.

DESCRIPTION OF THE RELATED TECHNOLOGY

Many devices include multiple image sensors that may be used for capturing one or more image frames. For example, a smartphone or tablet includes multiple image sensors to be used in generating images or video for different imaging applications. One or more image signal processors may process the image frames from the multiple image sensors. The processed image frames may be used by the imaging applications, such as for generating preview images, user photographs, recording video, performing augmented reality operations, and so on.

SUMMARY

The systems, methods, and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein.

One innovative aspect of the subject matter described in this disclosure can be implemented in a device. The device may include a processor and a memory coupled to the processor and storing instructions that, when executed by the processor, cause the device to perform operations. The operations may include capturing a first image of a scene using a first sensor at a first zoom ratio, capturing a second image of the scene using a second sensor at a second zoom ratio, the first image having a different field-of-view (FOV) than the second image, determining a translation matrix based on one or more spatial misalignments between the first image and the second image, determining a confidence associated with the translation matrix, in response to the confidence being greater than a confidence threshold, determining a weighting factor based on the first zoom ratio, the second zoom ratio, and a current zoom ratio of the device, applying the weighting factor to the translation matrix, and warping the first image to the second image using the weighted translation matrix.

In some implementations, the first image is captured concurrently with the second image when the current zoom ratio is within an overlap zone spanning between the first zoom ratio and the second zoom ratio. In some instances, the weighting factor is 0% when the current zoom ratio is equal to the first zoom ratio and the weighting factor is 100% when the current zoom ratio is equal to the second zoom ratio. In some other instances, the weighting factor corresponds to an identity matrix when the confidence is not greater than the confidence threshold. In some implementations, the first image has a wider FOV than the second image. In some instances, the first sensor is a wide-angle image sensor and the second sensor is a tele image sensor.

In some implementations, execution of the instructions causes the device to perform operations further including incrementally increasing the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio, and incrementally decreasing the weighting factor as the current zoom ratio decreases toward the first zoom ratio within the overlap zone. In some other implementations, execution of the instructions causes the device to perform operations further including generating a preview image based on the warped first image and the second image, and displaying the preview image to a user of the device.

In some implementations, execution of the instructions causes the device to perform operations further including scaling the first image to match the FOV of the second image, and identifying the one or more spatial misalignments between the scaled first image and the second image. In some other implementations, execution of the instructions causes the device to perform operations further including refraining from determining an additional translation matrix when the confidence is greater than the confidence threshold.

Another innovative aspect of the subject matter described in this disclosure can be implemented as a method for spatial alignment transform. The method may be performed by a device, and may include capturing a first image of a scene using a first sensor of the device at a first zoom ratio, capturing a second image of the scene using a second sensor of the device at a second zoom ratio, the first image having a different field-of-view (FOV) than the second image, determining a translation matrix based on one or more spatial misalignments between the first image and the second image, determining a confidence associated with the translation matrix, in response to the confidence being greater than a confidence threshold, determining a weighting factor based on the first zoom ratio, the second zoom ratio, and a current zoom ratio of the device, applying the weighting factor to the translation matrix, and warping the first image to the second image using the weighted translation matrix.

In some implementations, the first image is captured concurrently with the second image when the current zoom ratio is within an overlap zone spanning between the first zoom ratio and the second zoom ratio. In some instances, the weighting factor is 0% when the current zoom ratio is equal to the first zoom ratio and the weighting factor is 100% when the current zoom ratio is equal to the second zoom ratio. In some other instances, the weighting factor corresponds to an identity matrix when the confidence is not greater than the confidence threshold. In some implementations, the first image has a wider FOV than the second image. In some instances, the first sensor is a wide-angle image sensor and the second sensor is a tele image sensor.

In some implementations, the method further includes incrementally increasing the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio, and incrementally decreasing the weighting factor as the current zoom ratio decreases toward the first zoom ratio within the overlap zone. In some other implementations, the method further includes generating a preview image based on the warped first image and the second image, and displaying the preview image to a user of the device.

In some implementations, the method further includes scaling the first image to match the FOV of the second image, and identifying the one or more spatial misalignments between the scaled first image and the second image. In some other implementations, the method further includes refraining from determining an additional translation matrix when the confidence is greater than the confidence threshold.

Another innovative aspect of the subject matter described in this disclosure can be implemented in a non-transitory computer-readable medium. The non-transitory computer-readable medium can store instructions that, when executed by a processor of a device, cause the device to perform operations. In some implementations, the operations include capturing a first image of a scene using a first sensor of the device at a first zoom ratio, capturing a second image of the scene using a second sensor of the device at a second zoom ratio, the first image having a different field-of-view (FOV) than the second image, determining a translation matrix based on one or more spatial misalignments between the first image and the second image, determining a confidence associated with the translation matrix, in response to the confidence being greater than a confidence threshold, determining a weighting factor based on the first zoom ratio, the second zoom ratio, and a current zoom ratio of the device, applying the weighting factor to the translation matrix, and warping the first image to the second image using the weighted translation matrix.

In some implementations, the first image is captured concurrently with the second image when the current zoom ratio is within an overlap zone spanning between the first zoom ratio and the second zoom ratio. In some instances, the weighting factor is 0% when the current zoom ratio is equal to the first zoom ratio and the weighting factor is 100% when the current zoom ratio is equal to the second zoom ratio. In some other instances, the weighting factor corresponds to an identity matrix when the confidence is not greater than the confidence threshold. In some implementations, the first image has a wider FOV than the second image. In some instances, the first sensor is a wide-angle image sensor and the second sensor is a tele image sensor.

In some implementations, the operations further include incrementally increasing the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio, and incrementally decreasing the weighting factor as the current zoom ratio decreases toward the first zoom ratio within the overlap zone. In some other implementations, the operations further include generating a preview image based on the warped first image and the second image, and displaying the preview image to a user of the device.

In some implementations, the operations further include scaling the first image to match the FOV of the second image, and identifying the one or more spatial misalignments between the scaled first image and the second image. In some other implementations, the operations further include refraining from determining an additional translation matrix when the confidence is greater than the confidence threshold.

Another innovative aspect of the subject matter described in this disclosure can be implemented in a device. The device may include means for capturing a first image of a scene using a first sensor at a first zoom ratio, means for capturing a second image of the scene using a second sensor at a second zoom ratio, the first image having a different field-of-view (FOV) than the second image, means for determining a translation matrix based on one or more spatial misalignments between the first image and the second image, means for determining a confidence associated with the translation matrix, means for, in response to the confidence being greater than a confidence threshold, determining a weighting factor based on the first zoom ratio, the second zoom ratio, and a current zoom ratio of the device, means for applying the weighting factor to the translation matrix, and means for warping the first image to the second image using the weighted translation matrix.

In some implementations, the first image is captured concurrently with the second image when the current zoom ratio is within an overlap zone spanning between the first zoom ratio and the second zoom ratio. In some instances, the weighting factor is 0% when the current zoom ratio is equal to the first zoom ratio and the weighting factor is 100% when the current zoom ratio is equal to the second zoom ratio. In some other instances, the weighting factor corresponds to an identity matrix when the confidence is not greater than the confidence threshold. In some implementations, the first image has a wider FOV than the second image. The first and second sensor may be a wide-angle image sensor and a tele image sensor, respectively.

In some implementations, the device may further include means for incrementally increasing the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio, and incrementally decreasing the weighting factor as the current zoom ratio decreases toward the first zoom ratio within the overlap zone. In some other implementations, the device may further include means for generating a preview image based on the warped first image and the second image, and displaying the preview image to a user of the device.

In some implementations, the device may further include means for scaling the first image to match the FOV of the second image, and identifying the one or more spatial misalignments between the scaled first image and the second image. In some other implementations, the device may further include means for refraining from determining an additional translation matrix when the confidence is greater than the confidence threshold.

Details of one or more implementations of the subject matter described in this disclosure are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.

