CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a 35 U.S.C. § 371 National Phase conversion of International (PCT) Patent Application No. PCT/CN2019/078329, filed on Mar. 15, 2019, which claims priority of Chinese Patent Application No. 201810217087.6, filed on Mar. 16, 2018, the disclosure of which is incorporated by reference herein. The PCT International Patent Application was filed and published in Chinese.

FIELD OF THE INVENTION

The present invention relates generally to the field of video processing, and particularly relates to an anti-shake method for a panoramic video and a portable terminal.

BACKGROUND OF THE INVENTION

CMOS and CCD sensors are two types of image sensors that are currently commonly used. Both use photodiodes for photoelectric conversion to convert images into digital data. The main difference is the way in which digital data is transmitted.

The charge data of each pixel in each line of the CCD sensor will be transferred to the next pixel in turn, and be outputted from the bottom thereof, then be amplified and outputted by the amplifier at the edge of the sensor; while in the CMOS sensor, each pixel will be adjacent to one amplifier and an A/D conversion circuit, and data is outputted in a way similar to a memory circuit.

Generally, CMOS cameras use rolling shutter mode to obtain fisheye images through progressive exposure.

CMOS chips share the workload through many parallel A/Ds, but the entire array in the sensor must be converted one line at a time, which results in a small time delay between the readings of each line. Each individual line is usually able to start the exposure of the next frame when the readout of the previous frame is completed. Although the speed is very fast, the time delay between the readings of each line will be converted into the delay between the start of the exposure of each line, exposure of each line will not happen at the same time. The result is that each line in the frame will be exposed for the same time length, but the exposure will start at a different time point. Two frames are allowed to overlap exposure, and the final frame rate depends on the speed at which the scrolling readout process can be completed.

Such exposure mode causes a time difference between different lines of the same frame of image. If a handheld panoramic camera is used to shoot, when moving at high speeds, a jelly effect will occur due to the progressive exposure of the CMOS camera. The jelly effect is a problem that has not been well solved, especially for a VR panoramic video, and the anti-shake effect of video is poor.

SUMMARY OF THE INVENTION

The problem solved by the present invention is to provide an anti-shake method for a panoramic video, a computer-readable storage medium and a portable terminal, which aims to solve the jelly effect caused by the shaking of the picture due to the rolling shutter mode of the CMOS chip, and the poor anti-shake effect of the camera.

According to a first aspect, the present invention provides an anti-shake method for a panoramic video, comprises steps of:

obtaining, in real time, an output video frame, a fisheye image corresponding to the output video frame, a timestamp of the pixel in the video frame, and a corresponding camera gyroscope timestamp;

synchronizing the timestamp of the pixel in the video frame with the corresponding camera gyroscope timestamp, and calculating a rotation matrix of the camera movement in the camera gyroscope timestamp;

smoothing the camera movement and establishing a coordinate system of a smooth trajectory;

correcting the fisheye image distortion; and

rendering the fisheye image by means of forward rendering to generate a stable video.

According to a second aspect, the present invention provides an anti-shake method for a panoramic video, comprises steps of:

S201, obtaining, in real time, an output video frame, a fisheye image corresponding to the output video frame, a timestamp of the pixel in the video frame, and a corresponding camera gyroscope timestamp;

S202, synchronizing the timestamp in the video frame to the camera gyroscope acquisition system, and calculating a rotation matrix of the camera movement in the timestamp of the pixel in the current video frame as the approximate rotation matrix of the camera when collecting the current pixel;

S203, projecting the output video frame pixels to the corresponding grid points of the spherical model, and using the approximate rotation matrix to rotate the corresponding grid points of the spherical model to the grid points in the sensor coordinate system; using the fisheye image distortion correction method to establish the relationship between the grid points in the sensor coordinate system and the pixels in the fisheye image to obtain approximate reverse-mapping pixels;

S204, calculating the camera gyroscope timestamp of the approximate reverse-mapping pixels in the fisheye image, obtaining the accurate rotation matrix of the camera when collecting the current pixel, and using the accurate rotation matrix to rotate the corresponding grid points of spherical model again to obtain the second grid points in the sensor coordinate system;

S205, using the fisheye image distortion correction method to establish the relationship between the second grid points in the sensor coordinate system and the pixels in the fisheye image to obtain accurate pixels in the fisheye image, and using the mapping relationship between the output video frame pixels and the accurate pixels in the fisheye image, rendering the fisheye image in reverse rendering to generate a stable video.

According to a third aspect, the present invention provides a computer-readable medium that stores a computer program or computer programs including set of computer-executable instructions, which when being executed by a processor or processors, cause the processor or processors to perform steps of the above-mentioned anti-shake method for a panoramic video.

According to a fourth aspect, the present invention provides a portable terminal, comprising:

one or more processors;

a memory; and

one or more computer programs where the one or more computer programs including set of computer-executable instructions are stored in the memory and are configured to be executed by the one or more processors, and when executed by the one or more processors, cause the one or more processors to perform the above-mentioned anti-shake method for a panoramic video.

In the present invention, based on the timestamp of the pixel in the video frame, the fisheye image is pixel by pixel converted into the coordinate system of the smooth trajectory using the rotation matrix of the camera motion under the timestamp of the current video frame pixel in real time, and then rendered and output; thereby the distortion of the rolling shutter of the panoramic video sequence can be corrected. The invention can correct the image distortion caused by the CMOS rolling shutter and eliminate the jelly effect, and achieve better video image anti-shake effect;

Further, in the present invention, according to the timestamp of the video frame pixel, calculating the rotation matrix of the camera movement under the timestamp of the current video frame pixel as the approximate rotation matrix of the camera when collecting the current pixel, projecting the output video frame pixels to the corresponding grid points of the spherical model, and using the approximate rotation matrix to rotate the corresponding grid points of the spherical model to the grid points in the sensor coordinate system; using the fisheye image distortion correction method to establish the relationship between the grid points in the sensor coordinate system and the pixels in the fisheye image to obtain approximate reverse mapping pixels, calculating the camera gyroscope timestamp of the approximate reverse-mapping pixels in the fisheye image, obtaining the accurate rotation matrix of the camera when collecting the current pixel, and using the accurate rotation matrix to rotate the corresponding grid points of spherical model again to obtain the second grid points in the sensor coordinate system; using the fisheye image distortion correction method to establish the relationship between the second grid points in the sensor coordinate system and the pixels in the fisheye image to obtain accurate pixels in the fisheye image, and using the mapping relationship between the output video frame pixels and the accurate pixels in the fisheye image, rendering the fisheye image in reverse rendering to generate a stable video. Thereby, the distortion of the rolling shutter of the panoramic video sequence can be corrected. The invention can correct the image distortion caused by the CMOS rolling shutter and eliminate the jelly effect, thereby achieve a better video image anti-shake effect.

