CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No. PCT/CN2018/079033, filed on Mar. 14, 2018, which claims priority to Chinese Patent Application No. 201710179751.8, filed on Mar. 23, 2017. The disclosures of the aforementioned applications are hereby incorporated by reference in their entireties.

TECHNICAL FIELD

The present invention relates to the video processing field, and in particular, to a method and a client for playing back a panoramic video.

BACKGROUND

A panoramic video is also referred to as a 360-degree video, and means photographing an environment by using a plurality of camera lenses, compositing images corresponding to a plurality of angles into a panoramic video image by using technologies such as synchronization, splicing, and projection, and combining a plurality of frames of panoramic video images into a panoramic video. When viewing the panoramic video, a user may randomly change a viewing angle upwards, downwards, leftwards, or rightwards, to obtain better viewing experience.

The panoramic video includes information about a plurality of directions in a photographing environment, for example, the top, the bottom, the left, the right, the front, and the rear, and is limited to a field of view range (which is about 120 degrees) of human eyes and a field of view range of a display device (for example, currently, a field of view of a head-mounted device on the market is generally 96 degrees to 110 degrees). Therefore, when viewing the panoramic video, the user can actually view only content that corresponds to a small part of field of view and that is of the panoramic video. Usually, a region in which content actually viewed by the user currently is located in a panoramic video coordinate system is referred to as a user field of view. When viewing the panoramic video, the user can change the user field of view by dragging a mouse, turning the head, and the like, to view the panoramic video at different angles.

A client updates the user field of view in real time based on user interaction, and displays content in a corresponding region of the panoramic video. This brings the user with a higher degree of viewing freedom, but may also cause the user to miss some important/critical events. For example, when viewing a football match, the user is attracted to some episodes at the auditorium at a moment, and turns the head to the auditorium, but misses a goal on the court.

In the prior art, a server may identify, by using an image recognition technology (for example, object tracking), a region location at which a key event of the panoramic video occurs, generate a series of corresponding field of view information, and deliver the field of view information to the client. After receiving the field of view information (namely, a field of view recommended by the server) delivered by the server, the client automatically switches the user field of view to the recommended field of view, to display a corresponding key event.

Although the solution can prevent the user from missing a key event in a process of viewing the panoramic video, there are a series of disadvantages. First, the client automatically switches the user field of view of the user based on the recommended field of view delivered by the server. This causes sudden picture switching and affects user experience. In addition, because content corresponding to the recommended field of view is not necessarily content that the user is interested in, automatic switching interrupts previous viewing experience of the user and causes dislike to the user. For example, when the user views a football match, attention of the user is attracted to some episodes at the auditorium, and the user turns the head to the auditorium. At this moment, the user expects to know a user at the auditorium, and if the client forcibly switches to a goal picture, the user cannot view content that the user expects to view, and the user cannot freely select a field of view for viewing by the user. Consequently, user experience is reduced. In addition, this solution may cause confusion about a sense of space when the user views the panoramic video. For example, when the user wears a helmet to view the panoramic video, in a normal case, content on the left should be presented when the head of the user leans leftwards, so that the user has an immersive feeling. In this case, if a picture is suddenly switched to a recommended field of view that is on the right and that includes a key event, the head of the user leans leftwards, but the user actually views content on the right. This is contrary to the sense of space of the user.

SUMMARY

The present invention provides a method and a client for playing back a panoramic video, so that when playing back content corresponding to a user field of view, the client can intuitively indicate, to a user, occurrence of content corresponding to a recommended field of view, and the user can freely determine whether to switch a field of view.

To achieve the foregoing objective, the following technical solutions are used in embodiments of the present invention.

According to a first aspect, an embodiment of the present invention provides a method for playing back a panoramic video, including the following: A client obtains user field of view information and recommended field of view information of a panoramic video. The user field of view information is used to represent a user field of view of a user, that is, is used to represent a region in which content actually viewed by the user is located in a panoramic video coordinate system. When viewing the panoramic video, the user can change the user field of view by dragging a mouse, turning the head, and the like, to view the panoramic video at different angles. Usually, the client may obtain the view information of the user in real time based on user interaction. The recommended field of view information is used to represent a recommended field of view of the user, that is, is used to represent a region in which a key event is located in the panoramic video coordinate system. The recommended field of view is specified in advance by a panoramic video producer, or is identified by a server based on some technical means (for example, an image recognition technology such as object tracking). A manner of determining the recommended field of view is not limited in this embodiment of the present invention.

The client determines, based on the user field of view information and the recommended field of view information, that a field of view range corresponding to the recommended field of view is not within a field of view range corresponding to the user field of view. The client displays an image corresponding to the user field of view and an image corresponding to the recommended field of view.

In this embodiment of the present invention, in a panoramic video playback process, the client determines, based on whether the field of view range corresponding to the user field of view of the user includes the field of view range corresponding to the recommended field of view, whether the image corresponding to the recommended field of view also needs to be displayed while the image corresponding to the user field of view is displayed. In this way, the user is prevented from missing a key event that occurs within the field of view range corresponding to the recommended field of view, and a problem that the user has spatial-temporal confusion and flexibility of controlling the user field of view is reduced due to automatic field of view switching is also avoided.

The step of determining whether the field of view range corresponding to the recommended field of view is within the field of view range corresponding to the user field of view may be implemented in a plurality of manners based on different specific expression manners of the recommended field of view information and the user field of view information.

In a possible implementation, the user field of view information includes location information of the user field of view in a panoramic video image, and the recommended field of view information includes location information of the recommended field of view in the panoramic video image. For example, coordinate values of a field of view center may be used as the location information of each of the recommended field of view and the user field of view in the panoramic video image. Certainly, another location may be used, for example, an upper left corner or a lower right corner. The determining, by the client based on the user field of view information and the recommended field of view information, that a field of view range corresponding to the recommended field of view is not within a field of view range corresponding to the user field of view is specifically: determining a relative distance between the user field of view and the recommended field of view in the panoramic video image based on the location information of the user field of view in the panoramic video image and the location information of the recommended field of view in the panoramic video image; and when the relative distance is less than a preset threshold, determining that the field of view range corresponding to the user field of view includes the field of view range corresponding to the recommended field of view; or when the relative distance is not less than a preset threshold, determining that the field of view range corresponding to the user field of view does not include the field of view range corresponding to the recommended field of view.

Apparently, if the distance between the recommended field of view and the user field of view is larger, the two fields of view less likely overlap, and a key event that currently occurs less likely falls within the field of view range corresponding to the current user field of view. In this case, the image corresponding to the recommended field of view also needs to be displayed while the image corresponding to the user field of view is displayed, to prompt the user with occurrence of the key event.

In another possible implementation, the user field of view information includes location information of the user field of view in a panoramic video image and field of view magnitude information of the user field of view, and the recommended field of view information includes location information of the recommended field of view in the panoramic video image and field of view magnitude information of the recommended field of view. A field of view of a human is limited, and the field of view of the human is generally 90 degrees to 120 degrees. A coverage region (field of view range) of the field of view can be determined based on a location of the field of view and magnitude of the field of view. The determining, by the client based on the user field of view information and the recommended field of view information, that a field of view range corresponding to the recommended field of view is not within a field of view range corresponding to the user field of view is specifically: calculating display regions of the user field of view and the recommended field of view in a 2D plane based on the user field of view information and the recommended field of view information; determining a proportion of an overlapping area of the two display regions to the display region of the user field of view or the recommended field of view in the 2D plane, where preferably, a display region that corresponds to a field of view and that has a smaller area in the display regions of the user field of view and the recommended field of view in the 2D plane may be selected as an object for calculation; and when the proportion exceeds a preset threshold, determining that the field of view range corresponding to the user field of view includes the field of view range corresponding to the recommended field of view; or when the proportion does not exceed a preset threshold, determining that the field of view range corresponding to the user field of view does not include the field of view range corresponding to the recommended field of view.

Certainly, the user field of view and the recommended field of view each may be divided into a plurality of sub-regions in the 2D plane. Therefore, when the overlapping area is calculated, all the sub-regions need to be separately calculated.

If the overlapping part of the recommended field of view and the user field of view is smaller, a key event that currently occurs less likely falls within the field of view range corresponding to the current user field of view. In this case, the image corresponding to the recommended field of view also needs to be displayed while the image corresponding to the user field of view is displayed, to prompt the user with occurrence of the key event.

In another possible implementation, the user field of view information includes location information of the user field of view in a panoramic video image, and the recommended field of view information includes location information of the recommended field of view in the panoramic video image. In addition, the location information of each of the user field of view and the recommended field of view is represented by using angular coordinates. The determining, by the client based on the user field of view information and the recommended field of view information, that a field of view range corresponding to the recommended field of view is not within a field of view range corresponding to the user field of view is specifically: separately determining vectors from a sphere center to the user field of view and the recommended field of view; calculating an included angle between the two vectors; and when the included angle is less than a preset threshold, determining that the field of view range corresponding to the user field of view includes the field of view range corresponding to the recommended field of view; or when the included angle is not less than a preset threshold, determining that the field of view range corresponding to the user field of view does not include the field of view range corresponding to the recommended field of view.

In a process of viewing the panoramic video, in this embodiment of the present invention, a display location of the image corresponding to the recommended field of view is set, to provide the user with a most intuitive prompt, that is, to prompt the user with a specific key event that currently occurs in a specific direction at a current moment.

In a possible implementation, the client determines a relative location of the user field of view and the recommended field of view in the panoramic video image based on the user field of view information and the recommended field of view information; determines, based on the relative location, a display region of the image corresponding to the recommended field of view; and displays, in the display region, the image corresponding to the recommended field of view. The determining, based on the relative location, a display region of the image corresponding to the recommended field of view may be specifically: selecting an optimal display region from several predefined locations. For example, the recommended field of view can be displayed, by default, only at an upper left corner, an upper right corner, a lower left corner, and a lower right corner of the display region of the user field of view. If the recommended field of view is located on the left and the top of the user field of view, it is determined that the display region of the image corresponding to the recommended field of view is located at the upper left corner of the image corresponding to the user field of view. If the recommended field of view is located on the right and the bottom of the user field of view, it is determined that the display region of the image corresponding to the recommended field of view is located at the lower right corner of the image corresponding to the user field of view. Another case can be obtained by analogy. Certainly, a relative location may be calculated based on an actual included angle between the recommended field of view and the user field of view. For example, if the recommended field of view is located on the right of the user field of view, the display region of the image corresponding to the recommended field of view is located on the right of the image corresponding to the user field of view. Still further, a specific location that is of the display region of the image corresponding to the recommended field of view and that is on the right of the display region of the image corresponding to the user field of view is calculated based on an included angle between the recommended field of view and the user field of view in a vertical direction. In this embodiment of the present invention, the display region of the recommended field of view is determined based on the relative location of the user field of view and the recommended field of view in the panoramic video image, so that when the user expects to switch a field of view to the recommended field of view, the user can naturally switch to a direction of the display region based on a location of the display region of the recommended field of view. In particular, when the user wears a device such as a helmet, the relative location of the display region of the recommended field of view can be used to most intuitively prompt the user to switch the field of view.

In this embodiment of the present invention, that the client displays, in the display region, the image corresponding to the recommended field of view is also implemented in a plurality of manners based on whether the image corresponding to the recommended field of view and the image corresponding to the user field of view are displayed in a same playback window. In a possible implementation, the client superimposes, based on the display region, the image corresponding to the recommended field of view onto the image corresponding to the user field of view, and displays, in a current playback window, an image obtained after superimposition. In another possible implementation, the client displays, in a current playback window, the image corresponding to the user field of view; creates a recommended field of view playback window in the display region; and displays, in the recommended field of view playback window, the image corresponding to the recommended field of view, where the recommended field of view playback window is set above the user playback window.

In a process in which the user views the panoramic video, the user field of view may be constantly changing. In addition, one panoramic video may include a plurality of recommended fields of view, and different playback time correspond to different recommended fields of view. Therefore, a determining result of whether the field of view range corresponding to the recommended field of view is within the field of view range corresponding to the user field of view is also changing at any moment in the entire playback process.

The client may monitor a corresponding user interaction event, for example, the user drags a mouse or turns the head, to determine that the user field of view changes at a current moment. After the user field of view changes, the client obtains recommended field of view information corresponding to the current moment, and determines whether a field of view range corresponding to a recommended field of view at the current moment is within a field of view range corresponding to the changed user field of view. When the field of view range corresponding to the recommended field of view at the current moment is within the field of view range corresponding to the changed user field of view, the client does not display an image corresponding to the recommended field of view at the current moment.

Alternatively, the client may monitor a current playback time of the panoramic video, to determine that the recommended field of view changes at a current moment. After determining that the recommended field of view changes, the client obtains user field of view information corresponding to the current moment, and determines whether a field of view range corresponding to the changed recommended field of view is within a field of view range corresponding to a user field of view at the current moment. When the field of view range corresponding to the changed recommended field of view is within the field of view range corresponding to the user field of view at the current moment, the client does not display an image corresponding to the recommended field of view at the current moment.

In a possible implementation, the client further displays prompt information, to prompt the user to switch a field of view. The prompt information may be in a form of text or an image, for example, “double-click a left key on a remote control to directly switch to the recommended field of view”, or an arrow, where the arrow points to a field of view switching direction.

According to a second aspect, an embodiment of the present invention further provides a client for playing back a panoramic video, where the client has a function of implementing behavior of the client in the first aspect. The function may be implemented by hardware, or may be implemented by hardware by executing corresponding software. The hardware or the software includes one or more modules corresponding to the function.

In a possible design, a structure of the client includes a processor and a memory. The memory is configured to store a program that supports the client in performing the foregoing method. The processor is configured to execute the program stored in the memory. The client may further include a communications interface that is used by the client to communicate with another device or a communications network.

According to a third aspect, an embodiment of the present invention further provides a computer program product. When being executed, the computer product is configured to perform the method that is described above and that is performed by the client.

According to a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, where the computer readable storage medium stores an instruction, and the instruction is used to perform the method that is described above and that is performed by the client.

According to the method for playing back a panoramic video provided above, in the panoramic video playback process, when presentation of the user field of view of the user is not affected, the image corresponding to the recommended field of view can be prompted and presented, so that the user independently determines to switch a field of view. Therefore, a degree of freedom for the user to independently switch the field of view and continuous playback of the image corresponding to the user field of view are ensured, and the user does not miss important content, thereby improving user experience.

