RELATED APPLICATION DATA

This is the first patent application related to this matter.

FIELD

The present application generally relates to graphical user interfaces in extended reality environments, and in particular to devices, methods, systems, and media for selecting virtual objects for user interaction in an extended reality environment.

BACKGROUND

Augmented Realty (AR) and Virtual Reality (VR) have become increasingly popular in recent years. Although AR and VR technologies are unique in their own ways, AR may be considered an overlay of a virtual VR environment over a user's physical environment and thus several techniques and methods are applicable to both fields. One example is techniques for managing user interaction with graphical user interface (GUI) elements and other virtual objects that are spatially located in the virtual 3D space surrounding the user. Such interaction techniques are applicable to both AR and VR, and AR and VR may be referred to collectively as Extended Reality (XR).

In XR, virtual objects are spatially located in a virtual three-dimensional (3D) space surrounding the user as represented to the user through an XR display, unlike laptop or phone displays wherein the virtual objects are confined to a two-dimensional (2D) space close to the user. In some cases, XR GUI elements are deliberately located at a distance. This may be due to semantic anchoring information, for example an AR environment may include virtual signage affixed to a distant building in the physical environment or a wall-anchored planar AR display.

Interacting with virtual objects, such as GUI elements, at a distant location in the virtual 3D space is a fundamental user interface task in XR. Consequently, a significant amount of research has investigated this topic, ranging from novel pointing-and-selection techniques (i.e. techniques for selecting GUI elements or other virtual objects within the virtual 3D space in order to manipulate or otherwise interact with the selected virtual objects) to six degrees of freedom (DOF) manipulation (i.e. manipulation of virtual objects in three linear dimensions—height, width, depth—and three rotational dimensions—yaw, pitch, roll). Existing techniques for pointing and selection generally fall into two approaches. The first approach is ray casting based methods, which project a ray in a direction determined by user behavior, such as movement of the user's hand or user input to a VR controller, with virtual objects intercepting the ray being selected for interaction. However, as the distance between the user and the GUI elements or other virtual objects increases, the angular error of the ray casting direction increases, making it difficult to select individual virtual objects. Furthermore, because the selected virtual object is located at the tip of the ray, it is often difficult for users to rotate the object using intuitive controls. In addition, selecting a virtual object using ray casting tends to diminish the perception of tangibility of the selected virtual object by the user. The second approach to pointing and selection is arm extension based methods, which non-linearly map the user's physical hand to a distant virtual hand for interaction with distant virtual objects within the virtual 3D space. In arm extension methods, the user must adjust to a varying control-display (CD) gain function between the real hand and the virtual hand in different situations: i.e., movements of the user's physical arm and hand may be mapped to movements of the virtual hand with varying degrees of multiplication or magnification. Although arm extension methods may exhibit certain advantages over ray casting based methods, arm extension methods may be more socially awkward for users to perform, because they typically require a user to extend his or her arms outward in gross motor movements to manipulate distant virtual objects.

Overall, existing approaches to user interaction with distant virtual objects exhibit limitations with respect to the accuracy of interaction, the perceived tangibility of the virtual object, and the perception of depth in the layout of GUI elements or other virtual objects. Existing techniques to interact with distant virtual objects, such as ray casting and arm extension, suffer from angular error and CD gain function discrepancies respectively, leading to difficulties in selecting small virtual objects for interaction. Other existing approaches include methods for scaling the entire virtual environment, enlarging individual virtual objects, or zooming in on locations in the virtual 3D space (i.e. narrowing the user's field of view to shorten the perceived distance of distant locations in the virtual 3D space). However, these approaches typically result in decreasing the granularity of each virtual object, introducing CD gain discrepancy, and generally increasing the difficulty of interaction.

There thus exists a need for techniques enabling a user of an XR display to easily and intuitively interact with distant GUI elements or other virtual objects that overcome one or more of the limitations of existing approaches described above.

SUMMARY

The present disclosure describes devices, methods, systems, and media for selecting virtual objects for user interaction in an extended reality environment. Examples described herein bring selected distant virtual objects closer to the user within the virtual 3D space to situate the selected virtual object in virtual proximity to the user's hand for direct manipulation. A virtual object is selected by the user based on movements of the user's hand and/or head that are correlated or associated with an intent to select a specific virtual object within the virtual 3D space. As the user's hand moves in a way that is consistent with this intent, the virtual object is brought closer to the user's hand within the virtual 3D space.

In some embodiments, selection is performed by predicting an intended virtual object for selection based on the linear and angular velocities of head-worn and hand-worn sensors. Hand and head trajectory data is generated based on the velocity information received from the sensors, and the trajectory data is compared to a library of kinematic trajectory templates to identify a best-matched trajectory template. The best-matched template indicates a destination position of the hand, which can be used to determine an intent to select a virtual object based on ray casting.

The selected virtual object is then moved toward the user in the virtual 3D space. In some embodiments, the XR display renders the virtual object at a location closer to the user's hand, or toward a predicted future location of the user's hand, at a subsequent time step based on the predicted destination position of the hand. Once the virtual object is moved into virtual proximity to the user's hand, it can be directly manipulated by the user's hand without using the existing techniques and approaches described above for enabling user interaction with distant virtual objects. Examples described herein may therefore exhibit one or more advantages over the existing approaches described above, such as enabling users to interact with distant virtual objects without the need to adapt to changing CD gain, and with the tactility and high-resolution visual presentation of interacting with virtual objects in close proximity to the user's hand(s).

By bringing distant virtual objects into proximity with the user's hand, some embodiments may provide the user scope for direct manipulation. Such interaction in proximity to the hand may provide greater tangibility, higher precision, and greater granularity than other approaches, as it provides direct mapping to details of the selected virtual object, e.g. components of a selected collection of GUI elements. In addition, when the virtual object is moved towards the user, the range of motion of the user's hand is increased relative to approaches in which distant virtual objects are manipulated. The chance of accuracy problems in distal selection is therefore lowered, as is the risk of physical fatigue. Finally, hand-proximate interaction is more socially acceptable than distal ray casting interaction or arm-extension based interaction, because the user may maintain the hand(s) relatively close to the body rather than extending his or her arms outward to manipulate distant virtual objects.

In the present disclosure, the term “XR” refers to “extended reality”, an umbrella term referring to immersive technologies such as augmented reality (AR), virtual reality (VR), and mixed reality (MR). As used herein, XR refers to the presentation of a virtual 3D visual environment to a user through an XR display. An XR display is typically a stereoscopic display capable of simulating the effects of binocular vision in a 3D environment, but in some examples may be a monoscopic display such as a conventional 2D display screen capable of rendering 3D environments by projecting them onto two dimensions.

As used herein, the terms “virtual space”, “virtual 3D space”, “virtual environment”, and “virtual 3D environment” may be used interchangeably to mean a three-dimensional virtual space, containing virtual objects, generated by a computing device and visually presented to a user via an XR display. The virtual objects within the virtual environment are typically visually presented to the user so as to simulate viewing the object as though it were an actual physical object located at an actual physical distance from the user's point of view, with an orientation and relationship to other virtual and/or real objects that simulates the presence of the virtual object within a physical environment. In the case of VR, the user may be presented with a field of view that consists entirely of virtual objects within the virtual environment; in AR or MR, the user may be presented with a combination of the virtual objects of the virtual environment and real objects of the user's physical environment, with the virtual environment being “overlaid” over the physical environment such that the virtual objects appear to be present within the physical environment.

As used herein, the term “virtual location” refers to the representation of a virtual object within a virtual environment with respect to its apparent location relative to the user's point of view and to other virtual or physical objects. The “physical location” or “actual location” of a physical object in the user's environment refers to the actual physical location of the object relative to the user and to other physical objects in the user's physical environment. In AR or MR, virtual locations are typically defined in relation to physical locations; in VR, a physical object may be represented within the virtual environment by mapping the physical location of the physical object to a counterpart or avatar of the physical object, i.e. a virtual object representing the physical object, having a virtual location within the virtual environment that is mapped to the physical location of the object in the physical environment. The term “location” may refer to a virtual location in the context of virtual objects, and may refer to a physical location in the context of physical objects. In the case of physical objects represented in a virtual environment by an avatar (i.e. a virtual object), the “location” of the object may refer to either the physical location of the physical object or the virtual location of the corresponding avatar.

As used herein, “virtual object” may refer to any representation by a computing system of a phenomenon situated in a virtual environment at a virtual location such that a visually perceptible or otherwise perceptible stimulus is presented to a user as though the stimulus originated at the virtual location. Typically, virtual objects are visually presented to a user as 2D or 3D objects located within the 3D virtual environment. Virtual objects may include representations of real-world 3D objects (such as a 3D rendering of an airplane), 2D displays (such as a 2D rectangular screen showing a film being played), 2D sprites (such as a 2D button that can be pressed), or 3D sprites (such as a 3D slider for controlling audio volume). Virtual objects that correspond to the functionality of graphical user interface elements of a conventional computing system display—e.g., control widgets for receiving user input, display surfaces for displaying visual output such as text and images, and so on, including 3D versions thereof—may be referred to herein as “GUI elements”.

In the present disclosure, the term “position” refers to a location and orientation of a physical object or virtual object, such as a user's hand or head. “Location” may refer to the position of an object with respect to 3 linear dimensions (x, y, z), and “orientation” may refer to the position of an object with respect to 3 angular dimensions (roll, pitch, and yaw). “Distance” may refer to a linear distance (e.g. a 3-dimensional linear vector with x, y, and x elements), an angular distance (e.g. a 3-dimensional angular vector with roll, pitch, and yaw elements), or a combined linear and angular distance (e.g. a 6-dimensional vector). “Distance” may refer to a scalar value indicating the length of a vector, or to a vector value including directional information.

In some aspects, the present disclosure describes a method for presenting a virtual object of an extended reality (XR) environment to a user. Initial display information configured to control an XR display to present the virtual object to the user at an initial virtual location is generated. Hand movement information indicating a velocity of the hand is obtained from one or more hand movement sensors. The hand movement information is processed to generate trajectory data. The trajectory data is processed to determine that the user intends to select the virtual object. Updated display information configured to control the XR display to present the virtual object to the user at an updated virtual location closer to the hand than the initial virtual location is generated.

By moving a selected virtual object closer to the user's hand, the method may enable the user to interact with distant virtual objects without the need to adapt to changing CD gain. By bringing the virtual object into proximity with the user's hand, the user's interaction with the virtual object may exhibit the advantages of high tactility and high-resolution visual presentation.

In some aspects, the present disclosure describes a system for presenting a virtual object of an extended reality (XR) environment to a user. The system comprises a processor device, and a memory storing machine-executable instructions thereon. The machine-executable instructions, when executed by the processing device, cause the system to perform various steps. Initial display information configured to control an XR display to present the virtual object to the user at an initial virtual location is generated. Hand movement information indicating a velocity of the hand is obtained from one or more hand movement sensors. The hand movement information is processed to generate trajectory data. The trajectory data is processed to determine that the user intends to select the virtual object. Updated display information configured to control the XR display to present the virtual object to the user at an updated virtual location closer to the hand than the initial virtual location is generated.

In some examples, the method further comprises obtaining, from one or more head movement sensors, head movement information indicating a velocity of the head of the user, and wherein processing the hand movement information to generate the trajectory data comprises processing the hand movement information and the head movement information to generate the trajectory data.

In some examples, generating the trajectory data comprises retrieving previous hand movement information from a memory, storing the hand movement information obtained from the hand movement sensors in the memory, retrieving previous head movement information from a memory, storing the head movement information obtained from the head movement sensors in the memory, and processing the previous hand movement information, the hand movement information obtained from the hand movement sensors, the previous head movement information, and the head movement information obtained from the head movement sensors to generate the trajectory data.

In some examples, the hand movement information indicates a linear velocity and an angular velocity of the hand, and the head movement information indicates a linear velocity and an angular velocity of the head.

In some examples, the one or more hand movement sensors comprise an inertial measurement unit coupled to the hand, and the one or more head movement sensors comprise an inertial measurement unit coupled to the head.

In some examples, the inertial measurement unit coupled to the hand is an inertial measurement unit of a smart watch worn on the wrist of the hand, and the inertial measurement unit coupled to the head is an inertial measurement unit of a head-mounted display device worn on the head, the head-mounted display device comprising the XR display.

In some examples, processing the trajectory data to determine that the user intends to select the virtual object comprises comparing the trajectory data to a plurality of trajectory templates to identify a best-matched trajectory template of the plurality of trajectory templates corresponding to an intent to select the virtual object.

By using trajectory template profiles to predict user movements, some embodiments may customize the training of the prediction behavior of the system to a specific user, thereby potentially increasing accuracy of prediction and therefore consistency and correctness of behavior. Another potential benefit is that generating the trajectory templates may be easy to implement and fast relative to other training techniques for motion prediction systems. Furthermore, by using template matching to predict movement, a cumulative scoring function may be used to match candidate movements with a library of templates, resulting in fast, computationally inexpensive prediction that is easy to implement.

In some examples, the method further comprises processing the trajectory data to predict a future location of the hand. The updated virtual location is closer to the future location of the hand than the initial virtual location is.

In some examples, the initial display information is configured to control the XR display to present a plurality of virtual objects to the user on the XR display. Each virtual object is presented at a respective initial virtual location. The trajectory data is processed to determine that the user intends to select any one of two or more virtual objects of the plurality of virtual objects. The updated display information is configured to control the XR display to present the two or more virtual objects to the user on the XR display at a respective two or more updated virtual locations. Each respective updated virtual location is closer to the hand than each respective initial virtual location.

In some examples, the machine-executable instructions, when executed by the processing device, further cause the system to obtain, from one or more head movement sensors, head movement information indicating a velocity of the head of the user. Processing the hand movement information to generate the trajectory data comprises processing the hand movement information and the head movement information to generate the trajectory data.

In some examples, the hand movement information indicates a linear velocity and an angular velocity of the hand, and the head movement information indicates a linear velocity and an angular velocity of the head.

In some examples, the one or more hand movement sensors comprise an inertial measurement unit coupled to the hand, and the one or more head movement sensors comprise an inertial measurement unit coupled to the head.

In some examples, the system further comprises a head-mounted display device configured to be worn on the head, comprising the XR display and the inertial measurement unit coupled to the head.

In some examples, the system further comprises a smart watch configured to be worn on the wrist of the hand, comprising the inertial measurement unit coupled to the hand.

In some examples, the system further comprises the memory further stores a plurality of trajectory templates. Processing the trajectory data to determine that the user intends to select the virtual object comprises retrieving from the memory the plurality of trajectory templates, and comparing the trajectory data to the plurality of trajectory templates to identify a best-matched trajectory template of the plurality of trajectory templates corresponding to an intent to select the virtual object.

In some examples, the machine-executable instructions, when executed by the processing device, further cause the system to process the trajectory data to predict a future location of the hand. The updated virtual location is closer to the future location of the hand than the initial virtual location is.

In some aspects, the present disclosure describes a non-transitory processor-readable medium having machine-executable instructions stored thereon. The instructions, when executed by a processor device of a device, cause the device to generate initial display information configured to control an extended reality (XR) display to present a virtual object of an XR environment to a user at an initial virtual location; obtain, from one or more hand movement sensors, hand movement information indicating a velocity of the hand; process the hand movement information to generate trajectory data; process the trajectory data to determine that the user intends to select the virtual object; and generate updated display information configured to control the XR display to present the virtual object to the user at an updated virtual location closer to the hand than the initial virtual location.

In some aspects, the present disclosure describes a processor-readable medium having instructions tangibly stored thereon. The instructions, when executed by a processor device, cause the processor device to perform the method steps described above.

