TECHNICAL FIELD

The present disclosure relates generally to virtual reality systems and in particular to providing user input devices and other peripherals in a virtual reality environment.

BACKGROUND

Virtual, mixed, or augmented reality can be associated with a variety of applications that comprise immersive, highly visual, computer-simulated environments. These environments can simulate a physical presence of a user in a real or imagined world. The computer simulation of these environments can include computer rendered images, which can be presented by means of a graphical display. The display can be arranged as a head mounted display (HMD) that may encompass all or part of a user's field of view.

A user can interface with the computer-simulated environment by means of a user interface device or peripheral device, examples of which include a keyboard, game controller, joystick, mouse, stylus, speaker, and steering wheel. When immersed in the computer-simulated environment, the user may perform complex operations associated with the interface device, including actuation of one or more keys or other input elements of the interface device. However, there is need for improvement when operating within virtualized environments using physical interface devices, especially when performing productivity-based tasks.

SUMMARY

In some embodiments, a hub device for interfacing physical peripheral devices for a virtual environment can include a transceiver configured to communicate with a display system for presenting a virtual reality or augmented reality virtual reality environment to a user; a tracking subsystem configured to sense a location of a physical peripheral input device within a physical environment; and one or more processors coupled to the transceiver and the tracking subsystem, the one or more processors configured to: determine, via the tracking subsystem, the location of the physical peripheral input device within the physical environment; and transmit, via the transceiver, to the display system, the location of the physical peripheral input device within the physical environment. The one or more processors can be configured to: determine, via the tracking subsystem, the location of the physical peripheral input device in relation to the device; transform the location of the physical peripheral input device from a coordinate system used by the device to a coordinate system used by the display system; and transmit, via the transceiver, to the display system, the location of the physical peripheral input device using the coordinate system of the display system. In certain embodiments, the display system is a head-mounted display (HMD).

In certain embodiments, a hub device for determining a virtual workstation for use with a virtual environment can include a transceiver configured to communicate with a display system for presenting a virtual reality or augmented reality virtual reality environment to a user; one or more sensors configured to detect aspects of a physical environment in proximity to the device; and one or more processors configured to: determine, via the one or more sensors, the aspects of the physical environment in proximity to the device; based on determining the aspects of the physical environment, defining one or more zones for the user to interact with a physical peripheral input device and a virtual display provided by the display system; and transmit, via the transceiver, to the display system, information indicative of the one or more zones. The one or more zones can include a first zone that defines an area characterized in that a physical peripheral input device placed within the first zone is used by a user to interact with one or more virtual displays rendered by the display system. In some cases, the one or more zones can include a second zone characterized in that a virtual display can be perceived by the user as being located within the second zone and interaction with a physical peripheral device within the first zone can modify content observable by the user rendered on the virtual display within the second zone. In certain implementations, the one or more zones can include a third zone characterized in that content displayed within the third zone is viewable by a plurality of users interacting within an augmented reality or virtual reality virtual reality environment; and where content displayed within the second zone is viewable by only one user of the plurality of users. The display system may be a head-mounted display (HMD) or other suitable visual interface.

In some embodiments, a method of operating a hub to interact with one or more peripheral devices within an AR/VR workstation environment includes detecting a first physical peripheral device within a physical environment; determining a location of the first physical peripheral device within the physical environment; establishing a communicative coupling with, and receiving data from, the first physical peripheral device and an AR/VR-based display; facilitating a transfer of received data between the first physical peripheral device and the AR/VR-based display; determining an area to orient a virtual display relative to the first physical peripheral device within the physical environment based on the determined location of the first physical peripheral device; and generating control data configured to cause the AR/VR-based display to project the virtual display in the determined area. In some cases, the control data can be further configured to cause the AR/VR-based display to maintain a spatial relationship between the first physical peripheral device and the projected virtual display as the first physical peripheral device is moved within the physical environment. The method can further include receiving polling data from one or more sensors, the polling data corresponding to physical characteristics of a physical environment around the first physical peripheral device, where determining the area to orient the virtual display is further based on the physical characteristics of the physical environment. In some cases, a movement and location of a head-mounted display (HMD) of the AR/VR-based display and the hub can be tracked using a first tracking system, and the determining a location of the first physical peripheral device within the physical environment can be tracked via the hub using a second tracking system, the method further comprising: providing data corresponding to the determined location of the first physical peripheral device within the physical environment to the HMD, the provided data configured to cause an integration of tracking via the second tracking system with tracking via the first tracking system.

In certain embodiments, the method can further include detecting a second physical peripheral device within the physical environment; determining a location of the second physical peripheral device within the physical environment; establishing a communicative coupling with, and receiving data from the second physical peripheral device; and coalescing the received data from the first and second physical peripheral devices into aggregate peripheral data, where the aggregate peripheral data is transferred to the AR/VR-based display via the hub instead of the received data from each of the first and second physical peripheral devices. The display system may be a head-mounted display (HMD) or other suitable interface.

In further embodiments, a system includes one or more processors and one or more non-transitory computer-readable storage mediums containing instructions configured to cause the one or more processors to perform operations including: detecting a first physical peripheral device within a physical environment; determining a location of the first physical peripheral device within the physical environment; establishing a communicative coupling with, and receiving data from, the first physical peripheral device and an AR/VR-based display; facilitating a transfer of received data between the first physical peripheral device and the AR/VR-based display; determining an area to orient a virtual display relative to the first physical peripheral device within the physical environment based on the determined location of the first physical peripheral device; and generating control data configured to cause the AR/VR-based display to project the virtual display in the determined area. In some cases, the control data can be further configured to cause the AR/VR-based display to maintain a spatial relationship between the first physical peripheral device and the projected virtual display as the first physical peripheral device is moved within the physical environment. In certain implementations, the one or more non-transitory computer-readable storage mediums can further contain instructions configured to cause the one or more processors to perform operations including: receiving polling data from one or more sensors, the polling data corresponding to physical characteristics of a physical environment around the first physical peripheral device, where determining the area to orient the virtual display is further based on the physical characteristics of the physical environment.

In some embodiments, with respect to the system described above, a movement and location of a head-mounted display (HMD) of the AR/VR-based display and the hub can be tracked using a first tracking system, where the determining a location of the first physical peripheral device within the physical environment can be tracked via the hub using a second tracking system, and where the one or more non-transitory computer-readable storage mediums can further contain instructions configured to cause the one or more processors to perform operations including: providing data corresponding to the determined location of the first physical peripheral device within the physical environment to the HMD, the provided data configured to cause an integration of tracking via the second tracking system with tracking via the first tracking system. The one or more non-transitory computer-readable storage mediums may further contain instructions configured to cause the one or more processors to perform operations including: detecting a second physical peripheral device within the physical environment; determining a location of the second physical peripheral device within the physical environment; establishing a communicative coupling with, and receiving data from the second physical peripheral device; and coalescing the received data from the first and second physical peripheral devices into aggregate peripheral data, where the aggregate peripheral data is transferred to the AR/VR-based display via the hub instead of the received data from each of the first and second physical peripheral devices. The first and second physical peripheral devices may include at least one of a keyboard, computer mouse, stylus, AR/VR controller, remote, or presenter device.

In certain embodiments, a method of operating a virtual workstation can include determining a location of a physical peripheral input device within a physical environment and determining, based on the location of the physical peripheral input device within the physical environment, an orientation of a first virtual display to render to a user of the physical peripheral device, where content presented to the user via the rendered first virtual display can be modifiable in response to the user operating the physical peripheral input device. In some cases, the determined orientation of the first virtual display to render can be configured to remain at a fixed, perceived spatial relationship with respect to the physical peripheral input device as the physical peripheral input device moves within the physical environment. In certain embodiments, the determined orientation of the first virtual display to render can be configured to remain at a fixed, perceived spatial relationship with respect to the physical peripheral input device as the physical peripheral input device moves within the physical environment in response to a movement of the physical input device from an initial location that is greater than a threshold distance, and the determined orientation of the first virtual display to render can be configured to remain in place in response to a movement of the physical input device from the initial location that is less than the threshold distance.

In some embodiments, the method may further include determining, based on the location of the physical peripheral input device within the physical environment, an orientation of an interactive virtual display to render to the user of the physical peripheral device, where content presented to the user via the rendered interactive virtual display can be selectable by the user to modify an operation corresponding to use of the physical peripheral input device by the user, and where the interactive virtual display can be rendered to appear to the user to be in proximity to the physical peripheral device. The operation corresponding to use of the physical peripheral input device by the user may include changing a command corresponding to actuation of an input element of the physical peripheral input device, changing an appearance of the physical peripheral input device, an appearance of an input element of the physical peripheral input device, and/or changing a relationship between one or more rendered virtual displays and the physical peripheral input device.

In further embodiments, the relationship between the one or more rendered virtual displays and the physical peripheral input device may include selecting which of the one or more rendered virtual displays include content that is modifiable in response to the user operating the physical peripheral input device, selecting which of the one or more rendered virtual displays are anchored in position with respect to the physical peripheral input device, and/or selecting how one of the one or more rendered virtual displays are anchored in position with respect to the physical peripheral input device including: a threshold distance that the physical peripheral input device can be moved before the one of the one or more rendered virtual displays moves in response to the physical peripheral input device being moved; or whether the one or more rendered virtual displays move in response to the physical peripheral input device being moved. In some cases, the orientation of the interactive virtual display can be rendered to appear to the user to be at a location that is integrated on or within a periphery of the physical peripheral input device, or the interactive virtual display can be rendered to appear to the user to be located on an area of the physical peripheral input device including features that supplement functionality provided by the interactive virtual display.

In some embodiments, the features may include haptic feedback generated by a haptic feedback generator integrated within the physical peripheral input device to provide haptic feedback in response to user interaction with the interactive virtual display. The features can include a physical input element and where the content of the interactive virtual display is rendered to appear to the user to be selectable by the user by actuation of the physical input element. The interactive virtual display can be rendered to appear to the user to be located in an area in proximity to the physical peripheral input device. The content can be determined to be selected by the user, without physically contacting the physical peripheral input device, by a sensor of the physical peripheral input device. The method may further include determining a location of a second physical peripheral input device within the physical environment and determining, based on the location of the second physical peripheral input device within the physical environment, an orientation of a second interactive virtual display to render to a user of the second physical peripheral device. In such cases, the content presented to the user via the rendered display can be selectable by the user of the second physical peripheral input device to modify an operation to use of the second physical peripheral input device by the user of the second physical peripheral input device, and the second interactive virtual display can be rendered to appear to the user of the second physical peripheral input device to be in proximity to the second physical peripheral input device. In certain embodiments, the interactive virtual display can be rendered to appear to the user to be at a spatial location determined, at least in part, based on a biomechanical model of the user to enable the user to reach the interactive display with an appendage.

In further embodiments, the method may include generating control data to cause a head-mounted display device to render the first virtual display. The physical peripheral input device can be a computer keyboard, mini-tower, or other peripheral that is preferably stationary during use. The orientation of the first virtual display may be determined further based on physical characteristics of an environment that the physical peripheral input device is located within. The physical characteristics may include barriers that would block a line of sight of the user, and may be determined, in part, by a sensor of the physical peripheral input device. In some cases, the first virtual display can be rendered in response to actuation of a physical input element of the physical peripheral input device, in response to a determined proximity of the user to the physical peripheral input device, or in response to the physical peripheral input device being placed upon a working surface. Determination of the physical peripheral input device being placed upon the working surface may be based on a reading of a sensor attached to the physical peripheral input device.

In certain embodiments, the first virtual display can be rendered at a first orientation such that the first virtual display appears to the user to be at a first spatial orientation with respect to the physical peripheral input device in response to a determination that the physical peripheral input device is in a first state indicative of the user operating the physical peripheral device and the first virtual display may not be rendered at the first orientation in response to a determination that the physical peripheral input device is not in the first state. In some cases, the first virtual display can be rendered at a second orientation such that the first virtual display appears to the user to be at a second spatial orientation with respect to the physical peripheral input device in response to a determination that the physical peripheral input device is not in the first state. The first virtual display may be automatically rendered at the first orientation in response to a determination that the physical peripheral input device is in the first state. The physical peripheral input device may include features for determining the location of the physical peripheral input device within the physical environment by a tracking system and the tracking system can be used to determine an orientation of a physical display used to render the first virtual display. In some embodiments, the features can be selected from a list including a sensor, an emitter, and a marking. The sensor can be configured to detect or the emitter can be configured to emit: visible light, infrared light, ultrasound, magnetic fields, or radio waves. In some cases, the physical peripheral input device can include a plurality of the features to enable the physical peripheral input device to be tracked within the physical environment by any one of a plurality of tracking techniques. The physical peripheral input device may include an inertial measurement unit (IMU) and the location of the physical peripheral input device within the physical environment can be determined using the IMU. In certain embodiments, the orientation of the first virtual display can be determined based on a determined identity of the user of the physical peripheral input device and wherein the orientation of the first virtual display would be rendered differently for a differently identified user.

In some embodiments, a method of operating a peripheral device includes receiving polling data from one or more sensors, the polling data corresponding to physical characteristics of a physical environment around the peripheral device, determining an area to orient a virtual display relative to the peripheral device within the physical environment based on the physical characteristics, determining a spatial relationship between the peripheral device and the projected virtual display, and generating control data configured to cause an AR/VR-based head-mounted display (HMD) to project the virtual display in the determined area at a maintained spatial relationship between the peripheral device and the projected virtual display as the peripheral device is moved within the physical environment. The method can further include detecting that the peripheral device is placed on a surface or interfaced by a user, where receiving the polling data from the one or more sensors may occur in response to detecting that the peripheral device is placed on the surface or interfaced by the user. The method can include determining that the peripheral device is lifted off of the surface, and generating second control data to cause the HMD to change the spatial relationship between the peripheral device and the projected virtual display such that a volumetric area occupied by the peripheral device and the projected virtual display is reduced.

In some implementations, the method may further include determining a second area to orient a virtual interactive display relative to the peripheral device, where the interactive display can be configured to facilitate an augmentation of functional capabilities of the peripheral device, and determining a spatial relationship between the peripheral device and the projected interactive display, where the control data can be further configured to cause the HMD to project the interactive display in the determined second area and at a maintained spatial relationship between the peripheral device and the projected interactive display as the peripheral device is moved in the physical environment. The control data may cause the spatial relationship between the peripheral device and the virtual display to be maintained such that a movement of the peripheral device that is within a threshold distance from an initial location of the peripheral device does not cause the virtual display to move and a movement of the peripheral device that is greater than the threshold distance from the initial location of the peripheral device can cause the spatial relationship between the peripheral device and the projected interactive display to be fixed, where the spatial relationship between the peripheral device and the projected interactive display is fixed as the peripheral device is moved in the physical environment.

According to some embodiments, the method can include determining a third area on the peripheral device to orient a virtual overlay, where the virtual overlay can be configured to further facilitate the augmentation of the functional capabilities of the peripheral device, and determining a spatial relationship between the peripheral device and the projected virtual overlay, where the control data is further configured to cause the HMD to project the virtual overlay in the determined third area and at a maintained spatial relationship between the peripheral device and the projected interactive display as the peripheral device is moved in the physical environment.

In further embodiments, a hub device for interfacing physical peripheral devices for a virtual environment includes a transceiver configured to communicate with a display system for presenting a virtual reality or augmented reality virtual reality environment to a user, a tracking subsystem configured to sense a location of a physical peripheral input device within a physical environment, and one or more processors coupled to the transceiver and the tracking subsystem, the one or more processors configured to determine, via the tracking subsystem, the location of the physical peripheral input device within the physical environment, and transmit, via the transceiver, to the display system, the location of the physical peripheral input device within the physical environment. In some cases, the tracking subsystem can sense the location of the physical peripheral input device using a technique different from a technique used by the display system to track a location of the user or a head mounted display (HMD) worn by the user. The technique used by the tracking subsystem to track the physical peripheral input device and the technique used by the display system to track the location of the user or the HMD may be each selected from a list comprising: an ultrasonic emitter; an ultrasonic receiver; a visible light optical sensor; a visible light optical emitter; a non-visible light optical sensor; a non-visible light optical emitter; a magnetic field generator; a magnetic field sensor; a radio wave emitter; and a radio wave receiver. In some cases, the hub device can further comprising features for determining the location of the device within the physical environment by a system used to track a location of the user or a head mounted display (MHD) worn by the user. The features may be selected from a list including: a sensor; an emitter; and a marking, for example. The sensor can be configured to detect or the emitter is configured to emit: visible light, infrared light, ultrasound, magnetic fields, or radio waves.

In some embodiments, the one or more processors of the hub device can be configured to: determine, via the tracking subsystem, the location of the physical peripheral input device in relation to the device; transform the location of the physical peripheral input device from a coordinate system used by the device to a coordinate system used by the display system; and transmit, via the transceiver, to the display system, the location of the physical peripheral input device using the coordinate system of the display system. The coordinate system used by the display system may be used to track a location of the user or an HMD, or track a location of the user or an HMD. In some cases, the one or more processors of the hub device can be configured to receive, from the physical peripheral input device, a signal corresponding to actuation of an input element of the peripheral input device and transmit, via the transceiver, to the display system, a signal corresponding to the actuation of the input element.

In certain embodiments, the tracking subsystem of the hub device may be configured to sense locations each of a plurality of physical peripheral input devices within the physical environment; and the one or more processors can be configured to determine, via the tracking subsystem, the locations each of the plurality of physical peripheral input devices within the physical environment and transmit, via the transceiver, to the display system, the locations of each of the plurality of physical peripheral input devices within the physical environment. The hub device can further comprising a memory storing a plurality of profiles, each of the plurality of profiles corresponding to a different physical peripheral input device.

One of the plurality of profiles may include information for translating a signal received from a physical peripheral input device corresponding to the one of the plurality of profiles to a signal interpretable by the display system. The signal received from the physical peripheral input device corresponding to the one of the plurality of profiles may correspond to actuation of an input element of the physical peripheral input device corresponding to the one of the plurality of profiles. In some cases, one of the plurality of profiles can include information for rendering a visual representation of a physical peripheral input device corresponding to the one of the plurality of profiles, or information indicating features of a physical peripheral input device corresponding to the one of the plurality of profiles, where the features indicate capabilities of the physical peripheral input device to support augmented virtual content.

In some embodiments, the features indicate an area of the physical peripheral input device can include physical elements where augmented virtual content can be displayed to the user and wherein the elements support the augmented virtual content by providing a physical stimulus to a user interacting with the augmented virtual content or a tactile input element for interacting with the augmented virtual content. The location of the physical peripheral input device can be determined within a three-dimensional space. In some cases, the tracking subsystem may include, for determining a location of the physical peripheral input device, at least two of: an ultrasonic emitter or receiver; an optical sensor or receiver; a magnetic field generator or sensor; and a radio wave emitter or receiver.

According to certain embodiments, a hub device for determining a virtual workstation for use with a virtual environment can include a transceiver configured to communicate with a display system for presenting a virtual reality or augmented reality virtual reality environment to a user; one or more sensors configured to detect aspects of a physical environment in proximity to the device; and one or more processors configured to: determine, via the one or more sensors, the aspects of the physical environment in proximity to the device; based on determining the aspects of the physical environment, defining one or more zones for the user to interact with a physical peripheral input device and a virtual display provided by the display system; and transmit, via the transceiver, to the display system, information indicative of the one or more zones. The one or more sensors may include an optical, electromagnetic, or ultrasound imager to detect the aspects of the physical environment. The one or more zones can include a first zone that defines an area characterized in that a physical peripheral input device placed within the first zone is used by a user to interact with one or more virtual displays rendered by the display system. The first zone may be determined based on detecting a working surface that a physical peripheral input device can be placed upon. The working surface may include a substantially planar and horizontal surface. In some cases, the first zone can be determined based on identifying a location of a physical peripheral input device within the physical environment, or based on identifying one or more fiducial marks on a surface.

In certain embodiments, the one or more zones may include a second zone characterized in that a virtual display can be perceived by the user as being located within the second zone and interaction with a physical peripheral device within the first zone can modify content observable by the user rendered on the virtual display within the second zone. In some cases, the second zone can be selected to align with a vertical physical boundary detected within the physical environment. The second zone may be defined to align with a vertical physical boundary detected within the physical environment. The second zone can be defined based on a type of physical peripheral device detected within the first zone. In some implementations, the one or more zones can include a third zone, the third zone characterized in that content displayed on a virtual display can be perceived by the user as being located within the third zone and where interaction of the user with the content modifies operation of a physical peripheral input device located within the first zone. The location of the third zone can be determined based on a location of a physical peripheral input device being detected within the first zone. A plurality of third zones can be determined, each corresponding to a respective peripheral input device. The location of the third zone may be determined based on a biomechanical model of the user such that the location enables the user to physically reach the third zone using an appendage. In some cases, the information indicative of the one or more zones may include spatial coordinates of the one or more zones. The one or more zones may be determined based on a location of the hub device within the physical environment, or based on locations of each of a plurality of hub devices, where the hub device is included in the plurality of hub devices. The one or more zones can include a fourth zone characterized in that content displayed within the fourth zone is viewable by a plurality of users interacting within an augmented reality or virtual reality virtual reality environment; and where content displayed within the second zone is viewable by only one user of the plurality of users. In some cases, the one or more sensors can include an inertial measurement unit (IMU) and the one or more processors are configured to determine the location of the hub device within the physical environment via the IMU.

In some embodiments, a method of operating a hub to interact with one or more peripheral devices within an AR/VR workstation environment can include detecting a first physical peripheral device within a physical environment; determining a location of the first physical peripheral device within the physical environment; establishing a communicative coupling with, and receiving data from, the first physical peripheral device and an AR/VR-based display; facilitating a transfer of received data between the first physical peripheral device and the AR/VR-based display; determining an area to orient a virtual display relative to the first physical peripheral device within the physical environment based on the determined location of the first physical peripheral device; and generating control data configured to cause the AR/VR-based display to project the virtual display in the determined area. In certain embodiments, the control data can be further configured to cause the AR/VR-based display to maintain a spatial relationship between the first physical peripheral device and the projected virtual display as the first physical peripheral device is moved within the physical environment. The method can further include receiving polling data from one or more sensors, the polling data corresponding to physical characteristics of a physical environment around the first physical peripheral device, where determining the area to orient the virtual display can be further based on the physical characteristics of the physical environment.

In further embodiments, a movement and location of a head-mounted display (HMD) of the AR/VR-based display and the hub can be tracked using a first tracking system, and where the determining a location of the first physical peripheral device within the physical environment is tracked via the hub using a second tracking system, the method further comprising: providing data corresponding to the determined location of the first physical peripheral device within the physical environment to the HMD, the provided data configured to cause an integration of tracking via the second tracking system with tracking via the first tracking system. The method can further include detecting a second physical peripheral device within the physical environment; determining a location of the second physical peripheral device within the physical environment; establishing a communicative coupling with, and receiving data from the second physical peripheral device; and coalescing the received data from the first and second physical peripheral devices into aggregate peripheral data, where the aggregate peripheral data can be transferred to the AR/VR-based display via the hub instead of the received data from each of the first and second physical peripheral devices. In some cases, the hub may be integrated with one of a keyboard, smart device, wearable device, or standalone entity.

In some embodiments, a system to interface a user with a computer can include a display configured to present, to a user, one or more images to enable a virtual reality or augmented reality virtual reality environment for the user; and one or more processors coupled to the display, the one or more processors configured to: induce the display to render the one or more images to enable the virtual reality or augmented reality virtual reality environment for the user; and induce the display to render one or more layers to present, to the user, a representation of a physical interface device within the virtual reality or augmented reality virtual reality environment, wherein the representation appears to the user to be at a location within the virtual reality or augmented reality virtual reality environment corresponding to a physical spatial location of the physical interface device. The one or more layers to present the representation of the physical interface device may include an input element layer representing one or more input elements of the physical interface device; an indicia layer representing one or more indicia associated with the input elements; a feedback layer providing user feedback of a state of the one or more input element; or a body layer representing a body of the physical interface device. The one or more layers can include a proximal field layer representing a field proximal to the physical interface device, or a body part layer representing a body part interfacing with the physical interface device. In some implementations, one of the one or more layers can be represented as being partially transparent, where the one or more partially transparent layers can be superposed on another one of the one or more layers; or another one of the one or more layers is superposed on the one or more partially transparent layers. In some aspects, the one or more layers may include features representative of a depth of an object in relation to the physical interface device.

In certain embodiments, rendering the one or more of the layers or a change in rendering the one or more layers can be in response to trigger, the trigger including one or more of: a configuration of the physical interface device; an arrangement of the physical interface device; a detection of a physical object associated with the system; a contextual event associated with the system; and a user interaction with the system. The configuration of the physical interface device may include a change in a form factor of the physical interface device. In some aspects, the arrangement of the physical interface device can include a change in an orientation of the physical interface device and/or a location of the physical interface device. In some embodiments, the location can be a location upon a work surface; and the one or more layers can include a proximal field layer representative of the work surface. The physical object can be an interoperable user interface device with the physical interface device; and the detection of the physical object may include detecting a proximity of the physical object to the physical interface device.

In further embodiments, the user interaction with the system can include one or more of: a gesture made by the user; a user interfacing with the physical interface device; or detection of a user's intention to interface with the physical interface device. In some cases, the one or more of the layers can be user customizable. The representation of the physical interface device may include an image of the physical interface device bound to a physical size of the physical interface device; and one of the one or more of the layers can extend beyond the image of the physical interface device bound to the physical size of the physical interface device. The system can further including a sensor system configured to sense a real-world arrangement of the physical interface device, where the one or more processors can be configured to induce the display to render the one or more layers based on the sensor system sensing the real-world arrangement of the physical interface device.

