Movement, orientation, and motion: A WebXR example - Web APIs

Movement, orientation, and motion: A WebXR example
==================================================

In this article, we'll make use of information introduced in the previous articles in our WebXR tutorial series to construct an example which animates a rotating cube around which the user can move freely using a VR headset, keyboard, and/or mouse. This will help to solidify your understanding of how the geometry of 3D graphics and VR work, as well as to help ensure you understand the way the functions and data that are used during XR rendering work together.

**Figure: Screenshot of this example in action**
![Screenshot of example showing a textured cube the user can move around](/en-US/docs/Web/API/WebXR_Device_API/Movement_and_motion/xr-sample.png)

The core of this example—the spinning, textured, lighted cube—is taken from our WebGL tutorial series; namely, the penultimate article in the series, covering lighting in WebGL.

While reading this article and the accompanying source code, it's helpful to keep in mind that the display for a 3D headset is a single screen, divided in half. The left half of the screen is seen only by the left eye, while the right half is only seen by the right eye. Rendering the scene for immersive presentation requires multiple renders of the scene—once from the perspective of each eye.

When rendering the left eye, the `XRWebGLLayer` has its `viewport` configured to restrict drawing to the left half of the drawing surface. Contrarily, when rendering the right eye, the viewport is set to restrict drawing to the right half of the surface.

This example demonstrates this by showing the canvas on the screen, even when presenting a scene as an immersive display using an XR device.

Dependencies
------------

While we will not rely upon any 3D graphics frameworks such as `three.js` or the like for this example, we do use the `glMatrix` library for matrix math, which we've used in other examples in the past. This example also imports the WebXR polyfill maintained by the Immersive Web Working Group, which is the team responsible for the WebXR API's specification. By importing this polyfill, we allow the example to work on many browsers that don't yet have WebXR implementations in place, and we smooth out any transient deviations from the specification that occur during these still somewhat experimental days of the WebXR specification.

Options
-------

This example has a number of options you can configure by adjusting the values of constants before you load it in the browser. The code looks like this:

```js
const xRotationDegreesPerSecond = 25;
const yRotationDegreesPerSecond = 15;
const zRotationDegreesPerSecond = 35;
const enableRotation = true;
const allowMouseRotation = true;
const allowKeyboardMotion = true;
const enableForcePolyfill = false;
//const SESSION\_TYPE = "immersive-vr";
const SESSION\_TYPE = "inline";
const MOUSE\_SPEED = 0.003;
```

`xRotationDegreesPerSecond`

The number of degrees of rotation to apply around the X axis per second.

`yRotationDegreesPerSecond`

The number of degrees to rotate around the Y axis each second.

`zRotationDegreesPerSecond`

The number of degrees per second to rotate around the Z axis.

`enableRotation`

A Boolean indicating whether or not to enable the rotation of the cube at all.

`allowMouseRotation`

If `true`, you can use the mouse to pitch and yaw the view angle.

`allowKeyboardMotion`

If `true`, the W, A, S, and D keys move the viewer up, left, down, and to the right, while the up and down arrow keys move forward and backward. If `false`, only XR device changes to the view are permitted.

`enableForcePolyfill`

If this Boolean is `true`, the example will attempt to use the WebXR polyfill even if the browser actually has support for WebXR. If `false`, the polyfill is only used if the browser doesn't implement `navigator.xr`.

`SESSION_TYPE`

The type of XR session to create: `inline` for an inline session presented in the context of the document and `immersive-vr` to present the scene to an immersive VR headset.

`MOUSE_SPEED`

A multiplier used to scale the inputs from the mouse used for pitch and yaw control.

`MOVE_DISTANCE`

The distance to move in response to any of the keys used to move the viewer through the scene.

**Note:** This example always displays what it renders on the screen, even if using `immersive-vr` mode. This lets you compare any differences in rendering between the two modes, and lets you see output from immersive mode even if you don't have a headset.

Setup and utility functions
---------------------------

Next, we declare the variables and constants used throughout the application, starting with those used to store WebGL and WebXR specific information:

```js
let polyfill = null;
let xrSession = null;
let xrInputSources = null;
let xrReferenceSpace = null;
let xrButton = null;
let gl = null;
let animationFrameRequestID = 0;
let shaderProgram = null;
let programInfo = null;
let buffers = null;
let texture = null;
let mouseYaw = 0;
let mousePitch = 0;
```

This is followed by a set of constants, mostly to contain various vectors and matrices used while rendering the scene.

```js
const viewerStartPosition = vec3.fromValues(0, 0, -10);
const viewerStartOrientation = vec3.fromValues(0, 0, 1.0);

const cubeOrientation = vec3.create();
const cubeMatrix = mat4.create();
const mouseMatrix = mat4.create();
const inverseOrientation = quat.create();
const RADIANS\_PER\_DEGREE = Math.PI / 180.0;
```

The first two—`viewerStartPosition` and `viewerStartOrientation`—indicate where the viewer will be placed relative to the center of the space, and the direction in which they'll initially be looking. `cubeOrientation` will store the current orientation of the cube, while `cubeMatrix` and `mouseMatrix` are storage for matrices used during the rendering of the scene. `inverseOrientation` is a quaternion which will be used to represent the rotation to apply to the reference space for the object in the frame being rendered.

`RADIANS_PER_DEGREE` is the value to multiply an angle in degrees by to convert the angle into radians.

The last four variables declared are storage for references to the `<div>` elements into which we'll output the matrices when we want to show them to the user.

### Logging errors

A function called `LogGLError()` is implemented to provide an easily customized way to output logging information for errors that occur while executing WebGL functions.

```js
function LogGLError(where) {
  let err = gl.getError();
  if (err) {
    console.error(`WebGL error returned by ${where}: ${err}`);
  }
}
```

This takes as its only input a string, `where`, which is used to indicate what part of the program generated the error, since similar errors can have in multiple situations.

### The vertex and fragment shaders

The vertex and fragment shaders are both exactly the same as those used in the example for our article Lighting in WebGL. Refer to that if you're interested in the GLSL source code for the basic shaders used here.

Suffice it to say that the vertex shader computes the position of each vertex given the initial positions of each vertex and the transforms that need to be applied to convert them to simulate the viewer's current position and orientation. The fragment shader returns the color of each vertex, interpolating as needed from the values found in the texture and applying the lighting effects.

Starting up and shutting down WebXR
-----------------------------------

Upon initially loading the script, we install a handler for the `load` event, so that we can perform initialization.

```js
window.addEventListener("load", onLoad);

function onLoad() {
  xrButton = document.querySelector("#enter-xr");
  xrButton.addEventListener("click", onXRButtonClick);

  projectionMatrixOut = document.querySelector("#projection-matrix div");
  modelMatrixOut = document.querySelector("#model-view-matrix div");
  cameraMatrixOut = document.querySelector("#camera-matrix div");
  mouseMatrixOut = document.querySelector("#mouse-matrix div");

  if (!navigator.xr || enableForcePolyfill) {
    console.log("Using the polyfill");
    polyfill = new WebXRPolyfill();
  }
  setupXRButton();
}
```

The `load` event handler gets a reference to the button that toggles WebXR on and off into `xrButton`, then adds a handler for `click` events. Then references are obtained to the four `<div>` blocks into which we'll output the current contents of each of the key matrices for informational purposes while our scene is running.

Then we look to see if `navigator.xr` is defined. If it isn't—and/or the `enableForcePolyfill` configuration constant is set to `true`—we install the WebXR polyfill by instantiating the `WebXRPolyfill` class.

### Handling the startup and shutdown UI

Then we call the `setupXRButton()` function, which handles configuring the "Enter/Exit WebXR" button to enable or disable it as necessary depending on the availability of WebXR support for the session type specified in the `SESSION_TYPE` constant.

```js
function setupXRButton() {
  if (navigator.xr.isSessionSupported) {
    navigator.xr.isSessionSupported(SESSION\_TYPE).then((supported) => {
      xrButton.disabled = !supported;
    });
  } else {
    navigator.xr
      .supportsSession(SESSION\_TYPE)
      .then(() => {
        xrButton.disabled = false;
      })
      .catch(() => {
        xrButton.disabled = true;
      });
  }
}
```

The label of the button gets adjusted in the code that actually handles starting and stopping the WebXR session; we'll see that below.

The WebXR session is toggled on and off by the handler for `click` events on the button, whose label is appropriately set to either "Enter WebXR" or "Exit WebXR". This is done by the `onXRButtonClick()` event handler.

```js
async function onXRButtonClick(event) {
  if (!xrSession) {
    navigator.xr.requestSession(SESSION\_TYPE).then(sessionStarted);
  } else {
    await xrSession.end();

    if (xrSession) {
      sessionEnded();
    }
  }
}
```

This begins by looking at the value of `xrSession` to see if we already have a `XRSession` object representing an ongoing WebXR session. If we don't, the click represents a request to enable WebXR mode, so call `requestSession()` to request a WebXR session of the desired WebXR session type, and then call `sessionStarted()` to begin running the scene in that WebXR session.

If we already have an ongoing session, on the other hand, we call its `end()` method to stop the session.

The last thing we do in this code is to check to see if `xrSession` is still non-`NULL`. If it is, we call `sessionEnded()`, the handler for the `end` event. This code should not be necessary, but there appears to be an issue in which at least some browsers are not correctly firing the `end` event. By running the event handler directly, we complete the close-out process manually in this situation.

### Starting up the WebXR session

The `sessionStarted()` function handles actually setting up and starting the session, by setting up event handlers, compiling and installing the GLSL code for the vertex and fragment shaders, and attaching the WebGL layer to the WebXR session before kicking off the rendering loop. It gets called as the handler for the promise returned by `requestSession()`.

```js
function sessionStarted(session) {
  let refSpaceType;

  xrSession = session;
  xrButton.innerText = "Exit WebXR";
  xrSession.addEventListener("end", sessionEnded);

  let canvas = document.querySelector("canvas");
  gl = canvas.getContext("webgl", { xrCompatible: true });

  if (allowMouseRotation) {
    canvas.addEventListener("pointermove", handlePointerMove);
    canvas.addEventListener("contextmenu", (event) => {
      event.preventDefault();
    });
  }

  if (allowKeyboardMotion) {
    document.addEventListener("keydown", handleKeyDown);
  }

  shaderProgram = initShaderProgram(gl, vsSource, fsSource);

  programInfo = {
    program: shaderProgram,
    attribLocations: {
      vertexPosition: gl.getAttribLocation(shaderProgram, "aVertexPosition"),
      vertexNormal: gl.getAttribLocation(shaderProgram, "aVertexNormal"),
      textureCoord: gl.getAttribLocation(shaderProgram, "aTextureCoord"),
    },
    uniformLocations: {
      projectionMatrix: gl.getUniformLocation(
        shaderProgram,
        "uProjectionMatrix",
      ),
      modelViewMatrix: gl.getUniformLocation(shaderProgram, "uModelViewMatrix"),
      normalMatrix: gl.getUniformLocation(shaderProgram, "uNormalMatrix"),
      uSampler: gl.getUniformLocation(shaderProgram, "uSampler"),
    },
  };

  buffers = initBuffers(gl);
  texture = loadTexture(
    gl,
    "https://cdn.glitch.com/a9381af1-18a9-495e-ad01-afddfd15d000%2Ffirefox-logo-solid.png?v=1575659351244",
  );

  xrSession.updateRenderState({
    baseLayer: new XRWebGLLayer(xrSession, gl),
  });

  const isImmersiveVr = SESSION\_TYPE === "immersive-vr";
  refSpaceType = isImmersiveVr ? "local" : "viewer";

  mat4.fromTranslation(cubeMatrix, viewerStartPosition);

  vec3.copy(cubeOrientation, viewerStartOrientation);

  xrSession.requestReferenceSpace(refSpaceType).then((refSpace) => {
    xrReferenceSpace = refSpace.getOffsetReferenceSpace(
      new XRRigidTransform(viewerStartPosition, cubeOrientation),
    );
    animationFrameRequestID = xrSession.requestAnimationFrame(drawFrame);
  });

  return xrSession;
}
```

After storing the newly-created `XRSession` object into `xrSession`, the label of the button is set to "Exit WebXR" to indicate its new function after starting the scene, and a handler is installed for the `end` event, so we get notified when the `XRSession` ends.

Then we get a reference to the `<canvas>` found in our HTML—as well as its WebGL rendering context—which will be used as the drawing surface for the scene. The `xrCompatible` property is requested when calling `getContext()` on the element to gain access to the WebGL rendering context for the canvas. This ensures that the context is configured for use as a source for WebXR rendering.

Next, we add event handlers for the `mousemove` and `contextmenu`, but only if the `allowMouseRotation` constant is `true`. The `mousemove` handler will deal with the pitching and yawing of the view based upon the movement of the mouse. Since the "mouselook" feature functions only while the right mouse button is held down, and clicking using the right mouse button triggers the context menu, we add a handler for the `contextmenu` event to the canvas to prevent the context menu from appearing when the user initially begins their drag of the mouse.

Next, we compile the shader programs; get references to its variables; initialize the buffers that store the array of each position; the indexes into the position table for each vertex; the vertex normals; and the texture coordinates for each vertex. This is all taken directly from the WebGL sample code, so refer to Lighting in WebGL and its preceding articles Creating 3D objects using WebGL and Using textures in WebGL. Then our `loadTexture()` function is called to load the texture file.

Now that the rendering structures and data are loaded, we start preparing to run the `XRSession`. We connect the session to the WebGL layer so it knows what to use as a rendering surface by calling `XRSession.updateRenderState()` with a `baseLayer` set to a new `XRWebGLLayer`.

We then look at the value of the `SESSION_TYPE` constant to see whether the WebXR context should be immersive or inline. Immersive sessions use the `local` reference space, while inline sessions use the `viewer` reference space.

The `glMatrix` library's `fromTranslation()` function for 4x4 matrices is used to convert the viewer's start position as given in the `viewerStartPosition` constant into a transform matrix, `cubeMatrix`. The viewer's starting orientation, `viewerStartOrientation` constant, is copied into the `cubeOrientation`, which will be used to track the rotation of the cube over time.

`sessionStarted()` finishes up by calling the session's `requestReferenceSpace()` method to get a reference space object describing the space in which the object is being created. When the promise returned resolves to a `XRReferenceSpace` object, we call its `getOffsetReferenceSpace` method to obtain a reference space object to represent the object's coordinate system. The origin of the new space is located at the world coordinates specified by the `viewerStartPosition` and its orientation set to `cubeOrientation`. Then we let the session know we're ready to draw a frame by calling its `requestAnimationFrame()` method. We record the returned request ID in case we need to cancel the request later.

Finally, `sessionStarted()` returns the `XRSession` representing the user's WebXR session.

### When the session ends

When the WebXR session ends—either because it's being shut down by the user or by calling `XRSession.end()`—the `end` event is sent; we have set this up to call a function called `sessionEnded()`.

```js
function sessionEnded() {
  xrButton.innerText = "Enter WebXR";

  if (animationFrameRequestID) {
    xrSession.cancelAnimationFrame(animationFrameRequestID);
    animationFrameRequestID = 0;
  }
  xrSession = null;
}
```

We can also call `sessionEnded()` directly if we wish to programmatically end the WebXR session. In either case, the label of the button is updated to indicate that a click will start a session, and then, if there is a pending request for an animation frame, we cancel it by calling `cancelAnimationFrame`

Once that's done, the value of `xrSession` is changed to `NULL` to indicate that we're done with the session.

Implementing the controls
-------------------------

Now let's take a look at the code that handles turning keyboard and mouse events into something usable for controlling an avatar in a WebXR scenario.

### Moving using the keyboard

In order to allow the user to move through the 3D world even if they don't have a WebXR device with the inputs to perform movement through space, our handler for `keydown` events, `handleKeyDown()`, responds by updating offsets from the object's origin based on which key was pressed.

```js
function handleKeyDown(event) {
  switch (event.key) {
    case "w":
    case "W":
      verticalDistance -= MOVE\_DISTANCE;
      break;
    case "s":
    case "S":
      verticalDistance += MOVE\_DISTANCE;
      break;
    case "a":
    case "A":
      transverseDistance += MOVE\_DISTANCE;
      break;
    case "d":
    case "D":
      transverseDistance -= MOVE\_DISTANCE;
      break;
    case "ArrowUp":
      axialDistance += MOVE\_DISTANCE;
      break;
    case "ArrowDown":
      axialDistance -= MOVE\_DISTANCE;
      break;
    case "r":
    case "R":
      transverseDistance = axialDistance = verticalDistance = 0;
      mouseYaw = mousePitch = 0;
      break;
    default:
      break;
  }
}
```

The keys and their effects are:

* The
 `W`
 key moves the viewer upward by `MOVE_DISTANCE`.
* The
 `S`
 key moves the viewer downward by `MOVE_DISTANCE`.
* The
 `A`
 key slides the viewer to the left by `MOVE_DISTANCE`.
* The
 `D`
 key slides the viewer to the right by `MOVE_DISTANCE`.
* The up arrow key,
 `↑`
 , slides the viewer forward by `MOVE_DISTANCE`.
* The down arrow key,
 `↓`
 , slides the viewer backward by `MOVE_DISTANCE`.
* The
 `R`
 key resets the viewer to their starting position and orientation by resetting the input offsets all to 0.

These offsets will be applied by the renderer starting with the next frame drawn.

### Pitching and yawing with the mouse

We also have a `mousemove` event handler which checks to see if the right mouse button is down, and if so, calls the `rotateViewBy()` function, defined next, to calculate and store the new pitch (looking up and down) and yaw (looking left and right) values.

```js
function handlePointerMove(event) {
  if (event.buttons & 2) {
    rotateViewBy(event.movementX, event.movementY);
  }
}
```

Calculating the new pitch and yaw values is handled by the function `rotateViewBy()`:

```js
function rotateViewBy(dx, dy) {
  mouseYaw -= dx \* MOUSE\_SPEED;
  mousePitch -= dy \* MOUSE\_SPEED;

  if (mousePitch < -Math.PI \* 0.5) {
    mousePitch = -Math.PI \* 0.5;
  } else if (mousePitch > Math.PI \* 0.5) {
    mousePitch = Math.PI \* 0.5;
  }
}
```

Given as input the mouse deltas, `dx` and `dy`, the new yaw value is computed by subtracting from the current value of `mouseYaw` the product of `dx` and the `MOUSE_SPEED` scaling constant. You can, then, control how responsive the mouse is by increasing the value of `MOUSE_SPEED`.

Drawing a frame
---------------

Our callback for `XRSession.requestAnimationFrame()` is implemented in the `drawFrame()` function shown below. Its job is to obtain the viewer's reference space, compute how much movement needs to be applied to any animated objects given the amount of time that's elapsed since the last frame, and then to render each of the views specified by the viewer's `XRPose`.

```js
let lastFrameTime = 0;

function drawFrame(time, frame) {
  const session = frame.session;
  let adjustedRefSpace = xrReferenceSpace;
  let pose = null;

  animationFrameRequestID = session.requestAnimationFrame(drawFrame);
  adjustedRefSpace = applyViewerControls(xrReferenceSpace);
  pose = frame.getViewerPose(adjustedRefSpace);

  if (pose) {
    const glLayer = session.renderState.baseLayer;

    gl.bindFramebuffer(gl.FRAMEBUFFER, glLayer.framebuffer);
    LogGLError("bindFrameBuffer");

    gl.clearColor(0, 0, 0, 1.0);
    gl.clearDepth(1.0); // Clear everything
    gl.clear(gl.COLOR\_BUFFER\_BIT | gl.DEPTH\_BUFFER\_BIT);
    LogGLError("glClear");

    const deltaTime = (time - lastFrameTime) \* 0.001; // Convert to seconds
    lastFrameTime = time;

    for (const view of pose.views) {
      const viewport = glLayer.getViewport(view);
      gl.viewport(viewport.x, viewport.y, viewport.width, viewport.height);
      LogGLError(`Setting viewport for eye: ${view.eye}`);
      gl.canvas.width = viewport.width \* pose.views.length;
      gl.canvas.height = viewport.height;
      renderScene(gl, view, programInfo, buffers, texture, deltaTime);
    }
  }
}
```

The first thing we do is call `requestAnimationFrame()` to request that `drawFrame()` be called again for the next frame to be rendered. Then we pass the object's reference space into the `applyViewerControls()` function, which returns a revised `XRReferenceSpace` that transforms the position and orientation of the object to take into account the movement, pitch, and yaw applied by the user using the keyboard and mouse. Remember that, as always, the world's objects are moved and reoriented, not the viewer. The returned reference space makes it easy for us to do just that.

With the new reference space in hand, we get the `XRViewerPose` representing the viewer's point of view—for both of their eyes. If that's successful, we begin preparing to render by getting the `XRWebGLLayer` being used by the session and binding its frame buffer to be used as the WebGL frame buffer (so that rendering WebGL draws into the layer and therefore the XR device's display). With WebGL now configured to render to the XR device, we clear the frame to black and are ready to begin rendering.

The time elapsed since the last frame was rendered (in seconds) is computed by subtracting the previous frame's timestamp, `lastFrameTime`, from the current time as specified by the `time` parameter and then multiplying by 0.001 to convert milliseconds to seconds. The current time is then saved into `lastFrameTime`;

The `drawFrame()` function ends by iterating over every view found in the `XRViewerPose`, setting up the viewport for the view, and calling `renderScene()` to render the frame. By setting the viewport for each view, we handle the typical scenario in which the views for each eye are each rendered onto half of the WebGL frame. The XR hardware then handles ensuring that each eye only sees the portion of that image that is intended for that eye.

**Note:** In this example, we're visually presenting the frame both on the XR device *and* on the screen. To ensure that the on-screen canvas is the right size to allow us to do this, we set its width to be equal to the individual `XRView` width multiplied by the number of views; the canvas height is always the same as the viewport's height. The two lines of code that adjust the canvas size are not needed in regular WebXR rendering loops.

### Applying the user inputs

The `applyViewerControls()` function, which is called by `drawFrame()` before beginning to render anything, takes the offsets in each of the three directions, the yaw offset, and the pitch offset as recorded by the `handleKeyDown()` and `handlePointerMove()` functions in response to the user pressing keys and dragging their mouse with the right mouse button pressed. It takes as input the base reference space for the object, and returns a new reference space that alters the location and orientation of the object on match the result of the inputs.

```js
function applyViewerControls(refSpace) {
  if (
    !mouseYaw &&
    !mousePitch &&
    !axialDistance &&
    !transverseDistance &&
    !verticalDistance
  ) {
    return refSpace;
  }

  quat.identity(inverseOrientation);
  quat.rotateX(inverseOrientation, inverseOrientation, -mousePitch);
  quat.rotateY(inverseOrientation, inverseOrientation, -mouseYaw);

  let newTransform = new XRRigidTransform(
    { x: transverseDistance, y: verticalDistance, z: axialDistance },
    {
      x: inverseOrientation[0],
      y: inverseOrientation[1],
      z: inverseOrientation[2],
      w: inverseOrientation[3],
    },
  );
  mat4.copy(mouseMatrix, newTransform.matrix);

  return refSpace.getOffsetReferenceSpace(newTransform);
}
```

If all the input offsets are zero, we just return the original reference space. Otherwise, we create from the orientation changes in `mousePitch` and `mouseYaw` a quaternion specifying the inverse of that orientation, so that applying the `inverseOrientation` to the cube will correctly appear to reflect the viewer's movement.

Then it's time to create a new `XRRigidTransform` object representing the transform that will be used to create the new `XRReferenceSpace` for the moved and/or re-oriented object. The position is a new vector whose `x`, `y`, and `z` correspond to the offsets moved along each of those axes. The orientation is the `inverseOrientation` quaternion.

We copy the transform's `matrix` into `mouseMatrix`, which we'll use later to display the mouse tracking matrix to the user (so this is a step you normally can skip). Finally, we pass the `XRRigidTransform` into the object's current `XRReferenceSpace` in order to obtain the reference space that integrates this transform to represent the placement of the cube relative to the user given the user's movements. That new reference space is returned to the caller.

### Rendering the scene

The `renderScene()` function is called to actually render the parts of the world that are visible to the user at the moment. It's called once for each eye, with slightly different positions for each eye, in order to establish the 3D effect needed for XR gear.

Most of this code is typical WebGL rendering code, taken directly from the `drawScene()` function in the Lighting in WebGL article, and it's there that you should look for details on the WebGL rendering parts of this example [view the code on GitHub]. But here it begins with some code specific to this example, so we'll take a deeper look at that part.

```js
const normalMatrix = mat4.create();
const modelViewMatrix = mat4.create();

function renderScene(gl, view, programInfo, buffers, texture, deltaTime) {
  const xRotationForTime =
    xRotationDegreesPerSecond \* RADIANS\_PER\_DEGREE \* deltaTime;
  const yRotationForTime =
    yRotationDegreesPerSecond \* RADIANS\_PER\_DEGREE \* deltaTime;
  const zRotationForTime =
    zRotationDegreesPerSecond \* RADIANS\_PER\_DEGREE \* deltaTime;

  gl.enable(gl.DEPTH\_TEST); // Enable depth testing
  gl.depthFunc(gl.LEQUAL); // Near things obscure far things

  if (enableRotation) {
    mat4.rotate(
      cubeMatrix, // destination matrix
      cubeMatrix, // matrix to rotate
      zRotationForTime, // amount to rotate in radians
      [0, 0, 1],
    ); // axis to rotate around (Z)
    mat4.rotate(
      cubeMatrix, // destination matrix
      cubeMatrix, // matrix to rotate
      yRotationForTime, // amount to rotate in radians
      [0, 1, 0],
    ); // axis to rotate around (Y)
    mat4.rotate(
      cubeMatrix, // destination matrix
      cubeMatrix, // matrix to rotate
      xRotationForTime, // amount to rotate in radians
      [1, 0, 0],
    ); // axis to rotate around (X)
  }

  mat4.multiply(modelViewMatrix, view.transform.inverse.matrix, cubeMatrix);
  mat4.invert(normalMatrix, modelViewMatrix);
  mat4.transpose(normalMatrix, normalMatrix);

  displayMatrix(view.projectionMatrix, 4, projectionMatrixOut);
  displayMatrix(modelViewMatrix, 4, modelMatrixOut);
  displayMatrix(view.transform.matrix, 4, cameraMatrixOut);
  displayMatrix(mouseMatrix, 4, mouseMatrixOut);

  {
    const numComponents = 3;
    const type = gl.FLOAT;
    const normalize = false;
    const stride = 0;
    const offset = 0;
    gl.bindBuffer(gl.ARRAY\_BUFFER, buffers.position);
    gl.vertexAttribPointer(
      programInfo.attribLocations.vertexPosition,
      numComponents,
      type,
      normalize,
      stride,
      offset,
    );
    gl.enableVertexAttribArray(programInfo.attribLocations.vertexPosition);
  }

  {
    const numComponents = 2;
    const type = gl.FLOAT;
    const normalize = false;
    const stride = 0;
    const offset = 0;
    gl.bindBuffer(gl.ARRAY\_BUFFER, buffers.textureCoord);
    gl.vertexAttribPointer(
      programInfo.attribLocations.textureCoord,
      numComponents,
      type,
      normalize,
      stride,
      offset,
    );
    gl.enableVertexAttribArray(programInfo.attribLocations.textureCoord);
  }

  {
    const numComponents = 3;
    const type = gl.FLOAT;
    const normalize = false;
    const stride = 0;
    const offset = 0;
    gl.bindBuffer(gl.ARRAY\_BUFFER, buffers.normal);
    gl.vertexAttribPointer(
      programInfo.attribLocations.vertexNormal,
      numComponents,
      type,
      normalize,
      stride,
      offset,
    );
    gl.enableVertexAttribArray(programInfo.attribLocations.vertexNormal);
  }

  gl.bindBuffer(gl.ELEMENT\_ARRAY\_BUFFER, buffers.indices);
  gl.useProgram(programInfo.program);

  gl.uniformMatrix4fv(
    programInfo.uniformLocations.projectionMatrix,
    false,
    view.projectionMatrix,
  );
  gl.uniformMatrix4fv(
    programInfo.uniformLocations.modelViewMatrix,
    false,
    modelViewMatrix,
  );
  gl.uniformMatrix4fv(
    programInfo.uniformLocations.normalMatrix,
    false,
    normalMatrix,
  );
  gl.activeTexture(gl.TEXTURE0);
  gl.bindTexture(gl.TEXTURE\_2D, texture);

  gl.uniform1i(programInfo.uniformLocations.uSampler, 0);

  {
    const vertexCount = 36;
    const type = gl.UNSIGNED\_SHORT;
    const offset = 0;
    gl.drawElements(gl.TRIANGLES, vertexCount, type, offset);
  }
}
```

`renderScene()` begins by calculating how much rotation should occur around each of the three axes in the amount of time that has elapsed since the previous frame was rendered. These values let us adjust the rotation of our animating cube the right amount to ensure that its movement speed stays consistent regardless of variations in the frame rate that may occur due to system load. These values are calculated as the number of radians of rotation to apply given the elapsed time and stored into the constants `xRotationForTime`, `yRotationForTime`, and `zRotationForTime`.

After enabling and configuring depth testing, we check the value of the `enableRotation` constant to see if rotation of the cube is enabled; if it is, we use glMatrix to rotate the `cubeMatrix` (representing the cube's current orientation relative to the world space) around the three axes. With the cube's global orientation established, we then multiply that by the inverse of the view's transform matrix to get the final model view matrix—the matrix to apply to the object to both rotate it for its animation purposes, but to also move and reorient it to simulate the viewer's motion through the space.

Then the view's normal matrix is computed by taking the model view matrix, inverting it, and transposing it (swapping its columns and rows).

The last few lines of code added for this example are four calls to `displayMatrix()`, a function which displays the contents of a matrix for analysis by the user. The remainder of the function is identical or essentially identical to the older WebGL sample from which this code is derived.

### Displaying a matrix

For instructive purposes, this example displays the contents of the important matrices used while rendering the scene. The `displayMatrix()` function is used for this; this function uses MathML to render the matrix, falling back to a more array-like format if MathML isn't supported by the user's browser.

```js
function displayMatrix(mat, rowLength, target) {
  let outHTML = "";

  if (mat && rowLength && rowLength <= mat.length) {
    let numRows = mat.length / rowLength;
    outHTML =
      "<math xmlns='http://www.w3.org/1998/Math/MathML' display='block'>\n<mrow>\n<mo>[</mo>\n<mtable>\n";

    for (let y = 0; y < numRows; y++) {
      outHTML += "<mtr>\n";
      for (let x = 0; x < rowLength; x++) {
        outHTML += `<mtd><mn>${mat[x \* rowLength + y].toFixed(2)}</mn></mtd>\n`;
      }
      outHTML += "</mtr>\n";
    }

    outHTML += "</mtable>\n<mo>]</mo>\n</mrow>\n</math>";
  }

  target.innerHTML = outHTML;
}
```

This replaces the contents of the element specified by `target` with a newly-created `<math>` element which contains the 4x4 matrix. Each entry is displayed with up to two decimal places.

### Everything else

The rest of the code is identical to that found in the earlier examples:

`initShaderProgram()`

Initializes the GLSL shader program, calling `loadShader()` to load and compile each shader's program, then attaching each one to the WebGL context. Once they're compiled, the program is linked and returned to the caller.

`loadShader()`

Creates a shader object and loads the specified source code into it before compiling the code and checking to ensure that the compiler succeeded before returning the newly compiled shader to the caller. If an error occurs, `NULL` is returned instead.

`initBuffers()`

Initializes the buffers that contain data to be passed into WebGL. These buffers include the array of vertex positions, the array of vertex normals, the texture coordinates for each surface of the cube, and the array of vertex indices (specifying which entry in the vertex list represents each corner of the cube).

`loadTexture()`

Loads the image at a given URL and creates a WebGL texture from it. If the image's dimensions aren't both powers of two (see the `isPowerOf2()` function), mipmapping is disabled and wrapping is clamped to the edges. This is because optimized rendering of mipmapped textures only works for textures whose dimensions are powers of two in WebGL 1. WebGL 2 supports arbitrarily-sized textures for mipmapping.

`isPowerOf2()`

Returns `true` if the specified value is a power of two; otherwise returns `false`.

### Putting it all together

When you take all of this code and add in the HTML and the other JavaScript code not included above, you get what you see when you try out this example on Glitch. Remember: as you wander around, if you get lost, just hit the `R` key to reset yourself to the beginning.

A tip: if you don't have an XR device, you may be able to get some of the 3D effect if you bring your face very close to the screen, with your nose centered along the border between the left and right eye images in the canvas. By carefully focusing through the screen at the image, and slowly moving forward and backward, you should eventually be able to bring the 3D image into focus. It can take practice, and your nose may literally be touching the screen, depending on how sharp your eyesight is.

There are plenty of things you can do using this example as a starting point. Try adding more objects to the world, or improve the movement controls to move more realistically. Add walls, ceiling, and floor to enclose you in a space instead of having an infinite-seeming universe to get lost in. Add collision testing or hit testing, or the ability to change the texture of each face of the cube.

There are few limitations on what can be done if you set yourself to it.

See also
--------

* Learn WebGL (includes some great visualizations of the camera and how it relates to the virtual world)
* WebGL Fundamentals
* Learn OpenGL


Inputs and input sources - Web APIs

Inputs and input sources
========================

**Secure context:** This feature is available only in secure contexts (HTTPS), in some or all supporting browsers.

A full WebXR experience isn't just about showing the user a wholly virtual scene or augmenting reality by adding to or altering the world around them. In order to make an experience that's fulfilling and engaging, the user needs to be able to interact with it. To that end, WebXR provides support for a variety of kinds of input devices.

In this guide, we'll look at how to use WebXR's input device management features to determine what input sources are available and how to then monitor those sources for inputs in order to handle user interactivity with your virtual or augmented environment.

Inputs in WebXR
---------------

Fundamentally, inputs in WebXR fall into two basic categories: targeting and actions. Targeting is the specification of a point in space by the user's input. This may involve the user tapping on a spot on the screen, tracking their eyes, or the use of a joystick or motion-sensing controller to move a cursor.

Actions include both selection actions, such as clicking on a button, and squeeze actions, such as pulling a trigger or tightening your grip while wearing haptic gloves.

By combining these two types of input with the changing of viewing position and/or orientation through the headset or other mechanisms, you can create an interactive simulated environment.

### Input device types

WebXR supports a variety of different types of devices to handle targeting and action inputs. These devices include but aren't limited to:

* Screen taps (particularly but not necessarily only on phones or tablets) can be used to simultaneously perform both targeting and selection.
* Motion-sensing controllers, which use accelerometers, magnetometers, and other sensors for motion tracking and targeting and may additionally include any number of buttons, joysticks, thumbpads, touchpads, force sensors, and so on to provide additional input sources for both targeting and selection.
* Squeezable triggers or glove grip pads to provide squeeze actions.
* Voice commands using speech recognition.
* Spatially-tracked articulated hands, such as wired gloves can provide both targeting and squeeze actions, as well as selection if outfitted with buttons or other sources of selection actions.
* Single-button click devices.
* Gaze tracking (following the movements of the eye to choose targets).
### Input sources

Each source of WebXR input data is represented by an `XRInputSource` object which describes the input source and its current state. The information for each input source includes which hand it's held in (if applicable), what targeting method it uses, `XRSpace`s that can be used to draw the targeting ray and to find the targeted object or location as well as to draw objects in the user's hands, and profile strings specifying the preferred way to represent the controller in the user's viewing area as well as how the input operates.

The fundamental capabilities of an input source are:

Targeting

Monitoring directional controls (either a motion-sensing pointer or a joystick or trackpad, for example) to aim in a direction, possibly at a target, though targeting is left to you to implement yourself. See Facing and targeting for further information.

Selecting

Using the main "select" button or other input on the controller to choose the targeted direction (or the object at which it points), or to otherwise engage an action. For details on the primary action, see Primary action.

Squeezing

Squeezing the controller or a mechanism on the controller to engage a secondary action. The section Primary squeeze action describes this in more detail.

Any additional capabilities a WebXR controller may have are accessed through the input source's `gamepad` object. This object provides access to all of the buttons, axes, trackpads, and so forth that may be a part of the controller. See Advanced controllers using the gamepad object to learn how to use these controllers.

### Instance properties of input sources

Each individual `XRInputSource` has a set of properties that describe the input's available axes and buttons, which hand the user's holding it in, and how the input source is used to handle targeting within the 3D space.

#### Handedness

**Handedness**, indicated by the `XRInputSource` property `handedness`, is a string which indicates which of the viewer's hands the controller is in: `left` or `right`. It can also be set to `none` if the controller isn't hand-held, or if it's unknown which hand the controller is in.

Handedness can be used for various things, including selecting an appropriate mesh to use to represent the controller in the view and to help present it in the correct hand if drawing hands on the display. It can also be useful if your app uses the notion of "main hand" and "off hand" for determining the functionality of a controller; in a game, for example, the main hand controller may be the player's weapon while the off hand controller might be used to control the positioning of a shield.

#### Targeting ray mode

The targeting ray mode is a string found in the `targetRayMode` property. It describes the technique used to determine the targeting ray and how it should be shown to the user if presented visually.

When the targeting ray mode is `gaze`, the ray's origin is at the viewer and aims in the direction the user is facing. This gaze input method is fairly simple, and doesn't need any special controls, as it will be based on the facing direction reported by the headset or whatever device is used to determine what direction the viewer's face is pointing in. The target ray should extend outward from between the eyes in a direction perpendicular to the viewer's face.

More flexible is the `tracked-pointer` mode, in which the ray's origin is at a handheld controller or hand tracking system's origin and extends outward in the direction in which the controller is pointing. The ray extends outward in a direction defined by whatever platform and controller are being used, if that's defined; otherwise, the ray extends in the same direction the user is pointing in with their index finger, if it were currently outstretched.

The third and final target ray mode is most commonly found on mobile devices like smartphones and tablets. The `screen` mode indicates that the target ray is determined based on the user interacting with the WebXR context by engaging with the screen in some way—most likely by the viewer tapping the screen or dragging the target ray around with their fingers.

#### Target ray space

The `XRSpace` used to describe the position and orientation of the target ray is found in the `targetRaySpace` property. This space's native origin is located at the point from which the target ray is emitted (such as the front tip of the controller, or the end of a gun barrel if the controller is being rendered as a gun, for example), and the space's orientation vector extends outward along the path of the target ray.

You can easily obtain the target ray corresponding to the `targetRaySpace` from within the drawing handler for a given frame using `XRFrame`'s `getPose()` method. The returned `XRPose`'s `transform` is the transform corresponding to the target ray. Thus, for an input controller `primaryInput`:

```js
let targetRayPose = frame.getPose(primaryInput.targetRaySpace, viewerRefSpace);
let targetRayOrigin = targetRayPose.transform.position;
let targetRayVector = targetRayPose.transform.orientation;
```

With this, you now have the point from which the targeting ray is emitted (`targetRayOrigin`) and the direction in which it's pointing (`targetRayVector`), given in the viewer's reference space (`viewerRefSpace`). That's everything you need to be able to draw the targeting ray, determine what's being pointed at, do hit testing, and so forth.

#### Grip space

The input source's `gripSpace` property is an `XRSpace` you can use to render objects so they appear to be held in the viewer's hand.

**Figure: The coordinate system for the left hand's grip space.**
![A diagram showing how the grip space indicates the local coordinate system for the player's hand relative to the world.](/en-US/docs/Web/API/WebXR_Device_API/Inputs/dark_left.svg)
**Figure: The coordinate system for the right hand's grip space.**
![A diagram showing how the grip space indicates the local coordinate system for the player's hand relative to the world.](/en-US/docs/Web/API/WebXR_Device_API/Inputs/dark_right.svg)

The grip space's native origin, located around the center of the player's fist, is (0, 0, 0) within the input source's local coordinate system, while the `XRSpace` specified by `gripSpace` can be used at any time to convert coordinates or vectors from the input source's space into world coordinates (or vice versa).

This means that if you use a 3D model to represent your controller, your player's avatar's hands, or anything else representative of the controller's position in space, the `gripSpace` can be used as the transform matrix that correctly positions and orients the object's model for rendering. To do this, it's necessary to use the transform to convert the grip space to the world coordinate system used by WebGL for rendering purposes.

**Figure: Mapping the grip space to the world coordinate system. The distances *x*, *y*, and *z* together make up the world coordinates (*x*, *y*, z) corresponding to the origin of the grip space *G*.**
![A diagram showing the relationship between the grip space and the world space](/en-US/docs/Web/API/WebXR_Device_API/Inputs/gripspace-on-worldspace.svg)

In the diagram above, we see the grip space, whose origin is located at *G*, at the midpoint of the user's grip on the controller, which is pointing directly away from the user, parallel to the *z* axis. Relative to the origin of the world space, *W*, the grip space's origin is located *x* units to the right, *y* units above, and *z* units farther away. Given the directionality of the axes, the coordinates of the grip space can be expressed in world coordinates as (*x*, *y*, -*z*); *z* is negative since the grip space is farther away along the *z* axis, and is thus in the negative direction.

If the controller were instead positioned to the left of and closer to the user than the world space origin (or possibly behind the user, if the user is located at the origin, although that's an uncomfortable way to hold a controller), the coordinates would have a negative value for *x*, but a positive value for *z*. The value of *y* would still be positive unless the controller was moved below the world space origin.

This is shown in the diagram below, in which the controller is located down and to the left of the world space's origin, with the controller also moved to be closer to us than the origin. As a result, the values of *x* and *y* are both negative, while *z* is positive.

**Figure Mapping a grip space to the world origin when the controller is positioned below and to the left of the world origin, and closer to us than the world origin is.**
![The relationship between another grip space and the world space](/en-US/docs/Web/API/WebXR_Device_API/Inputs/gripspace-on-worldspace-diag.svg)

#### Gamepad record

Each input source has a `gamepad` property which, if not `NULL`, is a `Gamepad` object describing the various controls and widgets available on the controller. If the input device only has the primary movement sensors, a squeeze control, and a button, it may not have a `Gamepad` record. If, however, the `gamepad` is present, you can use it to identify and poll the buttons and axes available on the controller.

While the `Gamepad` record is defined by the Gamepad API specification, it's not actually managed by the Gamepad API, and doesn't function exactly the same way. See Advanced controllers using the gamepad object for more detailed information.

#### Profile strings

Each input source can have zero or more **input profile name** strings, found in the array `profiles`, each of which describes a preferred visual representation of the input source within the 3D world as well as how the input source functions. The use of these profiles is briefly described under Input profiles below.

### Transient input sources

Some devices may create **transient input sources** to be used in tandem with an action which doesn't really come from that device, but is being presented as if it is. For example, if an XR device provides a mode in which the mouse is used to simulate events on the device, a new `XRInputSource` object might be created to represent the simulated input source for the duration of handling the action.

This is necessary because of the separation that's maintained between standard input devices and XR input sources. An artificial source is used to represent the external source for the duration of each transient action.

Managing input sources
----------------------

When multiple input sources are available, you need to be able to obtain information about each one, including its position and orientation, its targeting ray (if applicable to your needs), and details that can help you decide how to present the input source visually, if at all. You also need to be able to determine which input source to use for what activities; for example, if the user has two controllers, which one will be tracked for manipulating UI elements, or will both?

To manage input sources, then, you need to be able to enumerate input sources, examine profile information about each input source, and decide how to use each input controller.

### Enumerating input sources

The WebXR session represented by the `XRSession` object has an `inputSources` property which is a *live* list of the WebXR input devices currently connected to the XR system.

```js
let inputSourceList = xrSession.inputSources;
```

Due to the fact that the contents of the `XRInputSource` objects representing each input source in the list are read-only, changes to these inputs are made by the WebXR system by deleting the source's record and adding a new one to replace it. An `inputsourceschange` event is sent to your `XRSession` whenever one or more of the input sources change, or when an input source is added to or removed from the list.

For example, if you need to keep up with which controller is held in each of the player's hands, you might do something like this:

```js
let inputSourceList = NULL;
let leftHandSource = NULL;
let rightHandSource = NULL;

xrSession.addEventListener("inputsourceschange", (event) => {
  inputSourceList = event.session.inputSources;

  inputSourceList.forEach((source) => {
    switch (source) {
      case "left":
        leftHandSource = source;
        break;
      case "right":
        rightHandSource = source;
        break;
    }
  });
});
```

The `inputsourceschange` event is also fired once when the session's creation callback first completes execution, so you can use it to fetch the input source list as soon as it's available at startup time. The event is delivered as an `XRInputSourcesChangeEvent`, which includes three properties of interest:

`session`

The `XRSession` for which the input sources have changed.

`added`

An array of zero or more `XRInputSource` objects indicating the input sources that have been newly added to the XR system.

`removed`

An array of zero or more `XRInputSource` objects indicating any input sources that have been removed from the XR system.

### Identifying the input's profile

Each input source has a `profiles` property, which contains a live list of the WebXR input profiles which apply to the input source, in order of specificity from most-specific to least.

In order to do anything meaningful involving scanning of profiles beyond basic identification of features, you may need to import the JSON profile database from the WebXR Input Profiles Registry.

See Input profiles for more specific details on working with input profiles.

### Choosing the primary controller

In order to avoid having problems introduced by multiple controllers trying to inadvertently manipulate the UI at the same time, your app may need to have a "primary" controller. Not only would this controller then take the responsibility of clicking through the user interface of your app, but it would also be considered the "main hand," while other controllers would then be off-hand or additional controllers.

**Note:** This doesn't mean your app *needs* to decide upon a primary controller. But if it does, these strategies may help.

There are a few ways you can decide upon a primary controller. We'll look at three.

#### Handedness

The most direct way to decide which controller is primary is to have a user-definable "Handedness" preference that the user sets to indicate which of their hands is dominant. You would then look at each input source and find one matching this, if available, falling back to another controller if no controller is in that hand.

```js
const primaryInputSource =
  xrSession.inputSources.find((src) => src.handedness === user.handedness) ??
  xrSession.inputSources[0];
```

This snippet of code starts by assuming that the first input source is the primary, but then looks for one whose `handedness` matches the one specified in the `user` object. If it matches, that input source is selected as the primary.

#### First-used

Another option is to use the first input the user triggers the select action on. The code below starts by assuming the first input source is the primary, then establishes a handler for the `select` event that records the event's source as the primary input source. Then the `select` event handler is replaced with the function `realSelectHandler()`, which will be used for handling all future `select` events. Then we pass the event through to `realSelectHandler()` to allow the event to be processed as normal.

```js
let primaryInputSource = xrSession.inputSources[0];

xrSession.onselect = (event) => {
  primaryInputSource = event.inputSource;
  xrSession.onselect = realSelectHandler;
  return realSelectHandler(event);
};
```

The effect is that we set the primary input source the first time a `select` event is received, regardless of which input source it comes from, handle the event as normal from there, and from then on handle the events as usual without any further worries about which input source is primary.

#### User-selected

The most complex way to determine a primary input source is highly flexible but can require a great deal of work to implement. In this scenario, you iterate over the list of input sources and their profiles to gather information about each input source, then present a user interface describing each input, allowing the user to assign uses to each of them. Doing this well could be a big job, but it might be useful for complex apps that may involve several user inputs.

Much of the information you need in order to implement this can be found in the section on Input profiles, below. Details are beyond the scope of this article, however.

Input profiles
--------------

As mentioned above, each input source has a list of input profile names that correspond to a set of information describing that input source and how it can be used. Those names are found in the input source's `profiles` property, and the official registry of these profile strings is maintained in the WebXR Input Profiles Registry on GitHub.

For example, the `generic-trigger-squeeze-touchpad` profile name can be used to locate the following JSON profile data by locating the `profileId` field that has the value `generic-trigger-squeeze-touchpad`.

```json
{
  "profileId": "generic-trigger-squeeze-touchpad",
  "fallbackProfileIds": [],
  "layouts": {
    "left-right-none": {
      "selectComponentId": "xr-standard-trigger",
      "components": {
        "xr-standard-trigger": { "type": "trigger" },
        "xr-standard-squeeze": { "type": "squeeze" },
        "xr-standard-touchpad": { "type": "touchpad" }
      },
      "gamepad": {
        "mapping": "xr-standard",
        "buttons": [
          "xr-standard-trigger",
          "xr-standard-squeeze",
          "xr-standard-touchpad"
        ],
        "axes": [
          { "componentId": "xr-standard-touchpad", "axis": "x-axis" },
          { "componentId": "xr-standard-touchpad", "axis": "y-axis" }
        ]
      }
    }
  }
}
```

This is a controller which regardless of which hand it's in (and even if it's not currently associated with a specific hand) has three components: a standard trigger, a standard squeeze input, and a touchpad. According to the `selectComponentId` property, the `xr-standard-trigger` component is the one used to perform a primary action.

Additionally, the `gamepad` object maps those inputs to the gamepad, assigning the trigger, squeeze, and touchpad tap to the input source's button list and the touchpad's "axes" to the axis list.

The list in `profiles` is in order of reverse specificity; that is, the most precise description is first, and the least precise description is last. The first entry in the list is typically indicative of the precise model of the controller, or of a model with which the controller is compatible.

For example, entry 0 in `profiles` for an Oculus Touch controller is `oculus-touch`. The next entry is `generic-trigger-squeeze-thumbstick`, indicating a generic device with a trigger, a squeeze control, and a thumbstick. While the Oculus Touch controller actually has a thumbpad rather than a thumbstick, the overall description is "close enough" that the details within the profile matching the name will let the controller be interpreted usefully.

Actions
-------

In WebXR, an **action** is a special type of event that's triggered by the user activating a special button on the controller. Any additional buttons (as well as things like axis controllers—joysticks, for example—and the like) are managed solely through the `XRInputSource` property `gamepad`. See Advanced controllers using the gamepad object below for further details on supporting these extra controls and buttons.

The **primary action** is the action that is triggered when the user engages the main control element that serves a special purpose. There are currently two types of primary action:

* The **primary action** is the action which is activated when the user activates the primary or "select" input on their controller. This input may be a button, trigger, trackpad tap or click, a voice command, or special hand gesture, or possibly some other form of input. For example, on a hand controller with a trackpad that can be clicked on, a trigger control, as well as back and "menu" buttons, clicking the trackpad is likely to be the primary action. Some controllers might have a button labeled "select." On a gamepad-style controller, the "A" button is likely to be the primary action.
* The **primary squeeze action** is the action which is engaged when the user squeezes the controller. This "squeezing" may be detected by literally using a pressure sensor in the controller, or may be simulated using a trigger, hand gesture, or other mechanism. For example, if the input controller is a haptic glove, it might report that the primary squeeze action has occurred when the user makes and clenches their fist.

While a given input source can have only one primary action and one primary squeeze action, there may be more than one control on the input device configured to trigger each primary action. For example, the user might have their controller set up so that both tapping and clicking the trackpad generate a primary action.

These types of input actions are described in more detail below.

### Primary action

Each input source should define a **primary action**. A primary action (which will sometimes be shortened to "select action") is a platform-specific action which responds to the user manipulating it by delivering, in order, the events `selectstart`, `select`, and `selectend`. Each of these events is of type `XRInputSourceEvent`.

**Note:** If an input source doesn't have a primary action, the input source is considered to be an **auxiliary input source**.

When the user points a device along a target ray in your 3D space and then triggers a select action, the following events are sent to the active `XRSession`:

1. A `selectstart` event, indicating that the user has performed the activity that begins the primary action. This may be a gesture, pressing a button, or the like.
2. If the primary action ends successfully (for example, due to the user releasing the button or trigger), rather than because of an error, the `select` event is sent.
3. After the `select` event is sent *or* if the controller on which the action is being performed is disconnected or otherwise becomes unavailable, the `selectend` event is sent.

Generally speaking, the `selectstart` and `selectend` events tell you when you might want to display something to the user indicating that the primary action is going on. This might be drawing a controller with the activated button in a new color, or showing the targeted object being grabbed and moved around, starting when `selectstart` arrives and stopping when `selectend` is received.

The `select` event, on the other hand, is the event that tells your code that the user has completed the action they want to complete. This might be as simple as throwing an object or pulling the trigger of a gun in a game, or as involved as placing an object that they were dragging around the world back down again in a new location.

If your primary action is a simple trigger action and you don't need to animate anything while the trigger is engaged, you can ignore the `selectstart` and `selectend` events and act on the `select` event.

```js
xrSession.addEventListener("select", (event) => {
  let inputSource = event.inputSource;
  let frame = event.frame;

  /\* handle the event \*/
});
```

Some actions may send these events very quickly, one after the other. The time that elapses between these events depends on both the hardware apparatus that causes the action as well as the software drivers that interpret the hardware action and turn it into a series of events. Do not presume these events will happen with any particular amount of time between them.

For example, if the hardware that causes the primary action to occur is a button, you would receive `selectstart` when the user presses the button, then `select` and `selectend` when the user releases it.

There are a number of examples showing how to handle `select` events throughout the documentation, such as in the section on Targeting and the targeting ray elsewhere in this article.

### Primary squeeze action

A **primary squeeze action** is a platform-specific action which sends the `XRSession` `squeezestart`, `squeezeend`, and `squeeze` events. This is typically generated by the user squeezing the controller, making a hand gesture that mimes grabbing something, or using (squeezing) a trigger.

The sequence of events is identical to those sent by the primary action, save for the name of each event:

1. An `squeezestart` event is sent to the `XRSession`, indicating that the user has begun a squeeze action.
2. If the primary squeeze action ends successfully, the session is sent a `squeeze` event.
3. Then, a `squeezeend` event is sent to indicate that the squeeze action is no longer underway. This is sent whether the squeeze action succeeded or not.

Two common uses for the primary squeeze action are to grasp and/or pick up objects in the 3D world and to squeeze a trigger to fire a weapon in a game or simulation.

#### Example

This sample code shows a set of squeeze event handlers that implement these events to manage picking up and holding objects from the scene. The code presumes the existence of an `avatar` object representing the character, as used in several other examples on this page, as well as the `pickUpObject()` and `dropObject()` functions, which handle transferring an object from the world to a particular hand and releasing an object from the hand and placing it back into the world.

##### Picking up an object: handling squeezestart events

```js
xrSession.addEventListener("squeezestart", (event) => {
  const targetRaySpace = event.inputSource.targetRaySpace;
  const hand = event.inputSource.handedness;

  let targetRayPose = event.frame.getPose(targetRaySpace, viewerRefSpace);
  if (!targetRayPose) {
    return;
  }

  let targetRayTransform = targetRayPose.transform;
  let targetObject = findTargetObject(targetRayTransform);

  if (targetObject) {
    if (avatar.heldObject[hand]) {
      dropObject(hand);
    }
    pickUpObject(targetObject, hand);
  }
});
```

The `squeezestart` event is handled by getting those pose and transform as usual, and getting the input source's `handedness` into the local constant `hand`. We'll use that to map the hand to the object held in that hand.

The code then identifies the targeted object, then, if an object is found along the targeting ray, picks it up. Picking up an object involves first looking to see if the hand represented by dropping any object already being held in the hand represented by `avatar.heldObject[hand]`. If an object is already held in that hand, it's dropped by calling the `dropObject()` function.

Then `pickUpObject()` is called, specifying the targeted object as the object to remove from the scene and place into the specified `hand`. `pickUpObject()` also records the original position of the object so it can be returned to that spot if the squeeze is canceled or aborted.

##### Dropping the object: the squeeze event handler

The `squeeze` event is received when the user ends the squeeze action by releasing their grip. In this example, we interpret that as releasing the currently-held object, dropping it into the scene at the targeted location.

This code presumes the existence of additional functions `findTargetPosition()`, which follows the target ray until it collides with something, then returns the coordinates at which the collision occurred, and `putObject()`, which places the object held in the specified `hand` at the given position, removing it from the hand.

```js
xrSession.addEventListener("squeeze", (event) => {
  const targetRaySpace = event.inputSource.targetRaySpace;
  const hand = event.inputSource.handedness;

  let targetRayPose = event.frame.getPose(targetRaySpace, viewerRefSpace);
  if (!targetRayPose) {
    return;
  }

  let targetRayTransform = targetRayPose.transform;
  let targetPosition = findTargetPosition(targetRayTransform);

  if (targetPosition) {
    if (avatar.heldObject[hand]) {
      putObject(hand, targetPosition);
      avatar.heldObject[hand] = null;
    }
  }
});
```

As in the `squeezestart` handler, this begins by collecting information needed about the event, including the hand that's dropping an object and the target ray's transform. The target ray transform is passed into the presumed `findTargetPosition()` function to get the coordinates at which to position the dropped object.

With the position in hand, we can then drop the object by calling the `putObject()` function, which takes as its inputs the `hand` and the destination position. This function's job is to remove the object from the specified hand and add it back to the scene, with its position set to place it atop the coordinates returned by `findTargetPosition()`.

##### Canceling the squeeze in the squeezeend handler

The `squeezeend` event is received after the squeeze is complete, even if it fails. We handle it by returning the currently-held object to where it was when it was picked up.

```js
xrSession.addEventListener("squeezeend", (event) => {
  const targetRaySpace = event.inputSource.targetRaySpace;
  const hand = event.inputSource.handedness;

  if (avatar.heldObject[hand]) {
    returnObject(hand);
    avatar.heldObject[hand] = null;
  }
});
```

Here, the `returnObject()` function is assumed to be one which knows how to return the object held in the specified `hand` to its initial position, as it was recorded by `pickUpObject()` in the `squeezestart` event handler.

Here, the `returnObject()` function is assumed to be one which knows how to return the object held in the specified `hand` to its initial position, as it was recorded by `pickUpObject()` in the `squeezestart` event handler.

### Transient actions

If an XR device uses the mouse to simulate a controller when in the `inline` mode, the approximate sequence of things takes place:

1. The user presses the mouse button while inside the `<canvas>` presenting the WebXR scene.
2. The mouse event is captured by the XR device's driver.
3. The device creates a new `XRInputSource` to represent the simulated XR input source. The `targetRayMode` is set to `screen`, and the other information is filled out as appropriate. This new input source is temporarily added to the list that's returned by the `XRSession` property `inputSources`.
4. The browser delivers `pointerdown` events corresponding to the action.
5. A primary action is generated and sent to the app in the form of a `selectstart` event, with its source set to the new `XRInputSource`. Or, if the mouse is being used as an off-hand or secondary controller, an auxiliary action is sent instead.
6. When the user releases the mouse button, the `select` event is sent to the `XRSession`, then the DOM receives a `click` event. The session then receives the `selectend` event indicating the completion of the action.
7. When the action is completed, the browser deletes the transient input source, and any appropriate `pointerup` events are sent.

Thus, the transient input source is indeed transient—it exists only for the duration of handling the input and will as such not be listed in the input source list.

Facing and targeting
--------------------

**Facing** is the direction in which the viewer is looking. This isn't provided using an input source. Instead, it's obtained using the `XRPose` obtained from the current animation frame's `XRFrame.getViewerPose()` method. The rotation component of the viewer pose's transform matrix is the viewer's facing direction.

You can learn more about how to use this the viewer pose to determine the facing direction in the article Viewpoints and viewers.

**Targeting** is the act of the user pointing in a particular direction using an input source. The input source's `targetRaySpace` is an `XRSpace` (and actually probably an `XRReferenceSpace`) which can be used to determine the orientation of the target ray relative to the viewer's facing direction.

This may or may not involve actually pointing at a particular object within the 3D world; you have to determine this yourself using hit testing—that is, checking to see if the target ray intersects with any objects in your scene.

### Targeting and the targeting ray

The targeting ray, which is a ray whose origin is located at the origin of the targeting ray space and points in the direction that the user is pointing the controller device. The targeting ray is defined using an `XRSpace` whose origin is located at the targeting ray's source (typically the outward-facing end of the controller or its representation in the 3D world), and whose orientation has -Z extending outward from the controller in the same direction as the `XRInputSource`'s `gripSpace`.

This space is found in the input source's `targetRaySpace` property. It can be used to determine the direction the controller is pointing and to determine the origin and orientation of the target ray. That can be accomplished by doing something like the following example, which implements a `select` event handler that needs this information. As usual, this code is assuming the use of glMatrix to perform the matrix and vector math:

```js
xrSession.addEventListener("select", (event) => {
  const targetRaySpace = event.inputSource.targetRaySpace;

  let targetRayPose = event.frame.getPose(targetRaySpace, viewerRefSpace);
  if (!targetRayPose) {
    return;
  }

  let targetRayTransform = targetRayPose.transform;
  let targetObject = findTargetObject(targetRayTransform);

  if (targetObject) {
    /\* do stuff with the targeted object \*/
  }
});
```

This obtains the origin of the targeting ray in the vector `targetSourcePoint` and the ray's orientation in the quaternion `targetDirection`. With either the

This starts by fetching the target ray's space into the local constant `targetRaySpace`. This is in turn used when calling the `XRFrame` method `getPose()` to create an `XRPose` object representing the position and orientation of the target ray in the viewer's reference space, `viewerRefSpace`. If this is `null`, the event handler returns without doing anything further.

The target ray's transform is obtained from the pose's `transform` property and stored in the local `targetRayTransform`. This is in turn used (in this case through a function called `findTargetObject()`) to find the first object the ray intersects with. If the targeting ray does intersect with an object in the scene, we can do whatever we need to with it.

If you need to peel out the actual position of the target ray's origin and the ray's directionality, you can do so like this:

```js
const targetRayOrigin = vec3.create();
const targetRayDirection = quat.create();
mat4.getTranslation(targetRayOrigin, viewerRefSpace);
mat4.getRotation(targetRayDirection, viewerRefSpace);
```

To determine what object is targeted, follow the targeting ray until it intersects an object. This process is called **hit testing** or **collision detection**. The approach you take to hit testing depends very much on the specific needs of your app. The first question is: are you detecting collisions with virtual objects or terrain, real-world objects or terrain, or both?

In any case, to identify the targeted object, you need to determine if the ray specified by the `XRInputSource` property `targetRaySpace` intersects any objects in the scene, whether they're virtual or real-world.

See Targeting and hit detection for a more detailed look at what's involved.

### Presenting hand-held objects

An input source's `gripSpace` property identifies an `XRSpace` describing the origin and orientation to use when rendering an object so that it appears to be held in the same hand as its input source. This space is intended to be used when drawing a model of the handheld WebXR input controller represented by the object `XRInputSource`, but can just as well be used to draw any object, such as a ball, a tool, or a weapon. We covered the grip space above, but let's see how it can be used to draw objects representative of the hand or in the hand.

Since the origin of the grip space is located at the center of the hand's grip, you can use it as a starting point for rendering your object. Apply any needed offset transform to shift the origin to the start point for rendering your object as you apply any rotation needed to orient your model correctly to align with the grip space's orientation.

Advanced controllers using the gamepad object
---------------------------------------------

An `XRInputSource` has a `gamepad` property whose value, if not `null`, is a `Gamepad` object which provides access to gamepad-style buttons, axis controllers (such as joysticks or thumbpads), and so forth. This may include the same buttons that trigger the standard `XRInputSource` actions, but may include any number of additional buttons and controls.

**Note:** While `Gamepad` is defined by the Gamepad API, it is not managed by the Gamepad API, so you must not attempt to use any Gamepad API methods with it. The object type is reused as a convenience.

If the value of `gamepad` is `null`, the input source doesn't define any controls using the `Gamepad` record, either because it doesn't support it or because it doesn't have any added controls on it.

This `gamepad` object is not only used to obtain access to specialty buttons, trackpads, and so forth, but also provides a way to more directly access and monitor the controls that serve as the primary select and squeeze inputs, since these are included in its `buttons` list.

Because this use of the `Gamepad` interface is a convenience rather than a true application of the Gamepad API, there are several differences between how it's used with WebXR and how it's used in Gamepad API applications. The most notable—but not the only—difference is that WebXR adds the `xr-standard` gamepad mapping, see the `XRInputSource.gamepad` property for additional differences. This gamepad mapping defines how the controls on a typical one-hand handheld VR controller are mapped to gamepad controls.

Incorporating input from non-WebXR sources
------------------------------------------

Sometimes, you need to have a way to let the user provide input using controllers which are external to WebXR. Most commonly, these inputs are from keyboards and mice, but you could also use non-XR gamepad devices, network inputs, or other sources of data to simulate user controls. While WebXR offers no support for directly interfacing these input devices with the XR scene, you can collect the input data yourself and apply it yourself.

Assuming inputs are used to control an avatar within the simulation, which is the most common use case, WebXR inputs are used to affect the avatar in the following ways, using data collected from the non-XR input device:

Position

The avatar's position is changed by applying a delta to the previously known position, then replacing the avatar's reference space with a new one whose transform reflects the new position.

Orientation

The avatar's orientation or facing direction is changed by applying a delta to its rotation around the three axes, updating its orientation vector, then recomputing its reference space.

Action

The avatar performs an action, such as using an object or weapon, jumping, or any other activity not related to basic movement and rotation.

Some inputs are instead used to control the application rather than the avatar. For example, a button might open an options menu used to configure the application. While that menu is open, inputs that would otherwise control the avatar might be instead applied to controlling the menu's interface.

### Using keyboard and mouse events

Capturing input from the keyboard and mouse is done just like it is in any web application. Set up handlers for the events that you need to handle in order to obtain the inputs you want. It's what you do with those inputs that's interesting.

Imagine an `avatar` object which we'll use to track information about the avatar and its worldview. We want the player to be able to use the `W`, `A`, `S`, and `D` keys to move forward, left, backward, and right. Because we are managing the avatar's position as defined by the keyboard and mouse in addition to anything the XR hardware might be doing, we need to maintain that information separately and apply it as a transform before rendering the avatar (or the world from the avatar's point of view).

To accomplish that, we include in the `avatar` object a `posDelta` property, of type `DOMPoint`, which includes the offsets to apply to all three axes in order to adjust the avatar's position (the viewer pose's reference space's origin) to include movement and rotation from the keyboard and mouse.

The corresponding code for keyboard input might look something like this:

```js
document.addEventListener("keydown", (event) => {
  switch (event.key) {
    case "a":
    case "A":
      avatar.posDelta.x -= ACCEL\_X;
      break;
    case "d":
    case "D":
      avatar.posDelta.x += ACCEL\_X;
      break;
    case "w":
    case "W":
      avatar.posDelta.y += ACCEL\_Y;
      break;
    case "s":
    case "S":
      avatar.posDelta.y -= ACCEL\_Y;
      break;
    default:
      break;
  }
});
```

This is a simple example, where acceleration is constant and not particularly realistic. You can enhance this greatly by applying some knowledge of physics to have the acceleration change over time based on how long the key is held down and other factors.

### Applying inputs to the scene

Now that we have the deltas that need to be applied to the position and orientation—in our example, in the `posDelta` and `orientDelta` properties of our `avatar` object—we can write code to apply those changes. Since we're already rendering the scene on a schedule, we can just add the code to apply these changes there, along with preparing and drawing the scene.

```js
function drawFrame(time, frame) {
  applyExternalInputs(avatar);
  let pose = frame.getViewerPose(avatar.referenceSpace);

  animationFrameRequest = session.requestAnimationFrame(drawFrame);

  /\* draw the frame here \*/
}
```

The `drawFrame()` function shown here is the callback invoked when it's time to draw the frame, as established by calling the `XRSession` method `requestAnimationFrame()`. It calls a function `applyExternalInputs()` which we'll define momentarily; it takes the `avatar` object and uses its information to update the avatar's reference frame.

After that, everything proceeds as usual, getting the viewer's pose from the updated reference frame, requesting the next frame callback through `requestAnimationFrame()`, and then continuing to set up WebGL and draw the scene. The drawing and other related code can be found in the example Movement, orientation, and motion.

The `applyExternalInputs()` method takes the `avatar` object replaces its `referenceSpace` property with a new reference space that incorporates the updated deltas.

```js
function applyExternalInputs(avatar) {
  if (!avatar.posDelta.x && !avatar.posDelta.y && !avatar.posDelta.z) {
    return; // Player hasn't moved with keyboard
  }

  let newTransform = new XRRigidTransform({
    x: avatar.posDelta.x,
    y: avatar.posDelta.y,
    z: avatar.posDelta.z,
  });
  avatar.referenceSpace =
    avatar.referenceSpace.getOffsetReferenceSpace(newTransform);
}
```

See also
--------

* Targeting and hit detection
* Geometry and reference spaces in WebXR
* Spatial tracking in WebXR
* Rendering and the WebXR frame animation callback
* Viewpoints and viewers: Simulating cameras in WebXR
* Movement, orientation, and motion: A WebXR example