Web audio spatialization basics - Web APIs

Web audio spatialization basics
===============================

 As if its extensive variety of sound processing (and other) options wasn't enough, the Web Audio API also includes facilities to allow you to emulate the difference in sound as a listener moves around a sound source, for example panning as you move around a sound source inside a 3D game.
 The official term for this is **spatialization**, and this article will cover the basics of how to implement such a system.

Basics of spatialization
------------------------

 In Web Audio, complex 3D spatializations are created using the `PannerNode`, which in layman's terms is basically a whole lotta cool maths to make audio appear in 3D space.
 Think sounds flying over you, creeping up behind you, moving across in front of you.
 That sort of thing.

 It's really useful for WebXR and gaming.
 In 3D spaces, it's the only way to achieve realistic audio. Libraries like three.js and A-frame harness its potential when dealing with sound.
 It's worth noting that you don't *have* to move sound within a full 3D space either — you could stick with just a 2D plane, so if you were planning a 2D game, this would still be the node you were looking for.

**Note:** There's also a `StereoPannerNode` designed to deal with the common use case of creating simple left and right stereo panning effects.
 This is much simpler to use, but obviously nowhere near as versatile.
 If you just want a simple stereo panning effect, our StereoPannerNode example (see source code) should give you everything you need.

3D boombox demo
---------------

 To demonstrate 3D spatialization we've created a modified version of the boombox demo we created in our basic Using the Web Audio API guide.
 See the 3D spatialization demo live (and see the source code also).

![A simple UI with a rotated boombox and controls to move it left and right and in and out, and rotate it.](/en-US/docs/Web/API/Web_Audio_API/Web_audio_spatialization_basics/web-audio-spatialization.png)

 The boombox sits inside a room (defined by the edges of the browser viewport), and in this demo, we can move and rotate it with the provided controls.
 When we move the boombox, the sound it produces changes accordingly, panning as it moves to the left or right of the room, or becoming quieter as it is moved away from the user or is rotated so the speakers are facing away from them, etc.
 This is done by setting the different properties of the `PannerNode` object instance in relation to that movement, to emulate spacialization.

**Note:** The experience is much better if you use headphones, or have some kind of surround sound system to plug your computer into.

Creating an audio listener
--------------------------

 So let's begin! The `BaseAudioContext` (the interface the `AudioContext` is extended from) has a `listener` property that returns an `AudioListener` object.
 This represents the listener of the scene, usually your user.
 You can define where they are in space and in which direction they are facing.
 They remain static. The `pannerNode` can then calculate its sound position relative to the position of the listener.

Let's create our context and listener and set the listener's position to emulate a person looking into our room:

```js
const AudioContext = window.AudioContext || window.webkitAudioContext;
const audioCtx = new AudioContext();
const listener = audioCtx.listener;

const posX = window.innerWidth / 2;
const posY = window.innerHeight / 2;
const posZ = 300;

listener.positionX.value = posX;
listener.positionY.value = posY;
listener.positionZ.value = posZ - 5;
```

We could move the listener left or right using `positionX`, up or down using `positionY`, or in or out of the room using `positionZ`. Here we are setting the listener to be in the middle of the viewport and slightly in front of our boombox. We can also set the direction the listener is facing. The default values for these work well:

```js
listener.forwardX.value = 0;
listener.forwardY.value = 0;
listener.forwardZ.value = -1;
listener.upX.value = 0;
listener.upY.value = 1;
listener.upZ.value = 0;
```

 The forward properties represent the 3D coordinate position of the listener's forward direction (e.g. the direction they are facing in), while the up properties represent the 3D coordinate position of the top of the listener's head.
 These two together can nicely set the direction.

Creating a panner node
----------------------

Let's create our `PannerNode`. This has a whole bunch of properties associated with it. Let's take a look at each of them:

 To start we can set the `panningModel`.
 This is the spacialization algorithm that's used to position the audio in 3D space. We can set this to:

`equalpower` — The default and the general way panning is figured out

`HRTF` — This stands for 'Head-related transfer function' and looks to take into account the human head when figuring out where the sound is.

Pretty clever stuff. Let's use the `HRTF` model!

```js
const panningModel = "HRTF";
```

 The `coneInnerAngle` and `coneOuterAngle` properties specify where the volume emanates from.
 By default, both are 360 degrees.
 Our boombox speakers will have smaller cones, which we can define.
 The inner cone is where gain (volume) is always emulated at a maximum and the outer cone is where the gain starts to drop away.
 The gain is reduced by the value of the `coneOuterGain`.
 Let's create constants that store the values we'll use for these parameters later on:

```js
const innerCone = 60;
const outerCone = 90;
const outerGain = 0.3;
```

The next parameter is `distanceModel` — this can only be set to `linear`, `inverse`, or `exponential`. These are different algorithms, which are used to reduce the volume of the audio source as it moves away from the listener. We'll use `linear`, as it is simple:

```js
const distanceModel = "linear";
```

 We can set a maximum distance (`maxDistance`) between the source and the listener — the volume will not be reduced anymore if the source moves further away from this point.
 This can be useful, as you may find you want to emulate distance, but volume can drop out and that's actually not what you want.
 By default, it's 10,000 (a unitless relative value). We can keep it as this:

```js
const maxDistance = 10000;
```

 There's also a reference distance (`refDistance`), which is used by the distance models.
 We can keep that at the default value of `1` as well:

```js
const refDistance = 1;
```

 Then there's the roll-off factor (`rolloffFactor`) — how quickly does the volume reduce as the panner moves away from the listener.
 The default value is 1; let's make that a bit bigger to exaggerate our movements.

```js
const rollOff = 10;
```

 Now we can start setting our position and orientation of our boombox.
 This is a lot like how we did it with our listener.
 These are also the parameters we're going to change when the controls on our interface are used.

```js
const positionX = posX;
const positionY = posY;
const positionZ = posZ;

const orientationX = 0.0;
const orientationY = 0.0;
const orientationZ = -1.0;
```

 Note the minus value on our z orientation — this sets the boombox to face us.
 A positive value would set the sound source facing away from us.

Let's use the relevant constructor for creating our panner node and pass in all those parameters we set above:

```js
const panner = new PannerNode(audioCtx, {
  panningModel,
  distanceModel,
  positionX,
  positionY,
  positionZ,
  orientationX,
  orientationY,
  orientationZ,
  refDistance,
  maxDistance,
  rolloffFactor: rollOff,
  coneInnerAngle: innerCone,
  coneOuterAngle: outerCone,
  coneOuterGain: outerGain,
});
```

Moving the boombox
------------------

 Now we're going to move our boombox around our 'room'. We've got some controls set up to do this.
 We can move it left and right, up and down, and back and forth; we can also rotate it.
 The sound direction is coming from the boombox speaker at the front, so when we rotate it, we can alter the sound's direction — i.e. make it project to the back when the boombox is rotated 180 degrees and facing away from us.

 We need to set up a few things for the interface.
 First, we'll get references to the elements we want to move, then we'll store references to the values we'll change when we set up CSS transforms to actually do the movement.
 Finally, we'll set some bounds so our boombox doesn't move too far in any direction:

```js
const moveControls = document
  .querySelector("#move-controls")
  .querySelectorAll("button");
const boombox = document.querySelector(".boombox-body");

// the values for our CSS transforms
const transform = {
  xAxis: 0,
  yAxis: 0,
  zAxis: 0.8,
  rotateX: 0,
  rotateY: 0,
};

// set our bounds
const topBound = -posY;
const bottomBound = posY;
const rightBound = posX;
const leftBound = -posX;
const innerBound = 0.1;
const outerBound = 1.5;
```

Let's create a function that takes the direction we want to move as a parameter, and both modifies the CSS transform and updates the position and orientation values of our panner node properties to change the sound as appropriate.

 To start with let's take a look at our left, right, up and down values as these are pretty straightforward.
 We'll move the boombox along these axis and update the appropriate position.

```js
function moveBoombox(direction) {
  switch (direction) {
    case "left":
      if (transform.xAxis > leftBound) {
        transform.xAxis -= 5;
        panner.positionX.value -= 0.1;
      }
      break;
    case "up":
      if (transform.yAxis > topBound) {
        transform.yAxis -= 5;
        panner.positionY.value -= 0.3;
      }
      break;
    case "right":
      if (transform.xAxis < rightBound) {
        transform.xAxis += 5;
        panner.positionX.value += 0.1;
      }
      break;
    case "down":
      if (transform.yAxis < bottomBound) {
        transform.yAxis += 5;
        panner.positionY.value += 0.3;
      }
      break;
  }
}
```

It's a similar story for our move in and out values too:

```js
case 'back':
  if (transform.zAxis > innerBound) {
    transform.zAxis -= 0.01;
    panner.positionZ.value += 40;
  }
  break;
case 'forward':
  if (transform.zAxis < outerBound) {
    transform.zAxis += 0.01;
    panner.positionZ.value -= 40;
  }
  break;
```

 Our rotation values are a little more involved, however, as we need to move the sound *around*.
 Not only do we have to update two axis values (e.g. if you rotate an object around the x-axis, you update the y and z coordinates for that object), but we also need to do some more maths for this.
 The rotation is a circle and we need `Math.sin` and `Math.cos` to help us draw that circle.

Let's set up a rotation rate, which we'll convert into a radian range value for use in `Math.sin` and `Math.cos` later, when we want to figure out the new coordinates when we're rotating our boombox:

```js
// set up rotation constants
const rotationRate = 60; // bigger number equals slower sound rotation

const q = Math.PI / rotationRate; //rotation increment in radians
```

We can also use this to work out degrees rotated, which will help with the CSS transforms we will have to create (note we need both an x and y-axis for the CSS transforms):

```js
// get degrees for CSS
const degreesX = (q \* 180) / Math.PI;
const degreesY = (q \* 180) / Math.PI;
```

Let's take a look at our left rotation as an example. We need to change the x orientation and the z orientation of the panner coordinates, to move around the y-axis for our left rotation:

```js
case 'rotate-left':
  transform.rotateY -= degreesY;

  // 'left' is rotation about y-axis with negative angle increment
  z = panner.orientationZ.value\*Math.cos(q) - panner.orientationX.value\*Math.sin(q);
  x = panner.orientationZ.value\*Math.sin(q) + panner.orientationX.value\*Math.cos(q);
  y = panner.orientationY.value;

  panner.orientationX.value = x;
  panner.orientationY.value = y;
  panner.orientationZ.value = z;
  break;
```

This *is* a little confusing, but what we're doing is using sin and cos to help us work out the circular motion the coordinates need for the rotation of the boombox.

We can do this for all the axes. We just need to choose the right orientations to update and whether we want a positive or negative increment.

```js
case 'rotate-right':
  transform.rotateY += degreesY;
  // 'right' is rotation about y-axis with positive angle increment
  z = panner.orientationZ.value\*Math.cos(-q) - panner.orientationX.value\*Math.sin(-q);
  x = panner.orientationZ.value\*Math.sin(-q) + panner.orientationX.value\*Math.cos(-q);
  y = panner.orientationY.value;
  panner.orientationX.value = x;
  panner.orientationY.value = y;
  panner.orientationZ.value = z;
  break;
case 'rotate-up':
  transform.rotateX += degreesX;
  // 'up' is rotation about x-axis with negative angle increment
  z = panner.orientationZ.value\*Math.cos(-q) - panner.orientationY.value\*Math.sin(-q);
  y = panner.orientationZ.value\*Math.sin(-q) + panner.orientationY.value\*Math.cos(-q);
  x = panner.orientationX.value;
  panner.orientationX.value = x;
  panner.orientationY.value = y;
  panner.orientationZ.value = z;
  break;
case 'rotate-down':
  transform.rotateX -= degreesX;
  // 'down' is rotation about x-axis with positive angle increment
  z = panner.orientationZ.value\*Math.cos(q) - panner.orientationY.value\*Math.sin(q);
  y = panner.orientationZ.value\*Math.sin(q) + panner.orientationY.value\*Math.cos(q);
  x = panner.orientationX.value;
  panner.orientationX.value = x;
  panner.orientationY.value = y;
  panner.orientationZ.value = z;
  break;
```

 One last thing — we need to update the CSS and keep a reference of the last move for the mouse event.
 Here's the final `moveBoombox` function.

```js
function moveBoombox(direction, prevMove) {
  switch (direction) {
    case "left":
      if (transform.xAxis > leftBound) {
        transform.xAxis -= 5;
        panner.positionX.value -= 0.1;
      }
      break;
    case "up":
      if (transform.yAxis > topBound) {
        transform.yAxis -= 5;
        panner.positionY.value -= 0.3;
      }
      break;
    case "right":
      if (transform.xAxis < rightBound) {
        transform.xAxis += 5;
        panner.positionX.value += 0.1;
      }
      break;
    case "down":
      if (transform.yAxis < bottomBound) {
        transform.yAxis += 5;
        panner.positionY.value += 0.3;
      }
      break;
    case "back":
      if (transform.zAxis > innerBound) {
        transform.zAxis -= 0.01;
        panner.positionZ.value += 40;
      }
      break;
    case "forward":
      if (transform.zAxis < outerBound) {
        transform.zAxis += 0.01;
        panner.positionZ.value -= 40;
      }
      break;
    case "rotate-left":
      transform.rotateY -= degreesY;

      // 'left' is rotation about y-axis with negative angle increment
      z =
        panner.orientationZ.value \* Math.cos(q) -
        panner.orientationX.value \* Math.sin(q);
      x =
        panner.orientationZ.value \* Math.sin(q) +
        panner.orientationX.value \* Math.cos(q);
      y = panner.orientationY.value;

      panner.orientationX.value = x;
      panner.orientationY.value = y;
      panner.orientationZ.value = z;
      break;
    case "rotate-right":
      transform.rotateY += degreesY;
      // 'right' is rotation about y-axis with positive angle increment
      z =
        panner.orientationZ.value \* Math.cos(-q) -
        panner.orientationX.value \* Math.sin(-q);
      x =
        panner.orientationZ.value \* Math.sin(-q) +
        panner.orientationX.value \* Math.cos(-q);
      y = panner.orientationY.value;
      panner.orientationX.value = x;
      panner.orientationY.value = y;
      panner.orientationZ.value = z;
      break;
    case "rotate-up":
      transform.rotateX += degreesX;
      // 'up' is rotation about x-axis with negative angle increment
      z =
        panner.orientationZ.value \* Math.cos(-q) -
        panner.orientationY.value \* Math.sin(-q);
      y =
        panner.orientationZ.value \* Math.sin(-q) +
        panner.orientationY.value \* Math.cos(-q);
      x = panner.orientationX.value;
      panner.orientationX.value = x;
      panner.orientationY.value = y;
      panner.orientationZ.value = z;
      break;
    case "rotate-down":
      transform.rotateX -= degreesX;
      // 'down' is rotation about x-axis with positive angle increment
      z =
        panner.orientationZ.value \* Math.cos(q) -
        panner.orientationY.value \* Math.sin(q);
      y =
        panner.orientationZ.value \* Math.sin(q) +
        panner.orientationY.value \* Math.cos(q);
      x = panner.orientationX.value;
      panner.orientationX.value = x;
      panner.orientationY.value = y;
      panner.orientationZ.value = z;
      break;
  }

  boombox.style.transform =
    `translateX(${transform.xAxis}px) ` +
    `translateY(${transform.yAxis}px) ` +
    `scale(${transform.zAxis}) ` +
    `rotateY(${transform.rotateY}deg) ` +
    `rotateX(${transform.rotateX}deg)`;

  const move = prevMove || {};
  move.frameId = requestAnimationFrame(() => moveBoombox(direction, move));
  return move;
}
```

Wiring up our controls
----------------------

Wiring up out control buttons is comparatively simple — now we can listen for a mouse event on our controls and run this function, as well as stop it when the mouse is released:

```js
// for each of our controls, move the boombox and change the position values
moveControls.forEach((el) => {
  let moving;
  el.addEventListener(
    "mousedown",
    () => {
      const direction = this.dataset.control;
      if (moving && moving.frameId) {
        cancelAnimationFrame(moving.frameId);
      }
      moving = moveBoombox(direction);
    },
    false,
  );

  window.addEventListener(
    "mouseup",
    () => {
      if (moving && moving.frameId) {
        cancelAnimationFrame(moving.frameId);
      }
    },
    false,
  );
});
```

Connecting our graph
--------------------

Our HTML contains the audio element we want to be affected by the panner node.

```html
<audio src="myCoolTrack.mp3"></audio>
```

We need to grab the source from that element and pipe it into the Web Audio API using the `AudioContext.createMediaElementSource`.

```js
// get the audio element
const audioElement = document.querySelector("audio");

// pass it into the audio context
const track = audioContext.createMediaElementSource(audioElement);
```

Next we have to connect our audio graph. We connect our input (the track) to our modification node (the panner) to our destination (in this case the speakers).

```js
track.connect(panner).connect(audioCtx.destination);
```

Let's create a play button, that when clicked will play or pause the audio depending on the current state.

```html
<button data-playing="false" role="switch">Play/Pause</button>
```

```js
// Select our play button
const playButton = document.querySelector("button");

playButton.addEventListener(
  "click",
  () => {
    // Check if context is in suspended state (autoplay policy)
    if (audioContext.state === "suspended") {
      audioContext.resume();
    }

    // Play or pause track depending on state
    if (playButton.dataset.playing === "false") {
      audioElement.play();
      playButton.dataset.playing = "true";
    } else if (playButton.dataset.playing === "true") {
      audioElement.pause();
      playButton.dataset.playing = "false";
    }
  },
  false,
);
```

For a more in depth look at playing/controlling audio and audio graphs check out Using The Web Audio API.

Summary
-------

 Hopefully, this article has given you an insight into how Web Audio spatialization works, and what each of the `PannerNode` properties do (there are quite a few of them).
 The values can be hard to manipulate sometimes and depending on your use case it can take some time to get them right.

**Note:** There are slight differences in the way the audio spatialization sounds across different browsers.
 The panner node does some very involved maths under the hood;
 there are a number of tests here so you can keep track of the status of the inner workings of this node across different platforms.

 Again, you can check out the final demo here, and the final source code is here.
 There is also a Codepen demo too.

 If you are working with 3D games and/or WebXR it's a good idea to harness a 3D library to create such functionality, rather than trying to do this all yourself from first principles.
 We rolled our own in this article to give you an idea of how it works, but you'll save a lot of time by taking advantage of work others have done before you.


Example and tutorial: Simple synth keyboard - Web APIs

Example and tutorial: Simple synth keyboard
===========================================

This article presents the code and working demo of a video keyboard you can play using the mouse. The keyboard allows you to switch among the standard waveforms as well as one custom waveform, and you can control the main gain using a volume slider beneath the keyboard. This example makes use of the following Web API interfaces: `AudioContext`, `OscillatorNode`, `PeriodicWave`, and `GainNode`.

Because `OscillatorNode` is based on `AudioScheduledSourceNode`, this is to some extent an example for that as well.

The video keyboard
------------------

### HTML

There are three primary components to the display for our virtual keyboard. The first is the musical keyboard itself. We draw this in a pair of nested `<div>` elements so that we can make the keyboard horizontally scrollable if all the keys don't fit on the screen, without having them wrap around.

#### The keyboard

First, we create space to build the keyboard into. We will be programmatically constructing the keyboard, because doing so gives us the flexibility to configure each key as we determine the appropriate data for the corresponding note. In our case, we get each key's frequency from a table, but it could be calculated algorithmically as well.

```html
<div class="container">
  <div class="keyboard"></div>
</div>
```

The `<div>` named `"container"` is the scrollable box that lets the keyboard be scrolled horizontally if it's too wide for the available space. The keys themselves will be inserted into the block of class `"keyboard"`.

#### The settings bar

Beneath the keyboard, we'll put some controls for configuring the layer. For now, we will have two controls: one to set the main volume and another to select what periodic waveform to use when generating notes.

##### The volume control

First we create the `<div>` to contain the settings bar, so it can be styled as needed. Then we establish a box that will be presented on the left side of the bar and place a label and an `<input>` element of type `"range"`. The range element will typically be presented as a slider control; we configure it to allow any value between 0.0 and 1.0, stepping by 0.01 each position.

```html
<div class="settingsBar">
  <div class="left">
    <span>Volume: </span>
    <input
 type="range"
 min="0.0"
 max="1.0"
 step="0.01"
 value="0.5"
 list="volumes"
 name="volume" />
    <datalist id="volumes">
      <option value="0.0" label="Mute"></option>
      <option value="1.0" label="100%"></option>
    </datalist>
  </div>
</div>
```

We specify a default value of 0.5, and we provide a `<datalist>` element which is connected to the range using the `name` attribute to find an option list whose ID matches; in this case, the data set is named `"volume"`. This lets us provide a set of common values and special strings which the browser may optionally choose to display in some fashion; we provide names for the values 0.0 ("Mute") and 1.0 ("100%").

##### The waveform picker

On the right side of the settings bar, we place a label and a `<select>` element named `"waveform"` whose options correspond to the available waveforms.

```html
  <div class="right">
    <span>Current waveform: </span>
    <select name="waveform">
      <option value="sine">Sine</option>
      <option value="square" selected>Square</option>
      <option value="sawtooth">Sawtooth</option>
      <option value="triangle">Triangle</option>
      <option value="custom">Custom</option>
    </select>
  </div>
</div>
```

### CSS

```css
.container {
  overflow-x: scroll;
  overflow-y: hidden;
  width: 660px;
  height: 110px;
  white-space: nowrap;
  margin: 10px;
}

.keyboard {
  width: auto;
  padding: 0;
  margin: 0;
}

.key {
  cursor: pointer;
  font:
    16px "Open Sans",
    "Lucida Grande",
    "Arial",
    sans-serif;
  border: 1px solid black;
  border-radius: 5px;
  width: 20px;
  height: 80px;
  text-align: center;
  box-shadow: 2px 2px darkgray;
  display: inline-block;
  position: relative;
  margin-right: 3px;
  user-select: none;
  -moz-user-select: none;
  -webkit-user-select: none;
  -ms-user-select: none;
}

.key div {
  position: absolute;
  bottom: 0;
  text-align: center;
  width: 100%;
  pointer-events: none;
}

.key div sub {
  font-size: 10px;
  pointer-events: none;
}

.key:hover {
  background-color: #eef;
}

.key:active,
.active {
  background-color: #000;
  color: #fff;
}

.octave {
  display: inline-block;
  padding: 0 6px 0 0;
}

.settingsBar {
  padding-top: 8px;
  font:
    14px "Open Sans",
    "Lucida Grande",
    "Arial",
    sans-serif;
  position: relative;
  vertical-align: middle;
  width: 100%;
  height: 30px;
}

.left {
  width: 50%;
  position: absolute;
  left: 0;
  display: table-cell;
  vertical-align: middle;
}

.left span,
.left input {
  vertical-align: middle;
}

.right {
  width: 50%;
  position: absolute;
  right: 0;
  display: table-cell;
  vertical-align: middle;
}

.right span {
  vertical-align: middle;
}

.right input {
  vertical-align: baseline;
}
```

### JavaScript

The JavaScript code begins by initializing a number of variables.

```js
const audioContext = new AudioContext();
const oscList = [];
let mainGainNode = null;
```

1. `audioContext` is set to reference the global `AudioContext` object (or `webkitAudioContext` if necessary).
2. `oscList` is set up to be ready to contain a list of all currently-playing oscillators. It starts off empty, since there are none playing yet.
3. `mainGainNode` is set to null; during the setup process, it will be configured to contain a `GainNode` which all playing oscillators will connect to and play through to allow the overall volume to be controlled using a single slider control.

```js
const keyboard = document.querySelector(".keyboard");
const wavePicker = document.querySelector("select[name='waveform']");
const volumeControl = document.querySelector("input[name='volume']");
```

References to elements we'll need access to are obtained:

* `keyboard` is the container element into which the keys will be placed.
* `wavePicker` is the `<select>` element used to choose the waveform to use for the notes.
* `volumeControl` is the `<input>` element (of type `"range"`) used to control the main audio volume.

```js
let noteFreq = null;
let customWaveform = null;
let sineTerms = null;
let cosineTerms = null;
```

Finally, global variables that will be used when constructing waveforms are created:

* `noteFreq` will be an array of arrays; each array represents one octave, each of which contains one entry for each note in that octave. The value for each is the frequency, in Hertz, of the note's tone.
* `customWaveform` will be set up as a `PeriodicWave` describing the waveform to use when the user selects "Custom" from the waveform picker.
* `sineTerms` and `cosineTerms` will be used to store the data for generating the waveform; each will contain an array that's generated when the user chooses "Custom".
### Creating the note table

The `createNoteTable()` function builds the array `noteFreq` to contain an array of objects representing each octave. Each octave, in turn, has one named property for each note in that octave; the property's name is the note's name (such as "C#" to represent C-sharp), and the value is the frequency, in Hertz, of that note.

```js
function createNoteTable() {
  const noteFreq = [];
  for (let i=0; i< 9; i++) {
    noteFreq[i] = [];
  }

  noteFreq[0]["A"] = 27.500000000000000;
  noteFreq[0]["A#"] = 29.135235094880619;
  noteFreq[0]["B"] = 30.867706328507756;

  noteFreq[1]["C"] = 32.703195662574829;
  noteFreq[1]["C#"] = 34.647828872109012;
  noteFreq[1]["D"] = 36.708095989675945;
  noteFreq[1]["D#"] = 38.890872965260113;
  noteFreq[1]["E"] = 41.203444614108741;
  noteFreq[1]["F"] = 43.653528929125485;
  noteFreq[1]["F#"] = 46.249302838954299;
  noteFreq[1]["G"] = 48.999429497718661;
  noteFreq[1]["G#"] = 51.913087197493142;
  noteFreq[1]["A"] = 55.000000000000000;
  noteFreq[1]["A#"] = 58.270470189761239;
  noteFreq[1]["B"] = 61.735412657015513;
  // …
```

Several octaves not shown for brevity.
```

noteFreq[2]["C"] = 65.406391325149658;
noteFreq[2]["C#"] = 69.295657744218024;
noteFreq[2]["D"] = 73.41619197935189;
noteFreq[2]["D#"] = 77.781745930520227;
noteFreq[2]["E"] = 82.406889228217482;
noteFreq[2]["F"] = 87.307057858250971;
noteFreq[2]["F#"] = 92.498605677908599;
noteFreq[2]["G"] = 97.998858995437323;
noteFreq[2]["G#"] = 103.826174394986284;
noteFreq[2]["A"] = 110.0;
noteFreq[2]["A#"] = 116.540940379522479;
noteFreq[2]["B"] = 123.470825314031027;

noteFreq[3]["C"] = 130.812782650299317;
noteFreq[3]["C#"] = 138.591315488436048;
noteFreq[3]["D"] = 146.83238395870378;
noteFreq[3]["D#"] = 155.563491861040455;
noteFreq[3]["E"] = 164.813778456434964;
noteFreq[3]["F"] = 174.614115716501942;
noteFreq[3]["F#"] = 184.997211355817199;
noteFreq[3]["G"] = 195.997717990874647;
noteFreq[3]["G#"] = 207.652348789972569;
noteFreq[3]["A"] = 220.0;
noteFreq[3]["A#"] = 233.081880759044958;
noteFreq[3]["B"] = 246.941650628062055;

noteFreq[4]["C"] = 261.625565300598634;
noteFreq[4]["C#"] = 277.182630976872096;
noteFreq[4]["D"] = 293.66476791740756;
noteFreq[4]["D#"] = 311.12698372208091;
noteFreq[4]["E"] = 329.627556912869929;
noteFreq[4]["F"] = 349.228231433003884;
noteFreq[4]["F#"] = 369.994422711634398;
noteFreq[4]["G"] = 391.995435981749294;
noteFreq[4]["G#"] = 415.304697579945138;
noteFreq[4]["A"] = 440.0;
noteFreq[4]["A#"] = 466.163761518089916;
noteFreq[4]["B"] = 493.883301256124111;

noteFreq[5]["C"] = 523.251130601197269;
noteFreq[5]["C#"] = 554.365261953744192;
noteFreq[5]["D"] = 587.32953583481512;
noteFreq[5]["D#"] = 622.253967444161821;
noteFreq[5]["E"] = 659.255113825739859;
noteFreq[5]["F"] = 698.456462866007768;
noteFreq[5]["F#"] = 739.988845423268797;
noteFreq[5]["G"] = 783.990871963498588;
noteFreq[5]["G#"] = 830.609395159890277;
noteFreq[5]["A"] = 880.0;
noteFreq[5]["A#"] = 932.327523036179832;
noteFreq[5]["B"] = 987.766602512248223;

noteFreq[6]["C"] = 1046.502261202394538;
noteFreq[6]["C#"] = 1108.730523907488384;
noteFreq[6]["D"] = 1174.659071669630241;
noteFreq[6]["D#"] = 1244.507934888323642;
noteFreq[6]["E"] = 1318.510227651479718;
noteFreq[6]["F"] = 1396.912925732015537;
noteFreq[6]["F#"] = 1479.977690846537595;
noteFreq[6]["G"] = 1567.981743926997176;
noteFreq[6]["G#"] = 1661.218790319780554;
noteFreq[6]["A"] = 1760.0;
noteFreq[6]["A#"] = 1864.655046072359665;
noteFreq[6]["B"] = 1975.533205024496447;
```

```js
  noteFreq[7]["C"] = 2093.004522404789077;
  noteFreq[7]["C#"] = 2217.461047814976769;
  noteFreq[7]["D"] = 2349.318143339260482;
  noteFreq[7]["D#"] = 2489.015869776647285;
  noteFreq[7]["E"] = 2637.020455302959437;
  noteFreq[7]["F"] = 2793.825851464031075;
  noteFreq[7]["F#"] = 2959.955381693075191;
  noteFreq[7]["G"] = 3135.963487853994352;
  noteFreq[7]["G#"] = 3322.437580639561108;
  noteFreq[7]["A"] = 3520.000000000000000;
  noteFreq[7]["A#"] = 3729.310092144719331;
  noteFreq[7]["B"] = 3951.066410048992894;

  noteFreq[8]["C"] = 4186.009044809578154;
  return noteFreq;
}
```

The result is an array, `noteFreq`, with an object for each octave. Each octave object has named properties in it where the property name is the name of the note (such as "C#" to represent C-sharp) and the property's value is the note's frequency in Hertz. In part, the resulting object looks like this:

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Octave | Notes |  |  |  |  |
| 0 | "A" ⇒ 27.5 | "A#" ⇒ 29.14 | "B" ⇒ 30.87 |  |  |  |  |  |  |  |  |  |
| 1 | "C" ⇒ 32.70 | "C#" ⇒ 34.65 | "D" ⇒ 36.71 | "D#" ⇒ 38.89 | "E" ⇒ 41.20 | "F" ⇒ 43.65 | "F#" ⇒ 46.25 | "G" ⇒ 49 | "G#" ⇒ 51.9 | "A" ⇒ 55 | "A#" ⇒ 58.27 | "B" ⇒ 61.74 |
| 2 | . . . |

With this table in place, we can find out the frequency for a given note in a particular octave quite easily. If we want the frequency for the note G# in octave 1, we use `noteFreq[1]["G#"]` and get the value 51.9 as a result.

**Note:** The values in the example table above have been rounded to two decimal places.
```

if (!Object.entries) {
  Object.entries = function entries(O) {
    return reduce(
      keys(O),
      (e, k) =>
        concat(
          e,
          typeof k === "string" && isEnumerable(O, k) ? [[k, O[k]]] : [],
        ),
      [],
    );
  };
}
```

### Building the keyboard

The `setup()` function is responsible for building the keyboard and preparing the app to play music.

```js
function setup() {
  noteFreq = createNoteTable();

  volumeControl.addEventListener("change", changeVolume, false);

  mainGainNode = audioContext.createGain();
  mainGainNode.connect(audioContext.destination);
  mainGainNode.gain.value = volumeControl.value;

  // Create the keys; skip any that are sharp or flat; for
  // our purposes we don't need them. Each octave is inserted
  // into a <div> of class "octave".

  noteFreq.forEach((keys, idx) => {
    const keyList = Object.entries(keys);
    const octaveElem = document.createElement("div");
    octaveElem.className = "octave";

    keyList.forEach((key) => {
      if (key[0].length === 1) {
        octaveElem.appendChild(createKey(key[0], idx, key[1]));
      }
    });

    keyboard.appendChild(octaveElem);
  });

  document
    .querySelector("div[data-note='B'][data-octave='5']")
    .scrollIntoView(false);

  sineTerms = new Float32Array([0, 0, 1, 0, 1]);
  cosineTerms = new Float32Array(sineTerms.length);
  customWaveform = audioContext.createPeriodicWave(cosineTerms, sineTerms);

  for (let i = 0; i < 9; i++) {
    oscList[i] = {};
  }
}

setup();
```

1. The table which maps note names and octaves to their frequencies is created by calling `createNoteTable()`.
2. An event handler is established (by calling our old friend `addEventListener()` to handle `change` events on the main gain control. This will update the main gain node's volume to the new value of the control.
3. Next, we iterate over each octave in the note frequencies table. For each octave, we use `Object.entries()` to get a list of the notes in that octave.
4. Create a `<div>` to contain that octave's notes (so we can have a small bit of space drawn between octaves), and set its class name to "octave"
5. For each key in the octave, we check to see if the note's name has more than one character. We skip these, because we're leaving out the sharp notes in this example. If the note's name is only one character, then we call `createKey()`, specifying the note string, octave, and frequency. The returned element is appended to the octave element created in step 4.
6. When each octave element has been built, it's appended to the keyboard.
7. Once the keyboard has been constructed, we scroll the note "B" in octave 5 into view; this has the effect of ensuring that middle-C is visible along with its surrounding keys.
8. Then a new custom waveform is built using `BaseAudioContext.createPeriodicWave()`. This waveform will be used any time the user selects "Custom" from the waveform picker control.
9. Finally, the oscillator list is initialized to ensure that it's ready to receive information identifying which oscillators are associated with which keys.

#### Creating a key

The `createKey()` function is called once for each key that we want to present in the virtual keyboard. It creates the elements that comprise the key and its label, adds some data attributes to the element for later use, and assigns event handlers for the events we care about.

```js
function createKey(note, octave, freq) {
  const keyElement = document.createElement("div");
  const labelElement = document.createElement("div");

  keyElement.className = "key";
  keyElement.dataset["octave"] = octave;
  keyElement.dataset["note"] = note;
  keyElement.dataset["frequency"] = freq;

  labelElement.innerHTML = `${note}<sub>${octave}</sub>`;
  keyElement.appendChild(labelElement);

  keyElement.addEventListener("mousedown", notePressed, false);
  keyElement.addEventListener("mouseup", noteReleased, false);
  keyElement.addEventListener("mouseover", notePressed, false);
  keyElement.addEventListener("mouseleave", noteReleased, false);

  return keyElement;
}
```

After creating the elements that will represent the key and its label, we configure the key's element by setting its class to "key" (which establishes its appearance). Then we add `data-*` attributes which contain the key's octave (attribute `data-octave`), string representing the note to play (attribute `data-note`), and frequency (attribute `data-frequency`) in Hertz. This will let us easily fetch that information as needed when handling events.

### Making music

#### Playing a tone

The `playTone()` function's job is to play a tone at the given frequency. This will be used by the handler for events triggering keys on the keyboard to start playing the appropriate notes.

```js
function playTone(freq) {
  const osc = audioContext.createOscillator();
  osc.connect(mainGainNode);

  const type = wavePicker.options[wavePicker.selectedIndex].value;

  if (type === "custom") {
    osc.setPeriodicWave(customWaveform);
  } else {
    osc.type = type;
  }

  osc.frequency.value = freq;
  osc.start();

  return osc;
}
```

`playTone()` begins by creating a new `OscillatorNode` by calling the `BaseAudioContext.createOscillator()` method. We then connect it to the main gain node by calling the new oscillator's `connect()` method;, which tells the oscillator where to send its output to. By doing this, changing the gain of the main gain node will affect the volume of all tones being generated.

Then we get the type of waveform to use by checking the value of the waveform picker control in the settings bar. If the user has it set to `"custom"`, we call `OscillatorNode.setPeriodicWave()` to configure the oscillator to use our custom waveform. Doing this automatically sets the oscillator's `type` to `custom`. If any other waveform type is selected in the wave picker, we set the oscillator's type to the value of the picker; that value will be one of `sine`, `square`, `triangle`, and `sawtooth`.

The oscillator's frequency is set to the value specified in the `freq` parameter by setting the value of the `OscillatorNode.frequency` `AudioParam` object. Then, at last, the oscillator is started up so that it begins to produce sound by calling the oscillator's inherited `AudioScheduledSourceNode.start()` method.

#### Playing a note

When the `mousedown` or `mouseover` event occurs on a key, we want to start playing the corresponding note. The `notePressed()` function is used as the event handler for these events.

```js
function notePressed(event) {
  if (event.buttons & 1) {
    const dataset = event.target.dataset;

    if (!dataset["pressed"] && dataset["octave"]) {
      const octave = Number(dataset["octave"]);
      oscList[octave][dataset["note"]] = playTone(dataset["frequency"]);
      dataset["pressed"] = "yes";
    }
  }
}
```

We start by checking whether the primary mouse button is pressed, for two reasons. First, we want to only allow the primary mouse button to trigger notes playing. Second, and more importantly, we are using this to handle `mouseover` for cases where the user is dragging from note to note, and we only want to start playing the note if the mouse is pressed when it enters the element.

If the mouse button is in fact down, we get the pressed key's `dataset` attribute; this makes it easy to access the custom data attributes on the element. We look for a `data-pressed` attribute; if there isn't one (which indicates that the note isn't already playing), we call `playTone()` to start playing the note, passing in the value of the element's `data-frequency` attribute. The returned oscillator is stored into `oscList` for future reference, and `data-pressed` is set to `yes` to indicate that the note is playing so we don't start it again next time this is called.

#### Stopping a tone

The `noteReleased()` function is the event handler called when the user releases the mouse button or moves the mouse out of the key that's currently playing.

```js
function noteReleased(event) {
  const dataset = event.target.dataset;

  if (dataset && dataset["pressed"]) {
    const octave = Number(dataset["octave"]);

    if (oscList[octave] && oscList[octave][dataset["note"]]) {
      oscList[octave][dataset["note"]].stop();
      delete oscList[octave][dataset["note"]];
      delete dataset["pressed"];
    }
  }
}
```

`noteReleased()` uses the `data-octave` and `data-note` custom attributes to look up the key's oscillator, then calls the oscillator's inherited `stop()` method to stop playing the note. Finally, the `oscList` entry for the note is cleared and the `data-pressed` attribute is removed from the key element (as identified by `event.target`), to indicate that the note is not currently playing.

#### Changing the main volume

The volume slider in the settings bar provides a simple interface to change the gain value on the main gain node, thereby changing the loudness of all playing notes. The `changeVolume()` method is the handler for the `change` event on the slider.

```js
function changeVolume(event) {
  mainGainNode.gain.value = volumeControl.value;
}
```

This sets the value of the main gain node's `gain` `AudioParam` to the slider's new value.

#### Keyboard support

The code below adds `keydown` and `keyup` event listeners to handle keyboard input. The `keydown` event handler calls `notePressed()` to start playing the note corresponding to the key that was pressed, and the `keyup` event handler calls `noteReleased()` to stop playing the note corresponding to the key that was released.

```js
const synthKeys = document.querySelectorAll(".key");
const keyCodes = [
  "Space",
  "ShiftLeft", "KeyZ", "KeyX", "KeyC", "KeyV", "KeyB", "KeyN", "KeyM", "Comma", "Period", "Slash", "ShiftRight",
  "KeyA", "KeyS", "KeyD", "KeyF", "KeyG", "KeyH", "KeyJ", "KeyK", "KeyL", "Semicolon", "Quote", "Enter",
  "Tab", "KeyQ", "KeyW", "KeyE", "KeyR", "KeyT", "KeyY", "KeyU", "KeyI", "KeyO", "KeyP", "BracketLeft", "BracketRight",
  "Digit1", "Digit2", "Digit3", "Digit4", "Digit5", "Digit6", "Digit7", "Digit8", "Digit9", "Digit0", "Minus", "Equal", "Backspace",
  "Escape",
];
function keyNote(event) {
  const elKey = synthKeys[keyCodes.indexOf(event.code)];
  if (elKey) {
    if (event.type === "keydown") {
      elKey.tabIndex = -1;
      elKey.focus();
      elKey.classList.add("active");
      notePressed({ buttons: 1, target: elKey });
    } else {
      elKey.classList.remove("active");
      noteReleased({ buttons: 1, target: elKey });
    }
    event.preventDefault();
  }
}
addEventListener("keydown", keyNote);
addEventListener("keyup", keyNote);
```

### Result

Put all together, the result is a simple but working point-and-click musical keyboard:

See also
--------

* Web Audio API
* `OscillatorNode`
* `GainNode`
* `AudioContext`


Visualizations with Web Audio API - Web APIs

Visualizations with Web Audio API
=================================

One of the most interesting features of the Web Audio API is the ability to extract frequency, waveform, and other data from your audio source, which can then be used to create visualizations. This article explains how, and provides a couple of basic use cases.

**Note:** You can find working examples of all the code snippets in our Voice-change-O-matic demo.

Basic concepts
--------------

To extract data from your audio source, you need an `AnalyserNode`, which is created using the `BaseAudioContext.createAnalyser` method, for example:

```js
const audioCtx = new AudioContext();
const analyser = audioCtx.createAnalyser();
```

This node is then connected to your audio source at some point between your source and your destination, for example:

```js
const source = audioCtx.createMediaStreamSource(stream);
source.connect(analyser);
analyser.connect(distortion);
distortion.connect(audioCtx.destination);
```

**Note:** you don't need to connect the analyser's output to another node for it to work, as long as the input is connected to the source, either directly or via another node.

The analyser node will then capture audio data using a Fast Fourier Transform (fft) in a certain frequency domain, depending on what you specify as the `AnalyserNode.fftSize` property value (if no value is specified, the default is 2048.)

**Note:** You can also specify a minimum and maximum power value for the fft data scaling range, using `AnalyserNode.minDecibels` and `AnalyserNode.maxDecibels`, and different data averaging constants using `AnalyserNode.smoothingTimeConstant`. Read those pages to get more information on how to use them.

To capture data, you need to use the methods `AnalyserNode.getFloatFrequencyData()` and `AnalyserNode.getByteFrequencyData()` to capture frequency data, and `AnalyserNode.getByteTimeDomainData()` and `AnalyserNode.getFloatTimeDomainData()` to capture waveform data.

These methods copy data into a specified array, so you need to create a new array to receive the data before invoking one. The first one produces 32-bit floating point numbers, and the second and third ones produce 8-bit unsigned integers, therefore a standard JavaScript array won't do — you need to use a `Float32Array` or `Uint8Array` array, depending on what data you are handling.

So for example, say we are dealing with an fft size of 2048. We return the `AnalyserNode.frequencyBinCount` value, which is half the fft, then call Uint8Array() with the frequencyBinCount as its length argument — this is how many data points we will be collecting, for that fft size.

```js
analyser.fftSize = 2048;
const bufferLength = analyser.frequencyBinCount;
const dataArray = new Uint8Array(bufferLength);
```

To actually retrieve the data and copy it into our array, we then call the data collection method we want, with the array passed as it's argument. For example:

```js
analyser.getByteTimeDomainData(dataArray);
```

We now have the audio data for that moment in time captured in our array, and can proceed to visualize it however we like, for example by plotting it onto an HTML `<canvas>`.

Let's go on to look at some specific examples.

Creating a waveform/oscilloscope
--------------------------------

To create the oscilloscope visualization (hat tip to Soledad Penadés for the original code in Voice-change-O-matic, we first follow the standard pattern described in the previous section to set up the buffer:

```js
analyser.fftSize = 2048;
const bufferLength = analyser.frequencyBinCount;
const dataArray = new Uint8Array(bufferLength);
```

Next, we clear the canvas of what had been drawn on it before to get ready for the new visualization display:

```js
canvasCtx.clearRect(0, 0, WIDTH, HEIGHT);
```

We now define the `draw()` function:

```js
function draw() {
```

In here, we use `requestAnimationFrame()` to keep looping the drawing function once it has been started:

```js
const drawVisual = requestAnimationFrame(draw);
```

Next, we grab the time domain data and copy it into our array

```js
analyser.getByteTimeDomainData(dataArray);
```

Next, fill the canvas with a solid color to start

```js
canvasCtx.fillStyle = "rgb(200 200 200)";
canvasCtx.fillRect(0, 0, WIDTH, HEIGHT);
```

Set a line width and stroke color for the wave we will draw, then begin drawing a path

```js
canvasCtx.lineWidth = 2;
canvasCtx.strokeStyle = "rgb(0 0 0)";
canvasCtx.beginPath();
```

Determine the width of each segment of the line to be drawn by dividing the canvas width by the array length (equal to the FrequencyBinCount, as defined earlier on), then define an x variable to define the position to move to for drawing each segment of the line.

```js
const sliceWidth = WIDTH / bufferLength;
let x = 0;
```

Now we run through a loop, defining the position of a small segment of the wave for each point in the buffer at a certain height based on the data point value from the array, then moving the line across to the place where the next wave segment should be drawn:

```js
for (let i = 0; i < bufferLength; i++) {
  const v = dataArray[i] / 128.0;
  const y = v \* (HEIGHT / 2);

  if (i === 0) {
    canvasCtx.moveTo(x, y);
  } else {
    canvasCtx.lineTo(x, y);
  }

  x += sliceWidth;
}
```

Finally, we finish the line in the middle of the right-hand side of the canvas, then draw the stroke we've defined:

```js
  canvasCtx.lineTo(WIDTH, HEIGHT / 2);
  canvasCtx.stroke();
}
```

At the end of this section of code, we invoke the `draw()` function to start off the whole process:

```js
draw();
```

This gives us a nice waveform display that updates several times a second:

![a black oscilloscope line, showing the waveform of an audio signal](/en-US/docs/Web/API/Web_Audio_API/Visualizations_with_Web_Audio_API/wave.png)

Creating a frequency bar graph
------------------------------

Another nice little sound visualization to create is one of those Winamp-style frequency bar graphs. We have one available in Voice-change-O-matic; let's look at how it's done.

First, we again set up our analyser and data array, then clear the current canvas display with `clearRect()`. The only difference from before is that we have set the fft size to be much smaller; this is so that each bar in the graph is big enough to actually look like a bar rather than a thin strand.

```js
analyser.fftSize = 256;
const bufferLength = analyser.frequencyBinCount;
const dataArray = new Uint8Array(bufferLength);

canvasCtx.clearRect(0, 0, WIDTH, HEIGHT);
```

Next, we start our `draw()` function off, again setting up a loop with `requestAnimationFrame()` so that the displayed data keeps updating, and clearing the display with each animation frame.

```js
function draw() {
  drawVisual = requestAnimationFrame(draw);

  analyser.getByteFrequencyData(dataArray);

  canvasCtx.fillStyle = "rgb(0 0 0)";
  canvasCtx.fillRect(0, 0, WIDTH, HEIGHT);
```

Now we set our `barWidth` to be equal to the canvas width divided by the number of bars (the buffer length). However, we are also multiplying that width by 2.5, because most of the frequencies will come back as having no audio in them, as most of the sounds we hear every day are in a certain lower frequency range. We don't want to display loads of empty bars, therefore we shift the ones that will display regularly at a noticeable height across so they fill the canvas display.

We also set a `barHeight` variable, and an `x` variable to record how far across the screen to draw the current bar.

```js
const barWidth = (WIDTH / bufferLength) \* 2.5;
let barHeight;
let x = 0;
```

As before, we now start a for loop and cycle through each value in the `dataArray`. For each one, we make the `barHeight` equal to the array value, set a fill color based on the `barHeight` (taller bars are brighter), and draw a bar at `x` pixels across the canvas, which is `barWidth` wide and `barHeight / 2` tall (we eventually decided to cut each bar in half so they would all fit on the canvas better.)

The one value that needs explaining is the vertical offset position we are drawing each bar at: `HEIGHT - barHeight / 2`. I am doing this because I want each bar to stick up from the bottom of the canvas, not down from the top, as it would if we set the vertical position to 0. Therefore, we instead set the vertical position each time to the height of the canvas minus `barHeight / 2`, so therefore each bar will be drawn from partway down the canvas, down to the bottom.

```js
for (let i = 0; i < bufferLength; i++) {
  barHeight = dataArray[i] / 2;

  canvasCtx.fillStyle = `rgb(${barHeight + 100} 50 50)`;
  canvasCtx.fillRect(x, HEIGHT - barHeight / 2, barWidth, barHeight);

  x += barWidth + 1;
}
```

Again, at the end of the code we invoke the `draw()` function to set the whole process in motion.

```js
draw();
```

This code gives us a result like the following:

![a series of red bars in a bar graph, showing intensity of different frequencies in an audio signal](/en-US/docs/Web/API/Web_Audio_API/Visualizations_with_Web_Audio_API/bar-graph.png)

**Note:** The examples listed in this article have shown usage of `AnalyserNode.getByteFrequencyData()` and `AnalyserNode.getByteTimeDomainData()`. For working examples showing `AnalyserNode.getFloatFrequencyData()` and `AnalyserNode.getFloatTimeDomainData()`, refer to our Voice-change-O-matic-float-data demo — this is exactly the same as the original Voice-change-O-matic, except that it uses Float data, not unsigned byte data. See this section of the source code for details


Advanced techniques: Creating and sequencing audio - Web APIs

Advanced techniques: Creating and sequencing audio
==================================================

In this tutorial, we're going to cover sound creation and modification, as well as timing and scheduling. We will introduce sample loading, envelopes, filters, wavetables, and frequency modulation. If you're familiar with these terms and looking for an introduction to their application with the Web Audio API, you've come to the right place.

**Note:** You can find the source code for the demo below on GitHub in the step-sequencer subdirectory of the MDN webaudio-examples repo. You can also see the live demo.

Demo
----

We're going to be looking at a very simple step sequencer:

![A sound sequencer application featuring play and BPM master controls and 4 different voices with controls for each.](/en-US/docs/Web/API/Web_Audio_API/Advanced_techniques/sequencer.png)

In practice, this is easier to do with a library — the Web Audio API was built to be built upon. If you are about to embark on building something more complex, tone.js would be an excellent place to start. However, we want to demonstrate how to create such a demo from first principles as a learning exercise.

The interface consists of master controls, which allow us to play/stop the sequencer, and adjust the BPM (beats per minute) to speed up or slow down the "music".

Four different sounds, or voices, can be played. Each voice has four buttons, one for each beat in one bar of music. When they are enabled, the note will sound. When the instrument plays, it will move across this set of beats and loop the bar.

Each voice also has local controls, allowing you to manipulate the effects or parameters particular to each technique we use to create those voices. The methods we are using are:

| Name of voice | Technique | Associated Web Audio API feature |
| --- | --- | --- |
| "Sweep" | Oscillator, periodic wave | `OscillatorNode`,
 `PeriodicWave` |
| "Pulse" | Multiple oscillators | `OscillatorNode` |
| "Noise" | Random noise buffer, Biquad filter | `AudioBuffer`,
 `AudioBufferSourceNode`,
 `BiquadFilterNode` |
| "Dial up" | Loading a sound sample to play | `BaseAudioContext/decodeAudioData`,
 `AudioBufferSourceNode` |

**Note:** We didn't create this instrument to sound good but to provide demonstration code. This demonstration represents a *very* simplified version of such an instrument. The sounds are based on a dial-up modem. If you are unaware of how such a device sounds, you can listen to one here.

Creating an audio context
-------------------------

As you should be used to by now, each Web Audio API app starts with an audio context:

```js
const audioCtx = new AudioContext();
```

The "sweep" — oscillators, periodic waves, and envelopes
--------------------------------------------------------

For what we will call the "sweep" sound, that first noise you hear when you dial up, we're going to create an oscillator to generate the sound.

The `OscillatorNode` comes with basic waveforms out of the box — sine, square, triangle, or sawtooth. However, instead of using the standard waves that come by default, we're going to create our own using the `PeriodicWave` interface and values set in a wavetable. We can use the `PeriodicWave()` constructor to use this custom wave with an oscillator.

### The periodic wave

First of all, we'll create our periodic wave. To do so, We need to pass real and imaginary values into the `PeriodicWave()` constructor:

```js
const wave = new PeriodicWave(audioCtx, {
  real: wavetable.real,
  imag: wavetable.imag,
});
```

**Note:** In our example, the wavetable is held in a separate JavaScript file (`wavetable.js`) because there are *so* many values. We took it from a repository of wavetables, found in the Web Audio API examples from Google Chrome Labs.

### The Oscillator

Now we can create an `OscillatorNode` and set its wave to the one we've created:

```js
function playSweep(time) {
  const osc = new OscillatorNode(audioCtx, {
    frequency: 380,
    type: "custom",
    periodicWave: wave,
  });
  osc.connect(audioCtx.destination);
  osc.start(time);
  osc.stop(time + 1);
}
```

We pass a time parameter to the function here, which we'll use later to schedule the sweep.

### Controlling amplitude

This is great, but wouldn't it be nice if we had an amplitude envelope to go with it? Let's create a simple one, so we get used to the methods we need to create an envelope with the Web Audio API.

Let's say our envelope has attack and release. We can allow the user to control these using range inputs on the interface:

```html
<label for="attack">Attack</label>
<input
 name="attack"
 id="attack"
 type="range"
 min="0"
 max="1"
 value="0.2"
 step="0.1" />

<label for="release">Release</label>
<input
 name="release"
 id="release"
 type="range"
 min="0"
 max="1"
 value="0.5"
 step="0.1" />
```

Now we can create some variables over in JavaScript and have them change when the input values are updated:

```js
let attackTime = 0.2;
const attackControl = document.querySelector("#attack");
attackControl.addEventListener(
  "input",
  (ev) => {
    attackTime = parseInt(ev.target.value, 10);
  },
  false,
);

let releaseTime = 0.5;
const releaseControl = document.querySelector("#release");
releaseControl.addEventListener(
  "input",
  (ev) => {
    releaseTime = parseInt(ev.target.value, 10);
  },
  false,
);
```

### The final playSweep() function

Now we can expand our `playSweep()` function. We need to add a `GainNode` and connect that through our audio graph to apply amplitude variations to our sound. The gain node has one property: `gain`, which is of type `AudioParam`.

This is useful — now we can start to harness the power of the audio param methods on the gain value. We can set a value at a certain time, or we can change it *over* time with methods such as `AudioParam.linearRampToValueAtTime`.

As mentioned above, we'll use the `linearRampToValueAtTime` method for our attack and release. It takes two parameters — the value you want to set the parameter you are changing to (in this case, the gain) and when you want to do this. In our case *when* is controlled by our inputs. So, in the example below, the gain increases to 1 at a linear rate over the time the attack range input defines. Similarly, for our release, the gain is set to 0 at a linear rate, over the time the release input has been set to.

```js
const sweepLength = 2;
function playSweep(time) {
  const osc = new OscillatorNode(audioCtx, {
    frequency: 380,
    type: "custom",
    periodicWave: wave,
  });

  const sweepEnv = new GainNode(audioCtx);
  sweepEnv.gain.cancelScheduledValues(time);
  sweepEnv.gain.setValueAtTime(0, time);
  sweepEnv.gain.linearRampToValueAtTime(1, time + attackTime);
  sweepEnv.gain.linearRampToValueAtTime(0, time + sweepLength - releaseTime);

  osc.connect(sweepEnv).connect(audioCtx.destination);
  osc.start(time);
  osc.stop(time + sweepLength);
}
```

The "pulse" — low-frequency oscillator modulation
-------------------------------------------------

Great, now we've got our sweep! Let's move on and take a look at that nice pulse sound. We can achieve this with a basic oscillator, modulated with a second oscillator.

### Initial oscillator

We'll set up our first `OscillatorNode` the same way as our sweep sound, except we won't use a wavetable to set a bespoke wave — we'll just use the default `sine` wave:

```js
const osc = new OscillatorNode(audioCtx, {
  type: "sine",
  frequency: pulseHz,
});
```

Now we're going to create a `GainNode`, as it's the `gain` value that we will oscillate with our second, low-frequency oscillator:

```js
const amp = new GainNode(audioCtx, {
  value: 1,
});
```

### Creating the second, low-frequency oscillator

We'll now create a second — `square` — wave (or pulse) oscillator to alter the amplification of our first sine wave:

```js
const lfo = new OscillatorNode(audioCtx, {
  type: "square",
  frequency: 30,
});
```

### Connecting the graph

The key here is connecting the graph correctly and also starting both oscillators:

```js
lfo.connect(amp.gain);
osc.connect(amp).connect(audioCtx.destination);
lfo.start();
osc.start(time);
osc.stop(time + pulseTime);
```

**Note:** We also don't have to use the default wave types for either of these oscillators we're creating — we could use a wavetable and the periodic wave method as we did before. There is a multitude of possibilities with just a minimum of nodes.

### Pulse user controls

For the UI controls, let's expose both frequencies of our oscillators, allowing them to be controlled via range inputs. One will change the tone, and the other will change how the pulse modulates the first wave:

```html
<label for="hz">Hz</label>
<input
 name="hz"
 id="hz"
 type="range"
 min="660"
 max="1320"
 value="880"
 step="1" />
<label for="lfo">LFO</label>
<input name="lfo" id="lfo" type="range" min="20" max="40" value="30" step="1" />
```

As before, we'll vary the parameters when the user changes the ranges values.

```js
let pulseHz = 880;
const hzControl = document.querySelector("#hz");
hzControl.addEventListener(
  "input",
  (ev) => {
    pulseHz = parseInt(ev.target.value, 10);
  },
  false,
);

let lfoHz = 30;
const lfoControl = document.querySelector("#lfo");
lfoControl.addEventListener(
  "input",
  (ev) => {
    lfoHz = parseInt(ev.target.value, 10);
  },
  false,
);
```

### The final playPulse() function

Here's the entire `playPulse()` function:

```js
const pulseTime = 1;
function playPulse(time) {
  const osc = new OscillatorNode(audioCtx, {
    type: "sine",
    frequency: pulseHz,
  });

  const amp = new GainNode(audioCtx, {
    value: 1,
  });

  const lfo = new OscillatorNode(audioCtx, {
    type: "square",
    frequency: lfoHz,
  });

  lfo.connect(amp.gain);
  osc.connect(amp).connect(audioCtx.destination);
  lfo.start();
  osc.start(time);
  osc.stop(time + pulseTime);
}
```

The "noise" — random noise buffer with a biquad filter
------------------------------------------------------

Now we need to make some noise! All modems have noise. Noise is just random numbers when it comes to audio data, so is, therefore, a relatively straightforward thing to create with code.

### Creating an audio buffer

We need to create an empty container to put these numbers into, however, one that the Web Audio API understands. This is where `AudioBuffer` objects come in. You can fetch a file and decode it into a buffer (we'll get to that later in the tutorial), or you can create an empty buffer and fill it with your data.

For noise, let's do the latter. We first need to calculate the size of our buffer to create it. We can use the `BaseAudioContext.sampleRate` property for this:

```js
const bufferSize = audioCtx.sampleRate \* noiseDuration;
// Create an empty buffer
const noiseBuffer = new AudioBuffer({
  length: bufferSize,
  sampleRate: audioCtx.sampleRate,
});
```

Now we can fill it with random numbers between -1 and 1:

```js
// Fill the buffer with noise
const data = noiseBuffer.getChannelData(0);
for (let i = 0; i < bufferSize; i++) {
  data[i] = Math.random() \* 2 - 1;
}
```

**Note:** Why -1 to 1? When outputting sound to a file or speakers, we need a number representing 0 dB full scale — the numerical limit of the fixed point media or DAC. In floating point audio, 1 is a convenient number to map to "full scale" for mathematical operations on signals, so oscillators, noise generators, and other sound sources typically output bipolar signals in the range -1 to 1. A browser will clamp values outside this range.

### Creating a buffer source

Now we have the audio buffer and have filled it with data; we need a node to add to our graph that can use the buffer as a source. We'll create an `AudioBufferSourceNode` for this, and pass in the data we've created:

```js
// Create a buffer source for our created data
const noise = new AudioBufferSourceNode(audioCtx, {
  buffer: noiseBuffer,
});
```

When we connect this through our audio graph and play it:

```js
noise.connect(audioCtx.destination);
noise.start();
```

You'll notice that it's pretty hissy or tinny. We've created white noise; that's how it should be. Our values are spread from -1 to 1, meaning we have peaks of all frequencies, which are actually quite dramatic and piercing. We *could* modify the function only spread values from 0.5 to -0.5 or similar to take the peaks off and reduce the discomfort; however, where's the fun in that? Let's route the noise we've created through a filter.

### Adding a biquad filter to the mix

We want something in the range of pink or brown noise. We want to cut off those high frequencies and possibly some lower ones. Let's pick a bandpass biquad filter for the job.

**Note:** The Web Audio API comes with two types of filter nodes: `BiquadFilterNode` and `IIRFilterNode`. For the most part, a biquad filter will be good enough — it comes with different types such as lowpass, highpass, and bandpass. If you're looking to do something more bespoke, however, the IIR filter might be a good option — see Using IIR filters for more information.

Wiring this up is the same as we've seen before. We create the `BiquadFilterNode`, configure the properties we want for it, and connect it through our graph. Different types of biquad filters have different properties — for instance, setting the frequency on a bandpass type adjusts the middle frequency. However, on a lowpass, it would set the top frequency.

```js
// Filter the output
const bandpass = new BiquadFilterNode(audioCtx, {
  type: "bandpass",
  frequency: bandHz,
});

// Connect our graph
noise.connect(bandpass).connect(audioCtx.destination);
```

### Noise user controls

On the UI, we'll expose the noise duration and the frequency we want to band, allowing the user to adjust them via range inputs and event handlers just like in previous sections:

```html
<label for="duration">Duration</label>
<input
 name="duration"
 id="duration"
 type="range"
 min="0"
 max="2"
 value="1"
 step="0.1" />

<label for="band">Band</label>
<input
 name="band"
 id="band"
 type="range"
 min="400"
 max="1200"
 value="1000"
 step="5" />
```

```js
let noiseDuration = 1;
const durControl = document.querySelector("#duration");
durControl.addEventListener(
  "input",
  (ev) => {
    noiseDuration = parseFloat(ev.target.value);
  },
  false,
);

let bandHz = 1000;
const bandControl = document.querySelector("#band");
bandControl.addEventListener(
  "input",
  (ev) => {
    bandHz = parseInt(ev.target.value, 10);
  },
  false,
);
```

### The final playNoise() function

Here's the entire `playNoise()` function:

```js
function playNoise(time) {
  const bufferSize = audioCtx.sampleRate \* noiseDuration; // set the time of the note

  // Create an empty buffer
  const noiseBuffer = new AudioBuffer({
    length: bufferSize,
    sampleRate: audioCtx.sampleRate,
  });

  // Fill the buffer with noise
  const data = noiseBuffer.getChannelData(0);
  for (let i = 0; i < bufferSize; i++) {
    data[i] = Math.random() \* 2 - 1;
  }

  // Create a buffer source for our created data
  const noise = new AudioBufferSourceNode(audioCtx, {
    buffer: noiseBuffer,
  });

  // Filter the output
  const bandpass = new BiquadFilterNode(audioCtx, {
    type: "bandpass",
    frequency: bandHz,
  });

  // Connect our graph
  noise.connect(bandpass).connect(audioCtx.destination);
  noise.start(time);
}
```

"Dial-up" — loading a sound sample
----------------------------------

It's straightforward enough to emulate phone dial (DTMF) sounds by playing a couple of oscillators together using the methods we've already used. Instead, we'll load a sample file in this section to look at what's involved.

### Loading the sample

We want to make sure our file has loaded and been decoded into a buffer before we use it, so let's create an `async` function to allow us to do this:

```js
async function getFile(audioContext, filepath) {
  const response = await fetch(filepath);
  const arrayBuffer = await response.arrayBuffer();
  const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
  return audioBuffer;
}
```

We can then use the `await` operator when calling this function, which ensures that we can only run subsequent code when it has finished executing.

Let's create another `async` function to set up the sample — we can combine the two async functions in a lovely promise pattern to perform further actions when this file is loaded and buffered:

```js
async function setupSample() {
  const filePath = "dtmf.mp3";
  const sample = await getFile(audioCtx, filePath);
  return sample;
}
```

**Note:** You can easily modify the above function to take an array of files and loop over them to load more than one sample. This technique would be convenient for more complex instruments or gaming.

We can now use `setupSample()` like so:

```js
setupSample().then((sample) => {
  // sample is our buffered file
  // …
});
```

When the sample is ready to play, the program sets up the UI, so it is ready to go.

### Playing the sample

Let's create a `playSample()` function similarly to how we did with the other sounds. This time we will create an `AudioBufferSourceNode`, put the buffer data we've fetched and decoded into it, and play it:

```js
function playSample(audioContext, audioBuffer, time) {
  const sampleSource = new AudioBufferSourceNode(audioContext, {
    buffer: audioBuffer,
    playbackRate,
  });
  sampleSource.connect(audioContext.destination);
  sampleSource.start(time);
  return sampleSource;
}
```

**Note:** We can call `stop()` on an `AudioBufferSourceNode`, however, this will happen automatically when the sample has finished playing.

### Dial-up user controls

The `AudioBufferSourceNode` comes with a `playbackRate` property. Let's expose that to our UI so that we can speed up and slow down our sample. We'll do that in the same sort of way as before:

```html
<label for="rate">Rate</label>
<input
 name="rate"
 id="rate"
 type="range"
 min="0.1"
 max="2"
 value="1"
 step="0.1" />
```

```js
let playbackRate = 1;
const rateControl = document.querySelector("#rate");
rateControl.addEventListener(
  "input",
  (ev) => {
    playbackRate = parseInt(ev.target.value, 10);
  },
  false,
);
```

### The final playSample() function

We'll then add a line to update the `playbackRate` property to our `playSample()` function. The final version looks like this:

```js
function playSample(audioContext, audioBuffer, time) {
  const sampleSource = new AudioBufferSourceNode(audioCtx, {
    buffer: audioBuffer,
    playbackRate,
  });
  sampleSource.connect(audioContext.destination);
  sampleSource.start(time);
  return sampleSource;
}
```

**Note:** The sound file was sourced from soundbible.com.

Playing the audio in time
-------------------------

A common problem with digital audio applications is getting the sounds to play in time so that the beat remains consistent and things do not slip out of time.

We could schedule our voices to play within a `for` loop; however, the biggest problem with this is updating while it is playing, and we've already implemented UI controls to do so. Also, it would be really nice to consider an instrument-wide BPM control. The best way to get our voices to play on the beat is to create a scheduling system, whereby we look ahead at when the notes will play and push them into a queue. We can start them at a precise time with the `currentTime` property and also consider any changes.

**Note:** This is a much stripped down version of Chris Wilson's A Tale Of Two Clocks (2013) article, which goes into this method with much more detail. There's no point repeating it all here, but we highly recommend reading this article and using this method. Much of the code here is taken from his metronome example, which he references in the article.

Let's start by setting up our default BPM (beats per minute), which will also be user-controllable via — you guessed it — another range input.

```js
let tempo = 60.0;
const bpmControl = document.querySelector("#bpm");

bpmControl.addEventListener(
  "input",
  (ev) => {
    tempo = parseInt(ev.target.value, 10);
  },
  false,
);
```

Then we'll create variables to define how far ahead we want to look and how far ahead we want to schedule:

```js
const lookahead = 25.0; // How frequently to call scheduling function (in milliseconds)
const scheduleAheadTime = 0.1; // How far ahead to schedule audio (sec)
```

Let's create a function that moves the note forwards by one beat and loops back to the first when it reaches the 4th (last) one:

```js
let currentNote = 0;
let nextNoteTime = 0.0; // when the next note is due.

function nextNote() {
  const secondsPerBeat = 60.0 / tempo;

  nextNoteTime += secondsPerBeat; // Add beat length to last beat time

  // Advance the beat number, wrap to zero when reaching 4
  currentNote = (currentNote + 1) % 4;
}
```

We want to create a reference queue for the notes that are to be played, and the functionality to play them using the functions we've previously created:

```js
const notesInQueue = [];

function scheduleNote(beatNumber, time) {
  // Push the note on the queue, even if we're not playing.
  notesInQueue.push({ note: beatNumber, time });

  if (pads[0].querySelectorAll("input")[beatNumber].checked) {
    playSweep(time);
  }
  if (pads[1].querySelectorAll("input")[beatNumber].checked) {
    playPulse(time);
  }
  if (pads[2].querySelectorAll("input")[beatNumber].checked) {
    playNoise(time);
  }
  if (pads[3].querySelectorAll("input")[beatNumber].checked) {
    playSample(audioCtx, dtmf, time);
  }
}
```

Here we look at the current time and compare it to the time for the following note; when the two match, it will call the previous two functions.

`AudioContext` object instances have a `currentTime` property, which allows us to retrieve the number of seconds after we first created the context. We will use it for timing within our step sequencer. It's extremely accurate, returning a float value accurate to about 15 decimal places.

```js
let timerID;
function scheduler() {
  // While there are notes that will need to play before the next interval,
  // schedule them and advance the pointer.
  while (nextNoteTime < audioCtx.currentTime + scheduleAheadTime) {
    scheduleNote(currentNote, nextNoteTime);
    nextNote();
  }
  timerID = setTimeout(scheduler, lookahead);
}
```

We also need a `draw()` function to update the UI, so we can see when the beat progresses.

```js
let lastNoteDrawn = 3;
function draw() {
  let drawNote = lastNoteDrawn;
  const currentTime = audioCtx.currentTime;

  while (notesInQueue.length && notesInQueue[0].time < currentTime) {
    drawNote = notesInQueue[0].note;
    notesInQueue.shift(); // Remove note from queue
  }

  // We only need to draw if the note has moved.
  if (lastNoteDrawn !== drawNote) {
    pads.forEach((pad) => {
      pad.children[lastNoteDrawn \* 2].style.borderColor = "var(--black)";
      pad.children[drawNote \* 2].style.borderColor = "var(--yellow)";
    });

    lastNoteDrawn = drawNote;
  }
  // Set up to draw again
  requestAnimationFrame(draw);
}
```

Putting it all together
-----------------------

Now all that's left to do is make sure we've loaded the sample before we can *play* the instrument. We'll add a loading screen that disappears when the file has been fetched and decoded. Then we can allow the scheduler to start using the play button click event.

```js
// When the sample has loaded, allow play
const loadingEl = document.querySelector(".loading");
const playButton = document.querySelector("#playBtn");
let isPlaying = false;
setupSample().then((sample) => {
  loadingEl.style.display = "none";

  dtmf = sample; // to be used in our playSample function

  playButton.addEventListener("click", (ev) => {
    isPlaying = !isPlaying;

    if (isPlaying) {
      // Start playing

      // Check if context is in suspended state (autoplay policy)
      if (audioCtx.state === "suspended") {
        audioCtx.resume();
      }

      currentNote = 0;
      nextNoteTime = audioCtx.currentTime;
      scheduler(); // kick off scheduling
      requestAnimationFrame(draw); // start the drawing loop.
      ev.target.dataset.playing = "true";
    } else {
      clearTimeout(timerID);
      ev.target.dataset.playing = "false";
    }
  });
});
```

Summary
-------

We've now got an instrument inside our browser! Keep playing and experimenting — you can expand on any of these techniques to create something much more elaborate.


Background audio processing using AudioWorklet - Web APIs

Background audio processing using AudioWorklet
==============================================

This article explains how to create an audio worklet processor and use it in a Web Audio application.

When the Web Audio API was first introduced to browsers, it included the ability to use JavaScript code to create custom audio processors that would be invoked to perform real-time audio manipulations. The drawback to `ScriptProcessorNode` was that it ran on the main thread, thus blocking everything else going on until it completed execution. This was far less than ideal, especially for something that can be as computationally expensive as audio processing.

Enter `AudioWorklet`. An audio context's audio worklet is a `Worklet` which runs off the main thread, executing audio processing code added to it by calling the context's `audioWorklet.addModule()` method. Calling `addModule()` loads the specified JavaScript file, which should contain the implementation of the audio processor. With the processor registered, you can create a new `AudioWorkletNode` which passes the audio through the processor's code when the node is linked into the chain of audio nodes along with any other audio nodes.

It's worth noting that because audio processing can often involve substantial computation, your processor may benefit greatly from being built using WebAssembly, which brings near-native or fully native performance to web apps. Implementing your audio processing algorithm using WebAssembly can make it perform markedly better.

High level overview
-------------------

Before we start looking at the use of AudioWorklet on a step-by-step basis, let's start with a brief high-level overview of what's involved.

1. Create module that defines an audio worklet processor class, based on `AudioWorkletProcessor` which takes audio from one or more incoming sources, performs its operation on the data, and outputs the resulting audio data.
2. Access the audio context's `AudioWorklet` through its `audioWorklet` property, and call the audio worklet's `addModule()` method to install the audio worklet processor module.
3. As needed, create audio processing nodes by passing the processor's name (which is defined by the module) to the `AudioWorkletNode()` constructor.
4. Set up any audio parameters the `AudioWorkletNode` needs, or that you wish to configure. These are defined in the audio worklet processor module.
5. Connect the created `AudioWorkletNode`s into your audio processing pipeline as you would any other node, then use your audio pipeline as usual.

Throughout the remainder of this article, we'll look at these steps in more detail, with examples (including working examples you can try out on your own).

The example code found on this page is derived from this working example which is part of MDN's GitHub repository of Web Audio examples. The example creates an oscillator node and adds white noise to it using an `AudioWorkletNode` before playing the resulting sound out. Slider controls are available to allow controlling the gain of both the oscillator and the audio worklet's output.

**See the code**

**Try it live**

Creating an audio worklet processor
-----------------------------------

Fundamentally, an audio worklet processor (which we'll refer to usually as either an "audio processor" or as a "processor" because otherwise this article will be about twice as long) is implemented using a JavaScript module that defines and installs the custom audio processor class.

### Structure of an audio worklet processor

An audio worklet processor is a JavaScript module which consists of the following:

* A JavaScript class which defines the audio processor. This class extends the `AudioWorkletProcessor` class.
* The audio processor class must implement a `process()` method, which receives incoming audio data and writes back out the data as manipulated by the processor.
* The module installs the new audio worklet processor class by calling `registerProcessor()`, specifying a name for the audio processor and the class that defines the processor.

A single audio worklet processor module may define multiple processor classes, registering each of them with individual calls to `registerProcessor()`. As long as each has its own unique name, this will work just fine. It's also more efficient than loading multiple modules from over the network or even the user's local disk.

### Basic code framework

The barest framework of an audio processor class looks like this:

```js
class MyAudioProcessor extends AudioWorkletProcessor {
  constructor() {
    super();
  }

  process(inputList, outputList, parameters) {
    // Using the inputs (or not, as needed),
    // write the output into each of the outputs
    // …
    return true;
  }
}

registerProcessor("my-audio-processor", MyAudioProcessor);
```

After the implementation of the processor comes a call to the global function `registerProcessor()`, which is only available within the scope of the audio context's `AudioWorklet`, which is the invoker of the processor script as a result of your call to `audioWorklet.addModule()`. This call to `registerProcessor()` registers your class as the basis for any `AudioWorkletProcessor`s created when `AudioWorkletNode`s are set up.

This is the barest framework and actually has no effect until code is added into `process()` to do something with those inputs and outputs. Which brings us to talking about those inputs and outputs.

### The input and output lists

The lists of inputs and outputs can be a little confusing at first, even though they're actually very simple once you realize what's going on.

Let's start at the inside and work our way out. Fundamentally, the audio for a single audio channel (such as the left speaker or the subwoofer, for example) is represented as a `Float32Array` whose values are the individual audio samples. By specification, each block of audio your `process()` function receives contains 128 frames (that is, 128 samples for each channel), but it is planned that *this value will change in the future*, and may in fact vary depending on circumstances, so you should *always* check the array's `length` rather than assuming a particular size. It is, however, guaranteed that the inputs and outputs will have the same block length.

Each input can have a number of channels. A mono input has a single channel; stereo input has two channels. Surround sound might have six or more channels. So each input is, in turn, an array of channels. That is, an array of `Float32Array` objects.

Then, there can be multiple inputs, so the `inputList` is an array of arrays of `Float32Array` objects. Each input may have a different number of channels, and each channel has its own array of samples.

Thus, given the input list `inputList`:

```js
const numberOfInputs = inputList.length;
const firstInput = inputList[0];

const firstInputChannelCount = firstInput.length;
const firstInputFirstChannel = firstInput[0]; // (or inputList[0][0])

const firstChannelByteCount = firstInputFirstChannel.length;
const firstByteOfFirstChannel = firstInputFirstChannel[0]; // (or inputList[0][0][0])
```

The output list is structured in exactly the same way; it's an array of outputs, each of which is an array of channels, each of which is a `Float32Array` object, which contains the samples for that channel.

How you use the inputs and how you generate the outputs depends very much on your processor. If your processor is just a generator, it can ignore the inputs and just replace the contents of the outputs with the generated data. Or you can process each input independently, applying an algorithm to the incoming data on each channel of each input and writing the results into the corresponding outputs' channels (keeping in mind that the number of inputs and outputs may differ, and the channel counts on those inputs and outputs may also differ). Or you can take all the inputs and perform mixing or other computations that result in a single output being filled with data (or all the outputs being filled with the same data).

It's entirely up to you. This is a very powerful tool in your audio programming toolkit.

### Processing multiple inputs

Let's take a look at an implementation of `process()` that can process multiple inputs, with each input being used to generate the corresponding output. Any excess inputs are ignored.

```js
process(inputList, outputList, parameters) {
  const sourceLimit = Math.min(inputList.length, outputList.length);

  for (let inputNum = 0; inputNum < sourceLimit; inputNum++) {
    const input = inputList[inputNum];
    const output = outputList[inputNum];
    const channelCount = Math.min(input.length, output.length);

    for (let channelNum = 0; channelNum < channelCount; channelNum++) {
      input[channelNum].forEach((sample, i) => {
        // Manipulate the sample
        output[channelNum][i] = sample;
      });
    }
  };

  return true;
}
```

Note that when determining the number of sources to process and send through to the corresponding outputs, we use `Math.min()` to ensure that we only process as many channels as we have room for in the output list. The same check is performed when determining how many channels to process in the current input; we only process as many as there are room for in the destination output. This avoids errors due to overrunning these arrays.

### Mixing inputs

Many nodes perform **mixing** operations, where the inputs are combined in some way into a single output. This is demonstrated in the following example.

```js
process(inputList, outputList, parameters) {
  const sourceLimit = Math.min(inputList.length, outputList.length);
  for (let inputNum = 0; inputNum < sourceLimit; inputNum++) {
    let input = inputList[inputNum];
    let output = outputList[0];
    let channelCount = Math.min(input.length, output.length);

    for (let channelNum = 0; channelNum < channelCount; channelNum++) {
      for (let i = 0; i < input[channelNum].length; i++) {
        let sample = output[channelNum][i] + input[channelNum][i];

        if (sample > 1.0) {
          sample = 1.0;
        } else if (sample < -1.0) {
          sample = -1.0;
        }

        output[channelNum][i] = sample;
      }
    }
  };

  return true;
}
```

This is similar code to the previous sample in many ways, but only the first output—`outputList[0]`—is altered. Each sample is added to the corresponding sample in the output buffer, with a simple code fragment in place to prevent the samples from exceeding the legal range of -1.0 to 1.0 by capping the values; there are other ways to avoid clipping that are perhaps less prone to distortion, but this is a simple example that's better than nothing.

Lifetime of an audio worklet processor
--------------------------------------

The only means by which you can influence the lifespan of your audio worklet processor is through the value returned by `process()`, which should be a Boolean value indicating whether or not to override the user agent's decision-making as to whether or not your node is still in use.

In general, the lifetime policy of any audio node is simple: if the node is still considered to be actively processing audio, it will continue to be used. In the case of an `AudioWorkletNode`, the node is considered to be active if its `process()` function returns `true` *and* the node is either generating content as a source for audio data, or is receiving data from one or more inputs.

Specifying a value of `true` as the result from your `process()` function in essence tells the Web Audio API that your processor needs to keep being called even if the API doesn't think there's anything left for you to do. In other words, `true` overrides the API's logic and gives you control over your processor's lifetime policy, keeping the processor's owning `AudioWorkletNode` running even when it would otherwise decide to shut down the node.

Returning `false` from the `process()` method tells the API that it should follow its normal logic and shut down your processor node if it deems it appropriate to do so. If the API determines that your node is no longer needed, `process()` will not be called again.

**Note:** At this time, unfortunately, Chrome does not implement this algorithm in a manner that matches the current standard. Instead, it keeps the node alive if you return `true` and shuts it down if you return `false`. Thus for compatibility reasons you must always return `true` from `process()`, at least on Chrome. However, once this Chrome issue is fixed, you will want to change this behavior if possible as it may have a slight negative impact on performance.

Creating an audio processor worklet node
----------------------------------------

To create an audio node that pumps blocks of audio data through an `AudioWorkletProcessor`, you need to follow these simple steps:

1. Load and install the audio processor module
2. Create an `AudioWorkletNode`, specifying the audio processor module to use by its name
3. Connect inputs to the `AudioWorkletNode` and its outputs to appropriate destinations (either other nodes or to the `AudioContext` object's `destination` property.

To use an audio worklet processor, you can use code similar to the following:

```js
let audioContext = null;

async function createMyAudioProcessor() {
  if (!audioContext) {
    try {
      audioContext = new AudioContext();
      await audioContext.resume();
      await audioContext.audioWorklet.addModule("module-url/module.js");
    } catch (e) {
      return null;
    }
  }

  return new AudioWorkletNode(audioContext, "processor-name");
}
```

This `createMyAudioProcessor()` function creates and returns a new instance of `AudioWorkletNode` configured to use your audio processor. It also handles creating the audio context if it hasn't already been done.

In order to ensure the context is usable, this starts by creating the context if it's not already available, then adds the module containing the processor to the worklet. Once that's done, it instantiates and returns a new `AudioWorkletNode`. Once you have that in hand, you connect it to other nodes and otherwise use it just like any other node.

You can then create a new audio processor node by doing this:

```js
let newProcessorNode = await createMyAudioProcessor();
```

If the returned value, `newProcessorNode`, is non-`null`, we have a valid audio context with its hiss processor node in place and ready to use.

Supporting audio parameters
---------------------------

Just like any other Web Audio node, `AudioWorkletNode` supports parameters, which are shared with the `AudioWorkletProcessor` that does the actual work.

### Adding parameter support to the processor

To add parameters to an `AudioWorkletNode`, you need to define them within your `AudioWorkletProcessor`-based processor class in your module. This is done by adding the static getter `parameterDescriptors` to your class. This function should return an array of `AudioParam` objects, one for each parameter supported by the processor.

In the following implementation of `parameterDescriptors()`, the returned array has two `AudioParam` objects. The first defines `gain` as a value between 0 and 1, with a default value of 0.5. The second parameter is named `frequency` and defaults to 440.0, with a range from 27.5 to 4186.009, inclusively.

```js
static get parameterDescriptors() {
  return [
   {
      name: "gain",
      defaultValue: 0.5,
      minValue: 0,
      maxValue: 1
    },
    {
      name: "frequency",
      defaultValue: 440.0,
      minValue: 27.5,
      maxValue: 4186.009
    }
  ];
}
```

Accessing your processor node's parameters is as simple as looking them up in the `parameters` object passed into your implementation of `process()`. Within the `parameters` object are arrays, one for each of your parameters, and sharing the same names as your parameters.

A-rate parameters

For a-rate parameters—parameters whose values automatically change over time—the parameter's entry in the `parameters` object is an array of `AudioParam` objects, one for each frame in the block being processed. These values are to be applied to the corresponding frames.

K-rate parameters

K-rate parameters, on the other hand, can only change once per block, so the parameter's array has only a single entry. Use that value for every frame in the block.

In the code below, we see a `process()` function that handles a `gain` parameter which can be used as either an a-rate or k-rate parameter. Our node only supports one input, so it just takes the first input in the list, applies the gain to it, and writes the resulting data to the first output's buffer.

```js
process(inputList, outputList, parameters) {
  const input = inputList[0];
  const output = outputList[0];
  const gain = parameters.gain;

  for (let channelNum = 0; channelNum < input.length; channelNum++) {
    const inputChannel = input[channelNum];
    const outputChannel = output[channelNum];

    // If gain.length is 1, it's a k-rate parameter, so apply
    // the first entry to every frame. Otherwise, apply each
    // entry to the corresponding frame.

    if (gain.length === 1) {
      for (let i = 0; i < inputChannel.length; i++) {
        outputChannel[i] = inputChannel[i] \* gain[0];
      }
    } else {
      for (let i = 0; i < inputChannel.length; i++) {
        outputChannel[i] = inputChannel[i] \* gain[i];
      }
    }
  }

  return true;
}
```

Here, if `gain.length` indicates that there's only a single value in the `gain` parameter's array of values, the first entry in the array is applied to every frame in the block. Otherwise, for each frame in the block, the corresponding entry in `gain[]` is applied.

### Accessing parameters from the main thread script

Your main thread script can access the parameters just like it can any other node. To do so, first you need to get a reference to the parameter by calling the `AudioWorkletNode`'s `parameters` property's `get()` method:

```js
let gainParam = myAudioWorkletNode.parameters.get("gain");
```

The value returned and stored in `gainParam` is the `AudioParam` used to store the `gain` parameter. You can then change its value effective at a given time using the `AudioParam` method `setValueAtTime()`.

Here, for example, we set the value to `newValue`, effective immediately.

```js
gainParam.setValueAtTime(newValue, audioContext.currentTime);
```

You can similarly use any of the other methods in the `AudioParam` interface to apply changes over time, to cancel scheduled changes, and so forth.

Reading the value of a parameter is as simple as looking at its `value` property:

```js
let currentGain = gainParam.value;
```

See also
--------

* Web Audio API
* Enter Audio Worklet (Chrome Developers blog)