Papers A
A-1- Challenges in the Development of an Easy-Accessed Mobile Phone Orchestra Platform
Anders Lind (Sounds like Lind)*; Björn Yttergren (Umeå); Håkan Gustafson (Studio With)
How can an easy-accessed Mobile Phone Orchestra (MPO)-platform be developed to facilitate high-end, flexible artistic expressions suitable for concert hall performances?. Moreover, What challenges and possibilities arises in the development process of a novel MPO-platform embracing school children as performers?. In this paper we discuss these questions, which were integrated in the development process of mobilephoneorchestra.com. In particular, we highlight the main challenge of developing a performance platform which both is mobile and artistically rewarding, but still easy accessed for school children. Mobilephoneorchestra.com is developed as a Web Audio API to enable large-scale concert hall performances of fixed polyphonic contemporary art music. A kind of philharmonic orchestra for electronic sounds, embracing school children as performers, using smart phones as instruments. Through an autoethnographic approach the research is carried out in multiple iterations, inspired by action research. In this paper we present findings from three iterations including performances of music involving MPO. School children/youths between ten- to seventeen years old were addressed as MPO-performers. The findings both reveal artistic challenges and possibilities with the platform. We specifically highlight the use of mobile music interfaces in combination with animated notation as a novel approach for a MPO-concept.
go to paper
A-2 - Build WebAudio and JavaScript Web Application using JSPatcher: A Web-based Visual Programming Editor
Shihong Ren (Université Jean Monnet)*; Laurent Pottier (CIEREC); Michel Buffa (Université Côte d'Azur, CNRS, INRIA, FRANCE)
Many visual programming languages (VPLs) such as Max or PureData provide a graphic canvas to allow developers to connect functions or data between them. This canvas, also known as a patcher, is basically a graph meant to be interpreted as dataflow computation by the system. Some VPLs are used for multimedia performance or content generation as the UI system is often an important part of the language. This paper presents a web-based VPL, JSPatcher, which allows not only to build audio graphs using the WebAudio API, but also to design graphically AudioWorklet DSPs with FAUST toolchain, or to create interactive programs with other language built-ins, Web APIs or any JavaScript modules.
A-3 - A Frontend for Adaptive Online Listening Tests
Johan Pauwels (Queen Mary University of London)*; Simon Dixon (Queen Mary University of London); Joshua D. Reiss (Queen Mary University of London)
A number of tools to create online listening tests are nowadays available. They provide an integrated platform consisting of a user-facing frontend and a backend to collect responses. These platforms provide an out-of-the box solution for setting up static listening tests, where questions and audio stimuli remain unchanged and user-independent. In this paper, we detail the changes we made to the webMUSHRA platform to convert it into a frontend for adaptive online listening tests. Some of the more advanced workflows that can be built around this frontend include session management to resume listening tests, server-based sampling of stimuli to enforce a certain distribution over all participants, and follow-up questions based on previous responses. The backends required for such workflows need a large amount of customisation based on the exact listining test specification, and are therefore deemed out of scope for this project. The proposed frontend therefore isn't meant as a replacement for the existing webMUSHRA platform, but as starting point to create custom listening tests. Nonetheless, a fair amount of the proposed changes are also beneficial for the creation of static listening tests.
A-4 - Glicol: A Graph-oriented Live Coding Language Developed with Rust, WebAssembly and AudioWorklet
Qichao Lan (University of Oslo)*; Alexander Refsum Jensenius (University of Oslo)
This paper introduces the new music live coding language Glicol (graph-oriented live coding language) and its web-based run-time environment. As the name suggests, this language is designed to represent directed acyclic graphs (DAG), using a syntax optimised for live music performances. The audio engine and the language interpreter are both developed with the Rust programming language. With the help of WebAssembly and AudioWorklet, this language can run in web browsers. It also supports co-performance with the support for collaborative editing. Taking advantages of the Rust programming language design, the run-time environment is both safe and efficient. Documentation and error handling messages can be accessed in the web browser. All in all, we see Glicol as an efficient and future-oriented language for collaborative text-based musicking.
A-5 - immaterial.cloud: Using peer-to-peer technologies for music
Tate Carson (Louisiana State University)*
immaterial.cloud is an immersive audiovisual installation that explores a possible networked future of peer-to-peer technologies, away from cloud computing. Participants experience the work via two to four smartphones placed in<br/>different locations in a room. As participants walk up to a phone, they see a representation of themselves through data. If the participant gets close enough, the phone triggers a change in the sound of immaterial.cloud and the other phones follow.
Papers B
B-1 - Putting Web Audio API to the test: Introducing WebAudioXML as a pedagogical platform
Hans Lindetorp (KMH)*; Kjetil Falkenberg (KTH Sound and Music Computing group)
Web technologies in general and Web Audio API in particular have a great potential as a learning platform for developing interactive sound and music applications. Earlier studies at the Royal College of Music in Stockholm have led to a wide range of student projects but have also indicated that there is a high threshold for novice programmers to understand and use Web Audio API. We developed the WebAudioXML coding environment to solve this problem, and added a statistics module to analyze student works. The current study is the first presentation and evaluation of the technology. Three classes of students with technical repsectively artistic background participated through online courses by building interactive, sound-based applications. We analysed the source code and self-reflective reports from the projects to understand the impact WebAudioXML has on creativity and the learning process. The results indicate that WebAudioXML can be a useful platform for eaching and learning how to build online audio applications. The platform makes mapping between user interactions and audio parameters accessible for novice programmer and supports artists in successfully realizing their design ideas. We show that templates can be a great help for the students to get started but also a limitation for them to expand ideas beyond the presented scope.
B-2 - JackTrip-WebRTC: Networked music experiments with PCM stereo audio in a Web browser
Matteo Sacchetto (Politecnico di Torino); Antonio Servetti (Politecnico di Torino)*; Chris Chafe (Stanford University)
A large number of web applications are available for videoconferencing and those have been very helpful during the lockdown periods caused by the COVID-19 pandemic. However, none of these offer high fidelity stereo audio for music performance, mainly because the current WebRTC RTCPeerConnection standard only supports compressed audio formats. This paper presents the results achieved implementing 16-bit PCM stereo audio transmission on top of the WebRTC RTCDataChannel with the help of Web Audio and AudioWorklets. Several measurements with different configurations, browsers, and operating systems are presented. They show that, at least on the loopback network interface, this approach can achieve better quality and lower latency than using RTCPeerConnection, i.e., latencies as low as 50-60~ms have been achieved on MacOS.
B-3 - Improvisation in Isolation: Quarentena Liv(r)e and Noise Symphony with the Playsound online music making tool
Ariane S Stolfi (Universidade Federal do Sul da Bahia)*; Diogo Pereira (Instituto Federal de Educação, Ciência e Tecnologia da Bahia); Mathieu Barthet (QMUL)
In this paper, we describe artistic practices with the web-based music making tool Playsound.space held over the last two years since the inception of the platform. After completing a first design phase, which is documented in previous publications, Playsound has been regularly used by the first author as her main musical instrument to perform in live improvisation contexts. The tool proved to be especially useful during the quarantine period due to Covid-19 in Brazil, by enabling the musician (i) to take part in performances with other musicians through online gatherings, and (ii) to compose solo pieces from home leveraging crowd-sourced sounds. We review these endeavours and provide a critical analysis exploring some of the benefits of online music making and related challenges yet to tackle. The use of Playsound "in vivo", in real artistic practices outside the laboratory, has enabled us to uncover playing strategies and user interface improvements which can inform the design of similar web-based music making tools.
B-4 - Choir Singers Pilot – An online platform for choir singers practice
Matan Gover (Voctro Labs)*; Álvaro Sarasúa (Voctro Labs); Hector Parra (Voctro Labs); Jordi Janer (Voctro Labs); Oscar Mayor (Voctro Labs); Helena Cuesta (Universitat Pompeu Fabra); Maria Pilar Pascual (Universitat Pompeu Fabra); Aggelos Gkiokas (Universitat Pompeu Fabra); Emilia Gomez (Universitat Pompeu Fabra)
We present the Choir Singers Pilot, a web-based system that assists choir singers in their individual learning and practice. Our system is built using modern web technologies and provides singers with an interactive view of the musical score along with an aligned audio performance created using state-of-the-art singing synthesis technology. The Web Audio API is used to dynamically mix the choir voices and give users control over sound parameters. In-browser audio latency compensation is used to keep user recordings aligned to the reference music tracks. The pitch is automatically extracted from user recordings and can then be analyzed using an assessment algorithm to provide intonation ratings to the user. The system also facilitates communication and collaboration in choirs by enabling singers to share their recordings with conductors and receive feedback. Our work, in the larger scope of the TROMPA project, aims to enrich musical activities and promote use of digital music resources. To that end, we also synthesize thousands of public-domain choir scores and make them available in a searchable repository alongside relevant metadata for public consumption.
B-5 - Online Audio Editor in Freesound
Roberto Pérez Sánchez (Universidad Pompeu Fabra)*; Frederic Font (Music Technology Group – Universitat Pompeu Fabra)
Freesound has a huge community dedicated to upload and share sounds online for other users to use those sounds in their productions. In some cases, the sounds will need to be transformed and users will have to go in their devices and edit that sound in order to get the exact version that fits their production the most. In this paper, an audio editor is developed and added into the Freesoud environment in order to have those two steps together in the same workflow. This approach will add a huge value to the Freesound experience and will differentiate it from its competitors. For the development of this editor, different Javascipt and HTML frameworks have been used, like Wavesurfer (based on Web Audio API) or Bootstrap, always trying to keep the interface and functionalities accessible and easy-to-use to a general audience. Finally, two evaluation methods have been developed, the first in order to get some general feedback about the use of the editor and the second one to be sure that all its different functionalities behave as expected. The first evaluation showed the need of a simpler interface and functionalities, which was developed and then tested using the second evaluation approach, showing that all different actions, filters and effects worked as expected.
Papers C
C-1 - CosmoNote: A Web-based Citizen Science Tool for Annotating Music Performances
Lawrence Fyfe (Centre National de la Recherche Scientifique–UMR9912 / STMS Laboratoire (IRCAM))*; Daniel Bedoya (IRCAM); Corentin Guichaoua (STMS, IRCAM, Sorbonne Université); Elaine Chew (CNRS-UMR9912/STMS IRCAM, Paris, France)
CosmoNote is a web-based citizen science tool for annotating musical structures, with a focus on structures created by the performer during expressive musical performance. The software interface enables the superimposition of synchronized discrete and continuous information layers which include note data representations, audio features such as loudness and tempo, and score features such as harmonic tension in a visual and audio environment. The tools provide the means for users to signal performance decisions such as segmentation and prominence using boundaries of varying strengths, regions, comments, and note groupings. User-friendly interaction features have been built in to facilitate ease of annotation; these include the ability to zoom in, listen to, and mark up specific segments of music. The data collected will be used to discover the vocabulary of performed music structures and to aid in the understanding of expressive choices and nuances.
C-2 - Play the place: Web-based experience of an architectural space using 3D audio
Clifford M Manasseh (Queen Mary University Of London)*; Stephan Zimmerli (Architect and Musician); Mathieu Barthet (QMUL)
Assessing the acoustics of an architectural space before its physical construction is of interest to architects and structural engineers. In this work, we present a browser-based interactive app enabling sonic interaction in a virtual environment modeled after a small studio architectural design for music ideation and creation made by architects Stephan &amp; Eric Zimmerli, called 'Studiolo'. We first describe our ideation and design process involving pilot testing in the New Atlantis online environment for a collaborative sound and music experience. We then describe the implementation of the 'Studiolo' web app prototype supported by Javascript packages (Tone, Resonance Audio, Three, and Tween) and the Web Audio API. Room acoustics is modeled using the EVERTims framework providing a real-time auralization engine for binaural room impulse response approximation. A critical analysis by one of the architects highlights the importance of multimodal factors, from visual mapping textures and lighting quality to more sensitive and responsive timbres addressing the "tactile dimension" of music creation.
C-3 - Collaborative Music Creation and Performance with Soundcool Online
Roger Dannenberg (School of Computer Science, Carnegie Mellon University)*; Huan Zhang (Carnegie Mellon University); Amit Meena (Indian Institute of Technology); Ankitkumar Joshi (University of Pittsburgh); Amey Patel (Indian Institute of Technology); Jorge Sastre (Universitat Politecnica de Valencia)
Soundcool Online is a Web Audio re-implementation of the original Max/MSP implementation of Soundcool, a system for collaborative music creation. Soundcool has many educational applications, and because Linux has been adopted in many school systems, we turned to Web Audio to enable Soundcool to run on Linux as well as many other platforms. An additional advantage of Soundcool Online is the elimination of a large download, allowing beginners to try the system more easily. Another advantage is the support for sharing provided by a centralized server, where projects can be stored and accessed by others. A cloud-based server also facilitates collaboration at a distance where multiple users can control the same project. In this scenario, local sound synthesis provides high-quality sound without the large bandwidth requirements of shared audio streams. Experience with Web Audio and latency measurements are re- ported.
C-4 - Essentia TensorFlow Models for Audio and Music Processing on the Web
Albin Correya (Universitat Pompeu Fabra); Pablo Alonso-Jiménez (Universitat Pompeu Fabra); Jorge Marcos-Fernández (Universitat Pompeu Fabra); Xavier Serra (Universitat Pompeu Fabra ); Dmitry Bogdanov (Universitat Pompeu Fabra)*
Recent advances in web-based machine learning (ML) tools empower a wide range of application developers in both industrial and creative contexts. The availability of pre-trained ML models and JavaScript (JS) APIs in frameworks like TensorFlow.js enabled developers to use AI technologies without demanding domain expertise. Nevertheless, there is a lack of pre-trained models in web audio compared to other domains, such as text and image analysis. Motivated by this, we present a collection of open pre-trained TensorFlow.js models for music-related tasks on the Web. Our models currently allow for different types of music classification (e.g., genres, moods, danceability, voice or instrumentation), tempo estimation, and music feature embeddings. To facilitate their use, we provide a dedicated JS add-on module essentia.js-model within the Essentia.js library for audio and music analysis. It has a simple API, enabling end-to-end analysis from audio input to prediction results on web browsers and Node.js. Along with the Web Audio API and web workers, it can also be used to build real-time applications. We provide usage examples, discuss possible use-cases, and report benchmarking results.
C-5 - kilobeat: low-level collaborative livecoding
Ian J Clester (Georgia Institute of Technology)*
This paper presents kilobeat, a collaborative, web-based, DSP-oriented livecoding platform. Players make music together by writing short snippets of code. Inspired by the practices of bytebeat, these snippets are low-level expressions representing digital audio signals. Unlike existing platforms, kilobeat does not adapt existing livecoding languages or introduce a new one: players write JavaScript expressions (using standard operators and math functions) to generate samples directly. This approach reduces the amount of background knowledge required to understand players' code and makes kilobeat amenable to synthesis and DSP pedagogy. To facilitate collaboration, players can hear each other's audio (distributed spatially in a virtual room), see each other's code (including edits and run actions), and depend on each other's output (by referencing other players as variables). Additionally, performances may be recorded and replayed later, including all player actions. For accessibility and ease of sharing, kilobeat is built on Web Audio and WebSockets.
Papers D
D-1 - A Time-Travel Debugger for Web Audio Applications
Andrew Thompson (Queen Mary University of London)*; Gyorgy Fazekas (Queen Mary University of London); Geraint A. Wiggins (Vrije Universiteit Brussel)
Developing real-time audio applications, particularly those with an element of user interaction, can be a difficult task. When things go wrong, it can be challenging to locate the source of a problem when many parts of the program are connected and interacting with one another in real-time.
We present a time-travel debugger for the Flow Web Audio framework that allows developers to record a session interacting with their program, playback that session with the original timing still intact, and step through individual events to inspect the program state at any point in time. In contrast to the browser's native debugging features, audio processing remains active while the time-travel debugger is enabled, allowing developers to listen out for audio bugs or unexpected behaviour.
We describe three example use-cases for such a debugger. The first is error reproduction using the debugger's JSON import/export capabilities to ensure the developer can replicate problematic sessions. The second is using the debugger as an exploratory aid instead of a tool for error finding. Finally, we consider opportunities for the debugger's technology to be used by end-users as a means of recording, sharing, and remixing ideas. We conclude with some options for future development, including expanding the debugger's program state inspector to allow for in situ data updates, visualisation of the current audio graph similar to existing Web Audio inspectors, and possible methods of evaluating the debugger's effectiveness in the scenarios described.
D-2 - Surfing with Narcissus: Updating Thea Musgrave's work for Solo Clarinet
Luan Luiz Gonçalves (Federal University of São João del-Rei (UFSJ))*; Rafael Andrade (Federal University of São João del-Rei (UFSJ)); Iura Sobrinho (Federal University of São João del-Rei (UFSJ)); Flávio Luiz Schiavoni (Federal University of São João del-Rei)
Narcissus is a work created in 1987 by Thea Musgrave for clarinet or flute and delayed audio, using a delay box. In order to recreate the piece using modern technology, our research group developed a web application to emulate the functions of the original equipament in Narcissus. "Surfing with Narcissus" is our report about the experience of recreating Musgrave's work using web audio and web music APIs.
D-3 - Composing and Improvising Using Sound Content-Based Descriptive Filtering
Austin Franklin (LSU)*; Dylan Burchett (Louisiana State University); William Thompson (Louisiana State University); Austin A Franklin (Louisiana State University)
The Freesound Player is a digital instrument developed in Max MSP that uses the Freesound API to make requests for as many as 16 sound samples which are filtered based on the sounds content. Once the samples are returned they are loaded into buffers and can be performed using a MIDI controller and processed in a variety of ways. The filters implemented will be discussed and demonstrated using three music compositions by the authors, along with considerations for composing and improvising using sound content-based descriptive filtering.
D-4 - Functional Reactive Programming and the Web Audio API
Mike Solomon (Meeshkan)*
Functional Reactive Programming (FRP) is a way to model discrete and continuous signals using events, which carry information corresponding to a precise moment in time, and behaviors, which represent time-varying values. This paper shows how the behavior pattern can be used to build sample-accurate interactive audio that blends the WebAudio API with other browser-based APIs, such as mouse events and MIDI events. It will start by presenting a brief history of FRP as well as definitions of polymorphic behavior and event types. It will then discuss the principal challenges of applying the behavior pattern to WebAudio, including named audio units, front-loaded evaluation, and scheduling precise temporal events using different clocks.
Talks A
A-1 - Rapidly Prototyping MIDI Controllers with Web MIDI
Max Vogler (Google)*
MIDI (Musical Instrument Digital Interface) is the standard communication protocol to connect audio hardware and software. The protocol is commonly used to send messages from hardware MIDI controllers to software that controls music synthesis and playback. Although a variety of hardware and software MIDI controllers exist, they typically use traditional, skeuomorphic input modes like keys, buttons, faders, and knobs. Since 2015, web browsers started supporting this protocol through the Web MIDI API, opening up an enormous untapped potential of integrations.
This talk guides viewers through quickly prototyping web-based MIDI controllers utilizing the power of web browsers and the JavaScript ecosystem. These prototypes showcase innovative ways of controlling audio software by sending MIDI messages from the web browser, e.g. using a phone's touchscreen with pressure detection as 3-d input to control audio effects.
The talk consists of a series of example prototypes, written in plain JavaScript and running in Chrome. Prior knowledge of JavaScript or the MIDI protocol is not required, but helpful in fully understanding the examples. The first example simply sends a single MIDI message from the browser to the audio software. Then, examples ramp up in interactivity and complexity, e.g. controlling audio effects through machine-learning models. The important parts of each prototype's code fit onto 1-2 slides and are followed by a live demonstration. The slides will be written in HTML and JavaScript themselves to allow neatless transitions from static slides to live examples. Every example contains a link to a fully-functional, open-source prototype (see https://github.com/max-vogler/midi) that viewers can run right in their browser.The talk aims to excite the audience about: * Unlocking novel input sources for audio software by using the web browser and ecosystem. * Iterating through prototypes quickly by using vanilla JavaScript with high-level APIs and libraries.
go to talk
A-2 - EMDM Online Concerts in the Pandemic: Bringing the Live to Live Streaming
Jesse Allison (Louisiana State University)*
To overcome the deficit of live events in 2020 and a lack of feeling of live performance through streaming, a trio of concerts were created using the web as a medium for performance. The first concert was an attempt to infuse live event dynamics into a series of recorded pieces. The second incorporated live performances streamed in novel ways to the audience like 360 degree video and a composite of browser based and live stream media. The third was done entirely live with 11 performers located in different spaces. <br/><br/>This talk will cover the novelty of using a website and distributed performance tools as a platform for delivering live performance and reintroducing the audience/performer connection in mediated performance.
Talks B
B-1 - Antropoloops online music remix tool for education: A Web Audio API Case Study
Daniel Gómez (Antropoloops)*
In this talk we will explain why and how we used modern web technologies, and specifically, the Web Audio API to build a creative digital tool intended to be used in the entry levels of our educational system (primary and secondary education). We are Rubén Alonso (artist and co-creator of "Antropoloops" project: a creative approach to ethnomusicology, where musical expressions from different cultures and times come into dialogue through electronic music) and Daniel Gómez (musician and software developer).We are part of a team made up of artists, educators and technicians, that asked ourselves how to combine sound collage, body movement and electronics to celebrate and engage cultural diversity in educational environments. Over the last three years, in collaboration with Carasso Foundation, we have developed both a methodology and an online remixing tool (https://play.antropoloops.com) that aims to explore the potential of music remixing as a way to work towards cultural inclusion at schools.
B-2 - A multitrack reactive music player for the browser
Dani Martín *
Accompanying systems provide musical accompaniment to a human performer. These systems can adapt to music contextual changes like the tempo or the tonality, which makes them an interesting tool for music students. In this paper I describe a browser-based multitrack reactive music player. This player enables bringing up automatic accompanying systems to the browser. The player is implemented in TypeScript, using the web audio API and MIDI.js soundfonts.
Talks C
C-1 - Declarative Web Audio API
Alexander Inkin (Tinkoff)*
In this talk, we will explore differences between imperative and declarative approach to Web Audio API. We will see what it takes to turn Web Audio API declarative with dependency injection pattern also known as inversion of control and what benefits it brings. We will create basic proof of concept using plain JavaScript and compare it to more advanced code based on Angular framework. All these techniques will help us write reusable structures and program music with Web Audio API. On top of that, we will look at Web MIDI API and create playable synthesizer using same declarative approach so we can perform music live. We will also touch on how using this knowledge and a little bit of web assembly we can create a Guitar Hero-like and Karaoke game which can be played directly in the browser, even on modern smartphones.
C-2 - Bringing sport fans back into the arena with web technology
Christoph Guttandin (Media Codings)*
From it's very beginning the Web Audio conference was a place which showcased systems using web technology to enable remote audience participation. Numerous talks, workshops and live performances were dedicated to this topic. But up until recently this was considered to be a nice add-on for live events outside of the Web Audio world. However the current pandemic changed this drastically.
Many events can't be attended in person anymore. This is true for small concerts and conferences but also for usually well attended sports events. In many cases remote participation became the only possible form of participation. For sports events this imposes a special problem since the reactions and chants from the stands are considered to be a part of the experience. Therefore many TV stations choose to play artificial background noise when broadcasting sports events without any onsite spectators.
I would like to present a system which uses web technologies to bring back the voices of sport fans into the arenas. It builds up on common rituals that many fans are used to. It's meant to be as less obtrusive as possible. It works by sending the announcements of the pa announcer to the fans at home. They can respond as they are used to. The responses get recorded as well and sent back. In the arena the players will hear the slightly delayed announcement together with the responses from their fans. The system is very scalable and can be used by as many people as would otherwise be in the stadium.
Talks D
D-1 - Testing MIDI applications
Sema Kachalo *
Testing is an important part of software development process.We introduce three open source tools to facilitate testing MIDI applications. midi-test ( https://github.com/jazz-soft/midi-test )This tool allows to open real MIDI ports (visible by all MIDI applications), and manipulate them via JavaScript: connect, disconnect, emit MIDI messages, monitor the received MIDI messages.
Combined with JZZ.jz, it can simulate Web MIDI API as well. web-midi-test ( https://github.com/jazz-soft/web-midi-test ) This is a fake Web MIDI API. Its ports a not visible by real MIDI applications, so they can be used only from JavaScript code. web-midi-test is useful for writing unit tests that don't require browser, and for modeling corner cases, i.g.: port is visible but is taken by another application, user allowed MIDI, but did not allow SysEx, etc… Both midi-test and web-midi-test share the same API that allow running both flavors of the same test. test-midi-files ( https://github.com/jazz-soft/test-midi-files ) This framework allows producing MIDI files from readable JavaScript code.
All above testing tools have been successfully used by the Open Source community.
D-2 - Collab-Hub: A system for creating collaborative telematic music performances with web-based instruments and creative development platforms
Anthony T Marasco (University of Texas Rio Grande Valley)*; Nick Hwang (University of Wisconsin-Whitewater); Eric Sheffield (Miami University)
Collab-Hub is a networking tool for sharing data across the internet for multimedia collaboration. Utilizing the Max graphical coding environment and its integrated Node.js functionality, Collab-Hub allows for multiple users to send data easily and remotely so they can collaborate with that data in real-time. <br/><br/>In this talk, we discuss the design of the Collab-Hub framework and provide examples of using it to create scalable and reconfigurable interaction layouts between Max patches and web-based instruments/interfaces. An analysis of pertinent historical precedents highlights the advantages Collab-Hub provides to artists who have little to no web development experience or to those who may be new to creating telematic and remote performance environments. A showcase of works created with Collab-Hub demonstrates the wide variety of artistic endeavors made possible through the framework.
Demo 1 - Live voice transformations for web conferencing using the browser
Gerard Llorach (University of Oldenburg)*; Mattes Ohlenbusch (University of Oldenburg); Thomas Brand (University of Oldenburg)
This demo consists of two elements: the web technology for real-time vocoding and its application for live voice transformation in web conferencing. This is an interesting approach, now more than ever in times of COVID-19, for modifying your speech in virtual conferences. We present a software-based solution that should be available to anyone doing web conferencing.
go to demo
Demo 2 - open.unmix.app - towards audio separation on the edge
Fabian-Robert Stöter (Inria, France)*; Maria Clara Machry Jacintho (Universidade Federal do Rio Grande do Sul); Delton de Andrade Vaz (Universidade Federal do Rio Grande do Sul); Stefan Uhlich (Sony European Technology Center); Yuki Mitsufuji (Sony Corporation); Antoine Liutkus (Inria)
Audio separation has recently become a mature technology, with state of the art reaching performance that permits wide audience application. While recent research continues to develop sophisticated methods to make results even better, we present our recent work in bringing the reference open-unmix model to the edge. This results in open.unmix.app, a web-based platform for audio separation. We demonstrate two contributions: First, we report on the difficulties of bringing a state-of-the-art deep learning-based model to javascript using TensorFlow.js including all necessary pre-and postprocessing. Solving them enables performing separation and enhancement of music and speech signals directly in the browser. Second, we present a platform that exploits this web-separation technology and allows users to share separated multitrack audio on the web using a custom, embeddable multi-track player.
Artwork 1 - Generative Music in the Web Audio API
Paul A Paroczai (Gold Saucer Studios)*
While there are many examples of interactive applications being built with the Web Audio API – from new and unique synthesis instruments, to emulators of popular signal processors, to audio engines for games – there are remarkably few that make use of the API's ability to implement systems which autonomously generate music. The work presented at the link below is a collection of six pieces in which all sounds are synthesized in real time: https://www.paulparoczai.net/#/webaudio/
go to artwork
Artwork 2 - immaterial.cloud
immaterial.cloud is an immersive audiovisual installation that explores a possible networked future of peer-to-peer technologies, away from the cloud. Participants experience the work via two to four smartphones placed in different locations in a room. As participants walk up to a phone, they see a representation of themselves through data. If the participant gets close enough, the phone triggers a change in the sound of immaterial.cloud and the other phones follow.
Artwork 3 - LudoTune
Dylan Turner (Dylan Turner)*
LudoTune is a browser-based musical toy that allows users to create music in a 3D environment by building cube-based structures. Users' creations can range from simple loops, to strange musical sculptures, to entire songs involving hundreds of cubes.
Artwork 4 - Radio-active Monstrosities
Angeliki Diakrousi *
In this paper, I describe the artwork Radioactive Monstrosities, a web-audio interface that addresses ways of listening to collective voices and certain female-sounding voices that are perceived as inappropriate or annoying — because of the quality of their sound, their gender, the medium's distortions but also stereotypes and collective memories that they often awake. These are verbal expressions that have been associated with forms of 'monstrosity' since ancient times. Visitors of the page are invited to record themselves and choose a type of distortion to participate in, forming new imaginaries around technologically-mediated voices, which through their technical 'monstrosity', can reveal other forms of speech.
Performances A
A-1 -Noise Symphony
Ariane S Stolfi (Universidade Federal do Sul da Bahia)*
Noise symphony is part of a series of pieces created from a selection of 690 sound sample taken by the author from https://freesound.org database using http://playsound.space software, from a query using the word "noise", that retrieved around 37000 results. Sound were selected based on their timber morphology, following a criteria of apparent visual difference. In this piece, sounds will be played together with solo extended vocal techniques
go to performance
A-2 - Aerial Glass: A Live Browser and Aerial Silk Performance
Paige Brown Jarreau; Jesse Allison (Louisiana State University)*; Austin Franklin (LSU); Nicholas Erickson (LSU)
Aerial Glass is a real-time performance, delivered through a website, of aerial silk acrobatics composited with a live browser performance of various video clips and audio composition. It uses the NexusHub framework to distribute a live in-browser performance of pre-rendered, transparent video clips, audio files, and web audio effects to everyone else visiting the website. A live physical performance on aerial silks is recorded in front of a green screen and live streamed to the website to be composited with the browser performance. <br/><br/>We propose a live performance of the work during the Web Audio Conference with performers at LSU, distributed to a worldwide audience through a website.
A-3 - AIrgots: Live Coding Performance
Francisco Bernardo (University of Sussex)*; Chris Kiefer (University of Sussex); Thor Magnusson (Music Department, University of Sussex)
We present a hybrid live coding musical performance using the latest version of Sema, the live coding language design and performance playground for the Web, with a custom Web Audio API signal engine. We explore machine learning, bespoke languages and interfaces as creative material in musical performance. Bernardo, Kiefer and Magnusson will collaborate as an ensemble of three, with networked and synchronized instances of Sema, in a performance which combines elements of improvisation and machine agency.
A-4 - SHP of Theseus
Anthony T Marasco (University of Texas Rio Grande Valley)*; Nick Hwang (University of Wisconsin-Whitewater); Eric Sheffield (Miami University))
SHP of THSEUS is an audiovisual composition we worked on collectively, drawing inspiration from the Greek myth in which, while on his adventures, Theseus would slowly replace parts of his ship until every part was eventually replaced. This is a philosophical quandary of identity. We navigate this concept through the use of Collab-Hub and our individual performance setups, in combination with the score.<br/>The composition uses a series of 10 graphical images, selected at random, that are sequentially presented to each performer. These graphical images are categorized as either 'sound' or 'control'. When performers receive a 'sound' image, they interpret their image and contribute sonically. When performers receive a 'control' image, they interpret their image and send control data to the performers which affect the gestures performed by others. As we alter and perform one another's patches over time, each of the individual performers relinquish control over some aspects of their own performance, even as they continue to steer forward with their interpretation of the score
Performances B
B-1 - Floop Jam
Gerard Roma (University of Huddersfield)*; Frederic Font (Music Technology Group – Universitat Pompeu Fabra); Antonio Ramires (Universitat Pompeu Fabra)
This submission proposes a web-based collaborative performance using Floop, a loop jamming system based on the Freesound database. The performers meet in a virtual room with a collection of loops that have been matched for rhythm, and organized according to timbre similarity. Their exploration develops as a DJ session, which is visible to the audience through the projected interface.
B-2 - Live Coding with Crowdsourced Sounds and A Virtual Agent Companion
Anna Xambó (De Montfort University)*
This performance combines machine learning algorithms with music information retrieval techniques to retrieve crowdsourced sounds from the online database Freesound.org. The use of a live coder virtual agent companion complements a human live coder in her/his practice. The core themes of legibility, agency and negotiability in performance are researched through the collaboration between the human live coder, the virtual agent live coder and the audience.
Performances C
C-1 - Collaborative Live Coding with Glicol Music Programming Language
Glicol is a graph-oriented live coding language developed with Rust, WebAssembly and AudioWorklet. This language can run in its web-based IDE that supports collaborative coding. In this performance, we invite the participants from the Glicol workshop to join the performance both at the scene or virtually. Each performer will be given the opportunity to write at least one node chain in the audio graph, and the first author will be in charge of when to execute the code. The music style will be improvised experimental/ambient techno.