BACKGROUND

In the music industry, composers or other artists often create synthesized orchestral music, commonly referred to as a “mock up.” A mock up is produced by an artist programming a digital score which can be played to produce the sound of multiple instruments. To produce this score, the mock up artist will often program not only the musical notes that the synthesized, or “virtual,” instruments are to play, but may also program other musical properties for the notes such as tone (also called timbre), vibrato and loudness.

Mock ups may be utilized in various ways in the music industry. In some cases, a mock up may be produced as a form of musical prototype so that an artist can listen to a musical composition before assembling live musicians to play the same composition. In some cases, a mock up may be played along with live musicians so that the combined sound simulates the sound that would be produced by a greater number of live musicians than are actually present.

SUMMARY

According to some aspects, a computer-implemented method of controlling the expressive behavior of a virtual musical instrument is provided, the method comprising obtaining an audio recording of a live musician playing an instrument, and generating, using at least one processor, at least part of a digital score representing audio of a virtual instrument having dynamic expression based on the audio recording, said generating comprising determining, for a first time position within the audio recording and based on the audio recording, a first value of an expression parameter, wherein the expression parameter is indicative of dynamic expression of the live musician, generating a first value of a control parameter based at least in part on the determined first value of the expression parameter, wherein the control parameter controls one or more aspects of the playback of audio samples for the virtual instrument, determining, for a second time position within the audio recording and based on the audio recording, a second value of the expression parameter, and generating a second value of the control parameter based at least in part on the determined second value of the expression parameter.

According to some embodiments, the method further comprises storing the first value of the control parameter within the digital score at one or more time positions based on the first time position, and storing the second value of the control parameter within the digital score at one or more time positions based on the second time position.

According to some embodiments, the method further comprises generating a first control message for the first time position within the audio recording, the first control message comprising an indication of the control parameter and an indication of the first value of the control parameter, and generating a second control message for the second time position within the audio recording, the second control message comprising an indication of the control parameter and an indication of the second value of the control parameter.

According to some embodiments, the method further comprises providing the first control message and the second control message to a sample library.

According to some embodiments, the method further comprises selecting, by the sample library from a plurality of audio samples, a first audio sample and a first value for at least one playback control of the first audio sample based at least in part on the first control message, and selecting, by the sample library from the plurality of audio samples, a second audio sample and a second value for the at least one playback control, different from the first value, of the second audio sample based at least in part on the second control message.

According to some embodiments, the audio recording of the live musician is obtained as a stream of audio data and the at least part of the digital score is generated in real-time based on the stream of audio data.

According to some embodiments, generating the at least part of the digital score is further based on sensor data generated by the live musician playing the instrument, the sensor data being produced concurrently with the audio recording.

According to some embodiments, determining the first value of an expression parameter comprises determining a peak amplitude of the audio recording at the first time position.

According to some embodiments, determining the first value of an expression parameter comprises determining a predetermined mixture of the peak amplitude of the audio recording at the first time position and a root mean square of the peak amplitude of the audio recording at the first time position.

According to some embodiments, generating the first value of the control parameter based at least in part on the determined first value of the expression parameter comprises scaling the first value of the expression parameter to an integer value within a predetermined range.

According to some embodiments, the predetermined range is between 0 and 127.

According to some embodiments, generating the first value of the expression parameter comprises determining a centroid of a frequency spectrum of the audio recording at the first time position.

According to some embodiments, generating the first value of the expression parameter comprises applying an offset to a measurement of the audio recording, the offset being selected based at least in part on a frequency value of the determined centroid.

According to some aspects, a non-transitory computer-readable medium is provided comprising instructions that, when executed by at least one processor, perform a method of controlling the expressive behavior of a virtual musical instrument, the method comprising obtaining an audio recording of a live musician playing an instrument, and generating, using at least one processor, at least part of a digital score representing audio of a virtual instrument having dynamic expression based on the audio recording, said generating comprising determining, for a first time position within the audio recording and based on the audio recording, a first value of an expression parameter, wherein the expression parameter is indicative of dynamic expression of the live musician, generating a first value of a control parameter based at least in part on the determined first value of the expression parameter, wherein the control parameter controls one or more aspects of the playback of audio samples for the virtual instrument, determining, for a second time position within the audio recording and based on the audio recording, a second value of the expression parameter, and generating a second value of the control parameter based at least in part on the determined second value of the expression parameter.

According to some embodiments, the method further comprises storing the first value of the control parameter within the digital score at one or more time positions based on the first time position, and storing the second value of the control parameter within the digital score at one or more time positions based on the second time position.

According to some embodiments, the method further comprises generating a first control message for the first time position within the audio recording, the first control message comprising an indication of the control parameter and an indication of the first value of the control parameter, and generating a second control message for the second time position within the audio recording, the second control message comprising an indication of the control parameter and an indication of the second value of the control parameter.

According to some embodiments, the method further comprises providing the first control message and the second control message to a sample library.

According to some embodiments, the method further comprises selecting, by the sample library from a plurality of audio samples, a first audio sample and a first value for at least one playback control of the first audio sample based at least in part on the first control message, and selecting, by the sample library from the plurality of audio samples, a second audio sample and a second value for the at least one playback control, different from the first value, of the second audio sample based at least in part on the second control message.

According to some embodiments, the audio recording of the live musician is obtained as a stream of audio data and the at least part of the digital score is generated in real-time based on the stream of audio data.

According to some embodiments, generating the at least part of the digital score is further based on sensor data generated by the live musician playing the instrument, the sensor data being produced concurrently with the audio recording.

The foregoing apparatus and method embodiments may be implemented with any suitable combination of aspects, features, and acts described above or in further detail below. These and other aspects, embodiments, and features of the present teachings can be more fully understood from the following description in conjunction with the accompanying drawings.

