<jupyter_start><jupyter_text>[&larr; Back to Index](index.html)  Novelty Functions To detect note onsets, we want to locate sudden changes in the audio signal that mark the beginning of transient regions. Often, an increase in the signal's amplitude envelope will denote an onset candidate. However, that is not always the case, for notes can change from one pitch to another without changing amplitude, e.g. a violin playing slurred notes.**Novelty functions** are functions which denote local changes in signal properties such as energy or spectral content. We will look at two novelty functions:1. Energy-based novelty functions (FMP, p. 306)2. Spectral-based novelty functions (FMP, p. 309)  Energy-based Novelty Functions Playing a note often coincides with a sudden increase in signal energy. To detect this sudden increase, we will compute an **energy novelty function** (FMP, p. 307):1. Compute the short-time energy in the signal.2. Compute the first-order difference in the energy.3. Half-wave rectify the first-order difference. First, load an audio file into the NumPy array `x` and sampling rate `sr`.<jupyter_code>x, sr = librosa.load('audio/simple_loop.wav')
print(x.shape, sr)<jupyter_output>(49613,) 22050<jupyter_text>Plot the signal:<jupyter_code>plt.figure(figsize=(14, 5))
librosa.display.waveplot(x, sr)<jupyter_output><empty_output><jupyter_text>Listen:<jupyter_code>ipd.Audio(x, rate=sr)<jupyter_output><empty_output><jupyter_text>RMS Energy [`librosa.feature.rmse`](http://librosa.github.io/librosa/generated/librosa.feature.rmse.htmllibrosa.feature.rmse) returns the root-mean-square (RMS) energy for each frame of audio. We will compute the RMS energy as well as its first-order difference.<jupyter_code>hop_length = 512
frame_length = 1024
rmse = librosa.feature.rmse(x, frame_length=frame_length, hop_length=hop_length).flatten()
rmse_diff = numpy.zeros_like(rmse)
rmse_diff[1:] = numpy.diff(rmse)
print(rmse.shape)
print(rmse_diff.shape)<jupyter_output>(97,)
(97,)<jupyter_text>To obtain an energy novelty function, we perform **half-wave rectification** (FMP, p. 307) on `rmse_diff`, i.e. any negative values are set to zero. Equivalently, we can apply the function $\max(0, x)$:<jupyter_code>energy_novelty = numpy.max([numpy.zeros_like(rmse_diff), rmse_diff], axis=0)<jupyter_output><empty_output><jupyter_text>Plot all three functions together:<jupyter_code>frames = numpy.arange(len(rmse))
t = librosa.frames_to_time(frames, sr=sr)
plt.figure(figsize=(15, 6))
plt.plot(t, rmse, 'b--', t, rmse_diff, 'g--^', t, energy_novelty, 'r-')
plt.xlim(0, t.max())
plt.xlabel('Time (sec)')
plt.legend(('RMSE', 'delta RMSE', 'energy novelty'))<jupyter_output><empty_output><jupyter_text>Log Energy The human perception of sound intensity is logarithmic in nature. To account for this property, we can apply a logarithm function to the energy before taking the first-order difference.Because $\log(x)$ diverges as $x$ approaches zero, a common alternative is to use $\log(1 + \lambda x)$. This function equals zero when $x$ is zero, but it behaves like $\log(\lambda x)$ when $\lambda x$ is large. This operation is sometimes called **logarithmic compression** (FMP, p. 310).<jupyter_code>log_rmse = numpy.log1p(10*rmse)
log_rmse_diff = numpy.zeros_like(log_rmse)
log_rmse_diff[1:] = numpy.diff(log_rmse)
log_energy_novelty = numpy.max([numpy.zeros_like(log_rmse_diff), log_rmse_diff], axis=0)
plt.figure(figsize=(15, 6))
plt.plot(t, log_rmse, 'b--', t, log_rmse_diff, 'g--^', t, log_energy_novelty, 'r-')
plt.xlim(0, t.max())
plt.xlabel('Time (sec)')
plt.legend(('log RMSE', 'delta log RMSE', 'log energy novelty'))<jupyter_output><empty_output><jupyter_text>Spectral-based Novelty Functions There are two problems with the energy novelty function: 1. It is sensitive to energy fluctuations belonging to the same note.2. It is not sensitive to spectral fluctuations between notes where amplitude remains the same. For example, consider the following audio signal composed of pure tones of equal magnitude:<jupyter_code>sr = 22050
def generate_tone(midi):
    T = 0.5
    t = numpy.linspace(0, T, int(T*sr), endpoint=False)
    f = librosa.midi_to_hz(midi)
    return numpy.sin(2*numpy.pi*f*t)
x = numpy.concatenate([generate_tone(midi) for midi in [48, 52, 55, 60, 64, 67, 72, 76, 79, 84]])<jupyter_output><empty_output><jupyter_text>Listen:<jupyter_code>ipd.Audio(x, rate=sr)<jupyter_output><empty_output><jupyter_text>The energy novelty function remains roughly constant:<jupyter_code>hop_length = 512
frame_length = 1024
rmse = librosa.feature.rmse(x, frame_length=frame_length, hop_length=hop_length).flatten()
rmse_diff = numpy.zeros_like(rmse)
rmse_diff[1:] = numpy.diff(rmse)
energy_novelty = numpy.max([numpy.zeros_like(rmse_diff), rmse_diff], axis=0)
frames = numpy.arange(len(rmse))
t = librosa.frames_to_time(frames, sr=sr)
plt.figure(figsize=(15, 4))
plt.plot(t, rmse, 'b--', t, rmse_diff, 'g--^', t, energy_novelty, 'r-')
plt.xlim(0, t.max())
plt.xlabel('Time (sec)')
plt.legend(('RMSE', 'delta RMSE', 'energy novelty'))<jupyter_output><empty_output><jupyter_text>Instead, we will compute a **spectral novelty function** (FMP, p. 309):1.  Compute the log-amplitude spectrogram.2.  Within each frequency bin, $k$, compute the energy novelty function as shown earlier, i.e. (a) first-order difference, and (b) half-wave rectification.3.  Sum across all frequency bins, $k$. Luckily, `librosa` has [`librosa.onset.onset_strength`](http://librosa.github.io/librosa/generated/librosa.onset.onset_strength.html) which computes a novelty function using spectral flux.<jupyter_code>spectral_novelty = librosa.onset.onset_strength(x, sr=sr)
frames = numpy.arange(len(spectral_novelty))
t = librosa.frames_to_time(frames, sr=sr)
plt.figure(figsize=(15, 4))
plt.plot(t, spectral_novelty, 'r-')
plt.xlim(0, t.max())
plt.xlabel('Time (sec)')
plt.legend(('Spectral Novelty',))<jupyter_output><empty_output><jupyter_text>Questions Novelty functions are dependent on `frame_length` and `hop_length`. Adjust these two parameters. How do they affect the novelty function? Try with other audio files. How do the novelty functions compare?<jupyter_code>ls audio<jupyter_output>125_bounce.wav                  jangle_pop.mp3
58bpm.wav                       latin_groove.mp3
README.md                       oboe_c6.wav
brahms_hungarian_dance_5.mp3    prelude_cmaj.wav
busta_rhymes_hits_for_days.mp3  simple_loop.wav
c_strum.wav                     simple_piano.wav
clarinet_c6.wav                 sir_duke_piano_fast.mp3
classic_rock_beat.wav           sir_duke_piano_slow.mp3
conga_groove.wav                sir_duke_trumpet_fast.mp3
[34mdrum_samples[m[m/                   sir_duke_trumpet_slow.mp3
funk_groove.mp3                 tone_440.wav
{'fraction_non_alphanumeric': 0.11918475852902083, 'fraction_numerical': 0.020233348102200562, 'mean_word_length': 5.425047438330171, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '76628', 'license': 'MIT', 'n_tokens_mistral': 2543, 'n_tokens_neox': 2343, 'n_words': 629, 'path': 'novelty_functions.ipynb', 'repo_name': 'mcferrenm/musicinformationretrieval.com', 'size': '6771'}