<jupyter_start><jupyter_text>Copyright 2020 Google LLC.Licensed under the Apache License, Version 2.0 (the "License");<jupyter_code># Copyright 2020 Google LLC. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================<jupyter_output><empty_output><jupyter_text>Train a DDSP Autoencoder on GPUThis notebook demonstrates how to install the DDSP library and train it for synthesis based on your own data using our command-line scripts. If run inside of Colab, it will automatically use a free Google Cloud GPU.At the end, you'll have a custom-trained checkpoint that you can download to use with the [DDSP Timbre Transfer Colab](https://colab.research.google.com/github/magenta/ddsp/blob/main/ddsp/colab/demos/timbre_transfer.ipynb). **Note that we prefix bash commands with a `!` inside of Colab, but you would leave them out if running directly in a terminal.**  Install DependenciesFirst we install the required dependencies with `pip`.<jupyter_code>!pip install -qU ddsp[data_preparation]==1.6.3

# Initialize global path for using google drive. 
DRIVE_DIR = ''<jupyter_output>/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pypi.ngc.nvidia.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings<jupyter_text>Setup Google Drive (Optional, Recommeded)This notebook requires uploading audio and saving checkpoints. While you can do this with direct uploads / downloads, it is recommended to connect to your google drive account. This will enable faster file transfer, and regular saving of checkpoints so that you do not lose your work if the colab kernel restarts (common for training more than 12 hours).   Login and mount your driveThis will require an authentication code. You should then be able to see your drive in the file browser on the left panel.<jupyter_code>from google.colab import drive
drive.mount('/content/drive')<jupyter_output><empty_output><jupyter_text>Set your base directory* In drive, put all of the audio (.wav, .mp3) files with which you would like to train in a single folder. * Typically works well with 10-20 minutes of audio from a single monophonic source (also, one acoustic environment). * Use the file browser in the left panel to find a folder with your audio, right-click **"Copy Path", paste below**, and run the cell.<jupyter_code>#@markdown (ex. `/content/drive/My Drive/...`) Leave blank to skip loading from Drive.
DRIVE_DIR = '' #@param {type: "string"}

import os
# assert os.path.exists(DRIVE_DIR)
# print('Drive Folder Exists:', DRIVE_DIR)<jupyter_output><empty_output><jupyter_text>Make directories to save model and data<jupyter_code>AUDIO_DIR = "../datasets/ddsp_instruments/haegeum"
AUDIO_FILEPATTERN = AUDIO_DIR + '/*'
SAVE_DIR = "./ddsp-haegeum"
# !mkdir -p $AUDIO_DIR
# if DRIVE_DIR:
#   SAVE_DIR = os.path.join(DRIVE_DIR, 'ddsp-solo-instrument')
# else:
#   SAVE_DIR = '/content/models/ddsp-solo-instrument'
# !mkdir -p "$SAVE_DIR"<jupyter_output><empty_output><jupyter_text>Prepare Dataset  Upload training audioUpload audio files to use for training your model. Uses `DRIVE_DIR` if connected to drive, otherwise prompts local upload.<jupyter_code>import glob
import os
# from ddsp.colab import colab_utils

# if DRIVE_DIR:
#   mp3_files = glob.glob(os.path.join(DRIVE_DIR, '*.mp3'))
#   wav_files = glob.glob(os.path.join(DRIVE_DIR, '*.wav'))
#   audio_files = mp3_files + wav_files
# else:
#   audio_files, _ = colab_utils.upload()

# for fname in audio_files:
#   target_name = os.path.join(AUDIO_DIR, 
#                              os.path.basename(fname).replace(' ', '_'))
#   print('Copying {} to {}'.format(fname, target_name))
#   !cp "$fname" $target_name<jupyter_output><empty_output><jupyter_text>Preprocess raw audio into TFRecord datasetWe need to do some preprocessing on the raw audio you uploaded to get it into the correct format for training. This involves turning the full audio into short (4-second) examples, inferring the fundamental frequency (or "pitch") with [CREPE](http://github.com/marl/crepe), and computing the loudness. These features will then be stored in a sharded [TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) file for easier loading. Depending on the amount of input audio, this process usually takes a few minutes.* (Optional) Transfer dataset from drive. If you've already created a dataset, from a previous run, this cell will skip the dataset creation step and copy the dataset from `$DRIVE_DIR/data`<jupyter_code>import glob
import os

TRAIN_TFRECORD = '../datasets/ddsp_instruments/haegeum/train.tfrecord'
TRAIN_TFRECORD_FILEPATTERN = TRAIN_TFRECORD + '*'


  # Make a new dataset.
# if not glob.glob(AUDIO_FILEPATTERN):
#   raise ValueError('No audio files found. Please use the previous cell to '
#                   'upload.')

# !ddsp_prepare_tfrecord \
#   --input_audio_filepatterns=$AUDIO_FILEPATTERN \
#   --output_tfrecord_path=$TRAIN_TFRECORD \
#   --num_shards=10 \
#   --alsologtostderr<jupyter_output><empty_output><jupyter_text>Save dataset statistics for timbre transferQuantile normalization helps match loudness of timbre transfer inputs to the loudness of the dataset, so let's calculate it here and save in a pickle file.<jupyter_code># from ddsp.colab import colab_utils
import ddsp.training

data_provider = ddsp.training.data.TFRecordProvider(TRAIN_TFRECORD_FILEPATTERN)
dataset = data_provider.get_dataset(shuffle=False)
PICKLE_FILE_PATH = os.path.join(SAVE_DIR, 'dataset_statistics.pkl')

# _ = colab_utils.save_dataset_statistics(data_provider, PICKLE_FILE_PATH, batch_size=1)<jupyter_output>2021-11-15 12:04:23.052418: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 12:04:23.058676: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 12:04:23.059491: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 12:04:23.060775: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compil[...]<jupyter_text>Let's load the dataset in the `ddsp` library and have a look at one of the examples.<jupyter_code># from ddsp.colab import colab_utils
import ddsp.training
from matplotlib import pyplot as plt
import numpy as np

data_provider = ddsp.training.data.TFRecordProvider(TRAIN_TFRECORD_FILEPATTERN)
dataset = data_provider.get_dataset(shuffle=False)

try:
  ex = next(iter(dataset))
except StopIteration:
  raise ValueError(
      'TFRecord contains no examples. Please try re-running the pipeline with '
      'different audio file(s).')


f, ax = plt.subplots(3, 1, figsize=(14, 4))
x = np.linspace(0, 4.0, 1000)
ax[0].set_ylabel('loudness_db')
ax[0].plot(x, ex['loudness_db'])
ax[1].set_ylabel('F0_Hz')
ax[1].set_xlabel('seconds')
ax[1].plot(x, ex['f0_hz'])
ax[2].set_ylabel('F0_confidence')
ax[2].set_xlabel('seconds')
ax[2].plot(x, ex['f0_confidence'])<jupyter_output><empty_output><jupyter_text>Train ModelWe will now train a "solo instrument" model. This means the model is conditioned only on the fundamental frequency (f0) and loudness with no instrument ID or latent timbre feature. If you uploaded audio of multiple instruemnts, the neural network you train will attempt to model all timbres, but will likely associate certain timbres with different f0 and loudness conditions.  First, let's start up a [TensorBoard](https://www.tensorflow.org/tensorboard) to monitor our loss as training proceeds. Initially, TensorBoard will report `No dashboards are active for the current data set.`, but once training begins, the dashboards should appear.<jupyter_code>%reload_ext tensorboard
import tensorboard as tb
tb.notebook.start('--logdir "{}"'.format(SAVE_DIR))<jupyter_output><empty_output><jupyter_text>We will now begin training. Note that we specify [gin configuration](https://github.com/google/gin-config) files for the both the model architecture ([solo_instrument.gin](TODO)) and the dataset ([tfrecord.gin](TODO)), which are both predefined in the library. You could also create your own. We then override some of the spefic params for `batch_size` (which is defined in in the model gin file) and the tfrecord path (which is defined in the dataset file).  Training Notes:* Models typically perform well when the loss drops to the range of ~4.5-5.0.* Depending on the dataset this can take anywhere from 5k-30k training steps usually.* The default is set to 30k, but you can stop training at any time, and for timbre transfer, it's best to stop before the loss drops too far below ~5.0 to avoid overfitting.* On the colab GPU, this can take from around 3-20 hours. * We **highly recommend** saving checkpoints directly to your drive account as colab will restart naturally after about 12 hours and you may lose all of your checkpoints.* By default, checkpoints will be saved every 300 steps with a maximum of 10 checkpoints (at ~60MB/checkpoint this is ~600MB). Feel free to adjust these numbers depending on the frequency of saves you would like and space on your drive.* If you're restarting a session and `DRIVE_DIR` points a directory that was previously used for training, training should resume at the last checkpoint.<jupyter_code>TRAIN_TFRECORD_FILEPATTERN = "/home/teo/userdata/datasets/datasets-master/train.tfrecord*"
from ddsp.training import (data, decoders, encoders, models, preprocessing, 
                           train_util, trainers)

sample_rate = 16000
TIME_STEPS = 1000

dataset = data_provider.get_batch(batch_size=32, shuffle=False)
batch = next(iter(dataset))
audio = batch['audio']
n_samples = audio.shape[1]

strategy = train_util.get_strategy()
# Create Neural Networks.
preprocessor = preprocessing.F0LoudnessPreprocessor(time_steps=TIME_STEPS)

decoder = decoders.RnnFcDecoder(rnn_channels = 256,
                                rnn_type = 'gru',
                                ch = 256,
                                layers_per_stack = 1,
                                input_keys = ('ld_scaled', 'f0_scaled'),
                                output_splits = (('amps', 1),
                                                 ('harmonic_distribution', 45),
                                                 ('noise_magnitudes', 45)))

# Create Processors.
harmonic = ddsp.synths.Harmonic(n_samples=n_samples, 
                                sample_rate=sample_rate,
                                name='harmonic')

noise = ddsp.synths.FilteredNoise(window_size=0,
                                  initial_bias=-10.0,
                                  name='noise')
add = ddsp.processors.Add(name='add')

# Create ProcessorGroup.
dag = [(harmonic, ['amps', 'harmonic_distribution', 'f0_hz']),
       (noise, ['noise_magnitudes']),
       (add, ['noise/signal', 'harmonic/signal'])]

processor_group = ddsp.processors.ProcessorGroup(dag=dag,
                                                 name='processor_group')


# Loss_functions
spectral_loss = ddsp.losses.SpectralLoss(loss_type='L1',
                                         mag_weight=1.0,
                                         logmag_weight=1.0)

with strategy.scope():
  # Put it together in a model.
  model = models.Autoencoder(preprocessor=preprocessor,
                             encoder=None,
                             decoder=decoder,
                             processor_group=processor_group,
                             losses=[spectral_loss])
  trainer = trainers.Trainer(model, strategy, learning_rate=1e-3)
dataset = trainer.distribute_dataset(dataset)
trainer.build(next(iter(dataset)))
dataset_iter = iter(dataset)

for i in range(10000):
  losses = trainer.train_step(dataset_iter)
  res_str = 'step: {}\t'.format(i)
  for k, v in losses.items():
    res_str += '{}: {:.2f}\t'.format(k, v)
  print(res_str)
start_time = time.time()
controls =  model(next(dataset_iter))
audio_gen = model.get_audio_from_outputs(controls)
print('Prediction took %.1f seconds' % (time.time() - start_time))
print('Original Audio')
play(audio)
print('Resynthesized Audio')
play(audio_gen)
print('Filtered Noise Audio')
audio_noise = controls['noise']['signal']
play(audio_noise)

specplot(audio)
specplot(audio_gen)
specplot(audio_noise)
batch_idx = 0
get = lambda key: ddsp.core.nested_lookup(key, controls)[batch_idx]

amps = get('harmonic/controls/amplitudes')
harmonic_distribution = get('harmonic/controls/harmonic_distribution')
noise_magnitudes = get('noise/controls/magnitudes')
f0_hz = get('f0_hz')
loudness = get('loudness_db')

audio_noise = get('noise/signal')

f, ax = plt.subplots(1, 2, figsize=(14, 4))
f.suptitle('Input Features', fontsize=16)
ax[0].plot(loudness)
ax[0].set_ylabel('Loudness')
ax[1].plot(f0_hz)
ax[1].set_ylabel('F0_Hz')

f, ax = plt.subplots(1, 2, figsize=(14, 4))
f.suptitle('Synth Params', fontsize=16)
ax[0].semilogy(amps)
ax[0].set_ylabel('Amps')
ax[0].set_ylim(1e-5, 2)
# ax[0].semilogy(harmonic_distribution)
ax[1].matshow(np.rot90(np.log10(harmonic_distribution + 1e-6)),
              cmap=plt.cm.magma, 
              aspect='auto')
ax[1].set_ylabel('Harmonic Distribution')
ax[1].set_xticks([])
_ = ax[1].set_yticks([])

f, ax = plt.subplots(1, 1, figsize=(7, 4))
# f.suptitle('Filtered Noise Params', fontsize=16)
ax.matshow(np.rot90(np.log10(noise_magnitudes + 1e-6)), 
           cmap=plt.cm.magma, 
           aspect='auto')
ax.set_ylabel('Filtered Noise Magnitudes')
ax.set_xticks([])
_ = ax.set_yticks([])

!/home/teo/.local/bin/ddsp_run \
  --mode=train \
  --alsologtostderr \
  --save_dir="$SAVE_DIR" \
  --gin_file=models/solo_instrument.gin \
  --gin_file=datasets/tfrecord.gin \
  --gin_param="TFRecordProvider.file_pattern='$TRAIN_TFRECORD_FILEPATTERN'" \
  --gin_param="batch_size=32" \
  --gin_param="train_util.train.num_steps=30000" \
  --gin_param="train_util.train.steps_per_save=300" \
  --gin_param="trainers.Trainer.checkpoints_to_keep=10"<jupyter_output>2021-11-14 22:28:52.273113: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-14 22:28:52.277900: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-14 22:28:52.278326: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I1114 22:28:52.278795 140090140157760 ddsp_run.py:180] Restore Dir: ./ddsp-flute
I1114 22:28:52.278987 140090140157760 ddsp_run.py:181] Save Dir: ./ddsp-flute
I1114 22:28:52.279093 140090140157760 resource_reader.py:50] system_path_file_exists:optimization/base.gin
E1114 22:28:52.279233 140090140157760 resource_reader.py:55] Path not foun[...]<jupyter_text>ResynthesisCheck how well the model reconstructs the training data<jupyter_code>from ddsp.colab.colab_utils import play, specplot
import ddsp.training
import gin
from matplotlib import pyplot as plt
import numpy as np

data_provider = ddsp.training.data.TFRecordProvider(TRAIN_TFRECORD_FILEPATTERN)
dataset = data_provider.get_batch(batch_size=1, shuffle=False)

try:
  batch = next(iter(dataset))
except OutOfRangeError:
  raise ValueError(
      'TFRecord contains no examples. Please try re-running the pipeline with '
      'different audio file(s).')

# Parse the gin config.
gin_file = os.path.join(SAVE_DIR, 'operative_config-0.gin')
gin.parse_config_file(gin_file)

# Load model
model = ddsp.training.models.Autoencoder()
model.restore(SAVE_DIR)

# Resynthesize audio.
outputs = model(batch, training=False)
audio_gen = model.get_audio_from_outputs(outputs)
audio = batch['audio']

print('Original Audio')
specplot(audio)
play(audio)

print('Resynthesis')
specplot(audio_gen)
play(audio_gen)<jupyter_output><empty_output><jupyter_text>Download CheckpointBelow you can download the final checkpoint. You are now ready to use it in the [DDSP Timbre Tranfer Colab](https://colab.research.google.com/github/magenta/ddsp/blob/main/ddsp/colab/demos/timbre_transfer.ipynb).<jupyter_code>from ddsp.colab import colab_utils
import tensorflow as tf
import os

CHECKPOINT_ZIP = 'my_solo_instrument.zip'
latest_checkpoint_fname = os.path.basename(tf.train.latest_checkpoint(SAVE_DIR))
!cd "$SAVE_DIR" && zip $CHECKPOINT_ZIP $latest_checkpoint_fname* operative_config-0.gin dataset_statistics.pkl
!cp "$SAVE_DIR/$CHECKPOINT_ZIP" ./
colab_utils.download(CHECKPOINT_ZIP)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10543970481355174, 'fraction_numerical': 0.02862414043718902, 'mean_word_length': 4.432128757971455, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 6, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '410010', 'license': 'MIT', 'n_tokens_mistral': 6063, 'n_tokens_neox': 5596, 'n_words': 1656, 'path': 'train_autoencoder.ipynb', 'repo_name': 'jdasam/dl_for_music', 'size': '17887'}