Neural Style Transfer has become a popular technique for
 generating images of distinct artistic styles using convolutional neural networks. This
 recent success in image style transfer has raised the question of
 whether similar methods can be leveraged to alter the “style” of musical
 audio. In this work, we attempt long time-scale high-quality audio transfer
 and texture synthesis in the time-domain that captures harmonic,
 rhythmic, and timbral elements related to musical style, using examples that
 may have different lengths and musical keys. We demonstrate the ability
 to use randomly initialized convolutional neural networks to transfer
 these aspects of musical style from one piece onto another using 3
 different representations of audio: the log-magnitude of the Short Time
 Fourier Transform (STFT), the Mel spectrogram, and the Constant-Q Transform
 spectrogram. We propose using these representations as a way of
 generating and modifying perceptually significant characteristics of
 musical audio content. We demonstrate each representation's
 shortcomings and advantages over others by carefully designing
 neural network structures that complement the nature of musical audio. Finally, we show that the most
 compelling “style” transfer examples make use of an ensemble of these
 representations to help capture the varying desired characteristics of
 audio signals. The problem we seek to explore in this paper is the transfer of artistic "style" from one musical audio example onto another. The definition and perception of an artistic style in visual art images (e.g., impressionist, pointilist, cubist) shown in Figure 1 is perhaps more straightforward than in the case musical audio. For images, a successful style transfer algorithm is capable of generating a novel image whose content information, or what is in the image, is matched as well as its stylistic information, or the artistic approach. In other words, it explores the question, "What would a rendering of scene A by artist B look like?" Figure 1 : Demonstration of image style transfer courtesy of BID7 .For our work, we similarly set out to develop an algorithm that explores the question, "What would it sound like if a musical piece by ensemble/artist A was performed by ensemble/artist B?" It should be noted that we do not approach the problem according to strict musicological definitions (e.g., melodic, harmonic, rhythmic, and structural elements), as one might proceed if given the musical notation of a composition. We do not presume access to the notation or any music theoretic analysis of a piece. We are instead interested in transferring the acoustic features related to harmonic, rhythmic, and timbral aspects of one musical piece onto another. Therefore , for the single instance "style" transfer algorithm we propose in this work, it is more accurate to pose the question as "What would a rendering of musical piece A (by artist A) using the harmonic and rhythmic patterns of piece B (by artist B) sound like?" In this paper, we define musical "style" transfer according to this type of audio content transformation, and will henceforth drop the use of quotation marks around "style". In texture generation, we instead ask "What would it sound like for a source musical piece to contain the same musical patterns and higher-order statistics without any of the same local, event-based information?" This can be achieved in the image or audio domain by only optimizing those terms of the loss function of a transfer algorithm associated with style, and not using any loss term associated with content.Currently, there are two types of approaches to image style transfer. The first method uses a learned generative model to manipulate the representation of the data such that it maintains its original content rendered into a new style. The second class of methods, which we investigate and apply in this paper, are concerned with synthesizing new data that matches the representations of data in a learned model in some specific way. Measuring the accuracy of such algorithms' abilities to transfer style is difficult, since most data is not able to be entirely disentangled into separate content and style components. This is especially true for musical style.There have been attempts for learning representations of musical style include the use of generative models which use a MIDI representation of audio BID14 . The advantages of using this representation are the ability to focus solely on a highly understandable representation of musical information in its harmonic and rhythmic components, but lacks the ability to capture other important sonic information like timbre.Our approach utilizes many interesting findings from recent research in image style transfer. We suggest that it is possible to use the same style transfer algorithm used for images for musical audio, but best performance requires a careful selection of how content and style is represented, given the task. FIG0 shows a spectral visualization of how a style transfer result contains both local, event based information from the content piece, while also having the characteristic nature of the style signal, as there is clearly more energy in the higher frequencies. However, it is important to note that despite this visualization in the log-magnitude STFT representation, the audio is ultimately synthesized in the time-domain. We introduce several improvements for performing musical style transfer on raw audio through the utilization of multiple audio representations. Our contributions can be summarized as follows: First, we have demonstrated that using additional representations of Mel and CQT spectrograms with accompanying neural structure improve in many cases the capture of musically meaningful style information. Secondly, we have proposed a novel, key-invariant content representation for musical audio. Finally we have shown that despite using log-magnitude spectrograms to capture the content and style information, we are still able to synthesize a target audio waveform in the time domain using the backpropogation of the STFT.While our proposed content representations work for audio in different keys, there still is no representation for tempo invariance. Other future work may include using learned generative models to perform musical style transfer and trying to perform style transfer entirely in the time-domain. This or the use of complex weights may be able to help improve representation of phase information in neural representations.
We present a long time-scale musical audio style transfer algorithm which synthesizes audio in the time-domain, but uses Time-Frequency representations of audio.
{"src": "scitldr-AIC_train", "paper_id": "BybQ7zWCb"}