Chord-Conditioned Melody Harmonization with Controllable Harmonicity

Melody harmonization has long been closely associated with chorales composed by Johann Sebastian Bach. Previous works rarely emphasised chorale generation conditioned on chord progressions, and there has been a lack of focus on assistive compositional tools. In this paper, we first designed a music representation that encoded chord symbols for chord conditioning, and then proposed DeepChoir, a melody harmonization system that can generate a four-part chorale for a given melody conditioned on a chord progression. With controllable harmonicity, users can control the extent of harmonicity for generated chorales. Experimental results reveal the effectiveness of the music representation and the controllability of DeepChoir.


INTRODUCTION
Chorales are vocal or instrumental music performed by a group of people, and the four-part setting for SATB (Soprano, Alto, Tenor, and Bass) is the standard for chorales. It is necessarily "polyphonic", i.e., consisting of two or more voices. It takes a lot of practice to establish a relationship between voices that is harmonically and rhythmically interdependent.
The task of chord-conditioned melody harmonization is aimed at automatically composing a four-part chorale based on a melody and a chord progression. Formally, given the soprano (melody) sequence, S 1:T = {S 1 , S 2 , ..., S T } and the chord sequence C 1:T = {C 1 , C 2 , ..., C T }, the goal is to infer sequences of the other three voices, i.e., A 1:T , T 1:T and B 1:T . All sequences (except for C 1:T ) share the same vocabulary V of size |V | ≈ 100. The phrase controllable harmonicity refers to the ability for users to encourage systems to generate more (or less) chord tones.
Studies of melody harmonization [1,2,3,4] have long focused on the Johann Sebastian Bach (JSB) Chorales dataset, and their goal is to make the music generated by systems sound as much like Bach as possible. For example, BachBotgenerated chorales [5] Fig. 1. The process of melody harmonization. Here, the alto, tenor, and bass are generated by the system. those of Bach, according to their subjective experiment results. Meanwhile, DeepBach [6], a system aimed at modelling polyphonic music, has been successfully applied to Monteverdi's five-part madrigals and Palestrina's masses, even if this system mainly focuses on JSB Chorales. As for non-neural network solutions like BacHMMachine [7], a probabilistic framework that integrates compositional principles, is capable of generating musically convincing chorales.
Although the above-mentioned works reproduce the style of Bach quite well, they lack chord conditioning (the learning of chord transitions), and most existing melody harmonization systems are uncontrollable.
In this paper, our goal is to provide an assistive compositional tool that can help users easily compose chorales. We first automatically labelled chord symbols for JSB Chorales, and encoded chords as chromagrams. We then added beat information into our music representation to make the system aware of time signatures. Furthermore, we applied gamma sampling [8], a sampling method for controlling language models, to achieve controllable harmonicity in the melody harmonization task, enabling users to steer the extent to which the system follows a given chord progression.   Fig. 2. For demonstration purposes, the time resolution of our proposed music representation is set to eighth notes.

Music Representation
As shown in Fig. 2b, our representation encodes each chorale into the following seven sequences of the same length.
• SATB Sequences: encode each voice as a piano-roll consisting of 130-D one-hot vectors (128 pitches, 1 rest, and 1 hold), with a sixteenth note time resolution.
• Fermata Sequence: consists of boolean values indicating whether a fermata symbol is present in each frame.
• Beat Sequence: based on time signatures, encodes the beat information into 4-D one-hot vectors, which correspond to non-beat, weak, medium-weight, and strong beats (ranging from 0 to 3).
• Chord Sequence: encodes chords as chromagrams (12-D multi-hot vectors), and each dimension corresponds to an activated pitch class of a chord.

DeepChoir
DeepChoir is a multi-headed autoregressive model with an encoder-decoder structure, as shown in Fig. 3. The two encoders use a stacked Bi-LSTM [9, 10] with a time-distributed dense layer, and the decoder uses a stacked dense layer.
Encoder. The structure of these two encoders is identical, but their weights are not shared. The input of the soprano encoder is the embedded soprano sequence, whereas the conditions encoder one is a concatenation of embedded beat, fermata, and chord sequences. The two encoders are formed by a stack of N = 3 identical blocks with two sub-layers. The first is a 128-hidden-size Bi-LSTM layer, and the second is a time-distributed dense layer. After the time-distributed dense layer, we use batch normalization [11] followed by dropout [12] with a rate of 0.2.
Decoder. The decoder, like the other two encoders, is composed of a stack of N = 3 identical blocks, but it is much simpler, including only a naive stacked dense layer with batch normalization and dropout. The three-head linearly transforms and softmaxes the output of the decoder, and autoregressively generates the ATB voices frame-by-frame.

Controllable Harmonicity
Based on the assumption that some attributes of the generated text are closely related to the occurrences of certain tokens, gamma sampling [8] achieves controllable text generation via scaling probabilities of attribute-related tokens during generation time: where Γ ∈[0,1] is the user-controllable parameter, p in/out is the input/output probability, A is the set of attribute-related tokens and p A is the sum of their probabilities, while p \A is the sum of the probabilities of tokens that are not in A. When Γ = 0.5, there is no change in the probability distribution, while when Γ < 0.5, the probabilities of the attributerelated tokens increase and vice versa. Since harmonicity depends on the proportion of chord tones present in the melody (the larger the proportion, the more harmonious it is), we define chord tones as the attributerelated tokens for controllable harmonicity. To make the controllable harmonicity more intuitive, we set the harmonicity parameter h = 1 − Γ.

Dataset
As we did not find any full JSB Chorales dataset with humanannotated chord symbols, and as performing harmonic analysis manually is too time-consuming, we carried out the following automated preprocessing.
• Chordifying: simplify a complex score with multiple voices into a succession of chords in one voice (see Fig.  4a) via a tool in music21 [13].
• Labelling: rank all chords based on beat strength, then only keep those with the highest scores and add them to sopranos (see Fig. 4b).
We ended up with a total of 366 chorales for training (90%) and validation (10%). Admittedly, these automatically labelled chord symbols are not authoritative from the perspective of harmonic analysis, but they are sufficient for our task.

Baselines
In addition to the proposed DeepChoir system, we conducted ablation studies to validate the importance of chord progression and beat information. For all systems below, h = 0.5.   Representations of two well-known melody harmonization systems [5,6] were chosen as baselines. It should note that their task (i.e., plain melody harmonization) is not exactly the same as DeepChoir. To the best of our knowledge, there are no other chord-conditioned melody harmonization systems available. Our purpose in choosing them as baselines is to evaluate the performance gains of introducing beat information without chords.
• BachBot [5]: only considers fermata information as the condition (no beat and chord sequences).
• DeepBach [6]: uses a subdivision list which includes indexes of sub-beats without considering time signatures (with fermata and beat-like sequences but no chord sequences).

Analyses of Music Representation
The validation set (38 chorales) reharmonized by different systems was used for this evaluation. We chose the token error rate [5] (TER, % of errors in token prediction) for quantitative evaluation. For the qualitative evaluation, we invited students from music schools to take a discrimination test, and received a total of 190 feedbacks. As shown in Fig. 5a, for TER, DeepChoir (w/o chord) has a slight improvement (about 1%) over DeepBach and Bach-Bot, which means that the encoded beat information is more effective than the subdivision list. After introducing the chord information, both DeepChoir (w/o beat) and DeepChoir (full) show a significant decrease (about 4%) in TER compared to DeepChoir (w/o chord). This demonstrates that the generation constrained by chords is more consistent with ground truth at the token level. In addition, the improvement is more noticeable on bass, as once chords are given, bass notes can be more deterministic, compared to alto and tenor.
The results of the discrimination test (see Fig. 5b) are quite clear: as music representation complexity rises, discrimination accuracy falls. When presented with a chorale generated by DeepChoir (full), about 40% of the listeners believed it was composed by Bach. Given the complexity of Bach's compositions and the fact that some of the listeners are already familiar with JSB Chorales, we consider this to be an excellent performance.

Metrics
Objective evaluations used three chord/melody harmonicity metrics proposed in [14], which have been extensively used in the melody harmonization task [15,16,17].
• Chord Tone to non-Chord Tone Ratio (CTnCTR): calculates the ratio of the number of the chord tones to the number of the non-chord tones.
• Pitch Consonance Score (PCS): calculated based on the interval between melody and chord notes.
• Melody-Chord Tonal Distance (MCTD): represents a melody note by a pitch class profile feature vector [18] and compares it against a chord symbol in the 6-D tonal space [19] to calculate the closeness between them.
For subjective evaluations, we invited students from music schools to rate all generated chorales from 1 to 5 based on their quality, and a total of 266 ratings were received. For presentation purposes, the scores were rescaled to 0.2 to 1 as a subjective metric, i.e., the Rescaled Score (RS).

Analyses of Controllable Harmonicity
We reharmonized the validation set (38 chorales) with h ranging from 0.5 (original) to 1.0 (blocking all non-chord tones). As shown in Fig. 6, h has strong positive correlations with the CTnCTR and PCS, with Pearson correlation coefficients of r = 0.9 and r = 0.88, respectively, but an obvious negative correlation with MCTD, where r = −0.86. These indicate that gamma sampling can significantly tune the harmonicity of generated chorales. However, according to RS, when h > 0.6, the listener's rating of the generated chorales decreases as h increases. We suggest that this is because setting h = 0.6 avoids the generation of some inappropriate non-chord tones, but further increasing h leads to the chorales becoming more bland and inconsistent with Bach's style. Fig. 7 provides examples generated by DeepChoir. We empirically tuned h in keeping with the original styles. Despite melodies and chords being taken from completely different styles compared to the training data, the generated chorales are still musically convincing. The expressive use of nonchord tones, especially in the bass, is quite impressive. With conditioned chords, these chorales not only match their original styles, but harmonically fit into existing accompaniments.

CONCLUSIONS
We proposed DeepChoir, a chord-conditioned melody harmonization system with controllable harmonicity. The experiment results demonstrate that DeepChoir-generated chorales are more consistent with ground truth than baselines, and tuning the controlling parameter h can significantly change their harmonicity. We expect that, with chord conditioning and controllable harmonicity, this system can be used to compose general polyphonic music that is not limited to the Bach style.