<jupyter_start><jupyter_text>User's Guide, Chapter 30: Examples 3Since the last set of examples in :ref:`Chapter 20 `, we've learned about sorting streams, graphs and plots, roman numerals, environment, post-tonal tools, iterators, grace notes, and spanners. Now let's put as many of these together as we can to analyze music computationally.The corpus of `music21` includes the over 1050 songs, reels, jigs, from _Ryan's Mammoth Collection of Fiddle Tunes,_ thanks to [<NAME>](http://trillian.mit.edu/~jc/homepage.html) and others. The collection was originally published in 1883 by <NAME> after <NAME>'s work. Let's use these, which are called "ryansMammoth" in the corpus, for some examples.(Note that some of the tunes in this corpus are based on uncredited tunes either by African-American composers or adapted from such tunes ["Minstrel Songs"], and for this reason were attacked in *Dwight's Journal* as too "trashy" (1855, p. 118) for cosmopolitan New England tastes.  The work of finding the 19th-century Black compositional geniuses behind the original tunes continues, and I hope we can supplement the adpated melodies in the corpus with originals in the near future.  In the meantime, I hope that the computational studies of these works can stick a finger in the face of <NAME>, if his myopic vision of American creativity could ever see it.)First we'll load `music21` and search the corpus for these tunes:<jupyter_code>from music21 import *
ryans = corpus.search('ryansMammoth')
ryans<jupyter_output><empty_output><jupyter_text>Let's look at one of these that I know is called 'Highland Regiment':<jupyter_code>highland = ryans.search('Highland Regiment')
highland
highlandParsed = highland[0].parse()
highlandParsed.measures(0, 8).show()<jupyter_output><empty_output><jupyter_text>Let's take our knowledge of spanners and see if the last note of a slur is generally higher or lower than the first note.  We'll use a `RecursiveIterator` with a `ClassFilter` of 'Slur' to find them all:<jupyter_code>highlandIterator = highlandParsed.recurse()
highlandSlurs = highlandIterator.getElementsByClass('Slur')

higher = 0
lower = 0
same = 0 # could happen for slurs longer than 2 notes

for sl in highlandSlurs:
    firstNote = sl.getFirst()
    lastNote = sl.getLast()
    psDiff = lastNote.pitch.ps - firstNote.pitch.ps
    if psDiff > 0:
        higher += 1
    elif psDiff < 0:
        lower += 1
    else:
        same += 1
(higher, lower, same)<jupyter_output><empty_output><jupyter_text>Hmmm... it looks like of the 49 slurs in this piece, more of them end lower than higher.  Let's do this on a sample of the first 20 pieces in the collection.  Let's aggregate our slur counting function a little bit, and make it safe in case a slur begins on a Chord, by taking the average pitch value of all the notes in the slur, introduce a Counter object from Python's `collections` module, and go to it:<jupyter_code>from statistics import mean
from collections import Counter

totalCounter = Counter()

def countOneSlur(sl, totalCounter):
    firstNote = sl.getFirst()
    lastNote = sl.getLast()
    if not hasattr(firstNote, 'pitches'):
        return
    if not hasattr(lastNote, 'pitches'):
        return

    firstNotePs = mean(p.ps for p in firstNote.pitches)
    lastNotePs = mean(p.ps for p in lastNote.pitches)
    psDiff = lastNotePs - firstNotePs
    if psDiff > 0:
        totalCounter['higher'] += 1
    elif psDiff < 0:
        totalCounter['lower'] += 1
    else:
        totalCounter['same'] += 1<jupyter_output><empty_output><jupyter_text>Now let's make a function that takes in an object from `corpus.search` and parses it and runs each slur through `countOneSlur`.<jupyter_code>def runOneScore(scCorpusSearchObject, totalCounter):
    scParsed = scCorpusSearchObject.parse()
    for sl in scParsed.recurse().getElementsByClass('Slur'):
        countOneSlur(sl, totalCounter)<jupyter_output><empty_output><jupyter_text>Always important to test to make sure we haven't broken anything.  This should give the same answer as before:<jupyter_code>runOneScore(highland[0], totalCounter)
totalCounter<jupyter_output><empty_output><jupyter_text>It works as before, though there's no "same" entry in `totalCounter` since we did not encounter such a case.  Let's reset our counter and run the first 20 pieces through the process:<jupyter_code>totalCounter = Counter()

stop = 20
for piece in ryans:
    runOneScore(piece, totalCounter)
    stop = stop - 1
    if stop == 0:
        break

totalCounter<jupyter_output><empty_output><jupyter_text>Hmmm... this still shows a few more "lower" cases than "higher" but not very much of a difference.  In fact, the entire difference of 10 can be attributed to the first test example, since "Highland Regiment" is in this test set.  Maybe the notion that slurs more often end on lower notes than higher notes can be rejected, but I think we'd like to run the whole data set first.  Simply remove the "stop" and "break" variables from above and the results will come back in a minute or two.  I ran it and got these results:    Counter({'higher': 3321, 'lower': 3637, 'same': 425})    Again, it's about 10% more times that a slur ends below the first note than above -- that's the same ratio as we saw for the first ten pieces, but it might be quite statistically significant given the much larger dataset.  After all, if you flip a coin three times, it's not that unusual to get all heads or all tails, but if you flip a coin one thousand times, even 550 heads and 450 tails is quite significant as an indicator of bias in the coin.  So let's write a little test program that treats `'higher'` and `'lower'` as if they are coin flips and see how often we get results outside the range (in either direction) as our counter results.  `numBiased` takes in a `Counter` object, and a number of trials to run, flips a coin as many times as the higher and lower values and prints out the fraction of the flip trials that lie outside the range.  Optionally, it can print out the exceptional results along the way:<jupyter_code>import random

def numBiased(inCounter, trials=100, printData=False):
    chanceTrials = 0
    totalSimulations = inCounter['higher'] + inCounter['lower']
    maxValue = max([inCounter['higher'], inCounter['lower']])
    minValue = min([inCounter['higher'], inCounter['lower']])
    for trialNum in range(trials):
        randCounter = Counter()
        for flipNum in range(totalSimulations):
            if random.randint(0, 1) == 1:
                outcome = 'higher'
            else:
                outcome = 'lower'
            randCounter[outcome] += 1
        if (randCounter['higher'] < maxValue
                and randCounter['higher'] > minValue):
            chanceTrials += 1
        elif printData:
            print(randCounter)
    return chanceTrials / trials<jupyter_output><empty_output><jupyter_text>We'll run it first on the data from "Highland Regiment", printing out the intermediate values along the way:<jupyter_code>numBiased(Counter({'higher': 19, 'lower': 30}), printData=True)<jupyter_output>Counter({'higher': 35, 'lower': 14})
Counter({'higher': 33, 'lower': 16})
Counter({'lower': 33, 'higher': 16})
Counter({'lower': 30, 'higher': 19})
Counter({'higher': 30, 'lower': 19})
Counter({'higher': 34, 'lower': 15})
Counter({'lower': 32, 'higher': 17})
Counter({'lower': 30, 'higher': 19})<jupyter_text>Most of the time, the results are within the range of Highland Regiment, but this is below the 0.95 threshold that would allow us to say that there's something happening here.  The results for the first ten pieces are even closer to a null hypothesis:<jupyter_code>numBiased(Counter({'higher': 80, 'lower': 90, 'same': 11}))<jupyter_output><empty_output><jupyter_text>But, as I noted above, much larger trials may be much more significant.  We will run it on the `Counter` object for the whole piece and see what the results are:<jupyter_code>totalCounter = Counter({'higher': 3321, 'lower': 3637, 'same': 425})
numBiased(totalCounter)<jupyter_output><empty_output><jupyter_text>Those are very significant results! There wasn't a single case where coins were flipped about 7000 times and we got fewer than 3321 or more than 3637 heads or tails! Thus we can definitely say that in this collection of fiddle tunes, you're more likely to slur down than up -- the size of the results is small, but the significance is high.  Roman Numeral Analysis on a MelodyThe majority of the pieces in _Ryan's Mammoth Collection_ are single-voice pieces, which might make Roman Numeral analysis difficult, but actually it can be even more interesting this way.  Let's take a tune from close to MIT's home, a reel called "The Boston":<jupyter_code>bostonMD = ryans.search('The Boston -- Reel')
bostonMD
boston = bostonMD[0].parse()
boston.measures(0, 9).show()<jupyter_output><empty_output><jupyter_text>Now let's create a chord from each beat in the piece (skipping the pickup notes), aggregating all the pitches from within that beat.  First, we'll create a new `Part` object that has the same framework of `Measures` as the original, along with time signatures, etc., but no notes.  We'll use the `.template()` method but tell it not to fill it with rests.<jupyter_code>bostonPart = boston.parts[0]
outPart = bostonPart.template(fillWithRests=False)<jupyter_output><empty_output><jupyter_text>Like we've done before, we'll start from the smallest part and work outwards.  Since we will be working with RomanNumerals, we'll have to know what key we are in; so we can analyze the piece; hopefully this will be F major.<jupyter_code>pieceKey = boston.analyze('key')
pieceKey<jupyter_output><empty_output><jupyter_text>Now let's make a routine that turns a list of Notes into a RomanNumeral object:<jupyter_code>def notesToRoman(notes):
    uniquePitches = set(n.pitch for n in notes)
    ch = chord.Chord(list(uniquePitches))
    return roman.romanNumeralFromChord(ch, pieceKey)<jupyter_output><empty_output><jupyter_text>Let's test it with the notes of measure 1:<jupyter_code>noteIterator = bostonPart.measure(1).notes
rn1 = notesToRoman(noteIterator)
rn1<jupyter_output><empty_output><jupyter_text>Great, this is exactly what we're looking for.  Now let's go into each measure that is full and analyze it separately. Now we'll go into each measure that is full, and analyze it separately.  We'll get everything from offset 0 to 1, then 1 to 2:<jupyter_code>inMeasures = list(bostonPart.getElementsByClass('Measure'))
outMeasures = list(outPart.getElementsByClass('Measure'))

for i in range(14):
    inMeasure = inMeasures[i]
    if inMeasure.duration.quarterLength != 2.0:
        continue
    outMeasure = outMeasures[i]
    
    for beatStart in (0, 1):
        beatNotes = inMeasure.getElementsByOffset(beatStart, 
                                                  beatStart + 1, 
                                                  includeEndBoundary=False)
        beatRN = notesToRoman(beatNotes)
        beatRN.lyric = beatRN.figure
        outMeasure.insert(beatStart, beatRN)
outPart.show()<jupyter_output><empty_output><jupyter_text>Well, that's a great way to earn a C-minus in any music analysis class!  Yes, all of these chords are correct, but only a few are useful for analysis.  So let's do two things: (1) filter out all notes that are on a very weak position (the second or fourth sixteenth note) and are a passing or neighbor tone or something like that, and (2) not show the figure for anything with the functionality of a "iii" chord or less. First let's redefine `notesToRoman` to do just that.<jupyter_code>def notesToRoman(notes):
    goodPitches = set()
    for n in notes:
        if (n.offset * 2) == int(n.offset * 2):
            goodPitches.add(n.pitch)
        else:
            nPrev = n.previous('Note')
            nNext = n.next('Note')
            if nPrev is None or nNext is None:
                continue
            prevInterval = interval.Interval(n, nPrev)
            nextInterval = interval.Interval(n, nNext)
            
            if (prevInterval.generic.undirected in (1, 2)
                and nextInterval.generic.undirected in (1, 2)):
                pass
            else:
                goodPitches.add(n.pitch)
    ch = chord.Chord(list(goodPitches))
    return roman.romanNumeralFromChord(ch, pieceKey)<jupyter_output><empty_output><jupyter_text>Now we can figure out that `functionalityScore` minimum:<jupyter_code>iii = roman.RomanNumeral('iii')
iii.functionalityScore<jupyter_output><empty_output><jupyter_text>And let's re-run it:<jupyter_code>outPart = bostonPart.template(fillWithRests=False)

inMeasures = list(bostonPart.getElementsByClass('Measure'))
outMeasures = list(outPart.getElementsByClass('Measure'))

for i in range(14):
    inMeasure = inMeasures[i]
    if inMeasure.duration.quarterLength != 2.0:
        continue
    outMeasure = outMeasures[i]
    
    for beatStart in (0, 1):
        beatNotes = inMeasure.getElementsByOffset(beatStart, 
                                                  beatStart + 1, 
                                                  includeEndBoundary=False)
        beatRN = notesToRoman(beatNotes)
        if beatRN.functionalityScore > 15:
            beatRN.lyric = beatRN.figure
        outMeasure.insert(beatStart, beatRN)
outPart.show()<jupyter_output><empty_output><jupyter_text>It's still not great, but we get a much better sense from this that the diminished chord on vii in root position is often being used in place of V as a dominant in this piece.  Let's run the routine one more time and this time put the lyrics directly back in the score:<jupyter_code>inMeasures = list(bostonPart.getElementsByClass('Measure'))

for i in range(14):
    inMeasure = inMeasures[i]
    if inMeasure.duration.quarterLength != 2.0:
        continue
    
    for beatStart in (0, 1):
        beatNotes = inMeasure.getElementsByOffset(beatStart, 
                                                  beatStart + 1, 
                                                  includeEndBoundary=False)
        beatRN = notesToRoman(beatNotes)
        if beatRN.functionalityScore > 15:
            beatNotes[0].lyric = beatRN.figure

bostonPart.measures(0, 13).show()<jupyter_output><empty_output><jupyter_text>This is looking pretty decent, but have no fears, we'll find more sophisticated score reduction tools in `music21` in the chapters to come. Let's visualize this whole piece as a piano roll:<jupyter_code>bostonPart.plot('pianoroll')<jupyter_output><empty_output><jupyter_text>Here the centrality of the low F and the C5 stand out, along with an avoidance of the low tonic during the middle section, from measures 10-15, which I presume emphasizes the dominant harmony, but due to the presence of B-flats, is not _in_ the dominant.<jupyter_code>bostonPart.measures(10, 15).analyze('key')<jupyter_output><empty_output><jupyter_text>It's clear that this piece is not chromatic at all, as a histogram of the pitch classes will show well:<jupyter_code>bostonPart.plot('histogram', 'pitchClass')<jupyter_output><empty_output><jupyter_text>Are there particular pitches -- perhaps exploiting the open strings -- that appear more often in this repertory?  We can stitch a bunch of pieces together and see.  We'll use an `Opus` Stream, which is a Stream that can hold other scores:<jupyter_code>manyScores = stream.Opus()

for i in range(50):
    sc = ryans[i].parse()
    manyScores.insert(0, sc)

manyScores.plot('histogram', 'pitchClass')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08443186933316185, 'fraction_numerical': 0.013954086553919363, 'mean_word_length': 3.9418493803622496, 'pattern_counts': {'":': 1, '<': 107, '<?xml version=': 0, '>': 109, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '299934', 'license': 'MIT', 'n_tokens_mistral': 4822, 'n_tokens_neox': 4424, 'n_words': 1762, 'path': 'documentation/source/usersGuide/usersGuide_30_examples3.ipynb', 'repo_name': 'cuthbertLab/music21', 'size': '15591'}