<jupyter_start><jupyter_text>This notebook contains a walk-through of the sound change simulation that was presented in Chapter 2 of my dissertation, *Functionalism, Lexical Contrast and Sound Change*.1. IntroductionFirst, we import some standard modules like Random, Numpy and Matplotlib.<jupyter_code>import random
import matplotlib.pyplot as plt
import numpy as np<jupyter_output><empty_output><jupyter_text>Next, we need to define an alphabet for our lexicon, and map our alphabet to indexes for computation purposes.I created two dictionaries that are used to encode consonant and vowel symbols.The vowel dictionary maps 15 different symbols onto different indexes.The consonant dictionary maps 26 different symbols onto different indexes.<jupyter_code>vowels = {'i': 0, 'e': 1, 'a': 2, 'o': 3, 'u': 4, 'ou':5, 'ei':6, 'ea':7, 'ee':8, 'oo':9, 'ai':10, 'oa':11,
          'oi':12, 'io':13, 'ie':14}

consonants = {'m': 0, 'p':1, 'b':2, 'f': 3, 'v': 4, 'd': 5, 't': 6, 'l': 7, 'n': 8, 'r': 9, 's': 10, 'k': 11,
              'y': 12, 'g': 13, 'j':14, 'h': 15, 'c':16, ' ':17, 'th':18, 'sh':19, 'wh':20, 'ch':21, 'tw':22,
              'x':23, 'w':24, 'z':25}<jupyter_output><empty_output><jupyter_text>With this dictionary, we can represent each word as an integer tuple. Words are initially read from a text filein the form of 3-dimensional tuples of the type (Onset, Nucleus, Coda). For instance the word "dog" will be processed in the form of a 3-dimensional tuple ```('d', 'o', 'g')```. Then, we can use the two dictionaries to transform each word into a integer vector, through a helper function.<jupyter_code>dog = ('d', 'o', 'g')

def vectorize(word):
    onset, nucleus, coda = word
    return consonants[onset], vowels[nucleus], consonants[coda]<jupyter_output><empty_output><jupyter_text>For instance, the word "dog" will be transformed in the tuple ```(5, 3, 13)```.<jupyter_code>print(vectorize(dog))<jupyter_output>(5, 3, 13)<jupyter_text>The lexicon is a list of words (=tuples), and is stored as a global variable, since we need to modify it as sound changeoccurs. A wordlist containing the words "dog", "cat", and "pig" is represented by the following variable.<jupyter_code>wordlist = [('d', 'o', 'g'), ('c', 'a', 't'), ('p', 'i', 'g')]<jupyter_output><empty_output><jupyter_text>Since we want to keep track of the number of symbols and the possible phonological environments, we also store three sets thatcontain the onsets, nuclei and codas available in the lexicon in its current state. These three sets are all global variables.<jupyter_code>def get_onset(wordlist):
    return {word[0] for word in wordlist}

def get_nucleus(wordlist):
    return {word[1] for word in wordlist}

def get_coda(wordlist):
    return {word[2] for word in wordlist}<jupyter_output><empty_output><jupyter_text>Let's extract the onsets, nuclei and codas of the current wordlist.<jupyter_code>onset, nucleus, coda = get_onset(wordlist), get_nucleus(wordlist), get_coda(wordlist)

print(onset)
print(nucleus)
print(coda)<jupyter_output>{'d', 'p', 'c'}
{'i', 'a', 'o'}
{'g', 't'}<jupyter_text>For illustratory purposes, we need a reverse-dictionary, which can be used to retrieve the symbols given their index,and a helper function to retrieve the word (in string format) given its integer vector representation:<jupyter_code>rev_vowels = {code: letter for letter, code in vowels.items()}
rev_consonants = {code: letter for letter, code in consonants.items()}


def vectorize_inverse(word):
    onset, nucleus, coda = word
    return rev_consonants[onset] + rev_vowels[nucleus] + rev_consonants[coda]<jupyter_output><empty_output><jupyter_text>By calling the function on the tuple ```(5,3,13)```, we will get the word "dog" back.<jupyter_code>dog = (5, 3, 13)
print(vectorize_inverse(dog))<jupyter_output>dog<jupyter_text>Another helper function that we need is a function that returns the average Levenshtein distance within a wordlist:<jupyter_code>def average(wordlist):
    av_length = []
    for index, word in enumerate(wordlist):
        for word2 in wordlist[index+1:]:
            lev = 0
            for i, letter in enumerate(word):
                if word2[i] != letter:
                    lev += 1
            av_length.append(lev)
    return round(sum(av_length)/len(av_length),3)<jupyter_output><empty_output><jupyter_text>The average distance of our current lexicon should be:[ d(dog,cat) + d(dog,pig) + d(cat,pig) ] /3:(3 + 2 + 3) / 3 : 8 / 3 = 2.667<jupyter_code>print(average(wordlist))<jupyter_output>2.667<jupyter_text>Now that the lexicon, the alphabet and the helper functions are defined, we can move to sound change! 2. Sound change functionsThis part defines the sound change functions. These functions modify the lexicon by simulating sound changes.The first function represents a sound change that targets the onset of the word. The comments on the function explain how we implement a sound change that targets the onset position of the artificial language (as described in 2.2.2). Depending on the outcome, the change might result in an unconditioned merger, a conditioned merger, a split, or a simple phonetic change. For more information, cf. 2.2.2 in the dissertation.<jupyter_code>def change_onset():
    #call the lexicon list and the onset set
    global lexicon, onset
    #prepare a new empty list, that will be filled with the form of the words after the sound change applies
    new_lexicon = []
    #pick an onset at random and name it target. This is the target of the sound change
    target = random.choice(list(onset))
    #pick an onset at random and name it outcome. This is the outcome of the sound change
    outcome = random.choice(list(rev_consonants))
    #select a random subset of nuclei as the conditioning environment
    environment = random.sample(nucleus, random.randint(0, len(nucleus) - 1))
    #apply the change to the lexicon
    for word in lexicon:
        #check words where target is the onset
        if word[0] == target:
            #determine whether the nucleus is in the conditioning environment
            if word[1] in environment:
                #if the nucleus is in the conditioning environment, then change target into outcome
                new_lexicon.append((outcome, word[1], word[2]))
            else:
                #if not, the change does not apply
                new_lexicon.append(word)
        else:
            #if the word does not start with target, the change does not apply
            new_lexicon.append(word)
    #this prints a line describing the change that happened
    print('/' + rev_consonants[target] + '/ becomes /' + rev_consonants[outcome] + '/ in onset before ['
          + ' '.join([rev_vowels[index] for index in environment]) + ']')
    #Update lexicon and onsets
    lexicon = new_lexicon
    onset = get_onset(lexicon)<jupyter_output><empty_output><jupyter_text>Let's try the function. All we need is to transform the words in the wordlist in their vectorizes representation, and store the type of onsets, nuclei and codas. Then we can apply some sound changes.<jupyter_code>wordlist = [('d', 'o', 'g'), ('c', 'a', 't'), ('p', 'i', 'g')]
lexicon = [vectorize(word) for word in wordlist]
onset, nucleus, coda = get_onset(lexicon), get_nucleus(lexicon), get_coda(lexicon)
for iteration in range(5):
  change_onset()<jupyter_output>/c/ becomes /tw/ in onset before [a i]
/d/ becomes /y/ in onset before [a]
/tw/ becomes /v/ in onset before [i a]
/v/ becomes /k/ in onset before []
/p/ becomes /f/ in onset before [o]<jupyter_text>Now let's see how the lexicon looks like after the change. Can you guess the form of the new lexicon?<jupyter_code>print([vectorize_inverse(word) for word in lexicon])<jupyter_output>['dog', 'vat', 'pig']<jupyter_text>With a lexicon of only three words, and changes that only apply to the first symbol of the words, the shape of the lexicon does not change radically.Now, it's time to add other sound change functions. The following two functions will apply a change to the nucleus of the word. The only difference between the two is whetherthe conditioning environment is the onset or the coda.<jupyter_code>def change_nucleus():
    #call the lexicon list and the nucleus set
    global lexicon, nucleus
    #prepare a new empty list, that will be filled with the form of the words after the sound change apply
    new_lexicon = []
    #pick a nucleus at random and name it target. This is the target of the sound change
    target = random.choice(list(nucleus))
    #pick a nucleus at random and name it outcome. This is the outcome of the sound change
    outcome = random.choice(list(rev_vowels))
    #select a random subset of onsets as the conditioning environment
    environment = random.sample(onset, random.randint(0, len(onset) - 1))
    #apply the change to the lexicon
    for word in lexicon:
        #check words where target is the nucleus
        if word[1] == target:
            #determine whether the onset is in the conditioning environment
            if word[0] in environment:
                #if the onset is in the conditioning environment, then change target into outcome
                new_lexicon.append((word[0], outcome, word[2]))
            else:
                #if not, the change does not apply
                new_lexicon.append(word)
        else:
            #if the word does not have target, the change does not apply
            new_lexicon.append(word)
    #this prints a line describing the change that happened
    print('/' + rev_vowels[target] + '/ becomes /' + rev_vowels[outcome] + '/ after ['
          + ' '.join([rev_consonants[index] for index in environment]) + ']')
    #Update lexicon and nuclei
    lexicon = new_lexicon
    nucleus = get_nucleus(lexicon)

def change_nucleus2():
    #call the lexicon list and the nucleus set
    global lexicon, nucleus
    #prepare a new empty list, that will be filled with the form of the words after the sound change apply
    new_lexicon = []
    #pick a nucleus at random and name it target. This is the target of the sound change
    target = random.choice(list(nucleus))
    #pick a nucleus at random and name it outcome. This is the outcome of the sound change
    outcome = random.choice(list(rev_vowels))
    #select a random subset of codas as the conditioning environment
    environment = random.sample(coda, random.randint(0, len(coda) - 1))
    #apply the change to the lexicon
    for word in lexicon:
        #check words where target is the nucleus
        if word[1] == target:
            #determine whether the coda is in the conditioning environment
            if word[2] in environment:
                #if the coda is in the conditioning environment, then change target into outcome
                new_lexicon.append((word[0], outcome, word[2]))
            else:
                #if not, the change does not apply
                new_lexicon.append(word)
        else:
            #if the word does not have target, the change does not apply
            new_lexicon.append(word)
    print('/' + rev_vowels[target] + '/ becomes /' + rev_vowels[outcome] + '/ before ['
          + ' '.join([rev_consonants[index] for index in environment]) + ']')
    #Update lexicon and nuclei
    lexicon = new_lexicon
    nucleus = get_nucleus(lexicon)<jupyter_output><empty_output><jupyter_text>The last function targets the coda of the word instead.<jupyter_code>def change_coda():
    #call the lexicon list and the coda set
    global lexicon, coda
    #prepare a new empty list, that will be filled with the form of the words after the sound change apply
    new_lexicon = []
    #pick a coda at random and name it target. This is the target of the sound change
    target = random.choice(list(coda))
    #pick a coda at random and name it outcome. This is the outcome of the sound change
    outcome = random.choice(list(rev_consonants))
    #select a random subset of nuclei as the conditioning environment
    environment = random.sample(nucleus, random.randint(0, len(nucleus) - 1))
    #apply the change to the lexicon
    for word in lexicon:
        #check words where target is the coda
        if word[2] == target:
            #determine whether the nucleus is in the conditioning environment
            if word[1] in environment:
                #if the nucleus is in the conditioning environment, then change target into outcome
                new_lexicon.append((word[0], word[1], outcome))
            else:
                #if not, the change does not apply
                new_lexicon.append(word)
        else:
            #if the word does not end with target, the change does not apply
            new_lexicon.append(word)
    print('/' + rev_consonants[target] + '/ becomes /' + rev_consonants[outcome] + '/ in coda after ['
          + ' '.join([rev_vowels[index] for index in environment]) + ']')
    #Update lexicon and onsets
    lexicon = new_lexicon
    coda = get_coda(lexicon)<jupyter_output><empty_output><jupyter_text>Now, let's repeat the previous experiments, with a larger vocabulary and a larger inventory of sound changes.<jupyter_code>wordlist = [('d', 'o', 'g'), ('c', 'a', 't'), ('p', 'i', 'g'), ('c','o','w'), ('r','a','t'), ('b', 'ee', ' '), ('f','i','sh')]

lexicon = [vectorize(word) for word in wordlist]
onset, nucleus, coda = get_onset(lexicon), get_nucleus(lexicon), get_coda(lexicon)
for iteration in range(5):
  change_onset()
  change_nucleus()
  change_nucleus2()
  change_coda()<jupyter_output>/f/ becomes / / in onset before [ee]
/o/ becomes /oo/ after [f b]
/ee/ becomes /i/ before [w sh]
/w/ becomes /m/ in coda after []
/f/ becomes /sh/ in onset before [o i a]
/a/ becomes /oo/ after [r]
/ee/ becomes /oa/ before [  t]
/sh/ becomes /wh/ in coda after []
/p/ becomes /m/ in onset before [o a oo i]
/a/ becomes /a/ after [c r b d]
/a/ becomes /o/ before [  w]
/ / becomes /g/ in coda after []
/sh/ becomes /k/ in onset before [oo]
/oo/ becomes /i/ after [sh b]
/i/ becomes /u/ before [t  ]
/ / becomes /m/ in coda after [oo o i a]
/m/ becomes /ch/ in onset before []
/i/ becomes /ai/ after []
/oa/ becomes /ei/ before [sh]
/ / becomes /r/ in coda after [i o oa oo]<jupyter_text>Now we see that more changes occurr, and as a consequence the lexicon also changes drastically.<jupyter_code>print([vectorize_inverse(word) for word in lexicon])<jupyter_output>['dog', 'cat', 'mig', 'cow', 'root', 'boar', 'shish']<jupyter_text>We can keep track of the average distance of the vocabulary by means of our ```average()``` function.<jupyter_code>print('The average distance at the beginning is %s' % average(wordlist))
print('The average distance at the end is %s' % average(lexicon))<jupyter_output>The average distance at the beginning is 2.714
The average distance at the end is 2.762<jupyter_text>3. SimulationFinally, we can run several sound change simulations, and keep track of the average distance and the number of phonemes in the lexicon during the simulations, by plotting them on a graph.In addition to the wordlist, we should just specify the number of iterations and the number sound changes to apply.<jupyter_code>wordlist = [('d', 'o', 'g'), ('c', 'a', 't'), ('p', 'i', 'g'), ('c','o','w'), ('r','a','t'), ('b', 'ee', ' '), ('f','i','sh')]
iterations = 5
n_changes = 10000

for iter in range(iterations):
  print("Iteration number %s started" % (iter+1))
  global onset, nucleus, coda, lexicon
  #this line loads the lexicon in the format described above: a list of integer tuples
  lexicon = [(consonants[word[0]], vowels[word[1]], consonants[word[2]]) for word in wordlist]
  #this line gets the onset, nucleus, and coda sets
  onset, nucleus, coda = get_onset(lexicon), get_nucleus(lexicon), get_coda(lexicon)
  #this line will be used to define the sound change functions used in the simulation and their weight
  #with this setting, each function is equally weighted
  functions = [change_onset, change_nucleus, change_nucleus2, change_coda]
  #we initialize lists that will keep track of the number of the iteration, the number of the phonemes,
  #and the average distance
  x_axis = [0]
  phonemes = [len(onset.union(coda)) + len(nucleus)]
  av_length = [average(lexicon)]
  for n in range(int(n_changes)):
    #this line selects a sound change function at random and applies it
    random.choice(functions)()
    #This line is needed to make the plot lighter. For the toy example in Figure 2.2, the sampling ratio has been reduced to '1'
    if n % 1000 == 0:
      #we update the lists that keep track of the number of the iteration, the number of the phonemes, and the
      #average distance
      x_axis.append(n+1)
      phonemes.append(len(onset.union(coda)) + len(nucleus))
      av_length.append(average(lexicon))
      #this loop prints the shape of the lexicon at the beginning of the simulation and after the sound changes applied
  for index, word in enumerate(lexicon):
    print(''.join(wordlist[index]) + '->' + ''.join(rev_consonants[word[0]] + rev_vowels[word[1]] + rev_consonants[word[2]]))
  print("Iteration number %s finished" % (iter+1))
  #After the simulation has ended, we plot the change in the number of phonemes and in the average distance
  #during the simulation
  #plot phoneme size
  plt.subplot(1, 2, 1)
  plt.plot(x_axis, phonemes)
  #plt.xticks(np.arange(1, 4, step=1)) #This is for the toy example in Figure 2.2
  #plt.yticks(np.arange(36, 39, step=1)) #This is for the toy example in Figure 2.2
  plt.title('Number of Phonemes')
  plt.xlabel('Iterations')
  plt.ylabel('Counts')
  #plot av_length
  plt.subplot(1, 2, 2)
  plt.plot(x_axis, av_length)
  #plt.xticks(np.arange(1, 4, step=1))  #This is for the toy example in Figure 2.2
  plt.title('Average Levenshtein Distance')
  plt.xlabel('Iterations')
  plt.ylabel('Counts')<jupyter_output>[1;30;43mStreaming output truncated to the last 5000 lines.[0m
/ie/ becomes /ou/ before []
/n/ becomes /y/ in coda after [o oo]
/r/ becomes /p/ in onset before [oo]
/y/ becomes /b/ in coda after [oo ie]
/oo/ becomes /ou/ before [b]
/l/ becomes /t/ in onset before [o]
/o/ becomes /ai/ after [l]
/x/ becomes /k/ in coda after [ie]
/ou/ becomes /ee/ after [r]
/ou/ becomes /u/ before []
/y/ becomes /wh/ in coda after [ie o]
/r/ becomes /ch/ in onset before []
/k/ becomes /h/ in coda after []
/r/ becomes /ch/ in onset before []
/ou/ becomes /oi/ before []
/ie/ becomes /oa/ after [l]
/oa/ becomes /ai/ before [k]
/k/ becomes /b/ in coda after [ai ou]
/o/ becomes /oi/ before []
/ou/ becomes /oo/ after [r]
/b/ becomes /j/ in coda after []
/r/ becomes /g/ in onset before []
/ai/ becomes /o/ before [b]
/wh/ becomes /ch/ in coda after [o]
/ch/ becomes /th/ in coda after []
/ou/ becomes /u/ before []
/r/ becomes /x/ in onset before []
/r/ becomes /w/ in onset before [ou]
/l/ becomes /ch/ in onset [...]
{'fraction_non_alphanumeric': 0.11093982564320647, 'fraction_numerical': 0.011269402509036784, 'mean_word_length': 3.6020058708414875, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 72, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '168902', 'license': 'MIT', 'n_tokens_mistral': 6182, 'n_tokens_neox': 5782, 'n_words': 2342, 'path': 'Chapter2/sound_change.ipynb', 'repo_name': 'AndreaCeolin/Functionalism_Contrast_Change', 'size': '18812'}