natural language toolkit tokenizers c 20012023 nltk project christopher hench chris l henchgmail com alex estes url https www nltk org for license information see license txt the sonority sequencing principle ssp is a language agnostic algorithm proposed by otto jesperson in 1904 the sonorous quality of a phoneme is judged by the openness of the lips syllable breaks occur before troughs in sonority for more on the ssp see selkirk 1984 the default implementation uses the english alphabet but the sonorityhiearchy can be modified to ipa or any other alphabet for the usecase the ssp is a universal syllabification algorithm but that does not mean it performs equally across languages bartlett et al 2009 is a good benchmark for english accuracy if utilizing ipa pg 311 importantly if a custom hierarchy is supplied and vowels span across more than one level they should be given separately to the vowels class attribute references otto jespersen 1904 lehrbuch der phonetik leipzig teubner chapter 13 silbe pp 185203 elisabeth selkirk 1984 on the major class features and syllable theory in aronoff oehrle eds language sound structure studies in phonology cambridge mit press pp 107136 susan bartlett et al 2009 on the syllabification of phonemes in hltnaacl pp 308316 syllabifies words based on the sonority sequencing principle ssp from nltk tokenize import syllabletokenizer from nltk import wordtokenize ssp syllabletokenizer ssp tokenize justification jus ti fi ca tion text this is a foobarlike sentence ssp tokenizetoken for token in wordtokenizetext this is a foo bar li ke sen ten ce param lang language parameter default is english en type lang str param sonorityhierarchy sonority hierarchy according to the sonority sequencing principle type sonorityhierarchy liststr sonority hierarchy should be provided in descending order if vowels are spread across multiple levels they should be passed assigned self vowels var together otherwise should be placed in first index of hierarchy assigns each phoneme its value from the sonority hierarchy note sentencetext has to be tokenized first param token single word or token type token str return list of tuples first element is characterphoneme and second is the soronity value rtype listtuplestr int ensures each syllable has at least one vowel if the following syllable doesn t have vowel add it to the current one param syllablelist single word or token broken up into syllables type syllablelist liststr return single word or token broken up into syllables with added syllables if necessary rtype liststr apply the ssp to return a list of syllables note sentencetext has to be tokenized first param token single word or token type token str return syllablelist single word or token broken up into syllables rtype liststr assign values from hierarchy if only one vowel return word sonority of previous focal and following phoneme focal phoneme these cases trigger syllable break no syllable break natural language toolkit tokenizers c 2001 2023 nltk project christopher hench chris l hench gmail com alex estes url https www nltk org for license information see license txt the sonority sequencing principle ssp is a language agnostic algorithm proposed by otto jesperson in 1904 the sonorous quality of a phoneme is judged by the openness of the lips syllable breaks occur before troughs in sonority for more on the ssp see selkirk 1984 the default implementation uses the english alphabet but the sonority_hiearchy can be modified to ipa or any other alphabet for the use case the ssp is a universal syllabification algorithm but that does not mean it performs equally across languages bartlett et al 2009 is a good benchmark for english accuracy if utilizing ipa pg 311 importantly if a custom hierarchy is supplied and vowels span across more than one level they should be given separately to the vowels class attribute references otto jespersen 1904 lehrbuch der phonetik leipzig teubner chapter 13 silbe pp 185 203 elisabeth selkirk 1984 on the major class features and syllable theory in aronoff oehrle eds language sound structure studies in phonology cambridge mit press pp 107 136 susan bartlett et al 2009 on the syllabification of phonemes in hlt naacl pp 308 316 syllabifies words based on the sonority sequencing principle ssp from nltk tokenize import syllabletokenizer from nltk import word_tokenize ssp syllabletokenizer ssp tokenize justification jus ti fi ca tion text this is a foobar like sentence ssp tokenize token for token in word_tokenize text this is a foo bar li ke sen ten ce param lang language parameter default is english en type lang str param sonority_hierarchy sonority hierarchy according to the sonority sequencing principle type sonority_hierarchy list str sonority hierarchy should be provided in descending order if vowels are spread across multiple levels they should be passed assigned self vowels var together otherwise should be placed in first index of hierarchy vowels nasals fricatives stops assigns each phoneme its value from the sonority hierarchy note sentence text has to be tokenized first param token single word or token type token str return list of tuples first element is character phoneme and second is the soronity value rtype list tuple str int if it s a punctuation or numbers assign 1 ensures each syllable has at least one vowel if the following syllable doesn t have vowel add it to the current one param syllable_list single word or token broken up into syllables type syllable_list list str return single word or token broken up into syllables with added syllables if necessary rtype list str apply the ssp to return a list of syllables note sentence text has to be tokenized first param token single word or token type token str return syllable_list single word or token broken up into syllables rtype list str assign values from hierarchy if only one vowel return word start syllable with first phoneme sonority of previous focal and following phoneme focal phoneme these cases trigger syllable break if it s a punctuation just break no syllable break append last phoneme
import re
import warnings
from string import punctuation

from nltk.tokenize.api import TokenizerI
from nltk.util import ngrams


class SyllableTokenizer(TokenizerI):
    

    def __init__(self, lang="en", sonority_hierarchy=False):
        
        
        
        
        
        if not sonority_hierarchy and lang == "en":
            sonority_hierarchy = [
                "aeiouy",  
                "lmnrw",  
                "zvsf",  
                "bcdgtkpqxhj",  
            ]

        self.vowels = sonority_hierarchy[0]
        self.phoneme_map = {}
        for i, level in enumerate(sonority_hierarchy):
            for c in level:
                sonority_level = len(sonority_hierarchy) - i
                self.phoneme_map[c] = sonority_level
                self.phoneme_map[c.upper()] = sonority_level

    def assign_values(self, token):
        
        syllables_values = []
        for c in token:
            try:
                syllables_values.append((c, self.phoneme_map[c]))
            except KeyError:
                if c not in "0123456789" and c not in punctuation:
                    warnings.warn(
                        "Character not defined in sonority_hierarchy,"
                        " assigning as vowel: '{}'".format(c)
                    )
                    syllables_values.append((c, max(self.phoneme_map.values())))
                    if c not in self.vowels:
                        self.vowels += c
                else:  
                    syllables_values.append((c, -1))
        return syllables_values

    def validate_syllables(self, syllable_list):
        
        valid_syllables = []
        front = ""
        vowel_pattern = re.compile("|".join(self.vowels))
        for i, syllable in enumerate(syllable_list):
            if syllable in punctuation:
                valid_syllables.append(syllable)
                continue
            if not vowel_pattern.search(syllable):
                if len(valid_syllables) == 0:
                    front += syllable
                else:
                    valid_syllables = valid_syllables[:-1] + [
                        valid_syllables[-1] + syllable
                    ]
            else:
                if len(valid_syllables) == 0:
                    valid_syllables.append(front + syllable)
                else:
                    valid_syllables.append(syllable)

        return valid_syllables

    def tokenize(self, token):
        
        
        syllables_values = self.assign_values(token)

        
        if sum(token.count(x) for x in self.vowels) <= 1:
            return [token]

        syllable_list = []
        syllable = syllables_values[0][0]  
        for trigram in ngrams(syllables_values, n=3):
            phonemes, values = zip(*trigram)
            
            prev_value, focal_value, next_value = values
            
            focal_phoneme = phonemes[1]

            
            if focal_value == -1:  
                syllable_list.append(syllable)
                syllable_list.append(focal_phoneme)
                syllable = ""
            elif prev_value >= focal_value == next_value:
                syllable += focal_phoneme
                syllable_list.append(syllable)
                syllable = ""

            elif prev_value > focal_value < next_value:
                syllable_list.append(syllable)
                syllable = ""
                syllable += focal_phoneme

            
            else:
                syllable += focal_phoneme

        syllable += syllables_values[-1][0]  
        syllable_list.append(syllable)

        return self.validate_syllables(syllable_list)