natural language toolkit tokenizers c 20012023 nltk project christopher hench chris l henchgmail com alex estes url https www nltk org for license information see license txt the legality principle is a language agnostic principle maintaining that syllable onsets and codas the beginning and ends of syllables not including the vowel are only legal if they are found as word onsets or codas in the language the english word admit must then be syllabified as admit since dm is not found wordinitially in the english language bartlett et al this principle was first proposed in daniel kahn s 1976 dissertation syllablebased generalizations in english phonology kahn further argues that there is a strong tendency to syllabify in such a way that initial clusters are of maximal length consistent with the general constraints on wordinitial consonant clusters consequently in addition to being legal onsets the longest legal onset is preferable onset maximization the default implementation assumes an english vowel set but the vowels attribute can be set to ipa or any other alphabet s vowel set for the usecase both a valid set of vowels as well as a text corpus of words in the language are necessary to determine legal onsets and subsequently syllabify words the legality principle with onset maximization is a universal syllabification algorithm but that does not mean it performs equally across languages bartlett et al 2009 is a good benchmark for english accuracy if utilizing ipa pg 311 references otto jespersen 1904 lehrbuch der phonetik leipzig teubner chapter 13 silbe pp 185203 theo vennemann on the theory of syllabic phonology 1972 p 11 daniel kahn syllablebased generalizations in english phonology phd diss mit 1976 elisabeth selkirk 1984 on the major class features and syllable theory in aronoff oehrle eds language sound structure studies in phonology cambridge mit press pp 107136 jeremy goslin and ulrich frauenfelder 2001 a comparison of theoretical and human syllabification language and speech 44 409436 susan bartlett et al 2009 on the syllabification of phonemes in hltnaacl pp 308316 christopher hench 2017 resonances in middle high german new methodologies in prosody uc berkeley syllabifies words based on the legality principle and onset maximization from nltk tokenize import legalitysyllabletokenizer from nltk import wordtokenize from nltk corpus import words text this is a wonderful sentence textwords wordtokenizetext lp legalitysyllabletokenizerwords words lp tokenizeword for word in textwords this is a won der ful sen ten ce param tokenizedsourcetext list of valid tokens in the language type tokenizedsourcetext liststr param vowels valid vowels in language or ipa representation type vowels str param legalfrequencythreshold lowest frequency of all onsets to be considered a legal onset type legalfrequencythreshold float gathers all onsets and then return only those above the frequency threshold param words list of words in a language type words liststr return set of legal onsets rtype setstr returns consonant cluster of word i e all characters until the first vowel param word single word or token type word str return string of characters of onset rtype str apply the legality principle in combination with onset maximization to return a list of syllables param token single word or token type token str return syllablelist single word or token broken up into syllables rtype liststr natural language toolkit tokenizers c 2001 2023 nltk project christopher hench chris l hench gmail com alex estes url https www nltk org for license information see license txt the legality principle is a language agnostic principle maintaining that syllable onsets and codas the beginning and ends of syllables not including the vowel are only legal if they are found as word onsets or codas in the language the english word admit must then be syllabified as ad mit since dm is not found word initially in the english language bartlett et al this principle was first proposed in daniel kahn s 1976 dissertation syllable based generalizations in english phonology kahn further argues that there is a strong tendency to syllabify in such a way that initial clusters are of maximal length consistent with the general constraints on word initial consonant clusters consequently in addition to being legal onsets the longest legal onset is preferable onset maximization the default implementation assumes an english vowel set but the vowels attribute can be set to ipa or any other alphabet s vowel set for the use case both a valid set of vowels as well as a text corpus of words in the language are necessary to determine legal onsets and subsequently syllabify words the legality principle with onset maximization is a universal syllabification algorithm but that does not mean it performs equally across languages bartlett et al 2009 is a good benchmark for english accuracy if utilizing ipa pg 311 references otto jespersen 1904 lehrbuch der phonetik leipzig teubner chapter 13 silbe pp 185 203 theo vennemann on the theory of syllabic phonology 1972 p 11 daniel kahn syllable based generalizations in english phonology phd diss mit 1976 elisabeth selkirk 1984 on the major class features and syllable theory in aronoff oehrle eds language sound structure studies in phonology cambridge mit press pp 107 136 jeremy goslin and ulrich frauenfelder 2001 a comparison of theoretical and human syllabification language and speech 44 409 436 susan bartlett et al 2009 on the syllabification of phonemes in hlt naacl pp 308 316 christopher hench 2017 resonances in middle high german new methodologies in prosody uc berkeley syllabifies words based on the legality principle and onset maximization from nltk tokenize import legalitysyllabletokenizer from nltk import word_tokenize from nltk corpus import words text this is a wonderful sentence text_words word_tokenize text lp legalitysyllabletokenizer words words lp tokenize word for word in text_words this is a won der ful sen ten ce param tokenized_source_text list of valid tokens in the language type tokenized_source_text list str param vowels valid vowels in language or ipa representation type vowels str param legal_frequency_threshold lowest frequency of all onsets to be considered a legal onset type legal_frequency_threshold float gathers all onsets and then return only those above the frequency threshold param words list of words in a language type words list str return set of legal onsets rtype set str returns consonant cluster of word i e all characters until the first vowel param word single word or token type word str return string of characters of onset rtype str apply the legality principle in combination with onset maximization to return a list of syllables param token single word or token type token str return syllable_list single word or token broken up into syllables rtype list str
from collections import Counter

from nltk.tokenize.api import TokenizerI


class LegalitySyllableTokenizer(TokenizerI):
    

    def __init__(
        self, tokenized_source_text, vowels="aeiouy", legal_frequency_threshold=0.001
    ):
        
        self.legal_frequency_threshold = legal_frequency_threshold
        self.vowels = vowels
        self.legal_onsets = self.find_legal_onsets(tokenized_source_text)

    def find_legal_onsets(self, words):
        
        onsets = [self.onset(word) for word in words]
        legal_onsets = [
            k
            for k, v in Counter(onsets).items()
            if (v / len(onsets)) > self.legal_frequency_threshold
        ]
        return set(legal_onsets)

    def onset(self, word):
        
        onset = ""
        for c in word.lower():
            if c in self.vowels:
                return onset
            else:
                onset += c
        return onset

    def tokenize(self, token):
        
        syllables = []
        syllable, current_onset = "", ""
        vowel, onset = False, False
        for char in token[::-1]:
            char_lower = char.lower()
            if not vowel:
                syllable += char
                vowel = bool(char_lower in self.vowels)
            else:
                if char_lower + current_onset[::-1] in self.legal_onsets:
                    syllable += char
                    current_onset += char_lower
                    onset = True
                elif char_lower in self.vowels and not onset:
                    syllable += char
                    current_onset += char_lower
                else:
                    syllables.append(syllable)
                    syllable = char
                    current_onset = ""
                    vowel = bool(char_lower in self.vowels)
        syllables.append(syllable)
        syllables_ordered = [syllable[::-1] for syllable in syllables][::-1]
        return syllables_ordered