<jupyter_start><jupyter_text>This notebook shows how to wrap a function with a `Process`, then to call it in a `Pipeline`  Make a new `Process`To understand how a `Process` works, we will create a new one here. We will make one specific for transliteration, then subclass that for a particular language.<jupyter_code>from cltk.core.data_types import Process
# this code in the CLTK takes the Anglo-Saxon runic alphabet and turns it into the Latin alphabet
from cltk.phonology.ang.transliteration import Transliterate
oe_runes = "ᚩᚠᛏ ᛋᚳᚣᛚᛞ ᛋᚳᛖᚠᛁᛝ ᛋᚳᛠᚦᛖᚾᚪ ᚦᚱᛠᛏᚢᛗ"  # type str
oe_latin = Transliterate().transliterate(text=oe_runes, mode="Latin")  # type str
print(oe_latin)
from dataclasses import dataclass
from copy import deepcopy
from boltons.cacheutils import cachedproperty
from cltk.core.exceptions import CLTKException
from cltk.core.data_types import Doc, Word
@dataclass
class OldEnglishTransliterationProcess(Process):
    """A simple ``Process`` for transliteration of 
    Old English in the runic alphabet.
    """
    
    language: str = None

    @cachedproperty
    def algorithm(self):
        """This is the algo to be sent over"""
        return Transliterate().transliterate

    def run(self, input_doc: Doc) -> Doc:
        output_doc = deepcopy(input_doc)
        
        for index, word_obj in enumerate(output_doc.words):
            oe_latin = self.algorithm(text=word_obj.string, mode="Latin")  # type str
            word_obj.phonetic_transcription = oe_latin
            output_doc.words[index] = word_obj

        return output_doc

oe_words = [Word(string=w) for w in oe_runes.split()]
cltk_doc_oe = Doc(words=oe_words)
translit_proc = OldEnglishTransliterationProcess
cltk_doc_oe = translit_proc().run(input_doc=cltk_doc_oe)
# now you can see that a value has been added to Word.phonetic_transcription
print(cltk_doc_oe.words[0])
print([(w.string, w.phonetic_transcription) for w in cltk_doc_oe.words])<jupyter_output>[('ᚩᚠᛏ', 'oft'), ('ᛋᚳᚣᛚᛞ', 'scyld'), ('ᛋᚳᛖᚠᛁᛝ', 'scefin'), ('ᛋᚳᛠᚦᛖᚾᚪ', 'sceathena'), ('ᚦᚱᛠᛏᚢᛗ', 'threatum')]<jupyter_text>Note that most ``Process``es in the CLTK library are more complex than this, as they allow for inheritance, which helps the project scale better. For instance:`Process` <--- `StemmingProcess` <--- {`LatinStemmingProcess`, `MiddleEnglishStemmingProcess`, `MiddleHighGermanStemmingProcess`, `OldFrenchStemmingProcess`}In these cases, the separation of `algorithm` from `run` allows for different functions to be called for each language.  Add a `Process` to a `Pipeline`Each `Process` takes a `Doc`, adds information to it and its `Word`s, then and returns the `Doc`. A `Process` like our new `OldEnglishTransliterationProcess` might belong at the end, so in the following we append it to the end of the `Pipeline`.<jupyter_code>from cltk import NLP
# Load the Old English NLP class
cltk_nlp = NLP(language="ang")
# Inspect the Pipline, which is contained in NLP
from pprint import pprint
pprint(cltk_nlp.pipeline.processes)
# Add the new custom Process to the end
cltk_nlp.pipeline.processes.append(OldEnglishTransliterationProcess)
# Now run the pipeline and see the results written to Word.phonetic_transcription
cltk_doc = cltk_nlp.analyze(text=oe_runes)
print(cltk_doc.words[0])<jupyter_output>Word(index_char_start=0, index_char_stop=3, index_token=0, index_sentence=None, string='ᚩᚠᛏ', pos=None, lemma='ᚩᚠᛏ', stem=None, scansion=None, xpos=None, upos=None, dependency_relation=None, governor=None, features={}, category={}, embedding=array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0[...]
{'fraction_non_alphanumeric': 0.15883040935672516, 'fraction_numerical': 0.041169590643274856, 'mean_word_length': 3.403707518022657, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 49, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '53612', 'license': 'MIT', 'n_tokens_mistral': 1793, 'n_tokens_neox': 1600, 'n_words': 527, 'path': 'notebooks/Make custom Process and add to Pipeline.ipynb', 'repo_name': 'pharos-alexandria/cltk', 'size': '4275'}