<jupyter_start><jupyter_text>You might want to consider the [start](search.ipynb) of this tutorial.Short introductions to other TF datasets:* [Dead Sea Scrolls](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/dss.ipynb),* [Old Babylonian Letters](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/oldbabylonian.ipynb),or the* [Q'uran](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/quran.ipynb)  TreesThe textual objects of the BHSA text are syntactic, but they are not syntax trees.The BHSA is the result of a data-driven parsing strategy with occasional human decisions.It results in functional objects such as sentences, clauses, and phrases,which are build from chunks called sentece-atoms, clause-atoms, and phrase-atoms.There is no deeper nesting of clauses within phrases, or even clauses within clauses or phrases within phrases.Instead, whenever objects are linguistically nested, there is an edge called `mother` between theobjects in question.For people that prefer to think in trees, we have unwrapped the `mother` relationship between clausesand made tree structures out of the data.The whole generation process of trees, including the quirks underway, is documentedin the notebook[trees.ipynb](https://nbviewer.jupyter.org/github/etcbc/trees/blob/master/programs/trees.ipynb).You see it done there for version 2017.We have used an ordinary Python program to generate trees for all  versions of the BHSA:[alltrees.py](https://github.com/etcbc/trees/blob/master/programs/alltrees.py)Those trees are available as a feature on sentence nodes, and you can load those featuresalongside the BHSA data.Here we show some examples of what you can do with it.<jupyter_code>%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>IncantationThe ins and outs of installing Text-Fabric, getting the corpus, and initializing a notebook areexplained in the [start tutorial](start.ipynb).<jupyter_code>from utils import structure, layout
from tf.app import use<jupyter_output><empty_output><jupyter_text>Note that we load the trees module.We also load the morphology of Open Scriptures for example usage later on.<jupyter_code>A = use("bhsa", mod="etcbc/trees/tf,etcbc/bridging/tf", hoist=globals())<jupyter_output><empty_output><jupyter_text>We first inspect the nature of these features, lets pick the first, last and middle sentence ofthe Hebrew Bible<jupyter_code>sentences = F.otype.s("sentence")
examples = (sentences[0], sentences[len(sentences) // 2], sentences[-1])<jupyter_output><empty_output><jupyter_text>We examine feature `tree`:<jupyter_code>for s in examples:
    print(F.tree.v(s))<jupyter_output>(S(C(PP(pp 0)(n 1))(VP(vb 2))(NP(n 3))(PP(U(pp 4)(dt 5)(n 6))(cj 7)(U(pp 8)(dt 9)(n 10)))))
(S(C(VP(vb 0))(NP(n 1))))
(S(C(CP(cj 0))(VP(vb 1))))<jupyter_text>Now  `treen`:<jupyter_code>for s in examples:
    print(F.treen.v(s))<jupyter_output>(S{1172308}(C{427559}(PP{651573}(pp 0)(n 1))(VP{651574}(vb 2))(NP{651575}(n 3))(PP{651576}(U{1300539}(pp 4)(dt 5)(n 6))(cj 7)(U{1300540}(pp 8)(dt 9)(n 10)))))
(S{1204166}(C{471249}(VP{782581}(vb 0))(NP{782582}(n 1))))
(S{1236024}(C{515689}(CP{904774}(cj 0))(VP{904775}(vb 1))))<jupyter_text>The structure of the trees is the same, but `treen` has numbers between braces in the tags of the nodes.These numbers are the Text-Fabric nodes of the sentences, clauses and phrases that the nodes of the treecorrespond to.  Using treesThese strings are not very pleasant to the eye.For one thing, we see numbers instead of words.They also seem a bit unwieldy to integrate with the usual text-fabric business.But nothing is farther from the truth.We show how to* produce a multiline view* see the words (in several representations)* add a gloss* add morphological data from an other project (**Open Scriptures**)Honesty compels us to note that we make use of a bunch of auxiliary functions in anaccompanying `utils` pacckage:<jupyter_code>passage = ("Job", 3, 16)
passageStr = "{} {}:{}".format(*passage)
verse = T.nodeFromSection(passage)
sentence = L.d(verse, otype="sentence")[0]
firstSlot = L.d(sentence, otype="word")[0]
stringTree = F.tree.v(sentence)
print(f"{passageStr} - first word = {firstSlot}\n\ntree =\n{stringTree}")<jupyter_output>Job 3:16 - first word = 336990

tree =
(S(C(Ccoor(CP(cj 0))(PP(pp 1)(U(n 2))(U(vb 3)))(NegP(ng 4))(VP(vb 5)))(Ccoor(PP(pp 6)(n 7)(Cattr(NegP(ng 8))(VP(vb 9))(NP(n 10)))))))<jupyter_text>ParsingKey to effective manipulation of tree strings is to parse them into tree structures: lists of lists.Here we use the generic utility `structure()`:<jupyter_code>tree = structure(stringTree)
tree<jupyter_output><empty_output><jupyter_text>Apply layoutHaving the real tree structure in hand, we can layout it in all kinds of ways.We use the generic utility `layout()` todisplay it a bit more friendly and to replace the numbers by real Text-Fabric slot numbers:<jupyter_code>print(layout(tree, firstSlot, str))<jupyter_output>S
    C
      Ccoor
        CP
          cj 336990
        PP
          pp 336991
          U
            n 336992
          U
            vb 336993
        NegP
          ng 336994
        VP
          vb 336995
      Ccoor
        PP
          pp 336996
          n 336997
          Cattr
            NegP
              ng 336998
            VP
              vb 336999
            NP
              n 337000<jupyter_text>That opens up the way to get the words in.The third argument of `layout()` above is `str`, which is a function that is applied to the slot numbers.It returns those numbers as string, and this is what ends up in the layout.  Fillin the wordsWe can pass any function, why not the function that looks up the word?Remember that `F.g_word_utf8.v` is a function that returns the full Hebrew word given a slot node.<jupyter_code>print(layout(tree, firstSlot, F.g_word_utf8.v))<jupyter_output>S
    C
      Ccoor
        CP
          cj אֹ֚ו
        PP
          pp כְ
          U
            n נֵ֣פֶל
          U
            vb טָ֭מוּן
        NegP
          ng לֹ֣א
        VP
          vb אֶהְיֶ֑ה
      Ccoor
        PP
          pp כְּ֝
          n עֹלְלִ֗ים
          Cattr
            NegP
              ng לֹא
            VP
              vb רָ֥אוּ
            NP
              n אֹֽור<jupyter_text>Add a gloss<jupyter_code>def gloss(n):
    lexNode = L.u(n, otype="lex")[0]
    return f'{F.g_word_utf8.v(n)} "{F.gloss.v(lexNode)}"'


print(layout(tree, firstSlot, gloss))<jupyter_output>S
    C
      Ccoor
        CP
          cj אֹ֚ו "or"
        PP
          pp כְ "as"
          U
            n נֵ֣פֶל "miscarriage"
          U
            vb טָ֭מוּן "hide"
        NegP
          ng לֹ֣א "not"
        VP
          vb אֶהְיֶ֑ה "be"
      Ccoor
        PP
          pp כְּ֝ "as"
          n עֹלְלִ֗ים "child"
          Cattr
            NegP
              ng לֹא "not"
            VP
              vb רָ֥אוּ "see"
            NP
              n אֹֽור "light"<jupyter_text>MorphologyIn 2018 I compared the morphology of Open Scriptures with that of the BHSA.See [brdiging](https://nbviewer.jupyter.org/github/ETCBC/bridging/blob/master/programs/BHSAbridgeOSM.ipynb).As a by-product I saved their morphology as a Text-Fabric feature on words.So we can add it to our trees.We also show the nesting depth in the resulting tree.<jupyter_code>def osmPhonoGloss(n):
    lexNode = L.u(n, otype="lex")[0]
    return (
        f'({F.osm.v(n)}) {F.g_word_utf8.v(n)} [{F.phono.v(n)}] "{F.gloss.v(lexNode)}"'
    )


print(layout(tree, firstSlot, osmPhonoGloss, withLevel=True))<jupyter_output>1  S
 2    C
 3      Ccoor
 4        CP
 5          cj (HC) אֹ֚ו [ˈʔô] "or"
 4        PP
 5          pp (HR) כְ [ḵᵊ] "as"
 5          U
 6            n (HNcmsa) נֵ֣פֶל [nˈēfel] "miscarriage"
 5          U
 6            vb (HVqsmsa) טָ֭מוּן [ˈṭāmûn] "hide"
 4        NegP
 5          ng (HTn) לֹ֣א [lˈō] "not"
 4        VP
 5          vb (HVqi1cs) אֶהְיֶ֑ה [ʔehyˈeh] "be"
 3      Ccoor
 4        PP
 5          pp (HR) כְּ֝ [ˈkᵊ] "as"
 5          n (HNcmpa) עֹלְלִ֗ים [ʕōlᵊlˈîm] "child"
 5          Cattr
 6            NegP
 7              ng (HTn) לֹא [lō-] "not"
 6            VP
 7              vb (HVqp3cp) רָ֥אוּ [rˌāʔû] "see"
 6            NP
 7              n (HNcbsa) אֹֽור [ʔˈôr] "light"
{'fraction_non_alphanumeric': 0.1122314247989437, 'fraction_numerical': 0.0339695114632097, 'mean_word_length': 2.9714013346043853, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '35104', 'license': 'MIT', 'n_tokens_mistral': 3254, 'n_tokens_neox': 2970, 'n_words': 879, 'path': 'bhsa/trees.ipynb', 'repo_name': 'annotation/tutorials', 'size': '8331'}