<jupyter_start><jupyter_text>SensesWe visualize the distribution of *sense labels* of verb occurrences, where verb occurrences select senses that depend on the number and nature of key constituents in their context.The verbal sense data has been computed in the[flowchart notebook](flowchart.ipynb).In this notebook we show how to make use of that data.For more information about the flowchart computations and the results,read the [wiki](https://github.com/ETCBC/valence/wiki/Workflows).Or go straight to the [legend](https://github.com/ETCBC/valence/wiki/Legend) of the sense labels.  ResultsWe have drawn the results of this notebook to the top.<jupyter_code># Run this cell as the last one.
for lex in frequentVerbLexs:  # noqa F821
    getSenseDistr(lex)  # noqa F821<jupyter_output>5277
3533
2468
1973
1912
1383
1106
1024
38
785<jupyter_text>[legend](https://github.com/ETCBC/valence/wiki/Legend)  Authors[<NAME> and <NAME>](https://github.com/ETCBC/valence/wiki/Authors)Last modified 2017-09-14.  DataWe carry out the valence project against the Hebrew Text Database of the BHSA, version `4b`.See the description of the [sources](https://github.com/ETCBC/valence/wiki/Sources).<jupyter_code>import collections  # noqa E402
from IPython.display import display  # noqa E402

import matplotlib.pyplot as plt  # noqa E402

%matplotlib inline

from tf.fabric import Fabric  # noqa E402

locations = "~/github/etcbc"
sources = ["bhsa", "valence"]
version = "2017"
modules = ["{}/tf/{}".format(s, version) for s in sources]
TF = Fabric(locations=locations, modules=modules)
api = TF.load(
    """
    sp vs
    lex gloss
    freq_lex
    predication sense
    mother
"""
)
api.makeAvailableIn(globals())<jupyter_output>0.00s loading features ...
   |     0.12s B sp                   from /Users/dirk/github/etcbc/bhsa/tf/2017
   |     0.12s B vs                   from /Users/dirk/github/etcbc/bhsa/tf/2017
   |     0.12s B lex                  from /Users/dirk/github/etcbc/bhsa/tf/2017
   |     0.00s B gloss                from /Users/dirk/github/etcbc/bhsa/tf/2017
   |     0.09s B freq_lex             from /Users/dirk/github/etcbc/bhsa/tf/2017
   |     0.06s B predication          from /Users/dirk/github/etcbc/valence/tf/2017
   |     0.02s B sense                from /Users/dirk/github/etcbc/valence/tf/2017
   |     0.20s B mother               from /Users/dirk/github/etcbc/bhsa/tf/2017
   |     0.00s Feature overview: 118 for nodes; 5 for edges; 1 configs; 7 computed
  5.20s All features loaded/computed - for details use loadLog()<jupyter_text>Overall sense label distributionLet's start with a distribution of the sense labels themselves.First collect the data. Here are the sense labels.<jupyter_code>senseLabels = sorted({F.sense.v(v) for v in F.otype.s("word")} - {None})
print(" ".join(senseLabels))<jupyter_output>-- -b -c -i -p c. d- db dc di dp i. k. l. n.<jupyter_text>We adjust the order of the sense labels manually, for clarity of presentation later on.<jupyter_code>senseLabels = "-- -i -b -p -c d- di db dp dc n. l. k. i. c.".split()
print(" ".join(senseLabels))<jupyter_output>-- -i -b -p -c d- di db dp dc n. l. k. i. c.<jupyter_text>Now we count how many times the labels occur.<jupyter_code>senseCount = collections.Counter()
noSense = []
isPredicate = {"regular", "copula"}

for v in F.sp.s("verb"):
    sense = F.sense.v(v)
    if sense is None:
        # skip words that are not verbs in the qal
        if F.vs.v(v) != "qal":
            continue
        # skip verbs in a phrase that is not a verb phrase, e.g. some participles
        # the criterion here is whether the value of feature `predication` is non trivial
        p = L.u(v, "phrase")
        if F.predication.v(p) not in isPredicate:
            continue
        noSense.append(v)
        continue
    senseCount[sense] += 1
TF.info("Counted {} senses".format(sum(senseCount.values())))
if noSense:
    TF.error("{} verb occurrences do not have a sense".format(len(noSense)))
    for v in noSense[0:10]:
        print(
            "{:<20} word {:>6} phrase {:>6} = {:<5}".format(
                "{} {}:{}".format(*T.sectionFromNode(v)),
                v,
                L.u(v, "phrase")[0],
                F.lex.v(v),
            )
        )
else:
    TF.info("All relevant verbs have been assigned a sense")<jupyter_output>15s Counted 47385 senses
    15s All relevant verbs have been assigned a sense<jupyter_text>Show the sense counts.<jupyter_code>x = range(len(senseLabels))
s = [senseCount[sl] for sl in senseLabels]
s<jupyter_output><empty_output><jupyter_text>Plot a chart with overall sense counts.<jupyter_code>fig = plt.figure()
plt.bar(x, s, tick_label=senseLabels, align="center")
plt.title("verb sense labels")<jupyter_output><empty_output><jupyter_text>[legend](https://github.com/ETCBC/valence/wiki/Legend)  Sense label distribution per verbWe show the sense label distribution for the most frequent verbs.First we need a function that plots a distribution, given a verb.The verb is given by its lexeme.<jupyter_code>x = range(len(senseLabels))


def getSenseDistr(lex):
    occurrences = [
        v for v in F.otype.s("word") if F.sense.v(v) and F.lex.v(v).rstrip("[=]") == lex
    ]
    print(len(occurrences))
    gloss = ", ".join(sorted({F.gloss.v(L.u(v, otype="lex")[0]) for v in occurrences}))
    senseCount = collections.Counter()
    for v in occurrences:
        sense = F.sense.v(v)
        senseCount[sense] += 1
    s = [senseCount[sl] for sl in senseLabels]
    plt.figure()
    plt.bar(x, s, tick_label=senseLabels, align="center")
    plt.title("sense of {} = {}".format(lex, gloss))
getSenseDistr(">MR")<jupyter_output>5277<jupyter_text>[legend](https://github.com/ETCBC/valence/wiki/Legend)<jupyter_code>getSenseDistr("DBR")<jupyter_output>38<jupyter_text>[legend](https://github.com/ETCBC/valence/wiki/Legend)  Sense label distribution frequent verbsWe produce the sense label distributions for the 10 most frequent verbs.<jupyter_code>frequentVerbNodes = sorted(
    (v for v in F.otype.s("word") if F.sp.v(v) == "verb" and F.vs.v(v) == "qal"),
    key=lambda v: -F.freq_lex.v(v),
)
len(frequentVerbNodes)
frequentVerbLexSet = set()
frequentVerbLexs = []
limit = 10
for v in frequentVerbNodes:
    if len(frequentVerbLexs) == 10:
        break
    lex = F.lex.v(v).rstrip("[=]")
    if lex in frequentVerbLexSet:
        continue
    frequentVerbLexSet.add(lex)
    frequentVerbLexs.append(lex)
frequentVerbLexs<jupyter_output><empty_output><jupyter_text>What's the matter with DBR?DBR is a frequent verb, why do we see so few sense assignments?That's easy: we do only the `qal` stems, and DBR is strong in `piel`s and weak in `qal`s:<jupyter_code>dbr = [v for v in F.sp.s("verb") if F.lex.v(v) == "DBR["]
dbrByStem = collections.Counter()
for v in dbr:
    dbrByStem[F.vs.v(v)] += 1

print(
    """
Frequency of the verb DBR: {:>4}
# Qal forms:               {:>4}
# Piel forms:              {:>4}
""".format(
        F.freq_lex.v(dbr[0]),
        dbrByStem["qal"],
        dbrByStem["piel"],
    )
)<jupyter_output>Frequency of the verb DBR: 1138
# Qal forms:                 41
# Piel forms:              1089<jupyter_text>What's the matter with >MR?*to speak* is a transitive verb, you would expect lots of occurrences with a direct object, so senses`d-`, `di`, `db`, `dp`, `dc`.Instead, we see an overwhelming number of `--`.Why is that?In many cases, this verbs introduces direct speech.The direct speech could be regarded as the direct object of `>MR` in those cases.But the creators of the database have encoded the direct speech as separate sentences, hence our algorithm does notdetect these kinds of direct objects.Let us have a look at a specific case, the clause starting with word 35.<jupyter_code>wn = 35
cn = L.u(wn, "clause")[0]
can = L.u(wn, "clause_atom")[0]
sn = L.u(wn, "sentence")[0]
vn = L.u(wn, "verse")[0]
directSpeech = T.text(L.d(cn, "word"))
sentence = T.text(L.d(sn, "word"))
wholeVerse = T.text(L.d(vn, "word"))
print(
    "Consider clause {} in sentence {} in verse {}".format(
        directSpeech, sentence, wholeVerse
    )
)<jupyter_output>Consider clause יְהִ֣י אֹ֑ור  in sentence יְהִ֣י אֹ֑ור  in verse וַיֹּ֥אמֶר אֱלֹהִ֖ים יְהִ֣י אֹ֑ור וַֽיְהִי־אֹֽור׃<jupyter_text>It might be handy to have a look at[this verse on SHEBANQ](https://shebanq.ancient-data.org/hebrew/text?qactive=hlcustom&qsel_one=white&qpub=x&qget=v&wactive=hlcustom&wsel_one=black&wpub=x&wget=v&nactive=hlcustom&nsel_one=black&npub=x&nget=v&chapter=1&lang=en&book=Genesis&qw=n&tr=hb&tp=txt_tb1&iid=Mnx2YWxlbmNl&verse=3&version=4b&mr=m&page=2&c_q1510=turquoise&c_w1BRAv=yellow&wd4_statfl=v&ph_arela=v&wd4_statrl=v&sn_an=x&cl=v&wd1_lang=x&wd1_subpos=x&wd2_person=v&sp_rela=v&wd1_pdp=x&sn_n=v&wd3_uvf=x&ph_fun=v&wd1_nmtp=x&gl=x&sp_n=v&pt=x&ph_an=v&ph_typ=x&cl_typ=x&tt=x&wd4_statro=x&wd3_vbs=x&wd1=v&tl=x&wd3=x&wd4=x&wd2_gender=v&ph=x&wd3_vbe=v&wd1_pos=x&ph_det=v&ph_rela=x&wd4_statfo=x&tl_tlv=x&wd2_stem=v&wd2_state=v&ht=v&ph_n=v&tl_tlc=x&cl_tab=x&wd3_nme=x&hl=x&cl_par=x&cl_an=x&cl_n=v&wd3_prs=v&wd3_pfm=x&sp=x&cl_code=x&ht_hk=x&wd2=x&hl_hlc=x&cl_rela=x&wd2_gnumber=v&wd2_tense=v&cl_txt=x&wd1_n=v&sn=v&ht_ht=v&hl_hlv=v&pref=alt).After loading the page, click on the verse number 3 to see it with some underlying data in view, in this case the word, clause and sentence numbers.The question is now, is there a link from this clause to the preceding וַיֹּ֥אמֶר ?Let's ask for the mother of this clause, or its enclosing sentence.<jupyter_code>print(
    """
mother of clause_atom: {}
mother of clause:      {}
mother of sentence:    {}
""".format(
        E.mother.f(can),
        E.mother.f(cn),
        E.mother.f(sn),
    )
)<jupyter_output>mother of clause_atom: (515658,)
mother of clause:      ()
mother of sentence:    ()<jupyter_text>Ah, the `clause_atom` has a mother link. Let's see where it leads.<jupyter_code>mother = E.mother.f(can)[0]
print(
    """
Mother of clause_atom {} is a node {} of type {}
{}
""".format(
        can, mother, F.otype.v(mother), T.text(L.d(mother, "word"))
    )
)<jupyter_output>Mother of clause_atom 515659 is a node 515658 of type clause_atom
וַיֹּ֥אמֶר אֱלֹהִ֖ים<jupyter_text>So it is a clause_atom to clause_atom link. Hm...<jupyter_code>verbsWithDaughters = []
caWithDaughters = []

for v in F.sp.s("verb"):
    if F.vs.v(v) != "qal":
        continue
    p = L.u(v, "phrase")[0]
    if F.predication.v(p) not in isPredicate:
        continue
    ca = L.u(v, "clause_atom")[0]
    daughters = E.mother.t(ca)
    if len(daughters) == 0:
        continue
    verbsWithDaughters.append(v)
    caWithDaughters.append(ca)
print(
    "{} occurences of {} verbs in clause atoms with daughters".format(
        len(verbsWithDaughters),
        len({F.lex.v(v) for v in verbsWithDaughters}),
    )
)
for ca in caWithDaughters[0:10]:
    print(
        "\n{:<15} mother {}".format(
            "{} {}:{}".format(*T.sectionFromNode(ca)),
            T.text(L.d(ca, "word")).rstrip("\n"),
        )
    )
    for da in E.mother.t(ca):
        print(
            "\t{:<15} daughter {}".format(
                "{} {}:{}".format(*T.sectionFromNode(da)),
                T.text(L.d(da, "word")).rstrip("\n"),
            )
        )<jupyter_output>Genesis 1:1     mother בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ 
	Genesis 1:2     daughter וְהָאָ֗רֶץ הָיְתָ֥ה תֹ֨הוּ֙ וָבֹ֔הוּ 
	Genesis 1:3     daughter וַיֹּ֥אמֶר אֱלֹהִ֖ים 

Genesis 1:2     mother וְהָאָ֗רֶץ הָיְתָ֥ה תֹ֨הוּ֙ וָבֹ֔הוּ 
	Genesis 1:2     daughter וְחֹ֖שֶׁךְ עַל־פְּנֵ֣י תְהֹ֑ום 

Genesis 1:3     mother וַיֹּ֥אמֶר אֱלֹהִ֖ים 
	Genesis 1:3     daughter יְהִ֣י אֹ֑ור 
	Genesis 1:3     daughter וַֽיְהִי־אֹֽור׃ 
	Genesis 1:4     daughter וַיַּ֧רְא אֱלֹהִ֛ים אֶת־הָאֹ֖ור 
	Genesis 1:6     daughter וַיֹּ֣אמֶר אֱלֹהִ֔ים 

Genesis 1:4     mother וַיַּ֧רְא אֱלֹהִ֛ים אֶת־הָאֹ֖ור 
	Genesis 1:4     daughter כִּי־טֹ֑וב 
	Genesis 1:4     daughter וַיַּבְדֵּ֣ל אֱלֹהִ֔ים בֵּ֥ין הָאֹ֖ור וּבֵ֥ין הַחֹֽשֶׁךְ׃ 

Genesis 1:5     mother וַיִּקְרָ֨א אֱלֹהִ֤ים׀ לָאֹור֙ יֹ֔ום 
	Genesis 1:5     daughter וְלַחֹ֖שֶׁךְ קָ֣רָא לָ֑יְלָה 
	Genesis 1:5     daughter וַֽיְהִי־עֶ֥רֶב 

Genesis 1:5     mother וַֽיְהִי־עֶ֥רֶב 
	Genesis 1:5     daughter וַֽיְהִי־בֹ֖קֶר 

Genesis 1:5    [...]
{'fraction_non_alphanumeric': 0.1505437893531769, 'fraction_numerical': 0.024531850519257502, 'mean_word_length': 3.570254110612855, 'pattern_counts': {'":': 2, '<': 55, '<?xml version=': 0, '>': 59, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '289344', 'license': 'MIT', 'n_tokens_mistral': 5262, 'n_tokens_neox': 5074, 'n_words': 1141, 'path': 'programs/senses.ipynb', 'repo_name': 'ETCBC/valence', 'size': '12237'}