<jupyter_start><jupyter_text>Lexemes and occurrencesHow do you find the word occurrences that are associated to a lexeme?  ExampleWe find an example of a lexeme with only 5 occurrences.<jupyter_code>rareLexemes = [lx for (lx, nOccs) in F.lex.freqList(nodeTypes={"word"}) if nOccs == 5]
len(rareLexemes)<jupyter_output><empty_output><jupyter_text>We just pick the one of these as an example, and get the nodes which have that lexeme, among which is the lexeme node.<jupyter_code>exampleLx = rareLexemes[100]
print(f'Lexeme "{exampleLx}"')
nodes = F.lex.s(exampleLx)
for n in nodes:
    print(f"{F.otype.v(n)} {n} {F.fullo.v(n)} {F.full.v(n)}")<jupyter_output>Lexeme "גֹּוג"
lex 1544074 None None
word 1626559 gwg גוג
word 1832664 [gwg  ]גוג
word 1974200 gwg גוג
word 2106500 gwg גוג
word 2106521 gwg גוג<jupyter_text>We identify the lexeme node as `lx` and keep the word occurrence nodes as `givenOccs`<jupyter_code>lx = nodes[0]
givenOccs = nodes[1:]
print(f"lexeme {lx} is {F.lexo.v(lx)} = {F.lex.v(lx)}")<jupyter_output>lexeme 1544074 is g…øwg = גֹּוג<jupyter_text>Now we can try out ways to get from `lx` to `givenOccs` and check whether we do it right.  Via `L.d()`If you are used to working with the [BHSA](https://github.com/etcbc/bhsa),you know that you can just use the locality-down operator to get from a lexeme node to the word nodes it contains.So let's try that here.<jupyter_code>lwords = L.d(lx, otype="word")
lwords<jupyter_output><empty_output><jupyter_text>That seems to work, but let's make it crystal clear:<jupyter_code>lwords == givenOccs<jupyter_output><empty_output><jupyter_text>We can also go upwards:<jupyter_code>for w in lwords:
    llxs = L.u(w, otype="lex")
    for llx in llxs:
        print(f"{w} => {llx} {llx == lx}")<jupyter_output>1626559 => 1544074 True
1832664 => 1544074 True
1974200 => 1544074 True
2106500 => 1544074 True
2106521 => 1544074 True<jupyter_text>Via E.occThere is an edge feature that links lexeme nodes to its word occurrence nodes: `occ`. It is directed from lexeme to occurrences.<jupyter_code>ewords = E.occ.f(lx)
ewords
ewords == givenOccs<jupyter_output><empty_output><jupyter_text>We can also go back:<jupyter_code>for w in ewords:
    elxs = E.occ.t(w)
    for elx in elxs:
        print(f"{w} => {elx} {elx == lx}")<jupyter_output>1626559 => 1544074 True
1832664 => 1544074 True
1974200 => 1544074 True
2106500 => 1544074 True
2106521 => 1544074 True<jupyter_text>QueriesWhen you are hand-coding, both methods are equivalent in outcome, and probablyequal in performance.Let's see how we can work with lexemes in queries. From lexeme to occurrenceWe find the occurrences for a lexeme.<jupyter_code>lresults = A.search(
    """
lex lexo=g…øwg
  word
"""
)
lresults
eresults = A.search(
    """
lex lexo=g…øwg
-occ> word
"""
)
eresults<jupyter_output>0.11s 5 results<jupyter_text>Again, equivalent results, but it seems that the query using the edge `occ` is faster.Let's check that by querying *all* lexemes and *all* occurrences!<jupyter_code>alresults = A.search(
    """
lex
  word
"""
)
len(alresults)
aeresults = A.search(
    """
lex
-occ> word
"""
)
len(aeresults)<jupyter_output>1.63s 470845 results<jupyter_text>Well, there seems to be no significant difference.  Occurrences with an uncertain signNow we make the query a bit more complex: we want the lexemes of occurrences with an uncertain sign<jupyter_code>lresults = A.search(
    """
lex
  word
    sign unc
"""
)
len(lresults)
eresults = A.search(
    """
lex
-occ> word
    sign unc
"""
)
len(eresults)<jupyter_output>1.57s 92247 results<jupyter_text>**Warning** The data in version 0.4 will lead to different results, due to generated empty words between brackets withoutany other material.In later versions of the data we don't do that anymore.You can use the incantation with `checkout=hot` to get the latest data.  From occurrence to lexemeSuppose we want to find the lexeme nodes of all words in a line that has an uncertain sign in it.This is more difficult, because the extra condition is not on the word, but on the line, and lexemes and linesdo not embed each other in general.We have to find the lines first, then find the words, and then the lexemes associated with it.We have to go from word to lexeme.<jupyter_code>lquery = """
line
  sign unc
  w:word

lex
  w
"""<jupyter_output><empty_output><jupyter_text>Explanation:We look for lines with an uncertain sign in it and an arbitrary word and its container lexeme.<jupyter_code>lresults = A.search(lquery)
equery = """
line
  sign unc
  w:word

lex
-occ> w
"""
eresults = A.search(equery)<jupyter_output>7.94s 1096153 results
{'fraction_non_alphanumeric': 0.10090948462537895, 'fraction_numerical': 0.04958856647899523, 'mean_word_length': 4.97542043984476, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 61, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '620887', 'license': 'MIT', 'n_tokens_mistral': 1885, 'n_tokens_neox': 1644, 'n_words': 557, 'path': 'dss/cookbook/lexemes.ipynb', 'repo_name': 'sethbam9/tutorials', 'size': '4618'}