<jupyter_start><jupyter_text>You might want to consider the [start](search.ipynb) of this tutorial.Short introductions to other TF datasets:* [Dead Sea Scrolls](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/dss.ipynb),* [Old Babylonian Letters](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/oldbabylonian.ipynb),or the* [Q'uran](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/quran.ipynb)<jupyter_code>%load_ext autoreload
%autoreload 2
from tf.app import use
A = use('bhsa', hoist=globals())
# A = use("bhsa:clone", checkout="clone", hoist=globals())<jupyter_output><empty_output><jupyter_text>RelationshipsThere are relationship between objects that are about their identities, the way theyoccupy space, and the way they are connected.Are two objects the same, do they occupy the same slots, do they overlap, is one embedded in the other,does one come before the other? Is there an edge from one to another?Although these relationships are easy to define, and even easy to implement,they may be very costly to use.When searching, most of them have to be computed very many times.Some of them have been precomputed and stored in an index, e.g. the embedding relationships.They can be used without penalty.Other relations are not suitable for pre-computing: most inequality relations are of that kind.It would require an enormous amount of storage to pre-compute for each node the set of nodes thatoccupy different slots. This type of relation will not be used in narrowing down the search space,which means that it may take more time to get the results.We are going to test all of our relationships here.Let us first see what relationships we have:<jupyter_code>S.relationsLegend()<jupyter_output>= left equal to right (as node)
                      # left unequal to right (as node)
                      < left before right (in canonical node ordering)
                      > left after right (in canonical node ordering)
                     == left occupies same slots as right
                     && left has overlapping slots with right
                     ## left and right do not have the same slot set
                     || left and right do not have common slots
                     [[ left embeds right
                     ]] left embedded in right
                     << left completely before right
                     >> left completely after right
                     =: left and right start at the same slot
                     := left and right end at the same slot
                     :: left and right start and end at the same slot
                     <: left immediately before right
                     :> left immediately after right
   [...]<jupyter_text>The top of the list are identity and spatial relationships.We are going to dicuss them. They are documented in[relationships](https://annotation.github.io/text-fabric/tf/about/searchusage.htmlrelational-operators)The bottom of the list are relationships defined by the edge features of your dataset.We have discussed them in [advanced](searchAdvanced.ipynb).  Identity and order = equal as nodeThe `=` means that both parts are the same node. Left and right are not two things with similar properties,no, they are one and the same thing.Useful if the thing you search for it part of two wildly different patterns.<jupyter_code>query = """
v1:verse
  sentence
    clause rela=Objc
      phrase
        word sp=verb gn=f nu=pl
v2:verse
  sentence
    c1:clause
    c2:clause
    c3:clause
    c1 < c2
    c2 < c3
v1 = v2
"""
results = A.search(query)<jupyter_output>0.81s 48 results<jupyter_text>We show the results with the first clause in magenta and the second sentence in cyan.Note that the first and the second sentence may be the same sentence!**And observe that the last clause is the same one as the first one, hence they have the same color.**<jupyter_code>skipCols = "1 6"  # the verses
colorMap = {2: "magenta", 6: "cyan"}
A.table(results, end=1, colorMap=colorMap, skipCols=skipCols)
A.show(results, end=1, colorMap=colorMap, skipCols=skipCols)<jupyter_output><empty_output><jupyter_text>unequal as node`n  m` if `n` and `m` are not the same node.If you write a template, and you know that one node should come before another one,consider using ``, which will constrain the results better.We have seen this in action in the search for gapped phrases.   canonical order`n < m` if `n` comes before `m` in the[canonical ordering](https://annotation.github.io/text-fabric/tf/core/nodes.html)of nodes.We have seen them in action before.  Space occupationWe show that the following relationships also work with custom sets, as introduced in the[searchAdvanced tutorial](searchAdvanced.ipynb).We make two custom sets, `common` and `rare`, consisting of nodes whose contained slots have allcommon lexemes or some rare lexemes, like we did in[searchAdvanced](searchAdvanced.ipynb).<jupyter_code>COMMON_RANK = 100
RARE_RANK = 500

common = set()
rare = set()

for n in N.walk():
    nTp = F.otype.v(n)
    if nTp == "lex":
        continue
    if nTp == "word":
        ranks = [F.rank_lex.v(n)]
    else:
        ranks = [F.rank_lex.v(w) for w in L.d(n, otype="word")]
    maxRank = max(ranks)
    minRank = min(ranks)
    if maxRank < COMMON_RANK:
        common.add(n)
    if maxRank > RARE_RANK:
        rare.add(n)

print(f"{len(common):>6} members in set common")
print(f"{len(rare):>6} members in set rare")<jupyter_output>669195 members in set common
425320 members in set rare<jupyter_text>Now we can do all kinds of searches within the domain of `common` and `rare` things.We give the names to all the sets and put them in a dictionary.<jupyter_code>customSets = dict(
    common=common,
    rare=rare,
)<jupyter_output><empty_output><jupyter_text>**Expert remark**Note that these sets contain both slot nodes and non-slot nodes.The code that implements the basic relationship is heavily optimized and contains case distinctions as to whether nodes are slotor non-slot.For ordinary node types, it is clear on beforehand whether its nodes are slot or non-slot, but custom sets may contain both.So our `frequent` and `infrequent` sets are good tests whether the basic relationships are correctly implemented.  == same slotsTwo objects are extensionally equal if they occupy exactly the same slots.<jupyter_code>query = """
v:verse
    s:sentence
v == s
"""
results = A.search(query)
A.table(results, end=7, skipCols="1")
A.show(results, start=1, end=1, skipCols="1")<jupyter_output><empty_output><jupyter_text>Now a similar query with the custom sets:<jupyter_code>query = """
v:common otype=verse
    s:common otype=sentence
v == s
"""
resultsCustom = A.search(query, sets=customSets)
A.table(resultsCustom, end=10, skipCols="1")
A.show(resultsCustom, start=1, end=1, skipCols="1")<jupyter_output>1.53s 135 results<jupyter_text>As a check we compute manually the maximum rank of the lexemes in the clauses yielded by the query without the custom sets:<jupyter_code>rejected = 0
for (verse, sentence) in results:
    maxRank = max(F.rank_lex.v(w) for w in L.d(verse, otype="word"))
    if maxRank >= COMMON_RANK:
        rejected += 1

print(f"Rejected {rejected} non-common results, leaving {len(results) - rejected} ones")<jupyter_output>Rejected 3448 non-common results, leaving 135 ones<jupyter_text>&& overlapTwo objects overlap if and only if they share at least one slot.This is quite costly to use in some cases.We are going to find the sentences that overlap with two verses.<jupyter_code>query = """
sentence
/with/
v1:verse
&& ..
v2:verse
&& ..
v1 < v2
/-/
"""<jupyter_output><empty_output><jupyter_text>Explanation: the query looks for sentences and delivers results that are tuples with only a sentence node.This is because the stuff within the `/with/` quantifier does not contribute to the result tuples.The `/with/` quantifier poses a few restrictions on its parent, the `sentence`.From within the quantifier you can refer to the parent by `..`.The condition is that there are verses `v1` and `v2` that have overlap with the sentence, and that `v1` comes before `v2`.The result is a tuple of exactly the sentences that span multiple verses.<jupyter_code>results = A.search(query)
A.table(results, end=5)
A.show(results, condensed=False, baseTypes="clause", start=1, end=3)<jupyter_output><empty_output><jupyter_text>Now with custom sets:<jupyter_code>query = """
common otype=sentence
/with/
v1:verse
&& ..
v2:verse
&& ..
v1 < v2
/-/
"""
resultsCommon = A.search(query, sets=customSets)
A.table(resultsCommon)<jupyter_output><empty_output><jupyter_text>The following query has the same results but is less insightful.<jupyter_code>query = """
sentence
  =: w1:word
  w2:word
  :=

v1:verse
  wv1:word

v2:verse
  wv2:word

w1 = wv1
w2 = wv2
v1 < v2
"""
fastResults = A.search(query)
sorted((x[0],) for x in fastResults) == sorted(results)<jupyter_output><empty_output><jupyter_text>not the same slotsTrue when the two objects in question do not occupy exactly the same set of slots.This is a very loose relationship.We look for sentences that start with a sentence atom that is not co-extensive with its sentence.<jupyter_code>query = """
s:sentence
=: sa:sentence_atom
s ## sa
"""
results = A.search(query)
A.table(results, end=5)
A.show(results, baseTypes="phrase", start=1, end=1)<jupyter_output>0.26s 702 results<jupyter_text>Now in the common domain.<jupyter_code>query = """
s:common otype=sentence
=: sa:sentence_atom
s ## sa
"""
resultsCustom = A.search(query, sets=customSets)
A.table(resultsCustom, end=5)
A.show(resultsCustom, baseTypes="phrase", start=1, end=1)<jupyter_output>0.77s 20 results<jupyter_text>|| disjoint slotsTrue when the two objects in question do not share any slots.This is a rather loose relationship.This can be used for locating gaps: a textual object that lies inside a gap of another object.See also [gaps](searchGaps.ipynb).Here we check whether there are phrases with disjoint subphrases.<jupyter_code>query = """
p:phrase
  s1:subphrase
  < s2:subphrase
s1 || s2
"""
results = A.search(query)
A.table(results, end=7)
A.show(results, start=1, end=1)<jupyter_output>1.52s 168504 results<jupyter_text>Now in the common domain.<jupyter_code>query = """
p:common otype=phrase
  s1:subphrase
  < s2:subphrase
s1 || s2
"""
resultsCustom = A.search(query, sets=customSets)
A.table(resultsCustom, end=7)
A.show(resultsCustom, start=1, end=1)<jupyter_output>1.06s 10999 results<jupyter_text>We see that Genesis 1:1 has fallen out. Let's check the ranks of the lexemes of its last phrase:<jupyter_code>firstSentence = F.otype.s("sentence")[0]
lastPhrase = L.d(firstSentence, otype="phrase")[-1]
[(F.g_word_utf8.v(w), F.rank_lex.v(w)) for w in L.d(lastPhrase, otype="word")]<jupyter_output><empty_output><jupyter_text>There you have it: the heavens are rare!  [[ and ]] embedding`n [[ m` if object `n` embeds `m`.`n ]] m` if object `n` lies embedded in `m`.These relations are used implicitly in templates when there is indentation:```s:sentence  p:phrase    w1:word gn=f    w2:word gn=m```The template above implicitly states the following embeddings:* `s ]] p`* `p ]] w1`* `p ]] w2`We have seen these relations in action.  Positioning > before and after (slot-wise)These relations test whether one object comes before or after an other,in the sense that the slotsoccupied by the one object lie completelybefore or after the slots occupied by the other object.<jupyter_code>query = """
sentence
  c1:clause
  p:phrase
  c2:clause
  c1 << p
  c2 >> p
"""
results = A.search(query)
colorMap = {2: "lightyellow", 3: "cyan", 4: "magenta", 5: "blue"}
A.table(results, end=5, baseTypes="phrase", colorMap=colorMap)
A.show(results, condensed=False, baseTypes="phrase", start=1, end=1, colorMap=colorMap)<jupyter_output>3.60s 136137 results<jupyter_text>In the common domain:<jupyter_code>query = """
sentence
  c1:common otype=clause
  p:rare otype=phrase
  c2:common otype=clause
  c1 << p
  c2 >> p
"""
resultsCustom = A.search(query, sets=customSets)
A.table(resultsCustom, end=5, baseTypes="phrase", colorMap=colorMap)
A.show(
    resultsCustom,
    condensed=False,
    baseTypes="phrase",
    start=1,
    end=1,
    colorMap=colorMap,
)<jupyter_output>2.01s 509 results<jupyter_text>=: same start slotsThis relation holds when the left and right hand sides are nodes that have the same first slot.It serves to enforce the the children of a parent are textually the first things inside thatparent. We have seen it in action before. := same end slotsThis relation holds when the left and right hand sides are nodes that have the same last slotIt serves to enforce the the children of a parent are textually the last things inside thatparent. We have seen it in action before. :: same boundary slotsThis relation holds when `=:` and `:=` both hold between the left and right hand sides.It serves to look for parents with single children, or at least, where the parent is textually spanned by a single child. Let us look for a phrase, whose start and end slots coincide with its containing clause.But only if the phrase does not conincide with its parent clause.<jupyter_code>query = """
c:clause
  :: p:phrase
c ## p
"""
results = A.search(query)
A.table(results, start=1, end=5, baseTypes="phrase")
A.show(results, start=1, end=5, condenseType="clause", baseTypes="phrase")<jupyter_output>0.36s 52 results<jupyter_text>Here you see an extra phrase in such clauses, lying embedded in the clause-spanning phrase.A nice case of **Mind the gap!**. In the common domain:<jupyter_code>query = """
c:common otype=clause
  :: p:common otype=phrase
c ## p
"""
resultsCustom = A.search(query, sets=customSets)
A.table(resultsCustom, start=1, end=10, baseTypes="phrase")
A.show(resultsCustom, start=1, end=5, condenseType="clause", baseTypes="phrase")<jupyter_output>1.52s 1 result<jupyter_text><: adjacent beforeThis relation holds when the left hand sides ends in a slot that lies before the first slot of the right hand side.It serves to enforce an ordering between siblings of a parent. :> adjacent afterThis relation holds when the left hand sides starts in a slot that lies after the last slot of the right hand side.<jupyter_code>query = """
clause
  phrase
  <: phrase
"""
results = A.search(query)
A.table(results, start=1, end=3, baseTypes="phrase")
query = """
clause
  phrase
  :> phrase
"""
results = A.search(query)
A.table(results, start=1, end=3, baseTypes="phrase")<jupyter_output>1.53s 162653 results<jupyter_text>Playing with common and rare:<jupyter_code>query = """
clause
  common otype=phrase
  <: rare otype=phrase
"""
resultsCommon = A.search(query, sets=customSets)
A.table(resultsCommon, start=1, end=3, baseTypes="phrase")
query = """
clause
  common otype=phrase
  :> rare otype=phrase
"""
resultsCommon = A.search(query, sets=customSets)
A.table(resultsCommon, start=1, end=3, baseTypes="phrase")<jupyter_output>1.54s 9684 results<jupyter_text>Another example: are there clauses with multiple clause atoms without a gap between the two?<jupyter_code>query = """
clause
  clause_atom
  <: clause_atom
"""
results = A.search(query)
A.table(results, start=1, end=10, baseTypes="clause_atom")<jupyter_output>0.40s 0 results<jupyter_text>Conclusion: there is always textual material between clause_atoms of the same clause.If we lift the adjacency to sequentially before (`<<`) we do get results:<jupyter_code>query = """
clause
  clause_atom
  << clause_atom
"""
results = A.search(query)
A.table(results, start=1, end=5, baseTypes="clause_atom")
A.show(results, start=1, end=1, baseTypes="clause_atom")<jupyter_output>0.41s 2707 results<jupyter_text>NearnessThe relations with `:` in their name always have a requirement somewhere that a slot of theleft hand node equals a slot of the right hand node, or that the two are adjacent.All these relationships can be relaxed by a **nearness number**.If you put a number *k* inside the relationship symbols, those restrictions will be relaxed to*the one slot and the other slot should have a mutual distance of at most k*. =k: same start within k slotsHere is an example.First we look for clauses, with a phrase in it that starts at thesame slot as the clause.<jupyter_code>results = A.search(
    """
chapter book=Genesis chapter=1
  clause
    =: phrase
"""
)<jupyter_output>0.40s 126 results<jupyter_text>Now we add a bit of freedom, but not much: 0. Indeed, this is no extrafreedom, and it should give the same number of results.<jupyter_code>results = A.search(
    """
chapter book=Genesis chapter=1
  clause
    =0: phrase
"""
)<jupyter_output>0.48s 126 results<jupyter_text>Now we add real freedom: 1 and 2<jupyter_code>results = A.search(
    """
chapter book=Genesis chapter=1
    clause
      =1: phrase
"""
)
results = A.search(
    """
chapter book=Genesis chapter=1
    clause
      =2: phrase
"""
)<jupyter_output>0.33s 315 results<jupyter_text>Let us see some cases:<jupyter_code>A.table(results, start=1, end=10, baseTypes="phrase", skipCols="1")
A.show(
    results,
    condensed=False,
    start=1,
    end=4,
    colorMap={2: "yellow", 3: "cyan"},
    baseTypes="phrase",
    skipCols="1",
)<jupyter_output><empty_output><jupyter_text>The first and second result show the same clause, with its first and second phrase respectively.Note that we look for phrases that lie embedded in their clause.So we do not get phrases of a preceding clause.But if we want, we can get those as well.<jupyter_code>results = A.search(
    """
chapter book=Genesis chapter=1
  c:clause
  p:phrase

  c =2: p
"""
)<jupyter_output>0.07s 485 results<jupyter_text>We have more results now. Here is a closer look:<jupyter_code>A.table(results, start=1, end=5, baseTypes="phrase", skipCols="1")
A.show(
    results,
    condensed=False,
    start=12,
    end=14,
    colorMap={2: "yellow", 3: "cyan"},
    baseTypes="phrase",
    skipCols="1",
)<jupyter_output><empty_output><jupyter_text>Here you see in results 13 and 14 a phrase of the previous clause. Lets also play with common and rare:<jupyter_code>resultsCommon = A.search(
    """
verse
  clause
    =: rare otype=phrase
""",
    sets=customSets,
)
resultsCommon = A.search(
    """
verse
  clause
    =0: rare otype=phrase
""",
    sets=customSets,
)
resultsCommon = A.search(
    """
verse
  clause
    =1: rare otype=phrase
""",
    sets=customSets,
)
resultsCommon = A.search(
    """
verse
  clause
    =2: rare otype=phrase
""",
    sets=customSets,
)<jupyter_output>1.30s 39382 results
{'fraction_non_alphanumeric': 0.1014227204138823, 'fraction_numerical': 0.01773011424876051, 'mean_word_length': 4.5294994040524434, 'pattern_counts': {'":': 2, '<': 136, '<?xml version=': 0, '>': 127, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '57072', 'license': 'MIT', 'n_tokens_mistral': 6049, 'n_tokens_neox': 5772, 'n_words': 2067, 'path': 'bhsa/searchRelations.ipynb', 'repo_name': 'sethbam9/tutorials', 'size': '18556'}