<jupyter_start><jupyter_text>Word sets by accentsWe make some classes of words, defined by the accents they contain, and save them as sets, to be used in queries.<jupyter_code>import re
from tf.app import use
from tf.lib import writeSets
A = use("bhsa:clone", hoist=globals())<jupyter_output><empty_output><jupyter_text>We define the accents and create a regular expression out of them.<jupyter_code>A_ACCENTS = set("04 24 33 63 70 71 72 73 74 93 94".split())
A_PAT = "|".join(A_ACCENTS)
A_RE = re.compile(f"(?:{A_PAT})")
A_RE<jupyter_output><empty_output><jupyter_text>We make two sets of words: words that contain one or more accents in `A_ACCENTS` and words that don't.The first set we call `word_a` and the other set `word_non_a`.We go through all words of the whole corpus.<jupyter_code>wordA = set()
wordNonA = set()

A.indent(reset=True)
A.info("Classifying words")

for w in F.otype.s("word"):
    translit = F.g_word.v(w)
    if A_RE.search(translit):
        wordA.add(w)
    else:
        wordNonA.add(w)

A.info(f"word_a has {len(wordA):>6} members")
A.info(f"word_non_a has {len(wordNonA):>6} members")<jupyter_output>0.00s Classifying words
  0.37s word_a has 140225 members
  0.37s word_non_a has 286359 members<jupyter_text>Collect the sets in a dictionary that assigns names to them:<jupyter_code>accents = dict(
    word_a=wordA,
    word_non_a=wordNonA,
)<jupyter_output><empty_output><jupyter_text>Test the set in a query:<jupyter_code>query = """
book book=Genesis
  word_a
    g_cons~^(?![KL]$)
    trailer~[^&]
"""
results = A.search(query, sets=accents)
A.table(results, end=5)
A.table(results, end=5, fmt="text-trans-full")
query = """
book book=Genesis
  word_non_a
    g_cons~^(?![KL]$)
    trailer~[^&]
"""
results = A.search(query, sets=accents)
A.table(results, end=5)
A.table(results, end=5, fmt="text-trans-full")<jupyter_output>0.85s 8034 results<jupyter_text>Now save the sets as a TF file in your Downloads folder (if you want it in an other place,tweak the variable `SET_DIR` below.We use the TF helper function[`writeSets`](https://annotation.github.io/text-fabric/tf/lib.htmltf.lib.writeSets)to do the work.<jupyter_code>SET_DIR = "~/Downloads"

writeSets(accents, f"{SET_DIR}/accents")<jupyter_output><empty_output><jupyter_text>Check:<jupyter_code>!ls -l ~/Downloads/accents<jupyter_output>-rw-r--r--@ 1 dirk  staff  730590 Jun 10 19:08 /Users/dirk/Downloads/accents
{'fraction_non_alphanumeric': 0.1342477140482128, 'fraction_numerical': 0.028678304239401497, 'mean_word_length': 5.418666666666667, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '490599', 'license': 'MIT', 'n_tokens_mistral': 983, 'n_tokens_neox': 920, 'n_words': 229, 'path': 'tutorial/cookbook/accents.ipynb', 'repo_name': 'ETCBC/bhs', 'size': '2406'}