<jupyter_start><jupyter_text>Getting startedIt is assumed that you have read[start](start.ipynb)and followed the installation instructions there.  CorpusThis is* `oldbabylonian` Old Babylonian Letters  First acquaintanceWe just want to grasp what the corpus is about and how we can find our way in the data.Open a terminal or command prompt and say one of the following```text-fabric oldbabylonian```Wait and see a lot happening before your browser starts up and shows you an interface on the corpus: Text-Fabric needs an app to deal with the corpus-specific things.It downloads/finds/caches the latest version of the **app**:```Using TF-app in /Users/dirk/text-fabric-data/annotation/app-oldbabylonian/code:	rv0.2=4bb2530bfb94dc93601f8b3df7722cb0e5df7a43 (latest release)```It downloads/finds/caches the latest version of the **data**:```Using data in /Users/dirk/text-fabric-data/Nino-cunei/oldbabylonian/tf/1.0.4:	rv1.4=43c36d148794e3feeb3dd39e105ce6a4df79c467 (latest release)```The data is preprocessed in order to speed up typical Text-Fabric operations.The result is cached on your computer.Preprocessing costs time. Next time you use this corpus on this machine, the startup time is much quicker.```TF setup done.```Then the app goes on to act as a local webserver serving the corpus that has just been downloadedand it will open your browser for you and load the corpus page``` * Running on http://localhost:8106/ (Press CTRL+C to quit)Opening oldbabylonian in browserListening at port 18986```   Help!Indeed, that is what you need. Click the vertical `Help` tab.From there, click around a little bit. Don't read closely, just note the kinds of information that is presented to you.Later on, it will make more sense!  BrowsingFirst we browse our data. Click the browse button.and then, in the table of *documents* (tablets), click on `obverse` Now you're looking at one side of tablet: the marks in an ASCII transcription. Now click the *Options* tab and select the `layout-orig-unicode` format to see the same tablet in cuneiform signs. You can click a triangle to see how a line is broken down:  SearchingSee that line, starting with the word `um-ma`, and whose last word ends in the sign `ma`?That is a pattern. Let's search for it.Enter this query in the search pad and press the search icon above it.```line  =: word    =: sign reading=um    <: sign reading=ma    :=  < sign reading=ma  :=``` In English:search all `line`s that contain a `word` and a `sign`  where:* `=:` the `word` starts where the `line` starts* the `word` contains a `sign` and a `sign` where:  * `=:` the first `sign` starts where the `word` starts  * `<:` the second sign follows the first sign immediately  * `:=` the second sign ends where the word ends*  `<` the `sign` comes after the word* `:=` the `sign` ends where the line ends You can expand results by clicking the triangle.You can see the result in context by  clicking the browse icon.You can go back to the result list by clicking the results icon.  ComputingWe see that this line comes at the start  of a tablet.In fact, this pattern corresponds to a heading of a letter.Question: of all 1274 results, how many are the first line, the second line, the third line, etc?*This is a typical question where you want to leave the search mode and enter computing mode*.Let's do that! If you have followed the installation instructions, you are set.Go to the browser window that opened when you gave the command `jupyter notebook` in your terminal.Then continue reading, and, ... executing.You can execute a cell by putting your cursor inside it and pressing `Shift Enter`. First we load the Text-Fabric module, as follows:<jupyter_code>import collections
from tf.app import use<jupyter_output><empty_output><jupyter_text>Now we load the TF-app for the corpus `oldbabylonian` and that app loads the corpus data.We give a name to the result of all that loading: `A`.<jupyter_code>A = use("oldbabylonian:clone", checkout="clone", hoist=globals())
# A = use('oldbabylonian', hoist=globals())<jupyter_output><empty_output><jupyter_text>Some bits are familiar from above, when you ran the `text-fabric` command in the terminal.Other bits are links to the documentation, they point to the same places as the links on the Text-Fabric browser.You see a list of all the data features that have been loaded.And a list of references to the API documentation, which tells you how you can use this data in your program statements.  Searching (revisited)We do the same search again, but now inside our program.That means that we can capture the results in a list for further processing.<jupyter_code>results = A.search(
    """
line
  =: word
    =: sign reading=um
    <: sign reading=ma
    :=
  < sign reading=ma
  :=
"""
)<jupyter_output>0.70s 1274 results<jupyter_text>In less than a second, we have all the results!Let's look at the first one:<jupyter_code>results[0]<jupyter_output><empty_output><jupyter_text>Each result is a list of numbers: for a1. line1. word1. sign1. sign1. sign Here is the  second one:<jupyter_code>results[1]<jupyter_output><empty_output><jupyter_text>And here the last one:<jupyter_code>results[-1]<jupyter_output><empty_output><jupyter_text>Now we want to find out something for each result line: which line number does it have among the lines on the same tablet face?Click the link `Feature docs` above, and read a bit under **Node type line**.There you see that the feature `ln` is of particular interest to us.First we get the line number of result 1000:<jupyter_code>node = results[999][0]
print(node)
lineNumber = F.ln.v(node)
print(lineNumber)<jupyter_output>252681
3<jupyter_text>Now we collect the set of all line numbers that our result lines have:<jupyter_code>{F.ln.v(result[0]) for result in results}<jupyter_output><empty_output><jupyter_text>What we really want to know is how the result lines are distributed over the line numbers.<jupyter_code>distribution = collections.Counter()

for result in results:
    lineNumber = F.ln.v(result[0])
    distribution[lineNumber] += 1

print(distribution)<jupyter_output>Counter({3: 834, 2: 110, 4: 102, 6: 42, 7: 37, 5: 33, 8: 31, 9: 16, 10: 13, 1: 11, 12: 9, 11: 8, 13: 8, 16: 5, 15: 4, 14: 4, 20: 2, 17: 2, 31: 1, 19: 1, 21: 1})<jupyter_text>An overwhelming majority has it on line 3Let's make the output a bit more friendly:<jupyter_code>for (lineNumber, amount) in sorted(distribution.items()):
    print(f"line {lineNumber:>2} is home to {amount:>3} results")<jupyter_output>line  1 is home to  11 results
line  2 is home to 110 results
line  3 is home to 834 results
line  4 is home to 102 results
line  5 is home to  33 results
line  6 is home to  42 results
line  7 is home to  37 results
line  8 is home to  31 results
line  9 is home to  16 results
line 10 is home to  13 results
line 11 is home to   8 results
line 12 is home to   9 results
line 13 is home to   8 results
line 14 is home to   4 results
line 15 is home to   4 results
line 16 is home to   5 results
line 17 is home to   2 results
line 19 is home to   1 results
line 20 is home to   2 results
line 21 is home to   1 results
line 31 is home to   1 results<jupyter_text>We can now inspect more closely what is going on, for example where results appear late in the tablet, after line 16:<jupyter_code>results16 = A.search(
    """
line ln>16
  =: word
    =: sign reading=um
    <: sign reading=ma
    :=
  < sign reading=ma
  :=
"""
)<jupyter_output>0.67s 7 results<jupyter_text>And we can show them here too:<jupyter_code>A.table(results16)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07958751983077737, 'fraction_numerical': 0.03291909042834479, 'mean_word_length': 4.637108792846497, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '339352', 'license': 'MIT', 'n_tokens_mistral': 2468, 'n_tokens_neox': 2236, 'n_words': 1069, 'path': 'lorentz2020/oldbabylonian.ipynb', 'repo_name': 'sethbam9/tutorials', 'size': '7564'}