<jupyter_start><jupyter_text>Loading AthenaeusWithout a TF-app, we use the more[primitive way](https://annotation.github.io/text-fabric/Api/Fabric/)to load a TF data set.Text-Fabric will not go online, but look on the local computer where we tell it to look.<jupyter_code>import os
import collections
from tf.fabric import Fabric<jupyter_output><empty_output><jupyter_text>LocationWe specify where on our system the data can be found.Most TF users have a directory `github` in their home directory.That directory is organized as in GitHub itself: first by org/user, then by repo.This way, we can borrow each other's code without modification.<jupyter_code>BASE = os.path.expanduser('~/github')
ORG = 'pthu'
REPO = 'patristics'
LOCATION1 = 'tf/1.1/athenaeus/Athenaeus/'
LOCATION2 = 'Deipnosophistae'<jupyter_output><empty_output><jupyter_text>The split of the relative path in `LOCATION1` and `LOCATION2` is arbitrary.In some scenarios it is handy to split a data set in a base set and a collection modules.Now we are going to call TF. We only ask it to search through the specified directories andread the metadata section of all `.tf` files.<jupyter_code>TF = Fabric(locations=f'{BASE}/{ORG}/{REPO}/{LOCATION1}', modules=LOCATION2)<jupyter_output>This is Text-Fabric 7.8.8
Api reference : https://annotation.github.io/text-fabric/Api/Fabric/

25 features found and 0 ignored<jupyter_text>Now we can load the data and receive an API for it.We want all loadable features, so we ask which they are.<jupyter_code>def tfInitiate():
    api = TF.load('', silent=False)
    allFeatures = TF.explore(silent=True, show=True)
    loadableFeatures = allFeatures['nodes'] + allFeatures['edges']
    TF.load(loadableFeatures, add=True, silent=True)
    return api<jupyter_output><empty_output><jupyter_text>The first time will see a precomputation step (levels, order, rank, ...)A lot of data is precomputed and stored in a hidden `.tf` directory, so that it loads quicker next time.The API relies on these precomputed data for efficientcy.<jupyter_code>api = tfInitiate()<jupyter_output>0.00s loading features ...
   |     0.11s T otype                from /Users/dirk/github/pthu/patristics/tf/1.1/athenaeus/Athenaeus//Deipnosophistae
   |     1.80s T oslots               from /Users/dirk/github/pthu/patristics/tf/1.1/athenaeus/Athenaeus//Deipnosophistae
   |     0.68s T norm                 from /Users/dirk/github/pthu/patristics/tf/1.1/athenaeus/Athenaeus//Deipnosophistae
   |     0.67s T main                 from /Users/dirk/github/pthu/patristics/tf/1.1/athenaeus/Athenaeus//Deipnosophistae
   |     0.00s T book                 from /Users/dirk/github/pthu/patristics/tf/1.1/athenaeus/Athenaeus//Deipnosophistae
   |     0.66s T orig                 from /Users/dirk/github/pthu/patristics/tf/1.1/athenaeus/Athenaeus//Deipnosophistae
   |     0.65s T plain                from /Users/dirk/github/pthu/patristics/tf/1.1/athenaeus/Athenaeus//Deipnosophistae
   |     0.60s T beta_plain           from /Users/dirk/github/pthu/patristics/tf/1.1/athenaeus/Athenaeus//Deipnosophi[...]<jupyter_text>Finally, we want to make the functions of the TF API available at the top level in this notebook<jupyter_code>docs = api.makeAvailableIn(globals())<jupyter_output><empty_output><jupyter_text>The variable `docs` is a list of function names that have been inserted in the global namespace:<jupyter_code>for member in docs:
  print(member)<jupyter_output>('Computed', 'computed-data', ('C Computed', 'Call AllComputeds', 'Cs ComputedString'))
('Features', 'edge-features', ('E Edge', 'Eall AllEdges', 'Es EdgeString'))
('Fabric', 'loading', ('ensureLoaded', 'TF', 'ignored', 'loadLog'))
('Locality', 'locality', ('L Locality',))
('Misc', 'messaging', ('cache', 'error', 'indent', 'info', 'isSilent', 'reset', 'setSilent', 'silentOff', 'silentOn', 'warning'))
('Nodes', 'navigating-nodes', ('N Nodes', 'sortKey', 'sortKeyTuple', 'otypeRank', 'sortNodes'))
('Features', 'node-features', ('F Feature', 'Fall AllFeatures', 'Fs FeatureString'))
('Search', 'search', ('S Search',))
('Text', 'text', ('T Text',))<jupyter_text>Now you can study as in the tutorials for DSS and Oldbabylonian: We can list all loaded features:<jupyter_code>Fall()<jupyter_output><empty_output><jupyter_text>Counting<jupyter_code>indent(reset=True)
info('Counting nodes ...')

i = 0
for n in N(): i += 1

info('{} nodes'.format(i))<jupyter_output>0.00s Counting nodes ...
  0.05s 303854 nodes<jupyter_text>Node types<jupyter_code>F.otype.slotType
F.otype.all
C.levels.data
for (typ, av, start, end) in C.levels.data:
  print(f'{end - start + 1:>7} {typ}s')<jupyter_output>1 _books
      1 heads
     15 books
     78 his
    167 cits
    275 nums
    789 adds
   1328 chapters
   1549 pbs
   1571 ps
   3126 quotes
   5264 bibls
  11267 ls
  12781 _sentences
 265642 words<jupyter_text>Feature statistics There are no linguistic features, as far as I can see, but there is `lemma`.  Word matters Top 20 frequent words<jupyter_code>for (w, amount) in F.lemma.freqList('word')[0:20]:
  print(f'{amount:>5} {w}')<jupyter_output>24736 ὁ,ὅς
12995 καί
11725 δέ
 5668 ἒ
 3139 φημί
 2916 οὗτος
 2696 αὐτός
 2558 ἐγώ
 2457 τις,ὁ,τίς,ὅς
 2369 εἰμί
 2056 οὐ
 2007 γάρ
 1932 τίς,τῷ,ὅς,ὁ,τις
 1856 ὁ,τίς,ὅς
 1807 ὡς,ὅς
 1742 περί
 1597 τε,σύ,τεός,τις
 1389 ἐπί
 1322 λέγω1,λέγω
 1312 εἰς,εἶμι,εἰμί<jupyter_text>Hapaxes<jupyter_code>hapaxes1 = sorted(lx for (lx, amount) in F.lemma.freqList('word') if amount == 1)
len(hapaxes1)
for lx in hapaxes1[0:20]:
  print(lx)<jupyter_output>*isgreek
*p
*ʼαγκυλητους
*ʼαδεσθαι
*ʼαδυφωνον
*ʼακουσομεθα
*ʼαναξαρχον
*ʼανδρομαχον
*ʼανθος
*ʼαντιφωντος
*ʼαπο
*ʼαποδιδωσι
*ʼαρπασθηναι
*ʼαφι
*ʼβρενθιν
*ʼβυζαντιους
*ʼγενη
*ʼγλαυκου
*ʼγραφει
*ʼγραφων<jupyter_text>Small occurrence baseThe occurrence base of a word are the books in which the word occurs.<jupyter_code>occurrenceBase = collections.defaultdict(set)

indent(reset=True)
info('compiling occurrence base ...')
for s in F.otype.s('book'):
  book = F.book.v(s)
  for w in L.d(s, otype='word'):
    occurrenceBase[F.lemma.v(w)].add(book)
info('done')
info(f'{len(occurrenceBase)} entries')<jupyter_output>0.00s compiling occurrence base ...
  0.22s done
  0.22s 23448 entries<jupyter_text>An overview of how many words have how big occurrence bases:<jupyter_code>occurrenceSize = collections.Counter()

for (w, books) in occurrenceBase.items():
  occurrenceSize[len(books)] += 1
  
occurrenceSize = sorted(
  occurrenceSize.items(),
  key=lambda x: (-x[1], x[0]),
)

for (size, amount) in occurrenceSize[0:10]:
  print(f'books {size:>4} : {amount:>5} words')
print('...')
for (size, amount) in occurrenceSize[-10:]:
  print(f'books {size:>4} : {amount:>5} words')<jupyter_output>books    1 : 12899 words
books    2 :  3627 words
books    3 :  1879 words
books    4 :  1179 words
books    5 :   781 words
books    6 :   571 words
books    7 :   435 words
books    8 :   375 words
books   15 :   347 words
books   10 :   294 words
...
books    6 :   571 words
books    7 :   435 words
books    8 :   375 words
books   15 :   347 words
books   10 :   294 words
books    9 :   288 words
books   11 :   218 words
books   12 :   205 words
books   14 :   176 words
books   13 :   174 words<jupyter_text>Let's give the predicate *private* to those words whose occurrence base is a single book.<jupyter_code>privates = {w for (w, base) in occurrenceBase.items() if len(base) == 1}
len(privates)<jupyter_output><empty_output><jupyter_text>Peculiarity of booksAs a final exercise with books, lets make a list of all books, and show their* total number of words* number of private words* the percentage of private words: a measure of the peculiarity of the book<jupyter_code>bookList = []

empty = set()
ordinary = set()

for d in F.otype.s('book'):
  book = F.book.v(d)
  words = {F.lemma.v(w) for w in L.d(d, otype='word')}
  a = len(words)
  if not a:
    empty.add(book)
    continue
  o = len({w for w in words if w in privates})
  if not o:
    ordinary.add(book)
    continue
  p = 100 * o / a
  bookList.append((book, a, o, p))

bookList = sorted(bookList, key=lambda e: (-e[3], -e[1], e[0]))

print(f'Found {len(empty):>4} empty books')
print(f'Found {len(ordinary):>4} ordinary books (i.e. without private words)')
print('{:<20}{:>5}{:>5}{:>5}\n{}'.format(
    'book', '#all', '#own', '%own',
    '-'*35,
))

for x in bookList[0:20]:
  print('{:<20} {:>4} {:>4} {:>4.1f}%'.format(*x))
print('...')
for x in bookList[-20:]:
  print('{:<20} {:>4} {:>4} {:>4.1f}%'.format(*x))<jupyter_output>book                 #all #own %own
-----------------------------------
3                    4719 1099 23.3%
14                   4777 1090 22.8%
15                   3821  840 22.0%
13                   4945 1078 21.8%
11                   4589  997 21.7%
7                    4457  948 21.3%
5                    4068  831 20.4%
4                    4777  955 20.0%
2                    3787  741 19.6%
9                    4109  797 19.4%
1                    3715  692 18.6%
6                    4453  824 18.5%
10                   4333  751 17.3%
12                   4165  712 17.1%
8                    3488  544 15.6%
...
3                    4719 1099 23.3%
14                   4777 1090 22.8%
15                   3821  840 22.0%
13                   4945 1078 21.8%
11                   4589  997 21.7%
7                    4457  948 21.3%
5                    4068  831 20.4%
4                    4777  955 20.0%
2                    3787  741 19.6%
9                    4109  797 19.4%[...]
{'fraction_non_alphanumeric': 0.129800629590766, 'fraction_numerical': 0.06967471143756558, 'mean_word_length': 3.1331309627059842, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 72, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '210085', 'license': 'MIT', 'n_tokens_mistral': 4023, 'n_tokens_neox': 3526, 'n_words': 970, 'path': 'athenaeus/start.ipynb', 'repo_name': 'ancient-data/tftutorials', 'size': '9530'}