<jupyter_start><jupyter_text>Old Babylonian in the browser:```text-fabric oldbabylonian```<jupyter_code>import sys, os, collections
from tf.app import use<jupyter_output><empty_output><jupyter_text>Incantation<jupyter_code>A = use('oldbabylonian', hoist=globals())
silentOff()<jupyter_output>connecting to online GitHub repo annotation/app-oldbabylonian ... connected
Using TF-app in /Users/dirk/text-fabric-data/annotation/app-oldbabylonian/code:
	rv0.2=#4bb2530bfb94dc93601f8b3df7722cb0e5df7a43 (latest release)
	connecting to online GitHub repo Nino-cunei/oldbabylonian ... connected
Using data in /Users/dirk/text-fabric-data/Nino-cunei/oldbabylonian/tf/1.0.4:
	rv1.4 (latest release)
   |     0.00s No structure info in otext, the structure part of the T-API cannot be used<jupyter_text>Counting<jupyter_code>indent(reset=True)
info('Counting nodes ...')

i = 0
for n in N(): i += 1

info('{} nodes'.format(i))<jupyter_output>0.00s Counting nodes ...
  0.06s 334667 nodes<jupyter_text>Node types<jupyter_code>F.otype.slotType
F.otype.maxSlot
F.otype.maxNode
F.otype.all
C.levels.data
for (typ, av, start, end) in C.levels.data:
  print(f'{end - start + 1:>7} {typ}s')<jupyter_output>1285 documents
   2834 faces
  27375 lines
  76505 words
  23449 clusters
 203219 signs<jupyter_text>Feature statistics  repeats<jupyter_code>F.repeat.freqList()<jupyter_output><empty_output><jupyter_text>type (cluster)<jupyter_code>F.type.freqList('cluster')<jupyter_output><empty_output><jupyter_text>type (sign)<jupyter_code>F.type.freqList('sign')<jupyter_output><empty_output><jupyter_text>flags<jupyter_code>F.flags.freqList()<jupyter_output><empty_output><jupyter_text>Word matters Top 20 frequent words<jupyter_code>for (w, amount) in F.sym.freqList('word')[0:20]:
  print(f'{amount:>5} {w}')<jupyter_output><empty_output><jupyter_text>Hapaxes<jupyter_code>hapaxes1 = sorted(x for (x, amount) in F.sym.freqList('word') if amount == 1)
len(hapaxes1)
for w in [w for (w, amount) in F.sym.freqList('word') if amount == 1][0:20]:
  print(f'"{w}"')<jupyter_output><empty_output><jupyter_text>Small occurrence baseThe occurrence base of a word are the documents in which occurs.<jupyter_code>occurrenceBase = collections.defaultdict(set)

for w in F.otype.s('word'):
  pNum = T.sectionFromNode(w)[0]
  occurrenceBase[F.sym.v(w)].add(pNum)<jupyter_output><empty_output><jupyter_text>An overview of how many words have how big occurrence bases:<jupyter_code>occurrenceSize = collections.Counter()

for (w, pNums) in occurrenceBase.items():
  occurrenceSize[len(pNums)] += 1
  
occurrenceSize = sorted(
  occurrenceSize.items(),
  key=lambda x: (-x[1], x[0]),
)

for (size, amount) in occurrenceSize[0:10]:
  print(f'tablets {size:>4} : {amount:>5} words')
print('...')
for (size, amount) in occurrenceSize[-10:]:
  print(f'tablets {size:>4} : {amount:>5} words')<jupyter_output><empty_output><jupyter_text>Let's give the predicate *private* to those words whose occurrence base is a single document.<jupyter_code>privates = {w for (w, base) in occurrenceBase.items() if len(base) == 1}
len(privates)<jupyter_output><empty_output><jupyter_text>Peculiarity of tabletsAs a final exercise with words, lets make a list of all documents, and show their* total number of words* number of private words* the percentage of private words: a measure of the peculiarity of the document<jupyter_code>docList = []

empty = set()
ordinary = set()

for d in F.otype.s('document'):
  pNum = T.documentName(d)
  words = {F.sym.v(w) for w in L.d(d, otype='word')}
  a = len(words)
  if not a:
    empty.add(pNum)
    continue
  o = len({w for w in words if w in privates})
  if not o:
    ordinary.add(pNum)
    continue
  p = 100 * o / a
  docList.append((pNum, a, o, p))

docList = sorted(docList, key=lambda e: (-e[3], -e[1], e[0]))

print(f'Found {len(empty):>4} empty documents')
print(f'Found {len(ordinary):>4} ordinary documents (i.e. without private words)')
print('{:<20}{:>5}{:>5}{:>5}\n{}'.format(
    'document', '#all', '#own', '%own',
    '-'*35,
))

for x in docList[0:20]:
  print('{:<20} {:>4} {:>4} {:>4.1f}%'.format(*x))
print('...')
for x in docList[-20:]:
  print('{:<20} {:>4} {:>4} {:>4.1f}%'.format(*x))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16016087059380174, 'fraction_numerical': 0.032410693162999764, 'mean_word_length': 5.942528735632184, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '353664', 'license': 'MIT', 'n_tokens_mistral': 1734, 'n_tokens_neox': 1630, 'n_words': 357, 'path': 'oldbabylonian/start.ipynb', 'repo_name': 'ancient-data/tftutorials', 'size': '4227'}