<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Start up1.1&nbsp;&nbsp;obverse1.2&nbsp;&nbsp;reverse1.3&nbsp;&nbsp;obverse1.4&nbsp;&nbsp;reverse1.5&nbsp;&nbsp;obverse1.6&nbsp;&nbsp;reverse  Numbers  Start upWe import the Python modules we need.<jupyter_code>%load_ext autoreload
%autoreload 2
import os
import collections
from IPython.display import display, Markdown

from tf.extra.cunei import Cunei

LOC = ("~/github", "Nino-cunei/uruk", "numbers")
A = Cunei(*LOC)
A.api.makeAvailableIn(globals())
def dm(markdown):
    display(Markdown(markdown))<jupyter_output><empty_output><jupyter_text>Using Text-Fabric SearchText-Fabric has a [search](https://annotation.github.io/text-fabric/tf/about/searchusage.html)facility, by which you can avoid a lot of programming. Let's get all the shinPP numerals on a reverse face.<jupyter_code>pNums = """
    P005381
    P005447
    P005448
""".strip().split()

pNumPat = "|".join(pNums)
shinPP = dict(
    N41=0.2,
    N04=1,
    N19=6,
    N46=60,
    N36=180,
    N49=1800,
)

shinPPPat = "|".join(shinPP)<jupyter_output><empty_output><jupyter_text>We query for shinPP numerals on the faces of selected tablets.The result of the query is a list of tuples `(t, f, s)` consisting ofa tablet node, a face node and a sign node, which is a shinPP numeral.<jupyter_code>query = f"""
tablet catalogId={pNumPat}
    face
        sign type=numeral grapheme={shinPPPat}
"""
results = list(S.search(query))
len(results)<jupyter_output><empty_output><jupyter_text>We have found 20 numerals.We group the results by tablet and by face.<jupyter_code>numerals = {}
for (tablet, face, sign) in results:
    numerals.setdefault(tablet, {}).setdefault(face, []).append(sign)<jupyter_output><empty_output><jupyter_text>We show the tablets, the shinPP numerals per face, and we add up the numerals per face.<jupyter_code>for (tablet, faces) in numerals.items():
    dm("---\n")
    display(A.lineart(tablet, withCaption="top", width="200"))
    for (face, signs) in faces.items():
        dm(f"### {F.type.v(face)}")
        distinctSigns = {}
        for s in signs:
            distinctSigns.setdefault(A.atfFromSign(s), []).append(s)
        display(A.lineart(distinctSigns))
        total = 0
        for (signAtf, signs) in distinctSigns.items():
            # note that all signs for the same signAtf have the same grapheme and repeat
            value = 0
            for s in signs:
                value += F.repeat.v(s) * shinPP[F.grapheme.v(s)]
            total += value
            amount = len(signs)
            shinPPval = shinPP[F.grapheme.v(signs[0])]
            repeat = F.repeat.v(signs[0])
            print(f"{amount} x {signAtf} = {amount} x {repeat} x {shinPPval} = {value}")
        dm(f"**total** = **{total}**")<jupyter_output><empty_output><jupyter_text>Frequency of Quads We count the number of quads, according to their ATF string.<jupyter_code>quadFreqs = collections.Counter()
for q in F.otype.s("quad"):
    quadFreqs[A.atfFromQuad(q)] += 1
for qAtf in quadFreqs:
    if "GISZ" in qAtf:
        print(f"{quadFreqs[qAtf]:>4} x {qAtf}")<jupyter_output>15 x |GISZxSZU2~b|
  42 x |GISZxSZU2~a|
  26 x |GISZ.TE|
   1 x |GIxGISZ@t|
   4 x |(GISZx(DIN.DIN))~a|
   3 x |NINDA2xGISZ|
   1 x |(GISZx(DIN.DIN))~c|
   7 x |DUG~bxGISZ|
   3 x |GISZ3~a&GISZ3~a|
   1 x |GISZ+SZU2~b|
   1 x |BU~axGISZ@t|
   1 x |GA2~a1xGISZ@t|
   1 x |(GISZx(DIN.DIN))~b|
   1 x |(GI&GI)xGISZ@t|<jupyter_text>Frequency of a particular quad:<jupyter_code>pQuad = "GI4~a"
print(f"The frequency of {pQuad} is {quadFreqs[pQuad]}")<jupyter_output>The frequency of GI4~a is 0<jupyter_text>Not what you expected? Let's broaden the question<jupyter_code>for qAtf in quadFreqs:
    if "GI4" in qAtf:
        print(f"{quadFreqs[qAtf]:>4} x {qAtf}")<jupyter_output>1 x |GI4~a&GI4~a|
   1 x |GI4~axA|
   1 x |GI4~b&GI4~b|<jupyter_text>Ah, `GI4~a` is a single sign, not a complex quad. Quads are by definition complex.That's why you want signs and quads in one table. Let's do it.<jupyter_code>quadSignFreqs = collections.Counter()
quadSignTypes = {"quad", "sign"}

for n in N():
    nType = F.otype.v(n)
    if nType not in quadSignTypes:
        continue
    atf = A.atfFromQuad(n) if nType == "quad" else A.atfFromSign(n)
    quadSignFreqs[atf] += 1
for qsAtf in quadSignFreqs:
    if "GISZ" in qsAtf or "GI4" in qsAtf:
        print(f"{quadSignFreqs[qsAtf]:>4} x {qsAtf}")
pQuad = "GI4~a"
print(f"The frequency of {pQuad} is {quadSignFreqs[pQuad]}")<jupyter_output>The frequency of GI4~a is 34<jupyter_text>Write frequencies You can make an output directory by hand, but we do it programmatically.<jupyter_code>reportDir = "reports"
os.makedirs(reportDir, exist_ok=True)
def writeFreqs(fileName, data, dataName):
    print(f"There are {len(data)} {dataName}s")

    for (sortName, sortKey) in (
        ("alpha", lambda x: (x[0], -x[1])),
        ("freq", lambda x: (-x[1], x[0])),
    ):
        with open(f"{reportDir}/{fileName}-{sortName}.txt", "w") as fh:
            for (item, freq) in sorted(data.items(), key=sortKey):
                if item != "":
                    fh.write(f"{freq:>5} x {item}\n")
writeFreqs("quad-signs", quadSignFreqs, "quad/sign")<jupyter_output>There are 2219 quad/signs<jupyter_text>Grabbing subcasesHow can we quickly grab cases at a certain level?There is a short answer and a long answer.Here is the short anwer: a new `A`-utility function that does it for you.<jupyter_code>subcases = A.casesByLevel(2, terminal=True)
len(subcases)<jupyter_output><empty_output><jupyter_text>A basic check: which numbers do we get?<jupyter_code>caseNumbers = collections.Counter()
for s in subcases:
    caseNumbers[F.number.v(s)] += 1
caseNumbers<jupyter_output><empty_output><jupyter_text>This is what happens under the hood.The next query picks every line in which there is a case in which there is a casesuch that the case is a direct child of the line and the subcase is a direct child of the case.This is what the `sub` edge is for. A case containing another case is very liberal. Cases not only containtheir subcases, but also their subsubcases and so on.Using `sub` brings is close to what we want.<jupyter_code>query = """
l:line
    w1:case
        w2:case
l -sub> w1
w1 -sub> w2
"""
results = list(S.search(query))
len(results)<jupyter_output><empty_output><jupyter_text>We still may have too much. Subcases may contain cases themselves, and in that case they do not correspond to a single line in thetranscription.If you want only subcases that are terminal cases, you have to filter the results by means of an extra line of code.A case is terminal if it has a feature `terminal` (with value `1`).<jupyter_code>subcases = [subcase for (line, case, subcase) in results if F.terminal.v(subcase)]
len(subcases)<jupyter_output><empty_output><jupyter_text>Let's just check whether the subcases have reasonable line numbers.<jupyter_code>caseNumbers = collections.Counter()
for subcase in subcases:
    cn = F.number.v(subcase)
    if cn:
        caseNumbers[cn] += 1
caseNumbers<jupyter_output><empty_output><jupyter_text>In order to get cases at deeper levels, we need to compose a query that is dependent onthe given depth. That is exactly what the `A.casesByLevel` function does.  Displaying subcasesWe show the ATF of nested subcases, together with their location in the corpus. We pick a deep level of cases, in order to make an inventory of the signs involved.<jupyter_code>sublevel4Nodes = A.casesByLevel(4, terminal=True)

for node in sublevel4Nodes:
    (pNum, column, lineNum) = T.sectionFromNode(node)
    srcLn = F.srcLn.v(node)
    print(f"{pNum}:{column}:{lineNum} = {srcLn}")<jupyter_output>P002694:reverse:1:3 = 3.b2B1. , X [...] 
P002694:reverse:1:3 = 3.b2B2. , X [...] 
P002694:reverse:1:3 = 3.b1B1. , [...] 
P325234:reverse:1:1 = 1.c5a1. 8(N01) , GAR U2~a 
P325234:reverse:1:1 = 1.c5a2. |SZU2.E2~b| 
P006036:obverse:1:2 = 2.b1B1. , (EN~a# PAP~a#)a 
P006036:obverse:1:2 = 2.b1B2. , (3(N57) GAN2)a 
P006036:obverse:1:2 = 2.b2B1. , (EN~a |SZU2.E2~b|)a 
P006036:obverse:1:2 = 2.b2B2. , (BU~a SZU)a 
P006036:obverse:1:2 = 2.b2B3. , (SAL BU~a)a 
P006036:obverse:1:2 = 2.b2B4. , (EN~a HI KASZ~c)a 
P218054:reverse:1:1 = 1.a1A1. [...] 5(N01)# , [...] UDU~a#? 
P218054:reverse:1:1 = 1.a1A2. [...] 7(N01)# , MASZ2 
P006160:obverse:1:2 = 2.b2B1. , ZATU836 BU~a SZE~a 
P006160:obverse:1:2 = 2.b2B2. , U8 LAGAB~a 
P005322:reverse:1:1 = 1.b2B1. 3(N20) 1(N05) 1(N42~a) , 
P005322:reverse:1:1 = 1.b2B2. 3(N18) 1(N03) 1(N40) , 
P005322:reverse:1:1 = 1.b4B1. 2(N05)? 2(N42~a)? , 1(N57)? SZE~a 
P005322:reverse:1:1 = 1.b4B2. 1(N03) 3(N40) , 
P005322:reverse:1:1 = 1.b1B1. 1(N20) 1(N42~a) 1(N25) , 
P005322:[...]<jupyter_text>We collect all sign nodes in these cases into a list.<jupyter_code>sublevel4Signs = []
for c4 in sublevel4Nodes:
    for sign in L.d(c4, otype="sign"):
        sublevel4Signs.append(sign)<jupyter_output><empty_output><jupyter_text>We count the signs occurring in these cases by their full ATF representation.<jupyter_code>signs4count = collections.Counter()
for s in sublevel4Signs:
    signs4count[A.atfFromSign(s)] += 1<jupyter_output><empty_output><jupyter_text>We print the counts, first sorted by atf representation.<jupyter_code>for (sign, amount) in sorted(signs4count.items()):
    print(f"{amount:>4} x {sign}")<jupyter_output>47 x ...
  16 x 1(N01)
   1 x 1(N02)
   2 x 1(N03)
   2 x 1(N05)
   1 x 1(N14)
   1 x 1(N18)
   1 x 1(N20)
   1 x 1(N24~a)
   1 x 1(N25)
   2 x 1(N40)
   2 x 1(N42~a)
  15 x 1(N57)
   5 x 2(N01)
   1 x 2(N05)
   3 x 2(N14)
   1 x 2(N34)
   1 x 2(N40)
   2 x 2(N42~a)
   1 x 2(N57)
   2 x 3(N01)
   1 x 3(N18)
   2 x 3(N20)
   1 x 3(N40)
   3 x 3(N57)
   4 x 4(N01)
   1 x 4(N18)
   1 x 5(N01)
   1 x 5(N03)
   2 x 6(N01)
   2 x 7(N01)
   1 x 7(N57)
   1 x 8(N01)
   1 x AB~a
   1 x ADAB
   2 x ALAN~b
   2 x AMA~a
   6 x AN
   1 x ANSZE~e
   2 x BA
   2 x BAD
   5 x BU~a
   2 x DI
   2 x DILMUN
   1 x DIM~a
   1 x DU
   1 x DU6~a
   2 x DUB~a
   1 x DUG~c
   1 x DUR2
   4 x E2~a
   2 x E2~b
  11 x EN~a
   1 x ERIN
   1 x GA2~a1
   1 x GAL~a
   1 x GAN2
   1 x GAN~c
   2 x GAR
   1 x GESZTU~a
   2 x GI
   1 x GI4~a
   2 x GI6
   1 x GU
   3 x HI
   1 x HI@g~a
   1 x IG~b
   1 x KASZ~c
   1 x KI
   1 x KISZIK~a
   1 x KU3~a
   2 x KUR~a
   1 x LAGAB~a
   1 x LAL3~a
   1 x MA
   3 x MASZ2
   [...]<jupyter_text>and now the same list, sorted by frequency.<jupyter_code>for (sign, amount) in sorted(
    signs4count.items(),
    key=lambda x: (-x[1], x[0]),
):
    print(f"{amount:>4} x {sign}")<jupyter_output>47 x ...
  16 x 1(N01)
  15 x 1(N57)
  11 x EN~a
  11 x X
   6 x AN
   5 x 2(N01)
   5 x BU~a
   5 x SAL
   5 x SZU
   5 x U4
   4 x 4(N01)
   4 x E2~a
   4 x NUN~a
   4 x SZE3
   3 x 2(N14)
   3 x 3(N57)
   3 x HI
   3 x MASZ2
   3 x PAP~a
   3 x SUHUR
   3 x SZA
   3 x TUR
   3 x U8
   2 x 1(N03)
   2 x 1(N05)
   2 x 1(N40)
   2 x 1(N42~a)
   2 x 2(N42~a)
   2 x 3(N01)
   2 x 3(N20)
   2 x 6(N01)
   2 x 7(N01)
   2 x ALAN~b
   2 x AMA~a
   2 x BA
   2 x BAD
   2 x DI
   2 x DILMUN
   2 x DUB~a
   2 x E2~b
   2 x GAR
   2 x GI
   2 x GI6
   2 x KUR~a
   2 x MUSZ3~a
   2 x NAB
   2 x NAGA~a
   2 x NIN
   2 x SZE~a
   2 x SZU2
   2 x TAK4~a
   2 x UDUNITA~a
   1 x 1(N02)
   1 x 1(N14)
   1 x 1(N18)
   1 x 1(N20)
   1 x 1(N24~a)
   1 x 1(N25)
   1 x 2(N05)
   1 x 2(N34)
   1 x 2(N40)
   1 x 2(N57)
   1 x 3(N18)
   1 x 3(N40)
   1 x 4(N18)
   1 x 5(N01)
   1 x 5(N03)
   1 x 7(N57)
   1 x 8(N01)
   1 x AB~a
   1 x ADAB
   1 x ANSZE~e
   1 x DIM~a
   1 x DU
   1 x DU6~a
   1 x DUG~c
   1 [...]<jupyter_text>TILL SO FARBelow are ways to classify tablets as to what number systems are present on them.This I did earlier. Specification of the Shin systems: just the bare minimum of info.<jupyter_code>numberSystems = dict(
    shinP=(40, 3, 18, 24, 45),
    shinPP=(4, 19, 36, 41, 46, 49),
    shinS=(25, 27, 28, 42, 5, 20, 47, 37),
)<jupyter_output><empty_output><jupyter_text>We turn the numbers into numeral graphemes:<jupyter_code>systems = {}

for (shin, numbers) in numberSystems.items():
    systems[shin] = {f"N{n:>02}" for n in numbers}<jupyter_output><empty_output><jupyter_text>Reality check<jupyter_code>systems<jupyter_output><empty_output><jupyter_text>We also want the opposite: given a numeral, which system is it?<jupyter_code>numeralMap = {}

for (shin, numerals) in systems.items():
    for n in numerals:
        if n in numeralMap:
            dm(f"**warning:** Numeral {n} in {shin} was already in {numeralMap[n]}")
        numeralMap[n] = shin

numeralMap<jupyter_output><empty_output><jupyter_text>Exercise:For each tablet, add three properties: hasShinP, hasShinPP, hasShinS.They will be True if and only if the tablet has a numeral in that category.Even better, instead of True or False, we let them record how many numerals in that set they have.<jupyter_code>tabletNumerics = collections.defaultdict(collections.Counter)

for tablet in F.otype.s("tablet"):
    pNum = F.catalogId.v(tablet)
    for sign in L.d(tablet, otype="sign"):
        if F.type.v(sign) == "numeral":
            numeral = F.grapheme.v(sign)
            system = numeralMap.get(numeral, None)
            if system is not None:
                tabletNumerics[pNum][system] += 1<jupyter_output><empty_output><jupyter_text>Now we write a csv file to the report directory, so that you can work with the data in Excel.We show the first few lines in the notebook<jupyter_code>filePath = f"{A.reportDir}/tabletNumerics.tsv"
lines = []
systemNames = sorted(systems)
fieldNames = "\t".join(systemNames)
for pNum in sorted(tabletNumerics):
    data = tabletNumerics[pNum]
    values = "\t".join(str(data[s]) for s in systemNames)
    lines.append(f"{pNum}\t{values}\n")
with open(filePath, "w") as fh:
    fh.write(f"tablet\t{fieldNames}\n")
    fh.write("".join(lines))

print("".join(lines[0:10]))<jupyter_output>P000148	0	0	1
P000245	2	0	0
P000266	0	1	0
P000308	2	0	0
P000434	2	0	0
P000511	1	0	0
P000517	1	0	0
P000550	2	0	0
P000734	2	0	0
P000735	2	0	1
{'fraction_non_alphanumeric': 0.12884312884312885, 'fraction_numerical': 0.06792506792506793, 'mean_word_length': 3.3832654340332184, 'pattern_counts': {'":': 2, '<': 96, '<?xml version=': 0, '>': 105, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '288142', 'license': 'MIT', 'n_tokens_mistral': 6382, 'n_tokens_neox': 5881, 'n_words': 1670, 'path': 'uruk/bits-and-pieces/numbers.ipynb', 'repo_name': 'sethbam9/tutorials', 'size': '13986'}