<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Data2&nbsp;&nbsp;Start up3&nbsp;&nbsp;Sign representations4&nbsp;&nbsp;Pairs per object4.1&nbsp;&nbsp;A battery of collocations4.2&nbsp;&nbsp;Simple versus refined5&nbsp;&nbsp;Results6&nbsp;&nbsp;Alternatives  CollocationWe want to get insights in the co-occurrences of signs on tablets in the[Uruk III/IV](http://cdli.ox.ac.uk/wiki/doku.php?id=proto-cuneiform)corpus (4000-3100 BC).These tablets have a poor archival context, since they come from rubbish pits, and may have been transportedfrom various different places than where they have been excavated.In order to get more information about their chronology and context, we need to study the evolution ofthe signs on the tablets. Collocation is one of the pre-requisites to do so.The tutorial ended with a first exercise in collocation, where we collated pairs of signsthat co-occur on tablets and used an unsophisticated distance measure.We repeat that exercise, and proceed to refine the collocation method step by step. DataWe have downloaded the transcriptions from the**Cuneiform Digital Library Initiative**[CDLI](https://cdli.ucla.edu),and converted them to[Text-Fabric](https://github.com/annotation/text-fabric).Read more about the details of the conversion in the[checks](http://nbviewer.jupyter.org/github/Nino-cunei/uruk/blob/master/programs/checks.ipynb) notebook.For an introduction to Text-Fabric, follow the[start](start.ipynb) tutorial. AuthorsJ. <NAME> and <NAME> (see the[README](https://github.com/annotation/Nino-cunei)of this repository).  Start upWe import the Python modules we need.<jupyter_code>%load_ext autoreload
%autoreload 2
import os
import collections
from IPython.display import Markdown, display

from tf.extra.cunei import Cunei

LOC = ("~/github", "/Nino-cunei/uruk", "collocation")
A = Cunei(*LOC)
A.api.makeAvailableIn(globals())
example = "P005381"
display(A.photo(example, width="200"))
display(A.lineart(example, width="200"))
GITHUB = "https://github.com"
REPO_REL = "annotation/Nino-cunei"
RESULT_GH = f"{GITHUB}/{REPO_REL}/blob/master/reports/collocation"
RESULT_DIR = f"{A.reportDir}/collocation"
TEMP_RESULT_DIR = f"{A.tempDir}/collocation"<jupyter_output><empty_output><jupyter_text>We create the temporary and report directories, if they do not exist already.<jupyter_code>for cdir in (TEMP_RESULT_DIR, RESULT_DIR):
    os.makedirs(cdir, exist_ok=True)<jupyter_output><empty_output><jupyter_text>We pick up where we left off in the [start](start.ipynb) tutorial: computing co-occurrencesby tablet. But we make the move to put our recipes into functions, that we will re-use and refine later on.  Sign representationsWe pre-compute the sign representations for each node.We also make an index of occurrences for each sign representation.<jupyter_code>NA = {"", "…", "X"}


def getSigns():
    signFromNode = dict()
    nodeFromSign = collections.defaultdict(list)

    for tablet in F.otype.s("tablet"):
        for s in L.d(tablet, otype="sign"):
            if F.grapheme.v(s) in NA:
                continue
            signRep = A.atfFromSign(s)
            signFromNode[s] = signRep
            nodeFromSign[signRep].append(s)
    print(
        f"computed {len(nodeFromSign)} distinct sign representations from {len(signFromNode)} nodes"
    )
    return (signFromNode, nodeFromSign)


(signFromNode, nodeFromSign) = getSigns()<jupyter_output>computed 1476 distinct sign representations from 91371 nodes<jupyter_text>Pairs per objectIn the [start](start.ipynb) tutorial we collected pairs per tablet, and we calculateda coarse distance between pairs, based on the distance of signs in the linear sequenceby which signs have been transcribed.We are going to write that process as a function, where we abstract from the level atwhich the pairs must co-occur. We also abstract from how we measure the distance. In addition to our coarse definition based on relative closeness, we alsowant a refined notion of closeness, one that does justice to thegeometry of a tablet.Instead of basing closeness on sign distance, we want to base it on quad distance.<jupyter_code>def coarseRelativeCloseness(i, j, signLength=None, quadInfo=None):
    return (signLength - abs(j - i)) / signLength


def refinedRelativeCloseness(i, j, signLength=None, quadInfo=None):
    (quadLength, outerQuadFromSign) = quadInfo
    return (quadLength - abs(outerQuadFromSign[i] - outerQuadFromSign[j])) / quadLength<jupyter_output><empty_output><jupyter_text>We write a function `getPairs(perType, measureName)` that computes co-occurent pairs on objectsof type `perType`. Here `measureName` is the name of a function that, given two sign nodes, computesa measure of closeness between those nodes.We also show the top pairs, and save all pairs to disk in a tsv file in your TEMP_RESULT_DIR.A shorter version of the results we write to your RESULT_DIR.<jupyter_code>SHOWPAIRS = 10
RESULTPAIRS = 1000


def getPairs(perType, measureName):
    pairs = collections.Counter()
    measure = globals()[measureName]

    for obj in F.otype.s(perType):
        signs = L.d(obj, otype="sign")
        signLength = signs[-1] - signs[0]

        outerQuads = A.getOuterQuads(obj)
        quadLength = len(outerQuads)
        outerQuadFromSign = {}
        for (i, outerQuad) in enumerate(outerQuads):
            if F.otype.v(outerQuad) == "sign":
                outerQuadFromSign[outerQuad] = i
            else:
                for s in L.d(outerQuad, otype="sign"):
                    outerQuadFromSign[s] = i
        quadInfo = (quadLength, outerQuadFromSign)

        thesePairs = {}
        for i in range(len(signs)):
            nodeI = signs[i]
            if nodeI not in signFromNode:
                continue
            signI = signFromNode[nodeI]
            for j in range(i + 1, len(signs)):
                nodeJ = signs[j]
                if nodeJ not in signFromNode:
                    continue
                signJ = signFromNode[nodeJ]
                if signJ == signI:
                    continue
                pair = (signI, signJ) if signI < signJ else (signJ, signI)
                closeness = measure(
                    nodeI, nodeJ, signLength=signLength, quadInfo=quadInfo
                )
                oldCloseness = thesePairs.get(pair, None)
                if oldCloseness is None or oldCloseness < closeness:
                    thesePairs[pair] = closeness
        for (pair, closeness) in thesePairs.items():
            pairs[pair] += closeness
    showPairs(pairs, perType, measureName)
    return pairs


def sortPairs(pairs):
    return sorted(
        pairs.items(),
        key=lambda x: (-x[1], x[0]),
    )


def sortPairsBare(pairs):
    return sorted(
        pairs,
        key=lambda x: (-pairs[x], x),
    )


def showPairs(pairs, perType, measureName):
    print(f"{len(pairs)} co-occurrences in {perType}s with measure {measureName}")
    sortedPairs = sortPairs(pairs)
    pairFile = f"per-{perType}-{measureName}.tsv"
    pairTempPath = f"{TEMP_RESULT_DIR}/{pairFile}"
    with open(pairTempPath, "w") as fh:
        fh.write(f"signI\tsignJ\t{measureName}\n")
        for ((signI, signJ), closeness) in sortedPairs:
            fh.write(f"{signI}\t{signJ}\t{closeness}\n")
    print(f"Written {len(pairs)} pairs to {pairFile} in _temp")

    pairPath = f"{RESULT_DIR}/{pairFile}"
    with open(pairPath, "w") as fh:
        fh.write("signI\tsignJ\t{measureName}\n")
        for ((signI, signJ), closeness) in sortedPairs[0:RESULTPAIRS]:
            fh.write(f"{signI}\t{signJ}\t{closeness}\n")
    print(f"Written {len(pairs)} pairs to {pairFile} in report")

    showPairs = sortedPairs[0:SHOWPAIRS]
    for ((signI, signJ), closeness) in showPairs:
        print(f"{signI:>10} <=~ {closeness:>7.2f} ~=> {signJ:<10}")
    if SHOWPAIRS < len(sortedPairs):
        print(f"...and {len(sortedPairs) - SHOWPAIRS} more")<jupyter_output><empty_output><jupyter_text>A battery of collocationsLet's do business with this function, and get our results back for tablets and a closenessfunction based on the size of the tablet.<jupyter_code>collocationPairs = {}

collocationObjects = """
    tablet
    face
    column
    line
""".strip().split()

closenessMethods = """
    coarseRelativeCloseness
    refinedRelativeCloseness
""".strip().split()
for obj in collocationObjects:
    for method in closenessMethods:
        collocationPairs.setdefault(obj, {})[method] = getPairs(obj, method)<jupyter_output>117270 co-occurrences in tablets with measure coarseRelativeCloseness
Written 117270 pairs to per-tablet-coarseRelativeCloseness.tsv in _temp
Written 117270 pairs to per-tablet-coarseRelativeCloseness.tsv in report
    1(N01) <=~  786.96 ~=> 2(N01)    
    1(N01) <=~  635.92 ~=> 1(N14)    
    1(N01) <=~  569.41 ~=> EN~a      
    1(N14) <=~  567.37 ~=> 2(N01)    
    1(N01) <=~  503.76 ~=> 3(N01)    
    2(N01) <=~  435.71 ~=> 3(N01)    
    1(N01) <=~  434.62 ~=> N         
    1(N14) <=~  410.67 ~=> 3(N01)    
    1(N01) <=~  400.25 ~=> AN        
    1(N01) <=~  386.27 ~=> GAL~a     
...and 117260 more
117270 co-occurrences in tablets with measure refinedRelativeCloseness
Written 117270 pairs to per-tablet-refinedRelativeCloseness.tsv in _temp
Written 117270 pairs to per-tablet-refinedRelativeCloseness.tsv in report
    1(N01) <=~  794.63 ~=> 2(N01)    
    1(N01) <=~  642.42 ~=> 1(N14)    
    1(N01) <=~  575.20 ~=> EN~a      
    1(N14) <=~  573.90 ~=> 2(N01)    
    1(N01) <=~  [...]<jupyter_text>Simple versus refinedDoes the refined measure give other results?<jupyter_code>def comparePairs(obj, method1, method2):
    pairs1 = sortPairsBare(collocationPairs[obj][method1])
    pairs2 = sortPairsBare(collocationPairs[obj][method2])
    if len(pairs1) != len(pairs2):
        print(
            f"{obj:<6}: !!! {method1:>24} => {len(pairs1):>6} =/= {len(pairs2):>6} <= {method2:<24}"
        )
    else:
        print(
            f"{obj:<6}:     {method1:>24} => {len(pairs1):>6} === {len(pairs2):>6} <= {method2:<24}"
        )
    firstDiff = -1
    for (i, (pair1, pair2)) in enumerate(zip(pairs1, pairs2)):
        if pair1 != pair2:
            firstDiff = i
            break
    if firstDiff < 0:
        if len(pairs1) == len(pairs2):
            print("\tIDENTICAL")
        else:
            methodSmall = method1 if len(pairs1) < len(pairs2) else method2
            methodBig = method2 if len(pairs1) < len(pairs2) else method1
            print(
                f"\tPREFIX: pairs by {methodSmall} are a prefix of pairs by {methodBig}"
            )
    else:
        print(f"\tFIRST DIFFERENCE at position {firstDiff}")

    topPairs1 = pairs1[0:RESULTPAIRS]
    topPairs2 = pairs2[0:RESULTPAIRS]
    setTop1 = set(topPairs1)
    setTop2 = set(topPairs2)

    if setTop1 == setTop2:
        print(f"\tEQUAL as set of the top-{RESULTPAIRS} pairs")
    else:
        common = setTop1 & setTop2
        print(f"\tSHARE {len(common)} pairs in their top-{RESULTPAIRS}")
    print("")
for obj in collocationObjects:
    comparePairs(obj, *closenessMethods)<jupyter_output>tablet:      coarseRelativeCloseness => 117270 === 117270 <= refinedRelativeCloseness
	FIRST DIFFERENCE at position 30
	SHARE 996 pairs in their top-1000

face  :      coarseRelativeCloseness => 104474 === 104474 <= refinedRelativeCloseness
	FIRST DIFFERENCE at position 25
	SHARE 996 pairs in their top-1000

column:      coarseRelativeCloseness =>  72494 ===  72494 <= refinedRelativeCloseness
	FIRST DIFFERENCE at position 26
	SHARE 989 pairs in their top-1000

line  :      coarseRelativeCloseness =>  36624 ===  36624 <= refinedRelativeCloseness
	FIRST DIFFERENCE at position 4
	SHARE 934 pairs in their top-1000<jupyter_text>**Observation**The more fine-grained your object of collocation is, the bigger the differences betweenthe refined and the coarse closeness measures.But all in all, the difference remains pretty small.  ResultsThe top {{RESULTPAIRS}} can be viewed online.<jupyter_code>resultLinks = []
for obj in collocationObjects:
    for method in closenessMethods:
        resultLinks.append(f"[{obj}-{method}]({RESULT_GH}/per-{obj}-{method}.tsv)\n\n")

RESULT_LINKS = "".join(resultLinks)
display(Markdown(RESULT_LINKS))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11188126885753893, 'fraction_numerical': 0.03131370790181848, 'mean_word_length': 3.430635838150289, 'pattern_counts': {'":': 1, '<': 65, '<?xml version=': 0, '>': 61, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '295311', 'license': 'MIT', 'n_tokens_mistral': 4265, 'n_tokens_neox': 3954, 'n_words': 1087, 'path': 'uruk/advanced/collocation.ipynb', 'repo_name': 'sethbam9/tutorials', 'size': '12274'}