<jupyter_start><jupyter_text>---To get started: consult [start](start.ipynb)--- Sharing data features Explore additional dataOnce you analyse a corpus, it is likely that you produce data that others can reuse.Maybe you have defined a set of proper name occurrences, or special numerals, or you have computed part-of-speech assignments.It is possible to turn these insights into *new features*, i.e. new `.tf` files with values assigned to specific nodes. Make your own dataNew data is a product of your own methods and computations in the first place.But how do you turn that data into new TF features?It turns out that the last step is not that difficult.If you can shape your data as a mapping (dictionary) from node numbers (integers) to values(strings or integers), then TF can turn that data into a feature file for you with one command. Share your new dataYou can then easily share your new features on GitHub, so that your colleagues everywherecan try it out for themselves.You can add such data on the fly, by passing a `mod={org}/{repo}/{path}` parameter,or a bunch of them separated by commas.If the data is there, it will be auto-downloaded and stored on your machine.Let's do it.<jupyter_code>%load_ext autoreload
%autoreload 2
import os
from tf.app import use
# A = use('dss', hoist=globals())
A = use("dss:clone", checkout="clone", hoist=globals())<jupyter_output><empty_output><jupyter_text>Making dataWe illustrate the data creation part by creating a new feature, `cert`.The idea is that we mark every consonant sign voor certainty.A certain consonant gets `cert=100`.If the consonant has the uncertain feature `unc`, then 10 times its value is subtracted from 100.If the consonant has the feature `rec`, it loses 45 points.Ancient removal `rem2` leads to minus 20, modern removal `rem` to minus 40.Ancient correction `cor2` leads to minus 12, modern correction `cor` to minus 18.Alternate marking `alt` leads to minus 25.The minimum is `1`.We extend the `cert` measure to words, fragments and scrolls by averaging over signs thathave received a `cert` measure.<jupyter_code>def measure(s):
    c = 100
    d = F.unc.v(s)
    if d:
        c -= 10 * d
    d = F.rec.v(s)
    if d:
        c -= 45
    d = F.rem.v(s)
    if d == 1:
        c -= 40
    elif d == 2:
        c -= 20
    d = F.cor.v(s)
    if d == 2 or d == 3:
        c -= 12
    elif d == 1:
        c -= 18
    d = F.alt.v(s)
    if d:
        c -= 25
    if c < 1:
        c = 1
    return c
CONS = "cons"
cert = {}

A.indent(reset=True)

for sc in F.otype.s("scroll"):
    fN = 0
    fSum = 0
    for f in L.d(sc, otype="fragment"):
        lN = 0
        lSum = 0
        for ln in L.d(f, otype="line"):
            wN = 0
            wSum = 0
            for w in L.d(ln, otype="word"):
                sN = 0
                sSum = 0
                for s in L.d(w, otype="sign"):
                    if F.type.v(s) != CONS:
                        continue
                    sCert = measure(s)
                    cert[s] = sCert
                    sN += 1
                    sSum += sCert
                if sN:
                    wCert = int(round(sSum / sN))
                    cert[w] = wCert
                    wN += 1
                    wSum += wCert
            if wN:
                lCert = int(round(wSum / wN))
                cert[ln] = lCert
                lN += 1
                lSum += lCert
        if lN:
            fCert = int(round(lSum / lN))
            cert[f] = fCert
            fN += 1
            fSum += fCert
    if fN:
        scCert = int(round(fSum / fN))
        cert[sc] = scCert

A.info(f"{len(cert)} certainties determined")<jupyter_output>5.49s 1625373 certainties determined<jupyter_text>Saving dataThe [documentation](https://annotation.github.io/text-fabric/tf/core/fabric.htmltf.core.fabric.FabricCore.save) explains how to save this data into a text-fabricdata file.We choose a location where to save it, the `exercises` folder in the `dss` repository in the `dss` organization.In order to do this, we restart the TF api, but now with the desired output location in the `locations` parameter.<jupyter_code>GITHUB = os.path.expanduser("~/github")
ORG = "etcbc"
REPO = "dss"
PATH = "exercises"
VERSION = A.version<jupyter_output><empty_output><jupyter_text>Note the version: we have built the version against a specific version of the data:<jupyter_code>A.version<jupyter_output><empty_output><jupyter_text>Later on, we pass this version on, so that users of our data will get the shared data in exactly the same version as their core data. We have to specify a bit of metadata for this feature:<jupyter_code>metaData = {
    "cert": dict(
        valueType="int",
        description="measure of certainty of material, between 1 and 100 (most certain)",
        creator="<NAME>",
    ),
}<jupyter_output><empty_output><jupyter_text>Now we can give the save command:<jupyter_code>TF.save(
    nodeFeatures=dict(cert=cert),
    metaData=metaData,
    location=f"{GITHUB}/{ORG}/{REPO}/{PATH}/tf",
    module=VERSION,
)<jupyter_output>0.00s Exporting 1 node and 0 edge and 0 config features to ~/github/etcbc/dss/exercises/tf/0.6:
   |     2.14s T cert                 to ~/github/etcbc/dss/exercises/tf/0.6
  2.14s Exported 1 node features and 0 edge features and 0 config features to ~/github/etcbc/dss/exercises/tf/0.6<jupyter_text>Sharing dataHow to share your own data is explained in the[documentation](https://annotation.github.io/text-fabric/tf/about/datasharing.html).Here we show it step by step for the `cert` feature.If you commit your changes to the exercises repo, and have done a `git push origin master`,you already have shared your data!If you want to make a stable release, so that you can keep developing, while your users fall backon the stable data, you can make a new release.Go to the GitHub website for that, go to your repo, and click *Releases* and follow the nudges.If you want to make it even smoother for your users, you can zip the data and attach it as a binary to the release just created.We need to zip the data in exactly the right directory structure. Text-Fabric can do that for us:<jupyter_code>%%sh

text-fabric-zip etcbc/dss/exercises/tf<jupyter_output>True
Create release data for etcbc/dss/exercises/tf
Found 4 versions
zip files end up in ~/Downloads/etcbc-release/dss
zipping etcbc/dss                  0.3 with   1 features ==> exercises-tf-0.3.zip
zipping etcbc/dss                  0.4 with   1 features ==> exercises-tf-0.4.zip
zipping etcbc/dss                  0.5 with   1 features ==> exercises-tf-0.5.zip
zipping etcbc/dss                  0.6 with   1 features ==> exercises-tf-0.6.zip<jupyter_text>All versions have been zipped, but it works OK if you only attach the newest version to the newest release.If a user asks for an older version in this release, the system can still find it.  Use the dataWe can use the data by calling it up when we say `use('dss', ...)`.Here is how:(use the line without `clone` if the data is really published,use the line with `clone` if you want to test your local copy of the feature).<jupyter_code># A = use('dss', hoist=globals(), mod='etcbc/dss/exercises/tf')
A = use(
    "dss:clone", checkout="clone", hoist=globals(), mod="etcbc/dss/exercises/tf:clone"
)<jupyter_output><empty_output><jupyter_text>Above you see a new section in the feature list: **etcbc/dss/exercises/tf** with our foreign feature in it: `cert`.Now, suppose did not know much about this feature, then we would like to do a few basic checks:<jupyter_code>F.cert.freqList()<jupyter_output><empty_output><jupyter_text>Which nodes have the lowest uncertainty?<jupyter_code>{F.otype.v(n) for n in N.walk() if F.cert.v(n) and F.cert.v(n) < 10}<jupyter_output><empty_output><jupyter_text>Only signs are this uncertain.Let's look for pretty uncertain fragments:<jupyter_code>results = A.search(
    """
fragment cert<50
"""
)
results = A.search(
    """
fragment cert<60
"""
)
A.table(results, start=1, end=20)<jupyter_output><empty_output><jupyter_text>Same for scrolls:<jupyter_code>results = A.search(
    """
scroll cert<50
"""
)
results = A.search(
    """
scroll cert<60
"""
)
A.show(results)<jupyter_output><empty_output><jupyter_text>Lines with certainty of 50:<jupyter_code>results = A.search(
    """
line cert<57
"""
)
A.show(results, start=100, end=102)<jupyter_output><empty_output><jupyter_text>With highlights and drilled down to sign level:<jupyter_code>highlights = {}

for s in F.otype.s("sign"):
    if not F.cert.v(s):
        continue
    color = "lightsalmon" if F.cert.v(s) < 56 else "mediumaquamarine"
    highlights[s] = color
A.show(
    results,
    start=100,
    end=102,
    withNodes=True,
    condensed=True,
    highlights=highlights,
    baseTypes="sign",
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10156604630049931, 'fraction_numerical': 0.016681797548797096, 'mean_word_length': 3.050091911764706, 'pattern_counts': {'":': 1, '<': 63, '<?xml version=': 0, '>': 59, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '488726', 'license': 'MIT', 'n_tokens_mistral': 2946, 'n_tokens_neox': 2772, 'n_words': 1018, 'path': 'dss/share.ipynb', 'repo_name': 'sethbam9/tutorials', 'size': '8817'}