<jupyter_start><jupyter_text>---To get started: consult [start](start.ipynb)--- RingsRings are literary structures of text composition, typically at the sub-sura level.They are marked by certain formal characteristics and a semantic structure where elements at the startare mirrored semantically at the end.The resulting effect is that the crux of the narrative/exposition appears at its center.This is not an easy concept to deal with computationally, because our dataset has only limited semantic information,and the observed patterns are the result of sensitivity of the human mind to a literary and religious text.So, do not expect to a recipe to identify rokings, but just a scratching of their surface.Moreover, this notebook is written from the view-point of a non-expert in Islam and Arabic.It serves as a testimony of how much perception such a person can muster.If you, as an expert, read on, it will become clear how much room for improvement there is.The challenge for you is then to strengthen your computational skills, so that youcan actually make those improvements.The notebook finishes off by showing how to save, export and share the data you create.<jupyter_code>%load_ext autoreload
%autoreload 2
import os
import collections
from IPython.display import Markdown, display

from tf.app import use

A = use("quran:clone", checkout="clone", hoist=globals())
# A = use('quran', hoist=globals())<jupyter_output><empty_output><jupyter_text>ExplorationWe follow an article by<NAME>, [*Surat al-Baqara: A Structural Analysis*](https://www.academia.edu/8642515/Surat_al-Baqarah_A_Structural_Analysis)  Trival detail: names of surasThe layman does not know what the Al-Baqara sura is. But there is data!Let's print the names of the suras.<jupyter_code>for s in F.otype.s("sura"):
    print(f"{F.number.v(s)}: {F.nameTrans.v(s)} ({F.name.v(s)})")<jupyter_output>1: Al-Faatiha (الفاتحة)
2: Al-Baqara (البقرة)
3: Aal-i-Imraan (آل عمران)
4: An-Nisaa (النساء)
5: Al-Maaida (المائدة)
6: Al-An'aam (الأنعام)
7: Al-A'raaf (الأعراف)
8: Al-Anfaal (الأنفال)
9: At-Tawba (التوبة)
10: Yunus (يونس)
11: Hud (هود)
12: Yusuf (يوسف)
13: Ar-Ra'd (الرعد)
14: Ibrahim (ابراهيم)
15: Al-Hijr (الحجر)
16: An-Nahl (النحل)
17: Al-Israa (الإسراء)
18: Al-Kahf (الكهف)
19: Maryam (مريم)
20: Taa-Haa (طه)
21: Al-Anbiyaa (الأنبياء)
22: Al-Hajj (الحج)
23: Al-Muminoon (المؤمنون)
24: An-Noor (النور)
25: Al-Furqaan (الفرقان)
26: Ash-Shu'araa (الشعراء)
27: An-Naml (النمل)
28: Al-Qasas (القصص)
29: Al-Ankaboot (العنكبوت)
30: Ar-Room (الروم)
31: Luqman (لقمان)
32: As-Sajda (السجدة)
33: Al-Ahzaab (الأحزاب)
34: Saba (سبإ)
35: Faatir (فاطر)
36: Yaseen (يس)
37: As-Saaffaat (الصافات)
38: Saad (ص)
39: Az-Zumar (الزمر)
40: Al-Ghaafir (غافر)
41: Fussilat (فصلت)
42: Ash-Shura (الشورى)
43: Az-Zukhruf (الزخرف)
44: Ad-Dukhaan (الدخان)
45: Al-Jaathiya (الجاثية)
46: Al-Ahqaf (الأحقاف)
47: Muhammad (محم[...]<jupyter_text>The article is about sura 2.<jupyter_code>sura2 = F.otype.s("sura")[1]<jupyter_output><empty_output><jupyter_text>SectionsThe article speaks about sections in sura2.Those sections apparently form a level between the sura and the aya.> The sura, 286 verses, consists of nine sections.```A    1 -  20B   21 -  39C   40 - 103D  104 - 141E  142 - 152D' 153 - 177C' 178 - 253B' 254 - 284A' 285 - 286```Can we identify those sections in the data?<jupyter_code>sections = [
    (1, 20),
    (21, 39),
    (40, 103),
    (104, 141),
    (142, 152),
    (153, 177),
    (178, 253),
    (254, 284),
    (285, 286),
]

nSections = len(sections)
sectionStarts = {b for (b, e) in sections}<jupyter_output><empty_output><jupyter_text>Apart from suras and ayas, there are other sectional objects in our data, see thefeature documentation (one of the links after the incantation):* juz* hizb* manzil* ruku* page* sajdaLet's see how our sura is divided into these units.Per unit type we list the how many of those units there are in this sura.<jupyter_code>UTYPES = """
juz
hizb
manzil
ruku
page
sajda
""".strip().split()
for uType in UTYPES:
    units = L.d(sura2, otype=uType)
    print(f"{uType.upper():<6}: {len(units):>2} x")<jupyter_output>JUZ   :  1 x
HIZB  : 18 x
MANZIL:  0 x
RUKU  : 40 x
PAGE  : 48 x
SAJDA :  0 x<jupyter_text>Now we want to now the exact aya intervals of the hizb, ruku and page units.<jupyter_code>for uType in UTYPES:
    units = L.d(sura2, otype=uType)
    if not len(units):
        continue
    print(f"{uType.upper():<6}: {len(units):>2} x")
    for unit in units:
        unitNum = F.number.v(unit)
        ayas = L.d(unit, otype="aya")
        firstAya = F.number.v(ayas[0])
        lastAya = F.number.v(ayas[-1])
        print(f"\t{unitNum:>2}: aya {firstAya:>3}-{lastAya:>3}")<jupyter_output>JUZ   :  1 x
	 2: aya 142-252
HIZB  : 18 x
	 2: aya  26- 43
	 3: aya  44- 59
	 4: aya  60- 74
	 5: aya  75- 91
	 6: aya  92-105
	 7: aya 106-123
	 8: aya 124-141
	 9: aya 142-157
	10: aya 158-176
	11: aya 177-188
	12: aya 189-202
	13: aya 203-218
	14: aya 219-232
	15: aya 233-242
	16: aya 243-252
	17: aya 253-262
	18: aya 263-271
	19: aya 272-282
RUKU  : 40 x
	 2: aya   1-  7
	 3: aya   8- 20
	 4: aya  21- 29
	 5: aya  30- 39
	 6: aya  40- 46
	 7: aya  47- 59
	 8: aya  60- 61
	 9: aya  62- 71
	10: aya  72- 82
	11: aya  83- 86
	12: aya  87- 96
	13: aya  97-103
	14: aya 104-112
	15: aya 113-121
	16: aya 122-129
	17: aya 130-141
	18: aya 142-147
	19: aya 148-152
	20: aya 153-163
	21: aya 164-167
	22: aya 168-176
	23: aya 177-182
	24: aya 183-188
	25: aya 189-196
	26: aya 197-210
	27: aya 211-216
	28: aya 217-221
	29: aya 222-228
	30: aya 229-231
	31: aya 232-235
	32: aya 236-242
	33: aya 243-248
	34: aya 249-253
	35: aya 254-257
	36: aya 258-260
	37: aya 261-266
	38: aya 267-273
	39: aya [...]<jupyter_text>It seems that the section boundaries all coincide with ruku boundaries.Let's check the degree in which the section boundaries respect the unit boundaries, for each unit type.We collect the start ayas for each unit.<jupyter_code>startAyas = collections.defaultdict(set)

for uType in UTYPES:
    units = L.d(sura2, otype=uType)
    if not len(units):
        continue
    for unit in units:
        unitNum = F.number.v(unit)
        ayas = L.d(unit, otype="aya")
        firstAya = F.number.v(ayas[0])
        startAyas[uType].add(firstAya)<jupyter_output><empty_output><jupyter_text>Now we count how many section starts are members of the startAyas for each unit type.<jupyter_code>agreement = {}

for uType in UTYPES:
    units = L.d(sura2, otype=uType)
    if not len(units):
        continue
    agreement[uType] = len(startAyas[uType] & sectionStarts)

for (uType, agreement) in agreement.items():
    print(
        f"{agreement} out of {nSections} section starts coincide with a {uType} start"
    )<jupyter_output>1 out of 9 section starts coincide with a juz start
1 out of 9 section starts coincide with a hizb start
7 out of 9 section starts coincide with a ruku start
2 out of 9 section starts coincide with a page start<jupyter_text>The rukus are the best match, but two sections cross a ruku boundary:<jupyter_code>sorted(sectionStarts - startAyas["ruku"])<jupyter_output><empty_output><jupyter_text>First and last linesWe explore the first and last ayas of each section.We make a list of tuples of their nodes.Note that the number of an aya is not the same as the node number of the aya.Node numbers are like barcodes: they identify objects uniquely within the whole universe.Aya numbers only identify an aya within its sura.But no two ayas have the same node number.All functions of TF require node numbers, so every now and then we have to move from theidentifications in front of our nose to the underlying node numbers.That is what the `T.nodeFromSection()` does: it takes a pair (sura number, aya number) andconverts it into a unique aya node number.<jupyter_code>boundaryAyaNodes = []

for (beginAnum, endAnum) in sections:
    beginA = T.nodeFromSection((2, beginAnum))
    endA = T.nodeFromSection((2, endAnum))
    boundaryAyaNodes.append((beginA, endA))<jupyter_output><empty_output><jupyter_text>Here is the tuple we constructed: an abstract ensemble of barcodes:<jupyter_code>boundaryAyaNodes<jupyter_output><empty_output><jupyter_text>Now we ask TF to show them in a more insightful way:<jupyter_code>A.table(boundaryAyaNodes)<jupyter_output><empty_output><jupyter_text>But I do not read Arabic, and I want to get a sense of what is going on.We display them more extensively by using `A.show()` instead.Before we make the call, we set up a[display parameter](https://annotation.github.io/text-fabric/tf/advanced/options.html)that calls up the `translation@en` feature for each aya.<jupyter_code>A.displaySetup(extraFeatures="translation@en", prettyTypes=False)<jupyter_output><empty_output><jupyter_text>If we run `A.show()` now, we get a display of ayas and words with translations and morphological information.<jupyter_code>A.show(boundaryAyaNodes, baseTypes="group")<jupyter_output><empty_output><jupyter_text>Vocative and resumption particlesWith my unaided eye I can already see that the start aya tends to start with a vocative particle and the end aya with a resumption particle. We now build a table of all ayas in this sura, where we highlight first words if they are a resumption or vocative particle.We also highlight the ayas that are the start and end ayas of the sections.  Building a tuple of the objects of interestWe proceed by building a list of aya nodes and their first words.We also make sets of the ayas where a section starts and where a section ends.<jupyter_code>sura2Ayas = L.d(sura2, otype="aya")

ayaTuples = [(a, L.d(a, otype="word")[0]) for a in sura2Ayas]

startAyaNodes = set()
endAyaNodes = set()

for (beginAnum, endAnum) in sections:
    beginA = T.nodeFromSection((2, beginAnum))
    endA = T.nodeFromSection((2, endAnum))
    startAyaNodes.add(beginA)
    endAyaNodes.add(endA)<jupyter_output><empty_output><jupyter_text>Defining highlightsWe can make a dictionary of nodes and the colors we want to display their objects in.We can pass that dictionary as a[display parameter](https://annotation.github.io/text-fabric/tf/advanced/options.html)<jupyter_code>highlights = {}

for a in sura2Ayas:
    firstWord = L.d(a, otype="word")[0]
    isVoc = F.posx.v(firstWord) == "vocative"
    isRes = F.posx.v(firstWord) == "resumption"
    if isVoc or isRes:
        highlights[firstWord] = "lime" if isVoc else "fuchsia"
    if a in startAyaNodes:
        highlights[a] = "gold"
    elif a in endAyaNodes:
        highlights[a] = "lightcoral"<jupyter_output><empty_output><jupyter_text>Color display of sections and particlesAyas that start a section are marked with a bar in *gold*, the ones that end a section have a *light coral* bar.Resumptive particles at the start of an aya are highlighted in *fuchsia*, the voactive ones in *lime*.<jupyter_code>A.table(ayaTuples, start=240, end=260, highlights=highlights)<jupyter_output><empty_output><jupyter_text>SemanticsThe article says:> interior sections correspond to each other: the second section corresponds to the second-to-last, and so on concentrically.and> By means of concentric patterning, ring composition calls attention to the center. We are drawn to look here for the essential message.If you don't read Arabic, a translation is the next best way to access the semantics.  Shuffled ayasFor each section, we will list the first three ayas (b1, b2, b3), the middle three (m1, m2, m3) and the last three (e1, e2, e3).We produce them in the order b1 - e3 - b2 - e2 - b3 - e1 - m1 - m3 - m2.We also put the first words, if they are particles of the vocative or resumption kind in the table.Everything will be formatted in a markdown table.<jupyter_code>def dm(markdownString):
    display(Markdown(markdownString))<jupyter_output><empty_output><jupyter_text>Construction of the tableWe walk through the sections and grab all the information we need, and wrap it into a bigmarkdown table.<jupyter_code>markdown = """
section | aya | position | kind | particle | translation
--- | --- | --- | --- | --- | ---
""".lstrip()

for (i, (beginAnum, endAnum)) in enumerate(sections):
    ayas = {}
    ayas["b1"] = T.nodeFromSection((2, beginAnum))
    ayas["e3"] = T.nodeFromSection((2, endAnum))
    ayas["m2"] = (ayas["b1"] + ayas["e3"]) // 2
    ayas["b2"] = ayas["b1"] + 1
    ayas["b3"] = ayas["b1"] + 2
    ayas["m1"] = ayas["m2"] - 1
    ayas["m3"] = ayas["m2"] + 1
    ayas["e1"] = ayas["e3"] - 2
    ayas["e2"] = ayas["e3"] - 1
    for aName in "b1 e3 b2 e2 b3 e1 m1 m3 m2".split():
        a = ayas[aName]
        w = L.d(a, otype="word")[0]
        posx = F.posx.v(w)
        kind = posx if posx in {"vocative", "resumption"} else None
        particle = None if kind is None else F.ascii.v(w)
        markdown += f"""
{i + 1} | {F.number.v(a)} | {aName} | {kind or '&nbsp;'} | {particle or '&nbsp;'} | {Fs('translation@en').v(a)}
""".lstrip()<jupyter_output><empty_output><jupyter_text>Shuffled table display<jupyter_code>dm(markdown)<jupyter_output><empty_output><jupyter_text>Sentiment Mining (primitive)As the article points out, sections often start by speaking to *good* people or actions, and end with condemning *bad*people or actions. Can we base sentiment mining on this idea?  Positivity located in the ringsIf a verb, noun or adjective occurs in an e1, e2, or e3 aya, its positivity is deccreased by 1, if it occurs in a b1, b2, b3 it is increased by 1.This is very crude, because we have taken the ring structure very rigidly.In the actual text, there might be ayas that are less pertinent to the ring structure, and the middle of the ringdoes not have to coincide with the exact middle aya of the section.It is up to the expert to devise additional heuristics that help to identify the rings in a more precise mannner.  Calculating positivityNow we are going to assign a positivity to all lemmas (verb, noun or adjective) of the sura.<jupyter_code>positivity = collections.Counter()
contentCats = {"verb", "noun", "word"}

for (i, (beginAnum, endAnum)) in enumerate(sections):
    ayas = {}
    ayas["b1"] = T.nodeFromSection((2, beginAnum))
    ayas["e3"] = T.nodeFromSection((2, endAnum))
    ayas["m2"] = (ayas["b1"] + ayas["e3"]) // 2
    ayas["b2"] = ayas["b1"] + 1
    ayas["b3"] = ayas["b1"] + 2
    ayas["m1"] = ayas["m2"] - 1
    ayas["m3"] = ayas["m2"] + 1
    ayas["e1"] = ayas["e3"] - 2
    ayas["e2"] = ayas["e3"] - 1
    for aName in "b1 e3 b2 e2 b3 e1 m1 m3 m2".split():
        a = ayas[aName]
        pos = 1 if aName.startswith("b") else -1 if aName.startswith("e") else 0
        for w in L.d(a, otype="word"):
            if F.pos.v(w) in contentCats:
                positivity[F.lemma.v(w)] += pos<jupyter_output><empty_output><jupyter_text>Table of lemmas and their positivityWe show the positivity of the lemmas of this sura.<jupyter_code>for (lemma, pos) in sorted(
    positivity.items(),
    key=lambda x: (-x[1], x[0]),
)[0:30]:
    print(f"{lemma:<20} has positivity {pos:>2}")
for (lemma, pos) in sorted(
    positivity.items(),
    key=lambda x: (-x[1], x[0]),
)[-30:]:
    print(f"{lemma:<20} has positivity {pos:>2}")<jupyter_output>jaA^'a               has positivity -2
kasaba               has positivity -2
katama               has positivity -2
l~ayosa              has positivity -2
malak                has positivity -2
mulok                has positivity -2
naAr                 has positivity -2
nafos                has positivity -2
qadiyr               has positivity -2
rajul                has positivity -2
sulayoma`n           has positivity -2
xa$iYa               has positivity -2
yataEal~amu          has positivity -2
yubodiYa             has positivity -2
{bon                 has positivity -2
{qotatala            has positivity -2
{xotalafa            has positivity -2
$aha`dap             has positivity -3
kita`b               has positivity -3
kul~                 has positivity -3
talaY`               has positivity -3
yumil~a              has positivity -3
A^taY                has positivity -4
kaAtib               has positivity -4
kafara               has positivity -4
kataba               has [...]<jupyter_text>Coloring the suraFinally, we display the whole sura with words colored according to their positivity or negativity.  Color gradientsNegative words marked red, and postive ones marked green.The farther from zero, the darker the color.The[HSL scheme](https://developer.mozilla.org/en-US/docs/Web/CSS/color_valueHSL_colors)of color specification is very convenient here.<jupyter_code>def posColor(p):
    if p <= -20:
        return "hsl(0, 100%, 50%)"
    if p <= -5:
        return "hsl(0, 90%, 60%)"
    if p <= -3:
        return "hsl(0, 80%, 70%)"
    if p == -2:
        return "hsl(30, 80%, 70%)"
    if p == -1:
        return "hsl(60, 80%, 70%)"
    if p == 0:
        return "hsl(90, 70%, 80%)"
    if p == 1:
        return "hsl(120, 80%, 70%)"
    if p == 2:
        return "hsl(120, 90%, 80%)"
    if p == 3:
        return "hsl(120, 100%, 90%)"
    if p == 4:
        return "hsl(120, 100%, 100%)"
    if p >= 4:
        return "hsl(150, 100%, 100%)"<jupyter_output><empty_output><jupyter_text>Computing highlightsWe just compute a highlight mapping, much like we did before, but now we are going to highlightall words in the sura that have a positivity or negativity.<jupyter_code>startAyaNodes = set()
endAyaNodes = set()

for (beginAnum, endAnum) in sections:
    beginA = T.nodeFromSection((2, beginAnum))
    endA = T.nodeFromSection((2, endAnum))
    startAyaNodes.add(beginA)
    endAyaNodes.add(endA)
highlights = {}

for a in sura2Ayas:
    words = L.d(a, otype="word")
    firstWord = words[0]
    isVoc = F.posx.v(firstWord) == "vocative"
    isRes = F.posx.v(firstWord) == "resumption"
    for w in words:
        if w == firstWord:
            if isVoc or isRes:
                highlights[firstWord] = "lime" if isVoc else "fuchsia"
        if F.lemma.v(w) in positivity:
            pos = positivity[F.lemma.v(w)]
            highlights[w] = posColor(pos)
    if a in startAyaNodes:
        highlights[a] = "gold"
    elif a in endAyaNodes:
        highlights[a] = "lightcoral"<jupyter_output><empty_output><jupyter_text>Colored sentiment display<jupyter_code>A.table(ayaTuples, start=240, end=260, highlights=highlights)<jupyter_output><empty_output><jupyter_text>Saving dataThe real power of TF comes now: we can save the sentiments as a new feature and use it as if it were given with the data set.We save it to a temporary location, in your downloads folder, and load it from there.We use[TF.save()](https://annotation.github.io/text-fabric/tf/core/fabric.htmltf.core.fabric.FabricCore.save) to save the feature.  Gathering the parametersWe set up all data that `TF.save()` needs.<jupyter_code>metaData = {
    "sentiment": {
        "valueType": "int",
        "description": "crude sentiment values for some words in sura 2",
    },
}

edgeFeatures = {}
nodeFeatures = {
    "sentiment": {
        w: positivity[F.lemma.v(w)]
        for w in L.d(sura2, otype="word")
        if F.lemma.v(w) in positivity
    },
}

location = os.path.expanduser("~/Downloads/q-ran/quran/tf")
module = A.version<jupyter_output><empty_output><jupyter_text>Performing the saveNow we do the save action:<jupyter_code>TF.save(
    metaData=metaData,
    edgeFeatures=edgeFeatures,
    nodeFeatures=nodeFeatures,
    location=location,
    module=module,
)<jupyter_output>0.00s Exporting 1 node and 0 edge and 0 config features to ~/Downloads/q-ran/quran/tf/0.4:
   |     0.01s T sentiment            to ~/Downloads/q-ran/quran/tf/0.4
  0.01s Exported 1 node features and 0 edge features and 0 config features to ~/Downloads/q-ran/quran/tf/0.4<jupyter_text>CheckingLet's check (maybe this does not work on Windows. Open the file explorer and locate the file yourself.<jupyter_code>%%sh
ls -lR ~/Downloads/q-ran/quran/tf<jupyter_output>total 0
drwxr-xr-x  3 dirk  staff  96 Jun 10 21:23 0.4

/Users/dirk/Downloads/q-ran/quran/tf/0.4:
total 40
-rw-r--r--  1 dirk  staff  19703 Jun 10 21:23 sentiment.tf<jupyter_text>Using new dataWe are going to load the Quran data again plus the sentiment data  Loading the sentiment dataWe use roughly the same statement as when we first loaded the data,but now we ask TF also to look into the directory where we saved the sentiments.We do that by supplying the `locations` and `modules` parameters.See also[A.use()](https://annotation.github.io/text-fabric/tf/advanced/app.htmltf.advanced.app.App)<jupyter_code>A = use(
    "quran:clone", checkout="clone", hoist=globals(), locations=location, modules=module
)
# A = use('quran', hoist=globals(), locations=location, modules=module)<jupyter_output><empty_output><jupyter_text>See: the feature `sentiment` got loaded.  Inspecting the sentiment dataLet's inspect it.<jupyter_code>F.sentiment.freqList()<jupyter_output><empty_output><jupyter_text>or, a bit more pleasing<jupyter_code>for (sent, amount) in F.sentiment.freqList():
    print(f"sentiment {sent:>3} occurs {amount:>3} times")<jupyter_output>sentiment  -1 occurs 710 times
sentiment   0 occurs 621 times
sentiment   1 occurs 475 times
sentiment  -2 occurs 317 times
sentiment -21 occurs 282 times
sentiment   2 occurs 159 times
sentiment   3 occurs  89 times
sentiment  -4 occurs  79 times
sentiment  -3 occurs  62 times
sentiment  -7 occurs  28 times
sentiment  -5 occurs  21 times
sentiment   4 occurs  13 times<jupyter_text>Displaying sentiment dataRemember the display where we showed the morphology and the translations?We can make the same display, but now with sentiments shown.We show only some of the ayas.We have to add the feature `sentiment` to the features to be displayed.<jupyter_code>A.displaySetup(extraFeatures="translation@en sentiment")
A.show(ayaTuples, start=10, end=20)<jupyter_output><empty_output><jupyter_text>Searching for sentiment dataWe can use the sentiment feature in searches.Let's look for negative words in ayas starting with a vocative particle.<jupyter_code>query = """
sura number=2
  aya
    =: word posx=vocative
    word sentiment<0
"""
results = A.search(query)<jupyter_output>0.24s 107 results<jupyter_text>Before we show the results, lets furnish an uncluttered[display](https://annotation.github.io/text-fabric/tf/advanced/display.htmltf.advanced.display.displaySetup)by suppressing a number of features.<jupyter_code>A.displaySetup(extraFeatures="translation@en", suppress="number posx")<jupyter_output><empty_output><jupyter_text>We want to see multiple negative words of an aya highlighted in the same aya, instead of displaying the aya over and over again for eachnegative word. Hence we'll pass `condensed=True`.<jupyter_code>A.show(results, start=1, end=5, condensed=True)<jupyter_output><empty_output><jupyter_text>Export to ExcelIt is convenient to export this to Excel, for more intense inspection, aided by all the tools that Excel has to offer for sorting, filtering and making charts.We use the function[A.export()]()to do this.This function takes a list of node tuples, retrieves data about those nodes, and writes it to a .tsv file (tab-separated), thatcan be opened by Excel (even with Arabic characters).<jupyter_code>A.export(results)<jupyter_output><empty_output><jupyter_text>The result is now in your Downloads folder, with name `results.tsv`<jupyter_code>%%sh
ls -l ~/Downloads/results.tsv
head -n 10 ~/Downloads/results.tsv<jupyter_output>-rw-r--r--@ 1 dirk  staff  204326 Jun 10 21:23 /Users/dirk/Downloads/results.tsv
��R	S1	S2	NODE1	TYPE1	number1	NODE2	TYPE2	TEXT2	NODE3	TYPE3	TEXT3	posx3	NODE4	TYPE4	TEXT4	sentiment4
1	2	21	218170	sura	2	128247	aya	JNpS#NJQOGN' qDFQN'3O q9R(O/OH'� 1N(QNCOEO qDQN0PI .NDNBNCOER HNqDQN0PJFN EPF BN(RDPCOER DN9NDQNCOER *N*QNBOHFN	433	word	JNpS	vocative	439	word	1N(QN	-1
2	2	21	218170	sura	2	128247	aya	JNpS#NJQOGN' qDFQN'3O q9R(O/OH'� 1N(QNCOEO qDQN0PI .NDNBNCOER HNqDQN0PJFN EPF BN(RDPCOER DN9NDQNCOER *N*QNBOHFN	433	word	JNpS	vocative	451	word[...]<jupyter_text>Not very pretty, because the file is in utf16 and the terminal does not handle that well.But this is the encoding that Excel can use, and there it looks allright:![nega](images/negativeExcel.png)  Share your dataYou can share the new `sentiment` feature in such a way that others can use it easily, even more easy than we just did.The thing to do is:* use text-fabric to store your features in a zip file* make or use a github repository* create a new release and add the zip file to itThese steps are explained in the [share](share.ipynb) tutorial.There we made an even cruder feature `sentiment`, with words in all suras, and stored it on GitHubin[q-ran/exercises/mining/tf](https://github.com/q-ran/exercises/tree/master/mining/tf/0.3).  Re-use your dataThat means that we can call up that data alongside the Quran data by passing an extra argument to `use()`:<jupyter_code>A = use(
    "quran:clone",
    checkout="clone",
    hoist=globals(),
    mod="q-ran/exercises/mining/tf:clone",
)
# A = use('quran', hoist=globals(), mod='q-ran/exercises/mining/tf')<jupyter_output><empty_output><jupyter_text>Note that you can now click on the `sentiment` feature above, which will bring you tothe repository where it is stored.If the creator of the data did a good job, you'll find documentation about `sentiment` in its readme fileor docs directory.We check the feature itself for its distribution.<jupyter_code>F.sentiment.freqList()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12293783564501796, 'fraction_numerical': 0.042035312753544794, 'mean_word_length': 3.9396946564885496, 'pattern_counts': {'":': 4, '<': 156, '<?xml version=': 0, '>': 161, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '265995', 'license': 'MIT', 'n_tokens_mistral': 10425, 'n_tokens_neox': 9517, 'n_words': 3014, 'path': 'quran/rings.ipynb', 'repo_name': 'sethbam9/tutorials', 'size': '25894'}