<jupyter_start><jupyter_text>CALAP from MQL into TF About CALAPCALAP stands for Computer-Assisted Linguistic Analysis of the Peshitta.It was a [project](https://openaccess.leidenuniv.nl/handle/1887/10866) at the University of Leiden.It has been followed up with another project, TURGAMA, which ended in 2010.Here is a [report](https://github.com/ETCBC/calap/blob/master/docs/pdf/TURGAMA2010-03.pdf).The [Peshitta](http://en.wikipedia.org/wiki/Peshitta) is a collection of Syriac texts. According to Wikipedia it is the standard version of the Bible in churches of the Syriac tradition. Resources can be found on [peshitta.org](http://www.peshitta.org).The text we use below comes from the [Peshitta Institute Leiden](http://www.hum.leiden.edu/religion/research/peshitta-institute/peshitta-institute.html), and has been prepared as an EMDROS database, which is now held by the [ETCBC](http://www.godgeleerdheid.vu.nl/etcbc). LAFFrom there is has been converted (2015?) to [LAF](http://www.iso.org/iso/catalogue_detail.htm?csnumber=37326)by <NAME>.The EMDROS result and LAF-data of the CALAP project has been archived at DANS:[DOI 10.17026/dans-zv9-w9d2](http://dx.doi.org/10.17026/dans-zv9-w9d2). Text-FabricThis notebook can read an[MQL](https://emdros.org/mql.html)dump of a version of the CALAP (Peshitta)and transform it in a Text-Fabric[Text-Fabric](https://github.com/Dans-labs/text-fabric)resource.**N.B.:**There is an error in the dump: the last monad (slot) is 53920, but the last book is declared to reachuntil the non-existent slot 53921:```CREATE OBJECTFROM MONADS =  { 35794-53920 }WITH ID_D = 144485[book  book := Sirach;]GO```The correction consists of replacing `53921` by `53920`, here, in one place only (line 503).We have corrected this manually and bzipped the result and stored it in the *sources* subdirectoryof this repo.The source only has the text in transliteration, in the feature `surface_consonants`.We provide a key to the transliteration and a feature `cons` that contains the backliterated unicode strings.We have an [overview of the features](https://github.com/ETCBC/calap/blob/master/docs/features.md)Once the conversion is done, you can run the [examples.ipynb](examples.ipynb) notebook to see what you can do with it.<jupyter_code>import os
from shutil import rmtree
from tf.fabric import Fabric
import utils<jupyter_output><empty_output><jupyter_text>Setting up the context: source file and target directoriesThe conversion is executed in an environment of directories, so that sources, temp files andresults are in convenient places and do not have to be shifted around.<jupyter_code>PROJECT = "calap"
VERSION = "2014"

repoBase = os.path.expanduser("~/github/etcbc")
thisRepo = "{}/{}".format(repoBase, PROJECT)

thisSource = "{}/source/{}".format(thisRepo, VERSION)
mqlzFile = "{}/{}.mql.bz2".format(thisSource, PROJECT)

thisTemp = "{}/_temp/{}".format(thisRepo, VERSION)
thisTempSource = "{}/source".format(thisTemp)
mqlFile = "{}/{}.mql".format(thisTempSource, PROJECT)
thisTempTf = "{}/tf".format(thisTemp)

thisTf = "{}/tf/{}".format(thisRepo, VERSION)<jupyter_output><empty_output><jupyter_text>Only update if and when needed, or force update of everything<jupyter_code>FORCE = False<jupyter_output><empty_output><jupyter_text>TestCheck whether this conversion is needed in the first place.Only when run as a script.<jupyter_code>testFile = "{}/.tf/otype.tfx".format(thisTf)
(good, work) = utils.mustRun(mqlzFile, "{}/.tf/otype.tfx".format(thisTf), force=FORCE)
print(f'The text fabric files {"must be updated" if good else "are up to date"}')<jupyter_output>|       0.00s 	Source /Users/dirk/github/etcbc/calap/source/2014/calap.mql.bz2 exists
|       0.00s 	Destination /Users/dirk/github/etcbc/calap/tf/2014/.tf/otype.tfx exists
|       0.00s 	Destination /Users/dirk/github/etcbc/calap/tf/2014/.tf/otype.tfx up to date
The text fabric files must be updated<jupyter_text>TF SettingsWe add some custom information here.* the MQL object type that corresponds to the Text-Fabric slot type, typically `word`;* a piece of metadata that will go into every feature; the time will be added automatically* suitable text formats for the `otext` feature of TF.The oText feature is very sensitive to what is available in the source MQL.It needs to be configured here.We save the configs we need per source and version.And we define a stripped down default version to start with.<jupyter_code>slotType = "word"

featureMetaData = dict(
    project="CALAP and TURGAMA",
    organizations="Peshitta Institute Leiden; Werkgroep Informatica at the Vrije Universiteit Amsterdam (WIVU)",
    projectPeriod="1999-2005, 2005-2010",
    projectUrl="http://www.nwo.nl/onderzoek-en-resultaten/onderzoeksprojecten/78/1900123778.html",
    encoders="<NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>",
    language="Syriac",
    iso639="syc",
    dataset="CALAP",
    version=VERSION,
    datasetName="Computer-Assisted Linguistic Analysis of the Peshitta",
    author="E<NAME>stra Centre for Bible and Computer",
    converter="Dirk Roorda (TF)",
)

oText = {
    "": {
        "": """
@fmt:text-trans-full={surface_consonants} 
@sectionFeatures=book,chapter,verse
@sectionTypes=book,chapter,verse
""",  # noqa W291
    },
    "2014": """
@fmt:text-trans-full={surface_consonants} 
@sectionFeatures=book,chapter,verse
@sectionTypes=book,chapter,verse
""",  # noqa W291
}<jupyter_output><empty_output><jupyter_text>The next function selects the proper otext material, falling back on a default if nothingappropriate has been specified in `oText`.<jupyter_code>thisOtext = oText.get(VERSION, oText[""])

if thisOtext is oText[""]:
    utils.caption(
        0, "WARNING: no otext feature info provided, using a meager default value"
    )
    otextInfo = {}
else:
    utils.caption(0, "INFO: otext feature information found")
    otextInfo = dict(
        line[1:].split("=", 1) for line in thisOtext.strip("\n").split("\n")
    )
    for x in sorted(otextInfo.items()):
        utils.caption(0, '\t{:<20} = "{}"'.format(*x))<jupyter_output>|       0.04s INFO: otext feature information found
|       0.04s 	fmt:text-trans-full  = "{surface_consonants} "
|       0.04s 	sectionFeatures      = "book,chapter,verse"
|       0.04s 	sectionTypes         = "book,chapter,verse"<jupyter_text>OverviewThe program has several stages:1. **prepare** the source (utils.bunzip if needed)1. **convert** convert the MQL file into a text-fabric dataset1. **differences** (informational)1. **deliver** the TF data at its destination directory1. **compile** all TF features to binary format  PrepareCheck the source, utils.bunzip it if needed, empty the result directory.<jupyter_code>if not os.path.exists(thisTempSource):
    os.makedirs(thisTempSource)

utils.caption(0, "bunzipping {} ...".format(mqlzFile))
utils.bunzip(mqlzFile, mqlFile)
utils.caption(0, "Done")

if os.path.exists(thisTempTf):
    rmtree(thisTempTf)
os.makedirs(thisTempTf)<jupyter_output>|       0.06s bunzipping /Users/dirk/github/etcbc/calap/source/2014/calap.mql.bz2 ...
|       0.06s 	NOTE: Using existing unzipped file which is newer than bzipped one
|       0.06s Done<jupyter_text>MQL to Text-FabricTransform the collected information in feature-like data-structures, and write it allout to `.tf` files.<jupyter_code>TF = Fabric(locations=thisTempTf, silent=True)
TF.importMQL(mqlFile, slotType=slotType, otext=otextInfo, meta=featureMetaData)<jupyter_output>0.00s Parsing mql source ...
  0.00s 		enum boolean
  0.00s 		enum lexical_set_e
  0.00s 		enum phrase_type_e
  0.00s 		enum phrase_atom_type_e
  0.00s 		enum phrase_function_e
  0.01s 		enum book_name_e
  0.01s 		enum part_of_speech_e
  0.01s 		enum gender_e
  0.01s 		enum number_e
  0.01s 		enum person_e
  0.01s 		enum stem_e
  0.01s 		enum state_e
  0.01s 		enum voice_e
  0.01s 		enum determination_e
  0.01s 		enum phrase_atom_relation_e
  0.01s 		enum subphrase_type_e
  0.01s 		enum subphrase_kind_e
  0.01s 		enum clause_type_e
  0.01s 		enum clause_atom_type_e
  0.01s 		enum clause_constituent_relation_e
  0.01s 		enum tense_e
  0.02s 		otype book
  0.02s 			feature book (str) =def= Genesis : node
  0.02s 		otype verse
  0.02s 			feature verse_label (str) =def= "" : node
  0.02s 			feature verse (int) =def= 0 : node
  0.02s 			feature chapter (int) =def= 0 : node
  0.02s 			feature book (str) =def= Genesis : node
  0.02s 		otype chapter
  0.02s 			feature chapter (int) =def= 0 :[...]<jupyter_text>DiffsCheck differences with previous versions.The new dataset has been created in a temporary directory,and has not yet been copied to its destination.Here is your opportunity to compare the newly created features with the older features.You expect some differences in some features.We check the differences between the previous version of the features and what has been generated.We list features that will be added and deleted and changed.For each changed feature we show the first line where the new feature differs from the old one.We ignore changes in the metadata, because the timestamp in the metadata will always change.<jupyter_code>utils.checkDiffs(thisTempTf, thisTf)<jupyter_output>..............................................................................................
.         14s Check differences with previous version                                        .
..............................................................................................
|         14s 	no features to add
|         14s 	2 features to delete
|         14s 		book@en
|         14s 		cons
|         14s 	37 features in common
|         14s analyzed_form             ... no changes
|         14s book                      ... no changes
|         14s chapter                   ... no changes
|         14s determination             ... no changes
|         14s emf                       ... no changes
|         14s frv                       ... no changes
|         14s gender                    ... no changes
|         14s is_apposition             ... no changes
|         14s lexeme                    ... no changes
|         14s morph_state               ... no changes
|         14s[...]<jupyter_text>DeliverCopy the new TF dataset from the temporary location where it has been created to its final destination.<jupyter_code>utils.deliverDataset(thisTempTf, thisTf)<jupyter_output>..............................................................................................
.         15s Deliver data set to /Users/dirk/github/etcbc/calap/tf/2014                     .
..............................................................................................<jupyter_text>Compile TFJust to see whether everything loads and the pre-computing of extra information works out.Moreover, if you want to work with these features, then the pre-computing has already been done, and everything is quicker in subsequent runs.We issue load statement to trigger the pre-computing of extra data.Note that all features specified text formats in the `otext` config feature,will be loaded, as well as the features for sections.At that point we have access to the full list of features.We grab them and are going to load them all!<jupyter_code>utils.caption(4, "Load and compile standard TF features")
TF = Fabric(locations=thisTf, modules=[""])
api = TF.load("")

utils.caption(4, "Load and compile all other TF features")
allFeatures = TF.explore(silent=False, show=True)
loadableFeatures = allFeatures["nodes"] + allFeatures["edges"]
api = TF.load(loadableFeatures)
api.makeAvailableIn(globals())<jupyter_output>..............................................................................................
.         15s Load and compile standard TF features                                          .
..............................................................................................
This is Text-Fabric 3.1.3
Api reference : https://github.com/Dans-labs/text-fabric/wiki/Api
Tutorial      : https://github.com/Dans-labs/text-fabric/blob/master/docs/tutorial.ipynb
Example data  : https://github.com/Dans-labs/text-fabric-data

37 features found and 0 ignored
  0.00s loading features ...
   |     0.15s T otype                from /Users/dirk/github/etcbc/calap/tf/2014
   |     0.83s T oslots               from /Users/dirk/github/etcbc/calap/tf/2014
   |     0.01s T book                 from /Users/dirk/github/etcbc/calap/tf/2014
   |     0.01s T chapter              from /Users/dirk/github/etcbc/calap/tf/2014
   |     0.01s T verse                from /Users/dirk/github/etcbc/calap/tf/2014
[...]<jupyter_text>Add unicode text and English book names<jupyter_code>syriacMapping = {
    ">": "\u0710",  # alaph
    "B": "\u0712",  # beth
    "G": "\u0713",  # gamal
    "D": "\u0715",  # dalat
    "H": "\u0717",  # he
    "W": "\u0718",  # waw
    "Z": "\u0719",  # zain
    "X": "\u071A",  # heth
    "V": "\u071B",  # teth
    "J": "\u071D",  # yudh
    "K": "\u071F",  # kaph
    "L": "\u0720",  # lamadh
    "M": "\u0721",  # mim
    "N": "\u0722",  # nun
    "S": "\u0723",  # semkath
    "<": "\u0725",  # e
    "P": "\u0726",  # pe
    "Y": "\u0728",  # sadhe
    "Q": "\u0729",  # qaph
    "R": "\u072A",  # rish
    "C": "\u072B",  # shin
    "T": "\u072C",  # taw
    "s": "\u0724",  # semkath final
    "p": "\u0727",  # pe reversed
}


def syriac(translit):
    return "".join(syriacMapping.get(c, c) for c in translit)
metaData = {
    "": featureMetaData,
    "cons": {
        "valueType": "str",
        "source": "feature surface_consonants",
        "method": "back transliteration",
    },
    "book@en": {
        "valueType": "str",
        "source": "feature book",
        "method": "copy",
    },
}

metaData["otext"] = dict()
metaData["otext"].update(T.config)
metaData["otext"].update({"fmt:text-orig-full": "{cons} "})

nodeFeatures = {}
edgeFeatures = {}
cons = {}
bookEn = {}
for n in N():
    sf = F.surface_consonants.v(n)
    if sf is not None:
        cons[n] = syriac(sf)
for n in F.otype.s("book"):
    bookEn[n] = F.book.v(n)
nodeFeatures["cons"] = cons
nodeFeatures["book@en"] = bookEn
TF = Fabric(locations=thisTf, silent=True)
TF.save(nodeFeatures=nodeFeatures, edgeFeatures=edgeFeatures, metaData=metaData)
utils.caption(4, "Load and compile standard TF features")
TF = Fabric(locations=thisTf, modules=[""])
api = TF.load("")

utils.caption(4, "Load and compile all other TF features")
allFeatures = TF.explore(silent=False, show=True)
loadableFeatures = allFeatures["nodes"] + allFeatures["edges"]
api = TF.load(loadableFeatures)
api.makeAvailableIn(globals())<jupyter_output>..............................................................................................
.         27s Load and compile standard TF features                                          .
..............................................................................................
This is Text-Fabric 3.1.3
Api reference : https://github.com/Dans-labs/text-fabric/wiki/Api
Tutorial      : https://github.com/Dans-labs/text-fabric/blob/master/docs/tutorial.ipynb
Example data  : https://github.com/Dans-labs/text-fabric-data

39 features found and 0 ignored
  0.00s loading features ...
   |     0.24s T cons                 from /Users/dirk/github/etcbc/calap/tf/2014
   |     0.00s M otext                from /Users/dirk/github/etcbc/calap/tf/2014
   |      |     0.02s C __sections__         from otype, oslots, otext, __levUp__, __levels__, book, chapter, verse
   |     0.00s T book@en              from /Users/dirk/github/etcbc/calap/tf/2014
   |     0.00s Feature overview: 36 for nodes; 1[...]
{'fraction_non_alphanumeric': 0.16056013811624784, 'fraction_numerical': 0.03286655156979346, 'mean_word_length': 2.7326968973747015, 'pattern_counts': {'":': 37, '<': 55, '<?xml version=': 0, '>': 54, 'https://': 11, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '120460', 'license': 'MIT', 'n_tokens_mistral': 5316, 'n_tokens_neox': 4935, 'n_words': 1411, 'path': 'programs/tfFromMql.ipynb', 'repo_name': 'ETCBC/calap', 'size': '15725'}