<jupyter_start><jupyter_text>You might want to consider the [start](search.ipynb) of this tutorial.Short introductions to other TF datasets:* [Dead Sea Scrolls](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/dss.ipynb),* [Old Babylonian Letters](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/oldbabylonian.ipynb),or the* [Q'uran](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/quran.ipynb)  Volume supportText-Fabric 9.0.0 introduces volume support.Read[here](https://annotation.github.io/text-fabric/tf/about/volumes.html)what that is and why you might want it.In this tutorial we show the practical side:how to *extract volumes* from works and *collect* several *volumes* into *collections*.<jupyter_code>%load_ext autoreload
%autoreload 2
import os
from tf.app import use
from tf.fabric import Fabric
from tf.volumes import extract, collect
from tf.core.helpers import unexpanduser as ux
GH = os.path.expanduser("~/github")
BH = f"{GH}/clariah/wp6-missieven"
VERSION = "1.0"
SOURCE = f"{BH}/tf/{VERSION}"
TARGET = f"{BH}/tf/{VERSION}/_local"<jupyter_output><empty_output><jupyter_text>Work and volumesWe use the Hebrew Bible as *work*.The *volumes* of a work are lists of its top-level sections.Volumes may have a name.We define three volumes out of the smallest books of the bible:<jupyter_code>VOLUMES = dict(
    middle=(8,),
    beginning=(1, 2),
    end=(13, 14),
)
COLLECTION = "selected"<jupyter_output><empty_output><jupyter_text>Volume supportWe can work with works through several TF apis:* the usual, high-level API using `A = use(work)`.* the basic, low-level API using `TF = Fabric(locations, modules)`* as plain functions `extract()` and `collect()`.We show all ways of doing it.  Load the workWe load the corpus in the advanced way:<jupyter_code>Aw = use("clariah/wp6-missieven:clone", checkout="clone")<jupyter_output><empty_output><jupyter_text>We can now extract volumes by using the `extract()` method on the `app` objectwhich is held in the variable `Aw`.Note: we are going to load several volumes and collections too, so instead storing thehandle to the API in a variable with the name `A`, we choose one with the name `Aw`.  Extract volumes<jupyter_code>volumes = Aw.extract(VOLUMES, overwrite=True)<jupyter_output>0.00s Check volumes ...
   |   Volume middle exists and will be recreated
   |   Volume beginning exists and will be recreated
   |   Volume end exists and will be recreated
   |   Work consists of 14 volumes:
   |   volume 1                   : with   352727 slots
   |   volume 2                   : with   400901 slots
   |   volume 3                   : with   439609 slots
   |   volume 4                   : with   366207 slots
   |   volume 5                   : with   412436 slots
   |   volume 6                   : with   434227 slots
   |   volume 7                   : with   393559 slots
   |   volume 8                   : with   125206 slots
   |   volume 9                   : with   464578 slots
   |   volume 10                  : with   637655 slots
   |   volume 11                  : with   499612 slots
   |   volume 12                  : with   342699 slots
   |   volume 13                  : with   371337 slots
   |   volume 14                  : with   736607 slots
  0.[...]<jupyter_text>Inspect the volumesThe `extract()` method returns basic information about the volumes:their location on disk.<jupyter_code>if volumes:
    for (name, info) in volumes.items():
        loc = info["location"]
        new = "(new)     " if info["new"] else "(existing)"
        print(f"volume {name:<7}: {new} at {ux(loc)}")
else:
    print(volumes)<jupyter_output>volume middle : (new)      at ~/github/clariah/wp6-missieven/tf/1.0/_local/middle
volume beginning: (new)      at ~/github/clariah/wp6-missieven/tf/1.0/_local/beginning
volume end    : (new)      at ~/github/clariah/wp6-missieven/tf/1.0/_local/end<jupyter_text>Load single volumesWe load the volumes separately.For each volume we get a handle, which we store in a dictionary `As`, keyed by its name.<jupyter_code>As = {}

for name in volumes:
    As[name] = use("clariah/wp6-missieven:clone", checkout="clone", volume=name)<jupyter_output><empty_output><jupyter_text>We see it reported that single volumes have been loaded instead of the whole work.The volume info can be obtained separately by reading the attribute `volumeInfo`,either on the `A` or on the `TF` object:<jupyter_code>for name in volumes:
    print(As[name].volumeInfo)<jupyter_output>middle:8
beginning:1-2
end:13-14<jupyter_text>Generated featuresWhen volumes are created, some extra features are generated, which have to do with the relationbetween the original work and the volume, and what happens at the boundaries of volumes.<jupyter_code>for name in volumes:
    print(name)
    for (feat, info) in As[name].isLoaded("owork ointerfrom ointerto", pretty=False).items():
        print(f"\t{feat}: {info['meta']['description']}")<jupyter_output>middle
	owork: mapping from nodes in the volume to nodes in the work
	ointerfrom: all outgoing inter-volume edges
	ointerto: all incoming inter-volume edges
beginning
	owork: mapping from nodes in the volume to nodes in the work
	ointerfrom: all outgoing inter-volume edges
	ointerto: all incoming inter-volume edges
end
	owork: mapping from nodes in the volume to nodes in the work
	ointerfrom: all outgoing inter-volume edges
	ointerto: all incoming inter-volume edges<jupyter_text>oworkNote that each volume has an extra feature: `owork`. Its value for each node in a volume datasetis the corresponding node in the *original work* from which the volume is taken.If you use the volume to compute annotations,and you want to publish these annotations against the original work,the feature `owork` provides the necessary information to do so.Suppose `annotvx` is a dict, mapping some nodes in the volume `x` to interesting values,then you apply them to the original work as follows``` python{F.owork.v(n): value for (n, value) in annotvx.items}```There is another important function of `owork`: when collecting volumes, we may encounter nodes in the volumesthat come from a single node in the work. We want to *merge* these nodes in the collected work.The information in `owork` provides the necessary information for that.  ointerto, ointerfromNote that we do have features `ointerto` and `ointerfrom`.They are used to store information that spans different volumes:edges from nodes in one volume  to nodes in another volume.  Make collections of volumesWe can collect volumes into new works by means of the `collect()` method on `Aw`.Let's collect all volumes just created.<jupyter_code>Aw.collect(
    tuple(volumes),
    COLLECTION,
    overwrite=True,
)<jupyter_output>Collection selected exists and will be recreated
  0.00s Loading volume middle                                                       from ~/github/clariah/wp6-missieven/tf/1.0/_local/middle ...
This is Text-Fabric 9.4.2
Api reference : https://annotation.github.io/text-fabric/tf/cheatsheet.html

46 features found and 0 ignored
  0.09s All features loaded/computed - for details use TF.isLoaded()
  0.09s Feature overview: 43 for nodes; 2 for edges; 1 configs; 9 computed
  0.02s All additional features loaded - for details use TF.isLoaded()
  0.12s Loading volume beginning                                                    from ~/github/clariah/wp6-missieven/tf/1.0/_local/beginning ...
This is Text-Fabric 9.4.2
Api reference : https://annotation.github.io/text-fabric/tf/cheatsheet.html

46 features found and 0 ignored
  0.42s All features loaded/computed - for details use TF.isLoaded()
  0.43s Feature overview: 43 for nodes; 2 for edges; 1 configs; 9 computed
  0.13s All additional featur[...]<jupyter_text>Load collectionWe can load the collection in the same way as a volume, but now using `collection=`:<jupyter_code>Ac = use("clariah/wp6-missieven:clone", checkout="clone", collection=COLLECTION)<jupyter_output><empty_output><jupyter_text>Which volumes have we got?<jupyter_code>Fc = Ac.api.F
Tc = Ac.api.T

for b in Fc.otype.s("volume"):
    print(Tc.sectionFromNode(b)[0])<jupyter_output>8
1
2
13
14
{'fraction_non_alphanumeric': 0.09125151883353584, 'fraction_numerical': 0.02612393681652491, 'mean_word_length': 3.608622620380739, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 39, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '560666', 'license': 'MIT', 'n_tokens_mistral': 2621, 'n_tokens_neox': 2424, 'n_words': 880, 'path': 'tutorial/volumes.ipynb', 'repo_name': 'CLARIAH/wp6-missieven', 'size': '8230'}