Model Card for ark-omikuji-eng-title-content

An Annif model, trained on historical titles and additional catalogue metadata for automatic subject indexing tasks. It classifies a given text into one or multiple subjects from the “Alter Realkatalog” (ARK) classification system. The model was developed in the research project Human.Machine.Culture at Staatsbibliothek zu Berlin – Berlin State Library (SBB).

Table of Contents

• Model Card for ark-omikuji-eng-title-content
• Table of Contents
• Model Details
  • Model Description
• Uses
  • Direct Use
  • Downstream Use
  • Out-of-Scope Use
• Bias, Risks, and Limitations
  • Recommendations
• Training Details
  • Training Data
  • Training Procedure
    • Preprocessing
    • Speeds, Sizes, Times
    • Training hyperparameters
    • Training results
• Evaluation
  • Testing Data, Factors and Metrics
    • Testing Data
    • Metrics
• Environmental Impact
• Technical Specifications
  • Model Architecture and Objective
    • Software
• Model Card Authors
• Model Card Contact
• How to Get Started with the Model

Model Details

Model Description

An Annif model, trained on historical titles and additional catalogue metadata for automatic subject indexing tasks. Subject indexing is a classical library task, aiming at describing the content of a resource. The model is intended to be used to automatically classify historical texts with a historical classification system developed in the 19th century to enrich those texts that have not been classified manually so far. For each input text, the model outputs one or multiple subjects from the ARK classification system. It is part of a collection of 5 models, created with the help of the Annif toolkit which addresses this task of automated subject indexing.

• Developed by: Sophie Schneider
• Shared by: Staatsbibliothek zu Berlin – Berlin State Library
• Model type: tree-based
• Language(s) (NLP): en (English)
• License: apache-2.0

Uses

Direct Use

This model can directly be used to automatically classify historical texts with the ARK classification scheme. It is intended to be used together with the Annif automated subject indexing toolkit version 0.60.0-1.1.0.

Downstream Use

Other/downstream uses outside of the Annif setting described above are not intended but also not excluded.

Out-of-Scope Use

The model is not intended for use on contemporary texts, as language and concept drifts will probably influence the results negatively and some terms from the vocabulary are not appropriate for more recent publications.

Bias, Risks, and Limitations

Since we are dealing with historical texts and especially with a historical classification system such as the ARK, the classes suggested for an input text might not be suitable for today’s understanding or might even be of an unethical nature (for more information, see also the datasheet accompanying the Metadata of the “Alter Realkatalog” (ARK of Berlin State Library) and the Datasheet for Machine-Readable Vocabulary Files of the ARK (Alter Realkatalog)).

Another limitation when using the ARK as a vocabulary arises from its hierarchical structure: the system contains multiple classes that do not describe the same content (e.g. they have different IDs) but are labeled identical (same name). This is due to the fact that the manual inspection of the whole path to a class, including all the upper level classes leading to it, delivers additional information that allows for contextualization. As duplicate label names seem to be - as expected - a challenge for lexical methods, we decided to focus on statistical rather than lexical algorithms.

Recommendations

Considering that the ARK classification scheme consists of 225.691 classes in total and that there is only limited training material at hand plus an overall unbalanced distribution of classes, we might describe this task as an Extreme Multi-Label Classification (XMC) problem. We recommend being aware of this limitation and, if available, use additional training data to improve the current model’s performance (e.g. by running annif learn, see CLI commands documentation).

Training Details

Training Data

Training data include a selection of metadata fields that were obtained via CBS export:

• Lehmann, J., & Schneider, S. (2024). Metadata of the "Alter Realkatalog" (ARK) of Berlin State Library (SBB) (Version 1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.12783813

The following title and content data fields have been extracted and combined from this dataset:

• "Abweichender Titel" (4212)
• "Abweichender Titel (Sucheinstieg)" (3260)
• "Ansetzungssachtitel" (3220)
• "Einheitssachtitel des beigefügten oder kommentierten Werkes" (4210)
• "Frühere/frühester Haupttitel (nur für fortlaufende und integrierende Ressourcen)" (4213)
• "Gesamttitel der Reproduktion" (4110)
• "Gesamttitel der fortlaufenden Ressource" (4170)
• "Gesamttitel der mehrteiligen Monografie" (4150)
• "Haupttitel, Titelzusatz, Verantwortlichkeitsangabe" (4000)
• "Normierter Zeitschriftenkurztitel" (3232)
• "Paralleltitel, paralleler Titelzusatz, parallele Verantwortlichkeitsangabe" (4002)
• "Titelkonkordanzen" (4245)
• "Titelzusätze und Verantwortlichkeitsangabe zur gesamten Vorlage" (4011)
• "Weitere Titel etc. bei Zusammenstellungen" (4010)
• "Weiterer Werktitel und sonstige unterscheidende Merkmale" (3211)
• "Werktitel und sonstige unterscheidende Merkmale des Werks" (3210)
• "Zusätzliche Sucheinstiege" (4200)
• "Veröffentlichungsart und Inhalt" (1140)
• "Sonstige Anmerkungen" (4201)
• "Zusammenfassende Register" (4203)
• "Inhaltliche Zusammenfassung" (4207 bzw. 9000)
• "Einleitender Text" (7124)

The vocabulary files themselves can be found here:

• Schneider, S., & Lehmann, J. (2024). Machine-Readable Vocabulary Files of the "Alter Realkatalog" (ARK) of Berlin State Library (SBB) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.13301020

Training Procedure

Training procedure includes loading the ARK vocabulary (see Datasheet for Machine-Readable Vocabulary Files of the ARK (Alter Realkatalog)) into Annif and training the Omikuji backend with the help of our training data. Further aspects on technical specifications can be found in the section Training hyperparameters.

Preprocessing

Besides merging and transforming the data described under Training Data to fit the corpus formats accepted by Annif, no further preprocessing of natural language or similar has been performed.

Speeds, Sizes, Times

Training takes from several minutes to a few hours on a V100, depending on the choice of dataset and algorithm as well as hyperparameter settings.

Training hyperparameters

For some of the ARK Annif models, a slight hyperparameter optimization has been conducted to identify the final hyperparameter settings stated below.

hyperparameter configuration (as it needs to be stated in the Annif projects.cfg file):

[ark-omikuji-eng-title-content]
name=ARK-DE-18 Omikuji
language=en
backend=omikuji
analyzer=snowball(english)
vocab=arktsv
cluster_k=2
collapse_every_n_layers=5

Training results

• Precision (--limit 1, --threshold 0): 0.3838
• Recall (--limit 1, --threshold 0): 0.3694
• F1 (--limit 1, --threshold 0): 0.3741
• NDCG (--limit 1, --threshold 0): 0.3795
• F1@5: 0.1839
• NDCG@5: 0.4702

Evaluation

Testing Data, Factors and Metrics

Testing Data

The dataset is described under Training Data. It was split into smaller subsets used for training, testing and validating (80%/10%/10% split).

Metrics

Model performance has been evaluated based on the following metrics: Precision, Recall, F1 and NDCG. These are standard metrics for machine learning and more specifically automatic subject indexing tasks and are directly provided in Annif by calling the annif eval statement. Evaluation parameters (--limit = maximum number of results to return; --threshold = minimum confidence for a suggestion to be considered) have been optimized before using the validation dataset and affect the results accordingly. We also state F1@5 and NDCG@5 scores reached without any evaluation parameters.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: V100
• Hours used: 0,5-5 hours
• Cloud Provider: No cloud.
• Compute Region: Germany.

Technical Specifications

Model Architecture and Objective

See Annif and Omikuji repositories on Github. Omikuji is an implementation of Partitioned Label Trees (Prabhu et al., 2018):

• Y. Prabhu, A. Kag, S. Harsola, R. Agrawal, and M. Varma, “Parabel: Partitioned Label Trees for Extreme Classification with Application to Dynamic Search Advertising,” in Proceedings of the 2018 World Wide Web Conference, 2018, pp. 993–1002.

Software

To run this model, Annif version 0.60.0 or higher (min. up to 1.1.0) must be installed.

Model Card Authors

Sophie Schneider and Jörg Lehmann

Model Card Contact

Questions and comments about the model can be directed to Sophie Schneider at [email protected], questions and comments about the model card can be directed to Jörg Lehmann at [email protected].

How to Get Started with the Model

Follow the Annif Getting Started page to set up and run Annif. Create a projects.cfg file (see section Training hyperparameters for details on the specific project configuration), load the ARK vocabulary (see Datasheet for Machine-Readable Vocabulary Files of the ARK (Alter Realkatalog)) via annif load-vocab command and copy the model folder over to data/projects.