task_categories:text-classification size_categories:1M<n<10M language:en license:apache-2.0 Ontologies Subsumption Inference Natural Language Inference Conceptual Knowledge LMs-as-KBs region:us
OntoLAMA: LAnguage Model Analysis datasets for Ontology Subsumption Inference.
@inproceedings{he2023language,
  title={Language Model Analysis for Ontology Subsumption Inference},
  author={He, Yuan and Chen, Jiaoyan and Jim{\'e}nez-Ruiz, Ernesto and Dong, Hang and Horrocks, Ian},
  booktitle={Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics},
  year={2023}
}
---
license: apache-2.0
task_categories:
- text-classification
tags:
- Ontologies
- Subsumption Inference
- Natural Language Inference
- Conceptual Knowledge
- LMs-as-KBs
pretty_name: OntoLAMA
size_categories:
- 1M<n<10M
language:
- en
dataset_info:
- config_name: schemaorg-atomic-SI
  features:
  - name: v_sub_concept
    dtype: string
  - name: v_super_concept
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': negative_subsumption
          '1': positive_subsumption
  - name: axiom
    dtype: string
  splits:
  - name: train
    num_bytes: 103485
    num_examples: 808
  - name: validation
    num_bytes: 51523
    num_examples: 404
  - name: test
    num_bytes: 361200
    num_examples: 2830
  download_size: 82558
  dataset_size: 516208
- config_name: doid-atomic-SI
  features:
  - name: v_sub_concept
    dtype: string
  - name: v_super_concept
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': negative_subsumption
          '1': positive_subsumption
  - name: axiom
    dtype: string
  splits:
  - name: train
    num_bytes: 15803053
    num_examples: 90500
  - name: validation
    num_bytes: 1978584
    num_examples: 11312
  - name: test
    num_bytes: 1977582
    num_examples: 11314
  download_size: 3184028
  dataset_size: 19759219
- config_name: foodon-atomic-SI
  features:
  - name: v_sub_concept
    dtype: string
  - name: v_super_concept
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': negative_subsumption
          '1': positive_subsumption
  - name: axiom
    dtype: string
  splits:
  - name: train
    num_bytes: 128737404
    num_examples: 768486
  - name: validation
    num_bytes: 16090857
    num_examples: 96060
  - name: test
    num_bytes: 16098373
    num_examples: 96062
  download_size: 28499028
  dataset_size: 160926634
- config_name: go-atomic-SI
  features:
  - name: v_sub_concept
    dtype: string
  - name: v_super_concept
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': negative_subsumption
          '1': positive_subsumption
  - name: axiom
    dtype: string
  splits:
  - name: train
    num_bytes: 152537233
    num_examples: 772870
  - name: validation
    num_bytes: 19060490
    num_examples: 96608
  - name: test
    num_bytes: 19069265
    num_examples: 96610
  download_size: 32379717
  dataset_size: 190666988
- config_name: bimnli
  features:
  - name: premise
    dtype: string
  - name: hypothesis
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': contradiction
          '1': entailment
  splits:
  - name: train
    num_bytes: 43363266
    num_examples: 235622
  - name: validation
    num_bytes: 4818648
    num_examples: 26180
  - name: test
    num_bytes: 2420273
    num_examples: 12906
  download_size: 19264134
  dataset_size: 50602187
- config_name: foodon-complex-SI
  features:
  - name: v_sub_concept
    dtype: string
  - name: v_super_concept
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': negative_subsumption
          '1': positive_subsumption
  - name: axiom
    dtype: string
  - name: anchor_axiom
    dtype: string
  splits:
  - name: train
    num_bytes: 2553731
    num_examples: 3754
  - name: validation
    num_bytes: 1271721
    num_examples: 1850
  - name: test
    num_bytes: 8926305
    num_examples: 13080
  download_size: 1064602
  dataset_size: 12751757
- config_name: go-complex-SI
  features:
  - name: v_sub_concept
    dtype: string
  - name: v_super_concept
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': negative_subsumption
          '1': positive_subsumption
  - name: axiom
    dtype: string
  - name: anchor_axiom
    dtype: string
  splits:
  - name: train
    num_bytes: 45328802
    num_examples: 72318
  - name: validation
    num_bytes: 5671713
    num_examples: 9040
  - name: test
    num_bytes: 5667069
    num_examples: 9040
  download_size: 5059364
  dataset_size: 56667584
---

# OntoLAMA: LAnguage Model Analysis for Ontology Subsumption Inference


### Dataset Summary

OntoLAMA is a set of language model (LM) probing datasets for ontology subsumption inference. 
The work follows the "LMs-as-KBs" literature but focuses on conceptualised knowledge extracted from formalised KBs 
such as the OWL ontologies. Specifically, the subsumption inference (SI) task is introduced and formulated in the
Natural Language Inference (NLI) style, where the sub-concept and the super-concept involved in a subsumption 
axiom are verbalised and fitted into a template to form the premise and hypothesis, respectively. 
The sampled axioms are verified through ontology reasoning. The SI task is further divided into Atomic SI and 
Complex SI where the former involves only atomic named concepts and the latter involves both atomic and complex concepts. 
Real-world ontologies of different scales and domains are used for constructing OntoLAMA and in total there are four Atomic 
SI datasets and two Complex SI datasets.

See dataset specifications: https://krr-oxford.github.io/DeepOnto/ontolama/

### Languages

The text in the dataset is in English, as used in the source ontologies. The associated BCP-47 code is `en`.

## Dataset Structure

### Data Instances


An example in the **Atomic SI** dataset created from the Gene Ontology (GO) is as follows:
```
{
    'v_sub_concept': 'ctpase activity',
    'v_super_concept': 'ribonucleoside triphosphate phosphatase activity',
    'label': 1,
    'axiom': 'SubClassOf(<http://purl.obolibrary.org/obo/GO_0043273> <http://purl.obolibrary.org/obo/GO_0017111>)'
}
```

An example in the **Complex SI** dataset created from the Food Ontology (FoodOn) is as follows:
```
{
    'v_sub_concept': 'ham and cheese sandwich that derives from some lima bean (whole)',
    'v_super_concept': 'lima bean substance',
    'label': 0,
    'axiom': 'SubClassOf(ObjectIntersectionOf(<http://purl.obolibrary.org/obo/FOODON_03307824> ObjectSomeValuesFrom(<http://purl.obolibrary.org/obo/RO_0001000> <http://purl.obolibrary.org/obo/FOODON_03302053>)) <http://purl.obolibrary.org/obo/FOODON_00002776>)',
    'anchor_axiom': 'EquivalentClasses(<http://purl.obolibrary.org/obo/FOODON_00002776> ObjectIntersectionOf(<http://purl.obolibrary.org/obo/FOODON_00002000> ObjectSomeValuesFrom(<http://purl.obolibrary.org/obo/RO_0001000> <http://purl.obolibrary.org/obo/FOODON_03302053>)) )'
}
```


An example in the **biMNLI** dataset created from the MNLI dataset is as follows:
```
{
    'premise': 'At the turn of the 19th century Los Angeles and Salt Lake City were among the burgeoning metropolises of the new American West.',
    'hypothesis': 'Salt Lake City was booming in the early 19th century.',
    'label': 1
}
```

### Data Fields

#### SI Data Fields
- `v_sub_concept`: verbalised sub-concept expression.
- `v_super_concept`: verbalised super-concept expression.
- `label`: a binary class label indicating whether two concepts really form a subsumption relationship (`1` means yes).
- `axiom`: a string representation of the original subsumption axiom which is useful for tracing back to the ontology.
- `anchor_axiom`: (for complex SI only) a string representation of the anchor equivalence axiom used for sampling the `axiom`.

#### biMNLI Data Fields
- `premise`: inheritated from the MNLI dataset.
- `hypothesis`: inheritated from the MNLI dataset.
- `label`: a binary class label indicating `contradiction` (`0`) or `entailment` (`1`).

### Data Splits

| Source     | #NamedConcepts | #EquivAxioms | #Dataset (Train/Dev/Test)                                              |
|------------|----------------|--------------|------------------------------------------------------------------------|
| Schema.org | 894            | -            | Atomic SI: 808/404/2,830                                               |
| DOID       | 11,157         | -            | Atomic SI: 90,500/11,312/11,314                                        |
| FoodOn     | 30,995         | 2,383        | Atomic SI: 768,486/96,060/96,062 <br /> Complex SI: 3,754/1,850/13,080 |
| GO         | 43,303         | 11,456       | Atomic SI: 772,870/96,608/96,610 <br /> Complex SI: 72,318/9,040/9,040 |
| MNLI       | -              | -            | biMNLI: 235,622/26,180/12,906                                          |


### Licensing Information

Creative Commons Attribution 4.0 International

### Citation Information

The relevant paper has been accepted at Findings of ACL 2023.

```
@inproceedings{he-etal-2023-language,
    title = "Language Model Analysis for Ontology Subsumption Inference",
    author = "He, Yuan  and
    Chen, Jiaoyan  and
    Jimenez-Ruiz, Ernesto  and
    Dong, Hang  and
    Horrocks, Ian",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.213",
    doi = "10.18653/v1/2023.findings-acl.213",
    pages = "3439--3453"
}
```