Edit model card

relbert/roberta-large-semeval2012-average-prompt-e-nce-classification-conceptnet-validated

RelBERT fine-tuned from roberta-large on
relbert/semeval2012_relational_similarity. Fine-tuning is done via RelBERT library (see the repository for more detail). It achieves the following results on the relation understanding tasks:

  • Analogy Question (dataset, full result):
    • Accuracy on SAT (full): 0.5213903743315508
    • Accuracy on SAT: 0.5222551928783383
    • Accuracy on BATS: 0.6292384658143413
    • Accuracy on U2: 0.4649122807017544
    • Accuracy on U4: 0.5277777777777778
    • Accuracy on Google: 0.768
  • Lexical Relation Classification (dataset, full result):
    • Micro F1 score on BLESS: 0.9121591080307367
    • Micro F1 score on CogALexV: 0.8328638497652581
    • Micro F1 score on EVALution: 0.652762730227519
    • Micro F1 score on K&H+N: 0.9641093413090353
    • Micro F1 score on ROOT09: 0.8827953619554998
  • Relation Mapping (dataset, full result):
    • Accuracy on Relation Mapping: 0.8159126984126984

Usage

This model can be used through the relbert library. Install the library via pip

pip install relbert

and activate model as below.

from relbert import RelBERT
model = RelBERT("relbert/roberta-large-semeval2012-average-prompt-e-nce-classification-conceptnet-validated")
vector = model.get_embedding(['Tokyo', 'Japan'])  # shape of (1024, )

Training hyperparameters

The following hyperparameters were used during training:

  • model: roberta-large
  • max_length: 64
  • mode: average
  • data: relbert/semeval2012_relational_similarity
  • split: train
  • data_eval: relbert/conceptnet_high_confidence
  • split_eval: full
  • template_mode: manual
  • template: I wasn’t aware of this relationship, but I just read in the encyclopedia that is ’s
  • loss_function: nce_logout
  • classification_loss: True
  • temperature_nce_constant: 0.05
  • temperature_nce_rank: {'min': 0.01, 'max': 0.05, 'type': 'linear'}
  • epoch: 30
  • batch: 128
  • lr: 5e-06
  • lr_decay: False
  • lr_warmup: 1
  • weight_decay: 0
  • random_seed: 0
  • exclude_relation: None
  • exclude_relation_eval: None
  • n_sample: 640
  • gradient_accumulation: 8

The full configuration can be found at fine-tuning parameter file.

Reference

If you use any resource from RelBERT, please consider to cite our paper.


@inproceedings{ushio-etal-2021-distilling-relation-embeddings,
    title = "{D}istilling {R}elation {E}mbeddings from {P}re-trained {L}anguage {M}odels",
    author = "Ushio, Asahi  and
      Schockaert, Steven  and
      Camacho-Collados, Jose",
    booktitle = "EMNLP 2021",
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
}
Downloads last month
7
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Dataset used to train research-backup/roberta-large-semeval2012-average-prompt-e-nce-classification-conceptnet-validated

Evaluation results