model update

README.md

@@ -0,0 +1,176 @@
+---
+datasets:
+- tner/tweetner7
+metrics:
+- f1
+- precision
+- recall
+model-index:
+- name: tner/twitter-roberta-base-2019-90m-tweetner7-random
+  results:
+  - task:
+      name: Token Classification
+      type: token-classification
+    dataset:
+      name: tner/tweetner7/test_2021
+      type: tner/tweetner7/test_2021
+      args: tner/tweetner7/test_2021
+    metrics:
+    - name: F1
+      type: f1
+      value: 0.6329255975760296
+    - name: Precision
+      type: precision
+      value: 0.6147809025506867
+    - name: Recall
+      type: recall
+      value: 0.6521739130434783
+    - name: F1 (macro)
+      type: f1_macro
+      value: 0.5849737353611323
+    - name: Precision (macro)
+      type: precision_macro
+      value: 0.5655720751091778
+    - name: Recall (macro)
+      type: recall_macro
+      value: 0.6073811457896877
+    - name: F1 (entity span)
+      type: f1_entity_span
+      value: 0.7735817294203468
+    - name: Precision (entity span)
+      type: precision_entity_span
+      value: 0.7513625463265751
+    - name: Recall (entity span)
+      type: recall_entity_span
+      value: 0.7971550826876374
+  - task:
+      name: Token Classification
+      type: token-classification
+    dataset:
+      name: tner/tweetner7/test_2020
+      type: tner/tweetner7/test_2020
+      args: tner/tweetner7/test_2020
+    metrics:
+    - name: F1
+      type: f1
+      value: 0.6428571428571428
+    - name: Precision
+      type: precision
+      value: 0.666110183639399
+    - name: Recall
+      type: recall
+      value: 0.6211728074727556
+    - name: F1 (macro)
+      type: f1_macro
+      value: 0.6067120703105228
+    - name: Precision (macro)
+      type: precision_macro
+      value: 0.6269481984991956
+    - name: Recall (macro)
+      type: recall_macro
+      value: 0.5890178249768797
+    - name: F1 (entity span)
+      type: f1_entity_span
+      value: 0.7620837808807734
+    - name: Precision (entity span)
+      type: precision_entity_span
+      value: 0.7896494156928213
+    - name: Recall (entity span)
+      type: recall_entity_span
+      value: 0.736377789309808
+
+pipeline_tag: token-classification
+widget:
+- text: "Get the all-analog Classic Vinyl Edition of `Takin' Off` Album from {{@Herbie Hancock@}} via {{USERNAME}} link below: {{URL}}"
+  example_title: "NER Example 1"
+---
+# tner/twitter-roberta-base-2019-90m-tweetner7-random
+
+This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on the 
+[tner/tweetner7](https://huggingface.co/datasets/tner/tweetner7) dataset (`train_random` split).
+Model fine-tuning is done via [T-NER](https://github.com/asahi417/tner)'s hyper-parameter search (see the repository
+for more detail). It achieves the following results on the test set of 2021:
+- F1 (micro): 0.6329255975760296
+- Precision (micro): 0.6147809025506867
+- Recall (micro): 0.6521739130434783
+- F1 (macro): 0.5849737353611323
+- Precision (macro): 0.5655720751091778
+- Recall (macro): 0.6073811457896877
+
+
+
+The per-entity breakdown of the F1 score on the test set are below:
+- corporation: 0.5055837563451777
+- creative_work: 0.41676942046855736
+- event: 0.45696539485359355
+- group: 0.599078341013825
+- location: 0.6480218281036835
+- person: 0.8302235359320156
+- product: 0.6381738708110735 
+
+For F1 scores, the confidence interval is obtained by bootstrap as below:
+- F1 (micro): 
+    - 90%: [0.6241107966406728, 0.6420422564843195]
+    - 95%: [0.6227081381578177, 0.6435080538043557] 
+- F1 (macro): 
+    - 90%: [0.6241107966406728, 0.6420422564843195]
+    - 95%: [0.6227081381578177, 0.6435080538043557] 
+
+Full evaluation can be found at [metric file of NER](https://huggingface.co/tner/twitter-roberta-base-2019-90m-tweetner7-random/raw/main/eval/metric.json) 
+and [metric file of entity span](https://huggingface.co/tner/twitter-roberta-base-2019-90m-tweetner7-random/raw/main/eval/metric_span.json).
+
+### Usage
+This model can be used through the [tner library](https://github.com/asahi417/tner). Install the library via pip   
+```shell
+pip install tner
+```
+and activate model as below.
+```python
+from tner import TransformersNER
+model = TransformersNER("tner/twitter-roberta-base-2019-90m-tweetner7-random")
+model.predict(["Jacob Collier is a Grammy awarded English artist from London"])
+```
+It can be used via transformers library but it is not recommended as CRF layer is not supported at the moment.
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+ - dataset: ['tner/tweetner7']
+ - dataset_split: train_random
+ - dataset_name: None
+ - local_dataset: None
+ - model: cardiffnlp/twitter-roberta-base-2019-90m
+ - crf: True
+ - max_length: 128
+ - epoch: 30
+ - batch_size: 32
+ - lr: 0.0001
+ - random_seed: 0
+ - gradient_accumulation_steps: 1
+ - weight_decay: 1e-07
+ - lr_warmup_step_ratio: 0.3
+ - max_grad_norm: 1
+
+The full configuration can be found at [fine-tuning parameter file](https://huggingface.co/tner/twitter-roberta-base-2019-90m-tweetner7-random/raw/main/trainer_config.json).
+
+### Reference
+If you use any resource from T-NER, please consider to cite our [paper](https://aclanthology.org/2021.eacl-demos.7/).
+
+```
+
+@inproceedings{ushio-camacho-collados-2021-ner,
+    title = "{T}-{NER}: An All-Round Python Library for Transformer-based Named Entity Recognition",
+    author = "Ushio, Asahi  and
+      Camacho-Collados, Jose",
+    booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations",
+    month = apr,
+    year = "2021",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2021.eacl-demos.7",
+    doi = "10.18653/v1/2021.eacl-demos.7",
+    pages = "53--62",
+    abstract = "Language model (LM) pretraining has led to consistent improvements in many NLP downstream tasks, including named entity recognition (NER). In this paper, we present T-NER (Transformer-based Named Entity Recognition), a Python library for NER LM finetuning. In addition to its practical utility, T-NER facilitates the study and investigation of the cross-domain and cross-lingual generalization ability of LMs finetuned on NER. Our library also provides a web app where users can get model predictions interactively for arbitrary text, which facilitates qualitative model evaluation for non-expert programmers. We show the potential of the library by compiling nine public NER datasets into a unified format and evaluating the cross-domain and cross- lingual performance across the datasets. The results from our initial experiments show that in-domain performance is generally competitive across datasets. However, cross-domain generalization is challenging even with a large pretrained LM, which has nevertheless capacity to learn domain-specific features if fine- tuned on a combined dataset. To facilitate future research, we also release all our LM checkpoints via the Hugging Face model hub.",
+}
+
+```

eval/metric.json

@@ -1 +0,0 @@
-{"random.dev": {"micro/f1": 0.6430903155603918, "micro/f1_ci": {}, "micro/recall": 0.6310731446876668, "micro/precision": 0.6555740432612313, "macro/f1": 0.5957476328680672, "macro/f1_ci": {}, "macro/recall": 0.5839083840160486, "macro/precision": 0.6084750980060624, "per_entity_metric": {"corporation": {"f1": 0.5758354755784061, "f1_ci": {}, "precision": 0.5714285714285714, "recall": 0.5803108808290155}, "creative_work": {"f1": 0.44236760124610586, "f1_ci": {}, "precision": 0.44654088050314467, "recall": 0.4382716049382716}, "event": {"f1": 0.391578947368421, "f1_ci": {}, "precision": 0.4043478260869565, "recall": 0.3795918367346939}, "group": {"f1": 0.622093023255814, "f1_ci": {}, "precision": 0.6257309941520468, "recall": 0.6184971098265896}, "location": {"f1": 0.6346153846153847, "f1_ci": {}, "precision": 0.6644295302013423, "recall": 0.6073619631901841}, "person": {"f1": 0.8595988538681949, "f1_ci": {}, "precision": 0.8754863813229572, "recall": 0.8442776735459663}, "product": {"f1": 0.6441441441441441, "f1_ci": {}, "precision": 0.6713615023474179, "recall": 0.6190476190476191}}}, "2021.test": {"micro/f1": 0.6329255975760296, "micro/f1_ci": {"90": [0.6241107966406728, 0.6420422564843195], "95": [0.6227081381578177, 0.6435080538043557]}, "micro/recall": 0.6521739130434783, "micro/precision": 0.6147809025506867, "macro/f1": 0.5849737353611323, "macro/f1_ci": {"90": [0.5763837020363822, 0.5947111988264823], "95": [0.5742937353061618, 0.5962661166656056]}, "macro/recall": 0.6073811457896877, "macro/precision": 0.5655720751091778, "per_entity_metric": {"corporation": {"f1": 0.5055837563451777, "f1_ci": {"90": [0.48112880503144656, 0.5305129060228105], "95": [0.47521529459960105, 0.5351315949731495]}, "precision": 0.4654205607476635, "recall": 0.5533333333333333}, "creative_work": {"f1": 0.41676942046855736, "f1_ci": {"90": [0.38737414150104255, 0.4471280364372469], "95": [0.381117185936746, 0.4518568526740165]}, "precision": 0.3793490460157127, "recall": 0.46238030095759236}, "event": {"f1": 0.45696539485359355, "f1_ci": {"90": [0.43451377299237326, 0.48098433064395635], "95": [0.4312793014907832, 0.48593287418465386]}, "precision": 0.4458874458874459, "recall": 0.46860782529572337}, "group": {"f1": 0.599078341013825, "f1_ci": {"90": [0.5799007693202035, 0.6205978336770757], "95": [0.5763590616997798, 0.6235043929757479]}, "precision": 0.5986842105263158, "recall": 0.5994729907773386}, "location": {"f1": 0.6480218281036835, "f1_ci": {"90": [0.6205056231580423, 0.6749297649471397], "95": [0.6175627543763568, 0.6813266130099813]}, "precision": 0.6333333333333333, "recall": 0.6634078212290503}, "person": {"f1": 0.8302235359320156, "f1_ci": {"90": [0.8202314881837767, 0.8410008455396478], "95": [0.8177612076669796, 0.8430586034166829]}, "precision": 0.8319141058867087, "recall": 0.8285398230088495}, "product": {"f1": 0.6381738708110735, "f1_ci": {"90": [0.6169357700291059, 0.6600499792631749], "95": [0.6131003018520063, 0.664773006102066]}, "precision": 0.6044158233670653, "recall": 0.6759259259259259}}}, "2020.test": {"micro/f1": 0.6428571428571428, "micro/f1_ci": {"90": [0.6218825133568143, 0.6608444099298781], "95": [0.6185707589214373, 0.665192796014874]}, "micro/recall": 0.6211728074727556, "micro/precision": 0.666110183639399, "macro/f1": 0.6067120703105228, "macro/f1_ci": {"90": [0.5847083370483181, 0.6258826342571657], "95": [0.5815028092972286, 0.6293428524288749]}, "macro/recall": 0.5890178249768797, "macro/precision": 0.6269481984991956, "per_entity_metric": {"corporation": {"f1": 0.5684754521963824, "f1_ci": {"90": [0.51461143224149, 0.6213723432170033], "95": [0.5048933935276239, 0.6302796758302875]}, "precision": 0.5612244897959183, "recall": 0.5759162303664922}, "creative_work": {"f1": 0.4887640449438202, "f1_ci": {"90": [0.42513867449237497, 0.5449838989348527], "95": [0.4129533747193237, 0.5546218487394958]}, "precision": 0.4915254237288136, "recall": 0.4860335195530726}, "event": {"f1": 0.45364891518737677, "f1_ci": {"90": [0.4032586558044806, 0.50105253276257], "95": [0.3941893490609532, 0.5113654082404082]}, "precision": 0.47520661157024796, "recall": 0.4339622641509434}, "group": {"f1": 0.5502645502645503, "f1_ci": {"90": [0.49639891028353794, 0.5996135198323417], "95": [0.4862892256873603, 0.610242001349384]}, "precision": 0.609375, "recall": 0.5016077170418006}, "location": {"f1": 0.7174603174603175, "f1_ci": {"90": [0.66, 0.7673751141712922], "95": [0.6483867450304767, 0.7759562841530054]}, "precision": 0.7533333333333333, "recall": 0.6848484848484848}, "person": {"f1": 0.8229166666666667, "f1_ci": {"90": [0.7947803203661327, 0.8469858791026068], "95": [0.7899859677143346, 0.8515427066348928]}, "precision": 0.8525179856115108, "recall": 0.7953020134228188}, "product": {"f1": 0.6454545454545455, "f1_ci": {"90": [0.6, 0.687823236296879], "95": [0.590903203014602, 0.695656930834364]}, "precision": 0.6454545454545455, "recall": 0.6454545454545455}}}, "2021.test (span detection)": {"micro/f1": 0.7735817294203468, "micro/f1_ci": {}, "micro/recall": 0.7971550826876374, "micro/precision": 0.7513625463265751, "macro/f1": 0.7735817294203468, "macro/f1_ci": {}, "macro/recall": 0.7971550826876374, "macro/precision": 0.7513625463265751}, "2020.test (span detection)": {"micro/f1": 0.7620837808807734, "micro/f1_ci": {}, "micro/recall": 0.736377789309808, "micro/precision": 0.7896494156928213, "macro/f1": 0.7620837808807734, "macro/f1_ci": {}, "macro/recall": 0.736377789309808, "macro/precision": 0.7896494156928213}}

eval/metric.test_2020.json

@@ -0,0 +1 @@
+{"micro/f1": 0.6428571428571428, "micro/f1_ci": {"90": [0.6218825133568143, 0.6608444099298781], "95": [0.6185707589214373, 0.665192796014874]}, "micro/recall": 0.6211728074727556, "micro/precision": 0.666110183639399, "macro/f1": 0.6067120703105228, "macro/f1_ci": {"90": [0.5847083370483181, 0.6258826342571657], "95": [0.5815028092972286, 0.6293428524288749]}, "macro/recall": 0.5890178249768797, "macro/precision": 0.6269481984991956, "per_entity_metric": {"corporation": {"f1": 0.5684754521963824, "f1_ci": {"90": [0.51461143224149, 0.6213723432170033], "95": [0.5048933935276239, 0.6302796758302875]}, "precision": 0.5612244897959183, "recall": 0.5759162303664922}, "creative_work": {"f1": 0.4887640449438202, "f1_ci": {"90": [0.42513867449237497, 0.5449838989348527], "95": [0.4129533747193237, 0.5546218487394958]}, "precision": 0.4915254237288136, "recall": 0.4860335195530726}, "event": {"f1": 0.45364891518737677, "f1_ci": {"90": [0.4032586558044806, 0.50105253276257], "95": [0.3941893490609532, 0.5113654082404082]}, "precision": 0.47520661157024796, "recall": 0.4339622641509434}, "group": {"f1": 0.5502645502645503, "f1_ci": {"90": [0.49639891028353794, 0.5996135198323417], "95": [0.4862892256873603, 0.610242001349384]}, "precision": 0.609375, "recall": 0.5016077170418006}, "location": {"f1": 0.7174603174603175, "f1_ci": {"90": [0.66, 0.7673751141712922], "95": [0.6483867450304767, 0.7759562841530054]}, "precision": 0.7533333333333333, "recall": 0.6848484848484848}, "person": {"f1": 0.8229166666666667, "f1_ci": {"90": [0.7947803203661327, 0.8469858791026068], "95": [0.7899859677143346, 0.8515427066348928]}, "precision": 0.8525179856115108, "recall": 0.7953020134228188}, "product": {"f1": 0.6454545454545455, "f1_ci": {"90": [0.6, 0.687823236296879], "95": [0.590903203014602, 0.695656930834364]}, "precision": 0.6454545454545455, "recall": 0.6454545454545455}}}

eval/metric.test_2021.json

@@ -0,0 +1 @@
+{"micro/f1": 0.6329255975760296, "micro/f1_ci": {"90": [0.6241107966406728, 0.6420422564843195], "95": [0.6227081381578177, 0.6435080538043557]}, "micro/recall": 0.6521739130434783, "micro/precision": 0.6147809025506867, "macro/f1": 0.5849737353611323, "macro/f1_ci": {"90": [0.5763837020363822, 0.5947111988264823], "95": [0.5742937353061618, 0.5962661166656056]}, "macro/recall": 0.6073811457896877, "macro/precision": 0.5655720751091778, "per_entity_metric": {"corporation": {"f1": 0.5055837563451777, "f1_ci": {"90": [0.48112880503144656, 0.5305129060228105], "95": [0.47521529459960105, 0.5351315949731495]}, "precision": 0.4654205607476635, "recall": 0.5533333333333333}, "creative_work": {"f1": 0.41676942046855736, "f1_ci": {"90": [0.38737414150104255, 0.4471280364372469], "95": [0.381117185936746, 0.4518568526740165]}, "precision": 0.3793490460157127, "recall": 0.46238030095759236}, "event": {"f1": 0.45696539485359355, "f1_ci": {"90": [0.43451377299237326, 0.48098433064395635], "95": [0.4312793014907832, 0.48593287418465386]}, "precision": 0.4458874458874459, "recall": 0.46860782529572337}, "group": {"f1": 0.599078341013825, "f1_ci": {"90": [0.5799007693202035, 0.6205978336770757], "95": [0.5763590616997798, 0.6235043929757479]}, "precision": 0.5986842105263158, "recall": 0.5994729907773386}, "location": {"f1": 0.6480218281036835, "f1_ci": {"90": [0.6205056231580423, 0.6749297649471397], "95": [0.6175627543763568, 0.6813266130099813]}, "precision": 0.6333333333333333, "recall": 0.6634078212290503}, "person": {"f1": 0.8302235359320156, "f1_ci": {"90": [0.8202314881837767, 0.8410008455396478], "95": [0.8177612076669796, 0.8430586034166829]}, "precision": 0.8319141058867087, "recall": 0.8285398230088495}, "product": {"f1": 0.6381738708110735, "f1_ci": {"90": [0.6169357700291059, 0.6600499792631749], "95": [0.6131003018520063, 0.664773006102066]}, "precision": 0.6044158233670653, "recall": 0.6759259259259259}}}

eval/metric_span.test_2020.json

@@ -0,0 +1 @@
+{"micro/f1": 0.7620837808807734, "micro/f1_ci": {}, "micro/recall": 0.736377789309808, "micro/precision": 0.7896494156928213, "macro/f1": 0.7620837808807734, "macro/f1_ci": {}, "macro/recall": 0.736377789309808, "macro/precision": 0.7896494156928213}

eval/metric_span.test_2021.json

@@ -0,0 +1 @@
+{"micro/f1": 0.7735817294203468, "micro/f1_ci": {}, "micro/recall": 0.7971550826876374, "micro/precision": 0.7513625463265751, "macro/f1": 0.7735817294203468, "macro/f1_ci": {}, "macro/recall": 0.7971550826876374, "macro/precision": 0.7513625463265751}

trainer_config.json

@@ -1 +1 @@
-{"data_split": "random.train", "model": "cardiffnlp/twitter-roberta-base-2019-90m", "crf": true, "max_length": 128, "epoch": 30, "batch_size": 32, "lr": 0.0001, "random_seed": 0, "gradient_accumulation_steps": 1, "weight_decay": 1e-07, "lr_warmup_step_ratio": 0.3, "max_grad_norm": 1}
+{"dataset": ["tner/tweetner7"], "dataset_split": "train_random", "dataset_name": null, "local_dataset": null, "model": "cardiffnlp/twitter-roberta-base-2019-90m", "crf": true, "max_length": 128, "epoch": 30, "batch_size": 32, "lr": 0.0001, "random_seed": 0, "gradient_accumulation_steps": 1, "weight_decay": 1e-07, "lr_warmup_step_ratio": 0.3, "max_grad_norm": 1}