Hugging Face's logo

CATIE-AQ
/
NERmembert-base-4entities

Token Classification
Transformers
TensorBoard
Safetensors
French
camembert
Inference Endpoints
Model card Files Metrics Community
Edit model card

NERmembert-base-4entities

Model Description

We present NERmembert-base-4entities, which is a CamemBERT base fine-tuned for the Name Entity Recognition task for the French language on four French NER datasets for 4 entities (LOC, PER, ORG, MISC).
All these datasets were concatenated and cleaned into a single dataset that we called frenchNER_4entities.
There are a total of 384,773 rows, of which 328,757 are for training, 24,131 for validation and 31,885 for testing.
Our methodology is described in a blog post available in English or French.

Dataset

The dataset used is frenchNER_4entities, which represents ~385k sentences labeled in 4 categories:

Label Examples
PER "La Bruyère", "Gaspard de Coligny", "Wittgenstein"
ORG "UTBM", "American Airlines", "id Software"
LOC "République du Cap-Vert", "Créteil", "Bordeaux"
MISC "Wolfenstein 3D", "Révolution française", "Coupe du monde de football"

The distribution of the entities is as follows:


Splits
O
PER
LOC
ORG
MISC

train
7,539,692
307,144
286,746
127,089
799,494

validation
544,580
24,034
21,585
5,927
18,221

test
720,623
32,870
29,683
7,911
21,760

Evaluation results

The evaluation was carried out using the evaluate python package.

frenchNER_4entities

For space reasons, we show only the F1 of the different models. You can see the full results below the table.


Model
PER
LOC
ORG
MISC

Jean-Baptiste/camembert-ner
0.971
0.947
0.902
0.663

cmarkea/distilcamembert-base-ner
0.974
0.948
0.892
0.658

NERmembert-base-3entities
0.978
0.957
0.904
0

NERmembert-large-3entities
0.980
0.963
0.919
0

NERmembert-base-4entities (this model)
0.978
0.958
0.903
0.814

NERmembert-large-4entities
0.982
0.964
0.919
0.834
Full results

Model
Metrics
PER
LOC
ORG
MISC
O
Overall

Jean-Baptiste/camembert-ner
Precision
0.952
0.924
0.870
0.845
0.986
0.976

Recall
0.990
0.972
0.938
0.546
0.992
0.976
F1
0.971
0.947
0.902
0.663
0.989
0.976

cmarkea/distilcamembert-base-ner
Precision
0.962
0.933
0.857
0.830
0.985
0.976

Recall
0.987
0.963
0.930
0.545
0.993
0.976
F1
0.974
0.948
0.892
0.658
0.989
0.976

NERmembert-base-3entities
Precision
0.973
0.955
0.886
0
X
X

Recall
0.983
0.960
0.923
0
X
X
F1
0.978
0.957
0.904
0
X
X

NERmembert-large-3entities
Precision
0.978
0.960
0.899
0
X
X

Recall
0.985
0.966
0.940
0
X
X
F1
0.980
0.963
0.919
0
X
X

NERmembert-base-4entities (this model)
Precision
0.973
0.951
0.888
0.850
0.993
0.984

Recall
0.983
0.964
0.918
0.781
0.993
0.984
F1
0.978
0.958
0.903
0.814
0.993
0.984

NERmembert-large-4entities
Precision
0.977
0.961
0.896
0.872
0.993
0.986

Recall
0.987
0.966
0.943
0.798
0.995
0.986
F1
0.982
0.964
0.919
0.834
0.994
0.986

In detail:

multiconer

For space reasons, we show only the F1 of the different models. You can see the full results below the table.


Model
PER
LOC
ORG
MISC

Jean-Baptiste/camembert-ner
0.940
0.761
0.723
0.560

cmarkea/distilcamembert-base-ner
0.921
0.748
0.694
0.530

NERmembert-base-3entities
0.960
0.887
0.877
0

NERmembert-large-3entities
0.965
0.902
0.896
0

NERmembert-base-4entities (this model)
0.960
0.890
0.867
0.852

NERmembert-large-4entities
0.969
0.919
0.904
0.864
Full results

Model
Metrics
PER
LOC
ORG
MISC
O
Overall

Jean-Baptiste/camembert-ner
Precision
0.908
0.717
0.753
0.620
0.936
0.889

Recall
0.975
0.811
0.696
0.511
0.938
0.889
F1
0.940
0.761
0.723
0.560
0.937
0.889

cmarkea/distilcamembert-base-ner
Precision
0.885
0.738
0.737
0.589
0.928
0.881

Recall
0.960
0.759
0.655
0.482
0.939
0.881
F1
0.921
0.748
0.694
0.530
0.934
0.881

NERmembert-base-3entities
Precision
0.957
0.894
0.876
0
X
X

Recall
0.962
0.880
0.878
0
X
X
F1
0.960
0.887
0.877
0
X
X

NERmembert-large-3entities
Precision
0.960
0.903
0.916
0
X
X

Recall
0.970
0.900
0.877
0
X
X
F1
0.965
0.902
0.896
0
X
X

NERmembert-base-4entities (this model)
Precision
0.954
0.893
0.851
0.849
0.979
0.954

Recall
0.967
0.887
0.883
0.855
0.974
0.954
F1
0.960
0.890
0.867
0.852
0.977
0.954

NERmembert-large-4entities
Precision
0.964
0.922
0.904
0.856
0.981
0.961

Recall
0.975
0.917
0.904
0.872
0.976
0.961
F1
0.969
0.919
0.904
0.864
0.978
0.961

multinerd

For space reasons, we show only the F1 of the different models. You can see the full results below the table.


Model
PER
LOC
ORG
MISC

Jean-Baptiste/camembert-ner
0.962
0.934
0.888
0.419

cmarkea/distilcamembert-base-ner
0.972
0.938
0.884
0.430

NERmembert-base-3entities
0.985
0.973
0.938
0

NERmembert-large-3entities
0.987
0.979
0.953
0

NERmembert-base-4entities (this model)
0.985
0.973
0.938
0.770

NERmembert-large-4entities
0.987
0.976
0.948
0.790
Full results

Model
Metrics
PER
LOC
ORG
MISC
O
Overall

Jean-Baptiste/camembert-ner
Precision
0.931
0.893
0.827
0.725
0.979
0.966

Recall
0.994
0.980
0.959
0.295
0.990
0.966
F1
0.962
0.934
0.888
0.419
0.984
0.966

cmarkea/distilcamembert-base-ner
Precision
0.954
0.908
0.817
0.705
0.977
0.967

Recall
0.991
0.969
0.963
0.310
0.990
0.967
F1
0.972
0.938
0.884
0.430
0.984
0.967

NERmembert-base-3entities
Precision
0.974
0.965
0.910
0
X
X

Recall
0.995
0.981
0.968
0
X
X
F1
0.985
0.973
0.938
0
X
X

NERmembert-large-3entities
Precision
0.979
0.970
0.927
0
X
X

Recall
0.996
0.987
0.980
0
X
X
F1
0.987
0.979
0.953
0
X
X

NERmembert-base-4entities (this model)
Precision
0.976
0.961
0.911
0.829
0.991
0.983

Recall
0.994
0.985
0.967
0.719
0.993
0.983
F1
0.985
0.973
0.938
0.770
0.992
0.983

NERmembert-large-4entities
Precision
0.979
0.967
0.922
0.852
0.991
0.985

Recall
0.996
0.986
0.974
0.736
0.994
0.985
F1
0.987
0.976
0.948
0.790
0.993
0.985

wikiner

For space reasons, we show only the F1 of the different models. You can see the full results below the table.


Model
PER
LOC
ORG
MISC

Jean-Baptiste/camembert-ner
0.986
0.966
0.938
0.938

cmarkea/distilcamembert-base-ner
0.983
0.964
0.925
0.926

NERmembert-base-3entities
0.970
0.945
0.878
0

NERmembert-large-3entities
0.972
0.953
0.893
0

NERmembert-base-4entities (this model)
0.970
0.945
0.876
0.872

NERmembert-large-4entities
0.975
0.953
0.896
0.893
Full results

Model
Metrics
PER
LOC
ORG
MISC
O
Overall

Jean-Baptiste/camembert-ner
Precision
0.986
0.962
0.925
0.943
0.998
0.992

Recall
0.987
0.969
0.951
0.933
0.997
0.992
F1
0.986
0.966
0.938
0.938
0.998
0.992

cmarkea/distilcamembert-base-ner
Precision
0.982
0.964
0.910
0.942
0.997
0.991

Recall
0.985
0.963
0.940
0.910
0.998
0.991
F1
0.983
0.964
0.925
0.926
0.997
0.991

NERmembert-base-3entities
Precision
0.971
0.947
0.866
0
X
X

Recall
0.969
0.943
0.891
0
X
X
F1
0.970
0.945
0.878
0
X
X

NERmembert-large-3entities
Precision
0.973
0.953
0.873
0
X
X

Recall
0.971
0.948
0.913
0
X
X
F1
0.972
0.953
0.893
0
X
X

NERmembert-base-4entities (this model)
Precision
0.970
0.944
0.872
0.878
0.996
0.986

Recall
0.969
0.947
0.880
0.866
0.996
0.986
F1
0.970
0.945
0.876
0.872
0.996
0.986

NERmembert-large-4entities
Precision
0.975
0.957
0.872
0.901
0.997
0.989

Recall
0.975
0.949
0.922
0.884
0.997
0.989
F1
0.975
0.953
0.896
0.893
0.997
0.989

Usage

Code 

from transformers import pipeline

ner = pipeline('token-classification', model='CATIE-AQ/NERmembert-base-4entities', tokenizer='CATIE-AQ/NERmembert-base-4entities', aggregation_strategy="simple")

results = ner(
"Le dévoilement du logo officiel des JO s'est déroulé le 21 octobre 2019 au Grand Rex. Ce nouvel emblème et cette nouvelle typographie ont été conçus par le designer Sylvain Boyer avec les agences Royalties & Ecobranding. Rond, il rassemble trois symboles : une médaille d'or, la flamme olympique et Marianne, symbolisée par un visage de femme mais privée de son bonnet phrygien caractéristique. La typographie dessinée fait référence à l'Art déco, mouvement artistique des années 1920, décennie pendant laquelle ont eu lieu pour la dernière fois les Jeux olympiques à Paris en 1924. Pour la première fois, ce logo sera unique pour les Jeux olympiques et les Jeux paralympiques."
)

print(results)

[{'entity_group': 'MISC', 'score': 0.9456432, 'word': 'JO', 'start': 36, 'end': 38},
{'entity_group': 'LOC', 'score': 0.9990527, 'word': 'Grand Rex', 'start': 75, 'end': 84},
{'entity_group': 'PER', 'score': 0.99884754, 'word': 'Sylvain Boyer', 'start': 165, 'end': 178},
{'entity_group': 'ORG', 'score': 0.99118334, 'word': 'Royalties & Ecobranding', 'start': 196, 'end': 219},
{'entity_group': 'PER', 'score': 0.9446552, 'word': 'Marianne', 'start': 299, 'end': 307},
{'entity_group': 'MISC', 'score': 0.97599506, 'word': 'Art déco', 'start': 438, 'end': 446},
{'entity_group': 'MISC', 'score': 0.99798834, 'word': 'Jeux olympiques', 'start': 550, 'end': 565},
{'entity_group': 'LOC', 'score': 0.7205312, 'word': 'Paris', 'start': 568, 'end': 573},
{'entity_group': 'MISC', 'score': 0.996698, 'word': 'Jeux olympiques', 'start': 635, 'end': 650},
{'entity_group': 'MISC', 'score': 0.9955608, 'word': 'Jeux paralympiques', 'start': 658, 'end': 676}]

Try it through Space

A Space has been created to test the model. It is available here.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 3

Training results

Training Loss Epoch Step Validation Loss Precision Recall F1 Accuracy
0.0407 1.0 41095 0.0547 0.9816 0.9816 0.9816 0.9816
0.0242 2.0 82190 0.0488 0.9843 0.9843 0.9843 0.9843
0.018 3.0 123285 0.0542 0.9844 0.9844 0.9844 0.9844

Framework versions

  • Transformers 4.36.2
  • Pytorch 2.1.2
  • Datasets 2.16.1
  • Tokenizers 0.15.0

Environmental Impact

Carbon emissions were estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). The hardware, runtime, cloud provider, and compute region were utilized to estimate the carbon impact.

  • Hardware Type: A100 PCIe 40/80GB
  • Hours used: 1h45min
  • Cloud Provider: Private Infrastructure
  • Carbon Efficiency (kg/kWh): 0.046 (estimated from electricitymaps for the day of January 4, 2024.)
  • Carbon Emitted (Power consumption x Time x Carbon produced based on location of power grid): 0.01 kg eq. CO2

Citations

NERmembert-base-4entities 

@misc {NERmembert2024,
    author       = { {BOURDOIS, Loïck} },  
    organization = { {Centre Aquitain des Technologies de l'Information et Electroniques} },  
    title        = { NERmembert-base-4entities },
    year         = 2024,
    url          = { https://huggingface.co/CATIE-AQ/NERmembert-base-4entities },
    doi          = { 10.57967/hf/1752 },
    publisher    = { Hugging Face }
}

multiconer 

@inproceedings{multiconer2-report,  
    title={{SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2)}},  
    author={Fetahu, Besnik and Kar, Sudipta and Chen, Zhiyu and Rokhlenko, Oleg and Malmasi, Shervin},  
    booktitle={Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)},  
    year={2023},  
    publisher={Association for Computational Linguistics}}

@article{multiconer2-data,  
    title={{MultiCoNER v2: a Large Multilingual dataset for Fine-grained and Noisy Named Entity Recognition}},  
    author={Fetahu, Besnik and Chen, Zhiyu and Kar, Sudipta and Rokhlenko, Oleg and Malmasi, Shervin},  
    year={2023}}

multinerd 

 @inproceedings{tedeschi-navigli-2022-multinerd,  
    title = "{M}ulti{NERD}: A Multilingual, Multi-Genre and Fine-Grained Dataset for Named Entity Recognition (and Disambiguation)",  
    author = "Tedeschi, Simone and  Navigli, Roberto",  
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",  
    month = jul,  
    year = "2022",  
    address = "Seattle, United States",  
    publisher = "Association for Computational Linguistics",  
    url = "https://aclanthology.org/2022.findings-naacl.60",  
    doi = "10.18653/v1/2022.findings-naacl.60",  
    pages = "801--812"}

pii-masking-200k 

@misc {ai4privacy_2023,  
    author = { {ai4Privacy} },  
    title = { pii-masking-200k (Revision 1d4c0a1) },  
    year = 2023,  
    url = { https://huggingface.co/datasets/ai4privacy/pii-masking-200k },  
    doi = { 10.57967/hf/1532 },  
    publisher = { Hugging Face }}

wikiann 

@inproceedings{rahimi-etal-2019-massively,  
    title = "Massively Multilingual Transfer for {NER}",  
    author = "Rahimi, Afshin and Li, Yuan and Cohn, Trevor",  
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",  
    month = jul,  
    year = "2019",  
    address = "Florence, Italy",  
    publisher = "Association for Computational Linguistics",  
    url = "https://www.aclweb.org/anthology/P19-1015",  
    pages = "151--164"}

wikiner 

@article{NOTHMAN2013151,  
    title = {Learning multilingual named entity recognition from Wikipedia},  
    journal = {Artificial Intelligence},  
    volume = {194},  
    pages = {151-175},  
    year = {2013},  
    note = {Artificial Intelligence, Wikipedia and Semi-Structured Resources},  
    issn = {0004-3702},  
    doi = {https://doi.org/10.1016/j.artint.2012.03.006},  
    url = {https://www.sciencedirect.com/science/article/pii/S0004370212000276},  
    author = {Joel Nothman and Nicky Ringland and Will Radford and Tara Murphy and James R. Curran}}

frenchNER_4entities 

@misc {frenchNER2024,  
    author       = { {BOURDOIS, Loïck} },  
    organization  = { {Centre Aquitain des Technologies de l'Information et Electroniques} },  
    title        = { frenchNER_4entities },  
    year         = 2024,  
    url          = { https://huggingface.co/CATIE-AQ/frenchNER_4entities },  
    doi          = { 10.57967/hf/1751 },  
    publisher    = { Hugging Face }  
}

CamemBERT 

@inproceedings{martin2020camembert,  
  title={CamemBERT: a Tasty French Language Model},  
  author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},  
  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},  
  year={2020}}

License

cc-by-4.0

Downloads last month
38
Safetensors
Model size
110M params
Tensor type
F32
·
Inference Examples
Token Classification
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for CATIE-AQ/NERmembert-base-4entities

Base model

almanach/camembert-base
Finetuned
(94)
this model

Dataset used to train CATIE-AQ/NERmembert-base-4entities

Space using CATIE-AQ/NERmembert-base-4entities 1

Collection including CATIE-AQ/NERmembert-base-4entities

Evaluation results

Metadata error: specify a dataset to view leaderboard