NERmemBERT2-3entities

Model Description

We present NERmemBERT2-3entities, which is a CamemBERT v2 base fine-tuned for the Name Entity Recognition task for the French language on five French NER datasets for 3 entities (LOC, PER, ORG).
All these datasets were concatenated and cleaned into a single dataset that we called frenchNER_3entities.
This represents a total of over 420,264 rows, of which 346,071 are for training, 32,951 for validation and 41,242 for testing.
Our methodology is described in a blog post available in English or French.

Evaluation results

frenchNER_3entities

For space reasons, we show only the F1 of the different models. You can see the full results below the table.

Model	Parameters	Context	PER	LOC	ORG
Jean-Baptiste/camembert-ner	110M	512 tokens	0.941	0.883	0.658
cmarkea/distilcamembert-base-ner	67.5M	512 tokens	0.942	0.882	0.647
NERmembert-base-4entities	110M	512 tokens	0.951	0.894	0.671
NERmembert-large-4entities	336M	512 tokens	0.958	0.901	0.685
NERmembert-base-3entities	110M	512 tokens	0.966	0.940	0.876
NERmembert2-3entities (this model)	111M	1024 tokens	0.967	0.942	0.875
NERmemberta-3entities	111M	1024 tokens	0.970	0.943	0.881
NERmembert-large-3entities	336M	512 tokens	0.969	0.947	0.890

The results of the 4-entity models on the 3-entity dataset are given for information only. They are not reported in the following.

Full results

Model	Metrics	PER	LOC	ORG	O	Overall
Jean-Baptiste/camembert-ner (110M)	Precision	0.918	0.860	0.831	0.992	0.974
	Recall	0.964	0.908	0.544	0.964	0.948
	F1	0.941	0.883	0.658	0.978	0.961
cmarkea/distilcamembert-base-ner (67.5M)	Precision	0.929	0.861	0.813	0.991	0.974
	Recall	0.956	0.905	0.956	0.965	0.948
	F1	0.942	0.882	0.647	0.978	0.961
NERmembert-base-3entities (110M)	Precision	0.961	0.935	0.877	0.995	0.986
	Recall	0.972	0.946	0.876	0.994	0.986
	F1	0.966	0.940	0.876	0.994	0.986
NERmembert2-3entities (111M) (this model)	Precision	0.964	0.935	0.872	0.995	0.985
	Recall	0.967	0.949	0.878	0.993	0.984
	F1	0.967	0.942	0.875	0.994	0.985
NERmemberta-3entities (111M)	Precision	0.966	0.934	0.880	0.995	0.985
	Recall	0.973	0.952	0.883	0.993	0.985
	F1	0.970	0.943	0.881	0.994	0.985
NERmembert-large-3entities (336M)	Precision	0.946	0.884	0.859	0.993	0.971
	Recall	0.955	0.904	0.550	0.993	0.971
	F1	0.951	0.894	0.671	0.988	0.971

In detail:

multiconer

For space reasons, we show only the F1 of the different models. You can see the full results below the table.

Model	PER	LOC	ORG
Jean-Baptiste/camembert-ner (110M)	0.940	0.761	0.723
cmarkea/distilcamembert-base-ner (67.5M)	0.921	0.748	0.694
NERmembert-base-3entities (110M)	0.960	0.887	0.876
NERmembert2-3entities (111M) (this model)	0.958	0.876	0.863
NERmemberta-3entities (111M)	0.964	0.865	0.859
NERmembert-large-3entities (336M)	0.965	0.902	0.896

Full results

Model	Metrics	PER	LOC	ORG	O	Overall
Jean-Baptiste/camembert-ner (110M)	Precision	0.908	0.717	0.753	0.987	0.947
	Recall	0.975	0.811	0.696	0.878	0.880
	F1	0.940	0.761	0.723	0.929	0.912
cmarkea/distilcamembert-base-ner (67.5M)	Precision	0.885	0.738	0.737	0.983	0.943
	Recall	0.960	0.759	0.655	0.882	0.877
	F1	0.921	0.748	0.694	0.930	0.909
NERmembert-base-3entities (110M)	Precision	0.957	0.894	0.876	0.986	0.972
	Recall	0.962	0.880	0.878	0.985	0.972
	F1	0.960	0.887	0.876	0.985	0.972
NERmembert2-3entities (111M) (this model)	Precision	0.951	0.906	0.853	0.984	0.967
	Recall	0.966	0.848	0.874	0.984	0.967
	F1	0.958	0.876	0.863	0.984	0.967
NERmemberta-3entities (111M)	Precision	0.962	0.859	0.862	0.985	0.970
	Recall	0.967	0.871	0.857	0.984	0.970
	F1	0.964	0.865	0.859	0.985	0.970
NERmembert-large-3entities (336M)	Precision	0.960	0.903	0.916	0.987	0.976
	Recall	0.969	0.900	0.877	0.987	0.976
	F1	0.965	0.902	0.896	0.987	0.976

multinerd

For space reasons, we show only the F1 of the different models. You can see the full results below the table.

Model	PER	LOC	ORG
Jean-Baptiste/camembert-ner (110M)	0.962	0.934	0.888
cmarkea/distilcamembert-base-ner (67.5M)	0.972	0.938	0.884
NERmembert-base-3entities (110M)	0.985	0.973	0.938
NERmembert2-3entities (111M) (this model)	0.985	0.972	0.933
NERmemberta-3entities (111M)	0.986	0.974	0.945
NERmembert-large-3entities (336M)	0.987	0.979	0.953

Full results

Model	Metrics	PER	LOC	ORG	O	Overall
Jean-Baptiste/camembert-ner (110M)	Precision	0.931	0.893	0.827	0.999	0.988
	Recall	0.994	0.980	0.959	0.973	0.974
	F1	0.962	0.934	0.888	0.986	0.981
cmarkea/distilcamembert-base-ner (67.5M)	Precision	0.954	0.908	0.817	0.999	0.990
	Recall	0.991	0.969	0.963	0.975	0.975
	F1	0.972	0.938	0.884	0.987	0.983
NERmembert-base-3entities (110M)	Precision	0.974	0.965	0.910	0.999	0.995
	Recall	0.995	0.981	0.968	0.996	0.995
	F1	0.985	0.973	0.938	0.998	0.995
NERmembert2-3entities (111M) (this model)	Precision	0.975	0.960	0.902	0.999	0.995
	Recall	0.995	0.985	0.967	0.995	0.995
	F1	0.985	0.972	0.933	0.997	0.995
NERmemberta-3entities (111M)	Precision	0.976	0.961	0.915	0.999	0.995
	Recall	0.997	0.987	0.976	0.996	0.995
	F1	0.986	0.974	0.945	0.997	0.995
NERmembert-large-3entities (336M)	Precision	0.979	0.970	0.927	0.999	0.996
	Recall	0.996	0.987	0.980	0.997	0.996
	F1	0.987	0.979	0.953	0.998	0.996

wikiner

For space reasons, we show only the F1 of the different models. You can see the full results below the table.

Model	PER	LOC	ORG
Jean-Baptiste/camembert-ner (110M)	0.986	0.966	0.938
cmarkea/distilcamembert-base-ner (67.5M)	0.983	0.964	0.925
NERmembert-base-3entities (110M)	0.969	0.945	0.878
NERmembert2-3entities (111M) (this model)	0.969	0.946	0.866
NERmemberta-3entities (111M)	0.971	0.948	0.885
NERmembert-large-3entities (336M)	0.972	0.950	0.893

Full results

Model	Metrics	PER	LOC	ORG	O	Overall
Jean-Baptiste/camembert-ner (110M)	Precision	0.986	0.962	0.925	0.999	0.994
	Recall	0.987	0.969	0.951	0.965	0.967
	F1	0.986	0.966	0.938	0.982	0.980
cmarkea/distilcamembert-base-ner (67.5M)	Precision	0.982	0.951	0.910	0.998	0.994
	Recall	0.985	0.963	0.940	0.966	0.967
	F1	0.983	0.964	0.925	0.982	0.80
NERmembert-base-3entities (110M)	Precision	0.971	0.947	0.866	0.994	0.989
	Recall	0.969	0.942	0.891	0.995	0.989
	F1	0.969	0.945	0.878	0.995	0.989
NERmembert2-3entities (111M) (this model)	Precision	0.971	0.946	0.863	0.994	0.988
	Recall	0.967	0.946	0.870	0.995	0.988
	F1	0.969	0.946	0.866	0.994	0.988
NERmemberta-3entities (111M)	Precision	0.972	0.946	0.865	0.995	0.987
	Recall	0.970	0.950	0.905	0.995	0.987
	F1	0.971	0.948	0.885	0.995	0.987
NERmembert-large-3entities (336M)	Precision	0.973	0.953	0.873	0.996	0.990
	Recall	0.990	0.948	0.913	0.995	0.990
	F1	0.972	0.950	0.893	0.996	0.990

wikiann

For space reasons, we show only the F1 of the different models. You can see the full results below the table.

Model	PER	LOC	ORG
Jean-Baptiste/camembert-ner (110M)	0.867	0.722	0.451
cmarkea/distilcamembert-base-ner (67.5M)	0.862	0.722	0.451
NERmembert-base-3entities (110M)	0.947	0.906	0.886
NERmembert2-3entities (111M) (this model)	0.950	0.911	0.910
NERmemberta-3entities (111M)	0.953	0.902	0.890
NERmembert-large-3entities (336M)	0.949	0.912	0.899

Full results

Model	Metrics	PER	LOC	ORG	O	Overall
Jean-Baptiste/camembert-ner (110M)	Precision	0.862	0.700	0.864	0.867	0.832
	Recall	0.871	0.746	0.305	0.950	0.772
	F1	0.867	0.722	0.451	0.867	0.801
cmarkea/distilcamembert-base-ner (67.5M)	Precision	0.862	0.700	0.864	0.867	0.832
	Recall	0.871	0.746	0.305	0.950	0.772
	F1	0.867	0.722	0.451	0.907	0.800
NERmembert-base-3entities (110M)	Precision	0.948	0.900	0.893	0.979	0.942
	Recall	0.946	0.911	0.878	0.982	0.942
	F1	0.947	0.906	0.886	0.980	0.942
NERmembert2-3entities (111M) (this model)	Precision	0.962	0.906	0.890	0.971	0.941
	Recall	0.938	0.917	0.884	0.982	0.941
	F1	0.950	0.911	0.887	0.976	0.941
NERmemberta-3entities (111M)	Precision	0.961	0.902	0.899	0.972	0.942
	Recall	0.946	0.918	0.881	0.982	0.942
	F1	0.953	0.902	0.890	0.977	0.942
NERmembert-large-3entities (336M)	Precision	0.958	0.917	0.897	0.980	0.948
	Recall	0.940	0.915	0.901	0.983	0.948
	F1	0.949	0.912	0.899	0.983	0.948

Usage

Code

from transformers import pipeline

ner = pipeline('token-classification', model='CATIE-AQ/NERmembert2-base-3entities', tokenizer='CATIE-AQ/NERmembert2-base-3entities', aggregation_strategy="simple")

result = ner(
"Le dévoilement du logo officiel des JO s'est déroulé le 21 octobre 2019 au Grand Rex. Ce nouvel emblème et cette nouvelle typographie ont été conçus par le designer Sylvain Boyer avec les agences Royalties & Ecobranding. Rond, il rassemble trois symboles : une médaille d'or, la flamme olympique et Marianne, symbolisée par un visage de femme mais privée de son bonnet phrygien caractéristique. La typographie dessinée fait référence à l'Art déco, mouvement artistique des années 1920, décennie pendant laquelle ont eu lieu pour la dernière fois les Jeux olympiques à Paris en 1924. Pour la première fois, ce logo sera unique pour les Jeux olympiques et les Jeux paralympiques."
)

print(result)

Try it through Space

A Space has been created to test the model. It is available here.

Environmental Impact

Carbon emissions were estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). The hardware, runtime, cloud provider, and compute region were utilized to estimate the carbon impact.

• Hardware Type: A100 PCIe 40/80GB
• Hours used: 1h51min
• Cloud Provider: Private Infrastructure
• Carbon Efficiency (kg/kWh): 0.055 (estimated from electricitymaps for the day of November 21, 2024.)
• Carbon Emitted (Power consumption x Time x Carbon produced based on location of power grid): 0.0255 kg eq. CO2

Citations

NERmemBERT2-3entities

@misc {NERmemberta2024,
    author       = { {BOURDOIS, Loïck} },  
    organization = { {Centre Aquitain des Technologies de l'Information et Electroniques} },  
    title        = { NERmemberta-3entities (Revision 989f2ee) },
    year         = 2024,
    url          = { https://huggingface.co/CATIE-AQ/NERmemberta-3entities },
    doi          = { 10.57967/hf/3640 },
    publisher    = { Hugging Face }
}

NERmemBERT

@misc {NERmembert2024,
    author       = { {BOURDOIS, Loïck} },  
    organization = { {Centre Aquitain des Technologies de l'Information et Electroniques} },  
    title        = { NERmembert-base-3entities },
    year         = 2024,
    url          = { https://huggingface.co/CATIE-AQ/NERmembert-base-3entities },
    doi          = { 10.57967/hf/1752 },
    publisher    = { Hugging Face }
}

CamemBERT

@inproceedings{martin2020camembert,  
  title={CamemBERT: a Tasty French Language Model},  
  author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},  
  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},  
  year={2020}}

CamemBERT 2.0

@misc{antoun2024camembert20smarterfrench,
      title={CamemBERT 2.0: A Smarter French Language Model Aged to Perfection}, 
      author={Wissam Antoun and Francis Kulumba and Rian Touchent and Éric de la Clergerie and Benoît Sagot and Djamé Seddah},
      year={2024},
      eprint={2411.08868},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.08868}, 
}

multiconer

@inproceedings{multiconer2-report,  
    title={{SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2)}},  
    author={Fetahu, Besnik and Kar, Sudipta and Chen, Zhiyu and Rokhlenko, Oleg and Malmasi, Shervin},  
    booktitle={Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)},  
    year={2023},  
    publisher={Association for Computational Linguistics}}

@article{multiconer2-data,  
    title={{MultiCoNER v2: a Large Multilingual dataset for Fine-grained and Noisy Named Entity Recognition}},  
    author={Fetahu, Besnik and Chen, Zhiyu and Kar, Sudipta and Rokhlenko, Oleg and Malmasi, Shervin},  
    year={2023}}

multinerd

@inproceedings{tedeschi-navigli-2022-multinerd,  
    title = "{M}ulti{NERD}: A Multilingual, Multi-Genre and Fine-Grained Dataset for Named Entity Recognition (and Disambiguation)",  
    author = "Tedeschi, Simone and  Navigli, Roberto",  
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",  
    month = jul,  
    year = "2022",  
    address = "Seattle, United States",  
    publisher = "Association for Computational Linguistics",  
    url = "https://aclanthology.org/2022.findings-naacl.60",  
    doi = "10.18653/v1/2022.findings-naacl.60",  
    pages = "801--812"}

pii-masking-200k

@misc {ai4privacy_2023,  
    author = { {ai4Privacy} },  
    title = { pii-masking-200k (Revision 1d4c0a1) },  
    year = 2023,  
    url = { https://huggingface.co/datasets/ai4privacy/pii-masking-200k },  
    doi = { 10.57967/hf/1532 },  
    publisher = { Hugging Face }}

wikiann

@inproceedings{rahimi-etal-2019-massively,  
    title = "Massively Multilingual Transfer for {NER}",  
    author = "Rahimi, Afshin and Li, Yuan and Cohn, Trevor",  
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",  
    month = jul,  
    year = "2019",  
    address = "Florence, Italy",  
    publisher = "Association for Computational Linguistics",  
    url = "https://www.aclweb.org/anthology/P19-1015",  
    pages = "151--164"}

wikiner

@article{NOTHMAN2013151,  
    title = {Learning multilingual named entity recognition from Wikipedia},  
    journal = {Artificial Intelligence},  
    volume = {194},  
    pages = {151-175},  
    year = {2013},  
    note = {Artificial Intelligence, Wikipedia and Semi-Structured Resources},  
    issn = {0004-3702},  
    doi = {https://doi.org/10.1016/j.artint.2012.03.006},  
    url = {https://www.sciencedirect.com/science/article/pii/S0004370212000276},  
    author = {Joel Nothman and Nicky Ringland and Will Radford and Tara Murphy and James R. Curran}}

frenchNER_3entities

@misc {frenchNER2024,  
    author       = { {BOURDOIS, Loïck} },  
    organization  = { {Centre Aquitain des Technologies de l'Information et Electroniques} },  
    title        = { frenchNER_3entities },  
    year         = 2024,  
    url          = { https://huggingface.co/CATIE-AQ/frenchNER_3entities },  
    doi          = { 10.57967/hf/1751 },  
    publisher    = { Hugging Face }  
}

License

MIT