<commit_before><h1>End-to-end evaluation</h1>

Individual models can be evaluated as explained in [Training the different models of Grobid](Training-the-models-of-Grobid.md).

For an end-to-end evaluation, covering the whole extraction process from the parsing of PDF to the end result of the cascading of several CRF models, GROBID includes two possible evaluation progresses:

* against a set of [PubMed Central](http://www.ncbi.nlm.nih.gov/pmc) articles. For its publications, PubMed Central provides both PDF and fulltext XML files in the [NLM](http://www.ncbi.nlm.nih.gov/pmc/pmcdoc/tagging-guidelines/article/style.html) format. Keeping in mind some limits described bellow, it is possible to estimate the ability of Grobid to extract and normalize the content of the PDF documents for matching the quality of the NLM file. 

* against TEI documents produced by [Pub2TEI](http://github.com/kermitt2/Pub2TEI). Pub2TEI is a set of XSLT that permit to tranform various _native_ XML publishers (including Elsevier, Wiley, Springer, etc. XML formats) into a common TEI format. This TEI format can be used as groundtruth structure information for evaluating GROBID output, keeping in mind some limits described bellow. 

## Getting PubMedCentral gold-standard data 

We are currently evaluating GROBID using the `PMC_sample_1943` dataset compiled by <NAME>. The dataset is available at this [url](https://grobid.s3.amazonaws.com/PMC_sample_1943.zip) (around 1.5GB in size). The sample dataset contains 1943 articles from 1943 different journals corresponding to the latest publications from a 2011 snapshot. 

Any similar PubMed Central set of articles could normally be used, as long they follow the same directory structure: one directory per article containing at least the corresponding PDF file and the reference NLM file. 

We suppose in the following that the archive is decompressed under `PATH_TO_PMC/PMC_sample_1943/`.

## Getting publisher gold-standard data 

Some publishers release publications in XML format complementary to PDF in Open Access, allowing text mining (see for instance the dedicated subset of PMC publications). On contractual basis, it is possible to acquire native XML from mainstream publishers. Unfortunately, each publisher uses a different XML schema and covering all these formats would be a very time-consuming work. To ease the processing of these XML documents, the projet [Pub2TEI](http://github.com/kermitt2/Pub2TEI) permits to transform the native XML formats of a dozen mainstream publishers into a common TEI format which is the same as the output of GROBID. 

See [Pub2TEI](http://github.com/kermitt2/Pub2TEI) for converting native publisher XML into usable TEI. 

## Directory structure

For running the evalution, the tool assumes that the files are organised in a set of directory in the following way: 

* a root directory containing one sub-directory per article
* each article sub-directory containing at least the PDF version and a gold XML structured version of the article (in NLM format for PubMedCentral evaluation or in TEI format for the Pub2TEI-based evaluation)  


## Running and evaluating 

### PubMedCentral

Under ```grobid-trainer/```, the following command line is used to run and evaluate Grobid on the dataset:
```bash
> mvn compile exec:exec -PPubMedCentralEval -Dpmc=*PATH_TO_PMC/PMC_sample_1943* -Drun=1<commit_msg>Add explanation on the directory structure for end-to-end evaluation<commit_after><h1>End-to-end evaluation</h1>

Individual models can be evaluated as explained in [Training the different models of Grobid](Training-the-models-of-Grobid.md).

For an end-to-end evaluation, covering the whole extraction process from the parsing of PDF to the end result of the cascading of several CRF models, GROBID includes two possible evaluation progresses:

* ├── files
│   ├── image.png
│   └── more-files
│       ├── file1.txt
│       ├── file2.txt
│       └── file3.txt
└── some-file.mdagainst a set of [PubMed Central](http://www.ncbi.nlm.nih.gov/pmc) articles. For its publications, PubMed Central provides both PDF and fulltext XML files in the [NLM](http://www.ncbi.nlm.nih.gov/pmc/pmcdoc/tagging-guidelines/article/style.html) format. Keeping in mind some limits described bellow, it is possible to estimate the ability of Grobid to extract and normalize the content of the PDF documents for matching the quality of the NLM file. 

* against TEI documents produced by [Pub2TEI](http://github.com/kermitt2/Pub2TEI). Pub2TEI is a set of XSLT that permit to tranform various _native_ XML publishers (including Elsevier, Wiley, Springer, etc. XML formats) into a common TEI format. This TEI format can be used as groundtruth structure information for evaluating GROBID output, keeping in mind some limits described bellow. 

## Getting PubMedCentral gold-standard data 

We are currently evaluating GROBID using the `PMC_sample_1943` dataset compiled by <NAME>. The dataset is available at this [url](https://grobid.s3.amazonaws.com/PMC_sample_1943.zip) (around 1.5GB in size). The sample dataset contains 1943 articles from 1943 different journals corresponding to the latest publications from a 2011 snapshot. 

Any similar PubMed Central set of articles could normally be used, as long they follow the same directory structure: one directory per article containing at least the corresponding PDF file and the reference NLM file. 

We suppose in the following that the archive is decompressed under `PATH_TO_PMC/PMC_sample_1943/`.

## Getting publisher gold-standard data 

Some publishers release publications in XML format complementary to PDF in Open Access, allowing text mining (see for instance the dedicated subset of PMC publications). On contractual basis, it is possible to acquire native XML from mainstream publishers. Unfortunately, each publisher uses a different XML schema and covering all these formats would be a very time-consuming work. To ease the processing of these XML documents, the projet [Pub2TEI](http://github.com/kermitt2/Pub2TEI) permits to transform the native XML formats of a dozen mainstream publishers into a common TEI format which is the same as the output of GROBID. 

See [Pub2TEI](http://github.com/kermitt2/Pub2TEI) for converting native publisher XML into usable TEI. 

## Directory structure

For running the evaluation, the tool assumes that the files are organised in a set of directory in the following way: 

* a root directory containing one sub-directory per article

* each article sub-directory containing at least the PDF version and a gold XML structured version of the article (in NLM format for PubMedCentral evaluation or in TEI format for the Pub2TEI-based evaluation). See the diagram bellow - the name of the sub-directory and the files is free.  

* extension for files generated with [Pub2TEI](http://github.com/kermitt2/Pub2TEI) is `.bib2tei.tei.xml`. Extension for GROBID-generated TEI is `.fulltext.tei.xml`. Extension for NLM files is `.nxlm`. 

```
├── article1
│   ├── article1.pdf
│   └── article1.bib2tei.tei.xml
│   └── article1.nxml
│  
└── articles2
│   ├── article2.pdf
│   └── article2.bib2tei.tei.xml
│   └── article2.nxml
...
```

## Running and evaluating 

### PubMedCentral

Under ```grobid-trainer/```, the following command line is used to run and evaluate Grobid on the dataset:
```bash
> mvn compile exec:exec -PPubMedCentralEval -Dpmc=*PATH_TO_PMC/PMC_sample_1943* -Drun=1
{'fraction_non_alphanumeric': 0.06654120177443204, 'fraction_numerical': 0.014383653716897432, 'mean_word_length': 4.662100456621005, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '788298', 'n_tokens_mistral': 2106, 'n_tokens_neox': 1945, 'n_words': 978}