task_categories:token-classification size_categories:10K<n<100K language:en license:cc-by-4.0 PIE idioms region:us
---
license: cc-by-4.0
dataset_info:
  features:
  - name: idiom
    dtype: string
  - name: is_pie
    dtype: bool
  - name: tokens
    sequence: string
  - name: ner_tags
    sequence:
      class_label:
        names:
          '0': O
          '1': B-PIE
          '2': I-PIE
  splits:
  - name: train
    num_bytes: 82950018
    num_examples: 46090
  - name: validation
    num_bytes: 10420303
    num_examples: 5761
  - name: test
    num_bytes: 10376839
    num_examples: 5762
  download_size: 19258913
  dataset_size: 103747160
task_categories:
- token-classification
language:
- en
tags:
- PIE
- idioms
size_categories:
- 10K<n<100K
pretty_name: Corpus of potentially idiomatic expressions (PIEs)
---
# Dataset Card for PIEs corpus

### Dataset Summary

This corpus is a collection of 57170 potentially idiomatic expressions (PIEs) based on the British National Corpus, prepaired for NER task.
Each of the objects is comes with a contextual set of tokens, BIO tags and boolean label.
The data sources are:
* [MAGPIE corpus](https://github.com/hslh/magpie-corpus)
* [PIE corpus](https://github.com/hslh/pie-annotation)

Detailed data preparation pipeline can be found [here](https://github.com/Gooogr/Idioms_spotter)
### Supported Tasks and Leaderboards

Token classification (NER)

### Languages

English

## Dataset Structure

### Data Instances
For each instance there is a string with target idiom, tokenized by word text with context of idiom usage, corresponded BIO tags 
and boolean label `is_pie`. This tag determines whether or not a collocation is considered an idiom in a given context. 
For a PIE dataset the choice was determined by the original PIE_label. For MAGPIE a threshold of 0.75 confidence coefficient was chosen.

An example from the train set looks like the following:
```
{'idiom': "go public"
'is_pie': True
'tokens': [ "Private", "dealers", "in", "the", "States", "go", "public" ]
'ner_tags': [ 0, 0, 0, 0, 0, 1, 2 ]
}
```
Where NER tags is {0: 'O', 1: 'B-PIE', 2: 'I-PIE'}

### Data Fields

* idiom: a string containg original PIE
* is_pie: a boolean label determining whether a PIE can be considered an idiom in a given context
* tokens: sequence of word tkenized string with PIE usage context
* ner_tags: corresponded BIO tags for word tokens

### Data Splits

The SNLI dataset has 3 splits: _train_, _validation_, and _test_. 

| Dataset Split | Number of Instances in Split |
| ------------- |----------------------------- |
| Train         | 45,736                       |
| Validation    | 5,717                        |
| Test          | 5,717                        |

## Dataset Creation

### Source Data

#### Initial Data Collection and Normalization

* [MAGPIE corpus](https://github.com/hslh/magpie-corpus)
* [PIE English corpus](https://github.com/hslh/pie-annotation)


## Additional Information

### Licensing Information

Corpus and it's sources are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

### Citation Information
[PIE Corpus](https://github.com/hslh/pie-annotation) (Haagsma, H. (Creator), Bos, J. (Contributor), Plank, B. (Contributor), University of Groningen.)<br>
[MAGPIE: A Large Corpus of Potentially Idiomatic Expressions](https://aclanthology.org/2020.lrec-1.35) (Haagsma et al., LREC 2020)