task_categories:sentence-similarity task_categories:text-classification task_ids:natural-language-inference task_ids:semantic-similarity-scoring annotations_creators:expert-generated language_creators:expert-generated multilinguality:translation size_categories:10K<n<100K source_datasets:extended|sick language:ja language:en license:cc-by-sa-4.0 semantic-textual-similarity sts region:us
Japanese Sentences Involving Compositional Knowledge (JSICK) Dataset.
JSICK is the Japanese NLI and STS dataset by manually translating the English dataset SICK (Marelli et al., 2014) into Japanese.
We hope that our dataset will be useful in research for realizing more advanced models that are capable of appropriately performing multilingual compositional inference.
(from official website)
@article{yanaka-mineshima-2022-compositional,
    title = "Compositional Evaluation on {J}apanese Textual Entailment and Similarity",
    author = "Yanaka, Hitomi and Mineshima, Koji",
    journal = "Transactions of the Association for Computational Linguistics",
    volume = "10",
    year = "2022",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://aclanthology.org/2022.tacl-1.73",
    doi = "10.1162/tacl_a_00518",
    pages = "1266--1284",
}
---
annotations_creators:
- expert-generated
language:
- ja
- en
language_creators:
- expert-generated
license:
- cc-by-sa-4.0
multilinguality:
- translation
pretty_name: JSICK
size_categories:
- 10K<n<100K
source_datasets:
- extended|sick
tags:
- semantic-textual-similarity
- sts
task_categories:
- sentence-similarity
- text-classification
task_ids:
- natural-language-inference
- semantic-similarity-scoring
---

# Dataset Card for JSICK

## Table of Contents
- [Dataset Card for JSICK](#dataset-card-for-jsick)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
      - [Japanese Sentences Involving Compositional Knowledge (JSICK) Dataset.](#japanese-sentences-involving-compositional-knowledge-jsick-dataset)
      - [JSICK-stress Test set](#jsick-stress-test-set)
    - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
      - [base](#base)
      - [stress](#stress)
    - [Data Fields](#data-fields)
      - [base](#base-1)
      - [stress](#stress-1)
    - [Data Splits](#data-splits)
    - [Annotations](#annotations)
  - [Additional Information](#additional-information)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://github.com/verypluming/JSICK
- **Repository:** https://github.com/verypluming/JSICK
- **Paper:** https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00518/113850/Compositional-Evaluation-on-Japanese-Textual
- **Paper:** https://www.jstage.jst.go.jp/article/pjsai/JSAI2021/0/JSAI2021_4J3GS6f02/_pdf/-char/ja

### Dataset Summary

From official [GitHub](https://github.com/verypluming/JSICK):

#### Japanese Sentences Involving Compositional Knowledge (JSICK) Dataset.

JSICK is the Japanese NLI and STS dataset by manually translating the English dataset [SICK (Marelli et al., 2014)](https://aclanthology.org/L14-1314/) into Japanese.
We hope that our dataset will be useful in research for realizing more advanced models that are capable of appropriately performing multilingual compositional inference.


#### JSICK-stress Test set

The JSICK-stress test set is a dataset to investigate whether models capture word order and case particles in Japanese.
The JSICK-stress test set is provided by transforming syntactic structures of sentence pairs in JSICK, where we analyze whether models are attentive to word order and case particles to predict entailment labels and similarity scores.

The JSICK test set contains 1666, 797, and 1006 sentence pairs (A, B) whose premise sentences A (the column `sentence_A_Ja_origin`) include the basic word order involving
ga-o (nominative-accusative), ga-ni (nominative-dative), and ga-de (nominative-instrumental/locative) relations, respectively.

We provide the JSICK-stress test set by transforming syntactic structures of these pairs by the following three ways:
- `scrum_ga_o`: a scrambled pair, where the word order of premise sentences A is scrambled into o-ga, ni-ga, and de-ga order, respectively.
- `ex_ga_o`:  a rephrased pair, where the only case particles (ga, o, ni, de) in the premise A are swapped
- `del_ga_o`: a rephrased pair, where the only case particles (ga, o, ni) in the premise A are deleted


### Languages

The language data in JSICK is in Japanese and English.


## Dataset Structure


### Data Instances
When loading a specific configuration, users has to append a version dependent suffix:

```python
import datasets as ds

dataset: ds.DatasetDict = ds.load_dataset("hpprc/jsick")
print(dataset)
# DatasetDict({
#     train: Dataset({
#         features: ['id', 'premise', 'hypothesis', 'label', 'score', 'premise_en', 'hypothesis_en', 'label_en', 'score_en', 'corr_entailment_labelAB_En', 'corr_entailment_labelBA_En', 'image_ID', 'original_caption', 'semtag_short', 'semtag_long'],
#         num_rows: 4500
#     })
#     test: Dataset({
#         features: ['id', 'premise', 'hypothesis', 'label', 'score', 'premise_en', 'hypothesis_en', 'label_en', 'score_en', 'corr_entailment_labelAB_En', 'corr_entailment_labelBA_En', 'image_ID', 'original_caption', 'semtag_short', 'semtag_long'],
#         num_rows: 4927
#     })
# })

dataset: ds.DatasetDict = ds.load_dataset("hpprc/jsick", name="stress")
print(dataset)
# DatasetDict({
#     test: Dataset({
#         features: ['id', 'premise', 'hypothesis', 'label', 'score', 'sentence_A_Ja_origin', 'entailment_label_origin', 'relatedness_score_Ja_origin', 'rephrase_type', 'case_particles'],
#         num_rows: 900
#     })
# })
```


#### base

An example of looks as follows:

```json
{
    'id': 1,
    'premise': '子供たちのグループが庭で遊んでいて、後ろの方には年を取った男性が立っている',
    'hypothesis': '庭にいる男の子たちのグループが遊んでいて、男性が後ろの方に立っている',
    'label': 1, // (neutral)
    'score': 3.700000047683716,
    'premise_en': 'A group of kids is playing in a yard and an old man is standing in the background',
    'hypothesis_en': 'A group of boys in a yard is playing and a man is standing in the background',
    'label_en': 1, // (neutral)
    'score_en': 4.5,
    'corr_entailment_labelAB_En': 'nan',
    'corr_entailment_labelBA_En': 'nan',
    'image_ID': '3155657768_b83a7831e5.jpg',
    'original_caption': 'A group of children playing in a yard , a man in the background .',
    'semtag_short': 'nan',
    'semtag_long': 'nan',
}
```

#### stress

An example of looks as follows:

```json
{
    'id': '5818_de_d',
    'premise': '女性火の近くダンスをしている',
    'hypothesis': '火の近くでダンスをしている女性は一人もいない',
    'label': 2,  // (contradiction)
    'score': 4.0,
    'sentence_A_Ja_origin': '女性が火の近くでダンスをしている',
    'entailment_label_origin': 2,
    'relatedness_score_Ja_origin': 3.700000047683716,
    'rephrase_type': 'd',
    'case_particles': 'de'
}
```

### Data Fields

#### base

A version adopting the column names of a typical NLI dataset.

| Name                       | Description                                                                                                                               |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| id                         | The ids (the same with original SICK).                                                                                                    |
| premise                    | The first sentence in Japanese.                                                                                                           |
| hypothesis                 | The second sentence in Japanese.                                                                                                          |
| label                      | The entailment label in Japanese.                                                                                                         |
| score                      | The relatedness score in the range [1-5] in Japanese.                                                                                     |
| premise_en                 | The first sentence in English.                                                                                                            |
| hypothesis_en              | The second sentence in English.                                                                                                           |
| label_en                   | The original entailment label in English.                                                                                                 |
| score_en                   | The original relatedness score in the range [1-5] in English.                                                                             |
| semtag_short               | The linguistic phenomena tags in Japanese.                                                                                                |
| semtag_long                | The details of linguistic phenomena tags in Japanese.                                                                                     |
| image_ID                   | The original image in [8K ImageFlickr dataset](https://www.kaggle.com/datasets/adityajn105/flickr8k).                                     |
| original_caption           | The original caption in [8K ImageFlickr dataset](https://www.kaggle.com/datasets/adityajn105/flickr8k).                                   |
| corr_entailment_labelAB_En | The corrected entailment label from A to B in English by [(Karouli et al., 2017)](http://vcvpaiva.github.io/includes/pubs/2017-iwcs.pdf). |
| corr_entailment_labelBA_En | The corrected entailment label from B to A in English by [(Karouli et al., 2017)](http://vcvpaiva.github.io/includes/pubs/2017-iwcs.pdf). |


#### stress

| Name                        | Description                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------- |
| id                          | Ids (the same with original SICK).                                                                |
| premise                     | The first sentence in Japanese.                                                                   |
| hypothesis                  | The second sentence in Japanese.                                                                  |
| label                       | The entailment label in Japanese                                                                  |
| score                       | The relatedness score in the range [1-5] in Japanese.                                             |
| sentence_A_Ja_origin        | The original premise sentences A from the JSICK test set.                                         |
| entailment_label_origin     | The original entailment labels.                                                                   |
| relatedness_score_Ja_origin | The original relatedness scores.                                                                  |
| rephrase_type               | The type of transformation applied to the syntactic structures of the sentence pairs.             |
| case_particles              | The grammatical particles in Japanese that indicate the function or role of a noun in a sentence. |


### Data Splits

| name            | train | validation |  test |
| --------------- | ----: | ---------: | ----: |
| base            | 4,500 |            | 4,927 |
| original        | 4,500 |            | 4,927 |
| stress          |       |            |   900 |
| stress-original |       |            |   900 |


### Annotations

To annotate the JSICK dataset, they used the crowdsourcing platform "Lancers" to re-annotate entailment labels and similarity scores for JSICK.
They had six native Japanese speakers as annotators, who were randomly selected from the platform.
The annotators were asked to fully understand the guidelines and provide the same labels as gold labels for ten test questions.

For entailment labels, they adopted annotations that were agreed upon by a majority vote as gold labels and checked whether the majority judgment vote was semantically valid for each example.
For similarity scores, they used the average of the annotation results as gold scores.
The raw annotations with the JSICK dataset are [publicly available](https://github.com/verypluming/JSICK/blob/main/jsick/jsick-all-annotations.tsv).
The average annotation time was 1 minute per pair, and Krippendorff's alpha for the entailment labels was 0.65.


## Additional Information

- [verypluming/JSICK](https://github.com/verypluming/JSICK)
- [Compositional Evaluation on Japanese Textual Entailment and Similarity](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00518/113850/Compositional-Evaluation-on-Japanese-Textual)
- [JSICK: 日本語構成的推論・類似度データセットの構築](https://www.jstage.jst.go.jp/article/pjsai/JSAI2021/0/JSAI2021_4J3GS6f02/_article/-char/ja)

### Licensing Information

CC BY-SA 4.0

### Citation Information

```bibtex
@article{yanaka-mineshima-2022-compositional,
    title = "Compositional Evaluation on {J}apanese Textual Entailment and Similarity",
    author = "Yanaka, Hitomi  and
      Mineshima, Koji",
    journal = "Transactions of the Association for Computational Linguistics",
    volume = "10",
    year = "2022",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://aclanthology.org/2022.tacl-1.73",
    doi = "10.1162/tacl_a_00518",
    pages = "1266--1284",
}

@article{谷中 瞳2021,
  title={JSICK: 日本語構成的推論・類似度データセットの構築},
  author={谷中 瞳 and 峯島 宏次},
  journal={人工知能学会全国大会論文集},
  volume={JSAI2021},
  number={ },
  pages={4J3GS6f02-4J3GS6f02},
  year={2021},
  doi={10.11517/pjsai.JSAI2021.0_4J3GS6f02}
}
```

### Contributions

Thanks to [Hitomi Yanaka](https://hitomiyanaka.mystrikingly.com/) and [Koji Mineshima](https://abelard.flet.keio.ac.jp/person/minesima/index-j.html) for creating this dataset.