task_categories:text-classification size_categories:1K<n<10K language:en language:ja license:other code region:us
This dataset determines whether a GitHub repository description relates to Japanese natural language processing (NLP). The labels are categorized as "Relevant (1)" and "Not Relevant (0)".
---
license: other
task_categories:
- text-classification
language:
- en
- ja
tags:
- code
size_categories:
- 1K<n<10K
---


# Dataset overview

This dataset identifies whether a GitHub repository description pertains to Japanese natural language processing (NLP).
The labels are categorized as **"Relevant (1)" and "Not Relevant (0)"**.

Problem Setting:
- Training Data: Repository descriptions from before 2022
- Test Data: Repository descriptions from 2023
- Objective: To detect repositories related to Japanese NLP

Data Collection:
- Positive Examples: Repositories listed in "[awesome-japanese-nlp-resources](https://github.com/taishi-i/awesome-japanese-nlp-resources)" as of September 9, 2023
- Negative Examples: Collected from the GitHub API and visually confirmed
- Note: The annotation process is subjective

Dataset Features:
- Subjective labeling
- Mixed English and Japanese descriptions
- Imbalanced label distribution

**These dataset features mirror real-world challenges and are ideal for evaluating models.**
Based on GitHub's terms of service, please use this dataset for research purposes only.


# How to use this dataset

How to load in Python.

```python
from datasets import load_dataset

dataset = load_dataset("taishi-i/awesome-japanese-nlp-classification-dataset")
```

Details of the dataset.

```python
DatasetDict({
    train: Dataset({
        features: ['label', 'text', 'url', 'created_at'],
        num_rows: 5496
    })
    validation: Dataset({
        features: ['label', 'text', 'url', 'created_at'],
        num_rows: 400
    })
    test: Dataset({
        features: ['label', 'text', 'url', 'created_at'],
        num_rows: 856
    })
})
```

# Baseline

Baseline trained with [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased).
Please use the baseline model from [here](https://huggingface.co/taishi-i/awesome-japanese-nlp-classification-model).
The F1-score for label 1 is important for this task.

| Label        | Precision | Recall | F1-Score | Support |
|--------------|-----------|--------|----------|---------|
| 0            | 0.98      | 0.99   | 0.98     | 796     |
| 1            | 0.79      | 0.70   | **0.74** | 60      |
| Accuracy     |           |        | 0.97     | 856     |
| Macro Avg    | 0.89      | 0.84   | 0.86     | 856     |
| Weighted Avg | 0.96      | 0.97   | 0.97     | 856     |



# Dataset stats

Label distribution:

| Dataset    | Label 0 (%) | Label 1 (%) |
|------------|-------------|-------------|
| Train      | 92.59       | 7.41        |
| Validation | 95.75       | 4.25        |
| Test       | 92.99       | 7.01        |

Relevant sample:

```python
{
    "label": 1,
    "text": "JGLUE: Japanese General Language Understanding Evaluation for huggingface datasets",
    "url": "https://github.com/shunk031/huggingface-datasets_JGLUE",
    "created_at": "2023-02-25T04:33:03Z"
}
```

Not Relevant sample:

```python
{
    "label": 0,
    "text": "Official repository of FaceLit: Neural 3D Relightable Faces (CVPR 2023)",
    "url": "https://github.com/apple/ml-facelit",
    "created_at": "2023-04-03T22:47:29Z"
}
```

Number of texts, average number of characters per text, minimum number of characters, maximum number of characters:

| Dataset    | Text Count | Average Length | Min Length | Max Length |
|------------|------------|----------------|------------|------------|
| Train      | 5496       | 58.05          | 2.0        | 609.0      |
| Validation | 400        | 54.33          | 8.0        | 226.0      |
| Test       | 856        | 58.85          | 3.0        | 341.0      |

Proportion of text languages:

| Dataset    | English (%) | Japanese (%) |
|------------|-------------|--------------|
| Train      | 89.34       | 10.66        |
| Validation | 82.00       | 18.00        |
| Test       | 83.18       | 16.82        |

Time range:

| Dataset | Start Date                | End Date                  |
|---------|---------------------------|---------------------------|
| Train   | 2008-02-11 22:55:26+00:00 | 2022-09-30 19:45:09+00:00 |
| Validation | 2022-10-01 06:02:56+00:00 | 2022-12-31 12:12:41+00:00 |
| Test | 2023-01-01 06:15:03+00:00 | 2023-08-21 15:30:53+00:00 |


# License

We collect and publish this dataset under [GitHub Acceptable Use Policies - 7. Information Usage Restrictions](https://docs.github.com/en/site-policy/acceptable-use-policies/github-acceptable-use-policies#7-information-usage-restrictions) and [GitHub Terms of Service - H. API Terms](https://docs.github.com/en/site-policy/github-terms/github-terms-of-service#h-api-terms) for research purposes. This dataset should be used solely for research verification purposes. Adhering to GitHub's regulations is mandatory.
