task_categories:text-generation task_categories:text2text-generation size_categories:10K<n<100K source_datasets:tatsu-lab/alpaca language:en license:cc-by-4.0 alpaca instruction generation region:us
---
license: cc-by-4.0
source_datasets: tatsu-lab/alpaca
task_categories:
- text-generation
- text2text-generation
language:
- en
tags:
- alpaca
- instruction generation
size_categories:
- 10K<n<100K
---

# 

The [`tatsu-lab/alpaca` dataset](https://huggingface.co/datasets/tatsu-lab/alpaca) was split into train/test/val with the goal of training text-to-text generation models to generate instruction prompts corresponding to arbitrary text.
To do this, you would use

- `output` as **the text2text model** input column
- `instruction` as the text2text model target/output column

## modifications & filtering 

Rows that used the column `input` in the original dataset, and rows where the `output` column contains less than 8 words were dropped.

Link to [function used to filter](https://gist.github.com/pszemraj/3633acb0cf3288d49b7bee550e756839) the original dataset after splitting
- The filter_dataset function reads datasets, counts tokens in specified columns, filters rows based on a minimum number of tokens, drops specified columns and/or rows with non-NaN values, and saves the modified datasets to a new directory. It returns summary statistics of the modified records.

## dataset info


Output of loading the dataset:

```python
DatasetDict({
    train: Dataset({
        features: ['instruction', 'output'],
        num_rows: 23167
    })
    test: Dataset({
        features: ['instruction', 'output'],
        num_rows: 2822
    })
    validation: Dataset({
        features: ['instruction', 'output'],
        num_rows: 2866
    })
})
```


## token counts in the `output` column


t5 

![output-tokens](https://i.imgur.com/cOCeBmA.png)


bart-base 

![output-tokens](https://i.imgur.com/y5CZiMa.png)


---