task_categories:text2text-generation task_categories:text-classification task_categories:text-generation size_categories:100K<n<1M language:en license:apache-2.0 in-context-learning symbol-tuning icl meta-icl meta-learning flan long-input instruction-tuning instruct metaicl arxiv:2110.15943 arxiv:2305.08298 arxiv:2301.05948 region:us
---
license: apache-2.0
task_categories:
- text2text-generation
- text-classification
- text-generation
language:
- en
tags:
- in-context-learning
- symbol-tuning
- icl
- meta-icl
- meta-learning
- flan
- long-input
- instruction-tuning
- instruct
- metaicl
dataset_info:
  features:
  - name: task
    dtype: string
  - name: inputs
    dtype: string
  - name: targets
    dtype: string
  - name: symbols
    sequence: string
  splits:
  - name: validation
    num_bytes: 42218685.0
    num_examples: 14970
  - name: test
    num_bytes: 43453364.0
    num_examples: 16204
  - name: train
    num_bytes: 1303015298.0
    num_examples: 452367
  download_size: 727062369
  dataset_size: 1388687347.0
size_categories:
- 100K<n<1M
---
# Description
Few-shot prompting demonstrates that language models can learn in context even though they were not trained to do. However, explicitly learning to learn in context [meta-icl](https://arxiv.org/abs/2110.15943) leads to better results. With symbol tuning, labels are replaced with arbitrary symbols (e.g. foo/bar), which makes learning in context a key condition to learn the instructions

We implement *symbol tuning*, as presented in the [Symbol tuning improves in-context learning](https://arxiv.org/pdf/2305.08298.pdf) paper with tasksource classification datasets.

An input is a shuffled sequence of 4 positive and 4 negative examples showing a particular label (replaced with a symbol - a random word), followed by an example to label.

This is the largest symbol-tuning dataset to date, with 279 datasets. Symbol tuning improves in-context learning, which tends to be degraded by instruction tuning.

# Usage
We limit input size to 50_000 characters. This is well enough to challenge long range modeling. But be careful to remove examples that are too long or to truncate from left, otherwise some examples might be unsolvable, as the "question" are at the end of the examples.


```python
dataset = load_dataset('tasksource/icl-symbol-tuning-instruct')
# assuming 4 characters per token and 1000 tokens
dataset = dataset.filter(lambda x:len(x['inputs'])<1000*4)
```

## References:
Code: https://github.com/sileod/tasksource


```
@article{sileo2023tasksource,
  title={tasksource: Structured Dataset Preprocessing Annotations for Frictionless Extreme Multi-Task Learning and Evaluation},
  author={Sileo, Damien},
  url= {https://arxiv.org/abs/2301.05948},
  journal={arXiv preprint arXiv:2301.05948},
  year={2023}
}

@article{wei2023symbol,
  title={Symbol tuning improves in-context learning in language models},
  author={Wei, Jerry and Hou, Le and Lampinen, Andrew and Chen, Xiangning and Huang, Da and Tay, Yi and Chen, Xinyun and Lu, Yifeng and Zhou, Denny and Ma, Tengyu and others},
  journal={arXiv preprint arXiv:2305.08298},
  year={2023}
}
```