@inproceedings{
brunet2023iclmarkup,
title={{ICL}-Markup: Structuring In-Context Learning using Soft-Token Tags},
author={Marc-Etienne Brunet and Ashton Anderson and Richard Zemel},
booktitle={R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models},
year={2023},
url={https://openreview.net/forum?id=lfIXPclVHj}
}
Large pretrained language models (PLMs) can be rapidly adapted to a wide variety of tasks
via a text-to-text approach, 
where the instruction and input are fed to the model in natural language.
Combined with in-context learning (ICL), 
this paradigm is impressively flexible and powerful.
However, it also burdens engineers with an overwhelming amount of choices,
many of them arbitrary.
Inspired by markup languages like HTML, 
we contribute a method of using soft-token (a.k.a tunable token)
tags to compose prompt templates.
This approach reduces arbitrary decisions
and streamlines the application of ICL.
Our method is a form of meta-learning for ICL;
it learns these tags in advance 
during a parameter-efficient fine-tuning ``warm-up'' process.
The tags can subsequently be used in templates for ICL on new,
unseen tasks without any additional fine-tuning. 
Our experiments with this approach yield promising initial results,
improving PLM performance in important enterprise applications such as few-shot and open-world intent detection, 
as well as text classification in news and legal domains.
Our method is a form of meta-learning for ICL;
it learns these tags in advance 
during a parameter-efficient fine-tuning "warm-up" process.
The tags can subsequently be used in templates for ICL on new,
unseen tasks without any additional fine-tuning. 
Our experiments with this approach yield promising initial results.
Improving PLM performance in important enterprise applications such as few-shot and open-world intent detection, 
as well as text classification in a legal domain.
ICL-Markup: Structuring In-Context Learning using Soft-Token Tags
Marc-Etienne Brunet Ashton Anderson Richard Zemel