With LLMs shifting their role from statistical modeling of language
to serving as general-purpose AI agents,
how should LLM evaluations change?
Arguably, a key ability of an AI agent
is to flexibly combine, as needed, the basic skills it has learned.
The capability to combine skills plays an important role in (human)  pedagogy
and also in a paper on emergence phenomena. This work introducesskill-mix,
a new evaluation to measure ability to combine skills.
Using a list ofskills the evaluator repeatedly picks random subsets ofskills
and asks the LLM to produce text combining that subset of skills.
Since the number of subsets grows like, for even modestthis evaluation will, with high probability,
require the LLM to produce text significantly different from any text in the training set.
The paper develops a methodology for
(a)  designing and administering such an evaluation, and
(b)  automatic grading (plus spot-checking by humans)  of the results using GPT-4 as well as the open LLaMA-2 70B model. Administering a version ofskill-mixto popular chatbots gave results that,
while generally in line with prior expectations, contained surprises.
Sizeable differences exist among model capabilities
that are not captured by their ranking on popular LLM leaderboards (“cramming for the leaderboard”) .
Furthermore, simple probability calculations indicate that
GPT-4’s reasonable performance onis suggestive of going beyond
“stochastic parrot” behavior, i.e., it combines skills in ways that it had not seen during training. We sketch how the methodology can lead to askill-mixbased eco-system of open evaluations for AI capabilities of future models.