TITLE: Commander's Intent: A Dataset and Modeling Approach for Human-AI Task Specification in Strategic Play 

ABSTRACT: Effective Human-AI teaming requires the ability to communicate the goals of
the team and constraints under which you need the agent to operate. Providing
the ability to specify the shared intent or operation criteria of the team can
enable an AI agent to perform its primary function while still being able to
cater to the specific desires of the current team. While significant work has
been conducted to instruct an agent to perform a task, via language or
demonstrations, prior work lacks a focus on building agents which can operate
within the parameters specified by a team. Worse yet, there is a dearth of
research pertaining to enabling humans to provide their specifications through
unstructured, naturalist language. In this paper, we propose the use of goals
and constraints as a scaffold to modulate and evaluate autonomous agents. We
contribute to this field by presenting a novel dataset, and an associated data
collection protocol, which maps language descriptions to goals and constraints
corresponding to specific strategies developed by human participants for the
board game Risk. Leveraging state-of-the-art language models and augmentation
procedures, we develop a machine learning framework which can be used to
identify goals and constraints from unstructured strategy descriptions. To
empirically validate our approach we conduct a human-subjects study to
establish a human-baseline for our dataset. Our results show that our machine
learning architecture is better able to interpret unstructured language
descriptions into strategy specifications than human raters tasked with
performing the same machine translation task (F(1,272.53) = 17.025, p < 0.001).
Effective Human-AI teaming requires the ability to communicate the goals of
the team and constraints under which you need the agent to operate. Providing
the ability to specify the shared intent or operation criteria of the team can
enable an AI agent to perform its primary function while still being able to
cater to the specific desires of the current team. While significant work has
been conducted to instruct an agent to perform a task, via language or
demonstrations, prior work lacks a focus on building agents which can operate
within the parameters specified by a team. Worse yet, there is a dearth of
research pertaining to enabling humans to provide their specifications through
unstructured, naturalist language. In this paper, we propose the use of goals
and constraints as a scaffold to modulate and evaluate autonomous agents. We
contribute to this field by presenting a novel dataset, and an associated data
collection protocol, which maps language descriptions to goals and constraints
corresponding to specific strategies developed by human participants for the
board game Risk. Leveraging state-of-the-art language models and augmentation
procedures, we develop a machine learning framework which can be used to
identify goals and constraints from unstructured strategy descriptions. To
empirically validate our approach we conduct a human-subjects study to
establish a human-baseline for our dataset. Our results show that our machine
learning architecture is better able to interpret unstructured language
descriptions into strategy specifications than human raters tasked with
performing the same machine translation task (F(1,272.53) = 17.025, p < 0.001).
Commander's Intent: A Dataset and Modeling Approach for Human-AI Task Specification in Strategic Play
http://arxiv.org/abs/2208.08374v1