Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments

Abstract: The capability of a reinforcement learning (RL) agent heavily depends on the diversity of the learning scenarios generated by the environment. Generation of diverse realistic scenarios is challenging for real-time strategy (RTS) environments. The RTS environments are characterized by intelligent entities/non-RL agents cooperating and competing with the RL agents with large state and action spaces over a long period of time, resulting in an infinite space of feasible, but not necessarily realistic, scenarios involving complex interaction among different RL and non-RL agents. Yet, most of the existing simulators rely on randomly generating the environments based on predefined settings/layouts and offer limited flexibility and control over the environment dynamics for researchers to generate diverse, realistic scenarios as per their demand. To address this issue, for the first time, we formally introduce the benefits of adopting an existing formal scenario specification language, SCENIC, to assist researchers to model and generate diverse scenarios in an RTS environment in a flexible, systematic, and programmatic manner. To showcase the benefits, we interfaced SCENIC to an existing RTS environment Google Research Football(GRF) simulator and introduced a benchmark consisting of 32 realistic scenarios, encoded in SCENIC, to train RL agents and testing their generalization capabilities. We also show how researchers/RL practitioners can incorporate their domain knowledge to expedite the training process by intuitively modeling stochastic programmatic policies with SCENIC.

\section{Submission of papers to NeurIPS 2021}

Please read the instructions below carefully and follow them faithfully.

\subsection{Style}

Papers to be submitted to NeurIPS 2021 must be prepared according to the
instructions presented here. Papers may only be up to {\bf nine} pages long,
including figures. Additional pages \emph{containing only acknowledgments and
references} are allowed. Papers that exceed the page limit will not be
reviewed, or in any other way considered for presentation at the conference.

The margins in 2021 are the same as those in 2007, which allow for $\sim$$15\%$
more words in the paper compared to earlier years.

Authors are required to use the NeurIPS \LaTeX{} style files obtainable at the
NeurIPS website as indicated below. Please make sure you use the current files
and not previous versions. Tweaking the style files may be grounds for
rejection.

\subsection{Retrieval of style files}

The style files for NeurIPS and other conference information are available on
the World Wide Web at
\begin{center}
  \url{http://www.neurips.cc/}
\end{center}
The file \verb+neurips_2021.pdf+ contains these instructions and illustrates the
various formatting requirements your NeurIPS paper must satisfy.

The only supported style file for NeurIPS 2021 is \verb+neurips_2021.sty+,
rewritten for \LaTeXe{}.  \textbf{Previous style files for \LaTeX{} 2.09,
  Microsoft Word, and RTF are no longer supported!}

The \LaTeX{} style file contains three optional arguments: \verb+final+, which
creates a camera-ready copy, \verb+preprint+, which creates a preprint for
submission to, e.g., arXiv, and \verb+nonatbib+, which will not load the
\verb+natbib+ package for you in case of package clash.

\paragraph{Preprint option}
If you wish to post a preprint of your work online, e.g., on arXiv, using the
NeurIPS style, please use the \verb+preprint+ option. This will create a
nonanonymized version of your work with the text ``Preprint. Work in progress.''
in the footer. This version may be distributed as you see fit. Please \textbf{do
  not} use the \verb+final+ option, which should \textbf{only} be used for
papers accepted to NeurIPS.

At submission time, please omit the \verb+final+ and \verb+preprint+
options. This will anonymize your submission and add line numbers to aid
review. Please do \emph{not} refer to these line numbers in your paper as they
will be removed during generation of camera-ready copies.

The file \verb+neurips_2021.tex+ may be used as a ``shell'' for writing your
paper. All you have to do is replace the author, title, abstract, and text of
the paper with your own.

The formatting instructions contained in these style files are summarized in
Sections \ref{gen_inst}, \ref{headings}, and \ref{others} below.

\section{General formatting instructions}
\label{gen_inst}

The text must be confined within a rectangle 5.5~inches (33~picas) wide and
9~inches (54~picas) long. The left margin is 1.5~inch (9~picas).  Use 10~point
type with a vertical spacing (leading) of 11~points.  Times New Roman is the
preferred typeface throughout, and will be selected for you by default.
Paragraphs are separated by \nicefrac{1}{2}~line space (5.5 points), with no
indentation.

The paper title should be 17~point, initial caps/lower case, bold, centered
between two horizontal rules. The top rule should be 4~points thick and the
bottom rule should be 1~point thick. Allow \nicefrac{1}{4}~inch space above and
below the title to rules. All pages should start at 1~inch (6~picas) from the
top of the page.

For the final version, authors' names are set in boldface, and each name is
centered above the corresponding address. The lead author's name is to be listed
first (left-most), and the co-authors' names (if different address) are set to
follow. If there is only one co-author, list both author and co-author side by
side.

Please pay special attention to the instructions in Section \ref{others}
regarding figures, tables, acknowledgments, and references.

\section{Headings: first level}
\label{headings}

All headings should be lower case (except for first word and proper nouns),
flush left, and bold.

First-level headings should be in 12-point type.

\subsection{Headings: second level}

Second-level headings should be in 10-point type.

\subsubsection{Headings: third level}

Third-level headings should be in 10-point type.

\paragraph{Paragraphs}

There is also a \verb+\paragraph+ command available, which sets the heading in
bold, flush left, and inline with the text, with the heading followed by 1\,em
of space.

\section{Citations, figures, tables, references}
\label{others}

These instructions apply to everyone.

\subsection{Citations within the text}

The \verb+natbib+ package will be loaded for you by default.  Citations may be
author/year or numeric, as long as you maintain internal consistency.  As to the
format of the references themselves, any style is acceptable as long as it is
used consistently.

The documentation for \verb+natbib+ may be found at
\begin{center}
  \url{http://mirrors.ctan.org/macros/latex/contrib/natbib/natnotes.pdf}
\end{center}
Of note is the command \verb+\citet+, which produces citations appropriate for
use in inline text.  For example,
\begin{verbatim}
   \citet{hasselmo} investigated\dots
\end{verbatim}
produces
\begin{quote}
  Hasselmo, et al.\ (1995) investigated\dots
\end{quote}

If you wish to load the \verb+natbib+ package with options, you may add the
following before loading the \verb+neurips_2021+ package:
\begin{verbatim}
   \PassOptionsToPackage{options}{natbib}
\end{verbatim}

If \verb+natbib+ clashes with another package you load, you can add the optional
argument \verb+nonatbib+ when loading the style file:
\begin{verbatim}
   \usepackage[nonatbib]{neurips_2021}
\end{verbatim}

As submission is double blind, refer to your own published work in the third
person. That is, use ``In the previous work of Jones et al.\ [4],'' not ``In our
previous work [4].'' If you cite your other papers that are not widely available
(e.g., a journal paper under review), use anonymous author names in the
citation, e.g., an author of the form ``A.\ Anonymous.''

\subsection{Footnotes}

Footnotes should be used sparingly.  If you do require a footnote, indicate
footnotes with a number\footnote{Sample of the first footnote.} in the
text. Place the footnotes at the bottom of the page on which they appear.
Precede the footnote with a horizontal rule of 2~inches (12~picas).

Note that footnotes are properly typeset \emph{after} punctuation
marks.\footnote{As in this example.}

\subsection{Figures}

\begin{figure}
  \centering
  \fbox{\rule[-.5cm]{0cm}{4cm} \rule[-.5cm]{4cm}{0cm}}
  \caption{Sample figure caption.}
\end{figure}

All artwork must be neat, clean, and legible. Lines should be dark enough for
purposes of reproduction. The figure number and caption always appear after the
figure. Place one line space before the figure caption and one line space after
the figure. The figure caption should be lower case (except for first word and
proper nouns); figures are numbered consecutively.

You may use color figures.  However, it is best for the figure captions and the
paper body to be legible if the paper is printed in either black/white or in
color.

\subsection{Tables}

All tables must be centered, neat, clean and legible.  The table number and
title always appear before the table.  See Table~\ref{sample-table}.

Place one line space before the table title, one line space after the
table title, and one line space after the table. The table title must
be lower case (except for first word and proper nouns); tables are
numbered consecutively.

Note that publication-quality tables \emph{do not contain vertical rules.} We
strongly suggest the use of the \verb+booktabs+ package, which allows for
typesetting high-quality, professional tables:
\begin{center}
  \url{https://www.ctan.org/pkg/booktabs}
\end{center}
This package was used to typeset Table~\ref{sample-table}.

\begin{table}
  \caption{Sample table title}
  \label{sample-table}
  \centering
  \begin{tabular}{lll}
    \toprule
    \multicolumn{2}{c}{Part}                   \\
    \cmidrule(r){1-2}
    Name     & Description     & Size ($\mu$m) \\
    \midrule
    Dendrite & Input terminal  & $\sim$100     \\
    Axon     & Output terminal & $\sim$10      \\
    Soma     & Cell body       & up to $10^6$  \\
    \bottomrule
  \end{tabular}
\end{table}

\section{Final instructions}

Do not change any aspects of the formatting parameters in the style files.  In
particular, do not modify the width or length of the rectangle the text should
fit into, and do not change font sizes (except perhaps in the
\textbf{References} section; see below). Please note that pages should be
numbered.

\section{Preparing PDF files}

Please prepare submission files with paper size ``US Letter,'' and not, for
example, ``A4.''

Fonts were the main cause of problems in the past years. Your PDF file must only
contain Type 1 or Embedded TrueType fonts. Here are a few instructions to
achieve this.

\begin{itemize}

\item You should directly generate PDF files using \verb+pdflatex+.

\item You can check which fonts a PDF files uses.  In Acrobat Reader, select the
  menu Files$>$Document Properties$>$Fonts and select Show All Fonts. You can
  also use the program \verb+pdffonts+ which comes with \verb+xpdf+ and is
  available out-of-the-box on most Linux machines.

\item The IEEE has recommendations for generating PDF files whose fonts are also
  acceptable for NeurIPS. Please see
  \url{http://www.emfield.org/icuwb2010/downloads/IEEE-PDF-SpecV32.pdf}

\item \verb+xfig+ "patterned" shapes are implemented with bitmap fonts.  Use
  "solid" shapes instead.

\item The \verb+\bbold+ package almost always uses bitmap fonts.  You should use
  the equivalent AMS Fonts:
\begin{verbatim}
   \usepackage{amsfonts}
\end{verbatim}
followed by, e.g., \verb+\mathbb{R}+, \verb+\mathbb{N}+, or \verb+\mathbb{C}+
for $\mathbb{R}$, $\mathbb{N}$ or $\mathbb{C}$.  You can also use the following
workaround for reals, natural and complex:
\begin{verbatim}
   \newcommand{\RR}{I\!\!R}
   \newcommand{\Nat}{I\!\!N}
   \newcommand{\CC}{I\!\!\!\!C}
\end{verbatim}
Note that \verb+amsfonts+ is automatically loaded by the \verb+amssymb+ package.

\end{itemize}

If your file contains type 3 fonts or non embedded TrueType fonts, we will ask
you to fix it.

\subsection{Margins in \LaTeX{}}

Most of the margin problems come from figures positioned by hand using
\verb+\special+ or other commands. We suggest using the command
\verb+\includegraphics+ from the \verb+graphicx+ package. Always specify the
figure width as a multiple of the line width as in the example below:
\begin{verbatim}
   \usepackage[pdftex]{graphicx} ...
   \includegraphics[width=0.8\linewidth]{myfile.pdf}
\end{verbatim}
See Section 4.4 in the graphics bundle documentation
(\url{http://mirrors.ctan.org/macros/latex/required/graphics/grfguide.pdf})

A number of width problems arise when \LaTeX{} cannot properly hyphenate a
line. Please give LaTeX hyphenation hints using the \verb+\-+ command when
necessary.

\begin{ack}
Use unnumbered first level headings for the acknowledgments. All acknowledgments
go at the end of the paper before the list of references. Moreover, you are required to declare
funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work).
More information about this disclosure can be found at: \url{https://neurips.cc/Conferences/2021/PaperInformation/FundingDisclosure}.

Do {\bf not} include this section in the anonymized submission, only in the final paper. You can use the \texttt{ack} environment provided in the style file to autmoatically hide this section in the anonymized submission.
\end{ack}

\section*{References}

References follow the acknowledgments. Use unnumbered first-level heading for
the references. Any choice of citation style is acceptable as long as you are
consistent. It is permissible to reduce the font size to \verb+small+ (9 point)
when listing the references.
Note that the Reference section does not count towards the page limit.
\medskip

{
\small

[1] Alexander, J.A.\ \& Mozer, M.C.\ (1995) Template-based algorithms for
connectionist rule extraction. In G.\ Tesauro, D.S.\ Touretzky and T.K.\ Leen
(eds.), {\it Advances in Neural Information Processing Systems 7},
pp.\ 609--616. Cambridge, MA: MIT Press.

[2] Bower, J.M.\ \& Beeman, D.\ (1995) {\it The Book of GENESIS: Exploring
  Realistic Neural Models with the GEneral NEural SImulation System.}  New York:
TELOS/Springer--Verlag.

[3] Hasselmo, M.E., Schnell, E.\ \& Barkai, E.\ (1995) Dynamics of learning and
recall at excitatory recurrent synapses and cholinergic modulation in rat
hippocampal region CA3. {\it Journal of Neuroscience} {\bf 15}(7):5249-5262.
}

\section*{Checklist}

The checklist follows the references.  Please
read the checklist guidelines carefully for information on how to answer these
questions.  For each question, change the default \answerTODO{} to \answerYes{},
\answerNo{}, or \answerNA{}.  You are strongly encouraged to include a {\bf
justification to your answer}, either by referencing the appropriate section of
your paper or providing a brief inline description.  For example:
\begin{itemize}
  \item Did you include the license to the code and datasets? \answerYes{See Section~\ref{gen_inst}.}
  \item Did you include the license to the code and datasets? \answerNo{The code and the data are proprietary.}
  \item Did you include the license to the code and datasets? \answerNA{}
\end{itemize}
Please do not modify the questions and only use the provided macros for your
answers.  Note that the Checklist section does not count towards the page
limit.  In your paper, please delete this instructions block and only keep the
Checklist section heading above along with the questions/answers below.

\begin{enumerate}

\item For all authors...
\begin{enumerate}
  \item Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
    \answerTODO{}
  \item Did you describe the limitations of your work?
    \answerTODO{}
  \item Did you discuss any potential negative societal impacts of your work?
    \answerTODO{}
  \item Have you read the ethics review guidelines and ensured that your paper conforms to them?
    \answerTODO{}
\end{enumerate}

\item If you are including theoretical results...
\begin{enumerate}
  \item Did you state the full set of assumptions of all theoretical results?
    \answerTODO{}
	\item Did you include complete proofs of all theoretical results?
    \answerTODO{}
\end{enumerate}

\item If you ran experiments...
\begin{enumerate}
  \item Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
    \answerTODO{}
  \item Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
    \answerTODO{}
	\item Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
    \answerTODO{}
	\item Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
    \answerTODO{}
\end{enumerate}

\item If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
\begin{enumerate}
  \item If your work uses existing assets, did you cite the creators?
    \answerTODO{}
  \item Did you mention the license of the assets?
    \answerTODO{}
  \item Did you include any new assets either in the supplemental material or as a URL?
    \answerTODO{}
  \item Did you discuss whether and how consent was obtained from people whose data you're using/curating?
    \answerTODO{}
  \item Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
    \answerTODO{}
\end{enumerate}

\item If you used crowdsourcing or conducted research with human subjects...
\begin{enumerate}
  \item Did you include the full text of instructions given to participants and screenshots, if applicable?
    \answerTODO{}
  \item Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
    \answerTODO{}
  \item Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
    \answerTODO{}
\end{enumerate}

\end{enumerate}

\section{Introduction}

Deep reinforcement learning (RL) has emerged as a powerful method to solve a variety of sequential decision-making problems,
including board games~\cite{silver2017mastering,silver2018general},
video games~\cite{mnih2015human,vinyals2019grandmaster},
and robotic manipulation \cite{kalashnikov2018qt}.
These successes rely heavily on good simulation environments and benchmarks~\cite{bellemare2013arcade, brockman2016openai, cobbe2020leveraging, duan2016benchmarking,dmcontrol_old}.
However, regardless of a long history of RL benchmarks,
the existing RL environments/simulators often fail to generate realistic scenarios portraying a rich variation of environment dynamics.
This lack of diversity severely limits the ability of reinforcement learning, primarily resulting in lack of generalization: one of the biggest challenges in deep reinforcement learning~\cite{cobbe2019quantifying, cobbe2020leveraging,lee2019network,lee2020context}.
To be able to train robust agents that generalizes over different variations of the environment dynamics, we must train RL agents on scenarios that represent the desired variation.

In literature, several techniques have been adopted to generate a rich variation of learning scenarios~\cite{cobbe2019quantifying,cobbe2020leveraging};
however, none of them provides a support to easily model and generate interactive scenarios. To properly benchmark the capabilities of trained RL agents, we need a systematic way to model and generate scenarios with varying levels of difficulties, which are determined by opponents' initial conditions and a fined-grained control over their behaviors. Furthermore, collecting quality demonstration data (e.g. to train RL agents or benchmark imitation learning algorithms) becomes expensive because humans have to play the game numerously to generate data, potentially with errors. Hence, there is a clear need to support users to intuitively model and generate various scenarios at their demands for accurate benchmark and quality dataset collection.

In this paper, we introduce our platform, Scenic4RL, which offers a systematic and transparent way of specifying and controlling environment dynamics.
We adopt an existing formal scenario specification language, {\sc Scenic}{}~\cite{scenic-static,scenic-dynamic}, to programmatically \textit{model} and \textit{generate} scenarios to benchmark RL algorithms.
{\sc Scenic}{} provides an intuitive abstraction over environment behaviors to easily model interactive scenarios. We interfaced {\sc Scenic}{} to Google Research Football~\cite{kurach2020google}, an stochastic environment that requires the agents to learn strategies at multiple levels of abstraction, from learning to pass, to high level tactics, and presents theoretically infinitely many variations of the environment dynamics (e.g., player positions and behaviors).

Our contributions are as follows:

\begin{itemize}
    \item We introduce a platform which supports intuitive modeling and generation of scenarios programmatically using {\sc Scenic}{} that enables creating diverse scenarios and easy generation of quality demonstration data, at users' demand.

    \item We introduce a dataset consisting of i) 13 mini-game scenarios along with reusable model and behavior libraries ii) semi-expert stochastic policies (and data generated from them) written in {\sc Scenic}\ for select 5 scenarios, and iii) 18 test scripts to evaluate generalization

    \item We evaluate three use cases of our platform and the dataset by benchmarking scenarios, behavior cloning with demonstration data from our stochastic policies, and testing generalization.

\end{itemize}

\section{Background}

\begin{figure*}[t!]
    \centering
    \begin{subfigure}[t]{0.2\textwidth}
        \centering
        \includegraphics[height=0.95in]{sections/figs/pass_and_shoot_birdeye.png}
        \caption{a bird-eye view of the scenario}
        \label{fig:pass_and_shoot_birdeye}
    \end{subfigure}
    ~
    \begin{subfigure}[t]{0.2\textwidth}
        \centering
        \includegraphics[width=1.2in,height=0.95in]{sections/figs/grf_snapshot.png}
        \caption{a snapshot of GRF environment}
        \label{fig:grf_snapshot}
    \end{subfigure}
    ~
    \begin{subfigure}[t]{0.45\textwidth}
        \centering
        \includegraphics[width=\linewidth]{sections/figs/grf_pass_and_shoot.png}
        \caption{GRF's scenario program}
        \label{fig:grf_scenario}
    \end{subfigure}
    ~
    \begin{subfigure}[t]{0.9\textwidth}
        \centering
        \includegraphics[width=\linewidth]{sections/figs/pass_and_shoot_scenic.png}
        \caption{{\sc Scenic}{} program of generalized pass-and-shoot scenario with distribution over players' initial condition and behaviors}
        \label{fig:scenic_program}
    \end{subfigure}
    \caption{Programs encoding the Google Research Football's (GRF) pass-and-shoot scenario}
    \label{fig:pass_and_shoot}
\end{figure*}

\subsection{Google Research Football Simulator}
The Google Research Football (GRF) simulator provides an environment to train and test RL agents to play soccer. The environment setup is as the following. All the players on the field are controlled by GRF's built-in, rule-based AI bots and RL agents. The simulator dynamically determines which of the RL team players are to be controlled by RL agents based on their vicinity to the ball. GRF provides 11 offense scenarios to benchmark RL agent performance.

\subsection{Scenario Specification Language: Scenic}\label{sec:scenic_background}
{\sc Scenic}{} is an object-oriented, probabilistic programming language which is designed to easily \textit{model} and \textit{generate} scenarios. In {\sc Scenic}, a scenario is a distribution over spatio-temporal configuration of agents and objects. Its natural English-like syntax and semantics let users intuitively specify complex spatial and temporal relations among objects (see  Sec.~\ref{scenic_for_RL} for an example). Furthermore, {\sc Scenic}{} allows users to construct object model, action, and behavior libraries and simply reference these from {\sc Scenic}{} programs. This feature expedites the modeling process because users do not have to encode a scenario from scratch. We cover the libraries in detail in Sec.~\ref{scenic_libraries}. Finally, users can specify distributions over states and player behaviors, making a {\sc Scenic}{} program much more expressive. Prior distribution is specified over an object's initial states like position and behavior according to its role. A scenario is generated by sampling an initial state at start and sampling an action for each object during simulation run time according to their specified behavior. If users instantiates an object but do not specify any initial states or behavior, those are sampled from the prior distribution on the object. For further details about the language please refer to \cite{scenic-dynamic}.
\section{The {\sc Scenic4RL}\ Platform for Reinforcement Learning}\label{scenic_for_RL}

\subsection{The Need for Scenario Specification Language for RL}

\begin{comment}
The capability of an agent depends largely on the quality of the scenarios it is trained on. However, as discussed previously, the existing simulators typically do not provide a systematic approach to generate  rich-diverse scenarios, limiting the abilities of reinforcement learning algorithms. To alleviate these problems, we propose our platform {\sc Scenic4RL}{}\ , which provides extensive control over the environment dynamics to generate scenarios on user-demand via {\sc Scenic}\ programs. {\sc Scenic4RL}\ offers a number of advantages over traditional simulators/environments:
\end{comment}

The capability of an agent depends largely on the quality of the scenarios it is trained on. However, the existing simulators typically do not provide a systematic way to generate rich-diverse scenarios, limiting the abilities of reinforcement learning algorithms. For example, the GRF's scenario python program, shown in Figure~\ref{fig:pass_and_shoot}(c), used to generate the Pass and Shoot scenario (Fig.~\ref{fig:pass_and_shoot}(a)), initiates each player and the ball at fixed hand-coded positions. There is no support to position the players intuitively by specifying their geometric relations (e.g. "there is a yellow player ahead of a blue player"). Rather the user has to precisely compute initial positions. Also, there are no easy-to-use interface to modify the opponent behaviors. Moreover, the dynamics of the existing environments often lack transparency, which can make reasoning about them difficult in some cases, e.g., it is often not clear how different (or, similar) the train and test environments are~\cite{cobbe2020leveraging}. To alleviate these problems, we propose our platform {\sc Scenic4RL}{}\ , which provides an \textit{extensive control over the environment dynamics} to generate scenarios on \textit{user-demand} via {\sc Scenic}\ programs. {\sc Scenic4RL}\ offers a number of advantages over traditional simulators/environments:

\begin{enumerate}
    \item \textbf{Programmable Environments on User-demand:} {\sc Scenic4RL} platform allows users to generate scenarios programmatically. Each environment/scenario is represented with a {\sc Scenic}\ program. This enables users to create scenarios as per their necessity.
    \item \textbf{Incorporation of Domain Knowledge:} {\sc Scenic4RL}\ allows writing stochastic policies which can be used to incorporate domain knowledge and generate offline data.
    \item \textbf{Enhanced Evaluation:} Users can define reward functions using {\sc Scenic}. They can also define arbitrary evaluation metrics to evaluate different aspect of a trained agent, e.g., ball possession, number of intercepts. {\sc Scenic4RL}\ also allows writing simple {\sc Scenic}\ policies to  compare with trained RL agents as baselines.
    \item \textbf{Reusability of libraries}: Users can easily modify or re-use existing {\sc Scenic}\ programs, models, actions, and behaviors to quickly model new scenarios.
    \item \textbf{Interpretability}: {\sc Scenic}{} provide an intuitive abstract syntax, resembling natural English, which makes the scenario programs interpretable.
    \item \textbf{Transparency}:
    Being able to specify/modify environment dynamics programmatically also improves transparency of the generated scenarios, e.g., users can reason about the difference/similarity of their train and test environments with more clarity.

\end{enumerate}

\subsection{Modeling Scenarios with {\sc Scenic}{}}

Formally, a scenario is a Markov Decision Process (MDPs)~\cite{sutton2018reinforcement} defined as a tuple $\left( \mathcal{S}, \mathcal{A}, p, r, \rho_0\right)$, with $\mathcal{S}$ denoting the state space, $\mathcal{A}$ the action space, $p\left(s^\prime|s,a\right)$ the transition dynamic, $r\left(s,a\right)$ the reward function, and $\rho_0$ the initial state distribution. Given the state and action spaces as defined by the GRF environment, a {\sc Scenic}{} program defines (i) the initial state distribution, (ii) the transition dynamics (specifically players' behaviors), and (iii) the reward function. Hence, users can exercise extensive control over the environment with {\sc Scenic}{}.

{\bf Modeling Initial State Distribution}
Users can intuitively specify initial state distributions with {\sc Scenic}{}'s high-level syntax. For example, refer to the full {\sc Scenic}{} program in Fig.~\ref{fig:pass_and_shoot}(d) which describes a more generalized version of the Pass and Shoot scenario modelled in Fig.~\ref{fig:pass_and_shoot}(c). In line 12-22, the initial state distribution is specified. The {\sc Scenic}{}'s spatial syntax highlighted in yellow let users to intuitively model spatial relations among objects, rather than having to hand-code positions as in the GRF's scenario. Here, \texttt{Left} represents the yellow team, \texttt{Right} the blue, and the two following abbreviated capital letters indicate the player role. As discussed in Sec.~\ref{sec:scenic_background}, object models (e.g. players, regions, and directions) that are undefined in the program are referenced from the model library, expediting the modeling process. For players instantiated without specification as in line 12 and 22, their state and behavior default to prior distribution, which saves repetitive state specification.

{\bf Modeling Transition Dynamics}
One can flexibly modify transition dynamics of the environment by specifying the behaviors of any players using {\sc Scenic}{}. Take the same example {\sc Scenic}{} program in Fig.~\ref{fig:pass_and_shoot}(d) as above. Line 1-10 models two new behaviors. A behavior can invoke another behavior(s) with syntax \texttt{do}, succinctly modeling a behavior in a hierarchical manner. Users can assign distribution over behaviors as in line 2. The interactive conditions are specified using try/interrupt block as in line 5-10. Semantically, the behavior specified in the try block is executed by default. However, if any interrupt condition is satisfied, then the default behavior is paused and the behavior in the interrupt block is executed until completion and then the default behavior resumes. These interrupts can be nested with interrupt below has higher priority. In such case, the same semantics is consistently applied.

{\bf Rewards}
{\sc Scenic}{} has a construct called \texttt{monitor}, which can be used to specify reward functions. The reward conditions in the \texttt{monitor} is checked at every simulation step and updates the reward accordingly.

\begin{comment}
\subsubsection{Why do We Need Scenario Specification Language?}
Let us visually motivate this point. Figure~\ref{fig:pass_and_shoot}(b) is a GRF's scenario program that models pass-and-shoot with keeper scenario, as visualized in Fig.~\ref{fig:pass_and_shoot}(a), where the RL agent's yellow team is to score against the opponent blue team. The program assigns fixed initial position, team, and role to each of the five players and places the soccer ball at a fixed location. No behavior is scripted. Instead, GRF's rule-based bots and an RL agent control the players. In fact, all of GRF's scenarios are modelled and generated this way.

This approach to model environment significantly increases the modeling time and restricts the model's expressiveness. First, there is a lack of support for assigning any behaviors and interactive conditions to any players. This issue creates reliance on fixed rule-based AI bots or another trained RL agents to generate behaviors in a blackbox manner, which may often not generate users' desired behaviors. Second, it is difficult to intuitively encode players' geometric relation at a high-level (e.g. there is a yellow player ahead of a blue player). Rather, the user has to precisely compute initial condition for all objects. Third, the program is not interpretable. Therefore, users have to execute the scenario program to visually observe and understand the scenario. Furthermore, this non-interpretability makes re-use or easy modification of scenarios very difficult, forcing users to write new scenarios from scratch. {\sc Scenic}{} addresses all of these challenges, allowing intuitive modeling of scenarios involving complex geometric relations among objects and interactive behaviors in an interpretable manner.

\end{comment}

\begin{comment}

\subsection{Programmable Environments}
The capability of an agent depends largely on the quality of the learning scenarios it is trained on.
With our platform we can generate environments programmatically, allowing generation of environments on user-demand. {\sc Scenic} allows low-level control over all the environment objects using a high level abstract semantic, which enables the ability to generate robust learning scenarios by defining distribution over different aspect of the environment dynamics.

\end{comment}

\subsection{Architecture}
Figure~\ref{fig:rlarch} shows an overview of the overall architecture of {\sc Scenic4RL}. The architecture can be divided into two parts, i) RL interface, through which the RL algorithms interact with {\sc Scenic4RL}\ and ii) the {\sc Scenic}\ Server, which executes a {\sc Scenic}{} program and governs the simulation by interacting with the underlying simulator. We implemented our interface using OpenAI Gym API~\cite{brockman2016openai}, which allows {\sc Scenic4RL}\ to be used seamlessly with all the existing standard RL frameworks.

For each simulation/episode, the {\sc Scenic}\ server first samples an initial state from the {\sc Scenic}{} program to start a new scenario in the GRF simulator, and updates its internal model of the world (e.g., player and ball positions). From then on, a round of communication occur between the RL algorithm and {\sc Scenic}\ server, with the RL interface at the middle. At each timestep, the gym interface takes in the action(s) for the RL agent and passes them to the {\sc Scenic}\ server. The {\sc Scenic}\ server in turn computes actions for all the remaining non-RL players---the players not controlled by the RL agent---and then executes all these actions (of both the RL and non-RL players) in the simulator. The {\sc Scenic}\ server then receives the observation and reward from the simulator, updates the internal world state, and then passes them back to the RL algorithm. This interaction goes on till any terminating conditions as specified in the scenario script is satisfied.

\begin{figure}[!htb]
  \centering
  \includegraphics[scale=0.4]{sections/figs/rl_arch.png}
  \caption{{\sc Scenic4RL}\ Architecture}
   \label{fig:rlarch}
\end{figure}

\begin{comment}

\subsection{Benefits of {\sc Scenic4RL}}

In this section we discuss the benefits of {\sc Scenic4RL} \

1. {Generation of environment user-demand}
Programmable interface so user can generate environments on demand for training and testing:

2. {Incorporation of domain knowledge via Scenic Policies}\label{scenic4rl:demonstration}

It is widely known that good quality offline data can facilitate RL training significantly. However, getting good quality offline data is often costly. In practice, the RL practitioners almost always have some domain knowledge on how to solve a particular environment, at least to some extent. Specifically if the environment is hard and sample-inefficient, offline data can significantly reduce training time. Hence, an easy way to generate demonstration data can be very beneficial in practice. Indeed, in the recent Kaggle competition \cite{?} for google soccer, many contestants have utilized offline data to tackle its humongous requirement of compute resources.  With Scenic’s interpretable and robust syntax one can write robust policies to generate high quality demonstration data and utilize it to facilitate the RL training, which is very beneficial for both RL practice and research.

3. {Enhanced Evaluation}

Creation of environment as per user demand. Plus use monitor to gather arbitrary metrics

5.
\end{comment}
\section{Evaluation}\label{sec:evaluation}
In this section, we evaluate our platform {\sc Scenic4RL}\  and the proposed datasets using three RL usecases. First, we benchmark our proposed scenario dataset---using the widely used reinforcement learning algorithm PPO~\cite{ppo}---to enable researchers to compare their algorithms. Then, we evaluate these PPO trained agents using our test dataset, which tests their generalization capabilities against varying initial state distribution. Finally, we evaluate the usefulness of the proposed {\sc Scenic}\ policies by showing significant improvement in training performance.

\subsection{Experimental Setup}\label{sec:exp_setup}
We run PPO on a single GPU machine (NVIDIA T4) with 16 parallel workers on Amazon AWS. Unless otherwise specified all the PPO training are run for 5M timesteps and repeated for 3 different seeds. For all the experiments, we use the stacked Super Mini Map representation for observations ---a 4x72x96 binary matrix representing positions of players from both team, the ball, and the active player---and the scores as rewards, i.e., $+1$ when scoring a goal and $-1$ upon conceding, from~\cite{kurach2020google}. Similar to the academy scenarios from~\cite{kurach2020google}, we also conclude a game when one of the following happens: either of the team scores, ball goes out of the field, or, the ball possession changes.  For further details, including hyperparameters and network architecture, we refer readers to the Supplementary Materials (Section \textit{Details on Experimental Setup and Training}).

\begin{comment}

\begin{figure}
     \centering
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{sections/figs/cross_easy.png}
         \caption{The $cross\_easy$ Scenario}
         \label{fig:cross_easy}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{sections/figs/fg_11v1.png}
         \caption{The $11v1$ Scenario}
         \label{fig:off_11v1}
     \end{subfigure}
    \caption{Two of the new offense Scenarios.}
        \label{fig:test_new_offense}
\end{figure}
\end{comment}
\subsection{Mini-game Scenario Benchmark}
We benchmark our mini-game scenarios by training agents with PPO for 5M timesteps. Figure \ref{fig:scenario-training} shows the average goal differences for all the scenarios. Note that we propose both offense and defense scenarios. For all these scenarios, we end the game if one of the teams score. Hence, the goal difference can range between -1 to +1. For the offense scenarios, a well trained agent is supposed to score consistently achieving a goal difference close to +1. On the other hand, a well trained agent should achieve a goal difference close to 0 for successfully defending the opponents. From the graph it can be seen that the proposed scenarios offer a varied levels of difficulties. For example, PPO consistently achieves goal difference of around 0.7 for the {\sc Easy Crossing} scenario, but barely learns anything for {\sc Hard Crossing}. In case of the defense scenarios, the results also show a varied range of difficulty, {\sc GK vs Opponent} scenario being be easiest. Due to the stochasticity of the GRF environment~\cite{kurach2020google}, we observed variations in training performance, with the most variation for the {\sc 11 vs GK} scenario, reaching 0.93 for one run while failing to learn anything for the other two runs.

\begin{figure}
  \centering
  \includegraphics[scale=0.28]{sections/figs/benchmark.png}
  \caption{Average Goal Difference of PPO agents on the proposed mini-game scenario Benchmark. The error bars represent 95\% bootstrapped confidence intervals}
   \label{fig:scenario-training}
\end{figure}

\subsection{Testing for Generalization}
As discussed in Sec.~\ref{sec:test_ds}, we introduce a dataset of test scripts to evaluate agents for their robustness against varying initial conditions. Fig.~\ref{fig:testing_for_gen} compares the trained agents' performance in training and test scenarios. As expected, we observe a noticeable drop of performance in the GRF academy and offense scenarios (Fig.~\ref{fig:testing_offense}). However, for defense scenarios, the drop was not as significant. To further investigate, we evaluated random RL policy performance in the test scenarios as shown in Fig.~\ref{fig:testing_defense}. Comparing the random and the trained RL agent's policies in the test scenarios, we observed that the trained RL agent noticeably outperforms the random policy across all test scenarios. This clearly indicates that the trained RL agent policy is generalizing to unseen test scenarios. Note that unlike most of the offense scenarios in Fig.~\ref{fig:testing_offense} with fixed initial state, all the defense scenarios assigned distributions over the initial state. We conjecture that this difference helped the RL agent to generalize better to unseen yet similar defense scenarios.

\begin{figure}

  \begin{subfigure}[t]{\textwidth}
  \includegraphics[scale=0.28]{sections/figs/testing_offense.png}
  \caption{Offense and select GRF academy scenarios}
   \label{fig:testing_offense}
  \end{subfigure}

  \begin{subfigure}[t]{\textwidth}
  \includegraphics[scale=0.28]{sections/figs/testing_defense_with_random.png}
  \caption{Defense scenarios}
   \label{fig:testing_defense}
  \end{subfigure}
  \caption{Evaluation of PPO agents' generalization against varying initial conditions. For most of the academy and offense scenarios we observe a significant drop in performance. However, for the defense scenarios the difference in train and test scenarios is not that significant. Hence, we compare the trained agents further with random agents in the test environment and observe clear sign of generalization.}
   \label{fig:testing_for_gen}
\end{figure}

\begin{comment}
\begin{figure}
     \centering
     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth]{sections/figs/ac_run_pass_and_shoot.png}
         \caption{The run, pass, and shoot Scenario from \cite{kurach2020google}. Average goal difference of 0.54}
         \label{fig:ac_rps}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth]{sections/figs/ac_run_pass_shoot_gen_1.png}
         \caption{All the players are now placed uniformly withing the shown boxes. Average goal difference: 0.45}
         \label{fig:ac_rps_gen}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth]{sections/figs/ac_run_pass_shoot_test.png}
         \caption{The player in the middle now has the ball and can shoot himself. Average goal difference: 0.03}
         \label{fig:ac_rps_shoot}
     \end{subfigure}
        \caption{The run, pass, and shoot Scenario from \cite{kurach2020google} along with two testing scenarios and the performance achieved by a PPO agent. All the scores are computed from an average of 500 independent runs.}
        \label{fig:test_ac_rps}
\end{figure}
\end{comment}

\begin{comment}

\begin{figure}
  \centering
  \includegraphics[scale=0.3]{sections/figs/testing.png}
  \caption{Evaluation of PPO agents' generalization against varying initial conditions.}
   \label{fig:testing_for_gen}
\end{figure}
\end{comment}

\subsection{Pretraining with Semi-expert Demonstrations}
In these section we evaluate the {\sc Scenic}\ policies and the demonstration data generated by them, as discussed in~\ref{dataset:behavior_cloning}. For each scenario, we first use behavior cloning to train an agent using 8K randomly selected samples from the 10K samples provided in the demonstration dataset . We then continue the training using PPO for 5M timesteps. Figure ~\ref{fig:pretrain} compares the training performance of these agents against the agents that were trained with PPO only. We notice that, even with such a low volume of demonstration data, we can train much better agents and can solve scenarios which were otherwise unsolved. The experimental results thus suggests, with probabilistic {\sc Scenic}{} policies we can generate rich quality demonstration data to substantially increase training performance, which can be particularly useful in practice for environments like GRF which requires a huge amount of compute resource.

\begin{figure}
  \centering
  \includegraphics[scale=0.28]{sections/figs/pretraining.png}
  \caption{Performance of PPO agents trained with and without any expert data, along with the performance of corresponding {\sc Scenic}\ and behavior cloned policies. We see significantly better training for three of the scenarios, while the rest two achieves comparable performance, highlighting the usefulness of the proposed {\sc Scenic}\ policies. }
   \label{fig:pretrain}

\end{figure}

\section{Constructing Dataset with the {\sc Scenic4RL}\ Platform}

Our dataset comprises of three categories of data: (1) {\sc Scenic}{} libraries, (2) {\sc Scenic}{} programs, and (3) demonstration data generated from semi-expert stochastic {\sc Scenic}{} Policies.

\subsection{{\sc Scenic}{} Libraries}\label{scenic_libraries}
As discussed in Sec.~\ref{sec:scenic_background}, {\sc Scenic}{} libraries consist of three types: model, action, and behavior libraries. The {\sc Scenic}{} libraries significantly reduces users' time to model complex scenarios by helping them re-use the set of models, actions, and behaviors in the libraries, rather than having to write scenario from scratch.

The \textit{model} library defines the state space. It defines players with distribution over their initial state according to their roles and GRF's AI bot is assigned by default to all player behavior. These prior distribution over the initial state and behavior can be overwritten in the {\sc Scenic}{} program. The model library also defines region objects such as goal and penalty box regions as well as directional objects in compass directions. The \textit{action} library defines the action space as determined by the GRF simulator. These action space consists of movement actions in eight compass directions, long/short/high pass, shoot, slide, dribble, and sprint.

The \textit{behavior} library consists of behaviors and helper functions that represent widely used basic skills in soccer. These behaviors include give-and-go, evasive zigzag dribble to avoid an opponent's  ball interception, dribbling to a designated point and shooting, shooting towards the left or right corner of the goal, etc. Additionally, the behavior library also include useful helper functions such as identifying nearest opponent or teammate, whether there is an opponent near the running direction of a dribbler, etc. Please refer to our open-sourced repository for more details.

\begin{figure*}[!htbp]
    \centering
    \begin{subfigure}[t]{0.23\textwidth}
        \centering
        \includegraphics[height=1.5in, width=1.1in]{sections/figs/new-scenario-1vs1.png}
        \caption{1 vs 1 evasive zig-zag towards the goal and shoot}
        \label{fig:defense_1vs1}
    \end{subfigure}
    ~
    \begin{subfigure}[t]{0.23\textwidth}
        \centering
        \includegraphics[height=1.5in, width=1.1in]{sections/figs/new-scenario-2vs2.png}
        \caption{2 vs 2 advance the ball towards goal via high pass and shoot}
        \label{fig:defense_2vs2}
    \end{subfigure}
    ~
    \begin{subfigure}[t]{0.23\textwidth}
        \centering
        \includegraphics[height=1.5in, width=1.1in]{sections/figs/new-scenario-3vs3.png}
        \caption{3 vs 3 left mid-fielder crosses to either player in the penalty box}
        \label{fig:defense_3vs3}
    \end{subfigure}
    ~
    \begin{subfigure}[t]{0.23\textwidth}
        \centering
        \includegraphics[height=1.5in, width=1.1in]{sections/figs/test_pass_and_shoot.png}
        \caption{generalization test scenario for pass and shoot scenario in Fig.~\ref{fig:pass_and_shoot_birdeye}}
        \label{fig:test_pass_and_shoot}
    \end{subfigure}
    \caption{Examples of our new defense scenarios with specific assigned behaviors (left three images) and a test scenario to assess generalization (rightmost). The RL team is yellow and the opponent, blue. The assigned opponent behaviors are highlighted with light blue arrows. Uniformly random distribution is assigned over a specific region for each player. These regions are highlighted boxes.}
    \label{fig:new_scenarios}
\end{figure*}

\subsection{{\sc Scenic}{} programs}
Please refer to our Supplement for detailed descriptions of all our scenarios and policies.

{\bf New Benchmark Scenarios}\label{sec:new_benchmark_scenarios}
We modelled 13 new benchmark scenarios. Nine of them are defense scenarios which are nice complement to GRF's offense-only scenarios. The remaining four scenarios are additional distinct offense scenarios that enriches GRF's benchmark scenarios. We varied the initial state distribution and assigned GRF's AI bot behavior by default. However, if the AI bot did not exhibit the behaviors we expected, we modelled the behaviors of environment agents to create such scenarios. For example, a few of our defense scenarios with specified behaviors are visualized in Fig.~\ref{fig:new_scenarios}. In these scenarios, the opponent AI bot did not exhibit expected behaviors so we specified behaviors in the {\sc Scenic}{} programs as described in the sub-captions. In 1vs1 scenario, the opponent AI bot was not aggressive enough to attack the goal and was easily pushed out of the penalty box. In 2vs2 scenario, the opponent initially with the ball did not quickly advance the ball to its teammate closer to the goal post via high pass. In 3vs3, the opponent player with the ball on the leftmost side of the field did not cross the ball to its teammates.

{\bf Scenarios for Testing Generalization}  \label{sec:test_ds}
We provide scripts to test generalization of GRF's 5 academy scenarios and all of our 13 new benchmark scenarios from our dataset. The scripts test how robust the trained agents are to perturbations in the initial state. We assigned distribution over the initial state while keeping the formation of players and their behaviors in each scenario intact. For example, for testing generalizability of an RL agent trained in the Pass and Shoot scenario (Fig.~\ref{fig:pass_and_shoot_birdeye}), we instantiated the yellow and the blue players on the symmetric right side of the field instead of the left and kept the other initial state distribution the same (Fig.~\ref{fig:test_pass_and_shoot}).

{\bf Probabilistic Scenic Policies and Offline Dataset}\label{dataset:behavior_cloning}
{\sc Scenic4RL}\ also allows users to incorporate their domain knowledge by writing probabilistic policies, which can be used as a means to generate rich demonstration data. Users can benefit from using our \textit{behavior} library to write their policies, instead of having to write them from scratch. In our work, we present simple {\sc Scenic}{}\ policies for three GRF academy scenarios~\cite{kurach2020google} and two of our new benchmark scenarios in which a proximal policy optimization (PPO) algorithm\cite{ppo} had relatively poor training performance. These policies are not to be considered "expert", as they do not solve the scenarios consistently. However, these simple policies can be considered as guidance/ semi-expert policies, which can be utilized for better training. We then generate very small datasets of demonstration from these policies, each containing ~10K samples collected from the successful (i.e. the agent scores) trajectories. Each sample is a $<o,a>$ pair, i.e., specifying the action $a$ the {\sc Scenic}{} policy took on the observation $o$. In our experiments we show that, such small offline dataset can expedite the RL training significantly.

\subsection{Towards Crowd Sourced Dataset}
A current limitation of our benchmark dataset is that it does not provide a comprehensive set of scenarios and policies that cover all realistic soccer scenarios and policies on the field. In fact, there are infinitely many different possible scenarios. We address this issue by open-sourcing our platform and dataset, \href{https://github.com/scenic4RL/scenic4rl/tree/v1}{https://github.com/scenic4RL/scenic4rl/tree/v1}, to allow other researchers to contribute their own scenarios and policies as {\sc Scenic}{} programs. A good crowdsourced dataset will cover much more scenarios than a research group can generate. In fact, the effort to crowdsource a dataset in RL domain has been low due to the lack of support for environment modeling. We share our platform to encourage it.

\section{Conclusion \& Future works}\label{sec:conclusion}

In this work we present {\sc Scenic4RL}{}: A platform for generation of diverse scenarios for Reinforcement Learning programmatically, using the {\sc Scenic} scenario specification language. We present datasets of mini-game scenarios, behaviors depicting common soccer short-term strategies, probabilistic {\sc Scenic}{} policies and the demonstration dataset generate from them. Lastly, our evaluation demonstrates that our platform and the dataset can be applied to train, test, and benchmark RL algorithms.

{\bf Limitations and future directions}. To the best of our knowledge {\sc Scenic4RL}\ is the first attempt to utilize formal scenario specification language to model environment dynamics in the domain of RL. There are a number of interesting future directions that we plan to pursue.

\begin{itemize}
    \item Interfacing other simulators: Currently, our platform only supports the Google Soccer Environment. In future we plan to include more simulators.

    \item As we are adding a layer over the base GRF simulator, we also add an overhead. In future we plan to decrease this overhead. Please refer to the Supplementary Materials for detailed experiment on {\sc Scenic4RL}\ performance.

    \item Multi-Agent Learning: While our platform does not restrict us to only single-agent RL settings, our current training experiments/benchmark do not include multi-agent training. In future, we would like to introduce scenarios and benchmarks for multi-agent setting.

    \item Multi-Task Learning: Currently we only train RL agents on a single scenario. However, many soccer skills are transferable. Hence, we want to experiment how we can solve multiple scenarios simultaneously in a multi-task setting

    \item Use of Formal Analysis Techniques. Using {\sc Scenic4RL}{} it is possible to implement verification and systematic testing techniques (e.g., see~\cite{fremont-cav20,fremont-itsc20}).
\end{itemize}

{\bf Negative Social Impact}. Because our work concerns simulated soccer environment and has no relation to humans in the real world, we do not foresee any negative social impact of this work.
\section{Related work}

\textbf{Improving Environment Dynamics for Generalization}: In literature, several techniques have been adopted to generate a rich variation of learning scenarios, primarily to promote or, ensure generalization. Techniques such as changing background with natural videos~\cite{zhang2018natural}, introducing sticky actions~\cite{machado2018revisiting} have been attempted, but are not robust enough. To ensure generalization, \cite{lee2020context} and \cite{seo2020trajectory} generated training and testing scenarios by randomly sampling from different regions of parameter space. Similar to supervised learning, the use of separate train and test sets have also been adopted~\cite{nichol2018gotta, cobbe2020leveraging, cobbe2019quantifying, justesen2018illuminating}, typically using techniques such as Procedural Content Generation, which has traditionally been used to automatically generate levels in video games. However, most of these focus on discrete domain, typically the dataset generation process is opaque, and it can be difficult to  quantify or, reason about how different (or, similar) these train and test sets are.

\textbf{Formal Scenario Specification Languages} A few scenario specification languages have been proposed in the autonomous driving domain since the introduction of {\sc Scenic}{}. Paracosm language ~\cite{paracosm} models dynamic scenarios with reactive and synchronous model of computation. The Measurable Scenario Description Language (M-SDL)~\cite{msdl} shares common features as {\sc Scenic}{} to model interactive scenarios. In contrast, however, {\sc Scenic}{} provides a much higher-level, probabilistic, declarative way of modeling, and it is not specific to autonomous driving domain.

\textbf{Datasets and Benchmarks}. There is a large body of work focused on designing benchmarks for RL~\cite{ beattie2016deepmind,bellemare2013arcade, brockman2016openai, cobbe2019quantifying, cobbe2020leveraging, duan2016benchmarking,fu2020d4rl, gulcehre2020rl,henderson2018deep, kurach2020google, ray2019benchmarking,  dmcontrol_old,dmcontrol_new, yu2020meta}.
The Arcade Learning Environment (ALE)~\cite{bellemare2013arcade} has becomes a popular benchmark to measure the progress of RL algorithms for discrete control tasks.
For continuous control tasks,
\citet{duan2016benchmarking} presented a benchmark with baseline implementations of various RL algorithms.
These benchmarks accelerate the progress, resulting advanced RL algorithms~\citep{sac,mnih2015human,trpo,schulman2015high,ppo}.

Recently, various RL benchmarks designed for various purposes have been proposed.
\citet{cobbe2020leveraging} presented a suite
of game-like environments where the train and test environments differ for evaluating generalization performance of RL agents.
\citet{kurach2020google} presented a Google Football simulator to study curriculum learning and multi-agent RL.
\citet{ray2019benchmarking} provided a Safety Gym for measuring progress towards RL agents that satisfy the safety constraints.
D4RL~\cite{fu2020d4rl} and RL Unplugged~\cite{gulcehre2020rl} have been proposed to evaluate and compare offline RL algorithms,
and \citet{yu2020meta} proposed Meta-world to study meta- and multi-task RL.

\section{DataSheets for Dataset}
\subsection{Motivation}
The dataset using {\sc Scenic4RL}{} platform is created to (i) benchmark training performance, (ii) test generalizability, and (iii) generate semi-expert demonstration data for reinforcement learning (RL) algorithms in Google Research Football (GRF) Environment~\cite{kurach2020google}. More importantly, we open-source our {\sc Scenic4RL}{} platform which enables researchers to intuitively \textit{model} distributions over initial states and transition dynamics over RL and non-RL (i.e. environment) players to \textit{generate} complex scenarios in GRF.

We share this platform for two reasons. First, despite advances in RL simulation environments, the lack of support to easily model and generate dynamic, interactive scenarios has become a bottleneck. Consequently, the benchmark scenarios have been restrictive (e.g. only varying the initial state distributions), and researcher had to rely on humans to generate quality demonstration data. In response, for the first time, we adopt an existing scenario specification language, {\sc Scenic}{}~\cite{scenic-static, scenic-dynamic}, to support researchers to intuitively model such complex environment. In our {\sc Scenic4RL}{} platform, researchers can use {\sc Scenic}{}'s high-level syntax and semantics to quickly create complex environment with a fine-grained control over players' behaviors (i.e. transition dynamics). Secondly, because the space of all possible scenarios is infinite in the soccer environment and, therefore, it is impossible for one research group to provide a comprehensive set of scenarios. We share our platform to encourage the RL community to collectively contribute to constructing a comprehensive set of benchmark scenarios and rich demonstration data using {\sc Scenic}{}.

This dataset is created by Abdus Salam Azad, Edward Kim, Kimin Lee, and Qiancheng Wu, who are currently researchers at UC Berkeley. This work is not funded by any specific organization.

\subsection{Composition}\label{sec:composition}
The dataset consists of 38 {\sc Scenic}{} programs and 5 sets of demonstration data that are generated from a subset of the programs. Here, scenarios are equivalent to {\sc Scenic}{} programs because {\sc Scenic}{} is a probabilistic programming language. Hence, a scenario encoded in {\sc Scenic}{} naturally becomes a program.

Of the 36 {\sc Scenic}{} programs, (i) 13 programs encode new offense and defense benchmark scenarios that the GRF environment does not provide, (ii) 18 programs model scenarios to test generalizability of RL agents trained from our 13 new benchmark scenarios and 5 GRF's benchmark scenarios with high (i.e. average scoring rate of 0.7 per game in each scenario) RL agent training performance, and (iii) 5 {\sc Scenic}{} programs specifying semi-expert RL stochastic policy to generate demonstration data. For these policies, we selected 5 offense scenarios from our new benchmark and GRF's benchmark scenarios where the proximal policy optimization (PPO) algorithm~\cite{ppo} performed relatively poorly, and scripted policy customized to each scenario. Hence, we provide 5 sets of demonstration data where each set consists of ~10K samples collected from the successful (i.e. scores a goal) trajectories. Each sample is a $<o,a>$ pair, i.e., specifying the action $a$ the {\sc Scenic}{} policy took on the observation $o$. In our benchmark scenarios, we used the same reward function as GRF environment, which is to score (i.e. +1 if scores, -1 if scored, 0 otherwise).

Our benchmark scenarios and semi-expert policies are stochastic. This is because {\sc Scenic}{} is a \textit{probabilistic} programming language, meaning users can specify distribution over states and transition dynamics. Hence, our provided scenarios and policies are expressive, capturing variety of initial state and transition dynamics that share certain spatial and temporal structures.

This dataset does not contain all possible benchmark scenarios or demonstration data in GRF environment since the space of possible scenarios are infinite. The scenarios and demonstration data we provide are only a subset which we modeled and generated using our domain knowledge of soccer.

Our dataset is self-contained, meaning we did \textit{not} rely on any external sources. As to our {\sc Scenic4RL}{} platform, we relied on three external open-sourced codebase: {\sc Scenic}{}, GRF environment, and OpenAI gym environment API~\cite{brockman2016openai}. Our dataset does \textit{not} contain any data that might be considered confidential, and it does \textit{not} relate to people nor contain any data that might be offensive.

\subsection{Collection Process}
The {\sc Scenic}{} programs in our dataset is manually modelled by us using our domain knowledge of soccer. We modelled our benchmark scenarios such that each scenario is qualitatively distinct. Broadly, there are two different scenario categories: offense and defense, both of which are crucial for solving the environment. Within each category, we made distinction in three ways. First, the number of players are varied. Second, if scenarios share the same number of players, then their initial state distributions are varied to create different scenarios. Lastly, if the first two features of the scenarios are the same, then we specified distinct transition dynamics to players to qualitatively differentiate scenarios by either (a) modeling transition dynamics in {\sc Scenic}{} or (b) assigning rule-based AI bots provided by GRF environment to control players.

We provide 18 {\sc Scenic}{} programs to test generalization of (i) five GRF benchmark scenarios where PPO algorithm had high (i.e. average scoring rate of 0.7 per game in each scenario) training performance as reported in ~\cite{kurach2020google}) and all of our new benchmark scenarios from our dataset. The scripts tests how robust the trained agents are to perturbations in the initial state. We assigned distribution over the initial state while keeping the formation of players and their transition dynamics in each scenario intact.

As to {\sc Scenic}{} programs specifying stochastic semi-expert RL policies, we chose 5 offense scenarios in which PPO performed relatively poorly (i.e. average scoring rate less than 0.7 goal per mini-game as specified by the scenarios) from our new benchmark and GRF's benchmark scenarios. We scripted a stochastic semi-expert RL policy customized to each of the five offense scenarios using our domain knowledge of soccer.

Our benchmark and test scenarios and semi-expert stochastic RL policies are executed in GRF environment using our {\sc Scenic4RL}{} platform, which interfaces the {\sc Scenic}{} and the GRF simulator. During simulation, the {\sc Scenic}{} program statically samples an initial state at start, and the {\sc Scenic}{} program and an RL algorithm dynamically samples players' actions according to specified transition dynamics in the program. Specifically, the {\sc Scenic}{} program, the RL algorithm (via OpenAI gym environment API), and the GRF simulator communicate the following information at every simulation timestep; the simulator sends the ground truth state information of the world such as each player position, direction, role, ball possession, etc., while the scenic program and the RL algorithm takes this ground truth and samples actions for each RL and non-RL players. These sampled actions are sent to the simulator to be executed for 1 simulation timestep, and this communication continues until termination conditions are satisfied. We used the same termination conditions as GRF environment: (a) ball goes out of the field, (b) either team scores, and (c) change in the ball possession across teams. For the demonstration data generation process, the RL agent is substituted with our manually scripted policy (i.e. transition dynamics). In such case, only the {\sc Scenic}{} program and the GRF simulator interacts to generate the demonstration data. For more detail, refer to Section 3.3 of our main paper.

We sample specific instances of scenarios (i.e. concrete initial state and transition dynamics) in accordance with the type of distributions specified in the {\sc Scenic}{} program. Currently, {\sc Scenic}{} supports a uniformly random distributions on discrete, categorical, and continuous variables and a normal distribution. No human entity was involved in these benchmark scenario or demonstration data generation process, since the generation is automatically handled by our {\sc Scenic4RL}{} platform. The benchmark scenarios are generated for 5 million simulation timesteps, equivalent to the benchmark setting provided in the GRF paper~\cite{kurach2020google}. As to demonstration data, since we only collected successful demonstration, the generation time varied widely according the accuracy of the policies, ranging from one to four hours to collect 10k demonstrations per set.

\subsection{Data Pre-processing}
For our demonstration dataset, we scripted customized policy for each of the five offense scenarios as discussed in Sec.~\ref{sec:composition}. We only collected successful trajectories where our policy ended in the RL team scoring against the opponent team. The non-successful demonstration data are discarded. Our code to generate this pre-processed demonstration data is open-sourced.

This non-successful demonstration can result from two causes. First, our scripted semi-expert policy may not always result in scoring. Second, the GRF environment has inherent stochasticity~\cite{kurach2020google} that occasionally either disregards or do not fully execute sampled actions from our {\sc Scenic}{} programs. This stochasticity is intentionally embedded in the simulator by the Google Research group to provide richer environment to RL agents.

\subsection{Uses}
We showcase the three use cases of our dataset and the {\sc Scenic4RL}{} platform: (i) benchmark training performance, (ii) test generalizability, and (iii) collect demonstration data to warm-start an RL agent. The experiment results for these use cases are included in the Evaluation Section of our main paper. Our dataset and the {\sc Scenic4RL}{} platform is new and, hence, it has not been used for any other task previously. Currently, our scenario and demonstration generation code implementation has been geared toward only single RL agent setting. We intend to extend our code base to accommodate multi-agent setting for our future work.

\subsection{Distribution}
Our dataset and the {\sc Scenic4RL}{} platform are publicly released on June, 2021 with BSD 3-Clause License. The code implementation for our platform, dataset generation, and experimentation as well as our dataset can be downloaded from \href{https://github.com/scenic4RL/scenic4rl/tree/v1}{https://github.com/scenic4RL/scenic4rl/tree/v1}. Our code base requires users to install the GRF simulator~\cite{kurach2020google}, the {\sc Scenic}{}~\cite{scenic-dynamic} repo, and the OpenAI's gym environment~\cite{brockman2016openai} API which are all open-sourced and free of charge.

\subsection{Maintenance}
The dataset and the platform will be hosted at UC Berkeley. All questions and comments can be sent to Abdus Salam Azad (\href{salam\_azad@berkeley.edu}{salam\_azad@berkeley.edu}) and Edward Kim (\href{ek65@berkeley.edu}{ek65@berkeley.edu}). All changes to the dataset and the platform will be announced through our mailing list. Those who would like to sign up should send an email to \href{scenic4rl@lists.berkeley.edu}{scenic4rl@lists.berkeley.edu}. All the code and dataset will continue to be supported unless otherwise communicated through our mailing list. At this point, we do not have any expiration date to our code nor dataset.

We encourage users to contribute to enhancing the platform and to construct a comprehensive scenarios to benchmark, train, and test generalizable RL agents. They can do so by creating pull requests with their modifications and explanations. Upon review, they will be merged into our repo.

\section{Description of Proposed Scenarios and Policies}\label{sec:dataset_descrition}
In this section we provide brief descriptions of all of the scenarios in our dataset.

\begin{figure}
     \centering
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/easy_cross.png}
         \caption{{\sc Easy Crossing}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_easy_cross.png}
         \caption{{\sc Generalized Easy Crossing}}
     \end{subfigure}

     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/hard_cross.png}
         \caption{{\sc Hard Crossing}}
         \label{fig:ds_off_cross_hard}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_hard_cross.png}
         \caption{{\sc Generalized Hard Crossing}}
     \end{subfigure}

     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/11_vs_gk.png}
         \caption{{\sc 11 vs GK}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_11vs1.png}
         \caption{{\sc Generalized 11 vs GK}}
     \end{subfigure}

     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/avoid_pass_shoot.png}
         \caption{{\sc Avoid, pass, and Shoot}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_avoid_pass_and_shoot.png}
         \caption{{\sc Generalized Avoid, pass, and Shoot}}
     \end{subfigure}

    \caption{New offense benchmark scenarios (left images) and corresponding generalized test scenarios (right images) in our dataset. The highlighted boxes represent the regions over which players' initial positions are uniformly randomly distributed. The opponent is in blue and the RL team in yellow.}
    \label{fig:scenarios}
\end{figure}

\begin{figure}
     \centering
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/goalkeeper_vs_opponent.png}
         \caption{{\sc Goalkeeper vs Opponent}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_goalkeeper_vs_opponent.png}
         \caption{{\sc Generalized Goalkeeper vs Opponent}}
     \end{subfigure}

     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/defender_vs_opponent_with_hesitation_dribble.png}
         \caption{{\sc Defender vs Opponent with Hesitant Dribble}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_defender_vs_opponent_with_hesitation_dribble.png}
         \caption{{\sc Generalized Defender vs Opponent with Hesitant Dribble}}
     \end{subfigure}

     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/defender_vs_opponent_with_zigzag_dribble.png}
         \caption{{\sc Defender vs Opponent with Zigzag Dribble}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_defender_vs_opponent_with_zigzag_dribble.png}
         \caption{{\sc Generalized Defender vs Opponent with Zigzag Dribble}}
     \end{subfigure}

     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/defense_2vs2.png}
         \caption{{\sc 2 vs 2}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_defense_2vs2.png}
         \caption{{\sc Generalized 2 vs 2}}
     \end{subfigure}

    \caption{New defense benchmark scenarios (left images) and corresponding generalized test scenarios (right images) in our dataset. The highlighted boxes represent the regions over which players' initial positions are uniformly randomly distributed. The opponent is in blue and the RL team in yellow.}
    \label{fig:defense_scenarios1}
\end{figure}

\begin{figure}
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/2vs2_counterattack.png}
         \caption{{\sc 2 vs 2 Counterattack}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_2vs2_counterattack.png}
         \caption{{\sc Generalized 2 vs 2 Counterattack}}
     \end{subfigure}

     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/2vs2_with_high_pass_forward.png}
         \caption{{\sc 2 vs 2 with High Pass Forward}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_2vs2_with_high_pass_forward.png}
         \caption{{\sc Generalized 2 vs 2 with High Pass Forward}}
     \end{subfigure}

    \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/3vs2_counterattack.png}
         \caption{{\sc 3 vs 2 Counterattack}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_3v2_counterattack.png}
         \caption{{\sc Generalized 3 vs 2 Counterattack}}
     \end{subfigure}

    \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/3vs3_cross_from_side.png}
         \caption{{\sc 3 vs 3 Cross from side}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_3vs3_cross_from_side.png}
         \caption{{\sc Generalized 3 vs 3 Cross from side}}
     \end{subfigure}

    \caption{New defense benchmark scenarios (left images) and corresponding generalized test scenarios (right images) in our dataset. The highlighted boxes represent the regions over which players' initial positions are uniformly randomly distributed. The opponent is in blue and the RL team in yellow.}
    \label{fig:defense_scenarios2}
\end{figure}

\begin{figure}
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/3vs3_side_buildup_play.png}
         \caption{{\sc 3 vs 3 side build up play}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/generalized_3vs3_side_buildup_play.png}
         \caption{{\sc Generalized 3 vs 3 side build up play}}
     \end{subfigure}
    \caption{New defense benchmark scenario (left images) and corresponding generalized test scenarios (right images) in our dataset. The highlighted boxes represent the regions over which players' initial positions are uniformly randomly distributed. The opponent is in blue and the RL team in yellow.}
    \label{fig:defense_scenarios3}
\end{figure}

\begin{figure}
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/run_to_score_with_keeper.png}
         \caption{{\sc run to score with a goal keeper}}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/pass_and_shoot.png}
         \caption{{\sc Pass and Shoot with a Goal Keeper}}
     \end{subfigure}

     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figs/easy_counter_attack.png}
         \caption{{\sc easy counterattack}}
     \end{subfigure}
    \caption{Google Research Football environment's scenarios for which we wrote semi-expert RL policies}
    \label{fig:grf_scenarios}
\end{figure}

\subsection{New Benchmark Scenarios}
In general, all our scenarios have the following three termination conditions: (i) ball goes off the field, (ii) change in ball possession across teams, (iii) one of the team scores. If any of these conditions are satisfied, then the scenario will terminate in simulation.

\subsubsection{Offense Scenarios}
In all of our four offense scenarios as shown in Fig.~\ref{fig:scenarios}, we explicitly modelled the initial state distribution in using {\sc Scenic} and implicitly specified the behaviors to environment players by assigning the rule-based AI bots provided by Google Research Football (GRF) to control all non-RL players.

\textbf{Hard Crossing}: A very common scenario in real soccer games: 2 of our players along are guarded by 3 of the opponent players, in an interleaved manner, along the line of the penalty box. Another of our player at the edge of the field is attempting a cross.

\textbf{11 vs GK}: Our team, with a full lineup of eleven players in a traditional 4-4-2 formation, needs to score against the opponent goalkeeper.

\textbf{Avoid, Pass, and Shoot}: Two of our players, one starting on the middle of the right half and the other inside the penalty box, tries to score. One opponent defender starts between our players to intercept direct pass.

\textbf{Easy Crossing}: An easy crossing scenario involving two of our players against opponent defender and goalkeeper in the penalty box.

\subsubsection{Defense Scenarios}
Like the offense scenarios, we assigned rule-based AI bots provided by GRF by default to control non-RL players in many of our defense scenarios as shown in Fig.~\ref{fig:defense_scenarios1},~\ref{fig:defense_scenarios2},~\ref{fig:defense_scenarios3}. However, if the AI bots do not exhibit expected behavior for our modelled scenarios, we specified non-RL players' behaviors in {\sc Scenic}{}. For these scenarios with specified behaviors in {\sc Scenic}{}, their behaviors are highlighted with light blue arrows.

\textbf{Goalkeeper vs Opponent}:
This scenario is designed to train an RL agent to be a defensive goalkeeper when it has to face an opponent one-on-one.

\textbf{Defender vs Opponent with Hesitant Dribble}:
The opponent dribbles, stop, then dribbles again in a repeated manner.

\textbf{Defender vs Opponent with Zigzag Dribble}:
This opponent aggressively evades the defender with zigzag dribble towards the goal and shoots.

\textbf{2 vs 2}:
Typical, 2 vs 2 setting where two defenders are already in place to fend off the two opponents near the penalty area with the ball.

\textbf{2 vs 2 Counterattack}:
An opponent attacking midfielder is already advanced deep into the left side of the field. The opponent right midfielder behind either short passes the ball to the attacking midfielder or dribbles up the field.

\textbf{2 vs 2 High Pass Forward}:
An opponent attacking midfielder is already advanced deep into the left side of the field. The opponent right midfielder quickly advances the ball to the attacking midfielder via high pass.

\textbf{3 vs 2 Counterattack}:
The defender near the penalty box is temporarily outnumbered by the opponent players due to a sudden counterattack.

\textbf{3 vs 3 Cross from Side}:
The opponent player on the side crosses the ball to either of the teammates in the middle who are running towards the penalty box area to receive the ball. \footnote{The {\sc Defense 3vs3 with cross} scenario doesn't conclude a game upon  a change in ball possession, unlike other scenarios}

\textbf{3 vs 3 Side Build Up Play}:
Instead of crossing, the opponent player on the side builds up a play by short passing to its teammates.

\subsection{Testing Generalization}
For all the new benchmark scenarios in our dataset as well as for the selected five GRF's scenarios, we generalized those scenarios to test the generalizability of the trained RL agent. Our test scenarios are juxtaposed to corresponding scenarios in Fig.~\ref{fig:scenarios}, ~\ref{fig:defense_scenarios1}, ~\ref{fig:defense_scenarios2}, ~\ref{fig:defense_scenarios3}. We modelled these test scenarios by either (i) adding distribution over the initial state or (ii) creating a symmetric opposite formation.

\subsection{Semi-Expert Stochastic Policies}
We selected five scenarios from GRF's and our benchmark scenarios. The selected GRF's scenarios are shown in Fig.~\ref{fig:grf_scenarios}. The following are the brief descriptions of policies encoded in {\sc Scenic}{} for each scenario.

\textbf{Pass and Shoot with a Goal Keeper}:
We randomly choose one of the two policies for the RL agent. In the first policy, the player dribbles to the penalty area and shoots once inside it. In the second policy, the player passes the ball to the teammate, who will then dribble towards the goal and shoot.

\textbf{Easy Counterattack}:
The first player will pass the ball to the right midfielder. Then, the player with the ball will run into the penalty area, and if there is an opponent player on the way, the player will pass the ball to the nearest teammate. The player will shoot at a corner of the goal once inside the penalty area.

\textbf{Run to Score with a Goal Keeper}:
The player with the ball will first sprint towards the goal and turn slightly to either left or right randomly to evade the opponent goalkeeper’s interception. Once the player bypasses the goalkeeper, or is inside the penalty area, the player will shoot.

\textbf{Avoid Pass and Shoot}:
The player decides to go towards 3 suitable regions of scoring: left edge/ middle/ right edge of the right goal post, by keeping as much distance possible to the opponent defender. At each time step it decides one of the three destination location. It first predicts its next position for all the three suitable destinations and pick the direction which keeps it farthest of the opponent. If the player comes near the defender it passes the ball to its teammate and if it can successfully go near the goal post, it attempts shooting.

\textbf{11 vs Goal Keeper}:
The player with the ball runs towards the right goalpost, if it reaches near the goal post it attempts to shoot. If the opponent goal keeper comes near (i.e. within seven meters) our player before it can reach near the right goal post, it stops running and shoots immediately.

\section{Details on Experimental Setup and Training}
We use the OpenAI Baselines'~\cite{baselines} implementation of PPO. The training was run for 5M timesteps with 16 parallel workers. All of our experiments are run on g4dn.4xlarge instances on Amazon AWS: a machine with a single NVIDIA T4 gpu, 16 virtual cores and 64GB RAM.

{\bf Network architecture \& Hyperparameters}
For the PPO training, we first experimented with the network architecture and hyperparameters from~\cite{kurach2020google} and was able to reproduce their result. \cite{kurach2020google} did an extensive  search to select their hyperparameters and hence we decided to use the same for our experiments. The architecture we used from ~\cite{kurach2020google} is similar to the architecture introduced in~\cite{impala}, with the exception of using four big blocks instead of three.

Table~\ref{tab:hp_ppo} provides specific values of the hyperparameters used in the PPO experiments.

\begin{table}[!h]
\begin{tabular}{lr}
\hline
\textbf{Parameter}           & \textbf{Value}   \\
\hline
Action Repetitions  & 1       \\
Clipping Range          & .115    \\
Discount Factor ($\gamma$)     & 0.997   \\
Entropy Coefficient & 0.00155 \\
GAE ($\lambda$)                 & 0.95    \\
Gradient Norm Clipping          & 0.76  \\
Learning Rate & 0.00011879\\
Number of Actors & 16\\
Optimizer & Adam \\
Training Epochs per Update & 2 \\
Training Min-batches per Update & 4 \\
Unroll Length/n-step & 512 \\
Value Function Coefficient & 0.5 \\
\hline
\end{tabular}
\caption{Training Parameters for PPO.}\label{tab:hp_ppo}
\end{table}

The parameters for behavior cloning is shown in Table~\ref{tab:hp_bc}. For the GRF academy scenarios the behavior cloning algorithm is run for 16 epochs while for the offense scenarios it was run for 5 epochs.

\begin{table}[!h]
\begin{tabular}{ll}
\hline
\textbf{Parameter}           & \textbf{Value}   \\
\hline
Learning Rate & 3e-4\\
Batch Size & 256\\
Optimizer & Adam \\
Epsilon(Adam) & 1e-5\\
\hline
\end{tabular}
\caption{Training Parameters for Imitation Learning.}\label{tab:hp_bc}
\end{table}

\section{Interface details and Reproducibility}

Our Platform follows the widely used OpenAI Gym API~\cite{brockman2016openai}.  For sample usage, we refer readers to our github repository:  \href{https://github.com/scenic4RL/scenic4rl/tree/v1}{https://github.com/scenic4RL/scenic4rl/tree/v1}.  The github repository contains detailed description of all our API with examples. It also contains necessary scripts to reproduce the results shown in paper.

\begin{comment}

{\bf Scenic GFootball Interface: API \& Sample Usage}
Below we show an example code of how to instantiate an environment from a scenario file along with other parameters in a dictionary and run a random agent on it.

\begin{verbatim}
from scenic.simulators.gfootball.rl.gfScenicEnv import GFScenicEnv
scenario = f"path/script.scenic"
env = GFScenicEnv(scenario=scenario, allow_render=False,
                  gf_env_settings={"rewards":'scoring', ...})
env.reset()
done = False
while not done:
    action = env.action_space.sample()
    observation, reward, done, info = env.step(action)
\end{verbatim}
\end{comment}

\section{Performance}

As we are adding an additional layer over the GRF simulator, we wanted to measure how much overhead we are adding over the base GRF simulator. We selected five GRF academy scenarios and ran a simulation of 20K timesteps with a random policy both in the GRF simulator and in {\sc Scenic4RL}. The simulation was ran sequentially, i.e., no parallelism was used. Across the scenarios, the GRF simulator took an average of 74.28 seconds for executing a simulation of 20K timesteps, while the {\sc Scenic4RL}\ interface took 222.07 seconds, showing a 2.99x drop in speed. Some of this overhead is inevitable however, we believe there are ways to speed up {\sc Scenic4RL}\ . For example, as we change the initial state every episode/simulation: we update the Python scenario file used by the GRF simulator for each episode/simulation. We plan to modify GRF interface to avoid such disk-access each simulation to speed up {\sc Scenic4RL}\ among other performance improvements. The scenarios we used for the experiments are namely: i) Empty Goal, ii) Empty Goal Close, iii) Pass and Shoot with Keeper, iv) Run, Pass, and Shoot with Keeper, and v) Run to Score with Keeper.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)