PatchDropout: Economizing Vision Transformers Using Patch Dropout

Abstract: Vision transformers have demonstrated the potential to outperform CNNs in a variety of vision tasks. But the computational and memory requirements of these models prohibit their use in many applications, especially those that depend on high-resolution images, such as medical image classification. Efforts to train ViTs more efficiently are overly complicated, necessitating architectural changes or intricate training schemes. In this work, we show that standard ViT models can be efficiently trained at high resolution by randomly dropping input image patches. This simple approach, PatchDropout, reduces FLOPs and memory by at least 50% in standard natural image datasets such as ImageNet, and those savings only increase with image size. On CSAW, a high-resolution medical dataset, we observe a 5 times savings in computation and memory using PatchDropout, along with a boost in performance. For practitioners with a fixed computational or memory budget, PatchDropout makes it possible to choose image resolution, hyperparameters, or model size to get the most performance out of their model.

\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the WACV 2023 web page
(\url{http://wacv2023.thecvf.com/submission/})
for a discussion of the policy on dual submissions.

\subsection{Paper length}
Papers must be no longer than eight pages, not including references.
Any pages in excess of eight pages must contain ONLY references ---
no text, figures, acknowledgements, tables, etc.

{\bf There will be no extra page charges for WACV 2023.}

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\wacvfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $087.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  In addition, in means you are extremely careful if you
share URLs to source code, github repositories, project websites,
datasets, etc. The URL or other information (e.g. your github user ID)
may identify you and would violate the anonymization policy.

For example, saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \etal [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors20} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2020 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a techreport for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors20b}''.  Then submit the techreport as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the WACV 70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\etal' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \etal.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors20} to
\cite{Alpher02,Alpher03,Authors20}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.

Page numbers should be in footer with page numbers, centered and .75
inches from the bottom of the page and make it start at the correct page
number rather than the 9876 in the example.  To do this find the secounter
line (around line 33 in this file) and update the page number as
\begin{verbatim}
\setcounter{page}{123}
\end{verbatim}
where the number 123 is your assigned starting page.

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors20}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
  \begin{center}
    {\small{
\begin{tabular}{llr}
\toprule
Method & Frobnability & Accuracy \\
\midrule
Theirs & Frumpy & 30.2\%\\
Yours & Frobbly & 45.2\%\\
Ours & Makes one's heart Frob & 99.9\%\\
\bottomrule
\end{tabular}
}}
\end{center}
\caption{Results.   Ours are better. If you prefer, you can put table
captions on top of the tables instead of on the bottom.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the WACV 2023 web page
(\url{http://wacv2023.thecvf.com/submission/})
for a discussion of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

Please direct any questions to the production editor in charge of these
proceedings at the IEEE Computer Society Press:
\url{https://www.computer.org/about/contact}.

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}

Please use this template for two separate purposes. (1) If you are
submitting a paper to the second round for a paper that received a
Revise decision in the first round, please use this template to
address the reviewer concerns and highlight the changes made since the
original submission. (2) If you submit a paper to the second round,
after you receive the reviews, you may optionally submit a rebuttal to
address factual errors or to supply additional information requested by
the reviewers (this rebuttal will be seen by Area Chairs but not by
the reviewers).  In either case, you are limited to a {\bf one page}
PDF file.  Please follow the steps and style guidelines outlined below
for submitting your author response.

The rebuttal must adhere to the same blind-submission as the original submission and must comply with this rebuttal-formatted template.

\subsection{Response length}
Author responses must be no longer than 1 page in length including any references and figures.  Overlength responses will simply not be reviewed.  This includes responses where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.  Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.

\section{Formatting your Response}

{\bf Make sure to update the paper title and paper ID in the appropriate place in the tex file.}

All text must be in a two-column format. The total allowable width of the text
area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch
(0.8 cm) space between them. The top margin should begin
1.0 inch (2.54 cm) from the top edge of the page.  The bottom margin should be
1-1/8 inches (2.86 cm) from the bottom edge of the page for $8.5 \times
11$-inch paper; for A4 paper, approximately 1-5/8 inches (4.13 cm) from the
bottom edge of the page.

Please number all of your sections and any displayed equations.  It is important
for readers to be able to refer to any particular equation.

Wherever Times is specified, Times Roman may also be used.  Main text should be
in 10-point Times, single-spaced. Section headings should be in 10 or 12 point
Times.  All paragraphs should be indented 1 pica (approx. 1/6 inch or 0.422
cm).  Figure and table captions should be 9-point Roman type as in
Figure~\ref{fig:onecol}.

List and number all bibliographical references in 9-point Times, single-spaced,
at the end of your response. When referenced in the text, enclose the citation
number in square brackets, for example~\cite{Alpher02}.  Where appropriate,
include the name(s) of editors of referenced books.

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{1in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to make is resolvable in a printed copy of the response.  Resize fonts in figures to match the font in the body text, and choose line widths which render effectively in print.  Many readers (and reviewers), even of an electronic copy, will choose to print your response in order to read it.  You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use \verb+\includegraphics+, and to specify the  figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

{\small
\bibliographystyle{ieee}

\section{Introduction}

\begin{figure}[t!]
\begin{center}
\vspace{-1mm}
\includegraphics[width=1.0\columnwidth]{figures/Fig1/Fig1.pdf}
\end{center}
\vspace{-5mm}
\caption{
    \emph{PatchDropout can be used to efficiently train off-the-shelf ViTs on high-resolution images.

    }
    A tiny proportion of the input patches is often enough for accurate prediction, and if larger resolution images are used performance can increase, despite the lost information.
    Here, training with different ratios of input patches affects the model's performance on \textsc{CSAW}\xspace, a real-world medical dataset with high resolution images.

    PatchDropout results in increased predictive performance under the same computational budget.
    Using a $16 \times$ larger image but keeping $5\%$ of the patches saves computation and memory, and improves performance.
    Further improvements can be achieved by increasing the keep rate, at the expense of computation.
    A similar trend is observed across standard image classification datasets.

    }
\label{fig:fig1}
\vspace{-10mm}
\end{figure}

Vision Transformers (ViTs) \cite{dosovitskiy2020image} have been recently introduced as a viable alternative to CNNs \cite{dosovitskiy2020image, liu2021swin, matsoukas2021time, shamshad2022transformers}.
However, promises of better performance have not yet been realized in many settings due to computational bottlenecks.
For instance, ViTs require large datasets to train on \cite{dosovitskiy2020image}, though this issue has been partially solved using pre-training on large datasets  \cite{dosovitskiy2020image,caron2021emerging}.
Memory and compute requirements add to this, since the self-attention mechanism introduces an element with quadratic complexity \emph{w.r.t.}~the number of tokens.
These bottlenecks can result in long training times, and for large images such as those encountered in medical image analysis, the computational and memory demands render off-the-shelf ViTs unsuitable.

These computational issues are acute in other domains as well, \textit{e.g.}~microscopy and remote sensing, especially when native resolution is not only a desired property but a requirement for accurate predictions.
Accordingly, several works focus on making vision transformers more efficient using a plethora of different approaches, which usually involve some kind of post-processing or architectural modifications \cite{rao2021dynamicvit, wang2021not, yang2021nvit, tang2022patch, yin2022vit}.
These methods prioritize efficiency during inference, \emph{e.g.}~for embedding in mobile devices, and have been shown to reduce run-time by 30 to 50\% without compromising performance.
However, the bottleneck in network training can not be overlooked.
Few works have addressed this topic, and those that do require architecture modifications or complex training schemes which limits their use \cite{wang2020linformer,liu2021swin,wang2021pyramid,renggli2022learning,ryoo2021tokenlearner}.
Efficient ViT training remains an important problem, especially for applications requiring large images, as all but the largest institutions are limited by computational resources to train ViTs.

In this work, we ask a fundamental question.
\textit{Are all input patches necessary during training, or can we randomly disregard a large proportion of them?}
An affirmative response entails a simple, yet efficient approach that reduces compute and memory footprint.
Our method, PatchDropout, randomly drops input tokens and results in
up to $5\times$ reduction in memory and compute during training when using high-resolution images, without compromising the model accuracy (Figure \ref{fig:fig1}).
This can be achieved with off-the-shelf vision transformers and a minimal implementation, owing to the nature of ViTs.
Furthermore, we show that given a fixed memory and computational budget, PatchDropout makes it possible to choose image resolution, hyperparameters, or model
size to get the most performance out of the model.
We conduct experiments on \textsc{CSAW}\xspace, a real-world medical dataset with high resolution images, and further validate our proposed method using three mainstream datasets: \textsc{ImageNet}\xspace, \textsc{CIFAR100}\xspace and \textsc{Places365}\xspace.
Through these experiments we show that:

\begin{itemize}
    \item We can randomly discard image patches during training without compromising performance and improve efficiency from $2 \times$ up to $5.6 \times$, depending on image size (see Figures \ref{fig:fig1} and \ref{fig:main_results}).

    \item Given the same computational budget, up-scaling the images and/or utilising a larger ViT variant while discarding a fraction of input tokens can improve the model's accuracy (see Table \ref{tab:token_number} and Table \ref{tab:model_variant}).
    \item PatchDropout can act as a regularization technique during training, resulting in increased model robustness (see Figure \ref{fig:robustness}).
\end{itemize}

These findings, along with additional ablation studies, suggest that PatchDropout can economize ViTs, allowing their utilization on high-resolution images, with potential gains in accuracy and robustness.
We will release the code to reproduce our experiments shortly.

\section{Related Work}
\label{related}

Several studies have examined how to obtain a lighter vision transformer model using an existing well-trained one to improve inference efficiency using \emph{e.g.}~pruning or a teacher for distillation.
DynamicViT \cite{rao2021dynamicvit} adds a prediction module for estimating the importance score of each patch progressively. The training is assisted by knowledge distillation and the patches whose contribution are minimal to the final prediction are pruned during inference.
Another pruning method, PatchSlimming \cite{tang2022patch} identifies less important patches from the last layer and removes them from previous ones.
DVT \cite{wang2021not} dynamically determines the number of patches by training a cascade of transformers using an increasing number of patches and then interrupts inference once the prediction is confident.

Another line of research focuses on making the training more efficient.
Several studies attempt optimization of network architectures through artificially designed modules \cite{wang2020linformer,liu2021swin,wang2021pyramid}, among which PatchMerger \cite{renggli2022learning} and TokenLearner \cite{ryoo2021tokenlearner} are designed specifically for reducing the number of tokens.
EViT \cite{liang2022not} learns to gradually preserve the attentive tokens and fuse the inattentive ones during training which results in a 0.3\% decrease in accuracy on \textsc{ImageNet}\xspace with a 50\% increase in speed of inference.
Compared to EViT, the proposed method of this study is complementary but with a much simpler mechanism that does not require substantial modifications.

A few recent works explore the possibility of learning expressive representations by selecting a subset of patches.
MAE \cite{he2022masked}, which is designed for more efficient self-supervised pre-training, proposes dropping a high proportion of patches and subsequently infer the missing patches through an autoencoder setup. Our work takes some inspiration from MAE, however, PatchDropout can be applied to target tasks directly using standard ViTs (unlike MAE).
In \cite{hao2022learnable}, the authors augment standard ViTs with additional patches that selectively attend to a subset of patches to improve transferability of ViTs.
Finally, \cite{naseer2021intriguing} shows that ViTs are robust to random occlusions.
However, it should be noted that occlusion does not result in efficiency improvement.

\section{Methods}

\label{methods}

Transformer models were originally developed for language-related tasks \cite{vaswani2017attention}, but their self-attention mechanism has been proven useful for vision tasks as well \cite{dosovitskiy2020image, touvron2021training, liu2021swin}.
An important difference between the two tasks is that visual data, often, contains considerable redundancy or correlation in appearance \cite{he2022masked} (see Figure \ref{fig:vis_red}).
This observation leads to the following question: \textit{Can we randomly omit input image patches during training? If yes, what are the benefits of doing so?}
Here, we aim to answer these questions, showing that vision transformers can indeed be trained using a fraction of the input data and perform well, while at the same time saving a significant amount of memory and compute.
Additionally, our simple training scheme may offer some desirable regularization effects.

\begin{figure}[t]
\begin{center}
\vspace{-2mm}
\begin{tabular}{@{}c@{}c@{}}
    \includegraphics[width=0.475\columnwidth]{figures/visual_redundancy/896_kr1.png} &
    \includegraphics[width=0.5\columnwidth]{figures/visual_redundancy/Fig2_umap2.pdf}\\
\end{tabular}
\end{center}
\vspace{-3mm}
\caption{
    \emph{Redundancy in mammographic images.}
    \textbf{(Left)} An example image from \textsc{CSAW}\xspace at $448\times448$ resolution.
    \textbf{(Right)} 2D projection of the extracted patches of the left image using UMAP \cite{mcinnes2018umap}.
    The red squares represent the randomly kept patches after PatchDropout with a keep rate of 0.25.
    The patches are clustered into 2 distinct groups: the air and the breast regions.
    Surprisingly, by simply sampling uniformly, the information needed for accurate classification is retained.
    }
\label{fig:vis_red}
\vspace{-4mm}
\end{figure}

\subsection{PatchDropout} \label{method}

Our core idea relies on the fact that the spatial redundancy encountered in image data can be leveraged to economize vision transformers.
If we randomly deny a fraction of the information to the model during training, we expect a diminished impact on the model's predictive performance.
PatchDropout implements this by randomly dropping a percentage of image tokens at the input level.
More specifically, before the patch embeddings are sent to transformer blocks, a subset of tokens is randomly sampled without replacement.
Positional embeddings are added prior to the random sampling so that the corresponding position information is retained.
If a {\fontfamily{qcr}\selectfont[CLS]}\xspace token exists, it is retained.
The sampled token sequence is sent to transformer blocks in the standard manner.
The proposed method is straightforward and trivial to implement, which makes it viable to be incorporated in most ViT models without substantial modifications.

\subsection{Complexity Analysis}\label{section:complexity}

Vision transformers operate on a series of tokens, where each token corresponds to a non-overlapping image patch and is represented by a linear projection of the patch summed with a positional embedding.
In practice, an image of size $H\times W$ is tiled into $N=HW/P^2$ patches, where $P$ is the patch size and it is typically defined by the user (often $8$ or $16$).
The resulting token sequence is fed into a series of consecutive transformer blocks that update the $d$-dimensional embedding of tokens and consist of a Multi-head Self-Attention (MSA) and Multi-Layer Perceptron modules (MLP).
The MSA itself includes a series of MLP layers that model the interactions between the tokens through attention.
A final MLP layer is responsible for projecting the output to have the same dimensions as its input, ready to be processed by the next transformer block.
Given this information, we can discuss the theoretical and empirical computational complexity of vision transformers.

\begin{figure}[t]
\centering
\scriptsize
\begin{tabular}{@{}c@{}c@{}c@{}}
\textbf{Keep rate 0.5 (Theoretical)} &
\textbf{Keep rate 0.5 (Empirical)} &\vspace{2mm}
\\
\includegraphics[width=0.5\linewidth]{figures/complexity/theoretical_complexity_0.5.pdf} &
\includegraphics[width=0.5\linewidth]{figures/complexity/empirical_complexity_0.5.pdf}\vspace{3mm}\\
\textbf{Keep rate 0.25 (Theoretical)} &
\textbf{Keep rate 0.25 (Empirical)} &\vspace{2mm}
\\
\includegraphics[width=0.5\linewidth]{figures/complexity/theoretical_complexity_0.25.pdf} &
\includegraphics[width=0.5\linewidth]{figures/complexity/empirical_complexity_0.25.pdf}\\

\end{tabular}

\normalsize
\caption{
\emph{Computational savings with PatchDropout increase for larger sequence length $N$.}
We illustrate the theoretical (left) and empirical (right) relative savings in computation when using PatchDropout with keep rates 0.5 and 0.25 for different input sequence lengths and ViT models.
Different sized images with a fixed patch size 16 result in different numbers of tokens (vertical dashed lines).
As discussed in Section \ref{section:complexity}, empirical savings do not always correspond to the theoretical analysis due to various factors.
However, the trend remains consistent: as the image size, and thus the number of input tokens $N$ increases, the observed computational savings approach the theoretical minimum $\lim_{N\to\infty}$.
}
\label{fig:complexity}
\vspace{-5mm}
\end{figure}

\paragraph{Theoretical complexity}
Given $L$ transformer blocks with $N$ tokens and $d$-dimensional embeddings, the computational cost of the self-attention within the MSA module is $\mathcal{O}(LN^2d)$, while the other MLP layers introduce a complexity of $\mathcal{O}(LNd^2)$.
In total, the computational complexity of a series of $L$ transformer blocks is:
\begin{equation}\label{eq1}
2LN^2d+4LNd^2.
\end{equation}

The compute is always linear to the depth $L$.
When $N\gg d$ the complexity reduces to the first term, when $N \ll d$ it reduces to the second term.

For high resolution images with small patch sizes, which is the focus of this work, the first term prevails.
This leads to a quadratic complexity with respect to the sequence length $N$.
Accordingly, removing a non-trivial portion of the input tokens can result in significant gains in compute.

\paragraph{Empirical complexity}
In practice, the observed computation cost may not exactly reflect the theoretical prediction.
A few factors can make the computational saving of PatchDropout less advantageous than the complexity analysis might suggest.
For instance, the patchifier, \ie the layer responsible for tokenizing and projecting the input image into a series of embedded tokens adds computational overhead.
The same is true for the classification head.
Nonetheless, as image size increases (and thus the input sequence length $N$), the gap between the theoretical and empirical relative computation should lessen.
This necessitates an empirical analysis of the savings in computation to confirm the theoretical predictions.

In Figure~\ref{fig:complexity} we illustrate the relative drop in computations, both according to Eq. \ref{eq1} and empirically according to the number of FLOPs.
We compare for different sequence lengths $N$ when using two keep rates of $0.5$ and $0.25$.
As discussed above, the computational saving of PatchDropout increases with increasing number of tokens $N$.
It can be seen that the drop in computation is similar to the keep rate for small $N$, but gradually converges to a quadratic savings with increasing $N$.
While the theoretical and empirical trends are similar, the empirical saving converges slower due to the additional computations discussed above.
Note that $N$ varies with image size $H\times W$ and patch size $P$.
The default image size across multiple vision benchmark datasets is $224\times224$.
At this scale, the relative computation stays close to the value of its keep rate -- the images are not large enough to benefit much from the quadratic savings.
However, many real-world tasks demand high-resolution images, like the ones typically encountered in the medical domain.
Here, we observe large computational savings as the sequence length increases.

Finally, an important factor to note is that the embedding dimensions $d$ varies between different ViT variants, affecting their computational savings.
In general, the relative computation of PatchDropout on smaller ViTs (with smaller $d$) decreases faster compared to larger ones.

\section{Experimental Setup}
\label{experiments}

We evaluate PatchDropout across a number of different ViT variants and datasets.
As representative ViT models, we selected \textsc{DeiT}s\xspace \cite{touvron2021training} of different capacity and \textsc{SWIN}s\xspace \cite{liu2021swin}, which are ViT variants that scale linearly with respect to the input sequence length by design.
For datasets, we selected three standard benchmark image classification datasets and a real-world medical dataset of high-resolution images.
Below, we describe the experimental settings in detail, and in Section \ref{results} we report our findings.

\subsection{Datasets}
\paragraph{Data selection}
In this work we attempt to economize vision transformers such that they can be utilized for tasks where high-resolution images are necessary for accurate predictions.
To this end, we select a subset of 190,094 high-resolution images from \textsc{CSAW}\xspace, a population based cohort which consists of millions of mammography scans primarily developed for breast cancer tasks \cite{dembrower2019multi,matsoukas2020adding, sorkhei2021csaw}.
Here, we focus on the breast cancer risk prediction, a sensitive classification task.
The data is split at the patient-level and the validation set contains balanced classes, resulting in 152,922 training images, 3,256 validation images, and 33,916 testing images.
Furthermore, to validate the applicability of PatchDropout in other domains and on conventional image sizes, we run experiments on 3 standard image classification datasets: \textsc{ImageNet}\xspace \cite{deng2009imagenet}, \textsc{CIFAR100}\xspace \cite{krizhevsky2009learning} and \textsc{Places365}\xspace \cite{zhou2017places}.
Adhering to standard practice, we report our results on the official validation splits of \textsc{ImageNet}\xspace and \textsc{Places365}\xspace and we use 1\% of the training data for validation.
On \textsc{CIFAR100}\xspace, 2\% of the training images comprise the validation set and the results are reported on the official test set.

\paragraph{Preprocessing}
Images from \textsc{CSAW}\xspace are in DICOM format, and require several pre-processing steps which are detailed below.
Using the DICOM metadata, we re-scale the intensity values and correct any images with inverted contrast.
Following  \cite{liu2020decoupling}, certain images are excluded according to a set of exclusion criteria.
The purpose is to filter out noisy images, images with implants, biopsies and mammograms with aborted exposure.
The mammograms with cancer signs are separated from those intended for risk estimation.
More specifically, cases with examination 60 days in advance of diagnosis are excluded in order to avoid risk conflation.
The images are resized to $896\times896$, $448\times448$, $224\times224$ and $112\times112$ for the purposes of this work.

For the other datasets we only resize their images to meet the needs of our experiments, no additional pre-processing was performed.
The image resolution on \textsc{ImageNet}\xspace varies and has an average resolution of $469\times387$.
For \textsc{Places365}\xspace we used images of size $256\times256$ and for \textsc{CIFAR100}\xspace $32\times32$.
We resize these to $448\times448$, $224\times224$, $128\times128$, $112\times112$ and $64\times64$ in our experiments using bi-linear interpolation.

\subsection{Models and training protocols}

\paragraph{Model selection}
In this study, we primarily use \textsc{DeiT}s\xspace \cite{touvron2021training} which are similar in spirit and computational complexity to the original ViTs \cite{dosovitskiy2020image}.
Unless otherwise specified, the model choice is \textsc{DeiT-B}\xspace trained on $16\times16$ patches (denoted as \textsc{DeiT-B}\xspace/16), and it is trained on input size $224\times224$.
Additionally, to show that PatchDropout is agnostic to architectural selections, we run ablations using \textsc{SWIN}s\xspace \cite{liu2021swin}.
The \textsc{SWIN}\xspace architecture was designed to reduce the computational complexity of the original ViTs.
They scale linearly with respect to the input size and they inherit some of the CNN's inductive biases by design.

\paragraph{Implementation details}
All models are initialized from \textsc{ImageNet-21k}\xspace  pre-trained weights and subsequently fine-tuned on the target task.
Hyper-parameters are selected through grid search based on the result from the validation set.
We use early stopping.
The batch size is consistently 128 for all experiments, and each model is trained with an SGD optimizer with momentum set to 0.9.
In every experiment, a linear learning rate warmup is utilized for the first 2 epochs.
The learning rates are $3\times10^{-4}$ on \textsc{ImageNet}\xspace, $5\times10^{-4}$ on \textsc{CIFAR100}\xspace and \textsc{Places365}\xspace, and $10^{-4}$ on \textsc{CSAW}\xspace, following the results of our grid search.
We use a weight decay of $1\times10^{-4}$ and label smoothing \cite{szegedy2016rethinking} of 0.1.
We apply randomly resized cropping, random horizontal flipping, random rotation and color jittering as augmentations.
Unless otherwise specified, each experiment was repeated three times and we report the mean.
All the experiments are conducted with PyTorch \cite{NEURIPS2019_9015}. The number of FLOPs is counted using the fvcore toolkit \cite{fvcore}, and the allocated memory is calculated on a single GPU with a batch size of 128, unless otherwise stated.

\section{Results and Discussion}
\label{results}

\begin{table}[t]
    \scriptsize
    \centering
        \begin{tabular}{@{}c@{\hspace{2mm}}c@{\hspace{2mm}}c@{\hspace{2mm}}c@{\hspace{2mm}} c@{\hspace{2mm}}@{}c}
        \toprule
        &\textbf{Input} &\textbf{Keep rate} & \textbf{Memory (GB)} & \textbf{GFLOPs} & \textbf{AUC} \\
        \midrule
        &224 &1 & 1.46 & 17.58 & 64.71\% \\
        \midrule
        &896 &0.05 & 1.50  & 15.65 &  65.27\%\\
        &896 &0.10 & 1.65 & 30.37 &  65.59\%\\
        &896 &0.25 & 2.51 & 79.96 &  66.63\%\\
        &896 &0.50 & 5.15 & 180.64 &  67.03\%\\
        &896 &1 & 14.86 & 449.98 & 66.47\%\\
        \bottomrule
        \end{tabular}
    \caption{\emph{Performance (AUC), memory and compute savings using PatchDropout on \textsc{CSAW}\xspace.} The memory is computed with a batch size of 1 on a single GPU.}
    \label{tab:csaw_keep_rates}
\vspace{-4mm}
\end{table}

We begin this section by demonstrating that not all input patches are necessary during training -- hence, we can randomly discard a large proportion of them.
Then, we show how PatchDropout can be used not only to save memory and compute but also to improve the model's predictive performance.
Finally, we analyze the regularization effects of PatchDropout and its role as an augmentation method.
Unless otherwise stated, each experiment is repeated 3 times and we report the mean value of the appropriate metric for each dataset.
For \textsc{ImageNet}\xspace, \textsc{CIFAR100}\xspace and \textsc{Places365}\xspace, we report top-1 accuracy and for \textsc{CSAW}\xspace the exam-level AUC, where the predictions take the average score of each mammogram in an examination.

\begin{figure}[t!]
\centering
\scriptsize
\begin{tabular}{@{}c@{}c@{}}
\hspace{5mm}\textsc{ImageNet}\xspace &
\hspace{5mm}\textsc{CIFAR100}\xspace

\\

\includegraphics[width=0.5\linewidth]{figures/Fig2/imagenet_main2.pdf} &
\includegraphics[width=0.5\linewidth]{figures/Fig2/cifar100_main2.pdf} \\
\hspace{5mm}\textsc{Places365}\xspace  &
\hspace{5mm}
\textsc{CSAW}\xspace
\\
\includegraphics[width=0.5\linewidth]{figures/Fig2/places_main2.pdf} &
\includegraphics[width=0.5\linewidth]{figures/Fig2/csaw_main2.pdf} \\

\end{tabular}
\vspace{-2mm}

\caption{\emph{Not all input patches are necessary to be present.} 50\% of the input patches are sufficient to preserve model performance for image size $224\times224$: it improves efficiency $2\times$ while the performance drop is contained at only 0.17\% on \textsc{ImageNet}\xspace, 0.07\% on \textsc{CIFAR100}\xspace and 0.38\% on \textsc{Places365}\xspace. On \textsc{CSAW}\xspace, keep around half of the input patches results in 0.25\% - 0.60\% increase in AUC compared with keeping all tokens. }
\label{fig:main_results}
\end{figure}

\addtolength{\tabcolsep}{-2.5mm}
\begin{table}[t]
\footnotesize

\centering
\scalebox{0.9}{
\begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}} c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}@{}c}
        \toprule
        & \makecell{\textbf{Keep} \\ \textbf{rate} }& \makecell{\textbf{Memory} \\ \textbf{(GB)}}  & \textbf{GFLOPs} & \textbf{\textsc{ImageNet}\xspace} & \textbf{\textsc{CIFAR100}\xspace} & \textbf{\textsc{Places365}\xspace}& \textbf{\textsc{CSAW}\xspace} \\
        \midrule
        & 1   & 20.96 & 17.58 & 83.17\%  & 93.33\% & 58.05\% & 64.71\% \\
        \midrule
        & 0.9 & 0.89$\times$ & 0.90$\times$ & $-$0.03\%  & $+$0.07\% & $-$0.09\%& $-$0.30\% \\
        & 0.8 & 0.78$\times$ & 0.79$\times$ & $+$0.09\%  & $+$0.11\% & $-$0.17\%& $-$0.13\% \\
        & 0.7 & 0.68$\times$ & 0.69$\times$ & $+$0.09\%  & $-$0.10\% & $-$0.12\% & $-$0.37\% \\
        & 0.6 & 0.57$\times$ & 0.59$\times$ & $+$0.03\%  & $-$0.07\% & $-$0.16\% & $+$0.60\% \\
        & 0.5 & 0.48$\times$ & 0.50$\times$ & $-$0.17\%  & $-$0.07\% & $-$0.38\% & $+$0.29\%  \\
        & 0.4 & 0.39$\times$ & 0.40$\times$& $-$0.59\%  & $-$0.27\% & $-$0.68\% & $+$0.25\% \\
        & 0.3 & 0.30$\times$& 0.40$\times$ & $-$1.41\%  & $-$0.58\% & $-$1.21\% & $-$0.04\% \\
        & 0.2 & 0.22$\times$ & 0.20$\times$ & $-$3.04\%  & $-$1.23\% &$-$2.19\% & $-$0.82\% \\
        & 0.1 & 0.14$\times$ & 0.10$\times$ & $-$7.28\%  & $-$4.22\% & $-$4.60\%& $-$5.41\% \\
        \bottomrule
\end{tabular} }
\caption{
\emph{Performance, memory and compute savings using PatchDropout on various datasets with $224\times224$ images.}
}
\label{tab:main_tab}
\vspace{-3mm}
\end{table}

\paragraph{Are all input patches necessary during training?}
To assess the impact of PatchDropout and determine whether all tokens are necessary for training ViTs, we conduct experiments where different percentages of the tokens are presented to the model.
As illustrated in Figure~\ref{fig:fig1} and Table~\ref{tab:csaw_keep_rates}, 25\% of the tokens are enough to train an accurate model on high-resolution \textsc{CSAW}\xspace images of $896\times896$ pixels, while consuming 82-83\% less memory and compute.
Interestingly, models trained with 25\% or 50\% of the tokens outperform a model that uses all tokens. This hints at a regularization effect for PatchDropOut that we will discuss later.

In Figure~\ref{fig:main_results} and Table~\ref{tab:main_tab} we explore how this trend translates to standard image classification benchmark datasets using $224\times224$ images.
We observe that performance varies as a function of the keep rate.
In all cases, a keep rate of 50\% or larger is sufficient to maintain good performance.
When using exactly 50\% of the tokens, the performance drop is contained to only 0.17\% on \textsc{ImageNet}\xspace, 0.07\% on \textsc{CIFAR100}\xspace and 0.38\% on \textsc{Places365}\xspace.
The reduction in memory and computation are significant and similar to the keep rate.

\begin{table}[t]

    \scriptsize
    \centering
\begin{tabular}{@{}c@{\hspace{1.5mm}}c@{\hspace{1.5mm}} c@{\hspace{1.5mm}}c@{\hspace{1.5mm}}c@{\hspace{1.5mm}}c@{\hspace{1.5mm}}c@{\hspace{1.5mm}}c@{\hspace{1.5mm}}@{}c}
        \toprule
        & \textbf{Input}& \textbf{Patch} & \textbf{Keep rate}&\textbf{GFLOPs} & \textbf{\textsc{ImageNet}\xspace} & \textbf{\textsc{CIFAR100}\xspace} & \textbf{\textsc{CSAW}\xspace} \\
        \midrule
        &64 & 16 & 1 & 1.46  & 66.78\%  &87.27\% &-\\
        &64 & 8 & 0.25 & 1.46  & 70.57\% &89.77\% &-\\
        &128 & 16 & 0.25 & 1.49  & \textbf{76.25\%} &\textbf{91.30\%} &-\\
        \midrule
        &112 & 16 & 1  & 4.33 &77.65\% & 91.98\% & 63.07\%\\
        &112 & 8 & 0.25 & 4.33 & 79.11\% & 92.38\% & 60.08\%\\
        &224 & 16 & 0.25 & 4.41 & \textbf{81.02\%} & \textbf{92.50\%} &\textbf{64.87\%}\\
        \midrule
        &224 & 16 & 1 &   17.58 & 83.17\% & \textbf{93.33\%} & 64.71\%\\
        &224 & 8 & 0.25 & 17.58 & \textbf{83.43\%} & 92.71\% & 64.28\%\\
        &448 & 16 & 0.25 & 17.93 & 83.26\% & 92.20\% & \textbf{65.59\%}\\
        \midrule
        &448 & 16 & 1 &  78.57 &-& -& 66.31\%\\
        &448 & 8 & 0.25 &  78.57 &-&- & 66.13\%\\
        &896 & 16 & 0.25& 79.96 &-&- & \textbf{66.63\%}\\
        \bottomrule
        \end{tabular}

\caption{\emph{Effect of varying image size and patch size.} The impact in terms of FLOPS and performance  of changing the image size and patch size is measured using PatchDropout over multiple datasets.}
\label{tab:token_number}
\vspace{-4mm}
\end{table}

\paragraph{Can we trade the savings in memory and compute introduced by PatchDropout
for more accurate predictions?
}
In the previous analysis we saw that PatchDropout allows for significant memory and computational savings without compromising the model's performance. This saving can enable a more elaborate model selection (\eg finer grid search) or a wider range of training choices (\eg larger batch size) or a more accurate but computationally-heavy architecture.
Therefore, the next question we ask is whether we could utilize the saved memory and compute to improve the model's predictive performance while keeping the computational budget similar to the one used for the full token sequence.
Our experiments show that this can be easily achieved with two simple design choices:
\begin{enumerate}[label={(\arabic*)}]
    \vspace{-1mm}
    \item Increasing the total token sequence by (a) using higher resolution images, or (b) decreasing the patch size.
    \vspace{-1mm}
    \item Employing models with greater capacity.
    \vspace{-1mm}
\end{enumerate}

\begin{table}[t]
    \scriptsize
    \centering

        \begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{2mm}} c@{\hspace{2mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}@{}c}
        \toprule
        & \textbf{Model}  &\makecell{\textbf{Keep} \\ \textbf{rate}}& \makecell{\textbf{Memory} \\ \textbf{(GB)}}& \textbf{GFLOPS} & \textbf{\textsc{ImageNet}\xspace} & \textbf{\textsc{CIFAR100}\xspace} & \textbf{\textsc{CSAW}\xspace}\\
        \midrule
        & \textsc{DeiT-T}\xspace & 1 & 5.06 & 1.26  & 75.22\%  &86.94\% & 63.45\%\\
        & \textsc{DeiT-S}\xspace & 0.25 & 2.46 & 1.15  & \textbf{78.09\%} & \textbf{90.30\%} & \textbf{63.76\%}\\
        \midrule
        & \textsc{DeiT-S}\xspace & 1 & 10.20 & 4.61  & 80.69\% & 91.08\% & 64.62\%\\
        & \textsc{DeiT-B}\xspace & 0.25 & 5.46 & 4.41 & \textbf{81.02\%}& \textbf{92.50\%} & \textbf{64.87\%}\\
        \midrule
        & \textsc{DeiT-B}\xspace & 1 & 20.96 & 17.58  & 83.17\% & 93.33\% & 64.71\%\\
        & \textsc{DeiT-L}\xspace & 0.25 &15.34 & 15.39 & \textbf{83.81\%}& \textbf{93.97\%} & \textbf{65.31\%}\\

        \bottomrule
        \end{tabular}
\caption{\emph{Impact of training larger ViT variants with PatchDropout using $224\times224$ images.}}
\label{tab:model_variant}
\end{table}

\begin{table}[t]
    \scriptsize
    \centering
        \begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{1mm}} c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}@{}c}
        \toprule
        & \textbf{Model} & \textbf{Depth}  &\makecell{\textbf{Keep} \\ \textbf{rate}}& \makecell{\textbf{Memory} \\ \textbf{(GB)}} &  \textbf{GFLOPS} & \textbf{\textsc{ImageNet}\xspace} & \textbf{\textsc{CIFAR100}\xspace} & \textbf{\textsc{CSAW}\xspace}\\
        \midrule
        & \textsc{DeiT-B}\xspace & 12  & 1 &20.96 & 17.58  & \textbf{83.17\%} & 93.33\% & 64.71\%\\
        & \textsc{DeiT-B}\xspace & 24  & 0.5 &19.73& 17.31 & 83.06\% & \textbf{93.40\%} & \textbf{65.42\%}\\
        & \textsc{DeiT-B}\xspace & 48  & 0.25 &20.95& 17.31 & 81.46\% & 92.71\% & 65.31\% \\
        \bottomrule
        \end{tabular}
\caption{\emph{Impact of training deeper models with PatchDropout using $224\times224$ images.}}
\label{tab:model_depth}
\vspace{-4mm}
\end{table}

\subparagraph{ -- Larger images or smaller patch size}
It has been proven that ViTs perform better on larger images and smaller patch sizes \cite{dosovitskiy2020image, touvron2021training, caron2021emerging}.
However, this comes with a large memory and computational overhead due to the increased input sequence length, as described in Section \ref{methods}.
PatchDropout mitigates this cost by reducing the sequence length, allowing for the utilization of larger images and smaller patches.
Table \ref{tab:token_number} illustrates the trade-off between the model's performance and the input sequence length for different settings on various datasets.

Trading the saved compute from PatchDropout for larger images yields a large performance boost for almost all setups (compare the \textit{1$^{st}$} and \textit{3$^{rd}$} row of each group).
For example, compare \textsc{ImageNet}\xspace for images of size $128\times128$ with keep rate of 0.25 compared to the $64\times64$ images with all tokens retained.
This simple trade-off results in a nearly $10\%$ absolute increase in accuracy for similar cost.
The performance gains lessen, however, for larger computational budgets.
This trend is observed across all the datasets.

Trading the savings obtained using PatchDropout for smaller patch sizes also yields significant performance gains.
However, the gains are not as consistent as for resolution. Compare the \textit{1$^{st}$} and \textit{2$^{nd}$} row of each group.
For the natural domain, smaller patch sizes improve model performance, as expected from  \cite{dosovitskiy2020image, touvron2021training}.
On \textsc{CSAW}\xspace, smaller patch size seems to negatively affect performance, but the effect diminishes as we move to higher resolutions.

Note that, in some cases, images are up-sampled in our experiments.
In general, we notice that higher resolution and smaller patch size is usually beneficial, but not always.
We speculate that as we move away from a dataset's native resolution, larger input size and smaller patch sizes might have a negative impact on model performance due to significant information loss.
Nevertheless, PatchDropout allows for the exploration of hyperparameter settings unfeasible to reach with the full token sequence.

\subparagraph{-- Models with larger capacity}
Increasing the model capacity is another way to attain better predictions.
The memory and compute saved from PatchDropout can be spent on training larger ViT variants.
In Table \ref{tab:model_variant}, we explore this trade-off. Interestingly, the trend is that larger models using PatchDropout are consistently better than the smaller variants of equivalent cost that use all tokens.
This trend follows across all data domains, with memory efficiency improving from $1.1 \times$ to $2.1 \times$.
Natural datasets are observed to win bigger gains with PatchDropout using larger models, as compared to increased image size or reducing token size.

An alternative way to increase the model capacity is to stack more transformer blocks to a particular ViT variant.
We explore this trade-off for a fixed computational budget by increasing the model's depth and varying the keep rate.
We report the results in table~\ref{tab:model_depth}.
When doubling the model's transformer blocks with PatchDropout we observe performance boosts for \textsc{CIFAR100}\xspace and \textsc{CSAW}\xspace.
However, we note worse performance on \textsc{ImageNet}\xspace when the model becomes too deep. We attribute this to the fact that ViT architectures are optimized for \textsc{ImageNet}\xspace.

Ensembles of models are yet another alternative to obtain more accurate predictions.
We explore how computational savings from PatchDropout can be spent training additional models for use in an ensemble.
In Table \ref{tab:ensemble}, we show that an ensemble of two networks trained with 50\% keep rate consistently outperforms a single model without PatchDropout.
However, the gains diminish when lower keep rates (25\%) are traded for an ensemble for four networks.

\begin{table}[t!]
    \scriptsize
    \centering
    \begin{tabular}{@{}c@{\hspace{1.5mm}} c@{\hspace{1.5mm}}c@{\hspace{1.5mm}}c@{\hspace{1.5mm}}c@{\hspace{1.5mm}}c@{\hspace{1.5mm}}@{}c}
        \toprule
        & \textbf{\#Networks} & \textbf{Keep rate}  & \textbf{GFLOPs} & \textbf{\textsc{ImageNet}\xspace} & \textbf{\textsc{CIFAR100}\xspace} & \textbf{\textsc{CSAW}\xspace} \\
        \midrule
        & 1 & 1 & 17.58 & 83.17\% & 93.33\% & 64.71\%\\
        & 2 & 0.5 & 17.44 & \textbf{83.48\%} & \textbf{93.74\%} & \textbf{65.26\%}\\
        & 4 & 0.25 & 17.66 & 82.20\% & 93.45\% & 64.92\%\\
        \bottomrule
        \end{tabular}
    \caption{\emph{Using PatchDropout compute savings to train an ensemble.}}
    \label{tab:ensemble}
    \vspace{-4mm}
\end{table}

\paragraph{Can PatchDropout be used as a regularisation method?}
Previously, we noticed that PatchDropout can result in increased performance compared to the same settings but with all tokens, \eg in Figure~\ref{fig:main_results}.
This has some implications about its regularization effects.
Thus we ask: \textit{(i)} Can PatchDropout be used as a regularizer? and \textit{(ii)} Does PatchDropout provide robustness against information removal?

To answer the first question we run experiments treating PatchDropout as an augmentation method.
In detail, at each iteration we uniformly sample a keep rate between 0.5 and 1 which we use to randomly select a subset of image patches.
We report the results in Table~\ref{tab:regularisation} and we conclude that PatchDropout is a useful augmentation method.
It provides regularization in the sense that generalization is improved across all the datasets.
This is not entirely surprising, as PatchDropout behaves similarly to known CNN regularization methods, like cutout \cite{devries2017improved}.

If PatchDropout has some regularization benefits, a further question is: \textit{can it provide robustness against information loss?}
To address this question, we evaluate models that have been trained with all image patches and models that have been trained using PatchDropout with different drop rates.
During test time, we randomly remove image content using different drop rates.
Our results are presented in Figure~\ref{fig:robustness}.
We find that, in all cases, models that have been trained with PatchDropout exhibit increased robustness against information removal (the green curve is consistently above the blue curve).
For completeness, we also report the curves when using all tokens at test time for models that have been trained with PatchDropout (purple curve in Figure~\ref{fig:robustness}).
These results further validates the regularization effects of our method.

\begin{table}[t]
    \scriptsize
    \centering
    \begin{tabular}{@{}c@{\hspace{3mm}} c@{\hspace{3mm}}c@{\hspace{3mm}}c@{\hspace{3mm}}@{}c}
        \toprule
        & \textbf{Keep rate} & \textbf{\textsc{ImageNet}\xspace} & \textbf{\textsc{CIFAR100}\xspace} & \textbf{\textsc{CSAW}\xspace} \\
        \midrule
        & 1 & 83.17\% & 93.33\% & 64.71\%\\
        & \{0.5,1\} & \textbf{83.32\%} & \textbf{93.57\%} & \textbf{65.04\%}\\
        \bottomrule
        \end{tabular}
    \caption{\emph{PatchDropout has regularization properties}. Rather than using all tokens, training ViTs with PatchDropout using random keep rates improves generalization.

    }
    \label{tab:regularisation}
\end{table}

\begin{figure}[t]
\centering
\scriptsize

\begin{tabular}{@{}c@{}c@{}}
\hspace{5mm}\textsc{ImageNet}\xspace &
\hspace{5mm}\textsc{CSAW}\xspace
\\
\includegraphics[width=0.5\linewidth]{figures/robustness/inference_efficiency_imagenet2.pdf}
 &
\includegraphics[width=0.5\linewidth]{figures/robustness/inference_efficiency_csaw2.pdf}
\end{tabular}
\normalsize
\caption{
\emph{PatchDropout improves model robustness.}
We deny information to the model during inference by randomly dropping input patches and measuring the change in performance.
The green curve shows the model's performance when training with a percentage of the input patches and evaluating using the same keep rate.
The blue curve represents the model's performance when training using all patches but evaluating on a subset of the input patches.
For completeness, the purple curve shows training with PatchDropout and inference with all patches.
When 50\% or more of the patches are kept, this results in a minimal performance drop on \textsc{ImageNet}\xspace and increased predictive performance on \textsc{CSAW}\xspace.
The trends show that models trained with PatchDropout are more robust to missing information during inference.
}

\label{fig:robustness}
\vspace{-4mm}
\end{figure}

\paragraph{Is PatchDropout constrained by the architectural choice?}
Throughout our work we used the \textsc{DeiT}\xspace model family as they are the most suitable for the purpose of our analysis.
This however, raises the question of whether PatchDropout is effective for other architectural choices.
To answer this question we run experiments using \textsc{SWIN}s\xspace \cite{liu2021swin} which are models purposely designed to reduce the computational complexity of \textsc{DeiT}s\xspace.
\textsc{SWIN}s\xspace operate using a window shifting approach and re-assignment of positional embeddings at each block.
This, necessitates a slightly different implementation of PatchDropout.
Instead of randomly sampling image patches, we apply structured sampling where we randomly sample column and row indices for each window to obtain the intersection tokens.
This maintains the spatial relationships between tokens and enables smooth window shifting for \textsc{SWIN}s\xspace.
The corresponding relative positional biases are sampled accordingly.

We report our findings in Figure~\ref{fig:swin} when using $224\times224$ images for both \textsc{ImageNet}\xspace and \textsc{CSAW}\xspace.
We discern similar patterns with the ones in Figure~\ref{fig:main_results} but the benefits from PatchDropout are not as distinct as before.
\textsc{CSAW}\xspace exhibits small performance gains when using PatchDropout with keep rates larger than 50\% while \textsc{ImageNet}\xspace displays a 1\%  drop at 50\% keep rate.
The applicability of PatchDropout however is still valid, even for this architecture which has been developed to economize \textsc{DeiT}s\xspace by design.

\begin{figure}[t]
\centering
\scriptsize
\begin{tabular}{@{}c@{}c@{}}
\hspace{5mm}\textsc{ImageNet}\xspace &
\hspace{5mm}\textsc{CSAW}\xspace
\\
\includegraphics[width=0.5\linewidth]{figures/swin/imagenet_swin_main2.pdf}
 &

\includegraphics[width=0.5\linewidth]{figures/swin/csaw_swin_main2.pdf}
\end{tabular}
\normalsize
\caption{\emph{PatchDropout also works for SWINs.} Despite their linear scaling via the reintroduction of CNN inductive biases, PatchDropout can be applied to \textsc{SWIN}s\xspace with a keep rate of 0.5 or higher without lowering performance.}
\label{fig:swin}
\vspace{-4mm}
\end{figure}

\paragraph{Other ablation studies}
We conclude our analysis with two ablation studies aiming to assess the effectiveness of random sampling and the role of large scale pretraining.
\subparagraph{-- How important is the sampling strategy in PatchDropout?}
To assess the efficacy of random sampling, which we use in our method, compared to other sampling methods, we conduct a small ablation study on \textsc{CIFAR100}\xspace where we train models using a fixed keep rate of 0.25 but we change the sampling method.
In Figure \ref{fig:sampling} we illustrate the sampling methods we used and we report the top-1 accuracy.
The results show that the effectiveness of the proposed method is not heavily dependent on the choice of sampling strategy.
This, along with the results from \textsc{SWIN}s\xspace where we used structured sampling (see previous paragraph) indicates that PatchDropout is a general approach which can be easily incorporated into other types of ViT models.

\subparagraph{-- Is PatchDropout sensitive to the initialization strategy?}
Throughout this work we utilized models pretrained on \textsc{ImageNet-21k}\xspace as vision transformers rely on large-scale pretraining, especially when working with small datasets \cite{dosovitskiy2020image, touvron2021training, matsoukas2021time}.
But, \textit{is PatchDropout useful when random initialization \cite{he2015delving} is used?}
To answer this question we train randomly initialized models on \textsc{CSAW}\xspace and we report the results in Table~\ref{tab:rand_init}.
Indeed, PatchDropout works with randomly initialized models, although with diminishing performance gains, suggesting that the proposed method is agnostic to the initialization strategy.

\begin{table}[t]
    \scriptsize
    \centering
    \begin{tabular}{@{}c@{\hspace{3mm}} c@{\hspace{3mm}}c@{\hspace{3mm}}@{}c}
        \toprule
        & \textbf{Keep rate}  & \textbf{\textsc{ImageNet-21k}\xspace init.} &\textbf{Random init. } \\
        \midrule
        & 1 & 64.71\% & 59.32\% \\
        & 0.50 & $+$0.29\% & $+$0.16\% \\
        & 0.25 & $+$0.16\% & $-$0.90\%\\
        \bottomrule
        \end{tabular}
    \caption{\emph{Impact of \textsc{ImageNet-21k}\xspace initialization for PatchDropout on \textsc{CSAW}\xspace.}}
    \label{tab:rand_init}

\end{table}

\begin{figure}[t]
\centering
\scriptsize
\begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{}}
   & Original &
   Uniform &
   Structured&
   Cropping&
   Random
\\
&\includegraphics[width=0.17\linewidth]{figures/sampling/original.png} &
\includegraphics[width=0.17\linewidth]{figures/sampling/224_kr0.25_ps16_samplinguniform_sampling.png} &
\includegraphics[width=0.17\linewidth]{figures/sampling/224_kr0.25_ps16_samplingrandom_indices.png} &
\includegraphics[width=0.17\linewidth]{figures/sampling/224_kr0.25_ps16_samplingcutout_sampling.png} &
\includegraphics[width=0.17\linewidth]{figures/sampling/224_kr0.25_ps16_samplingrandom.png} \\
  & 93.33\% &
  91.84\% &
  92.94\% &
  92.46\% &
  92.50\%

\end{tabular}
\normalsize
\caption{\emph{Impact of different patch sampling strategies}. Accuracy on \textsc{CIFAR100}\xspace is reported for keep rate 0.25 at  $224 \times 224$  resolution for various patch sampling strategies.}
\label{fig:sampling}
\vspace{-4mm}
\end{figure}

\section{Conclusion}
In this work, we rely on the fact that the spatial redundancy encountered in image data can be leveraged to economize vision transformers and we propose a simple yet efficient method, PatchDropout.
By dropping input tokens at random, our method results in significant memory and computation reduction, especially on high-resolution images.
In addition, we demonstrate how the saved compute introduced by PatchDropout can be exchanged for better predictive performance under the same memory and computational budget.
Finally, we show that PatchDropout can act as a regularization technique during training, resulting in increased model robustness.
PatchDropout requires minimal implementation and works with off-the-shelf vision transformers.
We believe that PatchDropout should be an essential tool in every practitioner’s toolkit
to reduce the computational and memory demands in transformer training.

\paragraph{Broader Impact}

Reducing computational requirements can help to democratize deep learning by making it cheaper to train models.
Achieving an equitable outcome for vulnerable and disadvantaged groups, who might lack access to sufficient funding and resources -- including but not limited to small academic groups, hospitals, and companies, necessitates a multitude of solutions.
Apart from social benefits, one might consider the positive impacts with regards to climate as well.
Data centers worldwide account for a substantial portion of energy consumption and GHG emissions which can be mitigated by shrinking computation during model development if applied widely. Despite our efforts, training state-of-the-art networks such as ViTs is still computationally expensive and thus has significant carbon footprints.

\paragraph{Acknowledgements.}
This work was partially supported by MedTechLabs (MTL), the Swedish Research Council (VR) 2017-04609, Region Stockholm HMT 20200958, and Wallenberg Autonomous Systems Program (WASP). The computations were enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.

Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)


Evidence for Pebbles in Comets

Abstract: When the EPOXI spacecraft flew by Comet 103P/Hartley 2, it observed large particles floating around the comet nucleus. These particles are likely low-density, centimeter- to decimeter-sized clumps of ice and dust. While the origin of these objects remains somewhat mysterious, it is possible that they are giving us important information about the earliest stages of our Solar System's formation. Recent advancements in planet formation theory suggest that planetesimals (or cometestimals) may grow directly from the gravitational collapse of aerodynamically concentrated small particles, often referred to as "pebbles." Here we show that the particles observed in the coma of 103P are consistent with the sizes of pebbles expected to efficiently form planetesimals in the region that this comet likely formed, while smaller pebbles are may be expected in the majority of comets, whose chemistry is often indicative of formation in the colder, outer regions of the protoplanetary disk.

\section{Introduction}
\label{sec:intro}

When the Epoxi spacecraft flew by Comet 103P/Hartley 2 it observed large
particles floating around the comet's nucleus \citep{AHearn.etal.2011}.  Based
on best estimates of the albedo of these objects, they appear to have a rather
steep size distribution in which the largest particles are thought to be
$\sim$ 10 cm in size \citep{Kelley.etal.2013,Harmon.etal.2004}.  These particles
are larger than has been generally detected in the coma of other comets.  While
it is possible that the size of these particles is determined simply by the
erosive process inside the comet \citep{Meech.Svoren.2004}, another intriguing
possibility is that these particles may be left over relics of the formation
process.

In the past it has generally been assumed that the icy bodies in the outer
solar system were all built up in a hierarchical fashion.  In this picture,
km-sized comets are built from binary collisions of smaller bodies, while the
larger Kuiper belt objects and even the cores of the giant planets were formed
by accretion of these small ``cometesimals''
\citep{Stern.1996,Stern.Colwell.1997,Kenyon.2002,Kenyon.etal.2008}.  However,
there are important challenges in forming comets via this mechanism.  First, it
may not be possible to grow to binary collisions up to km-sized objects.  As
particles grow their sticking efficiency is reduced, possibly leading to a
regime in which collisions are more likely to lead to ``bouncing'' rather than
accretion \citep{Zsom.etal.2010,Guttler.etal.2010}.  Additionally, as small
micron sized particles grow into macroscopic mm to m-sized particles they
attain larger relative velocities, increasing the likelihood that collisions
will be destructive \citep{Blum.Wurm.2000,Blum.Wurm.2008}.  And while
mass-transfer precesses may allow lucky particles to grow beyond these
barriers, the timescale for formation of planetesimals becomes long enough to
become problematic for subsequent planet formation \citep{Windmark.etal.2012}
Additionally, even if these barriers can be overcome and planetesimals can be
formed, to grow the larger Kuiper belt objects in the time alloted by this
process the disk must have been extremely dynamically cold and 100 to 1000
times more massive than it is today \citep{Kenyon.etal.2008}.

However, more recently there have been theoretical and observational reasons to
suggest that instead of forming in this bottom up manner, planetesimals may
form directly by the gravitational collapse of a dense cloud of small particle
embedded in the protoplanetary disk (the gravitational instability or GI
hypothesis).  In particular, recent breakthroughs in theory and computer
simulation have demonstrated that under reasonable conditions particles can be
concentrated to the point that they will gravitationally collapse, as suggested
by the GI hypothesis \citep[see reviews
by][]{Chiang.Youdin.2010,Johansen.etal.2014}.  In these models particles that
are small enough to have their orbits strongly perturbed by aerodynamic drag,
yet large enough to still be decoupled from the gas (mm to m sized objects
depending on the gas disk properties) can self-clump, forming gravitationally
bound objects of 10-1000 km in size
\citep[e.g.][]{Johansen.etal.2007,Johansen.etal.2015}.

Furthermore, new observational signatures may support the idea that
planetesimals form via GI rather than through hierarchical
growth. For example, the size distribution of planetesimals in the asteroid
belt appears to be inconsistent with the predictions of classical planetesimal
collisions, and instead may be showing us that the larger asteroids had to
directly form as relatively large objects (\citet{Morbidelli.etal.2009}, although see \citet{Weidenschilling.2011} for an alternative interpretation).
Additionally, the delicate wide binaries in the Kuiper Belt are unlikely to be
made by normal processes such as collisions or binary exchanges, instead they
are most easily made by a collapsing, fragmenting gravitationally bound pebble
``cloud'' \citep{Nesvorny.etal.2010,Parker.etal.2011}.
Furthermore, if one combines the observed activity of comets with the strength the dusty surface layers of comets, the dust on cometary surfaces must be in the form of relatively large particles in order for water sublimation to power cometary activity \citep{Skorov.Blum.2012,Blum.etal.2014}.  All of various lines of evidence combined paint a consistent picture in which small icy bodies in the outer-Solar system formed from collapsing clouds of small ``pebbles''.

If Kuiper belt objects did indeed form from the gravitational collapse of
clouds of small pebbles, there may be signatures of these formative pebbles
extant in comets today.  While some pebbles may destructively fragment during
the planetesimal formation process, so long as the initially formed
planetesimal was under 100 km in radius it is expected that most of the pebbles
are unlikely to collide at high enough speed to cause fragmentation
\citep{Jansson.Johansen.2014}.  Additionally, while thermal processing will
alter the cometary surface, creating lag deposits that mask the underlying
structure of the comet, only the surface layers of the comet likely have been
significantly altered by thermal processing \citep{Mumma.etal.1993}.  This may
leave pristine material beneath the lag layer.  With this in mind, in this
\emph{Note} we are interested in investigating the speculative proposition that
the large particles observed in the coma of Hartley 2 may be remnants of the
initial pebbles which formed the comet.  To this end we will address the issue
of the expected size of pebbles in different regions of the protoplanetary disk
and the possibility of predictable trends.

In this paper we look at the large particles in the coma of comet Hartley 2 and
show that their sizes are consistent with the size of pebbles expected to
efficiently form planetesimals.  In section \ref{sec:model} we describe from a
theoretical standpoint how large pebbles need to be in protoplanetary disks to
be concentrated by well-understood processes.  In section \ref{sec:comets} we
use the chemical composition of comet Hartley 2 to estimate its formation
location so that we can place the sizes of the particles in it coma in context
with the theoretical expectations.  Finally, in section \ref{sec:conclusions}
we summarize our results and discuss the implications and what future data may
help further elucidate the formation mechanism of comets.

\section{Theoretical Expectations of Pebble Sizes} \label{sec:model}

There are three main ideas for how pebbles can be concentrated to the degree
necessary for gravitational collapse in protoplanetary disks, turbulent eddies,
pressure bumps and vortices, and the streaming instability \citep[see][for a
review]{Johansen.etal.2014}.  Each of these processes will only work on
particles within a limited range of sizes, determined by the particles' Stokes
numbers ($\tau$).  The Stokes number is defined as $\tau \equiv t_s \Omega$,
where $t_s$ is the stopping time of the particle due to aerodynamic drag and
$\Omega \equiv \sqrt{G M_*/r^3}$ is the Keplerian orbital frequency around a
star of mass $M_*$ at a heliocentric distance $r$. The stopping time is
\begin{equation}
  t_s =
   \left\{
      \begin{array}{ll}
		  \frac{a \rho_s}{c_s \rho_g},& {\rm if}~a < \frac{3}{2}\lambda\\
		  \frac{2 a^2 \rho_s}{3 c_s \rho_g}, & {\rm otherwise}
	  \end{array}
   \right.
\end{equation}
\citep{Adachi.etal.1976} where $\lambda$ is the gas mean-free-path, $c_s$ is
the sound-speed, $\rho_s$ is the density of the solid particle, and $\rho_g$ is
the local gas density.

Turbulent eddies should concentrate particles whose stopping times are
comparable to the eddy turnover times, which corresponds roughly to $\tau\sim
10^{-5}-10^{-4}$ in typical protoplantary disks
\citep[e.g.][]{Cuzzi.etal.2008,Cuzzi.etal.2010}, however \citet{Pan.etal.2011}
found that strong clustering of particles in this size range may be too
difficult to form the number of required planetesimals.  Particle concentration
in pressure bumps
\citep[e.g.]{Whipple.1972,Haghighipour.Boss.2003,Kretke.etal.2009}, and vortices
\citep[e.g.]{Barge.Sommeria.1995,Lyra.etal.2008} is most effective for
particles with $\tau\sim 1$.  The streaming instability
\citep{Youdin.Goodman.2005,Johansen.etal.2007}  was shown to effectively
concentrate particles with $\tau$ in the range of between $10^{-2}$ and $1$
\citep{Bai.Stone.2010,Carrera.etal.2015}.  As the streaming instability is a
linear instability that has been robustly shown to function under physically
reasonable conditions, for the remainder of this paper we will take the
streaming instability range as the size scale of interest.

To determine the sizes of particles susceptible to the streaming instability,
we must know the midplane gas density and temperature profiles in
protoplanetary disks.  Unfortunately, these parameters cannot currently be
measured directly in the regions where comets are thought to have been formed
($\sim$5 to $\sim$ 50 AU), and theoretically depend upon parameters that are
expected to vary over the disk lifetime (such as the mass-accretion rate
through the disk) and/or are generally poorly constrained (such as the grain
opacity and the disk viscosity).  However, we can construct a reasonable
fiducial model based upon existing constraints and discuss how our results are
sensitive to our assumptions.

For the protoplanetary disk structure we assume a disk heated
by a combination of viscous heating and stellar irradiation.  We use the models
of \citet{Bitsch.etal.2015}, which utilize a full radiative transfer model to
calculate the 2D thermal structure of an axisymmetric protoplanetary disk.
This model has the advantage of more accurately calculating the temperature
profiles than more standard 1+1d or 2 layer calculations
\citep[e.g.][]{Chiang.etal.2001,DAlessio.etal.2005,Kretke.Lin.2010}, particularly in regions that may be self-shielding.

\begin{figure}
	\includegraphics[width=\textwidth]{TSigma.pdf}

	\caption{The disk surface density (upper panel) and temperature (lower panel) profiles assumed in this paper.  The disks have mass accretion rates of $5\times 10^{-8}$, $10^{-8}$ (fiducial), and $5\times 10^{-9}$ $M_\odot \rm{yr}^{-1}$ in red, green and blue, respectively.  The dashed-curves indicates how we modify the surface density to account for an exponential fall off as described in Eq.~\ref{eq:cutoff}.
	\label{fig:TSigma}}
\end{figure}

The green curves in figure~\ref{fig:TSigma} show the surface density and temperature profile for our fiducial disk model.  It has a mass accretion rate of $\dot{M} = 10^{-8} M_\odot {\rm yr}^{-1}$ around a 1 Myr old star.
The disk has a viscosity $\nu=\alpha c_s^2 \Omega^{-1}$ where $\alpha = 5.4\times 10^{-3}$.
We note that \citet{Bitsch.etal.2015} assume a steady-state accretion profile throughout the disk.  This means that the gas surface density, $\Sigma$, is directly determined by the mass accretion rate and viscosity by $\dot M = 3\pi\nu\Sigma$ \citep{Pringle.1981}.
The other solid curves in figure~\ref{fig:TSigma} show how the surface density and temperature depend on the mass-accretion rate. As accretion disks are expected to lose mass over time, these higher ($5\times 10^{-8}$ $M_\odot \rm{yr}^{-1}$) and lower ($5\times 10^{-9}$ $M_\odot \rm{yr}^{-1}$) mass accretion rates can be thought of as earlier and later evolutionary stages of the disk, respectively.

This steady-state assumption is expected for the inner region of any viscously evolving disk \citep{Pringle.1981}.  However, real disks do not extend forever, and as one enters into the outer region of the disk viscous evolution will lead to a smooth decrease in the disk surface density.  Observations of disks suggest that an exponential cut-off is an appropriate approximation for the outer disk surface density profile \citep{Andrews.etal.2010}.  This cutoff will modify the surface density of the disk significantly, but it will only make minor changes to the disk temperature profile.  That is because the thermal structure of the outer region of the protoplanetary disk is dominated by the passive re-radiation of the intercepted stellar light, a process with only very weak dependencies on the disk surface density.
Therefore we approximate the disk parameters in a disk with an exponential cutoff as the dashed curves in figure~\ref{fig:TSigma}, where the surface density is
\begin{equation}
	\Sigma = \frac{\dot{M}}{3\pi\nu}\exp\left(\frac{-r}{50\,{\rm AU}}\right).
	\label{eq:cutoff}
\end{equation}
and we assume the temperature profile of this disk is unchanged.

In figure \ref{fig:size} we show the sizes of such pebbles in our fiducial
disk, assuming $\rho_s = 0.5\,{\rm g\,cm}^{-3}$. The solid curves show the size of particles with a fixed $\tau$ (ranging from $10^{-2}$ to 1) in our fiducial disk with $\dot{M}=10^{-8} M_\odot\,\rm{yr}^{-1}$.  This range of $\tau$ is the size range most susceptible to the streaming instability, so we highlight it with gray shading.
The dashed-curves (and light gray region) shows how the Stokes numbers would vary in the disk if we modify our fiducial disk by assuming an exponential cutoff.
This figure demonstrates how the size of ``pebbles'' decreases with heliocentric distance in normal protoplanetary disks.
The sizes of those ``pebbles'' is sensitive to the disk properties.
In figure \ref{fig:size2} we highlight the same range of $\tau$ in disks with higher ($5\times 10^{-8}$
$M_\odot \rm{yr}^{-1}$)  and lower ($5\times 10^{-9}$ $M_\odot \rm{yr}^{-1}$) mass accretion rates, corresponds to earlier and later evolutionary stages of the disk, respectively.  From figure~\ref{fig:TSigma} we can see that the largest difference between these disks is the change in the surface density, however, at any given time, larger pebble are expected to be incorporated into comets in the inner disk as compared to the outer disk.

\section{``Pebbles'' in Hartley 2} \label{sec:comets}

In order to determine the aerodynamic properties of these pebbles, one must
know three things: the size and density of the pebbles, the properties of the gaseous
protoplanetary disk, and the location in the disk where the comet formed.

\citet{Kelley.etal.2013} calculated the size of the particles in the coma based
on the observations of individual particles.  They found that the maximum
particle radius of 30 cm if one assumes a bright, icy albedo ($A_p=0.67$) and a
maximum radius of 4 m if one assumed a dark, cometary albedo ($A_p=0.049$),
with a steep size distribution of $dn/da \propto a^{-4.7}$ down to the
detection limit.  \citeauthor{Kelley.etal.2013} take the dark solution to be
unlikely for a number of reasons.  First, if the particles are dark then the largest particles in the size distribution are 8 meters in diameter, which likely should have
been resolvable.  Additionally, the dark model implies that the coma consists of 0.6-14\% of the mass of the nucleus, which likely violates the constraints on the amount of mass lost each orbit \citep{Thomas.etal.2013}.  Furthermore, there is substantial amount of water vapor emitted isotropically around the coma, consistent with water sublimation from large particles in the coma.
However, we will include the dark solution for completeness and it
may be considered an extreme upper limit on the particle sizes.

We note that due to the long estimated lifetimes \citep{Beer.etal.2006} of
these ice balls and relatively high velocity of these particles
\citep{Hermalyn.etal.2013}, they likely did not sublimate enough to
significantly change their size between the time of ejection and detection.
Therefore we assume the observed size distribution is the same as the size
distribution at launch.  Additionally, while the turn-over size is not directly
observed, the presence of a turn over near the detection limit can be inferred
in the data because otherwise unresolved particles would cause a diffuse
emission that would have been detected \citep[see][for
details]{Kelley.etal.2013}.  Therefore in our plots we assume that the
differential particle size distribution is flat between the detection limit
($10^{-12}\,{\rm W\,m^{-2}\,\mu m^{-1}}$) and a factor of two below the
detection limit. This leads to most of the mass of the pebbles having sizes of
either 3 to 5 cm or 40 to 60 cm assuming a bright or dark albedo, respectively.

\begin{figure}[t!]
	\includegraphics[width=\textwidth]{pebblesize_fig1.pdf}

	\caption{The gray shaded region indicates the size range of particles with density $\rho_s = 0.5\,{\rm g\,cm}^{-3}$ susceptible to concentration via the streaming instability in the nominal disk.  The black curves indicate curves of constant $\tau$, of 1, 0.3, $10^{-1}$, $3\times 10^{-2}$, $10^{-2}$.  The dashed curves indicate how the $\tau$ values would change if we modify the surface density by adding an exponential cutoff as in Eq.~\ref{eq:cutoff}.  The color gradients within the blue and red boxes indicate the range of sizes of particles observed in the coma of comet Hartley 2 if a bright or dark albedo is assumed, respectively.  The shading corresponds to the normalized fraction of the pebble mass in each size bin.  The boxes are placed at a temperature range consistent with the comet's chemistry as described in Sec.~\ref{sec:comets}.
	\label{fig:size}}
\end{figure}

It is extremely difficult to determine the formation location of any comet.
Comet 103P is a Jupiter family comet (JFC), and from the inclination
distribution of this population it is clear that the most likely immediate
source region of these comets is the Kuiper Belt region \citep[R $>$ 30
AU,][]{Mumma.etal.1993}. However, modern models of solar system evolution
predict that there was significant dynamical mixing early on
\citep{Levison.etal.2008} so that planetesimals from the giant planet forming
region (5-14 AU) were scattered both into the Oort cloud and the scattered
disk, some of which later evolved on to JFC orbits.  Indeed, the varied
composition of JFCs (such as the wide range of D/H ratios
\citet{Hartogh.etal.2011,Altwegg.etal.2015}) supports the idea of mixing.

Fortunately, we have additional constraints on the formation location of this
comet---its composition.  Comet 103P has a D/H ratio of $1.4\times 10^{-4}$,
similar to that of the Earth \citep{Hartogh.etal.2011}.  This make it seem
reasonable that comet 103P formed closer to the sun than most other well
studied comets that have higher D/H ratios \citep{Delsemme.1998}.  Additionally,
comet 103P also has a notably high CO$_2$/CO ratio of around 100:1, compared to
the 1:1 or less more typical of other comets which have a high enough CO$_2$
ratio to be observed, suggesting that it may have formed significantly interior
to the CO snow line.  This is again in contrast to most well-characterized
comets \citep{AHearn.etal.2012}.  We would like to use this large CO$_2$/CO
ratio to give precise constraints on the comet's formation location.
Unfortunately, the exact pathway for CO$_2$ formation in comets is still not
well understood, and different mechanisms give rise to different formation
locations for 103P \citep[e.g.][]{Garrod.Pauly.2011,Noble.etal.2011}.  One
possibility is that the comet formed near or just outside of the CO$_2$ snow
line (at a formation temperature around 42-52K) \citep{Oberg.etal.2011}.
However, CO$_2$ is not prevalent in the gas phase, so direct condensation of
CO$_2$ may not actually be that efficient at producing CO$_2$ ice.
Therefore, another possible pathway for CO$_2$ ice production is by the
conversion of CO trapped in amorphous water ice.  This process has been
demonstrated to be efficient even up to 77 K \citep{Yokochi.etal.2012}.
Therefore, we will take the stance that the high CO$_2$/CO suggests that the
comet most likely formed in a region of the protoplanetary disk with a
temperature between 80 and 40 K.  We note that these temperatures are warmer
than the spin temperature of water measured in Hartley 2
\citep{Kawakita.etal.2004a}, however, that ratio may be modified by the
desorption process, therefore may not be reliable indicator of formation
location \citep{vanDishoeck.etal.2014}.

In figure \ref{fig:size} we overlay the size distribution of particles observed
in the coma of Hartley 2 at the range of temperatures we believe this comet
likely formed, between 80 and 40 K.  With the preferred ``bright'' albedo, we
find the particles would be consistent with a $\tau$ range from
$2\times10^{-2}$ to 1, consistent with expectations for the streaming
instability.  If the ``dark'' solution turns out to be correct, then the
largest particles in the coma are would likely have been larger than $\tau=1$, and thus is it not understood how particles of that size could have formed planetesimals.
The majority of the mass is still in the expected range
(particularly if the comet formed relatively close to the sun) so these large
particles could be aggregates of primordial pebbles, but more detailed
consideration would likely be necessary.  While the Stokes number of pebbles is
dependent on the disk properties, for our high and low mass-accretion rate disk
models in figure~\ref{fig:size2} the bright pebbles still agree quite well with
what is needed for the streaming instability.

\begin{figure}
	\includegraphics[width=\textwidth]{pebblesize_fig2.pdf}

	\caption{The size range of particles susceptible to concentration via the streaming instability in a high-mass accretion rate on the left, with a low-mass accretion rate on the right.  The curves and gradients are similar to those in Figure~\ref{fig:size}.
	\label{fig:size2}}
\end{figure}

If we assume that comets that formed farther from the sun have a similar range
in $\tau$ to theses pebbles, then they would have a maximum pebble size of
$\sim$5 cm, but, if the size distribution is similar to that of Hartley 2,
most of the mass would be in smaller, sub-centimeter sizes.  Radar
depolarization has indicated that there must be a significant population of
coma grains larger than 2 cm in size in comet C/2001 A2 (LINEAR)
\citep{Nolan.etal.2006}.  The spin temperature of this comet is colder than
that of Hartley 2 \citep{Kawakita.etal.2004a,DelloRusso.etal.2005}, but
unfortunately there is neither a D/H ratio nor a CO/CO$_2$ ratio to narrow
down the formation location.  In general, most comets do not have signatures
of such large particles.  There are a few exceptions.  For instance
\citet{Reach.etal.2000} found particles as large as 5 cm in the tail of comet
Encke a comet which does have detectable CO$_2$ \citep{Reach.etal.2013}, but
does not have detectable CO, although it is not clear how significant thermal
processing may be obsecuring the chemical composition
\citep{Fernandez.etal.2005}.  Additionally, other detections of large particles
consist of mm or up to cm sizes
\citep[e.g.][]{Harmon.etal.2004,Trigo-Rodriguez.etal.2008,Trigo-Rodriguez.etal.2010}.
This general lack of very large particles appears consistent with the idea that
these comets (which more likely formed in the outer regions of the Solar System
due to the D/H and CO/CO$_2$ ratios) should have been formed from smaller
pebbles.

\section{Conclusions and Discussion}\label{sec:conclusions}

Comet Hartley 2 is observationally a bit of an outlier in two different and
seemingly unrelated facets: it both is the comet with the largest particles
observed in its coma and it has chemistry indicating that it formed at a warmer
location than typical comets.  Given that current theoretical models suggest
that comets may have formed from the accretion of ``pebbles'' with Stokes
numbers of $10^{-2}$ to 1, we postulate that these two observations are not in
fact independent.  The large particles observed in the coma of Hartley 2 are
consistent with this size of pebbles for a range of reasonable disk models
given the likely formation location of the comet based on its chemistry.
Therefore, while we cannot conclusively state that these indeed are primordial
pebbles, it is at least an intriguing possibility that we may be seeing some
remnant of these initial pebbles.

If this is true then we expect a trend in constituent pebble size commensurate
with their formation location.  Therefore, we expect that the majority of
comets, which have chemistry consistent with formation in the outer regions of
the protoplantary disk, will have smaller pebbles than those seen in Hartley 2.
We note that much smaller pebbles cannot explain the activity of comets if
their activity is driven purely by water, however as these comets likely have
significant amount of CO this may not prove problematic
\citep{Blum.etal.2014,Blum.etal.2015}.  For example, based on the high D/H
ratio \citep{Altwegg.etal.2015} and the N$_2$/ CO ratio \citep{Rubin.etal.2015}
of comet 67P/Churyumov–Gerasimenko, we may expect that it formed in the
colder-outer regions of the Solar System and thus would have been likely formed
from small, perhaps cm or sub-cm sized pebbles (although we note that the high tensile strength of small particles may require that the pebbles be at least cm in size to explain the observed activity levels \citep{Gundlach.etal.2015}).
Indeed, there previous detections of dust in the coma of 67P suggest that the
tail and coma is dominated by large 100$\mu$m-mm sized particles
\cite{Kelley.etal.2008,Bauer.etal.2012,Rotundi.etal.2015} although there were
also detections of larger particles, in cm to meter sizes
\citep{Rotundi.etal.2015,Sierks.etal.2015}.  Under the arguments presented in
this paper we believe these larger, dusty objects are aggregates, pieces of the
cometary crust crust that have been ejected rather than primordial ``pebbles.''
Similarly, we expect the ``goose bumps'' observed on the surface of 67P
\citep{Sierks.etal.2015}
are not signatures of their component pebbles but instead have been formed by
some other mechanism modifying the cometary surface.

\section{Acknowledgments}
We would like to thank M.~Mumma, P.~Weissman and M.~Kelley for useful
discussions We would also like to thank J\"urgen Blum and anonymous referee for
their thoughtful suggestions, which have improved this manuscript.  This work
is supported by the Goddard Center for Astrobiology (a NASA Astrobiology
Institute).

Earth and Planetary Astrophysics (astro-ph.EP)

Evidence for Pebbles in Comets