<commit_before>\documentclass[preprint]{aastex}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\ifxetex
  \usepackage{fontspec,xltxtra,xunicode}
  \defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase}
\else
  \ifluatex
    \usepackage{fontspec}
    \defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase}
  \else
    \usepackage[utf8]{inputenc}
  \fi
\fi
%\usepackage{aastex}
\usepackage{graphicx}
\makeatletter
\ifxetex
  \usepackage[setpagesize=false, % page size defined by xetex
              unicode=false, % unicode breaks when used with xetex
              xetex,
              colorlinks=true,
              linkcolor=blue]{hyperref}
\else
  \usepackage[unicode=true,
              colorlinks=true,
              linkcolor=blue]{hyperref}
\fi

\newcommand{\todo}[1]{{\color[rgb]{0, .5, .5} TODO: #1}}

\begin{document}

\title{An Algorithmically-Derived Catalog of Bubbles in the Galaxy}
\shortauthors{Beaumont et al}
\author{<NAME>$^{1,2}$, <NAME>$^2$, <NAME>$^1$}
\affil{$^1$Institute for Astronomy, University of Hawai'i, 2680 Woodlawn Drive, Honolulu HI 96822;  <EMAIL>}
\affil{$^2$Harvard-Smithsonian Center for Astrophysics, 60 Garden St., Cambridge MA 02138}

\section{Introduction}
\label{sec:intro}

\subsection{Interstellar Bubbles}
\label{sec:bubbles}

\subsection{Previous Surveys}
\label{sec:bubbles}

\begin{figure*}
\includegraphics[trim= .7in 0 0 0, clip]{gallery}
\caption{Different astrophysical objects in the MWP catalog. a) ``Canonical'' wind-blown bubbles and HII regions. b) shells
without 8 $\mu$m PAH emission (likely supernovae and planetary nebulae). c) 24 $\mu$m-dark filaments and globules. d) Other structured
ISM, that isn't clearly a stellar-cleared}
\label{fig:gallery}
\end{figure*}

\subsection{Manual and Automatic Classification in Astronomy}
\label{sec:benefits}

In terms of accuracy, humans still outperform computers in most image-based pattern recognition tasks. Because of this, morphologically complex
structures in the ISM (including supernova remnants, outflows, bubbles, HII regions, molecular and infrared dark clouds, and planetary nebulae) are still
traditionally cataloged manually. Human classification has several disadvantages, however.

First, human classification is time consuming, and people-hours are a limited resource. Even by enlisting large communities of citizen scientists, data from next generation surveys will be too large to search exhaustively. For example, the $>$ 35,000 citizen scientists of the Milky Way Project classified roughly 45 GB of image data from the GLIMPSE I and II surveys. Current and next-generation astronomical datasets are many thousands of times larger than this, suggesting tens of millions of citizen scientists would be needed for similar exhaustive, manual searches.

Second, many scientifically important tasks are not suitable for enlisting the public. Part of the success of the Milky Way Project is due to the fact that the GLIMPSE images are beautiful, contain many bubbles, and are compelling to browse through. Searches for very rare or less aesthetic objects, on the other hand, are more tedious and less likely to entice large citizen science communities. 

Finally, manual classification is not easily repeatable, and hard to calibrate statistically. For example, while the MWP catalog does include a "hit-rate" denoting the fraction of users who agree upon the existence of each bubble, there is no quantitative estimate of the reliability of each bubble. Furthermore, the current set of classifications cannot be directly applied to new data.

Automatic classifications driven by machine learning techniques nicely complement human classification. Such an approach easily scales to large data volumes, and is immune to some of the factors that affect humans, like boredom and fatigue. Furthermore, because algorithmic classifications are systematic and repeatable, they are easier to interpret and statistically characterize. 

The principal downside to automated classification for ``hard'' tasks like bubble detection is that the algorithms typically require a ``training set'' of example objects to classify. Fortunately, the Milky Way Project catalogs just such a training set. We build an automatic bubble detector from these data in the present paper. There are several reasons to do this:

\begin{enumerate}
\item We can search for bubbles not detected by MWP citizen scientists. The main reason users may have missed bubbles is because they were presented with images at a fixed contrast. Thus, faint bubbles near bright sources may have gone undetected.

\item The automatic classifier is capable of producing quantitative reliability estimates for each bubble in the MWP catalog, potentially flagging non-bubble interlopers and leading to a cleaner catalog.

\item We can treat this task as a case study for complex classification tasks in future datasets, where exhaustive manual classification will not be feasible.

\end{enumerate}


\section{Classification Method}
\label{sec:method}

Our goal is to use the set of citizen-science classifications to build an automatic detector that, when presented with a region of the sky in the \textit{Spitzer} GLIMPSE and MIPSGAL surveys, accurately determines whether or not the image contains a bubble. This is a typical case of a supervised learning problem. Before we explain our approach in detail, here is a brief overview of the task:

\begin{enumerate}
\item Build a representative training set of examples of bubble and non-bubble images. This will be drived from the MWP dataset
\item Convert each example to a numerical \textit{feature vector} that describes each object and, hopefully, distinguishes between bubbles and non-bubbles
\item Feed the training data to a learning algorithm to build a model
\item Use a subset of the examples not used during training to optimize the tunable parameters (so called \textit{hyper parameters}) of the learning algorithm
\end{enumerate}

\subsection{Training Data}
\label{sec:method_training_data}
The MWP catalog is a natural choice to build a training dataset. However, as shown in Figure \ref{fig:gallery}, the catalog as a whole contains many examples of objects that are not bubbles, and have the potential to confuse the training process. Thus, we manually curated a list of 468 objects in the catalog which were clear examples of bubbles. 

We also require an example set of non-bubbles. In principle, any random region of the sky that doesn't overlap an object in the MWP catalog is a candidate negative example -- while some of these fields actually contain bubbles not in the MWP catalog, they are rare enough that they don't strongly impact the training. However, we found that selecting negative examples at random was sub-optimal. The reason for this is that most random fields are nearly empty, and easily distinguished from fields with bubbles. Classifiers generated from such a dataset incorrectly labeled nearly any field containing extended emission as a bubble.

A better set of negative examples includes more examples of fields containing extended emission (Figure \ref{fig:bootstrap_neg}). We built such a collection in a bootstrapped fashion. Starting with random negative examples, we trained a classifier and used it to scan a large number (20,000) of negative fields. We then discarded half of the initial negative examples (those classified most confidently as not containing bubbles), and replaced them with a random sample of the mis-classified examples. We repeated this process several times, but found that one iteration was usually sufficient to build a good set of training data.

\begin{figure}
\includegraphics{bootstrap_neg}
\caption{Examples of typical randomly selected negative examples (left), and ``hard''` examples (right)}
\label{fig:bootstrap_neg}
\end{figure}

\subsection{Feature Extraction}
\label{sec:method_feature_extraction}
Supervised classification algorithms distinguish between classes of objects based on numerical \textit{feature vectors} that summarize each object. Feature vector design is often the most important stage of supervised classification and, unfortunately, also the hardest. The ideal feature vector captures the differences between each class of objects, so that examples from different classes occupy different sub-regions of feature space. 

In the case of image classification, the most obvious choice for a feature vector is simply the numerical value of the pixels themselves. This turns out to be a poor choice, because extended objects like bubbles are characterized by correlations between hundreds or thousands of pixels, and any individual pixel is virtually irrelevant to the classification task. Machine learning algorithms generally perform better when individual elements of the feature vector are more important to the classification, and less dependent on the value of other feature vector elements.

Our feature vectors have been designed based on insights in the automated face detection literature \citep{ViolaJones}.  The basic strategy is to encode a very large number ($\sim$40,000) of generic statistics about extended structures at various positions and scales. This strategy seems to be more effective then trying to tune a feature vector to the specific object being identified. While most elements in the feature vector will have no predictive power in the classification, the appropriate learning algorithm will simply ignore this information.

To construct our feature vector for a given region in the sky, we first extract a (40x40) pixel postage stamp of the field from IRAC 8 $\mu$m and MIPS 24 $\mu$m images. Following a scheme similar to \cite{mwp1}, these images are intensity clipped at the 1 and 97th percentile, normalized to maximum intensity of 1, and passed through a square root transfer function. This scaling tends to do a good job of emphasizing ISM structure, making bubbles more visible to the eye. Then, the following properties were extracted from each postage stamp, and concatenated to form the feature vector:

\begin{enumerate}
\item The wavelet coefficients from the Discrete Cosine and Daub4 wavelet transforms. These coefficients encode how much power is in various spatial frequencies.
\item The image dot product\footnote{i.e., the sum of the pixel-by-pixel product of two images} of each image with template images of rings of different size and thickness. Bubbles are morphologically similar to these templates, and presumably have larger dot products than a random intensity distribution.
\item The byte size of each compressed postage stamp image -- images substantially more or less smooth than a bubble compress to smaller and larger files, respectively.
\item The DAISY features of the image \citep{DAISY}, which characterize the local gradients in an image and have been useful in other feature recognition contexts.
\end{enumerate}

\subsection{Training}
While there are many algorithms designed to learn class labels form a training set of feature vectors, we focus here on the Random Forest algorithm \citep{random forest}. Random Forests are aggregates of a simpler learning algorithm called a decision tree. A decision tree makes classifications by passing each feature vector through the nodes of a tree diagram. Each node consists of a simple boolean condition on one element of the feature vectors (e.g., $F_3 > 5$). Examples traverse the left or right paths through the tree based on whether each test is True or False, until they reach a terminal node. The terminal node contains the final classification (in the present context, ``Bubble'' or ``Not Bubble''). Decision trees are constructed in a greedy, stage-wise fashion: each boolean test is chosen to separate the set of training data into two subclasses that are more cleanly partitioned between classes (several different metrics can be used to measure the ``purity'' of the subgroups).

On their own, decision trees are prone to over-fitting the training data by adding too many nodes. Random Forests overcome this problem by building several trees on different subsets of the data. The final classification is made by taking the majority vote of each subtree. This mitigates over-fitting in the individual trees, since over-fitting is sensitive to the specific subset of data used, and hence averages away. Random Forests have proven effective in many machine learning contexts.

In addition to their good performance and robustness against over fitting, Random Forests have several other advantageous properties: it is possible to interpret how important each element in the feature vector is to the classification, the algorithm naturally ignores irrelevant elements of the feature vector, and it is conceptually and computationally simple.

\subsection{Hyperparameter Optimization}
The Random Forest algorithm has a small number of tunable parameters. We determined good settings for these hyper-parameters using cross validation. We partitioned the training data into two sets, and used the first set to train the algorithm with a given set of hyper-parameters. The second set of cross-validation data was used to evaluate the model. Then, the hyper parameters were adjusted and the process repeated. We chose the hyper parameters that maximized the performance on the cross-validation data.

\subsection{Final Model}

Classifiers cannot be used to re-classify the data used to train them, due to the possibility of over-fitting. Thus, to obtain valid classifications for all fields, we constructed three classifiers. Each classifier is trained using data from two-thirds of the sky, and used to classify the remaining third. The sky is partitioned into thirds based on the modulus of the nearest whole longitude: for example, the first classifier is trained on examples where $0.5 < (\ell \mod{3})  < 2.5$, and used to classify fields where ($\ell \mod{3}) < 0.5 ~ || ~(\ell \mod{3}) > 2.5 $

\section{Results}
\label{sec:results}

The output of the RandomForest classifier is a score, which represents the fraction of trees in the forest which classify a feature vector as a bubble. This score provides more information than a simple binary classification, as it gives some sense of the confidence of the classification. Furthermore, one can adjust the threshold that defines when an object is determined to be a bubble. Increasing the threshold increases the reliability of the catalog, at the cost of completeness.

One way to summarize the performance of a classifier is to plot the false positive rate (fraction of negative examples incorrectly classified) versus the true positive rate (fraction of bubbles currently classified) as the threshold varies. This is (unfortunately) referred to as the Receiver Operating Characteristic, and is shown for the three classifiers in Figure \ref{fig:roc}. The false positive rate is measured by classifying $\sim 15000$ random negative fields, and the true positive rate is measured using the high-quality subset of the MWP catalog described above.


Figure \ref{fig:false_pos} shows the negative fields with the highest scores -- that is, those most confidently classified as bubbles. 


\begin{figure}
\includegraphics{roc}
\caption{ROC}
\label{fig:roc}
\end{figure}


\begin{figure}
\includegraphics{fp}
\caption{FP}
\label{fig:fp}
\end{figure}
\subsection{Expert Validation}

\subsection{Comparison to Other Catalogs}
\label{sec:comparison}
Milky Way Projection

CP06, CP07

Pallidini



\section{Conclusion}
\label{sec:conclusion}

\end{document}<commit_msg>updated manuscript<commit_after>\documentclass[preprint]{aastex}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\ifxetex
  \usepackage{fontspec,xltxtra,xunicode}
  \defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase}
\else
  \ifluatex
    \usepackage{fontspec}
    \defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase}
  \else
    \usepackage[utf8]{inputenc}
  \fi
\fi
%\usepackage{aastex}
\usepackage{graphicx}
\makeatletter
\ifxetex
  \usepackage[setpagesize=false, % page size defined by xetex
              unicode=false, % unicode breaks when used with xetex
              xetex,
              colorlinks=true,
              linkcolor=blue]{hyperref}
\else
  \usepackage[unicode=true,
              colorlinks=true,
              linkcolor=blue]{hyperref}
\fi

\newcommand{\todo}[1]{{\color[rgb]{0, .5, .5} TODO: #1}}
\newcommand{\um}[0]{$\mu$m}

\begin{document}

\title{An Algorithmically-Derived Catalog of Bubbles in the Galaxy}
\shortauthors{Beaumont et al}
\author{<NAME>$^{1,2}$, <NAME>$^2$, <NAME>$^1$}
\affil{$^1$Institute for Astronomy, University of Hawai'i, 2680 Woodlawn Drive, Honolulu HI 96822;  <EMAIL>}
\affil{$^2$Harvard-Smithsonian Center for Astrophysics, 60 Garden St., Cambridge MA 02138}

\section{Introduction}
\label{sec:intro}

\subsection{Interstellar Bubbles}
\label{sec:bubbles}

\subsection{Previous Surveys}
\label{sec:bubbles}

\begin{figure*}
\includegraphics[trim= .7in 0 0 0, clip]{gallery}
\caption{Different astrophysical objects in the MWP catalog. a) ``Canonical'' wind-blown bubbles and HII regions. b) shells
without 8 $\mu$m PAH emission (likely supernovae and planetary nebulae). c) 24 $\mu$m-dark filaments and globules. d) Other structured
ISM, that isn't clearly a stellar-cleared}
\label{fig:gallery}
\end{figure*}

\subsection{Manual and Automatic Classification in Astronomy}
\label{sec:benefits}

In terms of accuracy, humans still outperform computers in most image-based pattern recognition tasks. Because of this, morphologically complex
structures in the ISM (including supernova remnants, outflows, bubbles, HII regions, molecular and infrared dark clouds, and planetary nebulae) are still
traditionally cataloged manually. Human classification has several disadvantages, however.

First, human classification is time consuming, and people-hours are a limited resource. Even by enlisting large communities of citizen scientists, data from next generation surveys will be too large to search exhaustively. For example, the $>$ 35,000 citizen scientists of the Milky Way Project classified roughly 45 GB of image data from the GLIMPSE I and II surveys. Current and next-generation astronomical datasets are many thousands of times larger than this, suggesting tens of millions of citizen scientists would be needed for similar exhaustive, manual searches.

Second, many scientifically important tasks are not suitable for enlisting the public. Part of the success of the Milky Way Project is due to the fact that the GLIMPSE images are beautiful, contain many bubbles, and are compelling to browse through. Searches for very rare or less aesthetic objects, on the other hand, are more tedious and less likely to entice large citizen science communities. 

Finally, manual classification is not easily repeatable, and hard to calibrate statistically. For example, while the MWP catalog does include a "hit-rate" denoting the fraction of users who agree upon the existence of each bubble, there is no quantitative estimate of the reliability of each bubble. Furthermore, the current set of classifications cannot be directly applied to new data.

Automatic classifications driven by machine learning techniques nicely complement human classification. Such an approach easily scales to large data volumes, and is immune to some of the factors that affect humans, like boredom and fatigue. Furthermore, because algorithmic classifications are systematic and repeatable, they are easier to interpret and statistically characterize. 

The principal downside to automated classification for ``hard'' tasks like bubble detection is that the algorithms typically require a ``training set'' of example objects to classify. Fortunately, the Milky Way Project catalogs just such a training set. We build an automatic bubble detector from these data in the present paper. There are several reasons to do this:

\begin{enumerate}
\item We can search for bubbles not detected by MWP citizen scientists. The main reason users may have missed bubbles is because they were presented with images at a fixed contrast. Thus, faint bubbles near bright sources may have gone undetected.

\item The automatic classifier is capable of producing quantitative reliability estimates for each bubble in the MWP catalog, potentially flagging non-bubble interlopers and leading to a cleaner catalog.

\item We can treat this task as a case study for complex classification tasks in future datasets, where exhaustive manual classification will not be feasible.

\end{enumerate}


\section{Classification Method}
\label{sec:method}

Our goal is to use the set of citizen-science classifications to build an automatic detector that, when presented with a region of the sky in the \textit{Spitzer} GLIMPSE and MIPSGAL surveys, accurately determines whether or not the image contains a bubble. This is a typical case of a supervised learning problem. Before we explain our approach in detail, here is a brief overview of the task:

\begin{enumerate}
\item Build a representative training set of examples of bubble and non-bubble images. This will be drived from the MWP dataset
\item Convert each example to a numerical \textit{feature vector} that describes each object and, hopefully, distinguishes between bubbles and non-bubbles
\item Feed the training data to a learning algorithm to build a model
\item Use a subset of the examples not used during training to optimize the tunable parameters (so called \textit{hyper parameters}) of the learning algorithm
\end{enumerate}

\subsection{Training Data}
\label{sec:method_training_data}
The MWP catalog is a natural choice to build a training dataset. However, as shown in Figure \ref{fig:gallery}, the catalog as a whole contains many examples of objects that are not bubbles, and have the potential to confuse the training process. Thus, we manually curated a list of 468 objects in the catalog which were clear examples of bubbles. 

We also require an example set of non-bubbles. In principle, any random region of the sky that doesn't overlap an object in the MWP catalog is a candidate negative example -- while some of these fields actually contain bubbles not in the MWP catalog, they are rare enough that they don't strongly impact the training. However, we found that selecting negative examples at random was sub-optimal. The reason for this is that most random fields are nearly empty, and easily distinguished from fields with bubbles. Classifiers generated from such a dataset incorrectly labeled nearly any field containing extended emission as a bubble.

A better set of negative examples includes more examples of fields containing extended emission (Figure \ref{fig:bootstrap_neg}). We built such a collection in a bootstrapped fashion. Starting with random negative examples, we trained a classifier and used it to scan a large number (20,000) of negative fields. We then discarded half of the initial negative examples (those classified most confidently as not containing bubbles), and replaced them with a random sample of the mis-classified examples. We repeated this process several times, but found that one iteration was usually sufficient to build a good set of training data.

\begin{figure}
\includegraphics{bootstrap_neg}
\caption{Examples of typical randomly selected negative examples (left), and ``hard''` examples (right)}
\label{fig:bootstrap_neg}
\end{figure}

\subsection{Feature Extraction}
\label{sec:method_feature_extraction}
Supervised classification algorithms distinguish between classes of objects based on numerical \textit{feature vectors} that summarize each object. Feature vector design is often the most important stage of supervised classification and, unfortunately, also the hardest. The ideal feature vector captures the differences between each class of objects, so that examples from different classes occupy different sub-regions of feature space. 

In the case of image classification, the most obvious choice for a feature vector is simply the numerical value of the pixels themselves. This turns out to be a poor choice, because extended objects like bubbles are characterized by correlations between hundreds or thousands of pixels, and any individual pixel is virtually irrelevant to the classification task. Machine learning algorithms generally perform better when individual elements of the feature vector are more important to the classification, and less dependent on the value of other feature vector elements.

Our feature vectors have been designed based on insights in the automated face detection literature \citep{ViolaJones}.  The basic strategy is to encode a very large number ($\sim$40,000) of generic statistics about extended structures at various positions and scales. This strategy seems to be more effective then trying to tune a feature vector to the specific object being identified. While most elements in the feature vector will have no predictive power in the classification, the appropriate learning algorithm will simply ignore this information.

To construct our feature vector for a given region in the sky, we first extract a (40x40) pixel postage stamp of the field from IRAC 8 $\mu$m and MIPS 24 $\mu$m images. Following a scheme similar to \cite{mwp1}, these images are intensity clipped at the 1 and 97th percentile, normalized to maximum intensity of 1, and passed through a square root transfer function. This scaling tends to do a good job of emphasizing ISM structure, making bubbles more visible to the eye. Then, the following properties were extracted from each postage stamp, and concatenated to form the feature vector:

\begin{enumerate}
\item The wavelet coefficients from the Discrete Cosine and Daub4 wavelet transforms. These coefficients encode how much power is in various spatial frequencies.
\item The image dot product\footnote{i.e., the sum of the pixel-by-pixel product of two images} of each image with template images of rings of different size and thickness. Bubbles are morphologically similar to these templates, and presumably have larger dot products than a random intensity distribution.
\item The byte size of each compressed postage stamp image -- images substantially more or less smooth than a bubble compress to smaller and larger files, respectively.
\item The DAISY features of the image \citep{DAISY}, which characterize the local gradients in an image and have been useful in other feature recognition contexts.
\end{enumerate}

\subsection{Training}
While there are many algorithms designed to learn class labels form a training set of feature vectors, we focus here on the Random Forest algorithm \citep{random forest}. Random Forests are aggregates of a simpler learning algorithm called a decision tree. A decision tree makes classifications by passing each feature vector through the nodes of a tree diagram. Each node consists of a simple boolean condition on one element of the feature vectors (e.g., $F_3 > 5$). Examples traverse the left or right paths through the tree based on whether each test is True or False, until they reach a terminal node. The terminal node contains the final classification (in the present context, ``Bubble'' or ``Not Bubble''). Decision trees are constructed in a greedy, stage-wise fashion: each boolean test is chosen to separate the set of training data into two subclasses that are more cleanly partitioned between classes (several different metrics can be used to measure the ``purity'' of the subgroups).

On their own, decision trees are prone to over-fitting the training data by adding too many nodes. Random Forests overcome this problem by building several trees on different subsets of the data. The final classification is made by taking the majority vote of each subtree. This mitigates over-fitting in the individual trees, since over-fitting is sensitive to the specific subset of data used, and hence averages away. Random Forests have proven effective in many machine learning contexts.

In addition to their good performance and robustness against over fitting, Random Forests have several other advantageous properties: it is possible to interpret how important each element in the feature vector is to the classification, the algorithm naturally ignores irrelevant elements of the feature vector, and it is conceptually and computationally simple.

\subsection{Hyperparameter Optimization}
The Random Forest algorithm has a small number of tunable parameters. We determined good settings for these hyper-parameters using cross validation. We partitioned the training data into two sets, and used the first set to train the algorithm with a given set of hyper-parameters. The second set of cross-validation data was used to evaluate the model. Then, the hyper parameters were adjusted and the process repeated. We chose the hyper parameters that maximized the performance on the cross-validation data.

\subsection{Final Model}

Classifiers cannot be used to re-classify the data used to train them, due to the possibility of over-fitting. Thus, to obtain valid classifications for all fields, we constructed three classifiers. Each classifier is trained using data from two-thirds of the sky, and used to classify the remaining third. The sky is partitioned into thirds based on the modulus of the nearest whole longitude: for example, the first classifier is trained on examples where $0.5 < (\ell \mod{3})  < 2.5$, and used to classify fields where ($\ell \mod{3}) < 0.5 ~ || ~(\ell \mod{3}) > 2.5 $

\section{Results}
\label{sec:results}

The output of the RandomForest classifier is a score, which represents the fraction of trees in the forest which classify a feature vector as a bubble. This score provides more information than a simple binary classification, as it gives some sense of the confidence of the classification. Furthermore, one can adjust the threshold that defines when an object is determined to be a bubble. Increasing the threshold increases the reliability of the catalog, at the cost of completeness.

One way to summarize the performance of a classifier is to plot the false positive rate (fraction of negative examples incorrectly classified) versus the true positive rate (fraction of bubbles currently classified) as the threshold varies. This is (unfortunately) referred to as the Receiver Operating Characteristic, and is shown for the three classifiers in Figure \ref{fig:roc}. The false positive rate is measured by classifying $\sim 15000$ random negative fields, and the true positive rate is measured using the high-quality subset of the MWP catalog described above. Recall that the set of bubbles in this test set were selected because they are very well defined. Thus, Figure \ref{fig:roc} overestimates the true positive rate when presented with more ambiguous bubbles. We compare the performance of the classifier on more representative bubbles in Section \ref{sec:case_study} and \ref{sec:user_study}.

\begin{figure}
\includegraphics{roc}
\caption{ROC}
\label{fig:roc}
\end{figure}


\begin{figure}
\includegraphics{fp}
\caption{FP}
\label{fig:fp}
\end{figure}


\subsection{Building a Catalog}
Using our automatic bubble classifier to conduct a full search for Bubbles in \textit{Spitzer} data is straightforward. The main task involves scanning the classifier over all images, at all scales. At each of these postage stamps, the classifier yields a score, representing its confidence that the given field contains a bubble. A simple threshold on this score identifies regions likely to contain a bubble -- from the discussion of Figure \ref{fig:roc} above, adjusting this threshold sets the tradeoff between completeness and reliability. 

In practice, the classifier is insensitive to small adjustments to the size or location of the field. As a result, bubbles are usually associated with a cluster of postage stamps that rise above the detection threshold (Figure \ref{fig:cluster}a). We follow a simple, greedy procedure to merge these individual detections. Given a collection of postage stamp descriptions (location and scale), this procedure identifies the two nearest locations at similar scales, and discards the location assigned a lower score by the classifier This process repeats until no two fields are close enough to discard. Two fields are close enough to merge if they are separated by less than the radius of either field, and have sizes that differ by less than 50\%. Figure \ref{fig:cluster}b) shows the result of clustering.

This practice works well for isolated bubbles, but occasionally fails for bubble complexes with multiple, overlapping bubbles. A more sophisticated clustering analysis may well perform better in this situation.

\subsection{Classifying the l=35 field}
\label{sec:case_study}
To better understand the successes and failures of our classifier, we have analyzed the longitude range $34.5^\circ < \ell < 35.5^\circ$ in detail. In our first analysis, we scanned for all bubbles above a minimum size $r _{\rm min} = 80''$ \todo{Is this r or d?}, with scores above $s_{\rm min} = 0.55$. This search yielded 25 bubbles after merging. 

Among the 25 automatic detections, 13 are unambiguous bubbles with clear counterparts in the MWP catalog. A further four objects appear to be bubbles as judged by the authors, but are part of bubble complexes. While the complexes are identified at least once in both catalogs, each catalog decomposes the complex slightly differently. Thus, these 4 objects don't have an obvious one-to-one relationship with an object in the MWP catalog. The automated classifier discovers two new candidate bubbles with no counterpart in the MWP catalog. \todo{Lookup what the MWP images for these regions look like. Why weren't they detected? My guess is contrast}. The remaining 6 objects in the automated catalog are false positives -- three are regions containing a bright and/or extended 24\um\, point source confused for a bubble interior, and three are regions with 8 \um\, nebulosity that don't appear shell-like.

In this same region, there are 32 objects in the MWP catalog, and 11 are unassociated with the automatic detections. Seven of these are judged by the authors to be false positives -- they are mostly regions of 8 \um\, nebulosity that don't appear to be shell-like or associated with young stars.  Five appear to be true bubbles, wrongly rejected by the automated classifier. The remaining six bubbles were not considered by the classifier -- three were too close to the image edge to be analyzed, and three were smaller than the minimum search scale (we return to these objects in the next subsection). 

These numbers allow us to estimate the reliability and completeness of both catalogs. Assuming that there are no other bubbles in this region with $r > s_{\rm min}$ missed by both classifiers (but still detectable from the image data), there are roughly 24 detectable bubbles in this region -- 13 bubbles present in both catalogs, 5 objects only in the MWP catalog, 2 objects only in the automated catalog, and $\sim$ 4 bubbles in bubble complexes. The automated catalog recovers 80\% (19/24) of the bubbles, and mis-classifies 24\% (6/25) of detections. The Milky Way Project performs similarly: it recovers 92\% of bubbles (22/24), and mis-classifies 23\% of its detections (7/31).

\subsubsection{Searching for small bubbles}
In the previous analysis, three bubbles in the MWP catalog were not included in the automated catalog because they were smaller than the smallest scales scanned over.  These objects would otherwise have been included -- that is, the classifier assigns each bubble a high score. We thus repeated the above analysis, with a lower value of $s_{\rm min}$. This smaller-scale scan recovers the three small bubbles missed by the first scan. In addition, it discovers 4 additional small bubbles not in the MWP catalog. \todo{Are these in the small object catalog?}. However, false positive rate is also higher -- at very small scales, the classifier more easily confuses bright 24 \um\, sources as bubble interiors, and find's more coincidental 8\um\, nebulosity that is vaguely ring-like. Overall, the false positive rate is 42\% when including a search for bubbles, and detections not in the MWP catalog are 4 times more likely to be false positives than legitimate bubbles. From this, we conclude that our classifier is not currently suitable for searching for small bubbles.

\subsection{Expert Validation}

To some extent, identifying bubbles in \textit{Spitzer} images is a subjective task. While the objects in Figure \ref{fig:gallery}a are unambiguously associated with star formation, other sources are not. In some cases -- particularly irregular bubbles, or those with faint or no 24\um\, emission -- it is unclear whether cavities seen in 8 \um\, data are actively sculpted by young stars, or merely the underlying structure of the undisturbed ISM. 

The automatic classification is potentially well-suited to quantify this degree of ambiguity, since the classifier assigns each region a 0-1 score based on how representative it is of the training data. It's reasonable to assume that this number correlates with the intrinsic ambiguity of a particular classification, and we can test this.

The MWP catalog is not an ideal dataset for assessing intrinsic classification ambiguity, since Citizen Scientists were not trained or instructed to discriminate between different astrophysical objects. Thus, while the MWP catalog does report the amount of consensus around each object, this consensus is more a measure of how shell-like an object appears, and not the consensus to which domain experts would concur a particular objects is a canonical bubble.


\subsection{Comparison to Other Catalogs}
\label{sec:comparison}
Milky Way Projection

CP06, CP07

Pallidini



\section{Conclusion}
\label{sec:conclusion}

\end{document}
{'fraction_non_alphanumeric': 0.04756132908223762, 'fraction_numerical': 0.006666491001554636, 'mean_word_length': 4.921672647838976, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5470208', 'n_tokens_mistral': 9100, 'n_tokens_neox': 8439, 'n_words': 5360}