<commit_before>%% Submissions for peer-review must enable line-numbering
%% using the lineno option in the \documentclass command.
%%
%% Preprints and camera-ready submissions do not need
%% line numbers, and should have this option removed.
%%
%% Please note that the line numbering option requires
%% version 1.1 or newer of the wlpeerj.cls file.

\documentclass[fleqn,10pt,lineno]{wlpeerj} % for journal submissions
% \documentclass[fleqn,10pt]{wlpeerj} % for preprint submissions



\newcommand{\cheng}[1]{\textcolor{green}{\textbf{Cheng: }{\footnotesize #1}}}

\title{Combining Active Learning Suggestions}

\author[1]{<NAME>}
\author[2]{<NAME>}
\affil[1]{Data to Decisions CRC, Kent Town, SA 5067, Australia}
\affil[2]{Machine Learning Research Group, Data61, CSIRO, Australia}

\keywords{machine learning, astronomy, active learning, bandit, rank aggregation}

\begin{abstract}
Recent advances in sensors and scientific instruments have led to an increasing use of machine learning techniques for managing the data deluge. Supervised learning has become a widely used paradigm in many big data applications. However,  labeled examples are required during the training phase of supervised machine learning algorithms, and the labeling has become a significant bottleneck. This paper explores the use of machine learning algorithms for identifying informative examples for labeling, the so-called active learning setting. We empirically compare several active learning heuristics on benchmark datasets, and focus on its application to photometric classification of the Sloan Digital Sky Survey. By considering each active learning heuristic as an expert recommendation of which example to label, we propose to combine them using bandit and rank aggregation algorithms. Our results show that combining active learning suggestions improves over each individual heuristic (including passive learning), and provides a promising practical approach.
\end{abstract}

\begin{document}

\flushbottom
\maketitle
\thispagestyle{empty}

\section*{Introduction}

Previous surveys~\cite{baram04onlcal,hsu15actll}. We are more comprehensive, and consider other types of combination.

There are three ideas which are often used for eliciting human
responses using machine learning predictors. At a high level they are
similar is spirit, but they have different foundations which lead to
different formulations. The ideas are active learning, bandits and
experimental design. Related to this but with literature from a different field is social choice theory, which looks at how individual preferences are aggregated.



\section*{Binary and Multiclass classification}

\begin{itemize}
  \item binary classification
  \item multiclass classification
\end{itemize}

Base learner: Logistic regression
\begin{itemize}
  \item Warm start?
  \item polynomial kernel of degree 2
\end{itemize}

Performance:
\begin{itemize}
  \item Accuracy
  \item Posterior mean balanced accuracy
  \item F1 score
\end{itemize}

Random sampling, figure out asymptote of performance.

\section*{Active Learning, Bandits, Choice and Design of Experiments}

\subsection*{Active Learning}

Active learning considers the setting where the agent interacts with
its environment to procure a training set, rather than passively
receiving i.i.d. samples from some underlying distribution.

It is often assumed that the environment is infinite (e.g. $R^d$) and
the agent has to choose a location, $x$, to query. The oracle then returns
the label $y$. It is often assumed that there is no noise in the label,
and hence there is no benefit of querying the same point $x$ again. In
many practical applications, the environment is considered to be
finite (but large). This is called the pool-based active learning.

The active learning algorithm is often compared to the passive
learning algorithm.

Reward: improvement in performance.

\begin{itemize}
  \item uncertainty sampling
    \begin{itemize}
      \item entropy of probability
      \item margin
    \end{itemize}
  \item version space reduction
    \begin{itemize}
      \item QBB margin
      \item QBB KL
    \end{itemize}
  \item (slow) Reduction of expected variance
  \item (slow) Reduction of expected entropy
  \item (maybe) information density http://burrsettles.com/pub/settles.emnlp08.pdf
\end{itemize}

\subsection*{Bandits}

A bandit problem is a sequential allocation problem defined by a set
of actions. The agent chooses an action at each time step, and the
environment returns a reward. The aim of the agent is to maximise reward.

In basic settings, the set of actions is considered to be
finite. There are three fundamental formalisations of the bandit
problem, depending on the assumed nature of the reward process:
stochastic, adversarial and Markovian. In all three settings the
reward is uncertain, and hence the agent may have to play a particular
action repeatedly.

The agent is compared to a static agent which has played the best
action. This difference in reward is called regret.
<commit_msg>Merging table from Alasdair<commit_after>%% Submissions for peer-review must enable line-numbering
%% using the lineno option in the \documentclass command.
%%
%% Preprints and camera-ready submissions do not need
%% line numbers, and should have this option removed.
%%
%% Please note that the line numbering option requires
%% version 1.1 or newer of the wlpeerj.cls file.

\documentclass[fleqn,10pt,lineno]{wlpeerj} % for journal submissions
% \documentclass[fleqn,10pt]{wlpeerj} % for preprint submissions


\newcommand{\cheng}[1]{\textcolor{green}{\textbf{Cheng: }{\footnotesize #1}}}

\usepackage{bm} % bold math symbols

% some convenient symbols
\DeclareMathAlphabet{\mathpzc}{OT1}{pzc}{m}{it}
\DeclareMathOperator{\Beta}{Beta}
\DeclareMathOperator{\Bin}{Bin}
\DeclareMathOperator{\arsinh}{arsinh}
\DeclareMathOperator{\tr}{tr}
\newcommand{\A}{\mathpzc{A}}
\newcommand{\B}{\mathcal{B}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\Y}{\mathcal{Y}}
\newcommand{\Ecal}{\mathcal{E}}
\newcommand{\Normal}{\mathcal{N}}
\newcommand{\Unlabelled}{\mathcal{U}}
\newcommand{\Labelled}{\mathcal{L}}
\newcommand{\R}{\mathcal{R}}
\newcommand*{\argmin}{\operatornamewithlimits{argmin}\limits}
\newcommand*{\argmax}{\operatornamewithlimits{argmax}\limits}


\title{Combining Active Learning Suggestions}

\author[1]{<NAME>}
\author[2]{<NAME>}
\affil[1]{Data to Decisions CRC, Kent Town, SA 5067, Australia}
\affil[2]{Machine Learning Research Group, Data61, CSIRO, Australia}

\keywords{machine learning, astronomy, active learning, bandit, rank aggregation}

\begin{abstract}
Recent advances in sensors and scientific instruments have led to an increasing use of machine learning techniques for managing the data deluge. Supervised learning has become a widely used paradigm in many big data applications. However,  labeled examples are required during the training phase of supervised machine learning algorithms, and the labeling has become a significant bottleneck. This paper explores the use of machine learning algorithms for identifying informative examples for labeling, the so-called active learning setting. We empirically compare several active learning heuristics on benchmark datasets, and focus on its application to photometric classification of the Sloan Digital Sky Survey. By considering each active learning heuristic as an expert recommendation of which example to label, we propose to combine them using bandit and rank aggregation algorithms. Our results show that combining active learning suggestions improves over each individual heuristic (including passive learning), and provides a promising practical approach.
\end{abstract}

\begin{document}

\flushbottom
\maketitle
\thispagestyle{empty}

\section*{Introduction}

Previous surveys~\cite{baram04onlcal,hsu15actll}. We are more comprehensive, and consider other types of combination.

There are three ideas which are often used for eliciting human
responses using machine learning predictors. At a high level they are
similar is spirit, but they have different foundations which lead to
different formulations. The ideas are active learning, bandits and
experimental design. Related to this but with literature from a different field is social choice theory, which looks at how individual preferences are aggregated.



\section*{Binary and Multiclass classification}

\begin{itemize}
  \item binary classification
  \item multiclass classification
\end{itemize}

Base learner: Logistic regression
\begin{itemize}
  \item Warm start?
  \item polynomial kernel of degree 2
\end{itemize}

Performance:
\begin{itemize}
  \item Accuracy
  \item Posterior mean balanced accuracy
  \item F1 score
\end{itemize}

Random sampling, figure out asymptote of performance.

\section*{Active Learning, Bandits, Choice and Design of Experiments}

\subsection*{Active Learning}

Active learning considers the setting where the agent interacts with
its environment to procure a training set, rather than passively
receiving i.i.d. samples from some underlying distribution.

It is often assumed that the environment is infinite (e.g. $R^d$) and
the agent has to choose a location, $x$, to query. The oracle then returns
the label $y$. It is often assumed that there is no noise in the label,
and hence there is no benefit of querying the same point $x$ again. In
many practical applications, the environment is considered to be
finite (but large). This is called the pool-based active learning.

The active learning algorithm is often compared to the passive
learning algorithm.

Reward: improvement in performance.

\begin{table}[h]
	\caption {Summary of active learning heuristics used in our experiments} \label{tab:heuristics}
	\centering
	\begin{tabular}{lll}
		\toprule
		{Name}  & Notation &  Objective  \\
		\midrule
		Entropy & $r_S(\bm{x}; h)$
			& $\argmax_{x \in \Ecal} \left\{-\sum_{y \in \Y} p(y | \bm{x}; h)
            \log \big[ p(y | \bm{x}; h) \big] \right\}$
			\\[2ex]
		Margin & $r_M(\bm{x}; h)$
			& $\argmin_{x \in \Ecal} \left\{ \max_{y \in \Y} p(y | \bm{x}; h) -
            \max_{z \in \Y \setminus \{y\}} p(z | \bm{x}; h)  \right\}$
			\\[2ex]
		QBB Margin & $r_{QM}(\bm{x}; h)$
			& $\argmin_{x \in \Ecal} \left\{ \max_{y \in \Y} p(y | \bm{x}; \B) -
            \max_{z \in \Y \setminus \{y\}} p(z | \bm{x}; \B)  \right\}$
			\\[2ex]
		QBB KL & $r_{QK}(\bm{x}; h)$
			& $\argmax_{x \in \Ecal} \left\{ \dfrac{1}{B}
               \sum_{b=1}^B D_{\mathrm{KL}}(p_b\|p_\B) \right\}$
			\\
		\bottomrule
	\end{tabular}
\end{table}



\begin{itemize}
  \item (slow) Reduction of expected variance
  \item (slow) Reduction of expected entropy
  \item (maybe) information density http://burrsettles.com/pub/settles.emnlp08.pdf
\end{itemize}

\subsection*{Bandits}

A bandit problem is a sequential allocation problem defined by a set
of actions. The agent chooses an action at each time step, and the
environment returns a reward. The aim of the agent is to maximise reward.

In basic settings, the set of actions is considered to be
finite. There are three fundamental formalisations of the bandit
problem, depending on the assumed nature of the reward process:
stochastic, adversarial and Markovian. In all three settings the
reward is uncertain, and hence the agent may have to play a particular
action repeatedly.

The agent is compared to a static agent which has played the best
action. This difference in reward is called regret.

{'fraction_non_alphanumeric': 0.08370467453036261, 'fraction_numerical': 0.005067715159458279, 'mean_word_length': 4.605288932419197, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '74717', 'n_tokens_mistral': 3191, 'n_tokens_neox': 3021, 'n_words': 1406}