<commit_before>\documentclass[runningheads]{llncs}
\usepackage{graphicx}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{ruler}
\usepackage{color}

%===========================================================
% Custom commands
\newcommand{\mat}{\mathbf}

%===========================================================
\begin{document}
\pagestyle{headings}
\mainmatter{}

\def\ACCV16SubNumber{***}  % Insert your submission number here

%===========================================================
\title{Author Guidelines for ACCV Submission} % Replace with your title
\titlerunning{ACCV-16 submission ID \ACCV16SubNumber}
\authorrunning{ACCV-16 submission ID \ACCV16SubNumber}

\author{Anonymous ACCV 2016 submission}
\institute{Paper ID \ACCV16SubNumber}

\maketitle

% Misc formatting notes:
% - Use "This is not a pipe~\cite{sourceYY}." to cite papers (not fancy natbib
%   stuff)
% - They want punctuated equations and a ~ space between the end of the equation
%   and the punctuation (either comma or full stop).

\begin{abstract}
The problem of human pose estimation in still images has been well-studied in
recent years, but making effective use of the temporal information inherent in
videos is still an open problem. We propose a new model which attempts to learn
temporal relationships by using a CNN to predict poses for several frames at a
time; our model caters to the detection capabilities of CNNs by casting pose
estimation as a problem of predicting the \textit{biposelets} which poses in
adjacent frames are composed of. Predicting poses in two frames at a time
reduces pose estimation over an entire video sequence to the problem of choosing
a pose for each frame from a set of high-scoring poses, which can be achieved by
minimising a trivial pairwise cost with dynamic programming. Experiments show
that our approach performs competitively with existing approaches on established
pose estimation benchmarks.
\end{abstract}

\section{Introduction}

% Things I want to say:
% Why pose estimation in general is interesting:
%  - Challenging task in general
%  - Use it for 3D pose estimation
%  - Use it for action recognition (P-CNN paper is relevant example)
%  - Use it for just about everything else
% Why video pose estimation in particular is interesting:
%  - Few results so far
%  - Intuitively seems like temporal information should lead to big improvement
%  - Hard to actually get solid improvement in practice
% How we approach the problem; briefly summarise the biposelet detection and
% sequence stitching stages here.
% Our results

\section{Related work}

% Work I want to mention:
% Some papers on poselets
% <NAME>'s stuff (graphical models!)
% Toshev & Szegedy (CNNs)
% Modeep
% Convolutional pose machines
% Anoop's stuff
% Flowing convnets
% "Recurrent human dynamics paper" (poor results, interesting approach)

\section{Detecting pose pairs}

% What I want to say:
% - Objective of this stage of pipeline is to find a pose in each frame of each
%   neighbouring frame pair.
% - Decompose pose into several subposes.
% - Intuitively: want to predict a configuration (poselet) for each subpose.
%   Choose configurations by applying K-means to all relative positions.
% - In our method: each configuration gives a location for each joint within
%   both frames. This allows configurations to capture both location and motion.
%   Call these configurations biposelets.
% - NOW talk about cost function.
% - Talk about pairwise costs and how the means are found for the pairwise
%   costs.
% - Talk about maximising cost function.
%   - Just message passing.
% - How unaries are produced and learnt by CNN (one follows from the other):
%   - CNN architecture
%   - CNN input (two frames + flow between them)
%   - CNN output
%   - CNN objective.
%   - Fully convolutional inference
%   - Learning (just SGD) --> maybe put the training details in the experiments
%     section.
% - How SSVM is learnt. Talk about learning target, how positives/negatives are
%   used, where supervision comes in (if anywhere), how things are optimised
%   (just cite ramanan2013dual).
% - How poses are recovered from a set of subpose locations and types for each
%   subpose.

In the first stage of our pipeline, we consider each neighbouring pair of frames
in a sequence and simultaneously infer the poses present in each. %TODO

\subsection{Biposelets}

We decompose pairs of poses in neighbouring frames into a fixed set of subposes,
each of which correspond to a subset of joints in the original pose skeleton.
Rather than representing the locations of each joint in each subpose directly,
we assign a biposelet type to each subpose %TODO

For each subpose, we discretise the space of configurations for joints within
the subpose into $K$ \textit{biposelets}. Biposelets are so named in analogy to
the \textit{poselets} of Bourdev and Malik~\cite{bourdev2009poselets}. Unlike
its namesake, however, the biposelet gives a location for each joint in a
subpose across two video frames instead of a just a single image. This allows
biposelets to represent both the relative positions of joints and their movement
over time. A representative set of biposelets learnt from MPII Cooking
Activities~\cite{rohrbach2012database} is shown in Figure~\ref{fig:biposelets}.

\begin{figure*}[t]
\begin{center}
\begin{tabular}{@{}c@{}c@{}c@{}c@{}c@{}c@{}c@{}c@{}}
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-1-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-2-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-3-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-4-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-5-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-6-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-7-f0.jpg}\\
%
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-1-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-2-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-3-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-4-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-5-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-6-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-7-f0.jpg}\\
%
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-1-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-2-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-3-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-4-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-5-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-6-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-7-f0.jpg}\\
\end{tabular}
\end{center}
\caption{Biposelets learnt for left elbows in the MPII Cooking Activities
dataset. Different rows show different biposelets. For brevity,
only the first frame associated with each biposelet is shown.}
\label{fig:biposelets}
\end{figure*}

\subsection{Frame pair model}

\begin{equation}
\label{eqn:full-cost}
C(\vec l, \vec t \mid \mat I)
= w_0 + \sum_{s \in \mathcal V} \phi_s(\vec l_s, t_s \mid \mat I)
+ \sum_{(s_1, s_2) \in \mathcal E}
    \psi_{s_1 s_2}(\vec l_{s_1}, \vec l_{s_2}, t_1, t_2),
\end{equation}

\begin{equation}
\label{eqn:unary-cost}
\phi_s(\vec l_s, t_s \mid \mat I) = w_s p(s, t_s \mid \mat I(\vec l_s))
\end{equation}

\begin{equation}
\label{eqn:pair-cost}
\psi_{s_1 s_2}(\vec l_{s_1}, \vec l_{s_2}, t_1, t_2)
= \langle
    \vec w_{s_1 s_2},
    \vec \Delta(\vec l_{s_1} - \vec l_{s_2}  - \vec m_{t_1 t_2})
\rangle,
\end{equation}
where $\vec \Delta(\begin{bmatrix}\delta_x & \delta_y\end{bmatrix}) =
\begin{bmatrix}\delta_x^2 & \delta_x & \delta_y^2 & \delta_y\end{bmatrix}$ is a
deformation feature.

\subsection{Pose recovery}

\section{Sequence stitching}

% Things I want to talk about:
%
% - TODO

In the second stage of our pipeline, we attempt to produce a temporally coherent
sequence of poses across every frame of a video. %TODO

\section{Experiments}

% Things I want to include:
% - Some sort of performance evaluation. Probably just measure eval time on
%   Poses in the Wild, or perhaps time-per-frame-pair for each pair in each of
%   the datasets, then mean recombination time per frame. Over all the test
%   sequences.

\subsection{MPII Cooking}

% TODO: PCKh graphs (raw PCK given elsewhere, but I can probably ignore that),
% PCP for each limb, qualitative results

% Other papers I will try to use:
% - This paper: http://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Cheron_P-CNN_Pose-Based_CNN_ICCV_2015_paper.pdf
%   ^ Very relevant! Seems to be action recognition paper where they've also
%     tackled pose estimation. Also check out http://jhmdb.is.tue.mpg.de/ (SHIT!)
%   ^ https://github.com/gcheron/P-CNN has a link to computed joint positions
%     over all of MPII cooking activities (!!)
%   ^ Oh wait, they're using Anoop's CVPR '14 paper to compute poses :( At least
%     they show that using the GT poses (which I'll hopefully be closer to)
%     improves performance by double-digits.<commit_msg>Work on frame pair part<commit_after>\documentclass[runningheads]{llncs}
\usepackage{graphicx}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{ruler}
\usepackage{color}

%===========================================================
% Custom commands
\newcommand{\mat}{\mathbf}

%===========================================================
\begin{document}
\pagestyle{headings}
\mainmatter{}

\def\ACCV16SubNumber{***}  % Insert your submission number here

%===========================================================
\title{Exploiting Temporal Relationships for Human Pose Estimation using Biposelets}
\titlerunning{ACCV-16 submission ID \ACCV16SubNumber}
\authorrunning{ACCV-16 submission ID \ACCV16SubNumber}

\author{Anonymous ACCV 2016 submission}
\institute{Paper ID \ACCV16SubNumber}

\maketitle

% Misc formatting notes:
% - Use "This is not a pipe~\cite{sourceYY}." to cite papers (not fancy natbib
%   stuff)
% - They want punctuated equations and a ~ space between the end of the equation
%   and the punctuation (either comma or full stop).
% - Don't feel obligated to go to 14 pages. A quick skim of accepted papers from
%   ACCV 2014 shows that going above 10 pages is the norm (shortest I saw was 12
%   pages), but that fewer than 14 pages of actual content is totally
%   acceptable.
% - Structure is flexible. For instance:
%   - There's no need for a separate discussion section (apart from discussion
%     in experiments section)
%   - Some authors don't include a separate section for related work, some do.
%   - Long sections are acceptable (but poor practice, IMO)

% Tentative list of figures to produce:
% - One flow chart for the entire pose estimation process (re-use images from
%   PowerPoint, but in Inkscape or something that plays nicely with LaTeX)
% - Big panel of biposelets
% - Illustration of stitching model for single frames (much like the one in my
%   PowerPoint)
% - Panels of good at bad predictions for MPII Cooking Activities, Poses in the
%   Wild and H3.6M
% - Possibly a small summary figure at the beginning showing some representative
%   detections to lend flavour to the rest of the paper.
% - Possibly CNN architecture diagram showing how flow is merged with RGB
%   images---would prefer not to do that, since it's not a well-explored aspect
%   of my approach.
% - Plots of PCK and strict PCP:
%   - Raw PCK for Poses in the Wild
%   - PCP-strict at various thresholds for PIW (include table for PCP@0.5)
%   - PCP-strict at various thresholds for MPII Cooking Activities (also needs
%     table for PCP@0.5)
%   - PCKh for H3.6M in same format as recurrent convnet paper

\begin{abstract}
The problem of human pose estimation in still images has been well-studied in
recent years, but making effective use of the temporal information inherent in
videos is still an open problem. We propose a new model which attempts to learn
temporal relationships by using a CNN to predict poses for several frames at a
time; our model caters to the detection capabilities of CNNs by casting pose
estimation as a problem of predicting the \textit{biposelets} which poses in
adjacent frames are composed of. Predicting poses in two frames at a time
reduces pose estimation over an entire video sequence to the problem of choosing
a pose for each frame from a set of high-scoring poses, which can be achieved by
minimising a trivial pairwise cost with dynamic programming. Experiments show
that our approach performs competitively with existing approaches on established
pose estimation benchmarks.
\end{abstract}

\section{Introduction}

% Things I want to say:
% Why pose estimation in general is interesting:
%  - Challenging task in general
%  - Use it for 3D pose estimation
%  - Use it for action recognition (P-CNN paper is relevant example)
%  - Use it for just about everything else
% Why video pose estimation in particular is interesting:
%  - Few results so far
%  - Intuitively seems like temporal information should lead to big improvement
%  - Hard to actually get solid improvement in practice
% How we approach the problem; briefly summarise the biposelet detection and
% sequence stitching stages here.
% Our results

\section{Related work}

% Work I want to mention:
% Some papers on poselets
% Xianjie Chen's stuff (graphical models!)
% Toshev & Szegedy (CNNs)
% Modeep
% Convolutional pose machines
% Anoop's stuff
% Flowing convnets
% "Recurrent human dynamics paper" (poor results, interesting approach)

\section{Detecting pose pairs}

% What I want to say:
% - Objective of this stage of pipeline is to find a pose in each frame of each
%   neighbouring frame pair.
% - Decompose pose into several subposes.
% - Intuitively: want to predict a configuration (poselet) for each subpose.
%   Choose configurations by applying K-means to all relative positions. This is
%   tied to the CNN training step, so I will have to explain it there.
% - In our method: each configuration gives a location for each joint within
%   both frames. This allows configurations to capture both location and motion.
%   Call these configurations biposelets.
% - NOW talk about cost function.
% - Talk about pairwise costs and how the means are found for the pairwise
%   costs.
% - Need to mention optical flow implementation in use
% - Talk about maximising cost function.
%   - Just message passing with distance transforms.
%   - Need complexity mentioned in there somewhere.
% - How unaries are produced and learnt by CNN (one follows from the other):
%   - CNN architecture
%   - CNN input (two frames + flow between them)
%   - CNN output
%   - CNN objective.
%   - Fully convolutional inference
%   - Learning (just SGD) --> maybe put the training details in the experiments
%     section.
% - How SSVM is learnt. Talk about learning target, how positives/negatives are
%   used, where supervision comes in (if anywhere), how things are optimised
%   (just cite ramanan2013dual). <-- Maybe put in learning section
% - How poses are recovered from a set of subpose locations and types for each
%   subpose.

\subsection{Subposes and biposelets}

In the first stage of our pipeline, we consider each neighbouring pair of frames
in a sequence and simultaneously infer the poses present in each. For the
purposes of this stage, we decompose a pair of poses in neighbouring frames into
a fixed set of subposes, each of which correspond to a subset of joints in the
original pose. For instance, a pose containing shoulder, elbow and wrist joints
could be decomposed into one subpose containing joints associated with the left
forearm, one subpose containing joints associated with the left upper arm, one
subpose containing both the left and right shoulders, etc.

Rather than representing the locations of each joint in each subpose directly,
we discretise the space of configurations for joints within each subpose into
$K$ \textit{biposelets}. Biposelets are so named in analogy to the
\textit{poselets} of Bourdev and Malik~\cite{bourdev2009poselets}. Unlike their
namesake, however, biposelets give a position for each joint in a subpose across
two video frames instead of a single one. This allows biposelets to represent
both the relative positions of joints and their movement over time. A
representative set of biposelets learnt for the MPII Cooking Activities pose
estimation dataset~\cite{rohrbach2012database} is shown in
Figure~\ref{fig:biposelets}.

\begin{figure*}[t]
\begin{center}
\begin{tabular}{@{}c@{}c@{}c@{}c@{}c@{}c@{}c@{}c@{}}
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-1-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-2-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-3-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-4-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-5-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-6-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-230/sample-7-f0.jpg}\\
%
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-1-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-2-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-3-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-4-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-5-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-6-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-289/sample-7-f0.jpg}\\
%
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-1-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-2-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-3-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-4-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-5-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-6-f0.jpg}\,&
\includegraphics[height=0.1375\linewidth]{figures/biposelets/poselet-297/sample-7-f0.jpg}\\
\end{tabular}
\end{center}
\caption{Biposelets learnt for left elbows in the MPII Cooking Activities
dataset. Different rows show different biposelets. For brevity,
only the first frame associated with each biposelet is shown.}
\label{fig:biposelets}
\end{figure*}

% TODO: Give motivation for decomposing into subposes and biposelets instead of:
% 1) Considering a single joint at a time
% 2) Considering the entire pose

\subsection{Frame pair model}

Formally, in the first stage of our pipeline, we are given a pair of frames
$\mat I_1$ and $\mat I_2$---which will be abbreviated $\mat D_{1,2}$ herein

\begin{equation}
\label{eqn:unary-cost}
\phi_s(\vec l_s, t_s \mid \mat I) = w_s p(s, t_s \mid \mat I(\vec l_s))
\end{equation}

\begin{equation}
\label{eqn:pair-cost}
\psi_{s_1 s_2}(\vec l_{s_1}, \vec l_{s_2}, t_1, t_2)
= \langle
    \vec w_{s_1 s_2},
    \vec \Delta(\vec l_{s_1} - \vec l_{s_2}  - \vec m_{t_1 t_2})
\rangle,
\end{equation}
where $\vec \Delta(\begin{bmatrix}\delta_x & \delta_y\end{bmatrix}) =
\begin{bmatrix}\delta_x^2 & \delta_x & \delta_y^2 & \delta_y\end{bmatrix}$ is a
deformation feature.

% Representing subpose locations as a vector is not strictly accurate
Given a pair of frames $\mat I_1$ and $\mat I_2$, the dense optical flow $\mat
F_{12}$ between those frames, a subpose location $\vec l_s$ for each subpose
$s$, and a biposelet $t_s$ for each subpose $s$, the full cost is
%
\begin{equation}
\label{eqn:full-cost}
C(\vec l, \vec t \mid \mat D_{12})
= w_0 + \sum_{s \in \mathcal V} \phi_s(\vec l_s, t_s \mid \mat D_{12})
+ \sum_{(s_1, s_2) \in \mathcal E}
    \psi_{s_1 s_2}(\vec l_{s_1}, \vec l_{s_2}, t_1, t_2),
\end{equation}
%
where $w_0$ is a bias and $\mat D_{12}$ is the concatenation of $\mat I_1$,
$\mat I_2$ and $\mat F_{12}$.

\subsection{Inference on frame pair model}

\section{Sequence stitching}

% Things I want to talk about:
%
% - How sequence stitching actually works
% - Why this is an improvement over other methods (fewer parameters, less
%   complicated implementation)
% - How we find our parameters (randomized search on validation set showed that
%   stitching is pretty much invariant to order-of-magnitude changes in relative
%   values of variables, so in practice you can probably pick coefficients by
%   hand)

In the second stage of our pipeline, we attempt to produce a temporally coherent
sequence of poses across every frame of a video.%TODO

After determining a location and biposelet type for each subpose, we can recover
a location for each joint in the original pose model%TODO

\section{Learning}

% Things I need to explain:
% - How CNN is learnt (what is the general learning setup? What augmentations do
%   we apply? What about SGD parameters?)
% - How biposelets are clustered (apply K-means to joints in CNN training
%   patches)
% - Structural SVM things

\subsection{Learning unaries}

\subsection{Clustering biposelets}

\subsection{Single-pair detector}

\subsection{Stitching weights}

\section{Experiments}

% Things I want to include:
% - Some sort of performance evaluation. Probably just measure eval time on
%   Poses in the Wild, or perhaps time-per-frame-pair for each pair in each of
%   the datasets, then mean recombination time per frame. Over all the test
%   sequences.

\subsection{MPII Cooking}

% TODO: PCKh graphs (raw PCK given elsewhere, but I can probably ignore that),
% PCP for each limb, qualitative results

% Other papers I will try to use:
% - This paper: http://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Cheron_P-CNN_Pose-Based_CNN_ICCV_2015_paper.pdf
%   ^ Very relevant! Seems to be action recognition paper where they've also
%     tackled pose estimation. Also check out http://jhmdb.is.tue.mpg.de/ (SHIT!)
%   ^ https://github.com/gcheron/P-CNN has a link to computed joint positions
%     over all of MPII cooking activities (!!)
%   ^ Oh wait, they're using Anoop's CVPR '14 paper to compute poses :( At least
%     they show that using the GT poses (which I'll hopefully be closer to)
%     improves performance by double-digits.
{'fraction_non_alphanumeric': 0.11166402127750848, 'fraction_numerical': 0.025095448500707822, 'mean_word_length': 4.547834364588291, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6168784', 'n_tokens_mistral': 7498, 'n_tokens_neox': 7106, 'n_words': 2592}