Towards Automated Performance Bug Identification in Python

Abstract: Context: Software performance is a critical non-functional requirement, appearing in many fields such as mission critical applications, financial, and real time systems. In this work we focused on early detection of performance bugs; our software under study was a real time system used in the advertisement/marketing domain.Goal: Find a simple and easy to implement solution, predicting performance bugs.Method: We built several models using four machine learning methods, commonly used for defect prediction: C4.5 Decision Trees, Na\"ıve Bayes, Bayesian Networks, and Logistic Regression.Results: Our empirical results show that a C4.5 model, using lines of code changed, file's age and size as explanatory variables, can be used to predict performance bugs (recall=0.73, accuracy=0.85, and precision=0.96). We show that reducing the number of changes delivered on a commit, can decrease the chance of performance bug injection.Conclusions: We believe that our approach can help practitioners to eliminate performance bugs early in the development cycle. Our results are also of interest to theoreticians, establishing a link between functional bugs and (non-functional) performance bugs, and explicitly showing that attributes used for prediction of functional bugs can be used for prediction of performance bugs.

\section{Introduction} \label{intro}

There exists large number of fields where performance of software is critical. For example, mission critical applications, financial and real time systems \cite{han2012performance, jin2012understanding, killian2010finding}.

Based on statistics from the International Advertising Bureau, mobile ad expenses for 2014 reached \$31.9 billion US dollars \cite{iab}, followed by a 20\% growth in 2015. This revenue is entirely managed by real time buying systems.

Real time systems (RTS) are software that subject to time constraints; they have to process information and provide an output within a pre-specified time threshold. A more specific sub-category of RTS is the application under study, which is a real time buying (or bidding) system (RTB), used in the advertisement/marketing domain. The RTB system, also known as Demand Side Platform (DSP), receives an \textit{http} request from the Sell Side Platform (SSP). This request, which is basically an auction being held by the SSP, contain information that is processed by the RTB. The RTB responds back with an advertisement, if some (or all) of the criteria are met. If the RTB system does not respond within a 100 milliseconds time constraint, the opportunity to bid is lost.

Code inefficiencies for this type of software can easily cause reduced throughput, increased latency, and wasted resources. All the above further translate to unreliable software that cannot respond within the desired time threshold (the aforementioned 100ms), leading to financial losses, due to unsuccessful bidding. Also, since performance bugs are not reported as often as functional bugs \cite{jin2012understanding, zaman2011security}, it is easier for them to accumulate and affect the system for a longer time. In this study (in line with \cite{jin2012understanding, killian2010finding, zaman2012qualitative}) we will define \textit{performance bug} as a software  defect,  where  after  relatively  simple  changes  are  applied,  the  performance of  the software  is  significantly  increased  without  affecting functionality.

\textbf{Problem:} Continuous integration practices are gaining momentum and are being used widely, especially by startups \cite{fowler2006continuous}. However, lack of time for sufficient performance testing and immaturity of the development team can cause performance bugs to be injected at all times. The problem in such cases, in contrast with regular bugs that affect the system’s outcome, is that such types of bugs are not easily diagnosed. Even if they are, developers can easily omit fixing/improving them in favor of delivering new features or other more significant tasks, such as fixing security bugs \cite{zaman2011security}. We need a way (preferably a simple and tractable one) to detect performance bugs using information that is either readily available to a developer or can be easily obtained. Static analysis and use of attributes extracted from source code repository are promising candidates.

Moreover, we want to understand which factors (i.e., attributes associated with committed code) affect injection of performance bugs, so that the management can take corrective actions.
Therefore, our \textbf{research question} is: \\

\emph{How can we detect performance bugs using static code analysis and attributes extracted from source code repository?} \\

\textbf{Approach:}  We tackle the research question by performing a case study; our software under study is a RTB system developed over a period of three years. Our goal was to determine if the commit metrics that we gathered, are associated with the probability of performance bugs to be injected. Approximately 2800 file commits (delivered over this three year period) were processed in order to extract a variety of attributes and metrics (analyzed in Section \ref{res_dataset}) for every file that was changed during development.

These attributes and metrics were categorized into three groups: a) Project attributes, b) Activity attributes, and c) Experience attributes. Then the data have been preprocessed (see Section \ref{data_prep} for details). We tried several approaches both in the manner of attributes selected as well as the classification algorithms that were used, in order to achieve the best outcome. We also quantified the contribution of each attribute on the efficiency of each model.

\textbf{Contributions:} Our main contributions from this work are: a) explicitly establishing that static code analysis and metrics can be used as predictors of performance bugs, similar to functional bugs, b) understanding the differences between classification algorithms that are commonly used to predict bugs, c) understanding the impact of each attribute on the models' efficiency.

Performance bugs are often caught while timing reference workloads. However this is typically done late in the development cycle. Our approach is complementary; it enables early detection (and thus early elimination) of the performance bugs, since prediction models operate on the source code and do not require code execution.

\textbf{Impact:} What we present in this paper is a starting point for practitioners seeking performance optimization and an efficient way of predicting performance bugs. A successful detection will have a significant impact, since in general, performance bugs need much more time to get fixed than non-performance (functional) bugs \cite{zaman2011security, zaman2012qualitative}.

\textbf{Structure of this work:} We provide a review of related work that has been conducted in this area, and identify the knowledge gap in Section \ref{rel_work}. Next, in Section \ref{methodology}, we present the methodology that we followed and the design of our study. Results of the study are given in Section \ref{results}. Threats to validity are identified in Section \ref{threats}. Finally, Section \ref{conclusion} summarizes our thoughts and outcomes, and specifies future research potentials.

\section{Related Work} \label{rel_work}
There are many studies in the area of defect prediction (see Hall et al. \cite{hall2012systematic} for review of 208 defect prediction models)  and quality assurance of software, since this is one of the most significant endeavors during a product’s life cycle. Reduced number of bugs assures better quality, thereby improving the product being delivered and allowing for better resource allocation \cite{fenton1999critique, hall2012systematic}.

There are several categories of tools and methods used to detect and predict defects, with the aim of delivering better software \cite{orso2014software}. However, static bug finders and statistical models \cite{rahman2014comparing} have become the two most prominent categories of defect prediction. The defects can be further categorized to create prediction models in order to distribute the resolution resources more efficiently \cite{caglayan2015predicting, caglayan2010usage, kan_metrics_2002}.
At the same time, efficiency, density, and principally the technical debt caused by defects, play critical roles as well \cite{vetro2012using}.

Failures after the release of a product have also momentous effects, especially for large-scale commercial distributions \cite{jin2012understanding}. Even though developers dedicate large amounts of effort and time to testing, they can never be sure that the system is absolutely reliable.

\emph{Performance}

Even though there are many in-depth studies trying to determine the best way of predicting functional bugs, there is not much work done in prediction of bugs causing performance degradation. Below we give an overview of research papers studying performance bugs.

Performance bugs are harder to expose during testing phase, because they do not cause fatal symptoms and do not affect the overall outcome \cite{molyneaux2014art}.  Furthermore, it is difficult to find the root cause of performance bugs (in comparison with other types of bugs); they also need more time to get resolved \cite{zaman2011security}.

The work that is closest to ours is by Jin et al. \cite{jin2012understanding}, which studied a set of 109 real-world performance bugs. They studied the bugs’ lifetime from inception to fix, their root causes and introduction mechanisms, in order to create rule-based detectors. We used a comparable approach to create detectors (referred to as "patterns" in our study) with the aim to identify similar performance bugs. They studied software written in Java, C, C++, and JavaScript; ours was written in Python\footnote{Programming language does affect performance of the models, as shown by \cite{weyuker_2010}.}. Moreover, we focused on finding an efficient method to automatically detect performance bugs based on data extracted through the use of patterns (197 real-world performance bugs in our case). Summarizing, Jin et al. \cite{jin2012understanding} focused on analyzing characteristics of performance bugs, while in this study, our focus is on understanding the contribution of each source code attribute, as extracted from the source code repository, to the predicting power of the several machine learning algorithms that we used. Lastly, our general approach and the methodology for creating the prediction model were meant to be easily reproducible in the future. Therefore, our work is complementary.

Automated detection of performance bugs have been implemented in the past \cite{foo2015, killian2010finding}. However, the authors used dynamic analysis tools, namely execution traces and historical performance data, to detect slowly executing code. In addition, dynamic analysis tools often require dedicated testing environment to get accurate performance readings. We, on the other hand, are leveraging code attributes extracted from source code repository and automatically detecting performance bugs before the code executes (and reaches production environment). Thus, our work is complementary.

Static analysis tools can also eliminate performance bugs~\cite{nistor2015}. However, this approach requires knowledge of pattern template, while ours does not (as we omit pattern type info in our models), hence the complementarity of our work.

There exist formal and thorough coding standards for RTS. However, the standards are, typically, language specific (e.g., C~\cite{misra-c} and C++~\cite{misra-cpp}). To the best of our knowledge, no thorough and formal RTS coding standard exists for Python.

Finally, since in our study we relate performance with the real-time nature of the system, one could argue that Python programming language might not be appropriate for such a case. Although this is a problem that falls out of the scope of this study, we understand its importance and validity. Python programming language (with a proper runtime \cite{paypal}) is used in well-known large-scale RTS, such as eBay, PayPal, and YouTube \cite{paypal, python_quotes}. The benefits of the language are best summarized by Cuong Do, Software Architect of YouTube: ``Python is fast enough for our site and allows us to produce maintainable features in record times, with a minimum of developers'' \cite{ python_quotes }.

\section{Methodology} \label{methodology}
In this section we discuss methodology of our case study. The section is structured as follows. We describe our software under study in Section \ref{e_study}. The dataset (extracted from the software under study) and its preprocessing are depicted in Sections \ref{res_dataset} and \ref{data_prep}, accordingly. Finally, we discuss building of prediction models in Section \ref{model_building}.

\subsection{Software Under Study} \label{e_study}
The software that we worked on is a RTB application, written in Python behind an asynchronous web framework. The system has been developed from scratch during the last three years. Although the history of the code might sound short, due to the continuous integration practices, the commit count is 2773 and that is only for the production (master) branch. Therefore, we had sufficient amount of information to work with.

The RTB application receives as input \textit{http} requests and responds back within a time constraint (typically, 100 milliseconds), that is set by the SSPs, which the RTB is connected with. The load of the system can grow tremendously, based on various factors, e.g., incoming request rate and current advertisement inventory.

It is the nature of the system that makes it vulnerable to small fluctuations of the response time, and that is where performance becomes critical. Although in common systems availability is measured by subtracting downtime, in our case, we have to append the total of timed-out sessions in order to calculate downtime. During a timeout, the system’s functionality is not affected. However, the strict time threshold is not allowing the system to complete a computational cycle successfully.

\subsubsection{Infrastructure}
Since performance is usually related to workload, we provide some information regarding the infrastructure associated with the software under study.
The cloud cluster undertaking the load consists of 22 virtual machines, each containing 32~vCPUs\footnote{Intel Xeon E5-2680 v2 (Ivy Bridge) Processors.} and 60~GB of RAM, each one of them capable of processing more than 3,000 requests per second. The cluster processes, on average, 10 billion transactions per day (going up to 30 billion transactions on a busy day). However, it is worth mentioning that the amount of machines deployed is dynamic and depends on many factors, such as high peak time periods of the year or the number of countries that the current ads are targeting (the larger the number of countries -- the higher the load). In such cases more computational power is added in order to accommodate the load.

\subsubsection{Spot the Bottlenecks}
As a first step of our methodology, we had to identify the bottlenecks of the system. This was achieved with the help of dynamic program analysis tools while reproducing conditions that happen in production environment and, in some cases, executing the analysis on the production environment itself. Although dynamic analysis tools -- the well-known \textit{profilers}, have been improved extensively and can provide accurate results, they cannot identify the performance bugs themselves \cite{jovic2011catch}. However, they can help in preliminary analysis and assist in easy verification of the experimental findings.

By identifying the bottlenecks, we were able to find alternative ways that optimize the performance of the system. Most of the times the changes that had to be applied were a few lines of code, which resulted in significant performance improvements. These findings were categorized and classified as \textit{patterns}.

\subsubsection{Patterns} \label{patterns}
We define patterns as coding practices/styles followed (coincidentally or deliberately) by individual developers.  These patterns introduced a logic that affected the performance, but not the overall correctness of the algorithm/system. We identified eleven patterns, described in Appendix~\ref{app:patterns}, causing performance degradation (PD). Identification was performed by detecting bottlenecks (i.e. code blocks causing PD) using profilers, followed by manual code inspection. All patterns always lead to PD, which we validated on four platforms and two Python versions. We fixed 197 instances/defects, based on these patterns; fixing patterns leads to 20-99\% improvement for a given code block; 99\% improvement was achieved by fixing pattern described in Appendix~\ref{p:2}.

These eleven unique patterns were generalized and used in scripts that we built, so that we could detect multiple areas of application. This type of static bug finders were used in the past for similar purposes \cite{jin2012understanding, livshits2005dynamine}; however, in this study they were mainly used to get an enriched dataset.

The practice of pattern identification was used before by Jin et al. \cite{jin2012understanding} (rule-based detectors) in order to achieve similar results, i.e., detect multiple areas that were ``affected'' by the same generalized model. However, their patterns were mainly connected to outdated or error prone API's and, in most of the cases, the bugs were product-specific. Contrariwise, the patterns that we identified are more general. Essentially, they can be re-used on any software written in the same programming language (Python) and are more related to the development techniques utilized by the software developers.\\

\emph{Benchmarking Patterns}\\

In terms of benchmarking and calculating the performance improvement of each pattern, we thoroughly tested the impact of each one of them. The performance improvements per pattern ranged between 20\% and 99\%.

Additionally, and in order to validate that a pattern is not endemic to a particular environment (e.g., Operating System, Hardware Platform, or Programming Language version) we tested them in multiple environments and observed that they lead to similar performance degradation in all environments.

\subsection{Resulting dataset} \label{res_dataset}
Having the patterns as a stepping-stone (and the static bug finder scripts using them), we could move forward and extract a dataset with the help of the  version control system. It is worth mentioning that the dataset that we extracted and utilized, contains information on the performance bug injection points. Such information is difficult to extract from a bug tracking system \cite{serrano2005bugzilla}, even when dealing with more general types of bugs.

Therefore, our dataset included performance bug injection commits and was further split by file. As a result we had a thorough dataset including the following attributes\footnote{This is not an exhaustive list of attributes that can be used to predict injection of a defect~\cite{ccauglayan2014factors, li2009analysis, caglayan2012factors}. However, we focused on the attributes that can be easily extracted by practitioners to simplify institutionalization of the prediction models. }: \\

\emph{Project attributes}
\renewcommand{\labelenumi}{\roman{enumi}}
\begin{enumerate}

\item{\textit{\textbf{Type:}} Indicates either if the entry injected\footnote{We manually traced injection point for each performance bug using history information obtained from the source code management system.} a performance bug into the code or not (not a defect at all).}
\item{\textit{\textbf{Commit ID:}} Unique identifier of the commit.

\textit{Total number of commits is 2773.}}
\item{\textit{\textbf{Owner:}} The user that submitted the commit.

\textit{13 unique owners under study.}}
\item{\textit{\textbf{File name:}} The file that was modified. Since each commit can contain several files, we split the entries to unique 2-tuples of commit id and file name (henceforth called tuple). In essence, the tuple serves as a unique identifier for a record in our dataset.

\textit{338 unique files, 4623 unique tuples (commit id and file name).}}
\end{enumerate}

\emph{Activity attributes}

\begin{enumerate}
\setcounter{enumi}{4}

\item{\textit{\textbf{Lines added:}} The number of lines that were added (or modified) for the corresponding file. Note that one modified line will be counted as one line removed and one line added (a-la diff). }
\item{\textit{\textbf{Lines removed:}} The number of lines that were removed (or modified) for the corresponding file.}
\item{\textit{\textbf{File age:}} Age of the file, computed as `modification date -- file creation date'.}
\item{\textit{\textbf{File size:}} Although two separate values were extracted to represent file size, we treat them as one attribute:}
\begin{itemize}
\item{\textit{Source lines of code.}
Lines of code without comments or blank lines.}
\item{\textit{Comment lines.}
Lines of code representing the developer's comments.}
\end{itemize}
\end{enumerate}

\emph{Experience attributes}

\begin{enumerate}
\setcounter{enumi}{8}

\item{\textit{\textbf{Maturity/Expertise of the owner:}} The amount of time an owner spent working on the product, computed as `modification date -- first commit date of the owner'.}
\item{\textit{\textbf{Time since last commit:}} How much time elapsed since the last commit (in any of the files).}
\end{enumerate}

Descriptive statistics for all the numerical attributes is shown in Table \ref{tab:attrs}. Correlation between the numerical attributes is given in Figure \ref{fig:cor}. As we can see, none of the variables, except\footnote{Which is typical for a software product.} `source lines of code' (SLOC) and `comment lines', are strongly correlated.

\begin{table*}[ht]
\centering
\caption{Descriptive statistics of numeric attributes.}

\begin{tabular}{|l|c|c|c|c|}
\hline
\textbf{Attribute} & \textbf{Range} & \textbf{Mean} & \textbf{Median} \\
\hline \hline
\rule{0pt}{2ex}
Lines added & 0 - 642 & 19.74 & 3 \\
\hline
\rule{0pt}{2ex}
Lines removed & 0 - 995 & 13.21 & 2  \\
\hline
\rule{0pt}{2ex}
File age (days) & 0 - 803 & 139.2 & 59.97  \\
\hline
\rule{0pt}{2ex}
SLOC & 0 - 1109 & 254.2 & 150  \\
\hline
\rule{0pt}{2ex}
Comment lines & 0 - 419 & 71.07 & 30  \\
\hline
\rule{0pt}{2ex}
Maturity/Expertise (days) & 0 - 997 & 416.9 & 435  \\
\hline
\rule{0pt}{2ex}
Time since last commit (hours) & 0 - 266.7 & 10.71 & 1.26 \\
& ($\approx 11$ days) & & \\
\hline
\end{tabular}

\label{tab:attrs}
\end{table*}

\begin{figure*}[!ht]
\centering
\includegraphics[width=\linewidth]{./fig/cor.pdf}
\caption{Correlogram visualizing correlation matrix \cite{corrgram} for numeric attributes. Diagonal shows attribute name. Lower triangle region shows confidence ellipse and smoothed line. Upper triangle region shows Pearson’s correlation coefficient $\rho$ and, in brackets, confidence interval for $\rho$. }
\label{fig:cor}
\end{figure*}

\subsection{Dataset Preprocessing} \label{data_prep}

\subsubsection{Cleansing}
We eliminated all files that did not go into production code base, namely: readme files, testing scripts, and help files.

Additionally, we removed a minor 0.2\% (9 out of 4623 unique tuples) of ``commit id -- file name'' records related to source code files. These records were outliers, extreme cases. For example, we excluded source files that were moved or removed. To be more specific, the  version control system by default identifies directory changes/refactorings as complete removals of the files themselves. Therefore, whenever a file is moved one or more levels up or down in the directory structure, we noticed abnormal numbers of lines added and/or removed. In some of theses cases (and especially in directories including large files) more than 10,000 lines were added or removed on a single commit. The cleansing described above, resulted in a more precise model creation, mainly because of the removal of entries that will not be seen on future commits \cite{gray2011misuse}.

\subsubsection{Re-balancing}\label{rebalancing}
After the initial cleansing we had our dataset in its final form. However, and as a result of the large number of commits that we have under study, it was expected that the non-defective entries would prevail over the defect set. In fact, we had 95.5\% (4426) of non-defective entries and 4.5\% (197) of defective ones. This type of imbalance is common, especially when dealing with real world problems \cite{he2009learning}. Therefore, in order to avoid biased results towards the majority class, we had to choose a re-balancing technique.

There exists several re-balancing methods, since this ubiquitous problem of imbalanced data is faced quite often. To overcome this obstacle and achieve valid results, researchers use over and under-sampling techniques, either in the minority or majority classes. While reviewing these methods and techniques that have been used towards this goal, we decided to use random over-sampling for our training dataset. Against the numerous (and in some cases combined and complex) methodologies, as studied by Batista et al. \cite{batista2004study}, random over-sampling provides competitive results.

\subsubsection{Overcoming Premature Over-fitting} \label{overcome_pof}

Given high imbalance of our data, we expected that models trained on rebalanced datasets would be prone to over-fitting (as seen by \cite{mease_2007, he2009learning}). This was confirmed during preliminary model building (discussed in Section~\ref{model_building}): we observed over-fitting of the models, no matter the classification method.

As described by Tan et al. \cite{tanonline}, over-fitting may occur even when one utilizes cross-validation, especially when dealing with extremely imbalanced datasets. Even after applying the re-balancing methods (described in Section~\ref{rebalancing}), the repetition of the entries might still result in false high precision \cite{tanonline}.

To overcome this issue, we used a combination of techniques, similar to the one that was introduced by Mease et al. \cite{mease_2007} and called ``JOUS-Boost''. They added independently and identically distributed noise to the minority class for which random oversampling created replicates. This approach was shown to be superior in dealing with the oversampling problem \cite{he2009learning}.

In our case, we decided to generate a ``jittered'' train set, after we split our data to 70\% train - 30\% validation. We added noise to the minority entries of the 70\% part, by randomly increasing or decreasing the values by 0 to 2\%. Formally, a variable $x$ becomes $x \times U(0.98, 1.02)$, where $U(0.98, 1.02)$ is a random value drawn from the uniform distribution with minimum value of $0.98$ and maximum value of $1.02$. This transformation could only apply to numerical attributes (listed in Table \ref{tab:attrs}). We do not add noise to the validation set, to mimic the actual defect prediction process; when a developer passes information about a new commit to the prediction `oracle' to predict if this commit contains a performance bug or not. Results of this experiment are discussed in the next section.

\subsection{Building the Models}\label{model_building}

\subsubsection{Classification Algorithms}
After re-balancing the dataset, it is time to move forward and start building the desired models. The first question that arises is: which classification algorithms should be used for building the models? Based on the literature and existing empirical work \cite{moser2008comparative} we decided to experiment with four popular machine learning methods (using Weka \cite{weka} and R \cite{R}). The algorithms chosen are: a) C4.5 decision tree (Weka's J48 class \cite{weka}), b) Na\"{\i}ve Bayes, c) Bayesian networks, and d) Logistic regression.

\subsubsection{Training and Testing the Models} \label{training_models}
We trained and tested the model on the 70\% of the data using 10-fold cross validation; we then validated the model on the remaining 30\% of the data (see Section \ref{overcome_pof} for description of data subsets). We used this approach to generate more precise efficiency metrics for the models. The details of the efficiency metrics --- namely, accuracy, precision, and recall --- are given below.

 \textbf{Accuracy:} accuracy  reflects the percentage of correctly classified performance bugs to the total number of performance  bugs. It is computed as follows \cite{witten2005data}:

\begin{equation}
    ACC  = \frac{(TP+TN)}{(TP+TN+FN+FP)},
    \label{eq:1}
\end{equation}
where $TP$ is the number of true positive results, $FP$ is the number of false positive results, $TN$ is is the number of true negative results, and $FN$ is the number of false negative results.

\textbf{Precision:} precision, or Positive Predictive Value is the fraction of retrieved instances that are relevant, and it is calculated as \cite{witten2005data}:

\begin{equation}
    PPV  = \frac{TP}{(TP+FP)}.
    \label{eq:2}
\end{equation}

\textbf{Recall :}  recall, also known as sensitivity or True Positive Rate, is the fraction of relevant instances that are retrieved, and it is calculated as \cite{witten2005data}:

\begin{equation}
    TPR  = \frac{TP}{(TP+FN)}.
    \label{eq:3}
\end{equation}

All three metrics range between 0 and 1; the closer the value to 1 -- the better.

\begin{table*}[ht]
\centering
\caption{Experiment metrics: JOUS-Boost. The models are cumulative: the first model describes the model using one explanatory variable: lines added; the second model – two variables: lines added and lines removed;  the third one – three variables: lines added, lines removed, and age; and so on. Size consists of two separate attributes, as described in Section \ref{res_dataset}.}
\label{table:jit}
\begin{tabular}{|c|l|c||C{2cm}|c|c|C{2cm}|}
  \hline

   & & & & & & \textbf{Logistic} \\

  \textbf{Row \#} & \multicolumn{1}{c|}{\textbf{Variable}} & \textbf{Metric} & \textbf{C4.5} & \textbf{Na\"{\i}ve Bayes} & \textbf{Bayesian Nets} &     \textbf{Regression} \\
  \hline
  \hline
    \rule{0pt}{2ex}
     & +Lines added & TPR: & 0.65 & 0.52 & 0.68 & 0.67 \\
    1 & \textit{jittered} & ACC: & 0.81 & 0.74 & 0.82 & 0.79 \\
     & & PPV: & 0.95 & 0.93 & 0.95 & 0.89 \\
  \hline
    \rule{0pt}{2ex}
     & +Lines removed & TPR: & 0.67 & 0.52 & 0.67 & 0.67 \\
    2 & \textit{jittered} & ACC: & 0.82 & 0.73 & 0.82 & 0.79 \\
     & & PPV: & 0.95 & 0.91 & 0.95 & 0.89 \\
  \hline
    \rule{0pt}{2ex}
     & +Age & TPR: & 0.70 & 0.52 & 0.67 & 0.68 \\
    3 & \textit{jittered} & ACC: & 0.84 & 0.73 & 0.82 & 0.80 \\
     & & PPV: & 0.96 & 0.91 & 0.95 & 0.89 \\
  \hline
    \rule{0pt}{2ex}
     & +Size & TPR: & 0.73 & 0.83 & 0.58 & 0.70 \\
    4 & (SLOC \& Comments) & ACC: & 0.85 & 0.73 & 0.79 & 0.80 \\
     & \textit{both jittered} & PPV: & 0.96 & 0.69 & 0.98 & 0.88 \\
  \hline
    \rule{0pt}{2ex}
     & +Experience & TPR: & 0.73 & 0.83 & 0.58 & 0.70 \\
    5 & \textit{jittered} & ACC: & 0.85 & 0.74 & 0.79 & 0.80 \\
     & & PPV: & 0.96 & 0.70 & 0.99 & 0.87 \\
  \hline
    \rule{0pt}{2ex}
     & +Time since last commit & TPR: & 0.72 & 0.88 & 0.52 & 0.70 \\
    6 & \textit{jittered} & ACC: & 0.84 & 0.72 & 0.76 & 0.80 \\
     & & PPV: & 0.96 & 0.67 & 0.99 & 0.88 \\
  \hline
    \rule{0pt}{2ex}
     & +Owner & TPR: & 0.68 & 0.90 & 0.52 & 0.77 \\
    7 & & ACC: & 0.83 & 0.73 & 0.76 & 0.83 \\
     & & PPV: & 0.97 & 0.68 & 0.99 & 0.87 \\
  \hline
    \rule{0pt}{2ex}
     & +File name & TPR: & 0.83 & 0.88 & 0.57 & 0.90 \\
    8 & & ACC: & 0.90 & 0.85 & 0.78 & 0.91 \\
     & & PPV: & 0.96 & 0.84 & 0.99 & 0.92 \\
  \hline
\end{tabular}

\end{table*}

\section{Results} \label{results}
The baseline of the attributes used for building each model, is the \textit{lines added} to each file on a single commit. Moving forward, new attributes are added (cumulatively) and their impact on the efficiency can be determined by the change of the metrics.

The performance of the models is given in Table~\ref{table:jit} and Figures~2-5. Bayesian Nets model yields the best results in the case of a single explanatory variable (\textit{lines added}) with $TPR = 0.68$, $ACC = 0.82$, and $PPV = 0.95$. However, as more explanatory variables are added to the model, C4.5 becomes the top performer. If we exclude the models that use \textit{file name} as the explanatory variable, the best model is based on C4.5 that is using \textit{lines added}, \textit{lines removed}, \textit{age}, and \textit{size} attributes as explanatory variable: $TPR = 0.73$, $ACC = 0.85$, and $PPV = 0.96$. Na\"{\i}ve Bayes and Logistic Regression models, in most cases, take second and third place in the performance competition (surpassing Bayesian Nets).

We will return to the models using \textit{file name} at the end of this section. Let us now analyse which factors affect injection of performance bugs, by analyzing the Logistic Regression (LR) models (because they are the most tractable and reasonably powerful) so that the management can take corrective actions. All the attributes in the LR models are statistically significant: $p$-values $< 0.01$. Recall and Accuracy of the LR models increase as illustrated by positive slopes in Figures~\ref{fig:tpr} and~\ref{fig:acc}, respectively. Precision of the LR models decreases with the addition of the attributes, with the exception of the \textit{file name} attribute (which we will discuss at the end of this section in details), as shown in Figure~\ref{fig:ppv}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\columnwidth]{./fig/tpr.pdf}
    \caption{Models' efficiency in term of Recall (True Positive Rate) for cumulatively added predictor variables. The labels of the $x$-axis correspond to the values in the `Row~\#' column of Table~\ref{table:jit}.}
    \label{fig:tpr}
\end{figure}

\begin{figure}[ht]
    \centering
    \includegraphics[width=\columnwidth]{./fig/acc.pdf}
    \caption{Models' efficiency in term of Accuracy for cumulatively added predictor variables. The labels of the $x$-axis correspond to the values in the `Row~\#' column of Table~\ref{table:jit}.}
    \label{fig:acc}
\end{figure}

As mentioned above, \textit{lines added} attribute has the highest predictive power. Peculiarly, addition of \textit{lines removed} to the model does not improve its predictive power significantly. This can be explained by the fact that bugs are mostly injected in new code rather than in modified\footnote{ A modified line is counted as one line removed and one line added.} or removed one. The sign of \textit{lines added} regression parameter is positive. This implies that in order to reduce the number of injected performance bugs, the management has to enforce reduction of the amount of changes delivered on a single code commit. This aligns with the findings that the data on the number of lines of code changed provides most of the information needed to predict functional bugs \cite{misirli2011different}. In essence, the more lines you change\footnote{In our case, we can narrow the type of change to code addition.}, the higher the probability that you will inject a bug. Our findings show that this statement is true not only for general functional bugs \cite{misirli2011different}, but also for performance bugs.

The attributes \textit{age} and \textit{size} improve performance of the models, but not significantly: e.g., compare $TPR = 0.67$ with $TPR = 0.70$. The sign of \textit{age} regression parameter is positive. From the management perspective, this suggests that a performance bug has higher probability of injections, as the software gets older, which is consistent with Lehman's laws of software evolution \cite{Lehman1980}. The signs of \textit{size} attributes \textit{SLOC} and \textit{comments} are negative, suggesting that larger files may have lower chances of injecting performance defect. However, given the low improvement in predictive power, the model may be capturing correlation rather than causality.

Merit of \textit{experience}, \textit{time since last commit}, and \textit{owner} is also arguable. They either do not enhance the predictive power of the model significantly or, in some cases, reduce it, as shown in Table~\ref{table:jit} and Figures~2-4. From the management perspective, this suggests that all developers (independent of their experience with the code base) have injected performance bugs in the past.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\columnwidth]{./fig/ppv.pdf}
    \caption{Models' efficiency in term of Precision for cumulatively added predictor variables. The labels of the $x$-axis correspond to the values in the `Row~\#' column of Table~\ref{table:jit}.}
    \label{fig:ppv}
\end{figure}

Inclusion of the \textit{file name} improves the performance: the best model is based on Logistic Regression with $TPR = 0.90$, $ACC = 0.91$, and $PPV = 0.92$. This is not surprising, given that our performance bugs reside in 9\% (32 out of 338) of the files. It implies that some files are more prone to performance bugs than others (as seen in other projects \cite{li2009analysis}). The management can focus quality assurance efforts on these 9\% of the files, e.g., making sure that code commits to these files are peer reviewed before merging them into the production code base.

However, if we reuse our model on a new project, inclusion of the file name attribute into the model may not be practical, since, at the beginning, information about files and new bugs will be limited~\cite{fukushima2014esj}. In this case we can resort to the general C4.5 based model mentioned above: it has lower recall and accuracy, but higher precision.

In the final model selection, comparison of True Positive and False Positive rates, shown in Figure~\ref{fig:tp_vs_fp}, might be of help. In this graph\footnote{This graph similar in nature to the receiver operating characteristic (ROC) curve \cite{bradley1997use}.}, we can measure the performance based on the True Positive rate, taking under consideration the drawback of the False Positive rate identification of the models. Since each case and application should be treated differently, we cannot explicitly propose a single model as the best performing one. We will discuss our final selection in Section~\ref{conclusion}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\columnwidth]{./fig/tp_vs_fp.pdf}
    \caption{Models' efficiency in term of True Positive Rate vs. False Positive Rate. The points on the graph represent models with different number of cumulatively added predictor variables. Points' labels correspond to the numbers given in the `Row~\#' column of Table~\ref{table:jit}.
    }
    \label{fig:tp_vs_fp}
\end{figure}

\section{Threats to Validity} \label{threats}
A number of tests are used to determine the quality of case studies. In this section we discuss four core tests: internal, construct, statistical, and external validity \cite{yin2013case}.

\textbf{Internal:} One of the authors of the study is affiliated with the company. In order to avoid researcher bias, we derived and followed strict automated processes for data extraction and processing. The results of extraction and processing were cross-validated by a researcher not affiliated with the company.
One of the most critical internal parts of our study is the patterns identification and validation, as explained in Section \ref{patterns}. Validating, benchmarking and leveraging the patterns was one of the most important parts of our study. We tested the patterns in multiple environments (by varying Operating System, Hardware Platform, and Programming Language version) and observed that the patterns lead to software slowdown in all environments.

Additionally and as already discussed in this paper, the real-time constraints that are externally set for the system under study, as well as the programming language that is used, might lead to bad interpretation of its real-time dynamic. However, this study begun and was applied on an existing, already developed and fully functioning software.

\textbf{Construct:} We construct our dataset based on the data collected from the source code management system. The system does not capture all events and activities happening in the organization. However, based on our results, even this incomplete set of data has strong predictive power.

\textbf{Statistical:} To prevent over-fitting of the models, we utilize 10-fold cross validation for model creation and the JOUS-Boost \cite{he2009learning} approach.

\textbf{External:} Generalizing our findings from a single-case design study to all situations is obviously not possible. However, this design is based on the rationale of the critical case \cite{yin2013case} of a Real Time System. Our results should be transferable to other researchers with well-designed and controlled experiments. In addition, the analysis can be replicated on open source software.

\section{Conclusion}\label{conclusion}
In this work we focused on detecting performance bugs that are important in multiple fields, such as mission critical applications, financial, and real time systems. Our software under study is a real time system used in the advertisement~/~marketing domain.

Our research question was: \emph{How can we detect performance bugs using static code analysis and attributes extracted from source code repository?} To answer the question, we described and followed a methodological approach that resulted in application of four different prediction algorithms, namely: C4.5 Decision Trees, Na\"{\i}ve Bayes, Bayesian Networks, and Logistic Regression.

The best model (not taking into account information about file names\footnote{A novel project may not have sufficient amount of file-name-related data~\cite{fukushima2014esj}.}) is obtained with C4.5, using lines of code added, lines of code removed, file age, and file size as explanatory variables ($TPR = 0.73$, $ACC = 0.85$, and $PPV = 0.96$). The best model, which  has file name available to it, is based on Logistic Regression with all the attributes included: $TPR = 0.90$, $ACC = 0.91$, and $PPV = 0.92$.

Furthermore, the analysis of the variables, based on Logistic Regression model, can aid management in setting corrective actions. In particular, injection of performance bugs can be reduced by decreasing the amount of changes delivered on a single commit\footnote{This will lead to increase of the number of commits. However (based on the authors' industrial experience as well as the literature~\cite{porter1998, kemerer2009}) individual commits will become easier to review, leading to improved defect removal.}, as well as by focusing  quality assurance resources on a small subset of error-prone files (9\% of the total number of files).

We believe that our methodological approach is of interest to practitioners, because it provides them with a simple and tractable model, using easily extractable code attributes for predicting performance bugs on a new code commit. In our models we omit pattern type information to show that the data, extracted automatically from version control system, is sufficient to predict performance bugs. This suggests that performance bugs mapped to non-functional requirements can be detected using the same attributes (particularly ``lines added'') as functional bugs.  This simplifies adoption of the models, since complexity and lack of human and/or hardware resources are often discouraging reasons. It also speeds up detection of a performance bug, as the model is applied to new code before it was committed to public code branch. If the model detects the performance bug, developer decides whether to review the code deeply or proceed as-is.

The resulting factors affecting projects may vary from project to project; however, practitioners may replicate model creation process, described in this study, tailoring it to their needs. Moreover, we describe how to process an extremely imbalanced dataset (where the number of non-defective records is $>20$ times larger than the number of defective ones), which can help practitioners facing similar issues.

The pattern identification that we conducted and briefly described in this paper, can also be considered as a guide for performance oriented best practices for the Python programming language. Additionally, these patterns can also be reused as static code analysis tools for identification of performance related fixes. They can also be considered as part of enhancement and development guidelines (e.g., Python's enhancement proposals \cite{pep}) for Python language.  Based on the methodology that we described, researchers can replicate similar pattern identification for other programming languages.

Our results are also of interest to theoreticians, since this work establishes a link between functional bugs and (non-functional) performance bugs, explicitly showing that attributes used for prediction of functional bugs can be used for prediction of performance bugs as well.

Going forward, we would like to extend our work to include other projects and company data.

\appendices

\section{Patterns}\label{app:patterns}

\subsection{Pattern 1}

An example of a Coding pattern is as follows. Python provides multiple methods to concatenate strings. For example,
\begin{lstlisting}
    foo =  'a_string'
    bar = 'abc' + 'def' + str1
\end{lstlisting}
is much slower than
\begin{lstlisting}
    foo = 'a_string'
    bar = '{0}{1}{2}'.format('abc', 'def', str1)
\end{lstlisting}

\subsection{Pattern 2} \label{p:2}
An example of a Design pattern is as follows. Python provides a default logging mechanism, which, in some cases, cannot avoid expensive function calls and/or calculations, even if these are finally not necessary/logged. A simple workaround is by doing a cheap function call before executing the actual log function. Essentially, the ``cheating'' function is just making sure that the current line will be logged before the execution of the expensive call. For example,
\begin{lstlisting}
    logger.setLevel(logging.CRITICAL)
    logger.debug(doSomethingExpensive())
\end{lstlisting}
the above example will spend all the time necessary to call the {\texttt{\small doSomethingExpensive()}} but finally won't use any of this information due to lower current logging level. However,
\begin{lstlisting}
    logger.setLevel(logging.CRITICAL)
    if logger.isEnabledFor(logging.DEBUG):
        logger.debug(doSomethingExpensive())
\end{lstlisting}
preliminary check of the log level can prevent unnecessary calculations.
In case of the if-block returning \texttt{FALSE}, the expense of calling \texttt{\small doSomethingExpensive()} is saved. We made sure that the combination of an if-block and the call of the \texttt{\small isEnabledFor(<level>)} is always cheaper or, at the very least, the same as the default logging function. It might be worth mentioning that the application of the above pattern, resulted in a very significant ($>15\%$) performance improvement - however we understand that this might not apply to every system.

Illustration of the remaining nine patterns are given below.

\subsection{Pattern 3}
Accessing global variables or built-in functions is slightly more time consuming than local.
\begin{lstlisting}
    def slightly_slower(asequence, adict):
        for x in asequence:
            adict[x] = hex(x)
\end{lstlisting}
\begin{lstlisting}
    def slightly_faster(asequence, adict):
        myhex = hex
        for x in asequence:
            adict[x] = myhex(x)
\end{lstlisting}
The latter is faster, because it is not accessing the built-in function globally in the second loop.

\subsection{Pattern 4}
Addition is faster than multiplication.
\begin{lstlisting}
    x+x
\end{lstlisting}
is faster than
\begin{lstlisting}
    x*2
\end{lstlisting}

\subsection{Pattern 5}
\texttt{if} statements are expensive in Python. It is recommended to use \texttt{if} statements only if you know that they will be executed only once. If they are going to be used more than one time (e.g., if statement within a for loop) it may be efficient to replace them (if possible).

\subsection{Pattern 6}
Loop unrolling - instead of a for loop:
\begin{lstlisting}
    def use_for():
        for i in range(1000):
            do(i)
\end{lstlisting}
use \texttt{map()}, if possible:
\begin{lstlisting}
    def use_map():
        map(do, range(1000))
\end{lstlisting}

\subsection{Pattern 7}
Built in functions are the way to go in all cases because they are written in \texttt{C}, which makes them faster than any other logic you might use \cite{python_functions}.
However, there may also be differences in between them, e.g. \texttt{type()} vs. \texttt{isinstance()}, and \texttt{range()} vs. \texttt{xrange()} (which is resolved in Python 3).

\subsection{Pattern 8}
Do not do:
\begin{lstlisting}
    for key in some_dict.keys()
\end{lstlisting}
but:
\begin{lstlisting}
    for key in some_dict
\end{lstlisting}

\subsection{Pattern 9}
Cache a method called in a loop (this is easily applied on methods that return consistent results). Method look-ups can be expensive.
So for example in:
\begin{lstlisting}
    for i in xrange(1000):
        myobj.compute(i)
\end{lstlisting}
you can eliminate the look-up:
\begin{lstlisting}
    compute = myobj.compute
    for i in xrange(1000):
        compute(i)
\end{lstlisting}

\subsection{Pattern 10}
Swapping (memory optimization). Avoid:
\begin{lstlisting}
    temp = x
    x = y
    y = temp
\end{lstlisting}
but do:
\begin{lstlisting}
    x, y = y, x
\end{lstlisting}

\subsection{Pattern 11}
Try to do operations in place, instead of creating a new instance (memory optimization). Do not do:
\begin{lstlisting}
    sortme = sorted(sortme)
\end{lstlisting}
but instead:
\begin{lstlisting}
    sortme.sort()
\end{lstlisting}

\section*{Acknowledgments}
The authors would like to thank the RTB development team of Addictive Mobility for the invaluable discussions, suggestions and feedback they provided during this research study. The research is supported in part by MITACS Accelerate Grant \# IT04452 and NSERC Discovery Grant RGPIN-2015-06075.

\bibliographystyle{IEEEtran}


Investigating naturalistic hand movements by behavior mining in long-term video and neural recordings

Abstract: Recent technological advances in brain recording and artificial intelligence are propelling a new paradigm in neuroscience beyond the traditional controlled experiment. Rather than focusing on cued, repeated trials, naturalistic neuroscience studies neural processes underlying spontaneous behaviors performed in unconstrained settings. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term. Here we describe an automated approach for analyzing simultaneously recorded long-term, naturalistic electrocorticography (ECoG) and naturalistic behavior video data. We take a behavior-first approach to analyzing the long-term recordings. Using a combination of computer vision, discrete latent-variable modeling, and string pattern-matching on the behavioral video data, we find and annotate spontaneous human upper-limb movement events. We show results from our approach applied to data collected for 12 human subjects over 7--9 days for each subject. Our pipeline discovers and annotates over 40,000 instances of naturalistic human upper-limb movement events in the behavioral videos. Analysis of the simultaneously recorded brain data reveals neural signatures of movement that corroborate prior findings from traditional controlled experiments. We also prototype a decoder for a movement initiation detection task to demonstrate the efficacy of our pipeline as a source of training data for brain-computer interfacing applications. Our work addresses the unique data analysis challenges in studying naturalistic human behaviors, and contributes methods that may generalize to other neural recording modalities beyond ECoG. We publicly release our curated dataset, providing a resource to study naturalistic neural and behavioral variability at a scale not previously available.

\section{Introduction}
Neuroscience has long been interested in understanding brain activity associated with spontaneous behaviors in freely behaving subjects.
Even so, hypotheses regarding brain function have typically been tested using carefully designed, well-controlled experimental tasks, where timing of cues, stimuli, and behavioral responses are known precisely.
Fortunately, recent technological advances have enabled us to study increasingly naturalistic and longer brain recordings, giving rise to a new paradigm called ``naturalistic neuroscience” \citep{nastase2020keep, huk_beyond_2018, gabriel_neural_2019-1, markowitz_striatum_2018, wang_unsupervised_2016} where neural computations associated with such spontaneous behaviors are studied.
Understanding such unstructured, long-term, and multi-modal data poses a substantial analytic challenge, due in part to the lack of \textit{a~priori} experimental design and the difficulty of isolating interpretable behavioral events.

\begin{figure*}[bthp!]
    \centering
    \includegraphics[width=1.0\linewidth]{fig_pipeline.pdf}
    \caption{
    Pipeline for behavioral video data processing.
     (a) Video frame showing estimated pose keypoints (colored dots) on human subject.
     (b) Autoregressive hidden Markov model (Section \ref{text:tss}) robustly segmented pose trajectory into \textit{rest} (shaded grey) and \textit{move} (shaded light blue) states.
     (c) Raster plot of pose states (\textit{move} in dark blue, \textit{rest} in white) for several video-clips for pattern matching at scale. Red box depicts one \textit{movement initiation} event matching a pattern of 15 contiguous \textit{rest} states (0.5s) followed by 15 contiguous \textit{move} states (0.5s).
    }
    \label{fig_pipeline}
\end{figure*}

\subsection{Related work}
Our work is related to several areas of active research in neuroscience, neuroengineering, and neuroethology that integrate techniques from machine-learning, computer-vision, and statistical modeling.
Many recent methodological innovations have addressed the automated analysis of \textit{non-human} animal behavior \citep{batty2019behavenet, pereira2019fast, nassar2018tree, mathis_markerless_2018, johnson2016nips, wiltschko_mapping_2015} (see also \citet{mathis2020deep} for a recent survey and \citet{anderson_toward_2014} for a perspective on this emerging area).
A typical non-human naturalistic neuroscience experiment \citep{johnson2020probabilistic, markowitz_striatum_2018, berman_measuring_2018} first collects simultaneously recorded behavioral video and neural activity data from one or more freely behaving subjects in an uncontrolled but sufficiently confined environment.
Next, the video recordings are processed through an extensive pipeline consisting of steps such as:
segmenting the subject(s) from the background,
transforming subject pose to common coordinates using affine transformations,
estimating pose of body-parts across frames,
and higher-level operations such as classifying pose or segmenting pose into actions.
Combined with the simultaneously recorded neural data, such naturalistic behavior data are being used to shed light on previously intractable questions in behavioral neuroscience, often at unprecedented scale.

Human action-recognition methods from mainstream computer vision \citep{ramasamy2018recent} are relevant but not directly applicable to the needs of naturalistic human neuroscience.
Traditionally, action-recognition research has concerned itself with discriminating activities at a coarse level, such as sitting vs. walking \citep{ghorbani2020movi}, and has often assumed the availability of a large corpus of labeled training data.
In contrast, to study the kinds of behaviors that interest neuroscientists and neuroengineers, we seek to localize fine-grained movements to sub-second temporal resolution, and ideally use the fewest behavioral labels possible \citep{seethapathi2019movement}.
Lastly, since it is not known which behaviors or behavioral characteristics will elicit neural responses worth studying further, a queryable representation that supports the flexibility to study several kinds of behaviors is desirable.
Recent work by \citep{fu_rekall_2019} develops such representations for semi-automated exploration of scenes in general videos.

Our work is most closely related to recent work in human naturalistic neuroscience that combine computer vision with opportunistic clinical brain recordings, including \citet{wang_unsupervised_2016}, \citet{alasfour2019coarse}, and \citet{chambers2019computer}.
In particular, we build on the work of \citet{wang2018ajile} using similar video data and estimate the pose of human upper-body keypoints using neural-networks.
\cite{gabriel_neural_2019-1} use optical flow and image partitioning to detect coarse limb movements from video taken in a clinical setting similar to ours and develop neural decoders for detecting these movements from brain data.
Compared to \citet{wang2018ajile}, who use a moving window heuristic on pose estimates to detect movements, we take a more principled approach to modeling the pose data.
This allows us to localize movement events with finer temporal-resolution and characterize entire movement trajectories, which in turn enables novel applications described later in the paper.

We also use newer, more efficient computer vision methods \citep{mathis2020deep, nath2019using} that allow us to process data at a scale that exceeds all of the aforementioned studies taken together in the number of subjects and duration of recordings analyzed.
Finally, we focus on curating, characterizing, and making our dataset available to the research community to foster further research and development in this area.
}

\subsection{Our approach}
We present a scalable behavior-mining approach to analyze simultaneously recorded naturalistic brain and behavior data, obtained opportunistically from human subjects undergoing long-term clinical monitoring prior to epilepsy surgery.
Our video processing pipeline (Figure~\ref{fig_pipeline}) first estimates the locations of keypoints (e.g. wrists and elbows) on the upper-body using a neural network trained on each subject~\citep{mathis_markerless_2018}.
We then segment the trajectory of each keypoint in time using discrete latent-variable models, building a discrete representation of pose dynamics.
Interestingly, having a discrete, sequential representation of upper-limb pose simplifies the problem of detecting behavioral \textit{events} to pattern-matching on strings.
Using regular-expressions corresponding to patterns of interest, we discover thousands of interpretable events per subject---an order of magnitude more observations than in a typical controlled human experiment.
To study the rich naturalistic variability associated with these events, we also extract metadata including movement angle, magnitude, and duration.

Next, we explore the use of these behavioral events for neuroscience and neuroengineering applications by analyzing the simultaneously recorded brain data.
Event-averaged spectrograms associated with our naturalistic human upper-limb movement initiation events corroborate and strengthen previous findings from controlled experiments \citep{miller_spectral_2007} (see also \citep{peterson2020behavioral}).
Preliminary investigations also suggest that our workflow could produce data useful for training brain-computer interface (BCI) decoders; due to the use of larger sample sizes of training data representative of naturalistic variability, such decoders may perform more robustly in real-world deployments.

Our key contributions in this paper are as follows.
First, we present a highly automated, novel workflow for analyzing simultaneously recorded naturalistic long-term human brain and behavioral video data.
Second, we develop a domain-relevant, robust, temporally precise, and queryable representation of human upper-limb pose.
Third, to showcase our workflow, we demonstrate example applications in neuroscience and neuroengineering, suggesting that our approach and results are of broad interest.
Finally, to support open science and facilitate further research in this area, we release our curated dataset consisting of annotated naturalistic events and associated neural recordings.

\section{Dataset}
\subsection{Human subjects and data collection}
Our dataset consists of human intracranial electrocorticography (ECoG) \citep{parvizi2018human} neural recordings and simultaneously recorded behavioral video recordings, obtained opportunistically from 12 patients with epilepsy for the duration of each patient's long-term (7--9 days) continuous clinical observation.
The University of Washington Institutional Review Board for the protection of human subjects approved our study and all patients provided their informed written consent.
Patient behavior was continuously recorded by a wall-mounted camera (RGB/infrared, 640 $\times$ 480 pixels) for real-time monitoring by an around-the-clock clinical team, except during intermittent equipment servicing or private times when the camera was switched off or turned away.
Patients were observed performing their daily activities (including talking, eating, watching TV, using a computer or phone, sleeping, receiving clinical care etc.) from the hospital bed while being tethered to a brain-recording interface.

Each patient had about 90 electrodes implanted under the skull and dura, directly on their brain surface, including either an \mbox{8 $\times$ 8} grid or two \mbox{8 $\times$ 4} grids of electrodes.
These electrodes were placed to monitor a subset of cortical regions, predominantly in one brain hemisphere, as determined by individual clinical need (right hemisphere for 5 and left hemisphere for 7 patients).
Grid electrodes are spaced 1 cm apart center-to-center.
The sampling rates for video and ECoG recordings were 30 fps and 1000 Hz respectively.
Together, the ECoG and video (approximately 18 million frames for a week) totaled about 250 GB of data per patient.

\subsection{ECoG and video data preprocessing}
We applied standard pre-processing steps to the ECoG data \citep{peterson2020behavioral, gabriel_neural_2019-1, miller2019library, miller_spectral_2007, schalk2007decoding}, including down-sampling to 500Hz, 60-Hz line noise removal, large-amplitude artifact removal, and  median-centering using a common reference across all electrodes.
All electrode positions were localized and converted to Montreal Neurological Institute and Hospital (MNI) coordinates using the Fieldtrip toolbox \citep{fieldtrip_ref, stolk2018} in MATLAB.
To aid interpretability, we restricted our analysis in this paper to the 64 grid electrodes covering one hemisphere per subject.
We excluded electrodes with recording issues, such as persistent presence of artifacts.
However, data for all available electrodes are provided in the publicly released dataset accompanying this paper.
In most cases, we also did not analyze the neurally and behaviorally atypical data from the first two days of a patient's hospital stay, since patients were usually heavily medicated during this time while recovering from electrode implantation surgery.

Before processing the video data, we manually inspected and annotated it at a coarse (every 3 minutes or so) level of granularity to create an \textit{omit-list} that was excluded from further processing.
The omit-list included long time-spans of sleep, times when a clinical or research team was actively working with the subject, private times, and times when applying computer-vision algorithms was impossible due to poor lighting conditions or severe occlusion of the subject's body.
Almost all camera movement occurs around times when the clinical team is actively working with the patient.
Removing these times results in a mostly steady recording configuration as seen in Figure \ref{fig_pipeline}(a).
We also labeled and excluded times when the clinical team had placed seizure restraints on the subjects' hands, since these limited mobility and gave rise to unnatural movements.
Completing these manual annotations took about 6--12 hours per subject, depending on their clinical treatment regime, activity and sleep schedule, and length of hospital stay.
When analysing ECoG and video data together, the two data-streams were synchronized using metadata extracted using the equipment manufacturer's (Natus Medical Incorporated) software.

\section{Methods}

\subsection{Precision extraction of behavioral events at scale}
We developed and validated a pipeline to extract temporally precise, interpretable movement events, by processing the video data through pose-estimation, pose time-series segmentation, event detection and finally, event metadata extraction (Figure~\ref{fig_pipeline}).

\subsubsection{Markerless pose estimation} \label{text:estimation}
To extract a subject's pose from raw video, we trained a state-of-the-art markerless pose estimation tool \citep{mathis_markerless_2018} known for its speed and data efficiency \citep{nath2019using}.
For training data, we manually annotated around 1000 frames per subject chosen randomly from the entire duration of a subject's video data, preferentially sampling active, daytime hours over times when the subject was asleep.
For each frame, we annotated up to 9 keypoints on the subject's body whenever visible \citep{PoseletsICCV09}.
These keypoints were the nose, both wrists, elbows, shoulders, and ears (Figure~\ref{fig_pipeline}a).

During prediction, the pose-estimation tool produced the $(x, y)$ coordinates and a \textit{confidence} estimate between $[0, 1]$ for each keypoint per frame (Figure~\ref{fig_pipeline}b).
To quantify the performance of keypoint tracking, we estimated the pixel-wise RMS error to be 1.54 $\pm$ 0.13 s.d. pixels on the training data and 5.97 $\pm$ 1.96 s.d. pixels on the holdout data, both averaged across 12 subjects;
holdout data is 5$\%$ of the manual annotations, excluding points below confidence threshold of $0.1$.
As an approximate scaling, $12$ pixels in the video span about $4$ cm in physical units, which is about the width of a human wrist.
We estimated this scale by comparing standard human measurements \citep{mcdowell2009anthropometric} with median distance between shoulder keypoints (in pixels) at movement onset for a few subjects.
On average, estimating pose for the entire duration of a subject's video took $400$ GPU-hours per subject using \texttt{AWS p2.16xlarge NVIDIA K80} GPUs.
We denoised the estimated pose trajectories by median filtering (window length $11$ frames) and smoothing (window length $11$, $2$nd order Savitzky-Golay \citep{schafer2011savitzky} filter).

\subsubsection{Segmentation of pose trajectories} \label{text:tss}
Next, we segmented the pose time-series into discrete, interpretable states while preserving the temporal precision of the keypoint tracking.
We applied a first-order autoregressive hidden semi-Markov model (ARHSMM) \citep{murphymlpp} with two latent states to each keypoint's time-series (Figure~\ref{fig_pipeline}b for left-wrist).
This model converts each keypoint's continuous pose dynamics into discretized dynamics consisting of \textit{rest} and \textit{move} \textbf{states}.
Using a semi-Markov, rather than a Markov model, accounts for the bias that limbs tend to be at \textit{rest} most of the time and mitigates unnecessary switching between latent states.
Similar to \citep{wiltschko_mapping_2015}, we fit the ARHSMM using the \texttt{pyhsmm-autoregressive} package in Python \citep{johnson2013hdphsmm}.
The resulting states are at video frame-rate resolution and the segmentation is relatively robust to variation in lighting, camera angle, and level of activity in the video.

\begin{figure}
    \centering
    \includegraphics[width=1.0\linewidth]{fig_cartoonbimanual.pdf}
    \caption{ Schematic of metadata extraction:
    [Top] Cartoon showing left-wrist movement at time of (A) movement initiation (B) maximum displacement from initiation (\textit{reach}) and (C) movement end.
    [Middle] Time course of left-wrist radial distance [px: pixels],

    and discretized state (R: \textit{rest}, M: \textit{move}).
    Extracted movement metadata include duration, start and end coordinates, among others (Sec. \ref{text:metadata}).

    [Bottom] Discretized state sequence for both left and right-wrists, now showing movement initiation in the right-wrist while the left-wrist is still in motion.
    The dashed line shows the window over which \textit{bimanual overlap} metadata is calculated, corresponding to the number of frames for which the opposing (left) wrist is in motion over that duration.
    }
    }
    \label{fig_cartoonbimanual}
\end{figure}

\subsubsection{Behavioral event mining} \label{text:mining}
Discretizing the pose trajectories facilitates the description of scientifically interesting behaviors performed spontaneously by the subject, even though they vary greatly in duration.
Specifically, the task of finding different types of behavioral \textbf{events} thus reduces to string pattern matching on the discretized dynamics.
For the behaviors we explore in the rest of this paper, we looked for \textit{movement initiation} events by matching a pattern of $15$ consecutive \textit{rest} states ($0.5$s), followed by at least $15$ consecutive \textit{move} states ($0.5$s).
Similarly, \textit{no-movement} events are state sequences of $90$ \textit{rest} states ($3.0$s) across both wrists and the nose.
To create our database of wrist movements, we use regular expressions to quickly find thousands of non-overlapping instances of such patterns in the discretized pose dynamics for each subject.

Parameters for smoothing, hyperparameters for the ARHSMM segmentation model, and the choice of regular expressions for event detection, were picked empirically by assessing performance on pose time-series derived from a small representative set of subject videos.
We confirmed that the temporal accuracy of event boundaries matched our expectations by manually inspecting a few dozen random events of each movement type for each subject.
}

\subsubsection{Event metadata extraction} \label{text:metadata}
For each detected movement event, we extracted several metadata features from the continuous pose-dynamics associated with the movement.
These include movement-associated metadata (Figure~\ref{fig_cartoonbimanual}) like the $(x,y)$ \textit{coordinates} of the keypoint at the start and end of the event,  \textit{duration} of the entire movement (up to next rest state), and \textit{rest duration} before and after movement.

Observed naturalistic hand movements often consisted of a hand reaching out, touching, or grabbing an object, then bringing the hand back to the body.
Therefore, we defined the \textit{reach} of a wrist movement to be its maximum radial displacement during the course of the event, as calculated from its location at the start of the event.
We extracted the \textit{magnitude}, \textit{angle}, and \textit{duration} for each reach.

To measure the \textit{shape} of a movement, we fit $1^{st}$, $2^{nd}$ and $3^{rd}$-degree polynomials to a keypoint's displacement trajectory.
Differences between the quality of the fit (as measured by $R^2$) to each polynomial type provide a rough measure of the ``curviness" of the movement trajectory.
We also estimated a movement's onset and offset \textit{speeds}, by calculating the keypoint's displacement change within short time windows around the start and end of the movement.

Since people often move both hands at the same time (i.e. ``bimanually"), we augmented each movement event with metadata about the opposing wrist's movement, if any (Figure~\ref{fig_cartoonbimanual}).
By juxtaposing the discrete state sequence of both wrists, we calculated when the opposing hand starts to move (\textit{lead/lag time difference}) and how long this movement overlaps with that of the primary hand (\textit{overlap duration}).

False positives in event discovery were still present in the data at this stage due to pose estimation failures and unusual pose states.
To compensate for failures in 2D pose estimation, we calculated movement-weighted \textit{confidence scores} for each event and removed those below a manually determined threshold.
To eliminate outlier pose states, we calculated mean \textit{distance} and mean \textit{angle} between \textit{shoulder} keypoints, then removed events from the top and bottom 5 percentiles of these quantities.

\begin{figure*}[tb!]
    \centering

    \includegraphics[width=1.0\linewidth]{fig_rastercounts_r.pdf}
    \caption{
    [Left] Number of right-wrist movement initiation events discovered per day for each of 12 subjects, totaling 475 to 3526 events per subject across their entire duration of clinical observation
    (268 $\pm$ 123 s.d. per day).
    [Right] Raster plot of right-wrist movement initiation event occurrences showing bursts of activity interspersed with periods of rest or omit-listed (Section \ref{text:estimation}) periods.
    See Figure A\ref{fig_rastercounts_l} for equivalent plots for left-wrist.
    }
    \label{fig_rastercounts_r}
\end{figure*}

\begin{figure}
    \centering
    \includegraphics[width=1.0\linewidth]{fig_trajectories.png}
    \caption{

    A sample of 50 typical right-wrist trajectories (px: pixels; translated to start at origin) showing diversity of naturalistic reach movements for a single subject (S10).
    Different colors represent different individual trajectories.
    Note the large variability in the movements, compared to what is normally captured by controlled experiments.}
    }
    \label{fig_trajectories}
\end{figure}

\begin{figure*}[t]
    \centering
    \includegraphics[width=1.0\linewidth]{fig_metadata_r.pdf}
    \caption{
    Histograms of right-wrist movement initiation event metadata per subject for their entire duration of clinical observation:
    (a) Reach magnitude shows a dominance of small movements,
    (b) Reach durations tended to be concentrated around $\approx$1s,
    (c) When both hands moved together (``bimanually"), they tended to start at about the same time.
    (d) Duration of time left hand was moving during a $\pm$0.5s window around time of right hand movement initiation.
    (e)
    Polar histograms show that many subjects primarily made upward-downward reaches.
    However, reaches in almost every other direction were also observed.
    See Figure A\ref{fig_metadata_l} for equivalent plots for left-wrist.
    }
    \label{fig_metadata_r}
\end{figure*}

\subsection{Neural correlates of movement} \label{text:encoding}
A core scientific question in systems neuroscience is how behaviors are encoded by the coordinated activation of brain regions.
To examine the neural correlates of naturalistic movement initiation, we performed a time-frequency (TF) analysis of the neural recordings \citep{cohen_analyzing_2014} by averaging event-locked spectrograms for each subject, using hundreds of movement initiation events chosen to match movement statistics (reach magnitude, onset velocity, and shape) of a previous controlled experimental study \citep{miller_spectral_2007}.
Using the aforementioned metadata to guide our search, we selected up to 200 events per day over 5 days for each of 12 subjects, and then further inspected the video for each event to remove any false positives (17.8$\%$ mean $\pm$ 9.9$\%$ s.d. events).

\subsection{Decoding naturalistic movement initiation} \label{text:decoding}
A grand challenge in neuroengineering is development of BCIs that can be used to predict spontaneous activity and intentions outside the lab, in everyday settings \citep{shanechi2019brain, smalley2019business, shanechi2018brain, warren2016recording, shenoy2012neural}.
Here we performed a preliminary study leveraging our pipeline as a source of training data for a BCI decoder that detects wrist movement initiation events.
Specifically, we trained separate classifiers, tailored to each subject, to discriminate between movement initiation events and no-movement epochs for each wrist using only features derived from the ECoG neural recordings.

Our decoder uses the Random Forest (RF) algorithm \citep{Breiman:2001:RF:570181.570182, murphymlpp}, which is typically considered one of the best off-the-shelf classification algorithms for small/medium sized datasets \citep{hastie2009elements}.
We used ECoG data $0.5$s before to $0.5$s after each event to compute TF spectrograms
at each of the 64 grid electrodes and used the flattened vector of TF bins as features for the classifier (TF bins were 200ms $\times$ 5Hz resolution, truncated at 150 Hz; approximately 9000 features total).

Given that the brain's response can drift over the course of days \citep{farshchian2018adversarial, klosterman2016day}, a reduced subset of events from 3 consecutive days (typically days 3 through 5 of clinical monitoring) were used.
We used events from the last day as the test set.
To eliminate the confound of movement initiation in the opposing wrist, we further filtered events to exclude those with significant movement ($\geq 0.2$ seconds) in the opposing wrist within the $\pm 0.5$s window used for ECoG data.
Positive (movement initiation) and negative (no-movement) examples were balanced by down-sampling negative examples.
This balancing eliminated bias in the training set and set up a baseline performance of $50 \%$ accuracy for test set performance.
Training and test supports were 633 $\pm$ 417 s.d. and 331 $\pm$ 203 s.d. examples, respectively.
We tuned the RF using a 20-trial randomized search over two hyperparameters: number of trees (range: $[50, 250]$) and maximum tree-depth (range: $[3, 15]$).
For each set of hyperparameters, 5-fold cross-validation holdout accuracy was used to measure performance.
Final performance reported is from training using best hyperparameters and corresponds to classifier accuracy on events from the withheld test day.

\section{Results}

\subsection{Characterizing naturalistic events} \label{text:analytics}
Our pipeline extracted 959 to 6745 individual wrist movement events per subject (487 $\pm$ 215 per day) across 12 subjects (Figures \ref{fig_rastercounts_r} and A\ref{fig_rastercounts_l}).
We found large variability between subjects in the number of events discovered, which we attribute to inter-subject differences in cycles of sleep and wakeful activity and clinical treatment regimes (Figure~\ref{fig_rastercounts_r} and A\ref{fig_rastercounts_l}).
We also observed rich within-subject variability in the event metadata (Figures \ref{fig_trajectories}, \ref{fig_metadata_r}, and A\ref{fig_metadata_l}), which further differentiates our dataset from those collected in controlled experiments.
Since our subjects received no instructions for when and how to move, we expect the observed movement statistics to closely reflect the natural statistics of human upper-limb movements while seated.

\begin{figure}[tb!]
    \centering
    \includegraphics[width=1.0\linewidth]{fig_spectrograms.png}
    \caption{
    Neural correlates of movement initiation:
    Event-locked spectrograms, averaged by brain region (cyan color in insets) across 12 subjects, showed movement-associated high-frequency power increase and low-frequency power decrease.
    These patterns corroborate and strengthen previous findings from controlled experiments \protect\citep{miller_spectral_2007}.
    See our companion preprint \protect\citet{peterson2020behavioral} for a deeper exploration of the behavioral and neural variability of these movements.}
    \label{fig_spectrograms}
\end{figure}

\subsection{Neural correlates of movement initiation}
We observed movement-associated power increases in a high-frequency band (76--100 Hz) and decreases in a low-frequency band (8--32 Hz) across many cortical areas (Figure~\ref{fig_spectrograms}).
This pattern is similar to what has been observed with controlled experimental trials in \citet{miller_spectral_2007}, \citet{volkova2019decoding}, and \citet{Yuan2014BrainComputerIU}.
Furthermore, we strengthen prior findings by showing that these movement-associated patterns hold across 5 consecutive days, well beyond the timescale of a typical controlled experiment (less than hours).
To the best of our knowledge, this is the first reported instance of a TF analysis of spontaneous naturalistic movements using events discovered by an automated workflow.
We elaborate on these results in our concurrently released preprint \citep{peterson2020behavioral}, where we further investigated the consequences of the relatively higher variance of naturalistic movement statistics and modeled the contributions of the various movement metadata to the observed neural responses.

\subsection{Decoding naturalistic movement initiation}
\begin{figure}[tb!]
    \centering
    \includegraphics[width=1.0\linewidth]{decoder_acc.png}
    \caption{Test set decoding accuracy for initiation of movement of contralateral (side opposite electrode implant) and ipsilateral (same side) wrists:
    As expected, decoding of contralateral movements is slightly more accurate than ipsilateral in almost all cases.
    }
    \label{fig_accuracy}
\end{figure}

\begin{figure*}[tbhp!]
    \centering
    \includegraphics[width=1.0\linewidth]{fig_nfi_e8.pdf}
    \caption{Contralateral and ipsilateral wrist movement initiation decoder feature importance scores aggregated by electrode (NFI$_E$),
    showing spatial contributions of different brain regions.
    Scores are normalized by dividing by highest electrode score for each decoder.
    Electrode coverage over motor cortex is highly correlated with decoder accuracy;
    for instance, subjects having good motor cortex coverage (S07, S06, S03 and S11) have the highest decoding performance (Figure~\ref{fig_accuracy}).
    See Figure A\ref{fig_nfi_e12} for plot with all 12 subjects.
    }
    \label{fig_nfi_e8}
\end{figure*}

\begin{figure}[tbhp!]
    \centering
    \includegraphics[width=0.99\linewidth]{fig_nfi_freq.pdf}
    \caption{

    Decoder feature importance scores aggregated by frequency and normalized by dividing by score of highest frequency bin per subject (NFI$_F$).
    Heatmaps show that the most relevant spectral features tend to come from a low-frequency band ($<$ 35Hz) and a high-frequency band (around 100 Hz), similar to prior findings \protect\citep{miller_spectral_2007}.
    When motor cortex electrode coverage is lacking (e.g. contralateral S02 and S05) or if ipsilateral wrist movement is being decoded, spectral feature contributions tend to be more broadly distributed across the frequency spectrum, and correlated with lower decoding accuracy.
    }
    }
    \label{fig_nfi_freq}
\end{figure}

Individual subject classifier performance varied widely between subjects, ranging from around chance levels to $80\%$ on test accuracy (Figure~\ref{fig_accuracy}), comparable to previously reported work \citep{gabriel_neural_2019-1,wang2018ajile}.
Classifier performance tended to be correlated with extent of motor cortex coverage (Figure~\ref{fig_nfi_e8} and A\ref{fig_nfi_e12}).
Due to hemispheric lateralization of brain function, decoding contralateral limb movements is expected to be more accurate than decoding ipsilateral movements \citep{tam2019human}.
Consequently, test set decoding accuracy (Figure~\ref{fig_accuracy}) was higher for the contralateral wrist in almost all subjects.
Since false positives (FPs) in the event data establish a ceiling on classifier accuracy, we estimated their prevalence by manually inspecting 100 randomly sampled events per event-type from each subject
(5$\%$ $\pm$ 10$\%$ s.d. for no-movement, 22$\%$ $\pm$ 16$\%$ s.d. for contralateral, and 14$\%$ $\pm$ 10$\%$ s.d. for ipsilateral events).
With more stringent rejection of FPs, we expect improved decoding performance and potentially more pronounced differences between contralateral and ipsilateral decoding accuracy.

To interpret the importance of spectral features in the decoder, we  visualized Random Forest feature importance scores  \citep{Breiman:2001:RF:570181.570182, hastie2009elements}.
We aggregated these scores in two ways to gain insight into their spatial (Figure~\ref{fig_nfi_e8} and A\ref{fig_nfi_e12}) and frequency (Figure~\ref{fig_nfi_freq}) components.
Spectral features are indexed by electrode, time, and frequency.
We define Feature Importance aggregated by Electrode $FI_E(e)$ for electrode $e \in \mathcal{E}$ as:
$${FI}_E(e) = \sum\limits_{ f \in \mathcal{F}, t \in \mathcal{T} } FI(t, f, e),$$
where $\mathcal{E}, \mathcal{F}$ and $\mathcal{T}$ are the sets of electrodes, frequency-bins, and time-bins over which the spectral features are calculated, respectively.
For the purpose of visualization, we normalized these values to get Normalized Feature Importance aggregated by Electrode $NFI_E(e)$:
$$NFI_E(e) = \frac{ FI_E(e) }{ \max_{e \in \mathcal{E} } FI_E(e) }. $$
As seen in Figures \ref{fig_nfi_e8} and A\ref{fig_nfi_e12}, we found that electrodes over sensorimotor cortex, when available, dominated feature importance (e.g. Subjects S07, S06, S03 and S11 in Figure~\ref{fig_accuracy}).
When motor cortex coverage is limited or unavailable, decoding is still possible because inter-region correlations are known to exist in the brain \citep{tam2019human,miller_spectral_2007,schalk2007decoding} and were likely exploited by the classifier, but with limited decoding capacity.

To understand the contributions of various frequency-bins to decoding, we define analogous formulas for feature importances aggregated by frequency-bin:

$$FI_F(f) = \sum\limits_{ e \in \mathcal{E}, t \in \mathcal{T} } FI(t, f, e)$$

$$NFI_F(f) = \frac{ FI_F(f) }{ \max_{f \in \mathcal{F} } FI_F(f)}$$

As seen in Figure~\ref{fig_nfi_freq}, we found that a low-frequency band ($<$ 35Hz) and a high-frequency band (around 100 Hz) dominates feature importance, similar to prior findings \citep{miller_spectral_2007}.
If ipsilateral wrist movement was being decoded, or when motor cortex electrode coverage was lacking (e.g. contralateral S02 and S05 in Figures \ref{fig_nfi_e8} and \ref{fig_nfi_e12}), spectral feature contributions tended to be more broadly distributed across the frequency spectrum and correlated with lower decoding accuracy.

\section{Discussion}

In summary, we have developed a highly automated and scalable approach for analyzing long-term datasets of simultaneously collected human brain and naturalistic behavior data.
Our workflow robustly uncovered and annotated thousands of human upper-limb movement events in behavior videos.
To detect movement events, we first discretized pose time-series for each wrist into two latent states, indicating movement or rest, and then used regular expressions to look for user-specified patterns in the latent state sequences.
This semi-supervised strategy allowed us to rapidly explore movements and their associated brain responses.
Importantly, our curated naturalistic dataset supported direct comparison with existing literature from controlled experiments.
To demonstrate the applicability of our workflow, we analyzed the brain data associated with the annotated events from two perspectives:
characterizing neural correlates of movement, and decoding naturalistic movement initiation using ECoG data.
Key to the success of our applications is the availability of a large number of repeated instances of movement initiation events, all available with high temporal precision, which is an essential requirement for generating event-averaged spectrograms \citep{cohen_analyzing_2014}.
The ability to select movements by magnitude, onset velocity, and complexity (using shape metadata) allowed us to match movement statistics between naturalistic and controlled experimental data, enabling a fair comparison.
Furthermore, the ability to select events without opposing wrist activity allowed us to disambiguate confounds when comparing movement decoders for opposing wrists.

\subsection{Limitations and future work}
Our work has a number of limitations that can be improved with further development.
First, our strategy of discretizing individual keypoint time-series to two latent states and then pattern-matching on latent state sequences may be challenging with more complex behaviors involving coordinated movement of more keypoints.
When we increased the number of latent states in the pose segmentation process, we also noticed that behavioral states were harder to interpret and associated ECoG responses were not easily separable.
The automated analysis of behavior for simple model organisms such as worms \citep{gupta2019context}, zebrafish \citep{johnson2020probabilistic}, flies \citep{berman2016predictability} and mice \citep{luxem2020identifying, markowitz_striatum_2018,datta2019q}, has advanced to the extent of being able to automatically extract hierarchies of coordinated behavioral sequences (or \textit{grammars}) from naturalistic videos.
Except for some very limited work \citep{SummersStay2012UsingAM, Yang2013ACS}, such progress has been elusive in human computer vision, possibly due to the sheer complexity and variability of human movements in various contexts.
Though not tailored to our temporal precision requirements, future research in fine-grained human action recognition in sports \citep{shao2020finegym, piergiovanni2018fine}, domestic \citep{rohrbach2012database} and industrial \citep{kobayashi2019fine} contexts could eventually provide methods that enable the collection of massive datasets of finely annotated human behavior.

All of our data was acquired opportunistically and videos were recorded from a single clinical monitoring camera.
Thus, a primary drawback of the event metadata generated by our pipeline is that they are derived from pose-estimation on single-camera RGB images, implying that all pose coordinates are 2D projections and that the fidelity of pose-derived metadata is limited.
However, one can still extract utility from the event metadata by coarse-graining or binning them.
The kinematic dataset could be made significantly richer by the use of additional hardware, such as an RGB-D (RGB with depth) camera or a stereoscopic camera system, that would enable pose-estimation and object-tracking in 3D \citep{karashchuk2020anipose, hansen2019fusing, sarafianos20163d}.

We controlled false positives in the event discovery process using a combination of pose-estimation confidence and a tedious manual omit-listing process.
We found the confidence estimate provided by our pose-estimation tool to  perform well under conditions of good visibility, but it was sensitive to variations arising from naturalistic lighting and occlusions.
One potential source of improvement could come from using pose-estimation algorithms that employ body models, such as OpenPose \citep{cao2018openpose}.
In our assessment, DeepLabCut \citep{mathis2020deep, nath2019using} offered a better speed (cost) vs. accuracy tradeoff at the scale we deployed for pose-estimation.
Future work is poised to take advantage of rapid innovations in computer vision, as more tools become available and accessible.
While manual creation and review of an omit-list cannot be completely avoided for compliance with human research protocols, we believe that a stereoscopic or depth based camera system could also help detect occlusions better and lead to a reduction in false positives.

Finally, two limitations arise from the opportunistic data-collection paradigm itself.
First, we have limited our study to a subject's wrists because they are relatively unconstrained and can perform spontaneous naturalistic movements compared to the rest of the subject's body.
Our subjects' heads are tethered to a brain recording device that partially restricts the movement of the rest of their upper body.
The study of more naturalistic, especially more active, behaviors would require wireless recording.
Second, ECoG data such as ours has been obtained opportunistically from a neuro-atypical patient population undergoing long-term monitoring preceding invasive epilepsy resection surgery.
We note with caution that conclusions from analyzing such data might not generalize well to the broader, neuro-typical population.

\subsection{Opportunities using curated dataset}
Accompanying our manuscript, we have publicly released our curated dataset comprising neural data and event metadata for over 40,000 instances of naturalistic human upper-limb movement events, and an equal number of rest events, across 12 subjects over about a week of clinical monitoring each.
We expect our dataset to be broadly applicable to BCI research like previously released datasets such as the BCI competitions I--IV datasets \citep{sajda2003data, blankertz2004bci, blankertz2006bci, tangermann2012review}, or other ECoG data libraries \citep{miller2019library} that were generated through controlled experimentation.
While the aforementioned datasets consist of 10s--100s of repeated instances of a behavior per subject, our dataset provides 1000s of instances per subject.
It captures rich naturalistic variability across multiple axes relating to the neural activity (subject, seizure foci, day of observation, electrode placement, and recording fidelity) and the behavior (subject activity profile, medication and treatment regime; wrist movement times, movement handedness and sequencing; and other event metadata).

We are working on refinements and extensions of the preliminary investigations presented here, and believe that our dataset could serve several other lines of scientific inquiry.
A thorough analysis of neural encoding of wrist movement initiation is available in our simultaneously released preprint \citep{peterson2020behavioral}.
As a follow-up to the limited prototype described here, we are currently exploring the use of neural networks to build more expressive decoders \citep{roy2019deep} for events and their associated event metadata.
The abundant availability of neural data also allows us to explore representation learning to obtain interpretable task-specific neural features \citep{pailla2019autoencoders, shiraishi2020neural}, and transfer learning to adapt decoders trained for one subject to another \citep{wu2020transfer, elango2017sequence, shenoy2007generalized}.

Our dataset is amenable to several types of modeling goals and approaches including
unsupervised latent factor modeling to extract single-trial neural dynamics \citep{pandarinath2018latent, ly2018electrocorticographic, cole2019cycle, pandarinath2018inferring, zhao2016interpretable},
dynamical modeling of the electrocorticographic spectrum \citep{chaudhuri2018random, beck2018state, haller2018parameterizing, brunton2016extracting},
probabilistic modeling to better understand neural data variability across trials, subjects and brain-regions \citep{omigbodun2016hidden, yang2019dynamic, abbaspourazad2018identifying, yang2017dynamic},
generative modeling to generate synthetic brain data \citep{hartmann2018eeg, aznan2019simulating}
and modeling the non-stationarity of the brain signal over long recording time spans \citep{farshchian2018adversarial, klosterman2016day, shenoy2006towards}.
We hope that our dataset will enable further research on models of neural function that incorporate naturalistic variability.

The presence of false positives in the data also motivates exploring algorithms for machine learning with noisy labels \citep{rolnick2017deep, natarajan2013learning, han2018co}.
This paradigm has been well studied for other applications of machine learning such as computer vision \citep{li2017learning}, where amassing large datasets with noisy labels is relatively inexpensive, but quality labeling is expensive to obtain.
The large behavioral variability associated with our neural data could also be used to investigate optimal training set selection, i.e. what types of and how much training data could be ideal for training a decoder \citep{wei2015submodularity, wei2014unsupervised, krause2008robust}.
Such characterizations could be used to inform the engineering of BCIs, making them significantly more robust to the variations present in real-world deployments.

\section*{Code and dataset release}
Code to reproduce several key plots in this manuscript is publicly available at: \url{https://github.com/BruntonUWBio/singh2020}.
Our complete curated dataset consisting of events and their metadata and associated neural data can also be downloaded following the instructions provided at the aforementioned URL.

\section*{Acknowledgements}

We thank John So for extensive help with manual annotation of the video data.
This work benefited from and was enabled by the groundwork laid by Nancy X. R. Wang towards study approval, initial clinical data procurement, preprocessing, and manual annotation, and establishing the plausibility of movement initiation prediction using a subset of this clinical data.
We thank the neurosurgeons Dr. Jeffrey G. Ojemann and Dr. Andrew Ko, and the staff and consenting patients at the University of Washington Harborview Medical Center in Seattle, for supporting this research.
We thank Pierre Karashchuk, Kameron D. Harris, James Wu, Nile Wilson, Ariel Rokem, Renshu Gu, David J. Caldwell, and Preston Jiang for helpful discussions and suggestions.
This work was funded by NSF award (1630178) and DOD/DARPA award (FA8750-18-2-0259) to BWB and RPNR, NSF award EEC-1028725 to RPNR, the Alfred P. Sloan Foundation (BWB), and the Washington Research Foundation (BWB).

\section*{Author Contributions}
SHS, RPNR, and BWB conceived of the study/analysis.
SHS and SMP performed the data analysis.
SHS, SMP, RPNR, and BWB interpreted the results.
SHS and BWB wrote the manuscript.
SHS, SMP, RPNR and BWB edited the manuscript.
RPNR and BWB acquired funding for the project.

}

\newcommand{\newblock}{}
\bibliographystyle{abbrvnamed}
\interlinepenalty=10000

Neurons and Cognition (q-bio.NC); Computer Vision and Pattern Recognition (cs.CV); Image and Video Processing (eess.IV)