<commit_before>% vim:syntax=tex

In this section we describe the design of a case study in which we
compare topic models trained on changesets to those trained on snapshots.
We describe the case study using the Goal-Question-Metric approach~\cite{Basili-etal:94}.
The data and source code for the case study is available in this paper's online
appendix\footnote{\textbf{REVIEWER URL ONLY:} \url{http://cscorley.students.cs.ua.edu/cfl/}}.

\subsection{Definition and Context}

% TODO
Our \textit{goal} is to evaluate the usefulness of topic models built
from changesets.
The \textit{quality focus} of the study is on informing development
decisions and policy changes that could lead to software with fewer
defects.
The \textit{perspective} of the study is of a researcher, developer, or
project manager who wishes to gain understanding of the concepts or
features implemented in the source code.
The \textit{context} of the study spans the version histories of fourteen
open source systems.

Toward achievement of our goal, we pose the following research questions:
\begin{description}[font=\itshape\mdseries,leftmargin=10mm,style=sameline]
% TODO
    \item[RQ1] How do changeset-based topic models perform for feature location?
    \item[RQ2] How do \emph{temporal} simulations of changeset-based topic models perform for feature location?
\end{description}
At a high level, we want to determine the feasibility in using changesets
to train topic models for feature location.

In the remainder of this section we introduce the subjects of our study,
describe the setting of our study, and report our data collection and analysis procedures.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Subject software systems}

All of our subject software systems come from two publicly-available
datasets.  The first is a dataset of four software systems by Dit et
al.~\cite{Dit:2013} and contains method-level goldsets.  The second is
a dataset of fourteen software systems by Moreno et
al.~\cite{Moreno:2014} and contains class-level goldsets. The four
software systems in the first dataset also appear in the second,
supplying us with both class- and method-level goldsets for the queries.

% TODO put a table describing systems (how many SLOC, corpus sizes) here
% TODO put paragraph describing each system in english here

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Setting}

Our document extraction process is shown on the left side of Figure~\ref{fig:extract}
We implemented our document extractor in Python v2.7
using the Dulwich library\footnote{\url{http://www.samba.org/~jelmer/dulwich/}}
for interacting with the source code repository and 
Taser\footnote{\url{http://xxxxxxxxxxx}} for parsing source code.
We extract documents from both a snapshot of the repository at a tagged
snapshot and each commit reachable from that tag's commit.
The same preprocessing steps are employed on all documents extracted.

For our document extraction from a snapshot, we first parse each Java file using our tool, Taser.
Taser is a text extractor implemented in Java using an open source Java 1.5 grammar and ANTLR v3.
The tool extracts documents from the chosen source code entity type,
either methods or classes, and treats inner entities to be distinct from the outer entity.
We consider interfaces, enumerations, and annotation types to also be a class.
The text of inner an entity (e.g., a method inside an anonymous class)
is only attributed to that entity, and not the containing one.
Comments, literals, and identifiers within a entity are considered as text of the entity.
Block comments immediately preceding an entity are also included in this text.

To extract text from the changesets, we look at the output of viewing
the \texttt{git diff} between two commits.
In our changeset text extractor, we extract all text related to the
change, e.g., context, removed, and added lines; metadata lines are ignored.
Note that we do not consider where the text originates from,
only that it is text changed by the commit.\footnote{
The Apache Lucene and Solr projects were merged into a single, new repository
during their development.
We only use changes that affect each project's subfolder in the merged repository,
and also include all changes from the two pre-merge repositories in each project's respective corpus.
}


After extracting tokens, we split the tokens based on camel case,
underscores, and non-letters.
We only keep the split tokens; original tokens are discarded.
We normalize to lower case before filtering non-letters, English stop words~\cite{StopWords}, Java keywords, and words shorter than three characters long.
We do not stem words. \attn{cite reason?}

Our modeling generation is shown on the right side of Figure~\ref{fig:extract}.
We implemented our modeling using the Python library Gensim~\cite{Gensim}.
Our evaluation is two-fold: we compare results for both LDA and LSI.
We do this to evaluate the performance of using changesets as input
documents regardless of chosen algorithm.

\subsubsection{LDA configuration}

Gensim's LDA implementation is based on an online LDA by Hoffman et al.~\cite{Hoffman-etal:2010}
and uses variational inference instead of a Collapsed Gibbs Sampler.
Unlike Gibbs sampling, in order to ensure that the model converges for each document,
we allow LDA to see each mini-batch $5$ times by setting Gensim's initialization parameter \texttt{passes} to this value
and allowing the inference step $1000$ iterations over a document.
We set the following LDA parameters for all fourteen systems:
$500$ topics ($K$), a symmetric $\alpha=1/K$, and a symmetric $\beta=1/K$.
These are default values for $\alpha$ and $\beta$ in Gensim.

For the temporal evaluation, we found it beneficial to consider two other parameters: $\kappa$ and $\tau_0$.
As noted in Hoffman et al.~\cite{Hoffman-etal:2010}, it is beneficial to
adjust $\kappa$ and $\tau_0$ to higher values for smaller mini-batches.
These two parameters control how much influence a new mini-batch has on the model when training.
However, the implementation of Gensim at the time of this paper did not
include a way to adjust $\tau_0$, so we only adjust $\kappa$.
We chose $\kappa=2.0$ for all systems, because the temporal evaluation
would often have mini-batch sizes in single digits.

\subsubsection{LSI configuration}

% TODO
Gensim's LSI implementation is based on an online LSI by {\v R}eh{\r u}{\v r}ek\cite{Radim:2011}.
We set the number of topics the same as LDA for each project.
The remaining parameters are left at default values, including those used
during the temporal evaluation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Data Collection and Analysis}

% TODO
We create two corpora for each of our fourteen subject systems.
We then model the documents into topics.

To answer RQ1, we take the traditional approach to evaluation:


We build a topic model for two corpora:
\begin{enumerate}
    \item Source code entities of a snapshot
    \item All changesets leading up to the snapshot
\end{enumerate}

The snapshot-based model is our baseline for comparison in our study.

To answer RQ2, we must take a different approach.
Since both LDA and LSI can be used as online topic models, we can
``simulate'' the results of a feature location technique over time.

Summary:
\begin{enumerate}
    \item Determine which commits to stop at and create mini-batches out of the commits in between
    \item For each mini-batch:
    \begin{enumerate}
        \item Update the model with the new document
        \item Extract a snapshot of the source code at that commit.
        \item Infer from the model the document-topics for each entity in the snapshot
        \item Query the model with the feature request text.
        \item Rank the source code entities by how similar their doc-topic is to the query's.
    \end{enumerate}
\end{enumerate}


However, only the Dit et al. dataset includes traceability links between
the queries and the commits the goldsets are extracted from.
We did a best-effort search for the Moreno et al. traceability links
using regular expressions.
We were able to recover portions of the links for all subject systems in
the Moreno et al. dataset with the exception of Hibernate, Pig (both versions), and Lucene.
The issues in these subject systems could not be found by searching the commit logs.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Results}


% TODO
RQ1 asks how well a topic model built upon changesets performs against
one built on source code entities.
We also wanted to know if there were any differences of performance
between two popular algorithms, LSI and LDA.
Table~\ref{table:rq1summary} summarizes our results for each system.

RQ2 asks how well a simulation of using a topic model would perform as
it were to be used in real-time.
This is a much closer evaluation of a feature location technique to it
being used in an actual development environment.
We also wanted to know if there were any differences of performance
between two popular algorithms, LSI and LDA.
Table~\ref{table:rq2summary} summarizes our results for each system.

<commit_msg>description of each system followed by its url<commit_after>% vim:syntax=tex

In this section we describe the design of a case study in which we
compare topic models trained on changesets to those trained on snapshots.
We describe the case study using the Goal-Question-Metric approach~\cite{Basili-etal:94}.
The data and source code for the case study is available in this paper's online
appendix\footnote{\textbf{REVIEWER URL ONLY:} \url{http://cscorley.students.cs.ua.edu/cfl/}}.

\subsection{Definition and Context}

% TODO
Our \textit{goal} is to evaluate the usefulness of topic models built
from changesets.
The \textit{quality focus} of the study is on informing development
decisions and policy changes that could lead to software with fewer
defects.
The \textit{perspective} of the study is of a researcher, developer, or
project manager who wishes to gain understanding of the concepts or
features implemented in the source code.
The \textit{context} of the study spans the version histories of fourteen
open source systems.

Toward achievement of our goal, we pose the following research questions:
\begin{description}[font=\itshape\mdseries,leftmargin=10mm,style=sameline]
% TODO
    \item[RQ1] How do changeset-based topic models perform for feature location?
    \item[RQ2] How do \emph{temporal} simulations of changeset-based topic models perform for feature location?
\end{description}
At a high level, we want to determine the feasibility in using changesets
to train topic models for feature location.

In the remainder of this section we introduce the subjects of our study,
describe the setting of our study, and report our data collection and analysis procedures.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Subject software systems}

All of our subject software systems come from two publicly-available
datasets.  The first is a dataset of four software systems by Dit et
al.~\cite{Dit:2013} and contains method-level goldsets.  The second is
a dataset of fourteen software systems by Moreno et
al.~\cite{Moreno:2014} and contains class-level goldsets. The four
software systems in the first dataset also appear in the second,
supplying us with both class- and method-level goldsets for the queries.

% TODO put a table describing systems (how many SLOC, corpus sizes) here
% TODO put paragraph describing each system in english here
ArgoUML is a UML CASE tool that supports standard UML diagrams \{url{http://argouml.tigris.org/
}}. Eclipse is an intergrated development environment to develop applications in various programming languages \{url{https://www.eclipse.org/home/index.php
}}. JabRef is a tool for managing bibliographical reference data \{url{JabRef is a tool for managing bibliographical reference data.  
}}. JEdit is a Java text editor \{url{http://www.jedit.org/
}}. MuCommander is a cross-platform file manager \{url{http://www.mucommander.com/
}}. Apache BookKeeper is a distributed logging service \{url{http://zookeeper.apache.org/bookkeeper/
}}. Apache Derby is a relational database management system \{url{http://db.apache.org/derby/
}}. Hibernate is a java package used to work with relational databases \{url{http://hibernate.org/
}}. Apache Lucene is an information retrieval library written in Java \{url{http://lucene.apache.org/core/
}}. Apache Mahout is a tool for scaleable machine learning \{url{https://mahout.apache.org/
}}. Apache OpenJPA is object relational mapping tool \{url{http://openjpa.apache.org/
}}. Apache Pig is a platform for analyzing large datasets consisting of high-level language \{url{http://pig.apache.org/
}}. Apache Solr is an enterprised search platform \{url{http://lucene.apache.org/solr/
}}. Apache Tika is a toolkit for extracting metadata and text from various types of files \{url{http://tika.apache.org/
}}. Apache ZooKeeper is a tool that works as a coordination service to help build distributed applications \{url{http://zookeeper.apache.org/bookkeeper/
}}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Setting}

Our document extraction process is shown on the left side of Figure~\ref{fig:extract}
We implemented our document extractor in Python v2.7
using the Dulwich library\footnote{\url{http://www.samba.org/~jelmer/dulwich/}}
for interacting with the source code repository and 
Taser\footnote{\url{http://xxxxxxxxxxx}} for parsing source code.
We extract documents from both a snapshot of the repository at a tagged
snapshot and each commit reachable from that tag's commit.
The same preprocessing steps are employed on all documents extracted.

For our document extraction from a snapshot, we first parse each Java file using our tool, Taser.
Taser is a text extractor implemented in Java using an open source Java 1.5 grammar and ANTLR v3.
The tool extracts documents from the chosen source code entity type,
either methods or classes, and treats inner entities to be distinct from the outer entity.
We consider interfaces, enumerations, and annotation types to also be a class.
The text of inner an entity (e.g., a method inside an anonymous class)
is only attributed to that entity, and not the containing one.
Comments, literals, and identifiers within a entity are considered as text of the entity.
Block comments immediately preceding an entity are also included in this text.

To extract text from the changesets, we look at the output of viewing
the \texttt{git diff} between two commits.
In our changeset text extractor, we extract all text related to the
change, e.g., context, removed, and added lines; metadata lines are ignored.
Note that we do not consider where the text originates from,
only that it is text changed by the commit.\footnote{
The Apache Lucene and Solr projects were merged into a single, new repository
during their development.
We only use changes that affect each project's subfolder in the merged repository,
and also include all changes from the two pre-merge repositories in each project's respective corpus.
}


After extracting tokens, we split the tokens based on camel case,
underscores, and non-letters.
We only keep the split tokens; original tokens are discarded.
We normalize to lower case before filtering non-letters, English stop words~\cite{StopWords}, Java keywords, and words shorter than three characters long.
We do not stem words. \attn{cite reason?}

Our modeling generation is shown on the right side of Figure~\ref{fig:extract}.
We implemented our modeling using the Python library Gensim~\cite{Gensim}.
Our evaluation is two-fold: we compare results for both LDA and LSI.
We do this to evaluate the performance of using changesets as input
documents regardless of chosen algorithm.

\subsubsection{LDA configuration}

Gensim's LDA implementation is based on an online LDA by Hoffman et al.~\cite{Hoffman-etal:2010}
and uses variational inference instead of a Collapsed Gibbs Sampler.
Unlike Gibbs sampling, in order to ensure that the model converges for each document,
we allow LDA to see each mini-batch $5$ times by setting Gensim's initialization parameter \texttt{passes} to this value
and allowing the inference step $1000$ iterations over a document.
We set the following LDA parameters for all fourteen systems:
$500$ topics ($K$), a symmetric $\alpha=1/K$, and a symmetric $\beta=1/K$.
These are default values for $\alpha$ and $\beta$ in Gensim.

For the temporal evaluation, we found it beneficial to consider two other parameters: $\kappa$ and $\tau_0$.
As noted in Hoffman et al.~\cite{Hoffman-etal:2010}, it is beneficial to
adjust $\kappa$ and $\tau_0$ to higher values for smaller mini-batches.
These two parameters control how much influence a new mini-batch has on the model when training.
However, the implementation of Gensim at the time of this paper did not
include a way to adjust $\tau_0$, so we only adjust $\kappa$.
We chose $\kappa=2.0$ for all systems, because the temporal evaluation
would often have mini-batch sizes in single digits.

\subsubsection{LSI configuration}

% TODO
Gensim's LSI implementation is based on an online LSI by {\v R}eh{\r u}{\v r}ek\cite{Radim:2011}.
We set the number of topics the same as LDA for each project.
The remaining parameters are left at default values, including those used
during the temporal evaluation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Data Collection and Analysis}

% TODO
We create two corpora for each of our fourteen subject systems.
We then model the documents into topics.

To answer RQ1, we take the traditional approach to evaluation:


We build a topic model for two corpora:
\begin{enumerate}
    \item Source code entities of a snapshot
    \item All changesets leading up to the snapshot
\end{enumerate}

The snapshot-based model is our baseline for comparison in our study.

To answer RQ2, we must take a different approach.
Since both LDA and LSI can be used as online topic models, we can
``simulate'' the results of a feature location technique over time.

Summary:
\begin{enumerate}
    \item Determine which commits to stop at and create mini-batches out of the commits in between
    \item For each mini-batch:
    \begin{enumerate}
        \item Update the model with the new document
        \item Extract a snapshot of the source code at that commit.
        \item Infer from the model the document-topics for each entity in the snapshot
        \item Query the model with the feature request text.
        \item Rank the source code entities by how similar their doc-topic is to the query's.
    \end{enumerate}
\end{enumerate}


However, only the Dit et al. dataset includes traceability links between
the queries and the commits the goldsets are extracted from.
We did a best-effort search for the Moreno et al. traceability links
using regular expressions.
We were able to recover portions of the links for all subject systems in
the Moreno et al. dataset with the exception of Hibernate, Pig (both versions), and Lucene.
The issues in these subject systems could not be found by searching the commit logs.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Results}


% TODO
RQ1 asks how well a topic model built upon changesets performs against
one built on source code entities.
We also wanted to know if there were any differences of performance
between two popular algorithms, LSI and LDA.
Table~\ref{table:rq1summary} summarizes our results for each system.

RQ2 asks how well a simulation of using a topic model would perform as
it were to be used in real-time.
This is a much closer evaluation of a feature location technique to it
being used in an actual development environment.
We also wanted to know if there were any differences of performance
between two popular algorithms, LSI and LDA.
Table~\ref{table:rq2summary} summarizes our results for each system.


{'fraction_non_alphanumeric': 0.08278014465471119, 'fraction_numerical': 0.00515208560388388, 'mean_word_length': 4.601276359600444, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '467913', 'n_tokens_mistral': 5147, 'n_tokens_neox': 4899, 'n_words': 2826}