Code to Comment "Translation": Data, Metrics, Baselining & Evaluation

Abstract: The relationship of comments to code, and in particular, the task of generating useful comments given the code, has long been of interest. The earliest approaches have been based on strong syntactic theories of comment-structures, and relied on textual templates. More recently, researchers have applied deep learning methods to this task, and specifically, trainable generative translation models which are known to work very well for Natural Language translation (e.g., from German to English). We carefully examine the underlying assumption here: that the task of generating comments sufficiently resembles the task of translating between natural languages, and so similar models and evaluation metrics could be used. We analyze several recent code-comment datasets for this task: CodeNN, DeepCom, FunCom, and DocString. We compare them with WMT19, a standard dataset frequently used to train state of the art natural language translators. We found some interesting differences between the code-comment data and the WMT19 natural language data. Next, we describe and conduct some studies to calibrate BLEU (which is commonly used as a measure of comment quality). using "affinity pairs" of methods, from different projects, in the same project, in the same class, etc; Our study suggests that the current performance on some datasets might need to be improved substantially. We also argue that fairly naive information retrieval (IR) methods do well enough at this task to be considered a reasonable baseline. Finally, we make some suggestions on how our findings might be used in future research in this area.

\section{Background \& Theory}

The value of comments in code comprehension has been well-established~\cite{takang1996effects}. However, developers
find it challenging to create \& maintain useful comments~\cite{de2005study,Fluri2007DoCA}. This has sparked a long line of research looking into the problem of comment generation.
An early line of work~\cite{sridhara2011automatically,sridhara2010towards,moreno2013automatic,buse2008automatic}
was rule-based, combining some form analysis of the source code to extract specific information, which could
then be slotted into different types of templates to produce comments. Another approach was
to use code-clone identification to produce comments for given code, using the comments associated
with a clone~\cite{wong2015clocom}.
Other approaches used keywords which programmers seem to
attend to in eye-tracking studies~\cite{rodeghero2014improving}. Still other approaches use topic analysis to organize descriptions
of code~\cite{mcburney2014improving}.

Most of the pioneeering approaches above
relied on specific features and rules hand-engineered for the task of
comment generation.
The advent of large open-source repositories with large volumes of source-code offered a novel, general, statistically rigorous,
possibility:  that these large datasets be mined for code-comment pairs, which could then
be used to train a model to produce comments from code. The success of classic statistical
machine translation~\cite{koehn2009statistical} offered a tempting preview of this: using
large amounts of aligned pairs of utterances in languages $A$ \& $B$, it was possible to learn a
conditional distribution of the form $p_t(b \mid a)$, where $a \in A,~$ and $~ b \in B$; given
an utterance $\beta \in B$, one could produce a possible translation $\alpha \in A$ by simply setting
\[ \alpha = \argmax_a p_t(a \mid \beta) \]
Statistical natural language translation approaches, which were already highly performant,
were further enhanced by deep-learning (DL).
Rather
than relying on specific inductive biases like phrase-structures in the case of classical SMT, DL
held the promise that the features relevant to translation
could themselves be learned from large volumes of data.
DL approaches have led to phenomenal
improvements in translation quality~\cite{vaswani2018tensor2tensor,klein2017opennmt}.
Several recent papers ~\cite{iyer2016summarizing,hu2017codesum,leclair2019neural} have explored
using these powerful DL approaches to the code-comment task.

Iyer \textit{et al.}~\cite{iyer2016summarizing} first applied DL to this task,
using code-English pairs mined from Stack Overflow---using simple attention over input code,
and an LSTM to generate outputs. Many other papers followed, which are
discussed below in~\cref{sec:related}.
We analyze the published literature, starting with the question of whether there
are notable distributional differences between the code-comment translation (CCT) and the statistical machine translation (WMT) data.
Our studies examine the distributions of the input and output data, and the dependence of the output on the input.

\label{sec:bg}

\RQ{rq:dist}{What are the differences between the translation (WMT) data, and code-comment (CCT) data?}

Next, we examine whether these differences actually affect the performance of translation models. In earlier work, Allamanis~\cite{allamanis2019adverse} pointed out the effects of data duplication on machine learning applications in software engineering. We study the effects of data duplication, as well as the effects of distributional differences on deep learning models.
One important aspect of SMT datasets is \emph{input-ouput dependence}. In translation \emph{e.g.} from German (DE) to English (EN), similar input DE sentences will to produce similar output EN sentences, and less similar DE sentences will tend
to produce less similar EN sentences. This same correlation might not apply in CCT datasets.

\RQ{rq:perf}{How the distributional differences in the SMT \& CCT data\-sets affect the measured performance?}

There's another important difference between code and natural language. Small differences, such
as substituting $*$ for $+$ and a  $1$ for a  $0$, can make the difference between a \emph{sum}
and a \emph{factorial} function; likewise changing one function identifier (\emph{mean}, rather than \emph{variance}).
These small changes should result in a large change in the associated comment. Likewise, there
are many different ways to write a sort function, all of which might entail the same comment.
Intuitively, this would appear to be less of an issue in natural languages; since as they have evolved
for consequential
communication in noisy environments, meaning should be robust to small changes.
Thus on the whole, we might expect that small changes in German should in general result in only
small changes in the English translation. Code, on the other hand, being a \emph{fiat} language,
might not be in general as robust, and so small changes in code may result in unpredictable changes in the
associated comment. Why does this matter? In general, modern machine translation methods use the generalized
function-approximation capability of deep-learning models. If natural language translation (WMT) has
a more functional dependency, and CCT doesn't, there is a suggestion that deep-learning models would find CCT a
greater challenge.

\RQ{rq:bivar}{Do similar inputs produce similar outputs in both WMT and CCT datasets?}

Prior work in natural language generation has shown that information retrieval (IR) methods can be effective
ways of producing suitable outputs. These methods match a new  input with semantically similar inputs in the training data, and return the associated output. These approaches can sometimes perform quite well~\cite{henderson2019polyresponse} and has been previously applied successfully to the task of comment generation \cite{10.1145/3238147.3240471, Zhang2020RetrievalbasedNS}.
Our goal here is to ask whether IR methods could be a relevant, useful baseline for CCT tasks.

\RQ{rq:IR}{How do the performance of naive Information Retrieval (IR) methods compare across WMT \& CCT datasets? }

Finally, we critically evaluate the use of BLEU scores in this task. Given the differences we found between
datasets used for training SMT translators and the code-comment datasets, we felt it would be important
to understand how BLEU is used in this task, and develop some empirical baselines to calibrate the observed
BLEU values in prior work. How good are the best-in-class BLEU scores (associated with the best current methods
for generating comments given the source of a method)? Are they only as good as simply retrieving a
comment associated with a random method in a different project? Hopefully they're much better.
How about the comment associated with a random method
from the same project? With a random method in the same class?
With a method that could reasonably be assumed quite similar?

\RQ{rq:calib}{How has BLEU been used in prior work for the code-comment task, and how should we view the measured performance?}

In the next section, we review the datasets that we use in our study.
\section{Datasets Used}

We examine the characteristics of four CCT data sets, namely CodeNN, DeepCom,  FunCom, \& DocString and one standard, widely-used machine-translation dataset, the WMT dataset.
We begin with a description of each dataset. Within some of the CCT datasets, we observe that the more popular ones can include several different variations: this is because follow-on work has sometimes gathered, processed, and
partitioned (training/validation/test)  the dataset differently.

\label{sec:thedata}
\mypara{CodeNN} Iyer \emph{et al}~\cite{iyer2016summarizing}  was an early CCT dataset, collected from StackOverflow, with code-comment
pairs for C\# and SQL. Stackoverflow posts consist of a title, a question, and a set of
answers which may contain code snippets.
Each pair consists of the \emph{title} and \emph{code snippet} from answers.
Iyer \emph{et al} gathered around a million pairs each for C\# and SQL; from these, focusing on just snippets in \emph{accepted} answers, they filtered down to 145,841 pairs for C\#  and 41,340 pairs for SQL. From these, they used a trained model (trained using a hand-labeled set) to filter out uninformative titles (\emph{e.g.,} ``How can make this complicated
query simpler") to 66,015 higher-quality pairs for C\# and 33,237 for SQL. In our analysis, we used only the C\# data.
 StackOverflow  has a well-known community norm to avoid redundant Q\&A; repeated questions are typically referred to the earlier post. As a result, this dataset has \emph{significantly less duplication}. The other  CCT datasets are different.

\mypara{DeepCom} \citet{hu2018deep} generate a CCT dataset by mining 9,714 Java projects. From this dataset, they filter out methods that have Javadoc comments, and select only those that have at least one-word descriptions. They also exclude getters, setters, constructors and test methods. This leaves them with 69,708 method-comment pairs. In this dataset, the methods (code)
are represented as serialized ASTs after parsing by Eclipse JDT.

Later, \citet{Hu2019DeepCC} updated their dataset and model, to a size of 588,108 examples. We refer to the former as DeepCom1 and obtain a copy online from followup work\footnote{\label{neuralcodesum}\url{https://github.com/wasiahmad/NeuralCodeSum/tree/d563e58/data}}. We refer to the latter as DeepCom2 and obtain a copy online\footnote{\url{https://github.com/xing-hu/EMSE-DeepCom/tree/98bd6a}}. In addition DeepCom2 is distributed with a 10-fold split in the cross-project setting (examples in the test set are from different projects). In \citet{Hu2019DeepCC} this is referred to the "RQ-4 split", but to avoid confusion with our research questions, we refer to it as DeepCom2f.

\mypara{Funcom}
\citet{leclair2019neural} started with the Sourcerer~\cite{bajracharya2006sourcerer} repo, with over 51M methods from 50K projects.
From this, they filtered out methods with Javadoc comments in English, and then also
the comments that were auto-generated. This leaves about 2.1M methods with patched
Javadoc comments. The source code was parsed into an AST.
They created two datasets, the \emph{standard}, which retained the original identifiers,
and \emph{challenge}, wherein the identifiers (except for Java API class names) were
replaced with a standardized token. They also made sure no data from
the same project was duplicated across training and/or validation and/or test. Notably, the FunCom dataset only considers the first sentence of the comment. Additionally, code longer than 100 words and comments longer 13 words were truncated.

Like for DeepCom, there are several versions of this dataset. We consider a version from \citet{leclair2019neural} as FunCom1 \ and the version from \citet{DBLP:journals/corr/abs-1904-02660} as FunCom2. These datasets are nearly identical, but FunCom2 has about 800 fewer examples and the two versions have reshuffled train/test/val splits. The Funcom1\footnote{\url{http://leclair.tech/data/funcom/}} and Funcom2\footnote{\url{https://s3.us-east-2.amazonaws.com/icse2018/index.html}}
datasets are available online.

\mypara{Docstring} \citet{baroneDocstring} collect Python methods and prefix comment "docstrings" by scraping GitHub. Tokenization was done using subword tokenization. They filtered the data for duplications, and also removed excessively long examples (greater than 400 tokens).  However, unlike other datasets, Barone \emph{et al.} do not limit to only the first sentence of the comments. This can result in relatively long desired outputs.

The dataset contains approximately 100k examples, but after filtering out very long samples,
as per Barone \emph{et al} preprocessing script\footnote{\url{https://bit.ly/2yDnHcS}}, this is reduced to 74,860 examples. We refer to this version as DocString1.

We also consider a processed version obtained from \citet{ahmad2020transformerbased} source\textsuperscript{\ref{neuralcodesum}} which was attributed to \citet{wei2019code}. We refer to this version as DocString2. Due to the processing choices, the examples in DocString2 are significantly shorter than DocString1.

\mypara{WMT19 News Dataset}
To benchmark the comment data with natural language, we used data from the Fourth Conference of Machine Translation (WMT19). In particular, we used the news dataset~\cite{barrault-etal-2019-findings}. After manual inspection, we determined this dataset offers a good balance of formal language that is somewhat domain specific to more loose language common in everyday speech. In benchmarking comment data with natural language, we wanted to ensure variety in the words and expressions used to avoid biasing results. We used the English-German translation dataset, and compared English in this dataset to comments in the other datasets (which were all in English) to ensure differences in metrics were not a result of differences in language.

\impara{Other CCT Datasets}
We tried to
capture most of the code-comment datasets that are used in the context of translation. However, there are some recent datasets which could be used in this context, but we did not explore \cite{husain2019codesearchnet, agashe2019juice}. While doing our work we noticed that some prior works provide the raw collection of code-comments for download, but not the exact processing and evaluations used~\cite{mooreConvComment}. Other works use published datasets like DocString, but processing and evaluation techniques are not now readily available~\cite{wang2020trans3, 9031440}. As we will discuss, unless the precise processing and evaluation code is available, the results may be difficult to compare.

\subsection{Evaluation Scores Used}
\label{sec:evalused}

A common metric used in evaluating text generation is BLEU score \citep{papineni2002bleu}. When comparing translations of natural language, BLEU score has been shown to correlate well with human judgements of translation quality~\cite{coughlin2003correlating}. In all the datasets we analyzed, the associated papers used
BLEU to evaluate the quality of the comment generation. However,
there are rather subtle differences in the way the BLEUs were calculated, which makes
the results rather difficult to compare. We begin this discussion with a
brief explanation of the BLEU score.

BLEU (as do related measures) indicates the closeness of a candidate translation output
to a ``golden" reference result.
BLEU \emph{per se} measures the \emph{precision} (as opposed to \emph{recall}) of a candidate, relative to the reference,
using constituent
\textit{n}-grams.
BLEU typically uses unigrams through 4-grams to measure the precision of the system output. If we define \textit:
\[ p_n = \frac{\text{number of \textit{n}-grams in both reference and candidate}}{\text{number of \textit{n}-grams in the candidate}} \]

BLEU combines the precision of each \textit{n}-gram using the geometric mean, $\exp({\frac{1}{N}\sum_{n=1}^{N} \log{p_n}) }$. With just this formulation, single word outputs or outputs that repeat common \textit{n}-grams could potentially have high precision. Thus, a \emph{``brevity penalty''} is used to scale the final score; furthermore each \textit{n}-gram in the reference can be used in the calculation just once. \citep{eisenstein2018natural}
These calculations are generally standard in all BLEU implementations, but several variations may arise.

\impara{Smoothing:} One variation arises when deciding how to deal with cases when $p_n = 0$, \emph{i.e.},  an $n$-gram in the candidate string is not in the reference string \cite{Chen2014ASC}.
With no adjustment, one has an undefined $\log{0}$. One can add a small epsilon to $p_n$ which removes undefined expressions.
However, because BLEU is a geometric mean of $p_n, n \in \{1, 2, 3, 4\}$ if $p_4$ is only epsilon above zero, it will result in a mean which is near zero. Thus, some implementations opt to smooth the $p_n$ in varying ways. To compare competing tools
for the same task, it would be preferable to use a standard measure.

\impara{Corpus vs. Sentence BLEU:}
When evaluating a translation system, one typically measures BLEU
(candidate \emph{vs} reference) across all the samples in the held-out test set. Thus another source of implementation variation is when deciding how to combine the results between all of the test set scores.
One option, which was proposed originally in Papineni \textit{et al.}~\cite{papineni2002bleu}, is a "corpus BLEU", sometimes referred to as C-BLEU. In this case the numerator and denominator of $p_n$ are accumulated across every example in the test corpus. This means as long as at least one example has a 4-gram overlap, $p_4$ will not be zero, and thus the geometric mean will not be zero
An alternative option for combining across the test corpus is referred to as "Sentence BLEU" or S-BLEU. In this setting BLEU score for the test set is calculated by simply taking the arithmetic mean the BLEU score calculated on each sentence in the set.

\impara{Tokenization Choices:} A final source of variation comes not from how the metric is calculated, but from the inputs it is given. Because the precision counts are at a token level, it has been noted that BLEU is highly sensitive to tokenization \citep{post2018clarity}. This means that when comparing to prior work on a dataset, one must be careful not only to use the same BLEU calculation, but also the same tokenization and filtering. When calculating scores on the datasets, we use the tokenization provided with the dataset.

Tokenization can be very significant for the resulting score. As a toy example, suppose a reference contained the string ``{\small\tt calls function foo()}'' and an hypothesis contained the string ``{\small\tt uses function foo()}''. If one chooses to tokenize by spaces, one has tokens [{\small\tt calls}, {\small\tt function}, {\small\tt foo()}] and [{\small\tt uses}, {\small\tt function}, {\small\tt foo()}]. This tokenization yields only one bigram overlap and no trigram or 4-gram overlaps. However, if one instead chooses to tokenize this as [{\small\tt calls}, {\small\tt function}, {\small\tt foo}, {\small\tt (}, {\small\tt )}] and [{\small\tt uses}, {\small\tt function}, {\small\tt foo}, {\small\tt (}, {\small\tt )}] we suddenly have three overlapping bigrams, two overlapping trigrams, and one overlapping 4-gram. This results in a swing of more than 15 BLEU-M2 points or nearly 40 BLEU-DC points (BLEU-M2 and BLEU-DC described below).

We now go through BLEU variants used by each of the datasets and assign a name to them. The name is not intended to be prescriptive or standard, but instead just for later reference in this document. All scores are the "aggregate" measures, which consider up to \textit{4}-grams.

\mypara{BLEU-CN} This is a Sentence BLEU metric. It applies a Laplace-like smoothing by adding 1 to both the numerator and denominator of $p_n$ for $n \geq 2$. The CodeNN authors' implementation was used \footnote{\url{https://github.com/sriniiyer/codenn/blob/0f7fbb8b298a8/src/utils/bleu.py}}.

\mypara{BLEU-DC} This is also a Sentence BLEU metric. The authors' implementation is based off NLTK~\cite{LoperBird02} using its "method 4" smoothing. This smoothing is more complex. It only applies when $p_n$ is zero, and sets  $p_n = 1 / ((n-1) + 5/\log{l_h})$ where $l_h$ is the length of the hypothesis. See the authors' implementation for complete details\footnote{\url{https://github.com/xing-hu/EMSE-DeepCom/blob/98bd6aac/scripts/evaluation.py}}.

\mypara{BLEU-FC} This is an unsmoothed corpus BLEU metric based on NLTK's implementation. Details are omitted for brevity, and can be found in the authors' source\footnote{\url{https://github.com/mcmillco/funcom/blob/41c737903/bleu.py\#L17}}.

\mypara{BLEU-Moses} The Docstring dataset uses a BLEU implementation by the Moses project\footnote{\url{https://bit.ly/2YF0hye}}. It is also an unsmoothed corpus BLEU. This is very similar to BLEU-FC (though note that due to differences in tokenization, scores presented by the two datasets are not directly comparable).

\mypara{BLEU-ncs} This is a sentence BLEU used in the implementation\footnote{\url{https://github.com/wasiahmad/NeuralCodeSum/blob/b2652e2/main/test.py\#L324}} of Ahmad et al. \cite{ahmad2020transformerbased}. Like BLEU-CN, it uses an add-one Laplace smoothing. However, it is subtly different than BLEU-CN as the add-one applies even for unigrams.

\mypara{SacreBLEU} The SacreBLEU implementation was created by \citet{post2018clarity} in an effort to help provide a standard BLEU implementation for evaluating on NL translation. We use the default settings which is a corpus BLEU metric with an exponential smoothing.

\mypara{BLEU-M2} This is a Sentence BLEU metric based on nltk "method 2" smoothing. Like BLEU-CN it uses a laplace-like add-one smoothing. This BLEU is later presented in plots for this paper.

\smallskip
We conclude by noting that the wide variety of BLEU measures used in prior work in code-comment translation carry some risks. We discuss further
below. \cref{tab:diffBLEU} provide some evidence suggesting that the variation is high enough to raise some concern about the true interpretation of claimed advances; as we argue below, the field can benefit from further standardization.

\subsection{Models \& Techniques}
~\label{sec:related}
In this section, we outline the various deep learning approaches that have been
applied to this code-comment task. We note that our goal in this paper is not to critique or improve upon
the specific technical methods, but to analyze the data \emph{per se} to gain some insights on
the distributions therein, and also to understand the most comment metric (BLEU) that is used,
and the implications of using this metric. However, for completeness, we list
the different approaches, and provide just a very brief overview of each technical
 approach.
All the datasets used below are described above in~\cref{sec:thedata}.

Iyer \emph{et al}~\cite{iyer2016summarizing} was an early attempt at this task, using a fairly standard seq2seq RNN model, enhanced with attention. Hu \emph{et al}~\cite{hu2018deep} also
used a similar RNN-based seq2seq model, but introduced a  ``tree-like" preprocessing of the input
source code. Rather than simply streaming in the raw tokens, they first parse it, and then serialize the resulting AST into a token stream
that is fed into the seq2seq model. A related approach~\cite{alon2018code2seq} digests a fixed-size random sample of paths through the AST of the input code (using LSTMs) and produces code
summaries. LeClair \emph{et al}~\cite{leclair2019neural} proposed an approach that combines
both structural and sequential representations of code; they have
also suggested the use of graph neural networks~\cite{alex2020improved}.  Wan \emph{et al}~\cite{wan2018improving} use a similar approach, but advocate using reinforcement learning to enhance the generation element. More recently, the use of function context~\cite{haque2020improved} has been reported to improve comment synthesis. Source-code vocabulary
proliferation is a well-known problem~\cite{karampatsis2020big}; previously unseen identifier or method names
in input code or output comments can diminish performance. New work by Moore \emph{et al}~\cite{mooreConvComment} approaches
this problem by using convolutions over individual \emph{letters} in the input and using
subtokens (by camel-case splitting) on the output. Very recently~\citet{Zhang2020RetrievalbasedNS} have
reported that combining sophisticated IR methods with deep-learning leads to further gains in the CCT task. For our purposes (showing that IR methods constitute a reasonable baseline) we use a very simple, vanilla, out-of-box Lucene
IR implementation, which already achieves nearly SOTA performance in many cases.

There are tasks related to generating comments from code: for example, synthesizing a commit
log given a code change~\cite{cortes2014automatically,liu2018neural,jiang2017automatically}, or generating
method names from the code~\cite{allamanis2016convolutional,alon2018code2seq}. Since these
are somewhat different tasks, with different data characteristics, we don't discuss them further.
In addition
code synthesis~\cite{yin2018learning,agashe2019juice} also uses matched pairs of natural language
and code; however, these datasets have not been used for generating English from code, and
are not used in prior work for this task; so we don't discuss them further here.

\subsection{Information Retrieval Baselines}
As can be see in ~\cref{fig:bivariate} and ~\cref{tab:ir-results}, datasets for the natural
language translation task show a smoother and more monotonic  input-output dependence;
by contrast, code-comment  datasets seem to have little or no input-output dependence.
This finding casts some doubt on the existence of a general sequence-to-sequence
$code \rightarrow comment$ function that
can be learned using a universal function approximator like a deep neural network. However it leaves
open the possibility that a more data-driven approach, that simply memorizes the training
data in some fashion, rather than trying to generalize from it, might also work. Thus, given
a code input, perhaps we can just try to find similar code in the training dataset, and
retrieve the comment associated with the similar code. This is a simple and naive information-retrieval
(IR) approach. We then compare this to the IR performance on NL translation.

\subsubsection{Method}
We use Apache Solr Version 8.3.1\footnote{https://lucene.apache.org/solr/}
to implement a straightforward IR approach. Apache Solr is a open source document search engine based on Apache Lucene. We simply construct an index of over the code parts
of the relevant datasets; given a code input, we use that as a ``query" over the index, find
the closest match, and return the comment associated with the closest matching code as
the ``generated comment".

We used the default parameters of Solr without tuning. This includes the default BM25 scoring function \cite{robertson1994some}.
For each dataset, we use always the same tokenization procedure used by authors. In addition,
we perform some additional pre-processing on the code, that is typically required
for IR approaches. For example, we remove highly frequent stop words from the code. Additionally, for datasets do not provide a tokenization phase
that actually splits  cammelCaseWords or snake\_case\_words, we include terms for indexing and searching which includes the split form of these words.
However, we note that the processing of stop words and word splitting only effects a minor change in performance.

\begin{table}[h]
    \centering
    \begin{tabular}{lllll}
        \toprule

        Dataset & Method Used & Score & Score Method \\
        \toprule

        DeepCom2f & IR-Baseline & 32.7 & BLEU-DC \\

        - & DeepCom (SBT) \cite{hu2018deep} & 38.2 & - \\
        - & Seq2Seq \cite{seq2seq} & 34.9 \cite{hu2018deep} & - \\

        \midrule

        DeepCom1 & IR-Baseline  & {\bf 45.6} & BLEU-ncs \\
        - & Transformer \cite{vaswani2017attention, ahmad2020transformerbased} & 44.6 & - \\

        \midrule
        FunCom1 & IR-Baseline  & 18.1 & BLEU-FC \\
        - & astattend-gru \cite{leclair2019neural} & 19.6 & - \\
        \midrule
        FunCom2 & IR-Baseline  & 18.2 & BLEU-FC \\
        - & astattend-gru \cite{leclair2019neural} & 18.7 \cite{alex2020improved} & - \\
        - & code2seq \cite{code2seq} & 18.8 \cite{alex2020improved}& - \\

        - & code+gnn+BiLSTM\cite{alex2020improved}& 19.9 & - \\

        \midrule
        CodeNN & IR-Baseline & \emph{7.6} & BLEU-CN \\
        - & IR Iyer \textit{et al}. & 13.7 \cite{iyer2016summarizing} & - \\
        - & CodeNN \cite{iyer2016summarizing} & 20.4 & - \\

        \midrule
        DocString2 & IR-Baseline  & {\bf 32.7} & BLEU-ncs \\
        - & Transformer \cite{vaswani2017attention, ahmad2020transformerbased} & 32.5 \cite{ahmad2020transformerbased} & - \\
        \midrule
        DocString1 & IR-Baseline & {\bf 27.6} & BLEU-Moses \\
        - & Seq2Seq & 14.0 \cite{baroneDocstring}  & - \\

        \midrule
        NL de-en & IR-Baseline  & \emph{2.2} & SacreBLEU \\
        - & FAIR Transformer\cite{ng2019facebook} & 42.7\footnotemark{} & - \\
        \bottomrule
    \end{tabular}

    \caption{\label{tab:ir-results} Measurements of a simple information retrieval
    baseline compared to various neural machine translation based methods. The scoring method we use
    mirrors the one used on the dataset (see Section \ref{sec:evalused}).}

\end{table}
\footnotetext{Value is for 2019 Model but on the 2018 test split}
\subsubsection{IR Results}
\label{sec:IRresults}
We find that on most datasets the simple IR baseline approaches the neural models,
and exceeds it for DeepCom1, DocString1, and DocString2.
However, IR does poorly on the WMT translation dataset, and also
on CodeNN. In both cases, we speculate that this may reflect the relative level of redundancy
in these datasets. CodeNN is drawn from StackOverflow, which tends to have fewer duplicated questions;
in the case of WMT, which is hand-curated, we expect there would be fewer duplications.

Prior work~\cite{10.1145/3238147.3240471,Zhang2020RetrievalbasedNS} has used very sophisticated
IR methods. We cannot claim to supersede these contributions; but will point out that a very naive IR method
does quite well, in some cases better than very recently published
methods on datasets/dataset-variations which currently lack IR baselines. We therefore view IR baselines as important calibration on model performance; by trying such a simple baseline first one can help find pathologies in the model or dataset which require further exploration.

We also note that there is variation results. In DeepCom2f, which includes 10 cross-project folds, we observe a wide range results ranging from a BLEU-DC of {\bf 20.6} to {\bf 48.4}! This level
of variation across folds is a cause for concern...this suggests depending on the split, a model with higher capacity to memorize the training data might do better or worse, potentially muddling the results if only doing one split. Similarly we notice that between different versions of FunCom scores vary quite a bit; this variation may
confound measurement of actual differences due to technical improvements.
\begin{minipage}{\columnwidth}
\begin{framed}
{\bf Recommendation: }{Since even naive IR methods provide competitive performance in many CCT datasets, they can be an important part for checking for issues in the new collection and new processing of CCT datasets. }
\end{framed}
\end{minipage}

\section{Introduction}
Programmers add comments to code to help  comprehension. The value of these comments is well understood and accepted.
A wide variety of comments exist~\cite{padioleau2009listening} in code, including prefix comments (standardized in frameworks
like Javadocs~\cite{kramer1999api}) which are inserted before functions or methods or modules, to describe their function. Given the value of comments, and the effort required to write them,
there has been considerable interest in providing automated
assistance to help developers to produce comments, and
a variety of approaches have been proposed~\cite{wong2015clocom,mcburney2014automatic,sridhara2010towards,rodeghero2014improving}.

\footnote{* Authors contributed equally}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.8\columnwidth]{zipf_plots/zipf_trigrams.png}
    \caption{Distribution of trigrams in English (blue) in the WMT~\cite{bojar2014findings}
    German-English machine-translation dataset, and in English  comments from several previously published
    Code-Comment datasets}
    \label{fig:firstzipf}
\end{figure}
Comments (especially prefix comments) are typically expected to be a useful summary of the function of the accompanying code. Comments
could be viewed as a restatement of the semantics of the code, in a different and more accessible natural language; thus, it is possible to view comment
generation as a kind of translation task, translating from one
(programming) language to a another (natural) language.
This view, together with the very large volumes of code
(with accompanying comments) available in open-source projects,
offers the very appealing possibility of leveraging decades
of research in statistical  natural language translation (NLT). If it's possible
to learn to translate from one language to another from data, why not learn
to synthesize comments from code?
Several recent papers~\cite{hu2018deep,ying2013code, leclair2019neural,iyer2016summarizing} have explored the idea
of applying Statistical Machine Translation (SMT)
methods to \emph{learn} to translate code to an English comments. But are these tasks really similar? We are interested to understand
in more detail how similar the task of generating comments from code is to the task of translating between natural languages.

Comments form a domain-specific dialect, which is highly structured, with a lot of very
repetitive templates. Comments often begin with patterns like \emph{"returns the"}, \emph{"outputs the"}, and \emph{"calculates the"}. Indeed, most of the earlier work (which wasn't based on
machine learning)
on this problem has
leveraged this highly templated nature of comments~\cite{sridhara2010towards,moreno2013automatic}.
We can see this phenomenon clearly using Zipf plots. Figure~\ref{fig:firstzipf}  compares the trigram frequencies
of the English language text in comments
(from the datasets~\cite{hu2018deep,leclair2019neural,iyer2016summarizing} that have been used to train deep-learning models for code-comment summarization)
and English language text in the WMT German-English translation
dataset: the x-axis orders the trigrams from most to least frequent using a log-rank scale,
and the y-axis is the log relative frequency of the trigrams in the corpus. The English
found in WMT dataset is the magenta line at the bottom. The comments
from code show consistently
higher slope in the (note, \emph{log-scaled}) y-axis of the Zipf plot, suggesting
that comments are far more saturated with repeating trigrams than is the English
found in the translation datasets. This observation motivates a closer examination
of the differences between code-comment and WMT datasets, and the implications
of using machine translation approaches for code-comment generation.

In this paper, we compare code-comment translation (CCT) datasets used with DL models for the task of comment generation, with  a popular natural translation (WMT) dataset used for training DL models for natural language translation. These were our results:
\begin{enumerate}
    \item We find that the desired outputs for the CCT task are much more repetitive.
    \item We find that the repetitiveness has a very strong effect on measured performance, much more
    so in the CCT datasets than the WMT dataset.
    \item We find that the WMT translation dataset has a smoother, more robust input-output dependency. Similar German inputs in WMT have a strong tendency to produce similar English outputs. However, this does appear to hold in the CCT datasets.
    \item We report that a naive  Information retrieval approach can meet or exceed reported numbers from neural models.
    \item We evaluate BLEU \emph{per se} as a measure of generated comment quality using
    groups of methods of varying "affinity"; this offers new perspectives on the BLEU  measure.
\end{enumerate}
Our findings have several implications for the future work in the area, in terms
of technical approaches, ways of measurement, for baselining, and for calibrating
BLEU scores. We begin below by first providing some background; we then describe
the datasets used in prior work. We then present an analysis of the datasets and
and an analysis of the evaluation metrics and baselines used. We conclude
after a detailed discussion of the implications of this work.

\vspace{0.013in}

\noindent\underline{\emph{But first, a disclaimer:}} this work does not offer any new models for or improvements on prior
results on the CCT task.
It is primarily retrospective, \emph{viz}, a critical
review of materials \& evaluations used in prior work in CCT, offered in a collegial spirit, hoping to advance
the way our community views the task of code-comment translation, and
how we might together make further advances in the measurement and evaluation
of innovations that are addressed in this task.

\input{background.tex}

\input{dataset_analysis.tex}

\input{MethodsFindings.tex}

\input{ir_discuss.tex}
\subsection{Calibrating BLEU Scores}

We now return our last research question, RQ~\ref{rq:calib}. How should we interpret the BLEU results reported
in prior work, and also the information retrieval BLEU numbers that we found (which are in the same range, see~\cref{tab:all-IR})?

To calibrate these reported BLEU scores, we conducted an observational study, using \emph{affinity groups} (AGs) of methods
that model different levels of expected similarity between the methods. For example, consider a random pair of methods,
so that both elements of the pair are \emph{methods from a different project}. This is our lowest-affinity group; we would expect
the comments to have very little in common, apart from both being utterances that describe code. The next higher
affinity group is a random pair of \emph{methods from the same project}. We would expect these to be a bit more
similar, since they are both concerned with the same application domain or function. The next higher level would
\emph{methods in the same class} which presumably are closer, although they would be describing different functions.
By taking a large number random pairs from each of these affinity groups, and measuring the BLEU for
pairs in each group,  we can get an estimate of BLEU for each group. For our experiment, we picked the
1000 largest projects from Github, and then chose 5000 random pairs from each of
the affinity groups. For each pair, we randomly picked one as the ``reference" output, and the other as
the ``candidate'' output, and the BLEU-M2 score. We report the results in two different
ways, in ~\cref{fig:violin} and in ~\cref{tab:diffBLEU}. For intraclass, we do not take more than six random pairs from a single class. In all AGs, we
removed all but one of the overloaded methods, and all getters and setters before our analysis. Without this filtering we see a difference of around 1-3  points.

\begin{figure}[h]
\includegraphics[width=0.6\columnwidth]{other_plots/affinity_violin2.png}
\caption{The distribution of BLEU scores between affinity groups. Red lines represent the means (i.e. the Sentence BLEU), and the dashed lines represent
quartiles.}
\label{fig:violin}
\end{figure}
First we describe~\cref{fig:violin} which shows the distribution of the BLEU scores in the 3 AGs.
As might be expected, the inter-project AG shows a fairly low mean, around 3. The intra-project AG
is a few BLEU points higher. Most notably, the intraclass AG has a BLEU score around 22, which
is close to the best-in-class values reported in prior work for some (but not all) datasets.

Note with the implemented experiment we cannot exactly compare these numbers, as each dataset is drawn from a different distribution.
Most CCT datasets provide the data in the traditional translation format of (source, target) pairs, making it difficult
to recover the other affinity group pairs of a given dataset example. This is why created our own new sampling based off of 1000 large Github repos. While not exactly comparable to existing datasets, new creations of CCT data could start with this simple
affinity group baseline to calibrate the reported results.

Another, stronger, affinity grouping would be methods that are semantically equivalent.
Rather than trying to identify such an affinity group ourselves by hand (which might
be subject to confirmation bias) we selected matched API calls from a recent
project, SimilarAPI~\footnote{\url{http://similarapi.appspot.com/}} by \citet{chensimilarapi} which used machine learning methods to match methods in different, but equivalent
APIs (e.g., \verb+Junit+ vs. \verb+testNG+). We extracted the descriptions of 40 matched pairs of high scoring matches from
different APIs and computed their BLEU.
We found that these BLEU scores
are on average about 8 points higher, with a mean around 32.
This number should be taken with caution, however, since
comments in this AG sample are substantially shorter than in the other groups.

\begin{framed}
{\bf Recommendation: }{Testing affinity groups can provide a baseline for calibrating BLEU results on a CCT dataset, as the simple trick of generating comments for a given method simply by retrieving the comments of a random other method in the same class possibly can approach SOTA techniques. }
\end{framed}

\noindent{\bf Postscript:} (\emph{BLEU Variability})
We noted earlier in~\Cref{sec:IRresults}, page~\pageref{sec:IRresults}, that there was
considerable intrinsic variation within a dataset, simply \emph{across different folds}; we reported that measured
BLEU-DC in DeepCom2f ranged from 20.6 to 48.4; similar results
were noted in the different variants of FunCom. This above affinity
group experiment with IR provided an opportunity to calibrate
another BLEU variability, \emph{across different
ways of calculating BLEU}.

\begin{table}[h]
\begin{tabular}{lr}
\toprule
 Function   &   intraclass \\
\midrule
 BLEU-FC    &        24.81 \\
 BLUE-CN    &        21.34 \\
 BLEU-DC    &        23.5  \\
 SacreBLEU  &        24.81 \\
 BLEU-Moses &        24.6  \\
 BLEU-ncs   &        21.49 \\
 BLEU-M2    &        21.22 \\
\bottomrule
\end{tabular}
\caption{The scores of samples of Java methods from the same class. }
\label{tab:diffBLEU}
\end{table}

We took the 5000-pair sample from the intraclass sample,
and measured the sentence BLEU for these pairs using the BLEU implementation variations used in the literature. The results are shown in~\cref{tab:diffBLEU}. The values range from around 21.2 to around 24.8; this range is actually rather high, compared to the gains reported
in recently published papers. This finding clarifies
the need to have a standardized measurement of performance.

\begin{framed}
{\bf Observation:} Measurements show substantial variation. The version of BLEU chosen, and sometimes even the folds in
the training/test split, can cause substantial variation in the measured performance, that may confound
the ability to claim clear advances over prior work.
\end{framed}

\begin{comment}

outline:

1) How are Code-Comm datasets different from translation datsets?   Univariate & bivariate
2) how is performance affected by frequently occurrring n-grams
3) How well does IR work
4) How is performance affected by duplicates, for IR & DL
\end{comment}

\section{Discussion}
We summarize our main findings and their implications.

\mypara{Comment Repetitiveness}
Our findings presented in Figure 1 show that comments in CCT datasets are far
more repetitive than the English found in the WMT dataset; figure 2 suggests
that this not merely a matter of greater vocabulary in distribution in comments,
but rather a function of how words are combined in comments. The highly-prevalent
patterns in comment have a substantially greater impact on the measured BLEU performance of
models trained with this CCT data, as shown in Figure 3. A closer look at the
CCT datasets shows that trigrams such as \litcode{creates a new}, \litcode{returns true if},
\litcode{constructor delegates to}, \litcode{factory method for} are very frequent. Getting
these right (or wrong) has a huge influence on performance.

\impara{Implications:} These findings suggest that getting just a few common patterns of comments right might
deceptively affect measured performance. So the actual performance of
comment generation might deviate a lot from measured values, much more so relative to natural language translation.
Repetition in comments might also mean that fill-in-the-blanks approaches~\cite{sridhara2011automatically}
might be revisited, with a more data-driven approach; classify code first, to find the right
template, and then fill-in-the-blanks, perhaps using an attention- or copy-mechanism.

\mypara{Input/Output Dependence}
When translating from one language to another, one would expect that more similar inputs
produce more similar outputs, and that this dependence is relatively smooth and monotonic. Our
findings in figure 4 and table 1, indicate that this property is indeed very strongly
true for general natural language outputs, but not as much for the comments.

\impara{Implications:} Deep-learning models are universal high-dimensional continuous function approximators.
Functions exhibiting a smooth input-output dependency, could be reasonably expected to be easier
to model. BLEU is a measure of lexical (token sequence) similarit); the rather non-functional nature
of the dependency suggested by figure 4 and table 1 indicate that token-sequence models that
work well for Natural language translation may be less performant for code; it may be that
other, non-sequential
models of code, such as tree-based or graph-based, are worth exploring further \cite{alex2020improved, 9031440}

\mypara{Baselining with IR}
Our experience suggests that simple IR approach provides BLEU performance
that is  comparable to current state of the art.

\impara{Implications} Our findings suggest that   a simple, standard, basic
IR approach would be a useful baseline for approaches to the CCT task. Especially
considering the range of different BLEU and tokenization approaches, this would be a
useful strawman baseline.

\mypara{Interpreting BLEU Scores}
BLEU, METEOR, ROGUE etc are measures that have been developed for different task
in natural language processing, such as translation \& summarization,
often after extensive, carefully designed, human subject studies.
Since BLEU is most commonly used in code-comment translation, we took an observational
approach calibrate the BLEU score. Our results, reported
in~\cref{fig:violin} and~\cref{tab:diffBLEU} indicate that the reported BLEU scores are not
that high.

\impara{Implications:} The best reported BLEU scores for the German-English translation tasks are currently are
in the low 40's. Our affinity group calibration suggests that on some datasets, the performance of models are comparable on average
to retrieving the comment of a random method from the same class. While this conclusion can't be explicitly drawn for a specific dataset without using the exact examples and processing from that specific dataset, but comparing results at an affinity group level can provide insight into minimum expected numbers for a new CCT datset.

\mypara{Learning From NLP Datasets}
We find that the current landscape of CCT datasets to be rather messy. There are often several different versions of the same dataset with different preprocessing, splits, and evaluation functions which all seem equivalent in name, but unless extra care is taken, might not be comparable.

However, some tasks in NLP do not seem to observe such variance within a task. We postulate this could be due to several reasons. For one, with the popularity of large open source repositories, it has become cheap and easy for a software engineering researcher to collect a large number of pairs of code and comments. This does not require hiring a human to label properties of text, and thus less effort might be taken on quality control compared to NLP data collection. Because researchers are domain experts in the datasets, they might be also more willing to apply their own version of preprocessing.

In addition, there are a wider array of tools to enforce consistency on various NLP tasks. For example the WMT conference on translation, a competition is ran with held out data and human evaluation. Other tasks, such as SQuAD\cite{DBLP:journals/corr/abs-1806-03822} for reading comprehension and GLUE\cite{DBLP:journals/corr/abs-1804-07461} for multitask evaluation allow for uploading code to a server which runs the proposed model on held out data. This ensure consistency in evaluation metrics and data.

We view adapting some these techniques as an interesting avenue for future work.

\section{Threats to Validity}
Our paper is a retrospective, and doesn't propose any new tools, metrics,\emph{etc}, still
some potential threats exist to our findings.

\impara{Fold Variance} With the exception of DeepCom2f we did not run measures over multiple folds or samples of the data. This makes it possible that there is variance in some of our reported numbers.

\impara{The Affinity Benchmarks} When collecting affinity groups, we collect full methods and process them using a set of filters. This means that when comparing these numbers, they might not be dirrectly comparable to a specific dataset. The numbers are presented only as estimate of similarity of the affinity groups.

\impara{Replication Threat} Whenever we had to , we did our best to replicate, and measure the quantities we reported using
the same code as the previous work. Still, it is possible that we failed to comprehend some subtleties in the provided code, and this
may be a threat to our findings.

\impara{Generalizability} We covered all the commonly used datasets and literature we could find. However, it may be that we have missed some where
cases our findings don't hold.

\section{Conclusion}
In this paper, we described a retrospective analysis of several research efforts
which used machine learning approaches, originally designed for the task of natural language
translation, for the task of generating comments from code. We examined the datasets,
the evaluation metrics, and the calibration thereof.
Our analysis pointed
out some key differences between general natural language corpora and comments: comments
are a lot more repetitive. We also found that a widely used natural language translation dataset shows a stronger,
smoother input-output relationships than natural language.
Turning then to a popular evaluation metric (BLEU score) we found considerable
variation based on the way it's calculated; in some cases this variation exceeded
claimed improvements. Looking at calibration of the reported BLEU scores,
first, we  found that simple
off-the-shelf information retrieval offers performance comparable to that reported previously.
Second, we found that the simple trick of retrieving a comment associated with a method in the same class
as a given method achieves an average performance comparable to current state-of-the-art. Our work suggests that future work in the area would benefit from a) other kinds
of translation models besides sequence-to-sequence encoder-decorder models b) more standardized
measurement of performance and c) baselining against Information Retrieval, and against some
very coarse foils (like retrieving a comment from a random other method in the same class).

Funding for this research was provided by National Science Foundation, under grant NSF 1414172,
\emph{SHF: Large: Collaborative Research: Exploiting the Naturalness of Software}.

Source code and data will be made available
at {\color{blue}{\url{https://bit.ly/3lBDegY}}}

\bibliographystyle{ACM-Reference-Format}

\section{Methods \&  Findings}
In the following section, we present our methods and results for each of the  RQs presented in \S~\ref{sec:bg}. In each case, we present some illustrative plots and (when applicable) the results of relevant statistical tests. All p-values have been corrected using family-wise (Benjamini-Hochberg) correction.
To examine the characteristics of each dataset, we constructed two types of plots: zipf plots and bivariate BLEU plots.
\subsection{Differences between CCT and WMT data}

\begin{figure}[h]

\includegraphics[width=0.74\columnwidth]{zipf_plots/zipf_unigrams.png}
\caption{Unigram (Vocabulary) distribution. Zipf plot for all datasets look similar. Difference from trigram Zipf
plot in Fig~\ref{fig:firstzipf} suggests greater repetitiveness in code comments.}
\label{fig:zipf1}
\end{figure}
The Zipf plots are a useful way to visualize
the skewness of textual data, where (in natural text) a few tokens (or ngrams) account for a large portion of the text. Each plot point is a (rank, relative-frequency) pair, both log-scaled. We use the plot to compare the relative skewness of the (English) comment data in the CCT data and the desired English outputs in the WMT NLT data. Examining the unigram
 Zipf plot above, it can be seen in both code comments and natural English, a few vocabulary words do  dominate. However, when we turn back to the trigram Zipf plots in Figure~\ref{fig:firstzipf}, we can see the
difference.
One is left with the clear
suggestion that: while the vocabulary distributions across the different datasets
aren't that different, the ways in which these vocabulary words
are \emph{combined into trigrams} are much more stylistic and templated in code comments.

\RSS{1}{Code comments are far more repetitive than the English found in Natural Language Translation datasets}

Given this relatively greater repetitive structure in code comments, we can expect that the performance
of translation tools will be strongly influenced by repeating (and/or very frequent) trigrams. If a few
frequent $n$-grams account for most of the desired output in a corpus, it would seem that these trigrams
would play a substantial, perhaps misleading role in measured performance.
\begin{figure*}[]

\includegraphics[width=1.25\columnwidth]{other_plots/side-by-side-do.pdf}
\caption{The effect of most frequent unigrams (left) and trigrams (right) on measured (smoothed) BLEU-4 performance. BLEU-4 was calculated after successive removals of of most frequent unigrams (right) and trigrams (left). The effect of removing frequent \emph{Unigrams} is by and large greater on the natural language dataset ("nl").
However, the effect of removing frequent \emph{trigrams}, on the comment datasets is generally stronger than on the "nl" dataset  due to high degree of repetition in the comment datasets. These apparent
visual differences are decisively confirmed by more rigorous statistical modeling.}

\label{fig:removal}
\end{figure*}
Figure~\ref{fig:removal} supports this analysis. The right-hand side plot shows the effect on BLEU-4
of replacing single words (unigrams) with
random tokens in the ``Golden" (desired) output in the various datasets. The left-hand plot shows
the effect of replacing trigrams. The index (1 to 100) on the x-axis shows the number
of most frequent n-grams replaced with random tokens. The y-axis shows the decrease in measured
BLEU-4 as the code is increasingly randomized.

The Unigrams plot suggests that the effect on the desired Natural language  ("nl") output,
as measured by BLEU is relatively greater when compared to most of the comment datasets.
This effect is reversed for Trigrams; the "nl" dataset is not affected as much by
the removal of frequent Trigrams as the comment datasets.
This analysis suggests that a tool that got the top few most frequent
trigrams wrong in the code-comment generation task would suffer a larger performance penalty than  a tool that
got the top-few $n$-grams wrong in a natural language  translation task. This visual evidence is strongly confirmed by rigorous statistical modeling, please see supplementary materials, bleu-cc.Rmd for the R code.
Frequent trigrams that have a big effect on the code comment BLEU include \emph{e.g.,} ``{\small\tt factory method for}'',
``{\small\tt delegates to the}'', and ``{\small\tt method for instantiating}''.
To put it another
way, one could boost the performance of a code-comment translation tool, perhaps misleadingly, by getting
a few such $n$-grams right.

\begin{comment}
To get a quantitative, statistical sense of the greater degree of repetition in the desired outputs in CCT datasets relative to NLT datasets, we measure the similarity between desired outputs using BLEU-1. BLEU-1 is a normalized measure of the number of overlapping unigrams (tokens), with 0 representing no similarity and 1 representing perfect overlap in tokens. BLEU-2 is like BLEU-1, but for bigrams, and so on.
To get a fair sample of similarities, we randomly select 100,000 pairs of outputs, from each dataset. For each pair, we compute the similarity using BLEU-1. Since the datasets aim to be free of duplicates,
generally we find a fair number of neglibible BLEU-1 values; for BLEU-2 and beyond most pairs show no similarity.
\end{comment}

\RSS{2}{Frequent n-grams could wield a much stronger effect on the measured BLEU performance on code-comment translation tasks than on natural language translation}

\begin{comment}

\subsection{Univariate Plots}

BLEU is a ubiquitous measure of the quality of a machine translation. In our setting, it is also used as measure of the similarity of two sentences. To investigate the similarity of code comments and natural language, we randomly sampled 100,000 pairs comments and natural language from each data set. We then calculated the sentence BLEUs for each of these pairs and plotted them if they exceeded a certain threshold (1e-5 in our case). We plotted natural language and comment BLEUs for BLEU1-4 on the same graph. As expected, most plots followed a normal uniform distribution's structure. We followed the same guidelines for reproducibility as in the zipf plots.

We then applied a Wilcoxon signed-rank test to every pair of BLEU scores from the comments data set and natural language data set. The Wilcoxon test is used to measure if the difference between the population means (or expected values) of two distributions are statistically different. In our setting, we seek to determine if the random pairwise BLEU scores of code comments and natural language are statistically different. A larger random BLEU for comments would indicate they are significantly more uniform and structured in nature. Finally, we also report the number of pairs in the 100,000 sample size that exceed the 1e-5 threshold.
\end{comment}

\subsection{Input-Output Similarity (RQ3)}

\begin{figure*}[]
    %
  \includegraphics[width=2.05\columnwidth]{other_plots/five_plots.png}

         \caption{Bivariate plots showing dependency of input-similarity to output-similarity. Pairs with similarity (BLEU-4) less than $10^{-5}$ are omitted. The natural language translation data have the strongest dependency: similar inputs have the strongest tendency to provide similar outputs}
\label{fig:bivariate}
\end{figure*}

An important property of natural language translation is that there is a general \emph{dependence of
input on output}. Thus, similar German sentences should translate to similar English sentences. For example
two German sentences with similar grammatical structure and vocabulary should in general result in two English
sentences whose grammatical structure and vocabulary resemble each other; likewise, in general, the more different two German sentences are in vocabulary and grammar, the more difference we expect in their English translations. Exceptions
are possible, since some similar constructions have different meanings:
(\emph{kicking the ball} vs. \emph{kicking the bucket}\footnote{The latter idiom indicates death;  Some automated translation engines (\emph{e.g.} Bing) seem to know the difference when translating from English}
). However, on average in large datasets, we should expect that more similar sentences give more similar translations.

When training a translation engine with a high-dimensional non-linear
function approximator like an encoder-decoder model using deep learning, this monotonic
dependence property is arguably useful. We would expect similar input sentences to be encode into similar
points in vector space, thus yielding more similar output sentences. How do natural language
translation (German-English) and code-comment datasets fare in this regard? To gauge this
phenomenon, we sampled 10,000 \emph{random pairs of input fragments} from each of
our datasets, and measured
their similarity using BLEU-M2, as well as the similarity of the corresponding Golden (desired)
output fragments. We then plot the \emph{input} BLEU-M2 similarity for each sampled pair
on the x-axis, and the
BLEU-M2 similarity of the corresponding pair of \emph{outputs} on the y-axis. We use a kernel-smoothed
2-d histogram
rather than a scatter plot, to make the frequencies more
visible, along with (we expect) an indication suggesting that similar inputs yield similar outputs.
Certainly, most inputs and outputs are different, so we expect to find a large number of highly dissimilar
pairs where
input and output pair BLEUs are virtually zero. So we considered our random samples, with and
without input and output similarities  bigger than
$\epsilon = 10^{-5}$. The highly dissimilar input-output pairs
are omitted in the plot. However, as an additional
quantitative test of correlation, we also considered Spearman's $\rho$.
both with and without the dissimilar pairs.

\begin{table}[h]
    \centering
    \begin{tabular}{lllc}
        \toprule

        Dataset & \emph{Spearman's} $\rho$ & Significance & \emph{Number of} \\

          &\emph{Correlation} &\emph{p-value} & \emph{pairs with}\\
                & $\rho$, $BLEU> \epsilon$  & p-value, $BLEU> \epsilon$  & BLEU $> \epsilon$ \\
                                & ($\rho$,all)  & (p-value, all)  &  \\
        \toprule
        NL & 0.70 (0.02) & 0.0 (0.055) & 2568 \\

        \midrule
        Deepcom1 & 0.056 (0.057) & 1.0e-6 (2.2e-8) & 7811 \\

        \midrule
        Deepcom2 & 0.036 (0.045) & 1.5e-3 (1.1e-5) & 7966 \\

        \midrule
        Docstring1 & 0.147 (0.16) & 6.7e-42 (4.6e-59) & 8585 \\
         \midrule
        Docstring2 & 0.041 (0.047) & 9.5e-5 (3.7e-6) & 9585 \\

        \midrule
        Funcom1 & 0.124 (0.083) & 1.30e-18 (3.4e-16) & 5026 \\

        \midrule
        Funcom2 & 0.122 (0.079) & 3.03e-18 (6.7e-15) & 5082 \\

        \midrule
        Codenn & 0.012 (0.0012) & 0.409 (0.904) & 2532 \\

        \bottomrule
    \end{tabular}
    \caption{Correlation values (Spearman's $\rho$, and significance, $p$-value,  for the plots  in Figure~\ref{fig:bivariate}. Values outside paranthesis are calculated with only the pairs having  pairwise BLEU $> 10^{-5}$; values in paranthesis include all pairs.
    $p$-values are adjusted with Benjamini-Hochberg familywise correction. In all cases, we chose 10,000 random
    pairs}
    \label{tab:all-IR}
\vspace{-0.35in}
\end{table}

The bivariate plots are shown in Fig~\ref{fig:bivariate} and the Spearmans $\rho$ in~\cref{tab:ir-results}.
We have split the 10,000 random samples
into two cases: one in which we do no further processing and one where both input/output BLEU similarities are both $> 10^{-5}$.
Spearman's $\rho$,
significance, and sample sizes are shown for non-zeroes in the columns (the numbers within
the parentheses include the highly dissimilar ones).
From the table (last column) we see that about 25-96\% of the pairs have some similarity on both inputs
and outputs, depending on the dataset.
The table also shows the Spearman's $\rho$ (first column) and significance (second column).
Each subplot in Fig~\ref{fig:bivariate}  shows one dataset, where
x-axis is the BLEU similarity of a pair's inputs, and y-axis is that of outputs. The plot is a binned 2D histogram,
using colored hexagons to represent counts in that bin. A representative variant of each dataset is plotted (as applicable); the omitted
ones are visually very similar.

We can  clearly see a stronger relationship between input and output BLEUs in the natural language setting. Particularly
for natural language data, this is further evidenced by the rather high Spearman correlation for
the non-zero BLEU pairs (\emph{0.70!!}), and the evident
visual dependence
between input-input similarity and output-output similarity is note worthy; this indicates
that there is strong, fairly monotonic relationship in natural language translation: the more similar the source,
the more similar the translation!

This analysis suggests that natural language data has a stronger, more specific input-output dependence; this
also suggests that translation between languages
is more amenable to learnable function-approximators like deep learners; this appears
to be \emph{substantially less true} for code-comment data.
This gives us the following conclusion with reference to RQ3.

\RSS{3}{
The natural language translation (WMT) shows a stronger input-output dependence than the CCT datasets in that
similar inputs are more likely to produce similar outputs.
}

\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigconf]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}

Software Engineering (cs.SE); Computation and Language (cs.CL)


Solar Chemical Peculiarities?

Abstract: Several investigations of FGK stars in the solar neighborhood have suggested that thin-disk stars with an iron abundance similar to the Sun appear to show higher abundances of other elements, such as silicon, titanium, or nickel. Offsets could arise if the samples contain stars with ages, mean galactocentric distances, or kinematics, that differ on average from the solar values. They could also arise due to systematic errors in the abundance determinations, if the samples contain stars that are different from the Sun regarding their atmospheric parameters. We re-examine this issue by studying a sample of 80 nearby stars with solar-like colors and luminosities. Among these solar "analogs", the objects with solar iron abundances exhibit solar abundances of carbon, silicon, calcium, titanium and nickel.

\section{Introduction}

Under the assumption that low-mass dwarf stars with convective envelopes
have a surface chemical composition which simply reflects that of their
natal clouds, one can use such stars to trace the chemical evolution of
the Galaxy. The Sun is then a convenient reference, but are there nearby
stars with solar composition? or in other words, is the solar abundance
pattern the norm in the local thin disk?

Inspection of some of the most recent synoptic studies of nearby stars
suggests that the Sun's metallicity is slightly off from average
(metallicity is here equated to the iron abundance
[Fe/H]
\footnote{[Fe/H]$\displaystyle = \log_{10} \frac{\rm N(Fe)}{\rm N(H)} +12$,
where N represents number density}).
Nordstr\"om et al. (2004) obtained metallicities from Str\"omgrem photometry
for nearly 14,000 F- and G-type stars within 70 pc, finding that their
distribution could be approximated by a Gaussian with a mean of
[Fe/H]$=-0.14$ and a $\sigma$ of 0.19 dex.
Allende Prieto et al. (2004) studied spectroscopically the stars more
luminous than $M_V=6.5$ ($M>0.76 M_{\odot}$) within 14.5 pc from the Sun
and concluded that their metallicity distribution is
centered at [Fe/H]$=-0.11$ and has a $\sigma$ of 0.18 dex.
Luck \& Heiter (2005) derived spectroscopic metallicities
for a sample of 114 FGK stars within 15 pc (similar to that analyzed by
Allende Prieto et al. 2004), finding a metallicity distribution with a
consistent width ($\sigma = 0.16$ dex), but centered at a value slightly
closer to solar ($-0.07$ for the complete sample, and $-0.04$
when thick-disk stars are excluded).
Haywood (2002) has argued that sample
selection based on spectral type discriminates against high-metallicity
stars, proposing a metallicity distribution
for the solar neighborhood (based on photometric indices)
that is centered at the solar value.

Inevitably, one must ask whether there is any reason to expect the
local metallicity distribution to be centered at the solar value.
Chemical differences among the Sun and its neighbors may be
reasonable if the age or the Galactic orbit of the Sun are
somewhat off from the average for nearby stars.
The age distribution or, equivalently, the star formation history of
the solar neighborhood is an unsolved problem, judging
from the discrepant results obtained from analyses of the Hipparcos
H-R diagram (Bertelli \& Nasi 2001, Vergely et al. 2002) and
studies of stellar activity (e.g. Rocha-Pinto et al. 2000).

What about abundance ratios? Should we expect the ratios
such as C/Fe to be fairly uniform at any given iron abundance?
Chemical uniformity requires the interstellar
medium to be extremely well mixed, but that is precisely what
local spectroscopic studies find. Reddy et al. (2003)
examined this issue based on high-dispersion spectra of a few hundred
stars and  were unable to detect any cosmic scatter.
The dispersion was as small as 0.03-0.04 dex for many elements, and could be
entirely accounted for considering the
uncertainties in the atmospheric parameters.
The immediate implication
is that the local interstellar medium is well mixed and has been well
mixed for many Ga. Such a conclusion is not contradicted by studies
of interstellar gas towards bright stars within and beyond the
local bubble (e.g. Oliveira et al. 2005).

In this situation it seems only
natural to expect the Sun to show similar abundance ratios as other
low-mass dwarfs in the solar vicinity with similar metallicity.
That is indeed the case for most elements, but there are some
striking offsets.
The landmark study by Edvardsson et al. (1993)
found  nearby FGK-type  stars with solar
iron abundance to be, on average, richer than the Sun
in Na, Al, and Si. Part of this trend, but not all, could
be linked to biases in other stellar parameters, such as mean
galactocentric distance and age.
More recent studies of nearby low-mass stars kept finding
offsets between the abundance ratios of stars with solar
iron abundances and the Sun. For example, Reddy et al. (2003)
found small offsets, in the same sense as Edvardsson et al.
for the ratios C/Fe, N/Fe, K/Fe, S/Fe, Al/Fe, and Si/Fe
(and perhaps Na/Fe), but opposite trends
for Mn/Fe and V/Fe. Allende Prieto et al. (2004) also found
similar patterns in their  sample for O/Fe, Si/Fe, Ca/Fe,
Sc/Fe, Ti/Fe,  Ni/Fe, and some neutron-capture elements (Na was not
studied).

The lack of consistency among different studies regarding the
existence and size of these chemical offsets is worrisome. Local samples
of stars span variable ranges in spectral type, which may be
associated with different systematic errors.
In order to explore further the nature of the observed offsets
we have observed a sample of solar analogs selected from the
{\it Hipparcos} color-magnitude diagram. We described the results
below.

\begin{figure}
\includegraphics[height=5.5in,angle=90]{callendeF1.ps}
\caption{
Stellar parameters for the sample. Two {\it metallicity} distributions
are shown: [M/H] indicates the values  derived from the analysis of
the spectral order that includes H$\beta$ (used to select the model
atmosphere), and [Fe/H] indicates the values subsequently derived from
the analysis of equivalent widths of Fe I lines. The surface gravities
shown here correspond to the spectroscopic values derived from the
H$\beta$ order, but the true gravities are likely tightly concentrated
around the solar value ($\log g \simeq 4.437$), given the narrow
distribution of the sample stars in $M_V$.
\label{f1}
}
\end{figure}

\section{Data and analysis}

To select solar analogs we used the Johnson  $M_V$ absolute magnitudes
and the $(B-V)$ color indices compiled by Allende Prieto
\& Lambert (1999) for 17,219 nearby stars ($d<100$pc)
included in the {\it Hipparcos} catalog. Stars were selected
to be within 0.07 mag from the adopted values for the Sun
$(B-V$,$M_V) = (0.65,4.85)$  and to be accessible to the 9.2m
Hobby-Eberly Telescope (HET) at McDonald
Observatory during the first observing period of 2005
(December 2004--March 2005), when the observations were obtained.
A list of 130 stars  were
placed on the HET queue, and 94 were spectroscopically observed.

The observations employed the High Resolution Spectrograph (HRS, Tull 1998),
a fiber-coupled spectrograph, using the first-order diffraction
grating g316 as cross disperser to get almost continuous
coverage between 407.6	 and 783.8 nm.
A fiber with a diameter of
2 arcsec fed the 0.625 arcsec wide slit of the spectrograph, providing
a FWHM resolving power of $R \sim 120,000$.  The data reduction
was made with an automated pipeline within IRAF, performing
bias correction, flat-fielding, scattered-light correction,
extraction, and wavelength calibration based on Th-Ar
hollow-cathode spectra.

The stellar effective temperatures, surface gravities, and overall
metallicity were derived by a $\chi^2$ fitting of the spectral order
containing H$\beta$ (Allende Prieto 2003). First, the procedure
was applied to the spectra of FG dwarfs included as part of the
Elodie library at a resolving power of $R\sim 10,000$, then the
residuals were fit by linear trends. After applying the linear corrections,
the rms scatter between our results and those in the Elodie catalog
are 1.5 \%, 0.16 dex and 0.07 dex for $T_{\rm eff}$, $\log g$, and
[Fe/H], respectively.

\begin{figure}
\includegraphics[height=3.5in,angle=0]{callendeF2a.ps}
\hfill
\includegraphics[height=3.5in,angle=0]{callendeF2b.ps}
\caption{
Abundances of Si and Ti, and their ratios to the iron abundances
for the sample stars. The broken lines mark the location of the
solar reference values.
\label{f2}
}
\end{figure}

The HRS spectra were processed in exactly the same manner, after
smoothing them to a resolution $R=10,000$, and the resulting parameters
were subjected to the linear corrections inferred from the comparison with
the Elodie  library.
Fig. \ref{f1} shows the distribution of the final atmospheric parameters.
Once the basic atmospheric parameters are constrained,
we measured and
made use of the equivalent widths of 46 Fe I lines to derive the appropriate
value of the microturbulence and the iron abundance using MOOG
(Sneden 2002). The iron linelist is
a subset of that described in Ram\'{\i}rez, Allende Prieto \& Lambert (2006),
and for other elements we used the same lines as Allende Prieto et al.
(2004). Abundances of C, Si, Ca, Ti, and Ni were also determined assuming LTE.

\section{Results and discussion}

Fig. \ref{f2} shows our results for silicon and titanium. The thin and
thick disk membership can be easily decided from these plots. The offset
from [Si/Fe]$=0$ at [Fe/H]$=0$ found in several previous surveys is
not apparent in the left-hand panels of Fig. \ref{f2}. Similarly, the
right-hand panels, do not confirm the offset found by Allende Prieto
et al. (2004) for titanium, and no significant
offsets were found either for carbon, calcium, titanium or nickel.
This result suggests that the offsets in previous analyses were
likely the result of systematic errors.

Samples of stars spanning a narrow range in atmospheric parameters
are well-suited to carry out differential studies  of chemical evolution.
With such samples, we can potentially minimize the impact of
shortcomings in the theory of stellar atmospheres and line formation
on the derived abundances, as the small scatter in the abundance ratios
shown in Fig. \ref{f2} suggest.

Solar Chemical Peculiarities?