Three errors and two problems in a recent paper: gazenet: End-to-end eye-movement event detection with deep neural networks (Zemblys, Niehorster, and Holmqvist, 2019)

Abstract: Zemblys et al. \cite{gazeNet} reported on a method for the classification of eye-movements ("gazeNet"). I have found 3 errors and two problems with that paper that are explained herein. \underline{\textit{\textbf{Error 1}}} The gazeNet classification method was built assuming that a hand-scored dataset from Lund University was all collected at 500 Hz, but in fact, six of the 34 recording files were actually collected at 200Hz. Of the six datasets that were used as the training set for the gazeNet algorithm, 2 were actually collected at 200Hz. \underline{\textit{\textbf{Problem 1}}} has to do with the fact that even among the 500Hz data, the inter-timestamp intervals varied widely. \underline{\textit{\textbf{Problem 2}}} is that there are many unusual discontinuities in the saccade trajectories from the Lund University dataset that make it a very poor choice for the construction of an automatic classification method. \underline{\textit{\textbf{Error 2}}} The gazeNet algorithm was trained on the Lund dataset, and then compared to other methods, not trained on this dataset, in terms of performance on this dataset. This is an inherently unfair comparison, and yet no where in the gazeNet paper is this unfairness mentioned. \underline{\textit{\textbf{Error 3}}} arises out of the novel event-related agreement analysis employed by the gazeNet authors. Although the authors intended to classify unmatched events as either false positives or false negatives, many are actually being classified as true negatives. True negatives are not errors, and any unmatched event misclassified as a true negative is actually driving kappa higher, whereas unmatched events should be driving kappa lower.

\section{Error Number 1}
\label{ErrorNumber1}
The gazeNet paper depends on the hand scoring of the eye-tracker data from Lund University (hereafter called the \textit{Lund2013}\footnote{Available for download at\\ http://www.humlab.lu.se/en/person/MarcusNystrom/}.)  The first error has to do with this dataset, on which the gazeNet algorithm was trained and developed.  Here is how the gazeNet authors describe the \textit{Lund2013}:
\\

\begin{adjustwidth}{1cm}{}
``It consists of monocular eye-movement data of participants viewing images, videos, and moving dots. The eye-movements of all participants were recorded with the SMI Hi-Speed 1250 eye-tracker, running at a sampling frequency of 500 Hz. Two domain experts then manually segmented data into fixations, saccades, post-saccadic oscillations (PSO), smooth pursuit, blinks and undefined events. A comprehensive description of the \textit{Lund2013} dataset and the coding process can be found in \cite{Larsson}.'' (page 3, left column, last paragraph)
\end{adjustwidth}

\noindent
\\\noindent
\\The 34 recordings where subjects were observing images were included in their analysis. Unfortunately, the gazeNet authors were unaware of the fact that in six\footnote{Higlighted in blue in Appendix 1}
of the 34 files  the median inter-timestamp intervals were 5000 microseconds, for a nominal sample rate of 200 Hz rather than the expected 500Hz.  Two of these 200 Hz datasets were included in the gazeNet training set (of a total of 6), one each was in the validation sets for rater RA and rater MN, and one each was in the test sets for rater RA and rater MN.  The gazeNet paper states that ``Trials were 4 or 10 s long.'' (column 2, page 3). This statement is false.  If one subtracts the first timestamp from the last timestamp, one will see that all of the files are almost exactly 10 seconds.  Since the gazeNet algorithm was trained and validated on improper data, it would seem to the present author that, at the least, the algorithm is not optimally trained. Since the gazeNet model was trained on improper data, there is basis for concern about the value of the reported gazeNet values in the gazeNet paper.

\section{Problem 1}
The gazeNet authors apply the MNH \cite{MNH} to the \textit{Lund2013}. The MNH \cite{MNH} was developed using the EyeLink 1000, at a 1000Hz sampling rate.  (I created the MNH.)  It is worth considering whether the application of the MNH to the \textit{Lund2013} is appropriate.  Most of the \textit{Lund2013} signals were sampled near 500 Hz, but as noted above, some of the data were sampled at 200 Hz.  Although the gazeNet authors correctly adjusted the sampling frequency parameter of the MNH to 500 Hz prior to application, they obviously did not adjust either the MNH or the data for the recordings at 200 Hz. Furthermore, even for studies sampled at 500Hz, there is significant variability in the inter-timestamp intervals in the \textit{Lund2013} (Figure 1).  According to technical staff at SR-Research, the inter-timestamp intervals for the EyeLink 1000 are all exactly 1 msec.  So, prior to applying the MNH to this data, this instability in the sampling rate would need to be addressed. \footnote{As the creator of the MNH, I cannot endorse the application of the MNH to data collected at 500 Hz.  I do not have extensive experience assessing its performance on such data. It was written for a reading task recorded with the EyeLink 1000 at 1000Hz.  However, I would suggest that those who intend to use it on such data anyway, interpolate their data to 1000 Hz using the “Piecewise Cubic Hermite Interpolating Polynomial” (PCHIP) method available in MATLAB.}

\begin{figure}[htbp]
\center
\frame{\includegraphics[width=0.4\textwidth]{Figures/Fig_01_Lund2013_ISI.pdf}}
\caption{Inter-Sample Intervals (ISI) for the \textit{Lund2013}.  This histogram reflects all the data from all the time stamps for the fourteen 500Hz (nominal) recordings that were coded by rater MN. At that sample rate, samples should be exactly 2000 microseconds apart. Note the bimodal nature of this histogram and the considerable variability around the nominal rate.}
\label{fig:1}
\end{figure}

\section{Problem 2}
There is a further problem with the \textit{Lund2013}.  A simple inspection of the recordings in the \textit{Lund2013} reveals many discontinuities in saccades. (See Figure 2 for examples.)  In a large number of samples during saccades (for this analysis only files scored by MN were used), there is a large gap in horizontal pixel position followed by a very short gap in pixel position. The blue dots in Figure 2 point to the sample after a large horizontal position move and followed by a small move.  To automatically detect these events, saccade samples were searched to find samples where the horizontal distance from the previous sample was greater than 3 times larger than the distance to the next sample.  Also, the distance to the next sample had to be less than 5.0 pixels.  Since such events can occur naturally at the end of saccades, the search was stopped 6 samples before the end of each saccade. There were 292 saccades in the entire dataset, and 109 (37\%) had a least 1 of these unusual events detected (see Figure 3).  The detection was not perfect, but the vast majority were indeed these discontinuities. Three saccades had 4 such events.  These discontinuities tend to occur more often in large saccades, and also at a time when eye-velocity is high.

\begin{figure*}[htbp]
\center
\frame{\includegraphics[width=1.0\textwidth]{Figures/FIG_02_WeirdEventsInSaccades.pdf}}
\caption{Eight examples, of 8 different saccades from 8 different files from the \textit{Lund2013}. File 1 = TH34-img-Europe-labelled-MN, File 2 = TH34-img-vy-labelled-MN, File 3 = TL20-img-konijntjes-labelled-MN, File 4 = TL28-img-konijntjes-labelled-MN, File 5 = UH27-img-vy-labelled-MN, File 6 = UH29-img-Europe-labelled-MN, File 7 = UH33-img-vy-labelled-MN, File 8 = UL43-img-Rome-labelled-MN.  The title for each plot contains the file number and the saccade number.  The hand scoring contained in the dataset was used to determine where saccades occur (shown in red). Non-Saccade data are shown in green, and the blue dots indicate the occurrence of the unusual discontinuities in this data (see text for explanation). These plots all employed the hand classification performed by rater``MN''. }
\label{fig:2}
\end{figure*}

\begin{figure}[htbp]
\includegraphics[width=0.5\textwidth]{Figures/WierdSaccadeHistogram.pdf}
\caption{Frequency histogram of number of discontinuous events per saccade.}
\label{WierdEventHistogram}
\end{figure}

These abnormalities are easily spotted when viewing the signals. I certainly think it is highly relevant for the reader to be informed about these unusual saccade artifacts, and their presence certainly should have been reported.  Given all of these issues with the Lund2013, I would not recommend its use, without correction, to develop and test a machine learning algorithm designed to classify eye-movements generally.

These kinds of discontinuities would obviously lead to an mis-estimation of saccade velocity, which the MNH very much depends on, as does the algorithm of \cite{Nystrom}.  It is not possible to accurately assess the classification accuracy of the MNH on the \textit{Lund2013} until this problem is solved, assuming it can be solved. \interfootnotelinepenalty=10000\footnote{There are classification issues with respect to the GazeCom and humanFixationEvaluation datasets that are also evaluated in the gazeNet paper.  Some of the issues with GazeCom are outlined in \cite{Shagen}, (see section 3.1.4, with 6 illustrations of  misclassified saccades).  This evaluation will be amplified in a future manuscript. For our analysis of the approach gazeNet applies to the humanFixationEvaluation dataset, see "GazeNetCritiqueAddendum.pdf" at  https://digital.library.txstate.edu/handle/10877/8649}

Although the gazeNet paper was developed using eye movement data that had several unusual properties, it is a general method that could, in theory, be applied to any similar eye-movement dataset.

\section{Error Number 2}
\label{ErrorNumber2}
On page 5 of the gazeNet paper, the authors state:

\begin{adjustwidth}{1cm}{}``...gazeNet was evaluated together with 4 other event detection algorithms on the Lund2013-image-test dataset, genSet and 2 other eye movement datasets.''
\end{adjustwidth}

\noindent On page 17, the authors state:

\begin{adjustwidth}{1cm}{}``The main exception is in the performance of MNH in detecting fixations: at the event-level it outperforms IRF (but not gazeNet) in the lund2013-image-test...''
\end{adjustwidth}

The data on this comparison is prominently displayed in Table 8.

This makes it clear that the MNH and other algorithms were compared to gazeNet on the Lund2013 dataset.

It should be noted that machine learning algorithms are trained to minimize classification error for the provided ground-truth labels. If a great many labeled saccades in the training set have the discontinuities illustrated above, then the network is expected to detect such unconventional events during inference.  As MNH was not designed to identify discontinuous saccades, there is no question that the gazeNet algorithm has an unfair advantage when classifying the Lund2013 dataset, where such discontinuous trajectories are labeled as standard saccades.  But the problem is even more extensive than this.  In addition to saccade discontinuities, the machine learning approach can “memorize” dataset specific patterns (saccade amplitude, frequency of saccades, general motion patterns, etc ...).  Any model comparison is unfair if only 1 algorithm was trained on the data used to compare methods and the other methods were not trained on this dataset.  Nowhere in the gazeNet paper is this inherent unfairness acknowledged.
\footnote{In the paper introducing the MNH \cite{MNH}, 3 other algorithms were compared to the MNH on EyeLink-1000 data collected at 1000 Hz. Two were developed or designed for such EyeLink data.  The third algorithm \cite{Nystrom} was developed on an SMI HiSpeed device. In this case, I received personal communication from Dr. Nyström indicating that that there should be no problem as long as the sampling rate parameter was modified.  Also, Dr. Holmqvist, who self-identified as one of the reviewers, raised no issues about the appropriateness of applying their algorithm to EyeLink data.}

\section{Error Number 3}
\label{ErrorNumber3}
The third error has to do with the calculation of the event-level agreement analysis in the gazeNet paper.\footnote{I received substantial initial assistance related to this error from a graduate student (Vladyslav Prokopenko). He is my colleague in Dr. Komogortsev's laboratory at Texas State University.   Before I got involved, he had looked carefully at the event-level analysis code and told me that he was confident that it did not do what the gazeNet authors intended, but he was not able to describe the exact nature of the error, nor was he able to point to the lines in the code which caused the error. Mr. Prokopenko also provided four Python code assignment statements that corrected the error in the original code.}  As part of that paper, the authors presented a new method to assess event-level agreement between 2 classifiers.\footnote{The Python code which performs the event-level agreement analysis for the gazeNet paper is available at https://github.com/r-zemblys/ETeval. } One classifier was referred to as “ground truth” (“gt”), and in the gazeNet paper, was a human rater. The other classifier was referred to as an algorithm which makes “predictions” (“pr”).  The final result was a Cohen's kappa for each comparison.

For ground truth (“gt”) I also used the \textit{Lund2013}, described above.  Specifically, the 34 recordings when subjects were looking at images are the focus of this report as was the case for the gazeNet paper. Like the gazeNet paper, I treated all of the \textit{Lund2013} as if it was collected at the nominal sampling rate of 500Hz.  Issues of data quality are not relevant to the point I am trying to make in this section.  The names of the original datasets and the names I used are listed in the Appendix.  Eye-movement data were classified into fixation, saccades, post-saccadic oscillations, smooth pursuit, blinks and undefined.

For my comparison, “predictive” algorithm, I employed the MNH \cite{MNH}.  As the creator of the MNH, it was easy to get the classifications.  I reiterate that data quality is not an issue for the point I want to make regarding event-level agreement. I am not trying to accurately estimate the classification accuracy of the MNH on the \textit{Lund2013}.  The MNH classifies data into fixations, saccades, post-saccadic oscillations, artifact, small RIONEPS noise \cite{MNH,RIONEPS} and large RIONEPS noise.  All sections of each dataset that were not labelled as fixation, saccade or PSO were removed from both the gt scored and the pr scored data.  Any dataset that contained less than 1000 remaining samples was not analyzed further.  This resulted in 28 datasets\footnote{Datasets are at https://doi.org/10.18738/T8/LKSTW4}.

The description of the algorithm in the gazeNet paper does not include pseudocode, nor does it precisely describe the actual steps in the analysis. Also, this description fails to introduce the concept of true negatives, which as we will see below, are critical to consider.  Therefore, I will take some time to introduce this method using my own figures and tables.  The input to the analysis is a two-column dataset, with one column listing the original classes (fixations, saccades and PSOs), on a sample by sample basis, from the gt method (\textit{Lund2013} hand scoring), and one column listing the original classes from the pr (MNH) algorithm.  Fixations were coded as 1, saccades as 2 and PSOs as 3.  Herein, I will be focused exclusively on obtaining kappas for fixation events, saccade events and PSO events.  For illustrative purposes,  I will be focused on the event-related kappa for PSO events for my 12th dataset (“Set\_12.csv”). To produce an evaluation of say, PSO events, all events that are not PSOs are combined into non-PSO events (Appendix II, Figure 1)

The next step was to match gt binary events to pr binary events.  This was done by determining the degree of overlap between events.  The amount of overlap between each gt event and each pr event was calculated.  Then each overlap was considered for matching from the longest to shortest and events were matched based on the overlap, if, and only if, both events were not already matched to other events.  A single pr event can only be matched to a single gt event.

The data format for input to the kappa function used in this code was two vectors, one representing the class (1 or 0) of the gt event, and the other representing the class (1 or 0) of the pr event.  Unmatched gt events and unmatched pr events are also given similar codes.  The final codes for the PSO evaluation for set number 12 are shown in Table 1.  Gt events that are matched to pr events are labelled as ``Matched''.  Although the codes for these matched events are typically either 1-1 or 0-0, this was not always the case.  In the gazeNet analysis, two events can be matched, but one could be a PSO and the other could be a “not-PSO”.  For example, event \# 14 in this list (Table 1) shows a match, where the gt event was a PSO and the pr event was a not-PSO.   Also, there are unmatched gt events and unmatched pr events.  Although the two vectors that are labelled in Table 1 as gt\_label and pr\_label are all that are required to compute kappa using the particular function employed in the original Python code, statisticians typically convert such information into a contingency table like that shown in Table 2. Once the data are in this form, Cohen's kappa can be easily calculated.

\renewcommand*{\thefootnote}{\fnsymbol{footnote}}
\begin{table*}[htbp]
\center
\caption{Codes for Events to be Input to the Kappa Function. PSO Analysis, Set\_12.csv}
\begin{minipage}{16cm}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
\hline
& & & & \textbf{Original} & \textbf{Corrected} & \textbf{Corrected} & \textbf{Correct}\\
\textbf{Count}  & \textbf{Status}  & \textbf{gt\_label} & \textbf{pr\_label} & \textbf{classification\footnote{TN = True Negative, FP = False Positive, FN = False Negative}} & \textbf{gt label} & \textbf{pr label} & \textbf{Classification} \\
\hline
1 & Matched & 0 & 0 & & & &  \\
\hline
2 & Matched & 1 & 1 & & & & \\
\hline
3 & Matched & 0 & 0 & & & &  \\
\hline
4 & Matched & 1 & 1 & & & &  \\
\hline
5 & Matched & 0 & 0 & & & &  \\
\hline
6 & Matched & 1 & 1 & & & &  \\
\hline
7 & Matched & 0 & 0 & & & &  \\
\hline
8 & Matched & 1 & 1 & & & &  \\
\hline
9 & Matched & 0 & 0 & & & &  \\
\hline
10 & Matched & 1 & 1 & & & & \\
\hline
11 & Matched & 0 & 0 & & & &  \\
\hline
12 & Matched & 1 & 1 & & & &  \\
\hline
13 & Matched & 0 & 0 & & & &  \\
\hline
14 & Matched & 1 & 0 & & & &  \\
\hline
15 & Matched & 0 & 0 & & & &  \\
\hline
16 & Matched & 1 & 1 & & & &  \\
\hline
17 & Matched & 0 & 0 & & & &  \\
\hline
18 & UnMatched\_GT & 0 & \cellcolor{blue!45}0 & \cellcolor{red!40}TN & 0  & 1  & FP  \\
\hline
19 & UnMatched\_GT & 1 & \cellcolor{blue!45}0 & FN & & & FN  \\
\hline
20 & UnMatched\_GT & 0 & \cellcolor{blue!45}0 & \cellcolor{red!40}TN & 0  & 1  & FP  \\
\hline
21 & UnMatched\_GT & 1 & \cellcolor{blue!45}0 & FN & & & FN  \\
\hline
22 & UnMatched\_GT & 1 & \cellcolor{blue!45}0 & FN & & & FN \\
\hline
23 & UnMatched\_GT & 0 & \cellcolor{blue!45}0 & \cellcolor{red!40}TN & 0  & 1  & FP \\
\hline
24 & UnMatched\_GT & 0 & \cellcolor{blue!45}0 & \cellcolor{red!40}TN & 0  & 1  & FP \\
\hline
25 & UnMatched\_GT & 1 & \cellcolor{blue!45}0 & FN & & & FN \\
\hline
26 & UnMatched\_PR & \cellcolor{green!45}0 & 1 & FP & & & FP  \\
\hline
27 & UnMatched\_PR & \cellcolor{green!45}0 & 0 & \cellcolor{red!40}TN & 1  & 0  & FN \\
\hline
28 & UnMatched\_PR & \cellcolor{green!45}0 & 1 & FP & & & FP  \\
\hline
29 & UnMatched\_PR & \cellcolor{green!45}0 & 1 & FP & & & FP \\
\hline
\end{tabular}
\end{minipage}
\end{table*}
\renewcommand*{\thefootnote}{\arabic{footnote}}

\begin{table}[htbp]
\center
\caption{Contingency Table}
\begin{tabular}{|c|c|c|}
\hline
& \textbf{PR=1} & \textbf{PR=0}\\
\hline
\textbf{GT=1} & TP(1,1) & FN(1,0) \\
\hline
\textbf{GT=0} & FP(0,1) & TN(0,0) \\
\hline
\end{tabular}
\end{table}

To calculate kappa for this binary form of the data, all events in each input file need to be classified as either True Positive (TP), True Negative (TN), False Positive (FP) or False Negative (FN).

The results of the matching process and code labelling is illustrated in Table 1 (see also Appendix II Figure 2).  In Table 1, note that some unmatched gt events are labelled 0-0.  In terms of the contingency table presented in Table 2, these would be interpreted as true negatives.  But all unmatched events must be considered as errors, and coded either 1-0 or 0-1.  True-negatives (0-0) are not errors.  I quote from the gazeNet paper:
\\
\begin{adjustwidth}{1cm}{}
``The remaining unmatched events are then labeled as false positives or false negatives, depending on whether they, respectively, occur in the ground truth or the algorithm event stream.'' (page 6, right column, 2nd line)\footnote{There are 4 potential types of mismatches, unmatched gt-true events, unmatched gt-false events, unmatched pr-true events and unmatched pr-false events.  Unfortunately, the gazeNet authors do not specify exactly how each of these should be coded.\\}
\end{adjustwidth}

\noindent
\\True negatives should only occur when non-PSO gt events are being correctly matched to non-PSO pr events. True negatives are an indication of good performance.  That unmatched events are being labelled as true negatives is the indication that the original code is not performing correctly.  If one looks at Table 1, one will see that all unmatched gt events are have pr codes of 0 (blue highlight).  You will also see that for all unmatched pr events, the corresponding gt code was 0 (green highlight).  Therefore, although all of the unmatched events are errors and should be coded as either 1-0 or 0-1 (false negatives or false positives), many of them (5 of 12 or 42\%) are actually coded as true negatives (red highlight).  Treating unmatched events as true negatives has the effect of increasing kappa toward 1.0, and therefore must be an error.
There are two lines in the code that are responsible for the error (see Table 3).

\begin{table*}[htbp]
\center
\caption{Python Assignment Statements}
\begin{tabular}{|l|c|}
\hline
\textbf{Original code snippets:}                            &  \\
\hline
evt\_pr = pd.concat((evt\_pr, pd.DataFrame(np.zeros(len(set\_gt)))))                            & (1) \\
\hline
evt\_gt = pd.concat((evt\_gt, pd.DataFrame(np.zeros(len(set\_pr)))))                            & (2) \\
\hline
\textbf{Corrected code snippets:}                                                                 &    \\
\hline
inverted\_gt = ($\sim$\_etdata\_gt.evt.loc{[}set\_gt, 'evt'{]}.values.astype(bool)).astype(int) & (3) \\
\hline
inverted\_pr = ($\sim$\_etdata\_pr.evt.loc{[}set\_pr, 'evt'{]}.values.astype(bool)).astype(int) & (4) \\
\hline
evt\_pr = pd.concat((evt\_pr, pd.DataFrame(inverted\_gt)))                                      & (5) \\
\hline
evt\_gt = pd.concat((evt\_gt, pd.DataFrame(inverted\_pr)))                                      & (6) \\
\hline
\end{tabular}
\end{table*}

Prior to the first line of code (1), evt\_pr is a list of the codes (0 or 1) for matched events. The variable “set\_gt” contains a list of all the unmatched gt events.  The part of assignment statement (1) that is:
\begin{center}$pd.DataFrame(np.zeros(len(set\_gt)))$\end{center}

\noindent
is appending a series of 0 codes to the evt\_pr list that is the same length as set\_gt.
Appending a list of 0 values is simply not correct.  I propose that what is needed is to append a list of codes that are opposite of the paired gt codes.  So, if an event is unmatched, it must be coded either 0-1 or 1-0.  I have prepared a corrected version of the event-level agreement Python code
\footnote{See
https://digital.library.txstate.edu/handle/10877/8649, where you can download ImprovedEventLevelKappa.zip.  This code reproduces the original erroneous kappas and the new, corrected kappas. It does not depend on the original ETeval code.  MATLAB code is also available to graphically illustrate the event-level agreement results, as in Appendix II, Figures 1 and 2.}.
In assignment statements (3) and (4) (Table 3), the codes that will be appended for unmatched events are created as inverted\_gt and inverted\_pr.  If the unmatched gt event has a code of 0, the pr code for the same event will be 1.  If the unmatched pr event has a code of 0, the gt code for the same event will be 1. The original code overestimates the number of true negatives and underestimates the number of false negatives and false positives.  Therefore, the original code produces kappa estimates that are too high.  In Fig. 4, I present distributions of incorrect Kappa\_gN (kappa\_gazeNet) values as well as the corrected Kappa\_Cor values for fixations, saccades and PSOs.

\begin{figure*}[htbp]
\center
\frame{\includegraphics[width=1.0\textwidth]{Figures/Fig_05_Violin_1.pdf}}
\caption{Violin plots of Cohen Kappa values before correction (“gN”, for gazeNet) and after correction (“Cor”, for correct). Mean values are shown in green.  The interquartile range is shown as a black vertical bar. For fixation, there was a decrease of 18.27\% after correction in mean kappa (t = 9.18, df = 19, p-value = 0.00000002, paired t-test, 2-tailed). For saccades, there was a decrease of 16.20\% after correction (t = 6.55, df = 22, p-value =0.000001).  For PSOs, there was a decrease of 33.78\% (t = 8.61, df = 26, p-value =0.000000004). Cases where kappa\_gN was equal to 1.0 were not included in this analysis (Fixation N = 8, Saccade N = 5, PSO N = 1), since there was nothing to correct in these cases. In three cases, all involving PSOs, the percent decrease was greater than 70\% (0.42 to 0.1, 0.35 to 0.08 and 0.42 to 0.03).  If cases in which kappa\_gN = 1.0 are included, the mean percent decrease across all events was 21.1\%, but in 8 cases the percent decrease was greater than 50\%.}
\label{fig:6}
\end{figure*}

The gazeNet authors make a strong case for using kappa over an F1-score for sample-level agreement statistics.  With regard to the event-level they state:\\

\begin{adjustwidth}{1cm}{}
``Note that when evaluating the detector’s performance at the event level instead of the sample level, the distribution of data across the classes is less unbalanced. Nevertheless, Cohen’s kappa is also a robust measure to use in this case, and will thus be used throughout the paper.''
\end{adjustwidth}
\noindent
\\There is an alternative way to describe the situation.  With binary classifications, every true event is followed by a false event. Thus, the distribution of classes is either perfectly balanced or only trivially unbalanced.  An F1-score would be appropriate. Other researchers use the F1-score to describe their event-level agreement results \cite{HoogeF1,Startsev}. There is no concept of a true negative in the context of the F1-score, and there is no ambiguity regarding the classification of false negatives (Misses) and false positives (False Alarms).  The gazeNet authors could have preferred one matching criteria over another, (e.g., “largest overlap” versus “overlapping test-events occurring earliest in time”), and still analyzed the performance using an F1-score.

Also, in the section in the gazeNet paper describing this event-level evaluation there were two additional errors.  The statement: \\

\begin{adjustwidth}{1cm}{}
``Hoppe and Bulling  \cite{HopBul} used a majority vote approach and calculated the F1-score.''\\
\end{adjustwidth}

\noindent
is false.  Hoppe and Bulling \cite{HopBul} do not express their event-level agreement results in terms of F1-scores.

Also, in this section, the gazeNet authors make the following statement:\\

\begin{adjustwidth}{1cm}{}
``Hooge et al. (2017) developed another event level F1-score and used it to compare human coders for assessing fixation coding performance. Hooge et al. looked for overlapping test-events occurring earliest in time and labeled them as hits. The remaining unmatched ground truth fixations were labeled as misses, while the remaining unmatched test fixations were labeled as false alarms. This works when there is a single event class, but when there is more than one event type to match, the Hooge et al. approach does not work well.'' (page 6, left column, near top)
\\
\end{adjustwidth}

\noindent
This notion does not make sense to me. The Hooge et al approach \cite{HoogeF1} is designed to compute an F1-score, and an F1-score is an average of precision and recall.  Precision and recall are only defined in the 2-class case.  They are not defined in the multi-class case.

\section{A Note on Nomenclature}
\label{Nomenclature} In the machine learning literature, including the gazeNet paper, criteria which emerge from human observation, intelligence, insight and intuition are pejoratively refered to as ``hand-crafted".  The hand has nothing to do with such criteria. Jewelry is handcrafted.  Bronze sculpture is handcrafted.  I propose that such criteria be properly referred to as criteria which emerge from human observation, intelligence, insight and intuition ($OI^3$).  Alternatively, criteria emerging from traditional approaches might be referred to as ``mindful'' whereas machine learning approaches might be referred to as ``mindless''.

    An earlier version of this manuscript is posted on arXiv \cite{arXiv}

\section{acknowledgements}
I would like to express my gratitude to Mr. Vladyslav Prokopenko for all of his assistance with this work.  Another graduate student colleague, Mr. Dmytro Katrychuk, made changes to my GitHub repository that facilitates setting up the working environment to use the code. He also significantly improved the Python script to increase its comprehensibility and to remove the dependency on the original ETeval code base.  I very much appreciate his assistance.

\section*{Conflict of interest}
I have no conflict of interest.


Collective Entity Disambiguation with Structured Gradient Tree Boosting

Abstract: We present a gradient-tree-boosting-based structured learning model for jointly disambiguating named entities in a document. Gradient tree boosting is a widely used machine learning algorithm that underlies many top-performing natural language processing systems. Surprisingly, most works limit the use of gradient tree boosting as a tool for regular classification or regression problems, despite the structured nature of language. To the best of our knowledge, our work is the first one that employs the structured gradient tree boosting (SGTB) algorithm for collective entity disambiguation. By defining global features over previous disambiguation decisions and jointly modeling them with local features, our system is able to produce globally optimized entity assignments for mentions in a document. Exact inference is prohibitively expensive for our globally normalized model. To solve this problem, we propose Bidirectional Beam Search with Gold path (BiBSG), an approximate inference algorithm that is a variant of the standard beam search algorithm. BiBSG makes use of global information from both past and future to perform better local search. Experiments on standard benchmark datasets show that SGTB significantly improves upon published results. Specifically, SGTB outperforms the previous state-of-the-art neural system by near 1\% absolute accuracy on the popular AIDA-CoNLL dataset.

\section{Acknowledgments}

We thank Prabhanjan Kambadur and other people in the Bloomberg AI team for their valuable comments on earlier version of this paper. We also thank the NAACL reviewers for their helpful feedback. This work also benefitted from discussions with Mark Dredze and Karl Stratos.

\section{Conclusion and future work}
\label{sec:con}

In this paper, we present a structured gradient tree boosting model for entity disambiguation. Entity coherence modeling is challenging, as exact inference is prohibitively expensive due to the pairwise entity relatedness terms in the objective function. We propose an approximate inference algorithm, BiBSG, that is designed specifically for SGTB to solve this problem. Experiments on benchmark ED datasets suggest that the expressive SGTB models are extremely good at dealing with the task of ED. SGTB significantly outperforms all previous systems on the AIDA-CoNLL dataset, and it also achieves SOTA results on many other ED datasets even in the cross-domain evaluation setting. SGTB is a family of structured learning algorithms that can be potentially applied to other core NLP tasks. In the future, we would like to investigate the effectiveness of SGTB on other information extraction tasks, such as relation extraction and coreference resolution.
\section{Experiments}
\label{sec:exp}

In this section, we evaluate SGTB on some of the most popular datasets for ED. After describing the experimental setup, we compare SGTB with previous state-of-the-art (SOTA) ED systems and present our main findings in~\autoref{sec:exp:results}.

\subsection{Data}

\begin{table} [t!]
\centering
\small
\begin{tabular}{llll}
    \toprule
    Dataset & \# mention & \# doc & \parbox{1.6cm}{\# mention \\ per doc}  \\ \midrule
    AIDA-train & 18,448 & 946 &19.5  \\
    AIDA-dev & 4,791 & 216 & 22.1 \\
    AIDA-test & 4,485 & 231 & 19.4 \\ \midrule
    AQUAINT & 727 & 50 & 14.5 \\
    MSNBC & 656 & 20 & 32.8 \\
    ACE & 257 & 36 & 7.1 \\ \midrule
    CWEB & 11,154 & 320 & 34.8 \\
    WIKI & 6,821 & 320 & 21.3 \\
    \bottomrule
\end{tabular}
\caption{Statistics of the ED datasets used in this work.}
\label{tab:data}
\end{table}

We use six publicly available datasets to validate the effectiveness of SGTB. AIDA-CoNLL~\cite{hoffart2011robust} is a widely adopted dataset for ED based on the CoNLL 2003 NER dataset~\cite{tjong2003introduction}. It is further split into training (AIDA-train), development (AIDA-dev), and test (AIDA-test) sets.\footnote{AIDA-dev and AIDA-test are also referred as AIDA-a and AIDA-b datasets in previous work.} AQUAINT~\cite{milne2008learning}, MSNBC~\cite{cucerzan2007large}, and ACE~\cite{ratinov2011local} are three datasets for Wikification, which also contain Wikipedia concepts beyond named entities. These datasets were recently cleaned and updated by~\newcite{guo2016robust}. WIKI and CWEB are automatically annotated datasets built from the ClueWeb and Wikipedia corpora by~\newcite{guo2016robust}. The statistics of these datasets are available in~\autoref{tab:data}.

\subsection{Experimental settings}

Following previous work~\cite{guo2016robust,ganea2017deep}, we evaluate our models on both \emph{in-domain} and \emph{cross-domain} testing settings. In particular, we train our models on AIDA-train set, tune hyperparameters on AIDA-dev set, and test on AIDA-test set (in-domain testing) and all other datasets (cross-domain testing). We follow prior work and report in-KB accuracies for AIDA-test and Bag-of-Title (BoT) F1 scores for the other test sets.

Two AIDA-CoNLL specific resources have been widely used in previous work. In order to have fair comparisons with these works, we also adopt them only for the AIDA datasets. First, we use a mention prior obtained from aliases to candidate entities released by~\newcite{hoffart2011robust} along with the two priors described in~\autoref{sec:imple:cand}. Second, we also experiment with PPRforNED, an entity candidate selection system released by~\newcite{pershina2015personalized}. It is unclear how candidates were pruned, but the entity candidates generated by this system have high recall and low ambiguity, and they contribute to some of the best results reported for AIDA-test~\cite{yamada2016joint,sil2018neural}.

\paragraph{Competitive systems}
We implement four competitive ED systems, and three of them are based on variants of our proposed SGTB algorithm.\footnote{Our implementations are based on the scikit-learn package~\cite{pedregosa2011scikit}.} \emph{Gradient tree boosting} is a local model that employs only local features to make independent decisions for every entity mention. Note that our local model is different from that presented by~\newcite{yamada2016joint}, where they treat ED as binary classification for each mention-entity pair. \emph{SGTB-BS} is a Structured Gradient Tree Boosting model trained with Beam Search with early update strategy. \emph{SGTB-BSG} uses Beam Search with Gold path training strategy presented in~\autoref{sec:model:bsg}. Finally, \emph{SGTB-BiBSG} exploits Bidirectional Beam Search with Gold path to leverage information from both past and future for better local search.

In addition, we compare against best published results on all the datasets. To ensure fair comparisons, we group results according to candidate selection system that different ED systems adopted.

\paragraph{Parameter tuning}

We tune all the hyperparameters on the AIDA-dev set. We use recommended hyperparameter values from scikit-learn to train regression trees, except for the maximum depth of the tree, which we choose from $\{3,5,8\}$. After a set of preliminary experiments, we select the beam size from $\{3,4,5,6\}$.  The best values for the two hyperparameters are $3$ and $4$ respectively. As mentioned in~\autoref{sec:model}, the learning rate is set to $1$. We train SGTB for at most $500$ epochs (i.e., fit at most $500$ regression trees). During training, we check the performance on the development set every $25$ epochs to perform early stopping.  Training takes $3$ hours for SGTB-BS and SGTB-BSG, and takes $9$ hours for SGTB-BiBSG on 16 threads.

\subsection{Results}
\label{sec:exp:results}

\input{results-tab}

\paragraph{In-domain results}
In-domain evaluation results are presented in~\autoref{tab:aida}. As shown, SGTB achieves much better performance than all previously published results. Specifically, SGTB-BiBSG outperforms the previous SOTA system~\cite{ganea2017deep} by $0.8\%$ accuracy, and improves upon the best published results when employing the PPRforNED candidate selection system by $1.9\%$ accuracy. Global information is clearly useful, as it helps to boost the performance by $2$-$4$ points of accuracy, depending on the candidate generation system. In terms of beam search training strategies, BiBSG consistently outperforms BSG and beam search with early update. By employing more point-wise functional gradients to train the regression trees and leveraging global information from both past and future to carry on local search, BiBSG is able to find better global solutions than alternative training strategies.

\paragraph{Cross-domain results}
As presented in~\autoref{tab:wned}, cross-domain experimental results are a little more mixed. SGTB-BS and SGTB-BSG perform quite competitively compared with SGTB-BiBSG. In a cross-domain evaluation setting, the test data is drawn from a different distribution as the training data. Therefore, less expressive models may be preferred as they may learn more abstract representations that will generalize better to out-of-domain data. Nevertheless, our SGTB models achieve better performance than best published results on three of the five popular ED datasets. Specifically, SGTB-BS outperforms the prior SOTA system by absolute $4\%$ F1 on the CWEB dataset, and SGTB-BiBSG performs consistently well across different datasets.

\section{Implementation}
\label{sec:imple}

We provide implementation details of our SGTB systems, including entity candidate generation, adopted local and global features, and some efforts to make training and inference faster.

\subsection{Candidate selection}
\label{sec:imple:cand}

We use a mention prior $\hat{p}(y | x)$ to select entity candidates for a mention $x$. Following~\newcite{ganea2017deep}, the prior is computed by averaging mention prior probabilities built from mention-entity hyperlink statistics from Wikipedia\footnote{We use a Wikipedia snapshot as of Feb. 2017.} and a large Web corpus~\cite{spitkovsky2012cross}. Given a mention, we select the top $30$ entity candidates according to $\hat{p}(y | x)$.

We also use a simple heuristic proposed by~\newcite{ganea2017deep} to improve candidate selection for persons: for a mention $x$, if there are mentions of persons that contain $x$ as a continuous subsequence of words, then we consider the candidate set obtained from the longest mention for the mention $x$.

\subsection{Features}
The feature function $\phi (\mathbf{x}, y_t, \mathbf{y_{1:t-1}})$ can be decomposed into the summation of a local feature function $\phi_L (\mathbf{x}, y_t)$ and a global feature function $\phi_G (y_t, \mathbf{y}_{1:t-1})$.

\paragraph{Local features}
We consider standard local features that have been used in prior work, including mention priors $p(y | x)$ obtained from different resources; entity popularity features based on Wikipedia page view count statistics;\footnote{We obtain the statistics of Feb. 2017 and Dec. 2011 from \url{https://dumps.wikimedia.org/other/pagecounts-ez/merged/} .} named entity recognition (NER) type features given by an in-house NER system trained on the CoNLL 2003 NER data~\cite{tjong2003introduction}; entity type features based on Freebase type information; and three textual similarity features proposed by~\newcite{yamada2016joint}.\footnote{We obtain embeddings jointly trained for words and entities from~\cite{ganea2017deep}.}

\iffalse
\begin{itemize}
    \item \textbf{Mention priors}: we use mention priors $p(e | m)$ obtained from different resources as separated features.
    \item \textbf{Entity popularity}: we adopt two entity popularity features based on Wikipedia page view count statistics.\footnote{We obtain the statistics of Feb. 2017 and Dec. 2011 from \url{https://dumps.wikimedia.org/other/pagecounts-ez/merged/} .}
    \item \textbf{NER types}: we include four named entity recognition (NER) type features given by an in-house NER system trained on the CoNLL 2003 NER data~\cite{tjong2003introduction}.
    \item \textbf{Entity types}: we map each Wikipedia page to one of the four NER types based on Freebase type information and Wikipedia to Freebase mappings.
    \item \textbf{Textual similarity}: we use three textual similarity features based on embeddings jointly trained for words and entities~\cite{ganea2017deep}. Following~\newcite{yamada2016joint}, we derive three textual vectors by averaging the embeddings of words of a mention $m$, of the sentence containing $m$, and of the document respectively. Then we consider three features that are cosine similarities between the textual vectors and the entity vector.
\end{itemize}
 \fi

\paragraph{Global features}
Three features are utilized to characterize entity-entity relationships: entity-entity co-occurrence counts obtained from Wikipedia, and two cosine similarity scores between entity vectors based on entity embeddings from~\cite{ganea2017deep} and Freebase entity embeddings released by Google\footnote{\url{https://code.google.com/archive/p/word2vec/}} respectively. We denote the entity-entity features between entities $y_t$ and $y_{t'}$  as $\phi_E (y_t, y_{t'})$.

At step $t$ of a training sequence, we quantify the coherence of $y_t$ with respect to previous decisions $\mathbf{y}_{1:y-1}$ by first extracting entity-entity features between $y_t$ and $y_{t'}$ where $1 \leq t' \leq t-1$, and then aggregating the information to have a global feature vector $\phi_G (y_t, \mathbf{y}_{1:t-1})$ of a fixed length:
\begin{align*}
\phi_G (y_t, \mathbf{y}_{1:t-1}) =& \sum_{t'=1}^{t-1} \frac{\phi_E (y_t, y_{t'})} {t-1} \\
&\oplus \max_{t'=1}^{t-1} \phi_E (y_t, y_{t'}),
\end{align*}
where $\oplus$ denotes concatenation of vectors.

\subsection{Efficiency}

Global models are powerful and effective, but often at a cost of efficiency. We discuss ways to speed up training and inference for SGTB models.

Many of the adopted features such as mention priors and entity-entity co-occurrences can be extracted once and retrieved later with just a hash map lookup. The most expensive features are the cosine similarity features based on word and entity embeddings. By normalizing the embeddings to have a unit norm, we can obtain the similarity features using dot products. We find this simple preprocessing makes feature extraction faster by two orders of magnitude.

SGTB training can be easily parallelized, as the computation of functional gradients are independent for different documents. During each training iteration, we randomly split training documents into different partitions, and then calculate the point-wise functional gradients for documents of different partitions in parallel.

\section{Introduction}
\label{sec:intro}

Entity disambiguation (ED) refers to the process of linking an entity mention in a document to its corresponding entity record in a reference knowledge base (e.g., Wikipedia or Freebase). As a core information extraction task, ED plays an important role in the language understanding pipeline, underlying a variety of downstream applications such as relation extraction~\cite{mintz2009distant,riedel2010modeling}, knowledge base population~\cite{ji2011knowledge,dredze2010entity}, and question answering~\cite{berant2013semantic,yih2015semantic}.
This task is challenging because of the inherent ambiguity between mentions and the referred entities. Consider, for example, the mention \example{Washington}, which can be linked to a city, a state, a person, an university, or a lake (\autoref{fig:sgb}).

Fortunately, simple and effective features have been proposed to capture the ambiguity that are designed to model the similarity between a mention (and its local context) and a candidate entity, as well as the relatedness between entities that co-occur in a single document. These are typically statistical features estimated from entity-linked corpora, and similarity features that are pre-computed using distance metrics such as cosine. For example, a key feature for ED is the \emph{prior probability} of an entity given a specific mention, which is estimated from mention-entity co-occurrence statistics. This simple feature alone can yield 70\% to 80\% accuracy on both news and Twitter texts~\cite{lazic2015plato,guo2013link}.

To capture the non-linear relationships between the low-dimensional dense features like statistical features, sophisticated machine learning models such as neural networks and gradient tree boosting are preferred over linear models. In particular, gradient tree boosting has been shown to be highly competitive for ED in recent work~\cite{yang2015smart,yamada2016joint}.
However, although achieving appealing results, existing gradient-tree-boosting-based ED systems typically operate on each individual mention, without attempting to jointly resolve entity mentions in a document together. Joint entity disambiguation has been shown to significantly boost performance when used in conjunction with other machine learning techniques~\cite{ratinov2011local,hoffart2011robust}. However, how to train a global gradient tree boosting model that produces coherent entity assignments for all the mentions in a document is still an open question.

In this work, we present, to the best of our knowledge, the first structured gradient tree boosting (SGTB) model for collective entity disambiguation. Building on the general SGTB framework introduced by~\newcite{yang2015smart}, we develop a globally normalized model for ED that employs a conditional random field (CRF) objective~\cite{lafferty2001conditional}. The model permits the utilization of global features defined between the current entity candidate and the entire decision history for previous entity assignments, which enables the global optimization for all the entity mentions in a document. As discussed in prior work~\cite{smith2007weighted,andor2016globally}, globally normalized models are more expressive than locally normalized models.

As in many other global models, our SGTB model suffers from the difficulty of computing the partition function (normalization term) for training and inference.  We adopt beam search to address this problem, in which we keep track of multiple hypotheses and sum over the paths in the beam. In particular, we propose Bidirectional Beam Search with Gold path (BiBSG) technique that is specifically designed for SGTB model training. Compared to standard beam search strategies, BiBSG reduces model variance and also enjoys the advantage in its ability to consider both past and future information when predicting an output.

Our contributions are:
\begin{itemize}
    \item We propose a SGTB model for collectively disambiguating entities in a document. By jointly modeling local decisions and global structure, SGTB is able to produce globally optimal entity assignments for all the mentions.
    \item We present BiBSG, an efficient algorithm for approximate bidirectional inference. The algorithm is tailored to SGTB models, which can reduce model variance by generating more point-wise functional gradients for estimating the auxiliary regression models.
    \item SGTB achieves state-of-the-art (SOTA) results on various popular ED datasets, and it outperforms the previous SOTA systems by $1$-$2\%$ absolute accuracy on the AIDA-CoNLL~\cite{hoffart2011robust}  dataset.

\end{itemize}

\section{Model}
\label{sec:model}

\begin{figure*}[t!]
\centering
\includegraphics[scale=.28]{./figs/sgb-figure.pdf}
\caption{(a) Example document $\mathbf{x}$ with entity candidates for each mention (gold entities are in \textbf{bold}); (b) the $m$-th SGTB update iteration: (i) conduct beam search to sample candidate entity sequences (\autoref{sec:train}), (ii) compute point-wise functional gradients for each candidate sequence, (iii) fit a regression tree to the negative functional gradient points with input features, $\phi$, (iv) update the factor scoring function, $F$, by adding the trained regression tree.}
\label{fig:sgb}
\end{figure*}

In this section, we present a SGTB model for collective entity disambiguation. We first formally define the task of ED, and then describe a structured learning formalization for producing globally coherent entity assignments for mentions in a document. Finally, we show how to optimize the model using functional gradient descent.

For an input document, assume that we are given all the mentions of named entities within it. Also assume that we are given a lexicon that maps each mention to a set of entity candidates in a given reference entity database (e.g., Wikipedia or Freebase).
The ED system maps each mention in the document to an entry in the entity database. Since a mention is often ambiguous on its own (i.e., the lexicon maps the mention to multiple entity candidates), the ED system needs to leverage two types of contextual information for disambiguation: local information based on the entity mention and its surrounding words, and global information that exploits the document-level coherence of the predicted entities. Note that modeling entity-entity coherence is very challenging, as the long-range dependencies between entities correspond to exponentially large search space.

We formalize this task as a structured learning problem. Let $\mathbf{x}$ be a document with $T$ target mentions, and $\mathbf{y} = \{ y_t \}_{t=1}^T$ be the entity assignments of the mentions in the document. We use $S(\mathbf{x}, \mathbf{y})$ to denote the joint scoring function between the input document and the output structure.  In traditional NLP tasks, such as part-of-speech tagging and named entity recognition, we often rely on low-order Markov assumptions to decompose the global scoring function into a summation of local functions. ED systems, however, are often required to model nonlocal phenomena, as any pair of entities is potentially interdependent. Therefore, we choose the following decomposition:
\begin{equation}
S (\mathbf{x}, \mathbf{y}) = \sum^T_{t=1} F (\mathbf{x}, y_t, \mathbf{y}_{1:t-1}),
\label{eq:joint-score}
\end{equation}
where $F (\mathbf{x}, y_t, \mathbf{y}_{1:t-1})$ is a factor scoring function. Specifically, a local prediction $y_t$ depends on all the \emph{previous decisions}, $\mathbf{y}_{1:t-1}$ in our model, which resembles recurrent neural network (RNN) models~\cite{elman1990finding,hochreiter1997long}.

We adopt a CRF loss objective, and define a distribution over possible output structures as follows:
\begin{equation}
    p (\mathbf{y} | \mathbf{x}) = \frac{\exp \{ \sum^T_{t=1} F (\mathbf{x}, y_t, \mathbf{y}_{1:t-1}) \}}{Z (\mathbf{x})},
    \label{eq:log-linear-model}
\end{equation}
where
\begin{equation*}
Z (\mathbf{x}) = \sum_{\mathbf{y}' \in Gen(\mathbf{x})} \exp \{ \sum^T_{t=1} F (\mathbf{x}, y'_t, \mathbf{y}'_{1:t-1}) \}
\end{equation*}
and $Gen(\mathbf{x})$ is the set of all possible sequences of entity assignments depending on the lexicon. $Z (\mathbf{x}) $ is then a global normalization term. As shown in previous work, globally normalized models are very expressive, and also avoid the label bias problem~\cite{lafferty2001conditional,andor2016globally}. The inference problem is to find
\begin{equation}
\argmax_{\mathbf{y} \in Gen(\mathbf{x})} p (\mathbf{y} | \mathbf{x}) = \argmax_{\mathbf{y} \in Gen(\mathbf{x})} \sum^T_{t=1} F (\mathbf{x}, y_t, \mathbf{y}_{1:t-1}).
\end{equation}

\subsection{Structured gradient tree boosting}

An overview of our SGTB model is shown in~\autoref{fig:sgb}. The model minimizes the negative log-likelihood of the data,
\begin{equation}
\begin{aligned}
L (\mathbf{y}^*, S (\mathbf{x}, \mathbf{y}))  &= - \log p (\mathbf{y}^* | \mathbf{x}) \\
                                                  &= \log Z(\mathbf{x}) - S(\mathbf{x}, \mathbf{y}^*),
\end{aligned}
\label{eq:loss}
\end{equation}
where $\mathbf{y}^*$ is the gold output structure.

In a standard CRF, the factor scoring function is typically assumed to have this form: $F (\mathbf{x}, y_t, \mathbf{y}_{1:t-1}) = \mathbf{\theta}^\top \phi (\mathbf{x}, y_t, \mathbf{y}_{1:t-1})$, where $\phi (\mathbf{x}, y_t, \mathbf{y}_{1:t-1})$ is the feature function and $\mathbf{\theta}$ are the model parameters.
The key idea of SGTB is that, instead of defining a parametric model and optimizing its parameters, we can directly optimize the factor scoring function $ F (\cdot)$ iteratively by performing gradient descent in function space. In particular, suppose $F (\cdot)=F_{m-1} (\cdot)$ in the $m$-th iteration, we will update $F (\cdot)$ as follows:
\begin{equation}
\begin{aligned}
F_m(\mathbf{x}, y_t, \mathbf{y}_{1:t-1}) &= F_{m-1}(\mathbf{x}, y_t, \mathbf{y}_{1:t-1}) \\
&- \eta_m g_m (\mathbf{x}, y_t, \mathbf{y}_{1:t-1}),
\end{aligned}
\end{equation}
where
\begin{equation}
\begin{aligned}
g_m (\mathbf{x}, y_t, &\mathbf{y}_{1:t-1}) = \frac{\partial L (\mathbf{y}^*, S (\mathbf{x}, \mathbf{y}))}{\partial F(\mathbf{x}, y_t, \mathbf{y}_{1:t-1}))} \\
&= p(\mathbf{y}_{1:t} | \mathbf{x}) - \mathbf{1}[\mathbf{y}_{1:t} = \mathbf{y}^*_{1:t}]
\end{aligned}
\label{eq:func-grad}
\end{equation}
is the functional gradient, $\eta_m$ is the learning rate, and $\mathbf{1} [\cdot]$ represents an indicator function, which returns $1$ if the predicted sequence matches the gold one, and $0$ otherwise. We initialize $F (\cdot)$ to $0$ ($F_0 (\cdot) = 0$).

We can approximate the negative functional gradient $-g_m(\cdot)$ with a regression tree model $h_m(\cdot)$ by fitting the training data $\{ \phi (\mathbf{x}^{(i)}, y_t^{(i)}, \mathbf{y}_{1:t-1}^{(i)}) \}$ to the point-wise negative functional gradients (also known as residuals) $\{ -g_m (\mathbf{x}^{(i)}, y_t^{(i)}, \mathbf{y}_{1:t-1}^{(i)}) \}$. Then the factor scoring function can be obtained by
\begin{equation}
F (\mathbf{x}, y_t, \mathbf{y}_{1:t-1}) = \sum_{m=1}^M \eta_m h_m (\mathbf{x}, y_t, \mathbf{y}_{1:t-1}) ,
\end{equation}
where $h_m (\mathbf{x}, y_t, \mathbf{y}_{1:t-1})$ is called a basis function. We set $\eta_m=1$ in this work.

\section{Related work}
\label{sec:related}

\paragraph{Entity disambiguation}

Most ED systems consist of a local component that models relatedness between a mention and a candidate entity, as well as a global component that produces coherent entity assignments for all mentions within a document. Recent research has largely focused on joint resolution of entities, which is usually performed by maximizing the global
topical coherence between entities. As discussed above, directly optimizing the coherence objective is computationally intractable, and several heuristics and approximations have been proposed to address the problem. \newcite{hoffart2011robust} use an iterative heuristic to remove unpromising mention-entity edges. \newcite{yamada2016joint} employ a two-stage approach, in which global information is incorporated in the second stage based on local decisions from the first stage. Approximate inference techniques have been widely adopted for ED. \newcite{cheng2013relational} use an integer linear program (ILP) solver. Belief propagation (BP) and its variant loopy belief propagation (LBP) have been used by~\newcite{ganea2016probabilistic} and~\newcite{ganea2017deep} respectively. We employ another standard approximate inference algorithm, beam search, in this work. To make beam search a better fit for SGTB training, we propose BiBSG that improves beam search training on stability and effectiveness.

\paragraph{Structured gradient tree boosting}

Gradient tree boosting has been used in some of the most accurate systems for a variety of classification and regression problems~\cite{babenko2011robust,wu2010adapting,yamada2016joint}. However, gradient tree boosting is seldom studied in the context of structured learning, with only a few exceptions. \newcite{dietterich2004training} propose TreeCRF that replaces the linear scoring function of a CRF with a scoring function given by a gradient tree boosting model. TreeCRF achieves comparable or better results than CRF on some linear chain structured prediction problems. \newcite{bagnell2007boosting} extend the Maximum Margin Planning (MMP; Ratliff et al., 2006\nocite{ratliff2006maximum}) algorithm to structured prediction problems by learning new features using gradient boosting machines. \newcite{yang2015smart} present a general SGTB framework that is flexible in the choice of loss functions and specific structures. They also apply SGTB to the task of tweet entity linking with a special non-overlapping structure. By decomposing the structures into local substructures, exact inference is tractable in all the aforementioned works. Our work shows that we can train SGTB models efficiently and effectively even with approximate inference. This extends the utility of SGTB models to a wider range of interesting structured prediction problems.

\section{Training}
\label{sec:train}

Training the SGTB model requires computing the point-wise functional gradients with respect to training documents and candidate entity sequences. This is challenging, due to the exponential output structure search space.  First, we are not able to enumerate all possible candidate entity sequences. Second, computing the conditional probabilities shown in~\autoref{eq:func-grad} is intractable, as it is prohibitively expensive to compute the partition function $Z(\mathbf{x})$ in~\autoref{eq:log-linear-model}. Beam search can be used to address these problems. We can compute point-wise functional gradients for candidate entity sequences in the beam, and approximately compute the partition function by summing over the elements in the beam.

In this section, we present a bidirectional beam search training algorithm that always keeps the gold sequence in the beam. The algorithm is tailored to SGTB, and improves standard training methods in two aspects: (1) it reduces model variance by collecting more point-wise function gradients to train a regression tree; (2) it leverages information from both past and future to conduct better local search.

\subsection{Beam search with gold path}
\label{sec:model:bsg}

The early update~\cite{collins2004incremental} and LaSO~\cite{daume2005learning, xu2007learning} strategies are widely adopted with beam search for updating model parameters in previous work. Both methods keep track of the location of the gold path in the beam while decoding a training sequence. A gradient update step will be taken if the gold path falls out of the beam at a specific time step $t$ or after the last step $T$. Adapting the strategies to SGTB training is straightforward. We will compute point-wise functional gradients for all candidate entity sequences after time step $T$ or when the gold sequence falls out the beam.
Both early update and LaSO are typically applied to online learning scenarios, in which model parameters are updated after passing one or a few training sequences.

SGTB training, however, fits the batch learning paradigm. In each training epoch, a SGTB model will be updated only once using the regression tree model fit on the point-wise negative functional gradients. The gradients are calculated with respect to the output sequences obtained from beam search.
We propose a simple training strategy that computes and collects point-wise functional gradients at every step of a training sequence. In addition, instead of passively monitoring the gold path, we always keep the gold path in the beam to ensure that we have valid functional gradients at each time step. The new beam search training method, Beam Search with Gold path (BSG), generates much more point-wise functional gradients than early update or LaSO, which can reduce the variance of the auxiliary regression tree model. As a result, SGTB trained with BSG consistently outperforms early update or LaSO in our exploratory experiments, and it also requires fewer training epochs to converge.\footnote{Early update and LaSO perform similarly, thus we only report results for early update in~\autoref{sec:exp}.}

\subsection{Bidirectional beam search}

During beam search, if we consider a decision made at time step $t$, the joint probability $p (\mathbf{y} | \mathbf{x})$ can be factorized around $t$ as follows:
\begin{equation}
\begin{aligned}
p (\mathbf{y} | \mathbf{x}) = p & (\mathbf{y}_{1:t-1} | \mathbf{x}) \cdot p(y_t | \mathbf{y}_{1:t-1}, \mathbf{x})  \\
                            \cdot & p(\mathbf{y}_{t+1:T} | y_t, \mathbf{y}_{1:t-1}, \mathbf{x}).
\end{aligned}
\label{eq:joint}
\end{equation}

Traditional beam search performs inference in a unidirectional (left-to-right) fashion. Since the beam search at time step $t$ considers only the beam sequences that were committed to so far, $\{\mathbf{y}_{1:t-1}\}$, it effectively approximates the above probability by assuming that all futures are equally likely, i.e. $p(\mathbf{y}_{t+1:T} | y_t, \mathbf{y}_{1:t-1}, \mathbf{x})$ is uniform. Therefore, at any given time, there is no information from the future when incorporating the global structure.

In this work, we adopt a Bidirectional Beam Search (BiBS) methodology that incorporates multiple beams to take future information into account~\cite{sun2017bidirectional}. It makes two simplifying assumptions that better approximate the joint probability above while remaining tractable: (1) future predictions are independent of past predictions given $y_t$; (2) $p(y_t)$ is uniform. These yield the following approximation:
\begin{equation}
\begin{aligned}
  p (\mathbf{y}_{t+1:T} | y_t, \mathbf{y}_{1:t-1}, \mathbf{x})
                    &= p (\mathbf{y}_{t+1:T} | y_t, \mathbf{x}) \\
                    \propto p(y_t  | \mathbf{y}_{t+1:T}, & \mathbf{x}) \cdot
                               p(\mathbf{y}_{t+1:T} |  \mathbf{x}).
\end{aligned}
\end{equation}
Substituting this back into~\autoref{eq:joint} therefore yields:
\begin{equation}
\begin{aligned}
p (\mathbf{y} | \mathbf{x})  \propto p & (\mathbf{y}_{1:t-1} | \mathbf{x}) \cdot p(y_t | \mathbf{y}_{1:t-1}, \mathbf{x}) \\
                             \cdot & p(y_t | \mathbf{y}_{t+1:t}, \mathbf{x}) \cdot p(\mathbf{y}_{t+1:T} | \mathbf{x}),
\end{aligned}
\label{eq:joint-appr}
\end{equation}
which decomposes into multiplication of a forward probability and a backward probability. In~\cite{sun2017bidirectional}, these are retrieved from forward and backward recurrent networks, whereas in our work we use the joint scores (log probabilities shown in~\autoref{eq:joint-score}) computed for partial sequences from forward and backward beams.

\begin{algorithm}[h]
  \small
  \setstretch{1.1}
  \SetKwInOut{Input}{Input}
  \SetKwInOut{Output}{Output}
  \Input{input document $\mathbf{x}$, candidate sequences $\{ \mathbf{y} \}$, \newline joint scoring function $S(\mathbf{x}, \mathbf{y}_{t_1:t_2})$}
  \Output{beam sequence set $C$}
 $C \leftarrow \emptyset$\\
 \While{not converged}{
  // \texttt{forward beam search}\\
  \For{$t = 1, \cdots, T$}{
   $C^{(F)} \leftarrow \text{top-B}_{\mathbf{y}_{1:t}} [S(\mathbf{x}, \mathbf{y}_{1:t}) + S(\mathbf{x}, \mathbf{y}_{T:t})]$\\
   // \texttt{add gold subsequence}\\
   $C^{(F)} \leftarrow C^{(F)} \cup \{ \mathbf{y}_{1:t}^* \}$\\
   $C \leftarrow C \cup C^{(F)}$
   }
   // \texttt{backward beam search}\\
   \For{$t = T, \cdots, 1$}{
   $C^{(B)} \leftarrow \text{top-B}_{\mathbf{y}_{T:t}} [S(\mathbf{x}, \mathbf{y}_{T:t}) + S(\mathbf{x}, \mathbf{y}_{1:t})]$\\
   // \texttt{add gold subsequence}\\
   $C^{(B)} \leftarrow C^{(B)} \cup \{ \mathbf{y}_{T:t}^* \}$\\
   $C \leftarrow C \cup C^{(B)}$
  }
 }
 \caption{Bidirectional Beam Search with Gold path (BiBSG)}
 \label{eq:bibsg}
\end{algorithm}

The full inference algorithm, Bidirectional Beam Search with Gold path (BiBSG), is presented in~\autoref{eq:bibsg}. When performing the forward pass to update the forward beam, forward joint scores, $S(\mathbf{x}, \mathbf{y}_{1:t})$, are computed with respect to current forward beam, and backward joint scores, $S(\mathbf{x}, \mathbf{y}_{T:t})$, are computed with respect to previous backward beam. A similar procedure is used for the backward pass. The search converges very fast, and we use two rounds of bidirectional search as a good approximation.
Finally, SGTB-BiBSG compares the conditional probabilities $p(\mathbf{y}^{\text{($\cdot$)}} | \mathbf{x})$ of the best scoring output sequences $\mathbf{y}^{\text{(F)}}$ and $\mathbf{y}^{\text{(B)}}$ obtained from the forward and backward beams. The final prediction is the sequence with the higher conditional probability score.


Herwig++ 2.2 Release Note

Abstract: A new release of the Monte Carlo program Herwig++ (version 2.2) is now available. This version includes a number of improvements including: matrix elements for the production of an electroweak gauge boson, W and Z, in association with a jet; several new processes for Higgs production in association with an electroweak gauge boson; and the matrix element correction for QCD radiation in Higgs production via gluon fusion.

\section{Introduction}

The last major public version (2.1) of \HWPP, is described in great detail
in \cite{Bahr:2008pv}. This release note therefore only lists the
changes which have been
made since the last release~(2.1). The manual has been updated to
reflect these changes and this release note is only intended to highlight these
new features and the other minor changes made since the last version.

Please refer to \cite{Bahr:2008pv} and the present paper if
using version 2.2 of the program.

The main new features of this version are the inclusion of matrix
elements for the production of an electroweak gauge boson,
$W^\pm$ and $Z^0$, in association with a jet in hadron-hadron collisions,
the addition of matrix
elements for the production of an electroweak gauge boson,
$W^\pm$ and $Z^0$, in association with the Higgs boson in both lepton-lepton
and hadron-hadron collisions, and the matrix element correction for the production
of QCD radiation in Higgs production via $gg\to h^0$.
In addition a number of other changes, such as the inclusion of
the option of a saturation model for the small-$x$ PDF and a restructuring of
the library structure, have been
made and a number of bugs have been fixed.

\subsection{Availability}
The new program, together  with other useful files and information,
can be obtained from the following web site:
\begin{quote}\tt
       \href{http://hepforge.cedar.ac.uk/herwig/}{http://hepforge.cedar.ac.uk/herwig/}
\end{quote}
  In order to improve our response to user queries, all problems and requests for
  user support should be reported via the bug tracker on our wiki. Requests for an
  account to submit tickets and modify the wiki should be sent to
  {\tt herwig@projects.hepforge.org}.

  \HWPP\ is released under the GNU General Public License (GPL) version 2 and
  the MCnet guidelines for the distribution and usage of event generator software
  in an academic setting, which are distributed together with the source, and can also
  be obtained from
\begin{quote}\tt
 \href{http://www.montecarlonet.org/index.php?p=Publications/Guidelines}{http://www.montecarlonet.org/index.php?p=Publications/Guidelines}
\end{quote}

\section{New Matrix Elements}

  A number of new matrix elements are included in this release:
\begin{itemize}
\item the \textsf{MEPP2WJet} and \textsf{MEPP2ZJet} classes for the simulation
      of $W^\pm$ and $Z^0$ production in association with a hard jet
	 in hadron-hadron collisions;
\item the \textsf{MEPP2WH} and \textsf{MEPP2ZH} classes for the simulation
      of $W^\pm$ and $Z^0$ production in association with a Higgs boson
	 in hadron-hadron collisions;
\item the \textsf{MEee2ZH} class for the production of a Higgs boson
	in association with a $Z^0$ boson in $e^+e^-$ collisions.
\end{itemize}

\section{Higgs Matrix Element Correction}

\begin{figure}[!!t]
\begin{center}
\includegraphics[width=0.49\textwidth,angle=90]{htoggA.ps}\hfill
\includegraphics[width=0.49\textwidth,angle=90]{htoggB.ps}
\end{center}
\caption{The transverse momentum of the Higgs boson at the LHC for a Higgs boson
	 with mass a) 115\,GeV and b) 300\,GeV. The results of the
         full correction~(solid), no correction~(dashed), only the soft matrix element
         correction to radiation generated by the parton shower~(dotted), and the
	 hard correction filling the `dead' region not filled by the parton shower~(dot-dashed) are shown.}
\label{fig:higgspt}
\end{figure}

  The matrix element correction for $gg\to h^0$ has been included using the same approach
  as for the Drell-Yan production of a $W^\pm$ or $Z^0$ boson. The $p_T$ spectra of
  Higgs bosons with masses of 115\,GeV and 300\,GeV are shown in Fig.\,\ref{fig:higgspt}.
  Only the $gg\to h^0g$, $qg\to h^0q$ and $\bar{q}g\to h^0\bar{q}$ partonic
  processes are included in the matrix element correction as these are
  the only processes which have a parton-shower interpretation. The
  $q\bar{q}\to h^0g$ process should therefore be included as an additional hard process
  with no cut on the $p_T$ of the outgoing gluon if all the partonic processes are
  required, in general this process gives a very small contribution.

\section{Other Changes}

A number of other more minor changes have been made.
The following changes have been made to improve the physics
simulation:
\begin{itemize}
\item A new class implementing a saturation model for the PDFs has been included.
\item A number of additional options for \textsf{HepMC} output have been added,
      including the units to be used for energies and distances.
\item The decays of the $\Sigma_b$ baryons have been changed so that only
      the decays $\Sigma_b\to\pi\Lambda_b$ are included.
\item A number of bugs in the hadronization have been fixed necessitating
      the retuning of the
      default hadronization parameters to LEP and B-factory data. The parameters
      for the underlying event have also been retuned.
\item The option of not building the BSM models has been added, by default
      all the models are built.
\item The particles for specific new physics models have now been moved to the files
      specifying the models.
\item The Higgs width is now automatically recalculated from its mass and
      the limits on the off-shell mass are taken to be proportional to the width.
\item The option of generating intrinsic $p_T$ according to an inverse quadratic
      distribution has been added.
\end{itemize}

The following bugs have been fixed:
\begin{itemize}
\item A major bug in the splitting function for $g\to gg$ branchings in the initial state
      which lead to
      too little radiation being generated from incoming gluons has been fixed.
\item The scale of the veto on the production of radiation in the parton shower
      has been changed from the $p_T$ of the particles in the hard process to
      their transverse mass, this has a significant effect for top production.
\item Intrinsic $p_T$ is no longer generated for the secondary scattering
      processes generated using the multiple parton-parton scattering model.
\item The directions of hadrons containing a quark from the
      perturbative stage of event were not correctly smeared, this has been corrected.
\item A bug in the selection of the hadrons in cluster decays which lead to $K_L^0$
      rather than $K_0$ and $\bar{K}_0$ mesons being produced has been fixed.
      A related problem which prevented more than one meson which
      has the same flavour composition and mass being produced has also been fixed.
\item A problem with the matrix element correction for top decay in
      rare cases where the off-shell $W$ boson mass is large has been fixed.
\item A workaround for problems with the built-in \textsf{gcc} {\tt abs} function
      has been added.
\item In order to fix some problems with the positions of particles we
      no longer include displacements for intermediates particles produced
      before the hadrons produced in the hadronization phase, all or which are
      assumed to be produced at the origin.
\item The limits on the off-shell mass of BSM particles is now set by default
      when the width is calculated or read from an input file.
\item In BSM models the branching ratios for
      Higgs decays are now correctly reset from the SM values.
\item A number of problems related to initializing and running the generator
      using the \texttt{Herwig++ read} command have been fixed.
\item The handling of problems when the unweighting of the matrix element in the
      generation of multiple parton-parton scattering fails has been improved.
\item Loop protection has been added to the four-body decays in the
      \textsf{Kinematics} class to prevent rare cases where an extremely large number
      of iterations were required. By default these decays now use the \textsf{MAMBO}
      algorithm to avoid this problem.
\item The value of $\pi$ used in various \textsf{AnalysisHandler}s are now
      consistently set using the value \textsf{Constants::pi}.
\end{itemize}

\section{Summary}

  \HWPP\,2.2 is the third version of the \HWPP\ program with a complete simulation of
  hadron-hadron physics and contains incremental changes with respect to the previous
  version. The program has been extensively tested against
  a large number of observables from LEP, Tevatron and B factories.
  All the features needed for realistic studies for
  hadron-hadron collisions are now present and  we look forward to
  feedback and input from users, especially
  from the Tevatron and LHC experiments.

  Our next major milestone is the release of version 3.0 which will be at least as
  complete as \HW\ in all aspects of LHC and linear collider simulation.
  Following the release of \HWPP\,3.0 we expect that support for the
  {\sf FORTRAN} program will cease.

\section*{Acknowledgements}

This work was supported by Science and Technology Facilities Council,
formerly the Particle Physics and Astronomy Research Council, the
European Union Marie Curie Research Training Network MCnet under
contract MRTN-CT-2006-035606 and the Helmholtz--Alliance ``Physics at
the Terascale''. The research of K.~Hamilton was supported
by the Belgian Interuniversity Attraction Pole, PAI, P6/11.  M. B\"ahr
acknowledges support from the ``Promotionskolleg am Centrum f\"ur
Elementarteilchenphysik und Astroteilchenphysik CETA'' and
Landesgraduiertenf\"orderung Baden-W\"urttemberg.  S. Pl\"atzer
acknowledges support from the Landesgraduiertenf\"orderung
Baden-W\"urttemberg.

  \providecommand{\href}[2]{#2}\begingroup\raggedright

High Energy Physics - Phenomenology (hep-ph)

Herwig++ 2.2 Release Note