<commit_before>\section{Background}
\label{sec:setup}
% (1 page)

% 1. Basic output specification of KBP.
%   - what are types, and entity links
In knowledge base population,
  each relation is a triple (\textsc{subject}, \textsc{predicate}, \textsc{object}) where \textsc{subject} and \textsc{object} are some globally unique entity identifiers (e.g.\, Wikipedia page titles), \textsc{predicate} is a relation belonging to a specified schema.\footnote{%
    The TAC KBP guidelines specify a total of 65 predicates (including inverses) such as 
     \texttt{per:title} or \texttt{org:founded\_on}, etc.
     etc.
    Subject entities can be people, organizations, geo-political entities, while object entities also include 
    dates, numbers and arbitrary string-values like job titles.
     }
A KBP system returns an output in the form of \textit{relation instances} (\textsc{subject}, \textsc{predicate}, \textsc{object}, \textsc{provenance}), where \textsc{provenance} is a description of where exactly in the document corpus the subject, object and relation were found. 
In the example shown in \figureref{example},
\entity{<NAME>} and \entity{<NAME>} are respectively identified as the subject and object of the predicate \relation{child of}, and the whole sentence is provided as provenance.
The provenance also identifies that \entity{<NAME>} is referenced by \mention{Fisher} within the sentence.
Note that the same relation can be expressed in different parts of the document corpus; each of these instances is described by a unique relation instance.

\paragraph{Pooled evaluation.}
\begin{figure}[t]
  \centering
  \includegraphics[width=0.9\columnwidth]{figures/pooling}
  \caption{\label{fig:pooling}
  In pooled evaluation, a evaluation dataset is constructed by labeling the relational tuples collected from the pooled systems (A and B) and those identified by a team of human annotators (Annotators).
 % Arrows indicate which systems extract which relation instances.
  However, when a new system (C) is evaluated on this dataset, some of its predictions are previously unseen and can not be fairly evaluated.
  Here, the precision and recall for C should be $\frac{3}{3}$ and $\frac{3}{4}$ respectively, but its evaluation scores are estimated to be $\frac{2}{3}$ and $\frac{2}{3}$.
  The discrepancy between these two scores is called \textit{pooling bias}.
  }
\end{figure}

% 2. How is KBP evaluated? 
The primary source of evaluation data for KBP comes from the annual TAC KBP competition organized by NIST \citep{ji2011kbp}.
As mentioned earlier, the dataset is constructed by evaluating every relation instance predicted by 
the participating systems for subject entities that belong to a held-out set of evaluation entities.
%A relational tuple for the relation (\entity{<NAME>}, \relation{child of}, \entity{<NAME>}) that specifies an ambiguous provenance like ``\mention{<NAME>} and \mention{<NAME>} arrived together at the awards show'' would be considered incorrect.
Separately, a team of annotators identify relations for the same evaluation entities by manually searching the document corpus within a time budget.
These two sets of labeled relational tuples are combined and released as the evaluation dataset.
In the example in \reffig{pooling}, systems A and B were used in constructing the pooling dataset, and there are 3 distinct relations in the dataset, between $s_1$ and $o_1$, $o_2$ and $o_3$.

A system is evaluated on the precision of its predicted relation instances for the evaluation entities and on the recall of the \textit{relations} it predicts for the same entities (see \reffig{pooling} for a worked example).
When using the evaluation data during system development, it is common practice to use the more lenient \anydoc{} score that ignores the provenance when checking if an relational tuple is true.
Under this metric, predicting the relation (\entity{<NAME>}, \relation{child of}, \entity{<NAME>}) from an ambiguous provenance like ``\mention{<NAME>} and \mention{<NAME>} arrived together at the awards show'' would be considered correct even though it would be marked wrong under the official metric.

%$t_5$ would be considered to be correct and system B would get a precision and recall of $\frac{2}{2}$ and $\frac{2}{3}$ respectively.

% TODO: do we really need to talk about duplicates?
%\footnote{%
%  The official scoring metric also penalizes systems that predict duplicated relations (e.g.\ if they also predict (\entity{<NAME>}, \relation{child of}, \entity{Debbie})); these typically arise due to an error in entity linking.
%}
%system A would get a precision of $\frac{2}{2}$ for predicting $t_2$ and $t_3$ while system B would get a precision of $\frac{1}{2}$ because $t_5$ specified an incorrect provenance.
%Recall is the fraction of true \textit{relations} that the system identifies, irrespective of which provenance it uses as long as the provenance is correct.
%Thus, system A would get a recall of $\frac{2}{3}$ while system B would only get a recall of $\frac{1}{3}$.
%Finally, \fone{} is the harmonic mean of precision and recall.

%\paragraph{Pooling bias.}
%As part of the competition, every relation predicted for the evaluation entities by the participating systems is labeled by annotators.
%However, when this dataset is used during system development, a new system (e.g.\ system C) may predict relational tuples that are not part of the dataset (e.g.\ $t_6$).
%These tuples can not be assessed without labeling the predictions and are considered to be wrong by default.
%As a result, system C would be evaluated to have a precision and recall of $\frac{2}{3}$ and $\frac{2}{3}$.
%On the other hand, if system C had participated in the pooling process, and $t_6$ were indeed correct, system C get higher precision and recall scores of $\frac{3}{3}$ and $\frac{3}{4}$ and would rank higher than system A.\footnote{Note that the recall scores of system A and B would also decrease because a new true relation had been found.}
%The discrepancy between the scores of a system because it did not participate in the pooling process is called pooling bias.

% vv OLD as of 4/8/17 

%In this section, we'll review the knowledge base population task, define appropriate notation, describe relevant evaluation metrics and discuss the standard evaluation methodology adopted by the TAC-KBP competition.
%
%\subsection{Knowledge base population}<commit_msg>Corrected some tuples -> instances.<commit_after>\section{Background}
\label{sec:setup}
% (1 page)

% 1. Basic output specification of KBP.
%   - what are types, and entity links
In knowledge base population,
  each relation is a triple (\textsc{subject}, \textsc{predicate}, \textsc{object}) where \textsc{subject} and \textsc{object} are some globally unique entity identifiers (e.g.\, Wikipedia page titles), \textsc{predicate} is a relation belonging to a specified schema.\footnote{%
    The TAC KBP guidelines specify a total of 65 predicates (including inverses) such as 
     \texttt{per:title} or \texttt{org:founded\_on}, etc.
     etc.
    Subject entities can be people, organizations, geo-political entities, while object entities also include 
    dates, numbers and arbitrary string-values like job titles.
     }
A KBP system returns an output in the form of \textit{relation instances} (\textsc{subject}, \textsc{predicate}, \textsc{object}, \textsc{provenance}), where \textsc{provenance} is a description of where exactly in the document corpus the subject, object and relation were found. 
In the example shown in \figureref{example},
\entity{<NAME>} and \entity{<NAME>} are respectively identified as the subject and object of the predicate \relation{child of}, and the whole sentence is provided as provenance.
The provenance also identifies that \entity{<NAME>} is referenced by \mention{Fisher} within the sentence.
Note that the same relation can be expressed in different parts of the document corpus; each of these instances is described by a unique relation instance.

\paragraph{Pooled evaluation.}
\begin{figure}[t]
  \centering
  \includegraphics[width=0.9\columnwidth]{figures/pooling}
  \caption{\label{fig:pooling}
  In pooled evaluation, an evaluation dataset is constructed by labeling the relation instances collected from the pooled systems (A and B) and those identified by a team of human annotators (Annotators).
 % Arrows indicate which systems extract which relation instances.
  However, when a new system (C) is evaluated on this dataset, some of its predictions are previously unseen and can not be fairly evaluated.
  Here, the precision and recall for C should be $\frac{3}{3}$ and $\frac{3}{4}$ respectively, but its evaluation scores are estimated to be $\frac{2}{3}$ and $\frac{2}{3}$.
  The discrepancy between these two scores is called \textit{pooling bias}.
  }
\end{figure}

% 2. How is KBP evaluated? 
The primary source of evaluation data for KBP comes from the annual TAC KBP competition organized by NIST \citep{ji2011kbp}.
As mentioned earlier, the dataset is constructed by evaluating every relation instance predicted by 
the participating systems for subject entities that belong to a held-out set of evaluation entities.
%A relational tuple for the relation (\entity{<NAME>}, \relation{child of}, \entity{<NAME>}) that specifies an ambiguous provenance like ``\mention{<NAME>} and \mention{<NAME>} arrived together at the awards show'' would be considered incorrect.
Separately, a team of annotators identify relations for the same evaluation entities by manually searching the document corpus within a time budget.
These two sets of labeled relation instances are combined and released as the evaluation dataset.
In the example in \reffig{pooling}, systems A and B were used in constructing the pooling dataset, and there are 3 distinct relations in the dataset, between $s_1$ and $o_1$, $o_2$ and $o_3$.

A system is evaluated on the precision of its predicted relation instances for the evaluation entities and on the recall of the \textit{relations} it predicts for the same entities (see \reffig{pooling} for a worked example).
When using the evaluation data during system development, it is common practice to use the more lenient \anydoc{} score that ignores the provenance when checking if a relation instance is true.
Under this metric, predicting the relation (\entity{<NAME>}, \relation{child of}, \entity{<NAME>}) from an ambiguous provenance like ``\mention{<NAME>} and \mention{<NAME>} arrived together at the awards show'' would be considered correct even though it would be marked wrong under the official metric.

%$t_5$ would be considered to be correct and system B would get a precision and recall of $\frac{2}{2}$ and $\frac{2}{3}$ respectively.

% TODO: do we really need to talk about duplicates?
%\footnote{%
%  The official scoring metric also penalizes systems that predict duplicated relations (e.g.\ if they also predict (\entity{<NAME>}, \relation{child of}, \entity{Debbie})); these typically arise due to an error in entity linking.
%}
%system A would get a precision of $\frac{2}{2}$ for predicting $t_2$ and $t_3$ while system B would get a precision of $\frac{1}{2}$ because $t_5$ specified an incorrect provenance.
%Recall is the fraction of true \textit{relations} that the system identifies, irrespective of which provenance it uses as long as the provenance is correct.
%Thus, system A would get a recall of $\frac{2}{3}$ while system B would only get a recall of $\frac{1}{3}$.
%Finally, \fone{} is the harmonic mean of precision and recall.

%\paragraph{Pooling bias.}
%As part of the competition, every relation predicted for the evaluation entities by the participating systems is labeled by annotators.
%However, when this dataset is used during system development, a new system (e.g.\ system C) may predict relational tuples that are not part of the dataset (e.g.\ $t_6$).
%These tuples can not be assessed without labeling the predictions and are considered to be wrong by default.
%As a result, system C would be evaluated to have a precision and recall of $\frac{2}{3}$ and $\frac{2}{3}$.
%On the other hand, if system C had participated in the pooling process, and $t_6$ were indeed correct, system C get higher precision and recall scores of $\frac{3}{3}$ and $\frac{3}{4}$ and would rank higher than system A.\footnote{Note that the recall scores of system A and B would also decrease because a new true relation had been found.}
%The discrepancy between the scores of a system because it did not participate in the pooling process is called pooling bias.

% vv OLD as of 4/8/17 

%In this section, we'll review the knowledge base population task, define appropriate notation, describe relevant evaluation metrics and discuss the standard evaluation methodology adopted by the TAC-KBP competition.
%
%\subsection{Knowledge base population}
{'fraction_non_alphanumeric': 0.07832026701855158, 'fraction_numerical': 0.00838314057284794, 'mean_word_length': 4.806219017575485, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6576615', 'n_tokens_mistral': 3327, 'n_tokens_neox': 3179, 'n_words': 1796}