<commit_before>% !TEX TS-program = pdflatexmk

\documentclass[10pt, a4paper]{article}
\usepackage{lrec2014}
\usepackage{graphicx}
\usepackage{microtype}
\usepackage{cleveref}
\usepackage{xcolor}
\usepackage{listings}

\lstset{
  language=Java,
  basicstyle=\scriptsize\ttfamily,
  keywordstyle=\color{red!60!blue}\ttfamily\bfseries,
  commentstyle=\color{green!50!black}\ttfamily,
}

\hyphenation{Berkeley-Parser}

\newcommand{\code}[1]{\texttt{\small #1}}

\title{ClearTK 2.0: Design Patterns for Machine Learning in UIMA}

\name{<NAME>$^1$, <NAME>$^2$, <NAME>$^2$}

\address{%
$^1$University of Alabama at Birmingham, Birmingham, AL, USA \\
$^2$University of Colorado at Boulder, Boulder, CO, USA \\
\texttt{<EMAIL>}, \texttt{<EMAIL>}, \texttt{<EMAIL>}
}

\hyphenation{Annotation-Handler Classifier-Annotator Chunker-Feature-Extractor Collection-Reader URI-Collection-Reader}

\abstract{%
ClearTK adds machine learning functionality to the UIMA framework, providing wrappers to popular machine learning libraries, a rich feature extraction library that works across different classifiers, and utilities for applying and evaluating machine learning models.
Since its inception in 2008, ClearTK has evolved in response to feedback from developers and the community.
This evolution has followed a number of important design principles including:
conceptually simple annotator interfaces,
readable pipeline descriptions,
minimal collection readers,
type system agnostic code,
modules organized for ease of import,
and assisting user comprehension of the complex UIMA framework.
\\ \newline
\Keywords{NLP frameworks, machine learning, UIMA}}


\begin{document}

\maketitleabstract

\section{Introduction}
The Unstructured Information Management Architecture (UIMA) framework for developing natural language processing pipelines has grown in popularity since it was open-sourced by IBM in 2005.  More recently, UIMA has gained recognition as the underlying architecture of the IBM Watson system that defeated human champions in the game show Jeopardy! \cite{ferrucci_building_2010}.
However, the framework only establishes an architecture for connecting NLP components and does not directly support constructing machine learning classifiers based on sets of features.

The ClearTK framework\footnote{http://www.cleartk.org/} was introduced to address this gap \cite{ogren-etal:2008:UIMA-LREC,ogren-etal:2009:UIMA-GSCL} by providing:
\begin{itemize}
\item A common interface and wrappers for popular machine learning libraries such as SVMlight, LIBSVM, LIBLINEAR, OpenNLP MaxEnt, and Mallet.
\item A rich feature extraction library that can be used with any of the machine learning classifiers. Under the covers, ClearTK understands each of the native machine learning libraries and translates features into a format appropriate to whatever model is being used.
\item Infrastructure for using and evaluating machine learning classifiers within the UIMA framework.
\end{itemize}

Since its inception in 2008, ClearTK has been adopted by multiple developers worldwide in both academia and industry (including University of Colorado, Technische Universit\"{a}t Darmstadt, Apache cTAKES, Thomson Reuters, and 3M) and has been employed on diverse domains including clinical text, social media and student writing.
ClearTK has been downloaded over 1700 times in just the past year, the project site receives over 100 new visits a month, and 78 developers have starred the project in Google Code.  
This growing user and developer base has provided a wealth of feedback that has led to a large number of changes.
In this paper, we reflect on key lessons learned over the last 5 years, and how they generally inform the design of natural language processing frameworks.

\section{Annotators should be conceptually simple}
\label{sec:annotators}
A core aspect of UIMA is the Annotator, which provides a \code{process} method that inspects the document (a \code{JCas} in UIMA), performs analyses, and stores resulting annotations.  This is a familiar construct, even to novice UIMA users, as there are analogs in other frameworks, e.g. Stanford CoreNLP's \code{CoreMap}\footnote{http://nlp.stanford.edu/software/corenlp.shtml} and GATE's \code{Document} \cite{Cunningham2011a}.  Our experience suggests that annotators' process methods should orchestrate the core analysis code in ways that are as straightforward and intuitive as possible.  ClearTK in previous iterations has suffered from over-abstraction of analysis code away from the process method -- resulting in difficult to understand code.

In the first versions of ClearTK, developers were required to write two separate annotators: one for writing training data and another for classification.  In practice, the bulk of the code in both annotators shared the same feature extraction steps.  Consequently, this overlapping functionality was structured to remove this redundancy.  The first version of this abstraction used a call-back approach where users only had to implement the feature extraction code.  However, users found the call-back style unintuitive as it diverged heavily from the UIMA \code{process} conventions.  
Thus we redesigned this abstraction into \code{ClassifierAnnotator}, which allows users to write their feature extraction code in the standard \code{process} method, but requires them to handle two cases: training mode and classification mode.  
An example of such an annotator is shown in \Cref{fig:bio-chunker-code}.
Though the change to \code{ClassifierAnnotator} requires additional coordination within a single annotator, it is conceptually simpler to learn.
\begin{figure*}
\begin{lstlisting}
public class NamedEntityChunker extends CleartkSequenceAnnotator<String> {
  ...
  private BioChunking<Token, NamedEntityMention> chunking = new BioChunking<>(
      Token.class, NamedEntityMention.class, "mentionType");
  ...
  public void process(JCas jCas) throws AnalysisEngineProcessException {
    for (Sentence sentence : JCasUtil.select(jCas, Sentence.class)) {
      // extract features for each token in the sentence
      List<Token> tokens = JCasUtil.selectCovered(jCas, Token.class, sentence);
      List<List<Feature>> featureLists = new ArrayList<>();
      for (Token token : tokens) {
        List<Feature> features = new ArrayList<>();
        features.addAll(this.extractor.extract(jCas, token));
        features.addAll(this.contextExtractor.extract(jCas, token));
        featureLists.add(features);
      }
      // during training, convert NamedEntityMentions in the CAS into expected classifier outcomes
      if (this.isTraining()) {
        // extract the gold (human annotated) NamedEntityMention annotations
        List<NamedEntityMention> namedEntityMentions = JCasUtil.selectCovered(
            jCas, NamedEntityMention.class, sentence);
        // convert the NamedEntityMention annotations into token-level BIO outcome labels
        List<String> outcomes = this.chunking.createOutcomes(jCas, tokens, namedEntityMentions);
        // write the features and outcomes as training instances
        this.dataWriter.write(Instances.toInstances(outcomes, featureLists));
      }
      // during classification, convert classifier outcomes into NamedEntityMentions in the CAS
      else {
        // get the predicted BIO outcome labels from the classifier
        List<String> outcomes = this.classifier.classify(featureLists);
        // create the NamedEntityMention annotations in the CAS
        this.chunking.createChunks(jCas, tokens, outcomes);
      }
    }
  }
}
\end{lstlisting}
\caption{The \code{process} method of a \code{CleartkAnnotator} for BIO-chunking}
\label{fig:bio-chunker-code}
\end{figure*}

We learned a similar lesson in designing an API for Begin-Inside-Outside (BIO) style chunking classification.
The original approach consisted of a subtype of \code{ClassifierAnnotator} that abstracted away aspects of chunking, such as feature extraction and converting labels from token-based chunk-based and back, into separate classes that were dynamically loaded at instantiation time.
In ClearTK 1.2 we replaced this architecture with a set of simple utility objects for converting between chunk labels and tokens labels, allowing for direct use of chunking within a standard \code{process} method (as shown in \Cref{fig:bio-chunker-code}) without need for specialized implementations of interfaces.

% Features like TF-IDF are in the Annotator, not in the encoder
% I decided to skip this; is seemed too complicated to explain 

\section{Pipelines should look like pipelines}
Once a user has developed a number of annotators, they typically string them together in a \emph{pipeline}, indicating the sequence in which these annotators analyze a text.
In ClearTK, users develop a variety of pipelines for different tasks such as training classifiers, making predictions with trained classifiers, testing classifier predictions against a gold standard, etc.  Our experience suggests that pipeline-based code should be structured to make it easy to quickly understand what annotators are running in what order.

Consider the case of model training and evaluation.
ClearTK's first abstraction separated evaluation into various classes and methods:
\begin{itemize}
\item The reader that loaded the training and testing data
\item The preprocessing portion of a pipeline
\item The classifier training portion of a pipeline
\item The classifier prediction portion of a pipeline
\item The evaluation portion of a pipeline
\end{itemize}
These items are easily separable and splitting them reduced code duplication. (For example, the preprocessing portion of the pipeline would be identical for training and testing).
However, because each of these items was implemented in a different class or method, it was often difficult for a reader to understand the big picture of what exactly was running in each pipeline.
In ClearTK 1.2, we simplified this abstraction, resulting in a single evaluation class with just three methods that must be defined:
\begin{enumerate}
\item Read a subset of data with a \code{CollectionReader}
\item Train a model given a \code{CollectionReader}
\item Test a model given a \code{CollectionReader}
\end{enumerate}
In exchange for some duplication (e.g. if training and testing used the same preprocessing) developers are rewarded with more interpretable, self-contained pictures of the training and testing pipelines.

Feature transformation (e.g. normalizing feature values to z-scores or scaling term counts by inverse document frequency) is another example of structuring concerns around pipelines.
In early versions of ClearTK, these kinds of transformations required running a specialized pipeline separately before the real pipeline to collect the sufficient statistics.
This was confusing to users because (1) two pipelines were required for what was conceptually a single pipeline and (2) feature transformations conceptually happen after training data is written, not before.
ClearTK 1.2 introduced \code{TrainableFeatureExtractor}s where a user instead:
\begin{itemize}
\item Runs the original pipeline for writing training data. The \code{TrainableFeatureExtractor} will flag features that need additional post-processing.
\item Ends the pipeline with an \code{InstanceDataWriter} that serializes the features for re-use.
\item Invokes the \code{TrainableFeaturesExtractor}'s \code{train} method on the serialized features to store sufficient statistics.
\item Uses the original data writer on the transformed features to write out training data for a classifier.
\end{itemize}
We found that this approach aligned better with the conceptual expectations of our users.


\section{Collection readers should be minimal}
In UIMA, a \code{CollectionReader} is the connection between the source (file, URL, etc.) and the UIMA document (\code{JCas}) object.
Early versions of ClearTK used the \code{CollectionReader} mechanism to both read in the text and import various annotation formats (TreeBank, PropBank, etc.).

However, as ClearTK developed support for importing more annotation formats, it became clear that this approach was problematic.
UIMA allows only a single \code{CollectionReader} at the beginning of each pipeline, so you cannot, for example, have both a \code{CollectionReader} for TreeBank and one for TimeML in the same pipeline, even if both layers of annotation exist for your document.
The solution to this problem is to view these TreeBank and TimeML readers not as \code{CollectionReader}s, but as regular UIMA annotators whose \code{process(JCas)} method utilizes external resources (e.g. a \code{.mrg} or a \code{.tml} file) to produce annotations to be added to the \code{JCas}.

ClearTK now recommends only one \code{CollectionReader}, \code{URICollectionReader}, which does nothing more than create a \code{JCas} containing the source's Uniform Resource Identifier (URI).
Reading the text or annotations over the text is the responsibility of subsequent annotators.
This approach to developing readers has several advantages, including more parallizable pipelines (which UIMA-AS can take advantage of) and added accessibility by leveraging users' existing familiarity with UIMA annotators.


\section{Code should be type system agnostic}
All UIMA annotators must declare a type system, which defines the annotations and attributes that an annotator may add to documents.
Due to varying requirements imposed by different domains and use cases, there is not yet a generally agreed upon NLP type system for UIMA, and thus many UIMA annotators cannot be combined easily.
In ClearTK, we have always been careful to decouple the machine learning framework from the type system. All of the machinery for creating classifier-based annotators including feature extraction, feature normalization, chunking, training, classification, etc. is completely type system independent.

%A case in point is a feature extractor called \code{CleartkExtractor}.  This feature extractor allows one to create features such as ``get the text of the token two tokens before the focus annotation'' or ``get the part-of-speech tag of the token three tokens after the focus annotation''.  These and many more related feature extraction scenarios are all accomplished by instantiating a CleartkExtractor with the types specific to a CleartkAnnotator's type system.  This eliminates the need to write many specialized feature extractors for different type systems.  This is true for all of the feature extractors provided the core ClearTK project including those that generate bag-of-words features and TF-IDF weighted features as well as the code that supports BIO-style chunking.  Thus, the only code that needs to know about a specific type system is the code written by a user of ClearTK who is implementing analysis engines with the framework.

However, other parts of ClearTK do depend on a specific type system, e.g. for reading different annotations from corpora, for wrapping the output produced by non-UIMA annotators, and for constructing state-of-the-art systems like ClearTK-TimeML \cite{bethard:2013:SemEval-2013}.
It is quite difficult to write a truly type-system agnostic UIMA annotator.
For example, the OpenNLP UIMA annotators are intended to be type system agnostic, but in fact make type-system specific assumptions, like representing the part-of-speech as a string-valued attribute of a token annotation.
To avoid this level of specific type system dependence, we have found it to be necessary to define interfaces for the various operations on tokens, sentences, parses, etc.
Such an approach has been implemented in ClearTK 2.0's wrappers for ClearNLP, and we plan to extend this to other areas in the future.


\section{Modules should match natural subsets}
ClearTK provides many different types of utilities (machine learning wrappers, readers for various corpora, UIMA wrappers for non-UIMA components like MaltParser or Stanford CoreNLP, etc.) and so it has been necessary to split ClearTK up into a small number of modules to allow users to depend on only those parts of ClearTK that they need.
In early versions of ClearTK, we structured these based on the types of annotations being processed, e.g. code for reading PennTreebank trees was put into the same module as our wrapper for OpenNLP's parser.
The idea was that if you were working on, say, parsing, you would want access to all the different parsing algorithms.
However, we found that this approach did not scale.
For example, very few users would want to include all of OpenNLP, MaltParser, BerkeleyParser, Stanford CoreNLP, etc. just to read trees from a PennTreebank file.<commit_msg>Adds evaluation pipeline example<commit_after>% !TEX TS-program = pdflatexmk

\documentclass[10pt, a4paper]{article}
\usepackage{lrec2014}
\usepackage{graphicx}
\usepackage{microtype}
\usepackage{cleveref}
\usepackage{xcolor}
\usepackage{listings}

\lstset{
  language=Java,
  basicstyle=\scriptsize\ttfamily,
  keywordstyle=\color{red!60!blue}\ttfamily\bfseries,
  commentstyle=\color{green!50!black}\ttfamily,
}

\hyphenation{Berkeley-Parser}

\newcommand{\code}[1]{\texttt{\small #1}}

\title{ClearTK 2.0: Design Patterns for Machine Learning in UIMA}

\name{<NAME>$^1$, <NAME>$^2$, <NAME>$^2$}

\address{%
$^1$University of Alabama at Birmingham, Birmingham, AL, USA \\
$^2$University of Colorado at Boulder, Boulder, CO, USA \\
\texttt{<EMAIL>}, \texttt{<EMAIL>}, \texttt{<EMAIL>}
}

\hyphenation{Annotation-Handler Classifier-Annotator Chunker-Feature-Extractor Collection-Reader URI-Collection-Reader}

\abstract{%
ClearTK adds machine learning functionality to the UIMA framework, providing wrappers to popular machine learning libraries, a rich feature extraction library that works across different classifiers, and utilities for applying and evaluating machine learning models.
Since its inception in 2008, ClearTK has evolved in response to feedback from developers and the community.
This evolution has followed a number of important design principles including:
conceptually simple annotator interfaces,
readable pipeline descriptions,
minimal collection readers,
type system agnostic code,
modules organized for ease of import,
and assisting user comprehension of the complex UIMA framework.
\\ \newline
\Keywords{NLP frameworks, machine learning, UIMA}}


\begin{document}

\maketitleabstract

\section{Introduction}
The Unstructured Information Management Architecture (UIMA) framework for developing natural language processing pipelines has grown in popularity since it was open-sourced by IBM in 2005.  More recently, UIMA has gained recognition as the underlying architecture of the IBM Watson system that defeated human champions in the game show Jeopardy! \cite{ferrucci_building_2010}.
However, the framework only establishes an architecture for connecting NLP components and does not directly support constructing machine learning classifiers based on sets of features.

The ClearTK framework\footnote{http://www.cleartk.org/} was introduced to address this gap \cite{ogren-etal:2008:UIMA-LREC,ogren-etal:2009:UIMA-GSCL} by providing:
\begin{itemize}
\item A common interface and wrappers for popular machine learning libraries such as SVMlight, LIBSVM, LIBLINEAR, OpenNLP MaxEnt, and Mallet.
\item A rich feature extraction library that can be used with any of the machine learning classifiers. Under the covers, ClearTK understands each of the native machine learning libraries and translates features into a format appropriate to whatever model is being used.
\item Infrastructure for using and evaluating machine learning classifiers within the UIMA framework.
\end{itemize}

Since its inception in 2008, ClearTK has been adopted by multiple developers worldwide in both academia and industry (including University of Colorado, Technische Universit\"{a}t Darmstadt, Apache cTAKES, Thomson Reuters, and 3M) and has been employed on diverse domains including clinical text, social media and student writing.
ClearTK has been downloaded over 1700 times in just the past year, the project site receives over 100 new visits a month, and 78 developers have starred the project in Google Code.  
This growing user and developer base has provided a wealth of feedback that has led to a large number of changes.
In this paper, we reflect on key lessons learned over the last 5 years, and how they generally inform the design of natural language processing frameworks.

\section{Annotators should be conceptually simple}
\label{sec:annotators}
A core aspect of UIMA is the Annotator, which provides a \code{process} method that inspects the document (a \code{JCas} in UIMA), performs analyses, and stores resulting annotations.  This is a familiar construct, even to novice UIMA users, as there are analogs in other frameworks, e.g. Stanford CoreNLP's \code{CoreMap}\footnote{http://nlp.stanford.edu/software/corenlp.shtml} and GATE's \code{Document} \cite{Cunningham2011a}.  Our experience suggests that annotators' process methods should orchestrate the core analysis code in ways that are as straightforward and intuitive as possible.  ClearTK in previous iterations has suffered from over-abstraction of analysis code away from the process method -- resulting in difficult to understand code.

In the first versions of ClearTK, developers were required to write two separate annotators: one for writing training data and another for classification.  In practice, the bulk of the code in both annotators shared the same feature extraction steps.  Consequently, this overlapping functionality was structured to remove this redundancy.  The first version of this abstraction used a call-back approach where users only had to implement the feature extraction code.  However, users found the call-back style unintuitive as it diverged heavily from the UIMA \code{process} conventions.  
Thus we redesigned this abstraction into \code{ClassifierAnnotator}, which allows users to write their feature extraction code in the standard \code{process} method, but requires them to handle two cases: training mode and classification mode.  
An example of such an annotator is shown in \Cref{fig:bio-chunker-code}.
Though the change to \code{ClassifierAnnotator} requires additional coordination within a single annotator, it is conceptually simpler to learn.
\begin{figure*}
\begin{lstlisting}
public class NamedEntityChunker extends CleartkSequenceAnnotator<String> {
  ...
  private BioChunking<Token, NamedEntityMention> chunking = new BioChunking<>(
      Token.class, NamedEntityMention.class, "mentionType");
  ...
  public void process(JCas jCas) throws AnalysisEngineProcessException {
    for (Sentence sentence : JCasUtil.select(jCas, Sentence.class)) {
      // extract features for each token in the sentence
      List<Token> tokens = JCasUtil.selectCovered(jCas, Token.class, sentence);
      List<List<Feature>> featureLists = new ArrayList<>();
      for (Token token : tokens) {
        List<Feature> features = new ArrayList<>();
        features.addAll(this.extractor.extract(jCas, token));
        features.addAll(this.contextExtractor.extract(jCas, token));
        featureLists.add(features);
      }
      // during training, convert NamedEntityMentions in the CAS into expected classifier outcomes
      if (this.isTraining()) {
        // extract the gold (human annotated) NamedEntityMention annotations
        List<NamedEntityMention> namedEntityMentions = JCasUtil.selectCovered(
            jCas, NamedEntityMention.class, sentence);
        // convert the NamedEntityMention annotations into token-level BIO outcome labels
        List<String> outcomes = this.chunking.createOutcomes(jCas, tokens, namedEntityMentions);
        // write the features and outcomes as training instances
        this.dataWriter.write(Instances.toInstances(outcomes, featureLists));
      }
      // during classification, convert classifier outcomes into NamedEntityMentions in the CAS
      else {
        // get the predicted BIO outcome labels from the classifier
        List<String> outcomes = this.classifier.classify(featureLists);
        // create the NamedEntityMention annotations in the CAS
        this.chunking.createChunks(jCas, tokens, outcomes);
      }
    }
  }
}
\end{lstlisting}
\caption{The \code{process} method of a \code{CleartkAnnotator} for BIO-chunking}
\label{fig:bio-chunker-code}
\end{figure*}

We learned a similar lesson in designing an API for Begin-Inside-Outside (BIO) style chunking classification.
The original approach consisted of a subtype of \code{ClassifierAnnotator} that abstracted away aspects of chunking, such as feature extraction and converting labels from token-based chunk-based and back, into separate classes that were dynamically loaded at instantiation time.
In ClearTK 1.2 we replaced this architecture with a set of simple utility objects for converting between chunk labels and tokens labels, allowing for direct use of chunking within a standard \code{process} method (as shown in \Cref{fig:bio-chunker-code}) without need for specialized implementations of interfaces.

% Features like TF-IDF are in the Annotator, not in the encoder
% I decided to skip this; is seemed too complicated to explain 

\section{Pipelines should look like pipelines}
Once a user has developed a number of annotators, they typically string them together in a \emph{pipeline}, indicating the sequence in which these annotators analyze a text.
In ClearTK, users develop a variety of pipelines for different tasks such as training classifiers, making predictions with trained classifiers, testing classifier predictions against a gold standard, etc.  Our experience suggests that pipeline-based code should be structured to make it easy to quickly understand what annotators are running in what order.

Consider the case of model training and evaluation.
ClearTK's first abstraction separated evaluation into various classes and methods:
\begin{itemize}
\item The reader that loaded the training and testing data
\item The preprocessing portion of a pipeline
\item The classifier training portion of a pipeline
\item The classifier prediction portion of a pipeline
\item The evaluation portion of a pipeline
\end{itemize}
These items are easily separable and splitting them reduced code duplication. (For example, the preprocessing portion of the pipeline would be identical for training and testing).
However, because each of these items was implemented in a different class or method, it was often difficult for a reader to understand the big picture of what exactly was running in each pipeline.
In ClearTK 1.2, we simplified this abstraction, resulting in a single evaluation class with just three methods that must be defined:
\begin{enumerate}
\item Read a subset of data with a \code{CollectionReader}
\item Train a model given a \code{CollectionReader}
\item Test a model given a \code{CollectionReader}
\end{enumerate}
A partial example of such an evaluation class is shown in \Cref{fig:evaluation-code}.
In exchange for some duplication (e.g. if training and testing used the same preprocessing) developers are rewarded with more interpretable, self-contained pictures of the training and testing pipelines.
\begin{figure*}
\begin{lstlisting}
public class EvaluateNamedEntityChunker extends
    Evaluation_ImplBase<File, AnnotationStatistics<String>> {

  ...
  protected CollectionReader getCollectionReader(List<File> files) throws Exception {
    return CollectionReaderFactory.createReader(UriCollectionReader.getDescriptionFromFiles(files));
  }
  ...
  public void train(CollectionReader collectionReader, File outputDirectory) throws Exception {
    // assemble the training pipeline
    AggregateBuilder aggregate = new AggregateBuilder();
    // an annotator that loads the text from the training file URIs
    aggregate.add(UriToDocumentTextAnnotator.getDescription());
    // an annotator that parses and loads MASC named entity annotations (and tokens)
    aggregate.add(MascGoldAnnotator.getDescription());
    // an annotator that adds part-of-speech tags
    aggregate.add(PosTaggerAnnotator.getDescription());
    // our NamedEntityChunker annotator, configured to write Mallet CRF training data
    aggregate.add(AnalysisEngineFactory.createEngineDescription(
        NamedEntityChunker.class,
        CleartkSequenceAnnotator.PARAM_IS_TRAINING,
        true,
        DirectoryDataWriterFactory.PARAM_OUTPUT_DIRECTORY,
        outputDirectory,
        DefaultSequenceDataWriterFactory.PARAM_DATA_WRITER_CLASS_NAME,
        MalletCrfStringOutcomeDataWriter.class));
    // run the pipeline over the training corpus
    SimplePipeline.runPipeline(collectionReader, aggregate.createAggregateDescription());
    // train a Mallet CRF model on the training data
    Train.main(outputDirectory);
  }
  ...
}
\end{lstlisting}
\caption{The \code{getCollectionReader} and \code{train} methods of a ClearTK evaluation class}
\label{fig:evaluation-code}
\end{figure*}

Feature transformation (e.g. normalizing feature values to z-scores or scaling term counts by inverse document frequency) is another example of structuring concerns around pipelines.
In early versions of ClearTK, these kinds of transformations required running a specialized pipeline separately before the real pipeline to collect the sufficient statistics.
This was confusing to users because (1) two pipelines were required for what was conceptually a single pipeline and (2) feature transformations conceptually happen after training data is written, not before.
ClearTK 1.2 introduced \code{TrainableFeatureExtractor}s where a user instead:
\begin{itemize}
\item Runs the original pipeline for writing training data. The \code{TrainableFeatureExtractor} will flag features that need additional post-processing.
\item Ends the pipeline with an \code{InstanceDataWriter} that serializes the features for re-use.
\item Invokes the \code{TrainableFeaturesExtractor}'s \code{train} method on the serialized features to store sufficient statistics.
\item Uses the original data writer on the transformed features to write out training data for a classifier.
\end{itemize}
We found that this approach aligned better with the conceptual expectations of our users.


\section{Collection readers should be minimal}
In UIMA, a \code{CollectionReader} is the connection between the source (file, URL, etc.) and the UIMA document (\code{JCas}) object.
Early versions of ClearTK used the \code{CollectionReader} mechanism to both read in the text and import various annotation formats (TreeBank, PropBank, etc.).

However, as ClearTK developed support for importing more annotation formats, it became clear that this approach was problematic.
UIMA allows only a single \code{CollectionReader} at the beginning of each pipeline, so you cannot, for example, have both a \code{CollectionReader} for TreeBank and one for TimeML in the same pipeline, even if both layers of annotation exist for your document.
The solution to this problem is to view these TreeBank and TimeML readers not as \code{CollectionReader}s, but as regular UIMA annotators whose \code{process(JCas)} method utilizes external resources (e.g. a \code{.mrg} or a \code{.tml} file) to produce annotations to be added to the \code{JCas}.

ClearTK now recommends only one \code{CollectionReader}, \code{URICollectionReader}, which does nothing more than create a \code{JCas} containing the source's Uniform Resource Identifier (URI).
Reading the text or annotations over the text is the responsibility of subsequent annotators.
This approach to developing readers has several advantages, including more parallizable pipelines (which UIMA-AS can take advantage of) and added accessibility by leveraging users' existing familiarity with UIMA annotators.
\Cref{fig:evaluation-code} shows an example usage of \code{URICollectionReader}.


\section{Code should be type system agnostic}
All UIMA annotators must declare a type system, which defines the annotations and attributes that an annotator may add to documents.
Due to varying requirements imposed by different domains and use cases, there is not yet a generally agreed upon NLP type system for UIMA, and thus many UIMA annotators cannot be combined easily.
In ClearTK, we have always been careful to decouple the machine learning framework from the type system. All of the machinery for creating classifier-based annotators including feature extraction, feature normalization, chunking, training, classification, etc. is completely type system independent.

%A case in point is a feature extractor called \code{CleartkExtractor}.  This feature extractor allows one to create features such as ``get the text of the token two tokens before the focus annotation'' or ``get the part-of-speech tag of the token three tokens after the focus annotation''.  These and many more related feature extraction scenarios are all accomplished by instantiating a CleartkExtractor with the types specific to a CleartkAnnotator's type system.  This eliminates the need to write many specialized feature extractors for different type systems.  This is true for all of the feature extractors provided the core ClearTK project including those that generate bag-of-words features and TF-IDF weighted features as well as the code that supports BIO-style chunking.  Thus, the only code that needs to know about a specific type system is the code written by a user of ClearTK who is implementing analysis engines with the framework.

However, other parts of ClearTK do depend on a specific type system, e.g. for reading different annotations from corpora, for wrapping the output produced by non-UIMA annotators, and for constructing state-of-the-art systems like ClearTK-TimeML \cite{bethard:2013:SemEval-2013}.
It is quite difficult to write a truly type-system agnostic UIMA annotator.
For example, the OpenNLP UIMA annotators are intended to be type system agnostic, but in fact make type-system specific assumptions, like representing the part-of-speech as a string-valued attribute of a token annotation.
To avoid this level of specific type system dependence, we have found it to be necessary to define interfaces for the various operations on tokens, sentences, parses, etc.
Such an approach has been implemented in ClearTK 2.0's wrappers for ClearNLP, and we plan to extend this to other areas in the future.


\section{Modules should match natural subsets}
ClearTK provides many different types of utilities (machine learning wrappers, readers for various corpora, UIMA wrappers for non-UIMA components like MaltParser or Stanford CoreNLP, etc.) and so it has been necessary to split ClearTK up into a small number of modules to allow users to depend on only those parts of ClearTK that they need.
In early versions of ClearTK, we structured these based on the types of annotations being processed, e.g. code for reading PennTreebank trees was put into the same module as our wrapper for OpenNLP's parser.
The idea was that if you were working on, say, parsing, you would want access to all the different parsing algorithms.
However, we found that this approach did not scale.
For example, very few users would want to include all of OpenNLP, MaltParser, BerkeleyParser, Stanford CoreNLP, etc. just to read trees from a PennTreebank file.
{'fraction_non_alphanumeric': 0.054331242583488724, 'fraction_numerical': 0.0043510199468836525, 'mean_word_length': 4.962769541778976, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '944378', 'n_tokens_mistral': 8911, 'n_tokens_neox': 8339, 'n_words': 4435}