<commit_before>### README is Work in Progress.

# Epic

(c) 2014 <NAME>.

Epic is a structured prediction framework for Scala. It also includes classes for training high-accuracy syntactic parsers, part-of-speech taggers, name entity recognizers, and more.

The current version is 0.2-SNAPSHOT

## Documentation

Documentation will live at the GitHub wiki: <https://github.com/dlwh/epic/wiki>

## Using Epic

### Command-line Usage

Epic bundles command line interfaces for using parsers, NER systems, and POS taggers (and more generally, segmentation and tagging systems). There are three classes, one for each kind of system:

* `epic.parser.ParseText` runs a parser.
* `epic.sequences.SegmentText` runs an NER system, or any kind of segmentation system.
* `epic.sequences.TagText` runs a POS tagger, or any kind of tagging system.

All of these systems expect plain text files as input, along with a path to a model file. The syntax is:

```bash
java -Xmx4g -cp /path/to/epic-assembly-0.2-SNAPSHOT.jar epic.parser.ParseText --model /path/to/model.ser.gz --nthreads <number of threads> [files]
```

Currently, all text is output to standard out. In the future, we will support output in a way that differentiates the files. In addition, we will add support for reading from stdin. By default, the system will use all available cores for execution.

Models can be downloaded from <http://www.scalanlp.org/models/> or from Maven Central. ([See below](#pre-trained-models).)

### Programmatic Usage

Epic also supports programmatic usage. All of the models assume that text has been segmented and tokenized.

#### Preprocessing text

To preprocess text so that the models can use them, you will need to segment out sentences and tokenize the sentences into individual words. Epic comes with classes to do both.

Once you have a sentence, you can tokenize it using a `epic.preprocess.TreebankTokenizer`, which takes a string and returns a sequence of tokens. All told, the pipeline looks like this:

```scala
val text = getSomeText();

val sentenceSplitter = MLSentenceSegmenter.bundled().get
val tokenizer = new epic.preprocess.TreebankTokenizer()

val sentences: IndexedSeq[IndexedSeq[String]] = sentenceSplitter(text).map(tokenizer).toIndexedSeq

for(sentence <- sentences) {
  // use the sentence tokens
}

```



#### Parser

To use the parser programmaticaly, deserialize a parser model--either using `epic.models.deserialize[Parser[AnnotatedLabel, String]](path)` or using the ParserSelector. Then, give the parser segmented and tokenized text:

```scala
val parser = epic.models.deserialize[Parser[AnnotataedLabel, String]](path)

// or:

val parser = epic.models.ParserSelector.loadParser("en") // or another 2 letter code.

val tree = parser(sentence)

println(tree.render(words))

```

Trees have a number of methods on them. See the class definition or [API docs](http://www.scalanlp.org/api/epic).

#### Part-of-Speech Tagger

Using a Part-of-Speech tagger is similar to using a parser: load a model, tokenize some text, run the tagger. All taggers are (currently) [linear chain conditional random fields](http://people.cs.umass.edu/~mccallum/papers/crf-tutorial.pdf), or CRFs. (You don't need to understand them to use them. They are just a machine learning method for assigning a sequence of tags to a sequence of words.)

```scala
val tagger = epic.models.deserialize[CRF[AnnotatedLabel, String]](path)

// or:

val tagger = epic.models.PosTaggerSelector.loadTagger("en") // or another 2 letter code.

val tags = tagger(sentence)

println(tags.render)

```


#### Named Entity Recognition

Using a named entity recognizer is similar to using a pos tagger: load a model, tokenize some text, run the recognizer. All NER systems are (currently) [linear chain semi-Markov conditional random fields](http://people.cs.umass.edu/~mccallum/papers/crf-tutorial.pdf), or SemiCRFs. (You don't need to understand them to use them. They are just a machine learning method for segmenting text into fields.

```scala

```scala
val tagger = epic.models.deserialize[SemiCRF[AnnotatedLabel, String]](path)

// or:

val tagger = epic.models.NerSelector.loadSegmenter("en") // or another 2 letter code.

val segments = tagger(sentence)

println(tags.render(tagger.outsideLabel))

```

The outside label of a SemiCRF is the label that is consider not part of a "real" segment. For instance, in NER, it is the label given to words that are not named entities.

### Pre-trained Models

Epic provides a number of pre-trained models. These are available as Maven artifacts from Maven Central, and can be loaded at runtime. To use a specific model, just depend on it (or alternatively download the jar file). You can then load the parser by calling, for example:

```scala
epic.parser.models.en.span.EnglishSpanParser.load()
```

This will load the  model and return a `Parser` object. If you want to not hardwire dependencies, either for internationalization or to potentially try different models, use `epic.models.ParserSelector.loadParser(language)`, where
language is the [two letter code for the language](http://www.loc.gov/standards/iso639-2/php/code_list.php) you want to use.

To following models are available at this time:


* Parser
  * English: 
    ```
    "org.scalanlp" %% "epic-parser-en-span" % "0.1"
    ```
  * Basque: 
    ```
    "org.scalanlp" %% "epic-parser-eu-span" % "0.1"
    ```
  * French: 
    ```
    "org.scalanlp" %% "epic-parser-fr-span" % "0.1"
    ```
  * German: 
    ```
    "org.scalanlp" %% "epic-parser-de-span" % "0.1"
    ```
  * Hungarian: 
    ```
    "org.scalanlp" %% "epic-parser-hu-span" % "0.1"
    ```
  * Korean: 
    ```
    "org.scalanlp" %% "epic-parser-ko-span" % "0.1"
    ```
  * Polish:
    ```
    "org.scalanlp" %% "epic-parser-pl-span" % "0.1"
    ```
  * Swedish: 
    ```
    "org.scalanlp" %% "epic-parser-sv-span" % "0.1"
    ```
* POS Taggers
  * English: 
    ```
    "org.scalanlp" %% "epic-pos-en" % "0.1"
    ```
  * Basque: 
    ```
    "org.scalanlp" %% "epic-pos-eu" % "0.1"
    ```
  * French: 
    ```
    "org.scalanlp" %% "epic-pos-fr" % "0.1"
    ```
  * German: 
    ```
    "org.scalanlp" %% "epic-pos-de" % "0.1"
    ```
  * Hungarian: 
    ```
    "org.scalanlp" %% "epic-pos-hu" % "0.1"
    ```
  * Polish: 
    ```
    "org.scalanlp" %% "epic-pos-pl" % "0.1"
    ```
  * Swedish: 
    ```
    "org.scalanlp" %% "epic-pos-sv" % "0.1"
    ```
* Named Entity Recognizers
  * English: 
    ```
    "org.scalanlp" %% "epic-ner-en-conll" % "0.1"
    ```


If you use any of the parser models in research publications, please cite:

> <NAME>, <NAME>, and <NAME>. 2014. Less Grammar, More Features. In ACL.

If you use the other things, just link to Epic.

## Building Epic

In order to do anything besides use pre-trained models, you will probably need to build Epic.

To build, you need a release of [SBT 0.13.2](http://www.scala-sbt.org/0.13.2/docs/Getting-Started/Setup.html)

then run 

<pre>
$ sbt assembly
</pre>

which will compile everything, run tests, and build a fatjar that includes all dependencies.

<commit_msg>make the model files be up to date<commit_after>### README is Work in Progress.

# Epic

(c) 2014 <NAME>.

Epic is a structured prediction framework for Scala. It also includes classes for training high-accuracy syntactic parsers, part-of-speech taggers, name entity recognizers, and more.

The current version is 0.2-SNAPSHOT

## Documentation

Documentation will live at the GitHub wiki: <https://github.com/dlwh/epic/wiki>

## Using Epic

### Command-line Usage

Epic bundles command line interfaces for using parsers, NER systems, and POS taggers (and more generally, segmentation and tagging systems). There are three classes, one for each kind of system:

* `epic.parser.ParseText` runs a parser.
* `epic.sequences.SegmentText` runs an NER system, or any kind of segmentation system.
* `epic.sequences.TagText` runs a POS tagger, or any kind of tagging system.

All of these systems expect plain text files as input, along with a path to a model file. The syntax is:

```bash
java -Xmx4g -cp /path/to/epic-assembly-0.2-SNAPSHOT.jar epic.parser.ParseText --model /path/to/model.ser.gz --nthreads <number of threads> [files]
```

Currently, all text is output to standard out. In the future, we will support output in a way that differentiates the files. In addition, we will add support for reading from stdin. By default, the system will use all available cores for execution.

Models can be downloaded from <http://www.scalanlp.org/models/> or from Maven Central. ([See below](#pre-trained-models).)

### Programmatic Usage

Epic also supports programmatic usage. All of the models assume that text has been segmented and tokenized.

#### Preprocessing text

To preprocess text so that the models can use them, you will need to segment out sentences and tokenize the sentences into individual words. Epic comes with classes to do both.

Once you have a sentence, you can tokenize it using a `epic.preprocess.TreebankTokenizer`, which takes a string and returns a sequence of tokens. All told, the pipeline looks like this:

```scala
val text = getSomeText();

val sentenceSplitter = MLSentenceSegmenter.bundled().get
val tokenizer = new epic.preprocess.TreebankTokenizer()

val sentences: IndexedSeq[IndexedSeq[String]] = sentenceSplitter(text).map(tokenizer).toIndexedSeq

for(sentence <- sentences) {
  // use the sentence tokens
}

```



#### Parser

To use the parser programmaticaly, deserialize a parser model--either using `epic.models.deserialize[Parser[AnnotatedLabel, String]](path)` or using the ParserSelector. Then, give the parser segmented and tokenized text:

```scala
val parser = epic.models.deserialize[Parser[AnnotataedLabel, String]](path)

// or:

val parser = epic.models.ParserSelector.loadParser("en") // or another 2 letter code.

val tree = parser(sentence)

println(tree.render(words))

```

Trees have a number of methods on them. See the class definition or [API docs](http://www.scalanlp.org/api/epic).

#### Part-of-Speech Tagger

Using a Part-of-Speech tagger is similar to using a parser: load a model, tokenize some text, run the tagger. All taggers are (currently) [linear chain conditional random fields](http://people.cs.umass.edu/~mccallum/papers/crf-tutorial.pdf), or CRFs. (You don't need to understand them to use them. They are just a machine learning method for assigning a sequence of tags to a sequence of words.)

```scala
val tagger = epic.models.deserialize[CRF[AnnotatedLabel, String]](path)

// or:

val tagger = epic.models.PosTaggerSelector.loadTagger("en") // or another 2 letter code.

val tags = tagger(sentence)

println(tags.render)

```


#### Named Entity Recognition

Using a named entity recognizer is similar to using a pos tagger: load a model, tokenize some text, run the recognizer. All NER systems are (currently) [linear chain semi-Markov conditional random fields](http://people.cs.umass.edu/~mccallum/papers/crf-tutorial.pdf), or SemiCRFs. (You don't need to understand them to use them. They are just a machine learning method for segmenting text into fields.

```scala

```scala
val tagger = epic.models.deserialize[SemiCRF[AnnotatedLabel, String]](path)

// or:

val tagger = epic.models.NerSelector.loadSegmenter("en") // or another 2 letter code.

val segments = tagger(sentence)

println(tags.render(tagger.outsideLabel))

```

The outside label of a SemiCRF is the label that is consider not part of a "real" segment. For instance, in NER, it is the label given to words that are not named entities.

### Pre-trained Models

Epic provides a number of pre-trained models. These are available as Maven artifacts from Maven Central, and can be loaded at runtime. To use a specific model, just depend on it (or alternatively download the jar file). You can then load the parser by calling, for example:

```scala
epic.parser.models.en.span.EnglishSpanParser.load()
```

This will load the  model and return a `Parser` object. If you want to not hardwire dependencies, either for internationalization or to potentially try different models, use `epic.models.ParserSelector.loadParser(language)`, where
language is the [two letter code for the language](http://www.loc.gov/standards/iso639-2/php/code_list.php) you want to use.

To following models are available at this time:


* Parser
  * English: 
    ```
    "org.scalanlp" %% "epic-parser-en-span" % "2014.6.3-SNAPSHOT"
    ```
  * Basque: 
    ```
    "org.scalanlp" %% "epic-parser-eu-span" % "2014.6.3-SNAPSHOT"
    ```
  * French: 
    ```
    "org.scalanlp" %% "epic-parser-fr-span" % "2014.6.3-SNAPSHOT"
    ```
  * German: 
    ```
    "org.scalanlp" %% "epic-parser-de-span" % "2014.6.3-SNAPSHOT"
    ```
  * Hungarian: 
    ```
    "org.scalanlp" %% "epic-parser-hu-span" % "2014.6.3-SNAPSHOT"
    ```
  * Korean: 
    ```
    "org.scalanlp" %% "epic-parser-ko-span" % "2014.6.3-SNAPSHOT"
    ```
  * Polish:
    ```
    "org.scalanlp" %% "epic-parser-pl-span" % "2014.6.3-SNAPSHOT"
    ```
  * Swedish: 
    ```
    "org.scalanlp" %% "epic-parser-sv-span" % "2014.6.3-SNAPSHOT"
    ```
* POS Taggers
  * English: 
    ```
    "org.scalanlp" %% "epic-pos-en" % "2014.6.3-SNAPSHOT"
    ```
  * Basque: 
    ```
    "org.scalanlp" %% "epic-pos-eu" % "2014.6.3-SNAPSHOT"
    ```
  * French: 
    ```
    "org.scalanlp" %% "epic-pos-fr" % "2014.6.3-SNAPSHOT"
    ```
  * German: 
    ```
    "org.scalanlp" %% "epic-pos-de" % "2014.6.3-SNAPSHOT"
    ```
  * Hungarian: 
    ```
    "org.scalanlp" %% "epic-pos-hu" % "2014.6.3-SNAPSHOT"
    ```
  * Polish: 
    ```
    "org.scalanlp" %% "epic-pos-pl" % "2014.6.3-SNAPSHOT"
    ```
  * Swedish: 
    ```
    "org.scalanlp" %% "epic-pos-sv" % "2014.6.3-SNAPSHOT"
    ```
* Named Entity Recognizers
  * English: 
    ```
    "org.scalanlp" %% "epic-ner-en-conll" % "2014.6.3-SNAPSHOT"
    ```


If you use any of the parser models in research publications, please cite:

> <NAME>, <NAME>, and <NAME>. 2014. Less Grammar, More Features. In ACL.

If you use the other things, just link to Epic.

## Building Epic

In order to do anything besides use pre-trained models, you will probably need to build Epic.

To build, you need a release of [SBT 0.13.2](http://www.scala-sbt.org/0.13.2/docs/Getting-Started/Setup.html)

then run 

<pre>
$ sbt assembly
</pre>

which will compile everything, run tests, and build a fatjar that includes all dependencies.


{'fraction_non_alphanumeric': 0.12374419732557333, 'fraction_numerical': 0.012748562322455484, 'mean_word_length': 3.677252106286455, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 2, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4565938', 'n_tokens_mistral': 4924, 'n_tokens_neox': 4614, 'n_words': 1601}