<commit_before>---
title: MeTA System Overview
layout: page
category: tut
order: 1
---

Our task is to improve upon and complement the current body of open source
machine learning and information retrieval software. The existing environment of
this open source software tends to be quite fragmented.

There is rarely a single location for a wide variety of algorithms; a good
example of this is the [LIBLINEAR](http://www.csie.ntu.edu.tw/~cjlin/liblinear/)
software package for SVMs. While this is the most cited of the open source
implementations of linear SVMs, it focuses solely on kernel-less methods. If
presented with a nonlinear classification problem, one would be forced to find a
different software package that supports kernels (such as the same authors'
[LIBSVM](http://www.csie.ntu.edu.tw/~cjlin/libsvm/) package). This places an
undue burden on the researchers---not only are they required to have a detailed
understanding of the research problem at hand, but they are now forced to
understand this fragmented nature of the open source software community, find
the appropriate tools in this mishmash of implementations, and compile and
configure the appropriate tool.

Even when this is all done, there is the problem of data formatting---it is
unlikely that the tools have standardized upon a single input format, so a
certain amount of "data munging" is now required. This all detracts from the
actual research task at hand, which has a marked impact on the speed of
discovery.

**We want to address these issues.** In particular, we want to provide a
unifying framework for text indexing and analysis methods, allowing researchers
to quickly run controlled experiments. We want to modularize the feature
generation, instance representation, data storage formats, and algorithm
implementations; this will allow for researchers to make seamless transitions
along any of these dimensions with minimal effort.

## How does MeTA store data?

All processed data is stored in a `disk_index`. Currently, we have two
index types: `forward_index` and `inverted_index`. The former is keyed by
document IDs, and the latter is keyed by term IDs.

 - `forward_index` is used for applications such as topic modeling and
   most classification tasks
 - `inverted_index` is used to create search engines, or do classification with
   *k*-nearest-neighbor

Since each MeTA application takes an index as input, all processed data is
interchangeable between all the components. This also gives a great advantage to
classification: **MeTA supports out-of-core classification by default!** If
your dataset is small enough (like most other toolkits assume), you can
load an index into a `dataset` object to keep it all in memory without
sacrificing any speed.

Index usage is explained in much more detail in the [Search
Tutorial]({{site.baseurl}}/search-tutorial.html), though it might make more sense
to read the about [Filters and
Analyzers]({{site.baseurl}}/analyzers-filters-tutorial.html) first.

## Corpus input formats

In the main toml configuration file, there are two parameters to specify a
corpus:

{% highlight toml %}
dataset = "dataset-name"
corpus = "name.toml"
{% endhighlight %}

The `dataset` key names a folder under the path `prefix` where the corpus
data resides. The key `corpus` is the path to the corpus configuration
file, relative to the corpus path.  Usually, we simply name the corpus
config file by the corpus input format type.  For example, if we are using
a line corpus (described below), we would say `corpus = "line.toml"`. The
file `line.toml` (or whatever it's called) contains corpus-specific
settings. Below is an example for a line-corpus:

{% highlight toml %}
# file: line.toml
type = "line-corpus"
encoding = "utf-8" # optional; this is the default
num-docs = 1000    # required for gz-corpus, optional for others
metadata = [{name = "path", type = "string"}]    # metadata explained later
{% endhighlight %}

There are currently four corpus input formats (and you can find examples of all
of these in the data directory for your MeTA checkout once you have built
the toolkit at least once):

 - `line_corpus`: each dataset consists of one or two files:
   * `corpusname.dat`: each document appears on one line.
   * `corpusname.dat.labels`: optional file that includes the class or label of
      the document on each line, again corresponding to the order in
      `corpusname.dat`. These are the labels that are used for the
      classification tasks.
 - `gz_corpus`: similar to `line_corpus`, but each of its files and
    metadata files are compressed using gzip compression:
    * `corpusname.dat.gz`: compressed version of `corpusname.dat`
    * `corpusname.dat.labels.gz`: compressed version of
      `corpusname.dat.labels`
    * In the `gz.toml` file, you will need to include the `num-docs` parameter.
 - `file_corpus`: each document is its own file. Your corpus configuration
    file (typically `file.toml`) should specify a `list` key to point MeTA
    at a list of all of the documents in your corpus. This file,
    `list-full-corpus.txt` (where list is whatever you set the `list` key
    to in your configuration file; typically just the name of the corpus
    again) contains (on each line) a required label for each document
    followed by the path to the file on disk (separated by a tab or a
    space). If the documents don't have specified class labels, just
    precede the line with `[none]` or similar.
 - `libsvm_corpus`: if only being used for classification, MeTA can also take
   libsvm-formatted input to create a `forward_index`.

The main toml configuration file has a few other options related to indexing.

{% highlight toml %}
uninvert = false          # default
indexer-ram-budget = 1024 # **estimated** total RAM budget for indexing in MB
                          # always set this lower than your physical RAM!
{% endhighlight %}

The key `uninvert` (if set to true) will create a `forward_index` by uninverting
an `inverted_index`. This is useful to set to true if you will be using both
indexes. If you'll only do classification or topic modeling, setting this to
false will not create an inverted index unless one is explicitly used.

The `indexer-ram-budget` is *approximately* how much total memory is
allocated during the indexing process. If there are four threads and
`indexer-ram-budget = 4096`, then each thread will store 1GB of processed
data before it flushes it to disk. Note that this limit is currently
heuristic-based and thus is not a hard limit (though the system does its
best to adhere to it).

## Metadata

The file `metadata.dat` may optionally accompany a corpus. The `metadata`
parameter in the corpus config file explains the schema of the file. Each line
in `metadata.dat` corresponds to the ordered documents in the corpus, and values
are tab-separated. Below is a simple `line.toml` corpus config file and
`metadata.dat` file.

{% highlight toml %}
# file: line.toml
type = "line"
num-docs = 4
metadata = [{name = "path", type = "string"},
            {name = "published", type = "uint"},
            {name = "conference", type = "string"}]
{% endhighlight %}

Below are the contents of `metadata.dat`:

~~~
papers/acl-short.txt	2015	ACL
papers/sigir-submission.txt	2016	SIGIR
papers/sigir-accepted.txt	2015	SIGIR
old-papers/nips.txt	2007	NIPS
~~~

Possible types are `int`, `uint`, `double`, and `string`. Note that the example
`path` variable isn't necessarily a path to an actual file (since this is a line
corpus), but it may be useful to the user to remember where the original files
came from.

To access the metadata of an indexed document, simply call

{% highlight cpp %}
doc_id d_id = // ...
auto mdata = idx->metadata(d_id);
std::cout << *mdata.get<std::string>("field-name") << std::endl;
{% endhighlight %}

## Datasets

There are several public datasets that we've converted to the `line_corpus`
format:

 - [20newsgroups](https://meta-toolkit.org/data/20newsgroups.tar.gz), originally
   from [here](http://qwone.com/~jason/20Newsgroups/).
 - [IMDB Large Movie Review Dataset](https://meta-toolkit.org/data/imdb.tar.gz),
   originally from [here](http://ai.stanford.edu/~amaas/data/sentiment/).
 - Any [libsvm-formatted
   dataset](http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/) can be
   used to create a `forward_index`.

## Tokenizers, Filters, and Analyzers

MeTA processes data with a system of tokenizers, filters and analyzers before it
is indexed.

**Tokenizers** come first. They define how to split a document's content into
tokens. Some examples are:

 - `icu_tokenizer`: converts documents into streams of tokens by following the
   unicode standards for sentence and word segmentation.<commit_msg>Add more examples of corpus configuration files.<commit_after>---
title: MeTA System Overview
layout: page
category: tut
order: 1
---

Our task is to improve upon and complement the current body of open source
machine learning and information retrieval software. The existing environment of
this open source software tends to be quite fragmented.

There is rarely a single location for a wide variety of algorithms; a good
example of this is the [LIBLINEAR](http://www.csie.ntu.edu.tw/~cjlin/liblinear/)
software package for SVMs. While this is the most cited of the open source
implementations of linear SVMs, it focuses solely on kernel-less methods. If
presented with a nonlinear classification problem, one would be forced to find a
different software package that supports kernels (such as the same authors'
[LIBSVM](http://www.csie.ntu.edu.tw/~cjlin/libsvm/) package). This places an
undue burden on the researchers---not only are they required to have a detailed
understanding of the research problem at hand, but they are now forced to
understand this fragmented nature of the open source software community, find
the appropriate tools in this mishmash of implementations, and compile and
configure the appropriate tool.

Even when this is all done, there is the problem of data formatting---it is
unlikely that the tools have standardized upon a single input format, so a
certain amount of "data munging" is now required. This all detracts from the
actual research task at hand, which has a marked impact on the speed of
discovery.

**We want to address these issues.** In particular, we want to provide a
unifying framework for text indexing and analysis methods, allowing researchers
to quickly run controlled experiments. We want to modularize the feature
generation, instance representation, data storage formats, and algorithm
implementations; this will allow for researchers to make seamless transitions
along any of these dimensions with minimal effort.

## How does MeTA store data?

All processed data is stored in a `disk_index`. Currently, we have two
index types: `forward_index` and `inverted_index`. The former is keyed by
document IDs, and the latter is keyed by term IDs.

 - `forward_index` is used for applications such as topic modeling and
   most classification tasks
 - `inverted_index` is used to create search engines, or do classification with
   *k*-nearest-neighbor

Since each MeTA application takes an index as input, all processed data is
interchangeable between all the components. This also gives a great advantage to
classification: **MeTA supports out-of-core classification by default!** If
your dataset is small enough (like most other toolkits assume), you can
load an index into a `dataset` object to keep it all in memory without
sacrificing any speed.

Index usage is explained in much more detail in the [Search
Tutorial]({{site.baseurl}}/search-tutorial.html), though it might make more sense
to read the about [Filters and
Analyzers]({{site.baseurl}}/analyzers-filters-tutorial.html) first.

The main toml configuration file has a few options related to indexing.

{% highlight toml %}
uninvert = false          # default
indexer-ram-budget = 1024 # **estimated** total RAM budget for indexing in MB
                          # always set this lower than your physical RAM!
{% endhighlight %}

The key `uninvert` (if set to true) will create a `forward_index` by uninverting
an `inverted_index`. This is useful to set to true if you will be using both
indexes. If you'll only do classification or topic modeling, setting this to
false will not create an inverted index unless one is explicitly used.

The `indexer-ram-budget` is *approximately* how much total memory is
allocated during the indexing process. If there are four threads and
`indexer-ram-budget = 4096`, then each thread will store 1GB of processed
data before it flushes it to disk. Note that this limit is currently
heuristic-based and thus is not a hard limit (though the system does its
best to adhere to it).

## Corpus input formats
In the main toml configuration file, there are two parameters to specify a
corpus:

{% highlight toml %}
dataset = "dataset-name"
corpus = "name.toml"
{% endhighlight %}

The `dataset` key names a folder under the path `prefix` where the corpus
data resides. The key `corpus` is the path to the corpus configuration
file, relative to the corpus path.  Usually, we simply name the corpus
config file by the corpus input format type.  For example, if we are using
a line corpus (described below), we would say `corpus = "line.toml"`. The
file `line.toml` (or whatever it's called) contains corpus-specific
settings. Refer to the corpus-specific documentation below for what it
should contain.

There are currently four corpus input formats (and you can find examples of all
of these in the data directory for your MeTA checkout once you have built
the toolkit at least once).

### Line Corpus
A `line_corpus` consists of one or two files:

 - `line.toml`: the corpus configuration file
 - `corpusname.dat`: each document appears on one line.
 - `corpusname.dat.labels`: optional file that includes the class or label of
   the document on each line, again corresponding to the order in
   `corpusname.dat`. These are the labels that are used for the
   classification tasks.

Below is an example of what `line.toml` might look like:

{% highlight toml %}
type = "line-corpus"
encoding = "utf-8" # optional; this is the default
num-docs = 1000    # optional
metadata = [{name = "path", type = "string"}]    # metadata explained later
{% endhighlight %}

### GZipped Line Corpus
A `gz_corpus` is simlar to a `line_corpus`, but each of its files are
compressed using gzip compression:

 - `gz.toml`: the corpus configuration file
 - `corpusname.dat.gz`: a compressed version of `corpusname.dat`
 - `corpusname.dat.labels.gz`: a compressed version of
   `corpusname.dat.labels`

Below is an example of what `gz.toml` might look like:

{% highlight toml %}
type = "gz-corpus"
encoding = "utf-8" # optional; this is the default
num-docs = 1000    # required
metadata = [{name = "path", type = "string"}]    # metadata explained later
{% endhighlight %}

### File Corpus
In a `file_corpus`, each document is its own file. Your corpus
configuration file (typically `file.toml`) should specify a `list` key to
point MeTA at a list of all of the documents in your corpus. This file,
`list-full-corpus.txt` (where list is whatever you set the `list` key to in
your configuration file; typically just the name of the corpus again)
contains (on each line) a required label for each document followed by the
path to the file on disk (separated by a tab or a space). If the documents
don't have specified class labels, just precede the line with `[none]` or
similar.

For example, your `file.toml` might look like:

{% highlight toml %}
type = "file-corpus"
list = "dataset"
encoding = "utf-8" # optional; this is the default
num-docs = 1000    # optional
metadata = [{name = "path", type = "string"}]    # metadata explained later
{% endhighlight %}

and the file `dataset-full-corpus.txt` might look like:

~~~
class-label	relative/path/to/document01.txt
class-label2	relative/path/to/document02.txt
class-label	relative/path/to/document03.txt
class-label3	relative/path/to/document04.txt
~~~

### LIBSVM Corpus
If only being used for classification, MeTA can also take libsvm-formatted
input to create a `forward_index`. A `libsvm_corpus` contains the following
files:

 - `libsvm.toml`: the corpus configuration file
 - `corpus.dat`: the LIBSVM-formatted data file

The `libsvm.toml` file is very straightforward:

{% highlight toml %}
type = "libsvm-corpus"
{% endhighlight %}

## Metadata

The file `metadata.dat` may optionally accompany a corpus. The `metadata`
parameter in the corpus config file explains the schema of the file. Each line
in `metadata.dat` corresponds to the ordered documents in the corpus, and values
are tab-separated. Below is a simple `line.toml` corpus config file and
`metadata.dat` file.

{% highlight toml %}
# file: line.toml
type = "line"
num-docs = 4
metadata = [{name = "path", type = "string"},
            {name = "published", type = "uint"},
            {name = "conference", type = "string"}]
{% endhighlight %}

Below are the contents of `metadata.dat`:

~~~
papers/acl-short.txt	2015	ACL
papers/sigir-submission.txt	2016	SIGIR
papers/sigir-accepted.txt	2015	SIGIR
old-papers/nips.txt	2007	NIPS
~~~

Possible types are `int`, `uint`, `double`, and `string`. Note that the example
`path` variable isn't necessarily a path to an actual file (since this is a line
corpus), but it may be useful to the user to remember where the original files
came from.

To access the metadata of an indexed document, simply call

{% highlight cpp %}
doc_id d_id = // ...
auto mdata = idx->metadata(d_id);
std::cout << *mdata.get<std::string>("field-name") << std::endl;
{% endhighlight %}

If you have no metadata, simply do not specify a `metadata` key in your
corpus configuration file.

## Datasets

There are several public datasets that we've converted to the `line_corpus`
format:

 - [20newsgroups](https://meta-toolkit.org/data/20newsgroups.tar.gz), originally
   from [here](http://qwone.com/~jason/20Newsgroups/).
 - [IMDB Large Movie Review Dataset](https://meta-toolkit.org/data/imdb.tar.gz),
   originally from [here](http://ai.stanford.edu/~amaas/data/sentiment/).
 - Any [libsvm-formatted
   dataset](http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/) can be
   used to create a `forward_index`.

## Tokenizers, Filters, and Analyzers

MeTA processes data with a system of tokenizers, filters and analyzers before it
is indexed.

**Tokenizers** come first. They define how to split a document's content into
tokens. Some examples are:

 - `icu_tokenizer`: converts documents into streams of tokens by following the
   unicode standards for sentence and word segmentation.
{'fraction_non_alphanumeric': 0.07870815955512779, 'fraction_numerical': 0.005133140840551813, 'mean_word_length': 4.221384701284199, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 7, 'https://': 4, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1316185', 'n_tokens_mistral': 5386, 'n_tokens_neox': 5023, 'n_words': 2536}