<commit_before># Library NPGe

This document is for developers and advanced users only.
End users should read [README](README.md) first.

## Installation

Requirements:

  * [Lua][lua] >= 5.1 or [LuaJIT][luajit]
  * [LuaRocks][luarocks]
  * [C++ compiler][gcc]
  * [Boost][boost]
  * NCBI [blast+][blast]

[lua]: http://www.lua.org/
[luajit]: http://luajit.org/
[luarocks]: https://luarocks.org/
[gcc]: https://gcc.gnu.org/
[boost]: http://www.boost.org/
[blast]: http://www.ncbi.nlm.nih.gov/BLAST/

```bash
$ git clone https://github.com/npge/lua-npge
$ cd lua-npge
$ sudo luarocks make
```

To install the library for current user only, use the following
command instead:

```bash
$ luarocks make --local
```

When installing for current user only, use the following
command to "enable" the library in Lua:

```bash
$ eval `luarocks path`
$ export PATH=~/.luarocks/bin:$PATH
```

You should repeat these command each time before using
npge. Otherwise you can append them to `~/.bashrc`.

Other important thing on Linux is to set environment
variable `LD_PRELOAD` to value
`/lib/x86_64-linux-gnu/libpthread.so.0`:
```bash
$ export LD_PRELOAD=/lib/x86_64-linux-gnu/libpthread.so.0
```
Otherwise it can crash in multithreaded mode.
(Not checked: for 32-bit Linux it should be
`/lib/i386-linux-gnu/libpthread.so.0`.)

Test if Lua can load NPGe:

```bash
$ lua -e 'require "npge"'
```

This command should produce no output in case of success.

On linux we recommend to use `luaprompt` as Lua interpreter:

```bash
$ sudo aptitude install libncurses5-dev
$ sudo luarocks install luaprompt \
    HISTORY_LIBDIR=/usr/lib/x86_64-linux-gnu/ \
    READLINE_LIBDIR=/usr/lib/x86_64-linux-gnu/
$ luap
>
```

Now you are ready to use NPGe:

```lua
> npge = require 'npge'
```

## Data structures provided by NPGe

See also [specs](spec/model).

Data structures provide low level access to NPG.
All data structures are immutable: no changes after creation.
Functions creating new instances of NPG data structures
live in table `npge.model`. Create an alias for it:

```lua
model = npge.model
```

Short description of NPG data structures:

 - `Sequence` stores string representing genome sequence
    (e.g., chromosome, plasmid or contig).
    It can represent temporary generated sequence
    (e.g., consensus of a block).
 - `Fragment` points to some fragment of Sequence
    (direct or reverse).
 - `Block` is a collection of fragments + alignment.
 - `BlockSet` is a collection of Sequences + a collection of
    Blocks. Fragments of the blocks belong to the sequences.

### Sequence

`Sequence` stores string representing genome sequence
(e.g., chromosome, plasmid or contig).
It can represent temporary generated sequence
(e.g., consensus of a block).

Sequence has the following properties:

  * `name` (e.g., "BRUAB&chr1&c", "consensus0001")
  * `text` (e.g., "ATTCCC")
  * `description` (e.g., "Brucella")

To create a sequence, call the following function:

```lua
>  seq = model.Sequence("BRUAB&chr1&c", "ATTCCC", "Brucella")
```
Text can contain gaps, they are ignored.
All letters are uppercased.
Letters from [IUPAC code for incomplete nucleic acid
specification][n] ("N", "R", "Y", "M", "K", "S", "W",
"H", "B", "V", "D") are replaced with "N".
All unknown letters are skipped.

Description is optional. Default description is "".

[n]: http://www.chem.qmul.ac.uk/iubmb/misc/naseq.html

Methods:

```lua
>  seq:name()
"BRUAB&chr1&c"

>  seq:text()
"ATTCCC"

>  seq:length()
6

>  seq:description()
"Brucella"

>  seq:genome()
"BRUAB"

>  seq:chromosome()
"chr1"

>  seq:circular()
true

>  seq:sub(0, 2)
"ATT"
```

Methods `genome()`, `chromosome()` and `circular()` work if
only the name's format is "GENOME&CHROMOSOME&CIRCULARITY",
because they get data from the name. Genome or chromosome
can't be empty in this case.

Sequences can be compared with operator `==`. Comparison
ignores all properties but sequence name.

Sequence has nice string representation:
```
>  tostring(seq)
"Sequence BRUAB&chr1&c of length 6"
```

### Fragment

Fragment points to some fragment of Sequence
(direct or reverse).

Fragment has the following properties:

  * `sequence` -- a pointer to Sequence object
  * `start` -- 0-based integer
  * `stop` -- 0-based integer
  * `ori` -- `1` or `-1`

Creation:
```lua
>  start = 0
>  stop = 2
>  ori = 1
>  fr = model.Fragment(seq, start, stop, ori)
>  fr
Fragment BRUAB&chr1&c_0_2_1 of length 3
```

Reverse fragment:
```lua
>  start = 2
>  stop = 0
>  ori = -1
>  fr = model.Fragment(seq, start, stop, ori)
>  fr
Fragment BRUAB&chr1&c_2_0_-1 of length 3
```

A fragment crossing the "beginning" of a circular sequence
is called *parted fragment*:
```lua
>  seq = model.Sequence("BRUAB&chr1&c", "ATTCCC", "Brucella")
>  start = 4
>  stop = 1
>  ori = 1
>  fr = model.Fragment(seq, start, stop, ori)
>  fr
Fragment BRUAB&chr1&c_4_1_1 of length 4 (parted)
```

Methods:

```lua
>  fr
Fragment BRUAB&chr1&c_4_1_1 of length 4 (parted)

>  fr:sequence()
Sequence BRUAB&chr1&c of length 6

>  fr:sequence() == seq
true

>  fr:start()
4

>  fr:stop()
1

>  fr:ori()
1

>  fr:id()
"BRUAB&chr1&c_4_1_1"

>  fr:text()
"CCAT"

>  fr:parted()
true
```

Method `:parts()` returns two fragments. First of them has
start position of the parent fragment. This method can be
applied only to a parted fragment.

```lua
>  seq = model.Sequence("BRUAB&chr1&c", "ATTCCC")
>  parted = model.Fragment(seq, 4, 1, 1)
>  part1, part2 = parted:parts() -- only for parted fragments

>  part1
Fragment BRUAB&chr1&c_4_5_1 of length 2
>  part1:text()
"CC"

>  part2
Fragment BRUAB&chr1&c_0_1_1 of length 2
>  part2:text()
"AT"
```

Fragments can be compared with operator `==`. Comparison
succeeds iff all properties of fragments are equal.

```lua
>  part1 == model.Fragment(seq, 4, 5, 1)
true

>  part1 == part2
false
```

Fragments can be compared with operators `<`, `>`, `<=`, `>=`.
Fragments are compared as tuples
(sequence.name, min-pos, max-pos, ori). Min-pos is minimum of
start and stop. Max-pos is maximum of start and stop.
*Note*. For parted fragments the same definition is used.

```lua
>  part1 < part2
false
```

Counting shared positions of two fragments:

```lua
-- create fragments
>  seq = model.Sequence("BRUAB&chr1&c", "ATTCCC")
>  parted = model.Fragment(seq, 4, 1, 1)
>  fr2 = model.Fragment(seq, 1, 4, 1)
>  part1, part2 = parted:parts() -- only for parted fragments

>  parted:common(part1)
2

>  part1:common(parted)
2

>  part1:common(part2)
0

-- parted fragment may have two overlapping areas
-- they are counted correctly
>  parted:common(fr2)
2
```

### Block

Block is a collection of fragments + alignment.

There are two methods to create a block:

```lua
>  bl1 = model.Block({part1, part2})

>  bl1:text(part1)
"CC"

>  bl1:text(part2)
"AT"

>  bl1
Block of 2 fragments, length 2
```

```lua<commit_msg>API.md: move a definition closer to code using it<commit_after># Library NPGe

This document is for developers and advanced users only.
End users should read [README](README.md) first.

## Installation

Requirements:

  * [Lua][lua] >= 5.1 or [LuaJIT][luajit]
  * [LuaRocks][luarocks]
  * [C++ compiler][gcc]
  * [Boost][boost]
  * NCBI [blast+][blast]

[lua]: http://www.lua.org/
[luajit]: http://luajit.org/
[luarocks]: https://luarocks.org/
[gcc]: https://gcc.gnu.org/
[boost]: http://www.boost.org/
[blast]: http://www.ncbi.nlm.nih.gov/BLAST/

```bash
$ git clone https://github.com/npge/lua-npge
$ cd lua-npge
$ sudo luarocks make
```

To install the library for current user only, use the following
command instead:

```bash
$ luarocks make --local
```

When installing for current user only, use the following
command to "enable" the library in Lua:

```bash
$ eval `luarocks path`
$ export PATH=~/.luarocks/bin:$PATH
```

You should repeat these command each time before using
npge. Otherwise you can append them to `~/.bashrc`.

Other important thing on Linux is to set environment
variable `LD_PRELOAD` to value
`/lib/x86_64-linux-gnu/libpthread.so.0`:
```bash
$ export LD_PRELOAD=/lib/x86_64-linux-gnu/libpthread.so.0
```
Otherwise it can crash in multithreaded mode.
(Not checked: for 32-bit Linux it should be
`/lib/i386-linux-gnu/libpthread.so.0`.)

Test if Lua can load NPGe:

```bash
$ lua -e 'require "npge"'
```

This command should produce no output in case of success.

On linux we recommend to use `luaprompt` as Lua interpreter:

```bash
$ sudo aptitude install libncurses5-dev
$ sudo luarocks install luaprompt \
    HISTORY_LIBDIR=/usr/lib/x86_64-linux-gnu/ \
    READLINE_LIBDIR=/usr/lib/x86_64-linux-gnu/
$ luap
>
```

Now you are ready to use NPGe:

```lua
> npge = require 'npge'
```

## Data structures provided by NPGe

See also [specs](spec/model).

Data structures provide low level access to NPG.
All data structures are immutable: no changes after creation.
Functions creating new instances of NPG data structures
live in table `npge.model`. Create an alias for it:

```lua
model = npge.model
```

Short description of NPG data structures:

 - `Sequence` stores string representing genome sequence
    (e.g., chromosome, plasmid or contig).
    It can represent temporary generated sequence
    (e.g., consensus of a block).
 - `Fragment` points to some fragment of Sequence
    (direct or reverse).
 - `Block` is a collection of fragments + alignment.
 - `BlockSet` is a collection of Sequences + a collection of
    Blocks. Fragments of the blocks belong to the sequences.

### Sequence

`Sequence` stores string representing genome sequence
(e.g., chromosome, plasmid or contig).
It can represent temporary generated sequence
(e.g., consensus of a block).

Sequence has the following properties:

  * `name` (e.g., "BRUAB&chr1&c", "consensus0001")
  * `text` (e.g., "ATTCCC")
  * `description` (e.g., "Brucella")

To create a sequence, call the following function:

```lua
>  seq = model.Sequence("BRUAB&chr1&c", "ATTCCC", "Brucella")
```
Text can contain gaps, they are ignored.
All letters are uppercased.
Letters from [IUPAC code for incomplete nucleic acid
specification][n] ("N", "R", "Y", "M", "K", "S", "W",
"H", "B", "V", "D") are replaced with "N".
All unknown letters are skipped.

Description is optional. Default description is "".

[n]: http://www.chem.qmul.ac.uk/iubmb/misc/naseq.html

Methods:

```lua
>  seq:name()
"BRUAB&chr1&c"

>  seq:text()
"ATTCCC"

>  seq:length()
6

>  seq:description()
"Brucella"

>  seq:genome()
"BRUAB"

>  seq:chromosome()
"chr1"

>  seq:circular()
true

>  seq:sub(0, 2)
"ATT"
```

Methods `genome()`, `chromosome()` and `circular()` work if
only the name's format is "GENOME&CHROMOSOME&CIRCULARITY",
because they get data from the name. Genome or chromosome
can't be empty in this case.

Sequences can be compared with operator `==`. Comparison
ignores all properties but sequence name.

Sequence has nice string representation:
```
>  tostring(seq)
"Sequence BRUAB&chr1&c of length 6"
```

### Fragment

Fragment points to some fragment of Sequence
(direct or reverse).

Fragment has the following properties:

  * `sequence` -- a pointer to Sequence object
  * `start` -- 0-based integer
  * `stop` -- 0-based integer
  * `ori` -- `1` or `-1`

Creation:
```lua
>  start = 0
>  stop = 2
>  ori = 1
>  fr = model.Fragment(seq, start, stop, ori)
>  fr
Fragment BRUAB&chr1&c_0_2_1 of length 3
```

Reverse fragment:
```lua
>  start = 2
>  stop = 0
>  ori = -1
>  fr = model.Fragment(seq, start, stop, ori)
>  fr
Fragment BRUAB&chr1&c_2_0_-1 of length 3
```

A fragment crossing the "beginning" of a circular sequence
is called *parted fragment*:
```lua
>  seq = model.Sequence("BRUAB&chr1&c", "ATTCCC", "Brucella")
>  start = 4
>  stop = 1
>  ori = 1
>  fr = model.Fragment(seq, start, stop, ori)
>  fr
Fragment BRUAB&chr1&c_4_1_1 of length 4 (parted)
```

Methods:

```lua
>  fr
Fragment BRUAB&chr1&c_4_1_1 of length 4 (parted)

>  fr:sequence()
Sequence BRUAB&chr1&c of length 6

>  fr:sequence() == seq
true

>  fr:start()
4

>  fr:stop()
1

>  fr:ori()
1

>  fr:id()
"BRUAB&chr1&c_4_1_1"

>  fr:text()
"CCAT"

>  fr:parted()
true
```

Method `:parts()` returns two fragments. First of them has
start position of the parent fragment. This method can be
applied only to a parted fragment.

```lua
>  seq = model.Sequence("BRUAB&chr1&c", "ATTCCC")
>  parted = model.Fragment(seq, 4, 1, 1)
>  part1, part2 = parted:parts() -- only for parted fragments

>  part1
Fragment BRUAB&chr1&c_4_5_1 of length 2
>  part1:text()
"CC"

>  part2
Fragment BRUAB&chr1&c_0_1_1 of length 2
>  part2:text()
"AT"
```

Fragments can be compared with operator `==`. Comparison
succeeds iff all properties of fragments are equal.

```lua
>  part1 == model.Fragment(seq, 4, 5, 1)
true

>  part1 == part2
false
```

Fragments can be compared with operators `<`, `>`, `<=`, `>=`.
Fragments are compared as tuples
(sequence.name, min-pos, max-pos, ori). Min-pos is minimum of
start and stop. Max-pos is maximum of start and stop.
*Note*. For parted fragments the same definition is used.

```lua
>  part1 < part2
false
```

Counting shared positions of two fragments:

```lua
-- create fragments
>  seq = model.Sequence("BRUAB&chr1&c", "ATTCCC")
>  parted = model.Fragment(seq, 4, 1, 1)
>  part1, part2 = parted:parts() -- only for parted fragments

>  parted:common(part1)
2

>  part1:common(parted)
2

>  part1:common(part2)
0

-- parted fragment may have two overlapping areas
-- they are counted correctly
>  fr2 = model.Fragment(seq, 1, 4, 1)
>  parted:common(fr2)
2
```

### Block

Block is a collection of fragments + alignment.

There are two methods to create a block:

```lua
>  bl1 = model.Block({part1, part2})

>  bl1:text(part1)
"CC"

>  bl1:text(part2)
"AT"

>  bl1
Block of 2 fragments, length 2
```

```lua
{'fraction_non_alphanumeric': 0.1405042996647719, 'fraction_numerical': 0.02040518874799592, 'mean_word_length': 3.9865552325581395, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 127, 'https://': 6, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '24654', 'n_tokens_mistral': 5272, 'n_tokens_neox': 4871, 'n_words': 1635}