<commit_before>Stockdb
=======


This library is a storage for Stock Exchange quotes.

It is an append-only online-compress storage, that supports failover, indexing and simple lookups of stored quotes.

It can compress 400 000 daily quotes into 4 MB of disk storage.

Usage
=====

First install and compile it. Include it as a rebar dependency.

It can be used either as an appender, either as reader. You cannot mix these two modes now.

Appender
------

    {ok, Appender} = stockdb:open_appender('NASDAQ.AAPL', "2012-01-15", [{depth, 2}]),
    {ok, Appender1} = stockdb:append({md, 1326601810453, [{450.1,100},{449.56,1000}], [{452.43,20},{454.15,40}]}),
    stockdb:close(Appender1).


Now lets explain, what is happening.

* Open appender. Stock name should be a symbol, date should either erlang date `{YYYY,MM,DD}`, either a string `"YYYY-MM-DD"`.
* Don't forget to specify proper depth. If you skip it, default depth is 1 and you will save only best bid and best ask
* Specify also `{scale, 1000}` option, if you want to store quotes with precision less than 1 cent. Stockdb stores your prices as int: `round(Price*Scale)`
* Now append market data.
* Market data is following: ```{md, UtcMilliseconds, [{L1BidPrice,L1BidSize},{L2BidPrice,L2BidSize}..], [{L1AskPrice,L2AskSize}..]}```
* You can include ```-include_lib("stockdb/include/stockdb.hrl").``` to use ```#md{}``` and ```#trade{}``` records

Now take a look at db/stock folder. There you can see new file `db/stock/NASDAQ.AAPL-2012-01-15.stock` and now you can read back stocks from it.


Reader
-----

The most simple way is just to read all daily events to replaying them

    {ok, Events} = stockdb:events('NASDAQ.AAPL', "2012-01-15").

But there are possible more enhanced ways of limiting amount of loaded data.


Iterator
-----

If you need something more complex than just getting all data from DB for stock/date pair, you can use iterators.
Iterator is database opened for read-only with (optionally) filters applied on it.
You can read iterator's events one-by-one, saving memory by not keeping extracted data.

Basic iterator is created as follows:

    {ok, Iterator} = stockdb:init_reader('NASDAQ.AAPL', {2012, 8, 7}, []).

Here first argument is stock, second is date, third is list of filters (empty for basic case).

You can read events one-by-one using `stockdb:read_event/1` function:

    {Event1, Iterator1} = stockdb:read_event(Iterator),
    {Event2, Iterator2} = stockdb:read_event(Iterator1).

Also, you can call `stockdb:events(Iterator)` to get all events from it.


Iterator filters
----------------

Iterator filters currently may be `{range, Start, End}` or `{filter, FilterFun, FilterState0}`.
`FilterFun` may be function name from module `stockdb_filters` (currently only `candle`) or
any function with arity 2 which returns list of emitted events. Filter must accept events from
previous filter (or `#md{}` and `trade{}` from DB) and `eof` to handle end of underlying source.
Return value is tuple with list of emitted events on first place and next state on second.

For example, this simple function drops every second event:

    FilterFun = fun
        (eof, _State) -> {[], eof};
        (Event, true) -> {[Event], false};
        (_Event, _Other) -> {[], true}
      end.

And, we can see it is working:

    25> length(stockdb:events(Iterator)).          
    20703
    27> {ok, FIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{filter, FilterFun, false}]),
    27> length(stockdb:events(FIterator)).
    10351

To use pre-defined filters you can just specify filter name:

    28> {ok, CIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{filter, candle, [{period, 120000}]}]),
    28> length(stockdb:events(CIterator)).
    2242

StockDB index is optimized for fast timestamp seeking, so you can use `{range, Start, End}` pseudo-filter. Start and End
are both millisecond timestamps or erlang-style `{HH, MM, SS}` tuples (tuples will work only over DB source, not over other iterator):

    31> {ok, RIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{range, {14,0,0}, {15,0,0}}]),
    31> length(stockdb:events(RIterator)).
    5139

Of course, you may specify multiple filters:

    32> {ok, RFCIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{range, {14,0,0}, {15,0,0}}, {filter, FilterFun, false}, {filter, candle, [{period, 120000}]}]),
    32> length(stockdb:events(RFCIterator)).
    372                                   

Also, iterators may cascade:

    35> {ok, RIterator_F} = stockdb:init_reader(RIterator, [{filter, FilterFun, false}]),
    35> {ok, RIterator_F_C} = stockdb:init_reader(RIterator_F, [{filter, candle, [{period, 120000}]}]),
    35> length(stockdb:events(RIterator_F_C)).
    372
    36> stockdb:events(RIterator_F_C) == stockdb:events(RFCIterator).
    true
   <commit_msg>stockdb: readme: about open_read/2<commit_after>Stockdb
=======


This library is a storage for Stock Exchange quotes.

It is an append-only online-compress storage, that supports failover, indexing and simple lookups of stored quotes.

It can compress 400 000 daily quotes into 4 MB of disk storage.

Usage
=====

First install and compile it. Include it as a rebar dependency.

It can be used either as an appender, either as reader. You cannot mix these two modes now.

Writing database: Appender
==========================

Typical workflow when appending data to DB:

    {ok, Appender} = stockdb:open_appender('NASDAQ.AAPL', "2012-01-15", [{depth, 2}]),
    {ok, Appender1} = stockdb:append({md, 1326601810453, [{450.1,100},{449.56,1000}], [{452.43,20},{454.15,40}]}),
    stockdb:close(Appender1).


Now lets explain, what is happening.

* Open appender. Stock name should be a symbol, date should either erlang date `{YYYY,MM,DD}`, either a string `"YYYY-MM-DD"`.
* Don't forget to specify proper depth. If you skip it, default depth is 1 and you will save only best bid and best ask
* Specify also `{scale, 1000}` option, if you want to store quotes with precision less than 1 cent. Stockdb stores your prices as int: `round(Price*Scale)`
* Now append market data.
* Market data is following: ```{md, UtcMilliseconds, [{L1BidPrice,L1BidSize},{L2BidPrice,L2BidSize}..], [{L1AskPrice,L2AskSize}..]}```
* You can include ```-include_lib("stockdb/include/stockdb.hrl").``` to use ```#md{}``` and ```#trade{}``` records

Now take a look at db/stock folder. There you can see new file `db/stock/NASDAQ.AAPL-2012-01-15.stock` and now you can read back stocks from it.


Reading database
================

Read whole DB
-------------

The most simple way is just to read all daily events to replaying them

    {ok, Events} = stockdb:events('NASDAQ.AAPL', "2012-01-15").

But there are possible more enhanced ways of limiting amount of loaded data.


Iterator
--------

If you need something more complex than just getting all data from DB for stock/date pair, you can use iterators.
Iterator is database opened for read-only with (optionally) filters applied on it.
You can read iterator's events one-by-one, saving memory by not keeping extracted data.

Basic iterator is created as follows:

    {ok, Iterator} = stockdb:init_reader('NASDAQ.AAPL', {2012, 8, 7}, []).

Here first argument is stock, second is date, third is list of filters (empty for basic case).

You can read events one-by-one using `stockdb:read_event/1` function:

    {Event1, Iterator1} = stockdb:read_event(Iterator),
    {Event2, Iterator2} = stockdb:read_event(Iterator1).

When there are no more events, `eof` event is returned. Make sure your code handles it well!

Also, you can call `stockdb:events(Iterator)` to get all events from it.


Iterator filters
----------------

Iterator filters currently may be `{range, Start, End}` or `{filter, FilterFun, FilterState0}`.
`FilterFun` may be function name from module `stockdb_filters` (currently only `candle`) or
any function with arity 2 which returns list of emitted events. Filter must accept events from
previous filter (or `#md{}` and `trade{}` from DB) and `eof` to handle end of underlying source.
Return value is tuple with list of emitted events on first place and next state on second.

For example, this simple function drops every second event:

    FilterFun = fun
        (eof, _State) -> {[], eof};
        (Event, true) -> {[Event], false};
        (_Event, _Other) -> {[], true}
      end.

And, we can see it is working:

    25> length(stockdb:events(Iterator)).          
    20703
    27> {ok, FIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{filter, FilterFun, false}]),
    27> length(stockdb:events(FIterator)).
    10351

To use pre-defined filters you can just specify filter name:

    28> {ok, CIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{filter, candle, [{period, 120000}]}]),
    28> length(stockdb:events(CIterator)).
    2242

StockDB index is optimized for fast timestamp seeking, so you can use `{range, Start, End}` pseudo-filter. Start and End
are both millisecond timestamps or erlang-style `{HH, MM, SS}` tuples (tuples will work only over DB source, not over other iterator):

    31> {ok, RIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{range, {14,0,0}, {15,0,0}}]),
    31> length(stockdb:events(RIterator)).
    5139

Of course, you may specify multiple filters:

    32> {ok, RFCIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{range, {14,0,0}, {15,0,0}}, {filter, FilterFun, false}, {filter, candle, [{period, 120000}]}]),
    32> length(stockdb:events(RFCIterator)).
    372                                   

Also, iterators may cascade:

    35> {ok, RIterator_F} = stockdb:init_reader(RIterator, [{filter, FilterFun, false}]),
    35> {ok, RIterator_F_C} = stockdb:init_reader(RIterator_F, [{filter, candle, [{period, 120000}]}]),
    35> length(stockdb:events(RIterator_F_C)).
    372
    36> stockdb:events(RIterator_F_C) == stockdb:events(RFCIterator).
    true


Self-sufficient read-only state
-------------------------------

Function `stockdb:init_reader/3` currently accesses file directly. If you have distributed setup, it will fail. Stockdb is able to bypass this by using a lower-level `stockdb:open_read/2`.
`open_read/2` returns in-memory read-only database state with full buffer and file descriptor closed. Actually, `stockdb:init_reader/3` first opens DB using `stockdb:open_read/2` and then calls `stockdb:init_reader/2` on it. So does `stockdb:events/2`. You can do the same:

    42> {ok, S} = stockdb:open_read('NASDAQ.AAPL', {2012, 8,7}),
    42> {ok, Iterator} = stockdb:init_reader(S, []),
    42> stockdb:events(S) == stockdb:events(Iterator).
    true

Note that we still use the same Iterator which matches perfectly. `stockdb:init_reader(S, [])` can be called when original file is unavailable allowing to minimize network load when DB content is needed on other node.
{'fraction_non_alphanumeric': 0.13409153444006236, 'fraction_numerical': 0.04200678712281024, 'mean_word_length': 4.048148148148148, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3415684', 'n_tokens_mistral': 3744, 'n_tokens_neox': 3401, 'n_words': 1292}