<commit_before># Design

[Go](http://golang.org/) for the server-side ([Martini](http://martini.codegangsta.io/) for the webapp), [Redis](http://redis.io/) for temporary data, [SSDB](https://github.com/ideawu/ssdb) (Backed by [LevelDB](https://code.google.com/p/leveldb/)) for persistent data.

The initial import is still slow, I already spent a lot of time trying to improve it (it took 1+ week on a small dual-core/6GB RAM server at block 247000), and the database takes around 69GB (at block 247000). I think the minimal requirement are 100+GB HD and 4+GB RAM.

## Unconfirmed transactions

Bitcoind memory pool is synced every 500ms in Redis, along with every transactions.

The most recent memory pool sync is stored in a sorted set (with time as score, in ``btcplex:rawmempool``), allowing them to be "replayed" via SSE on the unconfirmed transactions page.
Each unconfirmed transaction is stored as JSON in a key ``btcplex:utx:%v`` (hash), the key is destroyed when it get removed from the memory pool.
The sync is performed by keeping two sets: ``btcplex:rawmempool:%v`` (unix time):

- one containing the previous state of memory pool (500ms ago)
- the current state of memory pool

The diff of the two sets is computed, and old unconfirmed transactions are removed. 

## New block

BTCplex relies on ``bitcoind`` blocknotify callback, each time the best block changes, it will be processed (via the RPC API) and immediately available. 

## Available keys in SSDB

I will try to keep an updated list of how data is stored in SSDB by data type.

### Strings

Blocks, transactions, TxIns, TxOuts are stored in JSON format (SSDB support Redis protocol but it doesn't support MULTI, so I can't use hashes if I can't retrieve multiple hashes in one call, also, some JSON objects are nested, so I stick to JSON.

- ``block:height:%v`` (height) -> Contains the hash for the given height
- ``block:%v`` (hash) -> Block data in JSON format
- ``tx:%v`` (hash) -> Transaction data in JSON format
- ``txi:%v:%v`` (hash, index) -> TxIn data in JSON format
- ``txo:%v:%v`` (hash, index) -> TxOut data in JSON format
- ``txo:%v:%v:spent`` (hash, index) -> Spent data in JSON format
- ``btcplex:utx:%v`` (hash) -> Unconfirmed transaction (with TxOuts/TxIns) in JSON format


### Hashes

BTCplex keeps one hashes per address (``addr:%v:h`` (address)) containing TotalSent, TotalReceived. 

Hash keys for address data:

- ``ts`` -> TotalSent
- ``tr`` -> TotalReceived


### Sorted Sets

BTCplex keeps three sorted set per address (``addr:%v`` (address), ``addr:%v:received`` (address), ``addr:%v:sent`` (address)) containing Tx hash for every transaction involving the address sorted by BlockTime.

It also store one sorted for each block containing transaction references sorted by index (``block:%v:txs`` (hash)).

Bitcoind memory pool is "synced" in a sorted set: ``btcplex:rawmempool``.


### Webapp

- [Martini](http://martini.codegangsta.io/)


## Backend notes<commit_msg>Updated DESIGN.md<commit_after># Design

[Go](http://golang.org/) for the server-side ([Martini](http://martini.codegangsta.io/) for the webapp), [Redis](http://redis.io/) for temporary data, [SSDB](https://github.com/ideawu/ssdb) (Backed by [LevelDB](https://code.google.com/p/leveldb/)) for persistent data.

The initial import is still slow, I already spent a lot of time trying to improve it (it took 1+ week on a small dual-core/6GB RAM server at block 247000), and the database takes around 69GB (at block 247000). I think the minimal requirement are 100+GB HD and 4+GB RAM.

## Unconfirmed transactions

Bitcoind memory pool is synced every 500ms in Redis, along with every transactions.

The most recent memory pool sync is stored in a sorted set (with time as score, in ``btcplex:rawmempool``), allowing them to be "replayed" via SSE on the unconfirmed transactions page.
Each unconfirmed transaction is stored as JSON in a key ``btcplex:utx:%v`` (hash), the key is destroyed when it get removed from the memory pool.
The sync is performed by keeping two sets: ``btcplex:rawmempool:%v`` (unix time):

- one containing the previous state of memory pool (500ms ago)
- the current state of memory pool

The diff of the two sets is computed, and old unconfirmed transactions are removed. 

## New block

BTCplex relies on ``bitcoind`` blocknotify callback, each time the best block changes, it will be processed (via the RPC API) and immediately available. 

## Available keys in SSDB

I will try to keep an updated list of how data is stored in SSDB by data type.

### Strings

Blocks, transactions, TxIns, TxOuts are stored in JSON format (SSDB support Redis protocol but it doesn't support MULTI, so I can't use hashes if I can't retrieve multiple hashes in one call, also, some JSON objects are nested, so I stick to JSON.

- ``block:height:%v`` (height) -> Contains the hash for the given height
- ``block:%v`` (hash) -> Block data in JSON format
- ``block:%v`` (hash) -> Block data along with its transactions in JSON format
- ``tx:%v`` (hash) -> Transaction data in JSON format
- ``txi:%v:%v`` (hash, index) -> TxIn data in JSON format
- ``txo:%v:%v`` (hash, index) -> TxOut data in JSON format
- ``txo:%v:%v:spent`` (hash, index) -> Spent data in JSON format
- ``btcplex:utx:%v`` (hash) -> Unconfirmed transaction (with TxOuts/TxIns) in JSON format


### Hashes

BTCplex keeps one hashes per address (``addr:%v:h`` (address)) containing TotalSent, TotalReceived. 

Hash keys for address data:

- ``ts`` -> TotalSent
- ``tr`` -> TotalReceived


### Sorted Sets

BTCplex keeps three sorted set per address (``addr:%v`` (address), ``addr:%v:received`` (address), ``addr:%v:sent`` (address)) containing Tx hash for every transaction involving the address sorted by BlockTime.

It also store one sorted for each block containing transaction references sorted by index (``block:%v:txs`` (hash)).

Bitcoind memory pool is "synced" in a sorted set: ``btcplex:rawmempool``.


### Webapp

- [Martini](http://martini.codegangsta.io/)


### Server-Sent Events

One goroutine per server-sent events, broadcasted (using [bcast](https://github.com/grafov/bcast)) to multiple channels efficiently.

List of Redis PubSub channels:

- btcplex:utxs -> Rely unconfirmed transactions in JSON format
- btcplex:blocknotify -> Send best block hash when it changes (blocknotify callback)


## Backend notes
{'fraction_non_alphanumeric': 0.11406127258444619, 'fraction_numerical': 0.00816967792615868, 'mean_word_length': 4.554973821989528, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 24, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6128347', 'n_tokens_mistral': 1941, 'n_tokens_neox': 1854, 'n_words': 859}