<commit_before># Fast Dot Products on Pretty Big Data

[![Build Status](https://travis-ci.org/pinleague/bdot.svg)](https://travis-ci.org/pinleague/bdot)

Bdot makes your ram bigger on the inside. Since it's based on [Bcolz](https://github.com/Blosc/bcolz/)
you also get (mostly) transparant compressed disk based storage for free.

![Bigger on the Inside](https://31.media.tumblr.com/dcd82ee9cc541ef6774572e9110de082/tumblr_inline_n3eq30Vjhh1rnbe7i.gif)


## Install
`pip install bdot`

or build from source (requires bcolz >= 0.9.0)

```python
python setup.py build_ext --inplace
python setup.py install
```

## Usage

### Matrix . Vector

```python
import bdot
import bcolz
import numpy as np

matrix = np.random.random_integers(0, 12000, size=(300000, 100))
bcarray = bdot.carray(matrix, chunklen=2**13, cparams=bcolz.cparams(clevel=2))

v = bcarray[0]

result = bcarray.dot(v)
expected = matrix.dot(v)

# should return True
(expected == result).all()

```

### Matrix . Matrix

```python

import bdot
import bcolz
import numpy as np

matrix = np.random.random_integers(0, 120, size=(1000, 100))
bcarray1 = bdot.carray(matrix, chunklen=2**9, cparams=bcolz.cparams(clevel=2))
bcarray2 = bdot.carray(matrix, chunklen=2**9, cparams=bcolz.cparams(clevel=2))


result = bcarray1.dot(bcarray2)
expected = matrix.dot(matrix.T)

# should return True
(expected == result).all()

```

## Simple Benchmarks

Benchmarks were done on data structures generated by the above code, are very informal, and vary a bit across data sets.

### Space

* bcarray ~64MB
* matrix ~229MB

compression ratio: 3.55 
(this tells you what multiple of the RAM in your machine you can operate with)


compression ratio on real world (non-random) data tends to be better (~5x)

### Time
```python
%timeit -n30 -r2 bcarray.dot(v)

30 loops, best of 2: 48.2 ms per loop
```
<commit_msg>note about using second matrix transpose<commit_after># Fast Dot Products on Pretty Big Data

[![Build Status](https://travis-ci.org/pinleague/bdot.svg)](https://travis-ci.org/pinleague/bdot)

Bdot makes your ram bigger on the inside. Since it's based on [Bcolz](https://github.com/Blosc/bcolz/)
you also get (mostly) transparant compressed disk based storage for free.

![Bigger on the Inside](https://31.media.tumblr.com/dcd82ee9cc541ef6774572e9110de082/tumblr_inline_n3eq30Vjhh1rnbe7i.gif)


## Install
`pip install bdot`

or build from source (requires bcolz >= 0.9.0)

```python
python setup.py build_ext --inplace
python setup.py install
```

## Usage

### Matrix . Vector

Multiply a matrix (stored in a `carray`) with a vector (`numpy.ndarray`)

```python
import bdot
import bcolz
import numpy as np

matrix = np.random.random_integers(0, 12000, size=(300000, 100))
bcarray = bdot.carray(matrix, chunklen=2**13, cparams=bcolz.cparams(clevel=2))

v = bcarray[0]

result = bcarray.dot(v)
expected = matrix.dot(v)

# should return True
(expected == result).all()

```

### Matrix . Matrix

Multiply a matrix (stored in a `carray`) with the transpose of another matrix (also in a `carray`).

```python

import bdot
import bcolz
import numpy as np

matrix = np.random.random_integers(0, 120, size=(1000, 100))
bcarray1 = bdot.carray(matrix, chunklen=2**9, cparams=bcolz.cparams(clevel=2))
bcarray2 = bdot.carray(matrix, chunklen=2**9, cparams=bcolz.cparams(clevel=2))

# calculates bcarray1 x bcarray2.T (transpose)
result = bcarray1.dot(bcarray2)
expected = matrix.dot(matrix.T)

# should return True
(expected == result).all()

```

## Simple Benchmarks

Benchmarks were done on data structures generated by the above code, are very informal, and vary a bit across data sets.

### Space

* bcarray ~64MB
* matrix ~229MB

compression ratio: 3.55 
(this tells you what multiple of the RAM in your machine you can operate with)


compression ratio on real world (non-random) data tends to be better (~5x)

### Time
```python
%timeit -n30 -r2 bcarray.dot(v)

30 loops, best of 2: 48.2 ms per loop
```

{'fraction_non_alphanumeric': 0.1268958543983822, 'fraction_numerical': 0.0455005055611729, 'mean_word_length': 4.802052785923753, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '583848', 'n_tokens_mistral': 1534, 'n_tokens_neox': 1390, 'n_words': 428}