<commit_before>numpush
=======

Efficient distribution for scientific Python data structures and code.

Numpush is not a clustering or map-reduce solution, it is merely
a library of common functions that are often needed when building
distributed scientific Python applications.


Target Infastructure
--------------------

* Network transports
    * TIPC
    * ZeroMQ
    * TCP
* IO Operations
    * sendfile
    * splice
    * mmap
    * read/write
* Libraries
    * numpy
    * pandas
    * numba
    * bitey
    * llvm-py
    * theano
    * numexpr
    * pytables
    * carray
* Datastore Backends
    * MooseFS
    * Redis

Data
----

The core idea is based on the observation that most scientific Python
data structures can be represented by two parts:

1. Large block of indexable memory
2. Small bits of metadata

(1) is a heavy object to move around, so we use fast transports (i.e.
ZeroMQ, TIPC, UDP Lite ) that all act on a universal memoryview
interface doing low-level POSIX IO calls ( mmap, splice, sendfile )
that don't require needless copying in userspace to send data and
can delegate scheduling to the Linux kernel.

(2) are trivial to move around, these are pickled or msgpacked and moved
around easily since they are most often PyObject's or can be encoded as
PyObjects.

For example a Pandas dataframe is a Numpy array with some extra
metadata and logical methods. The logic exists wherever the library is
installed, to the move the dataframe we need then only encode
the metadata and buffer the internal array.

```python
>>> df = DateFrame({'a': [1,2,3], 'b': [4,5,6]})
   a  b
0  1  4
1  2  5
2  3  6

>>> index = df.index.tolist()
>>> columns = df.columns.tolist()
>>> dtype = df.values.dtype
>>> shape = df.values.shape

>>> nd = buffer(df.values)

>>> df2 = np.frombuffer(nd, dtype=dtype).reshape(shape)
```

Code<commit_msg>Slightly better usage.<commit_after>numpush
=======

Efficient distribution for scientific Python data structures and code.

Numpush is not a clustering or map-reduce solution, it is merely
a library of common functions that are often needed when building
distributed scientific Python applications.


Target Infastructure
--------------------

* Network transports
    * TIPC
    * ZeroMQ
    * TCP
* IO Operations
    * sendfile
    * splice
    * mmap
    * read/write
* Libraries
    * numpy
    * pandas
    * numba
    * bitey
    * llvm-py
    * theano
    * numexpr
    * pytables
    * carray
* Datastore Backends
    * MooseFS
    * Redis

Data
----

The core idea is based on the observation that most scientific Python
data structures can be represented by two parts:

1. Large block of indexable memory
2. Small bits of metadata

(1) is a heavy object to move around, so we use fast transports (i.e.
ZeroMQ, TIPC, UDP Lite ) that all act on a universal memoryview
interface doing low-level POSIX IO calls ( mmap, splice, sendfile )
that don't require needless copying in userspace to send data and
can delegate scheduling to the Linux kernel.

(2) are trivial to move around, these are pickled or msgpacked and moved
around easily since they are most often PyObject's or can be encoded as
PyObjects.

For example a Pandas dataframe is a Numpy array with some extra
metadata and logical methods. The logic exists wherever the library is
installed, to the move the dataframe we need then only encode
the metadata and buffer the internal array.

```python
>>> df = DateFrame({'a': [1,2,3], 'b': [4,5,6]})
   a  b
0  1  4
1  2  5
2  3  6

>>> index = df.index.tolist()
>>> columns = df.columns.tolist()
>>> dtype = df.values.dtype
>>> shape = df.values.shape

>>> nd = buffer(df.values)

... sent over the wire ..

>>> nd2 = np.frombuffer(nd, dtype=dtype).reshape(shape)

>>> df2 = DataFrame(nd2, index=index, columns=columns)
```

Code
{'fraction_non_alphanumeric': 0.09110057925223802, 'fraction_numerical': 0.011058451816745656, 'mean_word_length': 3.254199328107503, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1576296', 'n_tokens_mistral': 1155, 'n_tokens_neox': 1121, 'n_words': 501}