<commit_before># megaman: Scalable manifold learning

[![build status](http://img.shields.io/travis/mmp2/megaman/master.svg?style=flat)](https://travis-ci.org/mmp2/megaman)
[![version status](http://img.shields.io/pypi/v/megaman.svg?style=flat)](https://pypi.python.org/pypi/megaman)
[![license](http://img.shields.io/badge/license-BSD-blue.svg?style=flat)](https://github.com/mmp2/megaman/blob/master/LICENSE)

``megaman`` is a scalable manifold learning package implemented in
python. It has a front-end API designed to be familiar
to [scikit-learn](http://scikit-learn.org/) but harnesses
the C++ Fast Library for Approximate Nearest Neighbors (FLANN)
and the Sparse Symmetric Positive Definite (SSPD) solver
Locally Optimal Block Precodition Gradient (LOBPCG) method
to scale manifold learning algorithms to large data sets.
On a personal computer megaman can embed 1 million data points
with hundreds of dimensions in 10 minutes.
megaman is designed for researchers and as such caches intermediary
steps and indices to allow for fast re-computation with new parameters.

Documentation can be found at http://mmp2.github.io/megaman/

## Examples

See documentation and example.py for usage.

## Installation

megaman has the following dependencies:

- [python](http://python.org) tested with versions 2.7, 3.4, and 3.5
- [numpy](http://numpy.org) version 1.8 or higher
- [scipy](http://scipy.org) version 0.16.0 or higher
- [scikit-learn](http://scikit-learn.org)
- [FLANN](http://www.cs.ubc.ca/research/flann/)

Additionally, installation from source requires [cython](http://cython.org/) and a C++ compiler such as ``gcc``/``g++``

Optional requirements include

- [pyamg](http://pyamg.org/), which allows for faster decompositions of large matrices
- [pyflann](https://github.com/primetang/pyflann) which offers another method of computing distance matrices

The package uses ``nose`` for unit tests

### Installing from source pip and conda

The above requirements can be installed on Linux and Mac OSX using [conda](http://conda.pydata.org/miniconda.html):

```
$ conda install -c https://conda.anaconda.org/jakevdp pip nose coverage cython numpy scipy scikit-learn flann pyamg gcc
$ pip install pyflann
```

Finally, install the ``megaman`` package itself:
```
$ python setup.py install
```

## Unit Tests
megaman uses ``nose`` for unit tests. With nosetests installed, type

    $ make test

to run the unit tests. The tests are run on Python versions 2.7

## Authors
- [<NAME>](http://www.stat.washington.edu/people/jmcq/)
- [<NAME>](http://www.stat.washington.edu/mmp/)
- [<NAME>](https://github.com/Jerryzcn)
- [<NAME>](http://www.vanderplas.com)

## Future Work

We have the following planned updates for upcoming releases:

- Native support for K-Nearest Neighbors distance (in progress)
- Lazy R-metric (only calcualte on selected points)
- Make cover_plotter.py work more generally with rmetric.py
<commit_msg>DOC: update install instructions in README<commit_after># megaman: Scalable manifold learning

[![build status](http://img.shields.io/travis/mmp2/megaman/master.svg?style=flat)](https://travis-ci.org/mmp2/megaman)
[![version status](http://img.shields.io/pypi/v/megaman.svg?style=flat)](https://pypi.python.org/pypi/megaman)
[![license](http://img.shields.io/badge/license-BSD-blue.svg?style=flat)](https://github.com/mmp2/megaman/blob/master/LICENSE)

``megaman`` is a scalable manifold learning package implemented in
python. It has a front-end API designed to be familiar
to [scikit-learn](http://scikit-learn.org/) but harnesses
the C++ Fast Library for Approximate Nearest Neighbors (FLANN)
and the Sparse Symmetric Positive Definite (SSPD) solver
Locally Optimal Block Precodition Gradient (LOBPCG) method
to scale manifold learning algorithms to large data sets.
On a personal computer megaman can embed 1 million data points
with hundreds of dimensions in 10 minutes.
megaman is designed for researchers and as such caches intermediary
steps and indices to allow for fast re-computation with new parameters.

Documentation can be found at http://mmp2.github.io/megaman/

## Examples

See documentation and example.py for usage.

## Installation

megaman has the following dependencies:

- [python](http://python.org) tested with versions 2.7, 3.4, and 3.5
- [numpy](http://numpy.org) version 1.8 or higher
- [scipy](http://scipy.org) version 0.16.0 or higher
- [scikit-learn](http://scikit-learn.org)
- [FLANN](http://www.cs.ubc.ca/research/flann/)

Additionally, installation from source requires [cython](http://cython.org/) and a C++ compiler such as ``gcc``/``g++``

Optional requirements include

- [pyamg](http://pyamg.org/), which allows for faster decompositions of large matrices
- [pyflann](https://github.com/primetang/pyflann) which offers another method of computing distance matrices

The package uses ``nose`` for unit tests

### Installing from source pip and conda

The above requirements can be installed on Linux and Mac OSX using [conda](http://conda.pydata.org/miniconda.html):

```
$ conda install -c https://conda.anaconda.org/jakevdp pip nose coverage cython numpy scipy scikit-learn pyflann pyamg gcc
```

Finally, install the ``megaman`` package itself:
```
$ python setup.py install
```

## Unit Tests
megaman uses ``nose`` for unit tests. With nosetests installed, type

    $ make test

to run the unit tests. The tests are run on Python versions 2.7

## Authors
- [<NAME>](http://www.stat.washington.edu/people/jmcq/)
- [<NAME>](http://www.stat.washington.edu/mmp/)
- [<NAME>](https://github.com/Jerryzcn)
- [<NAME>](http://www.vanderplas.com)

## Future Work

We have the following planned updates for upcoming releases:

- Native support for K-Nearest Neighbors distance (in progress)
- Lazy R-metric (only calcualte on selected points)
- Make cover_plotter.py work more generally with rmetric.py

{'fraction_non_alphanumeric': 0.12583120204603582, 'fraction_numerical': 0.007161125319693095, 'mean_word_length': 4.673114119922631, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 12, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2905715', 'n_tokens_mistral': 1933, 'n_tokens_neox': 1819, 'n_words': 592}