<commit_before>Zipline
=======

Zipline is a realtime stream processing system.

System Setup
==============

Running
-------

Initial `virtualenv` setup::

    $ mkvirtualenv zipline
    $ workon zipline
	# Go get coffee, the following will compile a heap of C/C++ code
	$ ./etc/ordered_pip.sh etc/requirements.txt
	# And optionally:
	$ ./etc/ordered_pip.sh etc/requirements_dev.txt


Develop
-------

To run tests::

    $ nosetests

Tooling hints
================
zipline relies heavily on scientific Python components (numpy, scikit, pandas, matplotlib, ipython, etc). Tooling up can be a pain, and it often involves managing a configuration including your OS, c/c++/fortran compilers, python version, and versions of numerous modules. I've found the following tools absolutely indispensable: 

- some kind of package manager for your platform. package managers generally give you a way to search, install, uninstall, and check currently installed packages. They also do a great job of managing dependencies.
   - Linux: yum/apt-get
   - Mac OS: homebrew/macport/fink (I highly recommend homebrew: https://github.com/mxcl/homebrew) 
   - Windows: probably best if you use a complete distribution, like: enthought, ActiveState, or Python(x,y)
- Python also provides good package management tools to help you manage the components you install for Python.
   - pip
   - easy_install/setuptools. I have always used setuptools, and I've been quite happy with it. Just remember that setuptools is coupled to your python version. 
- virtualenv and virtualenvwrapper are your very best friends. They complement your python package manager by allowing you to create and quickly switch between named configurations.
    - *Install all the versions of Python you like to use, but install setuptools, virtualenv, and virtualenvwrapper with the very latest python.* Use the latest python to install the latest setuptools, and the latest setuptools to install virtualenv and virtualenvwrapper. virtualenvwrapper allows you to specify the python version you wish to use (mkvirtualenv -p <python executable> <env name>), so you can create envs of any python denomination.

Mac OS hints
-------------

Scientific python on the Mac can be a bit confusing because of the many independent variables. You need to have several components installed, and be aware of the versions of each:

- XCode. XCode includes the gcc and g++ compilers and architecture specific assemblers. Your version of XCode will determine which compilers and assemblers are available. The most common issue I encountered with scientific python libraries is compilation errors of underlying C code. Most scientific libraries are optimized with C routines, so this is a major hurdle. In my environment (XCode 4.0.2 with iOS components installed) I ran into problems with -arch flags asking for power pc (-arch ppc passed to the compiler). Read this stackoverflow to see how to handle similar problems: http://stackoverflow.com/questions/5256397/python-easy-install-fails-with-assembler-for-architecture-ppc-not-installed-on
- gfortran 	- you need this to build numpy. With brew you can install with just: ```brew install gfortran```
- umfpack 	- you need this to build scipy. ```brew install umfpack```<commit_msg>Adjusts indentation of lists.<commit_after>Zipline
=======

Zipline is a realtime stream processing system.

System Setup
==============

Running
-------

Initial `virtualenv` setup::

    $ mkvirtualenv zipline
    $ workon zipline
	# Go get coffee, the following will compile a heap of C/C++ code
	$ ./etc/ordered_pip.sh etc/requirements.txt
	# And optionally:
	$ ./etc/ordered_pip.sh etc/requirements_dev.txt


Develop
-------

To run tests::

    $ nosetests

Tooling hints
================
zipline relies heavily on scientific Python components (numpy, scikit, pandas, matplotlib, ipython, etc). Tooling up can be a pain, and it often involves managing a configuration including your OS, c/c++/fortran compilers, python version, and versions of numerous modules. I've found the following tools absolutely indispensable: 

- some kind of package manager for your platform. package managers generally give you a way to search, install, uninstall, and check currently installed packages. They also do a great job of managing dependencies.
    - Linux: yum/apt-get
    - Mac OS: homebrew/macport/fink (I highly recommend homebrew: https://github.com/mxcl/homebrew) 
    - Windows: probably best if you use a complete distribution, like: enthought, ActiveState, or Python(x,y)
- Python also provides good package management tools to help you manage the components you install for Python.
    - pip
    - easy_install/setuptools. I have always used setuptools, and I've been quite happy with it. Just remember that setuptools is coupled to your python version. 
- virtualenv and virtualenvwrapper are your very best friends. They complement your python package manager by allowing you to create and quickly switch between named configurations.
    - *Install all the versions of Python you like to use, but install setuptools, virtualenv, and virtualenvwrapper with the very latest python.* Use the latest python to install the latest setuptools, and the latest setuptools to install virtualenv and virtualenvwrapper. virtualenvwrapper allows you to specify the python version you wish to use (mkvirtualenv -p <python executable> <env name>), so you can create envs of any python denomination.

Mac OS hints
-------------

Scientific python on the Mac can be a bit confusing because of the many independent variables. You need to have several components installed, and be aware of the versions of each:

- XCode. XCode includes the gcc and g++ compilers and architecture specific assemblers. Your version of XCode will determine which compilers and assemblers are available. The most common issue I encountered with scientific python libraries is compilation errors of underlying C code. Most scientific libraries are optimized with C routines, so this is a major hurdle. In my environment (XCode 4.0.2 with iOS components installed) I ran into problems with -arch flags asking for power pc (-arch ppc passed to the compiler). Read this stackoverflow to see how to handle similar problems: http://stackoverflow.com/questions/5256397/python-easy-install-fails-with-assembler-for-architecture-ppc-not-installed-on
- gfortran 	- you need this to build numpy. With brew you can install with just: ```brew install gfortran```
- umfpack 	- you need this to build scipy. ```brew install umfpack```
{'fraction_non_alphanumeric': 0.07335671801128565, 'fraction_numerical': 0.0030501753850846426, 'mean_word_length': 4.229665071770335, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4678724', 'n_tokens_mistral': 1698, 'n_tokens_neox': 1605, 'n_words': 862}