<commit_before>PySparkling and Spark Version
=============================
There exist multiple pySparkling packages, each is intended to be used with different Spark version.

 - h2o_pysparkling_1.6 - for Spark 1.6.x
 - h2o_pysparkling_1.5 - for Spark 1.5.x
 - h2o_pysparkling_1.4 - for Spark 1.4.x

So for example, to install pySparkling for Spark 1.6, the command would look like:

.. code-block:: bash

    pip install h2o_pysparkling_1.6

Setup and Installation
======================

Prerequisites:
    
  - Python 2.7
  - Numpy 1.9.2

For windows users, please grab a .whl from http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy

In order to use PySparkling, it requires the following runtime python dependencies to be available on the system: *requests*, *tabulate*, *six* and *future* modules, all of which are available on PyPi:

.. code-block:: bash

  $ pip install requests
  $ pip install tabulate
  $ pip install six
  $ pip install future
  
The required packages are installed automatically in case when PySparkling is installed from PyPi.



The Sparkling-Water Python Module
=================================

Prepare the environment
-----------------------
1. Either clone and build Sparkling Water project

.. code-block:: bash

    git clone http://github.com/h2oai/sparkling-water
    cd sparkling-water
    ./gradlew build -x check


or download and unpack sparkling water release from  `here
<http://www.h2o.ai/download/sparkling-water/choose>`_.

2. Configure the location of Spark distribution and cluster:

.. code-block:: bash

    export SPARK_HOME="/path/to/spark/installation"
    export MASTER='local[*]'


Run PySparkling interactive shell
---------------------------------

1. Ensure you are in the Sparkling Water project directory and run PySparkling shell:

.. code-block:: bash

    bin/pysparkling


The *pysparkling* shell accepts common *pyspark* arguments.


For running on YARN and other supported platforms please see `Running Sparkling Water on supported platforms
<https://github.com/h2oai/sparkling-water/blob/master/DEVEL.md#TargetPlatforms>`_.


2. Initialize H2OContext

.. code:: python

      from pysparkling import *
      import h2o
      hc = H2OContext(sc).start()


Run IPython Notebook with PySparkling
-------------------------------------
.. code-block:: bash

    IPYTHON_OPTS="notebook" bin/pysparkling


Run IPython with PySparkling
----------------------------
.. code-block:: bash

    PYSPARK_PYTHON="ipython" bin/pysparkling


Use PySparkling as Spark Package
--------------------------------
.. code-block:: bash

	$SPARK_HOME/bin/spark-submit
	--packages ai.h2o:sparkling-water-core_2.10:1.6.1
	--py-files $SPARKLING_HOME/py/dist/pySparkling-1.6.1-py2.7.egg  ./py/examples/scripts/ChicagoCrimeDemo.py


Use PySparkling in Databricks Cloud
-----------------------------------

To use PySparkling in Databricks cloud, add PySparkling egg file as a library and attach this library to an existing cluster. After this step you can use PySparkling in a notebook
attached to the cluster in exactly same way as from pySparkling shell.

The PySparkling egg file is available in *py/dist* directory in both built Sparkling Water project and downloaded Sparkling Water release.

	
An introduction to PySparkling
==============================

What is H2O?
------------

H2O is an opensource, in-memory, distributed, fast and scalable machine learning and predictive analytics platform that provides capability to build machine learning models on big data and allow easy productionalization of them in an enterprise environment. 
<commit_msg>[SW-148] Mention in pysparkling readme which python modules are required when using PySparkling in Databricks cloud<commit_after>PySparkling and Spark Version
=============================
There exist multiple pySparkling packages, each is intended to be used with different Spark version.

 - h2o_pysparkling_1.6 - for Spark 1.6.x
 - h2o_pysparkling_1.5 - for Spark 1.5.x
 - h2o_pysparkling_1.4 - for Spark 1.4.x

So for example, to install pySparkling for Spark 1.6, the command would look like:

.. code-block:: bash

    pip install h2o_pysparkling_1.6

Setup and Installation
======================

Prerequisites:
    
  - Python 2.7
  - Numpy 1.9.2

For windows users, please grab a .whl from http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy

In order to use PySparkling, it requires the following runtime python dependencies to be available on the system: *requests*, *tabulate*, *six* and *future* modules, all of which are available on PyPi:

.. code-block:: bash

  $ pip install requests
  $ pip install tabulate
  $ pip install six
  $ pip install future
  
The required packages are installed automatically in case when PySparkling is installed from PyPi.



The Sparkling-Water Python Module
=================================

Prepare the environment
-----------------------
1. Either clone and build Sparkling Water project

.. code-block:: bash

    git clone http://github.com/h2oai/sparkling-water
    cd sparkling-water
    ./gradlew build -x check


or download and unpack sparkling water release from  `here
<http://www.h2o.ai/download/sparkling-water/choose>`_.

2. Configure the location of Spark distribution and cluster:

.. code-block:: bash

    export SPARK_HOME="/path/to/spark/installation"
    export MASTER='local[*]'


Run PySparkling interactive shell
---------------------------------

1. Ensure you are in the Sparkling Water project directory and run PySparkling shell:

.. code-block:: bash

    bin/pysparkling


The *pysparkling* shell accepts common *pyspark* arguments.


For running on YARN and other supported platforms please see `Running Sparkling Water on supported platforms
<https://github.com/h2oai/sparkling-water/blob/master/DEVEL.md#TargetPlatforms>`_.


2. Initialize H2OContext

.. code:: python

      from pysparkling import *
      import h2o
      hc = H2OContext(sc).start()


Run IPython Notebook with PySparkling
-------------------------------------
.. code-block:: bash

    IPYTHON_OPTS="notebook" bin/pysparkling


Run IPython with PySparkling
----------------------------
.. code-block:: bash

    PYSPARK_PYTHON="ipython" bin/pysparkling


Use PySparkling as Spark Package
--------------------------------
.. code-block:: bash

	$SPARK_HOME/bin/spark-submit
	--packages ai.h2o:sparkling-water-core_2.10:1.6.1
	--py-files $SPARKLING_HOME/py/dist/pySparkling-1.6.1-py2.7.egg  ./py/examples/scripts/ChicagoCrimeDemo.py


Use PySparkling in Databricks Cloud
-----------------------------------
In order to use PySparkling in Databricks cloud, PySparkling module has to be added as a library to current cluster.
Pysparkling can be added as library in two ways. You can either upload PySparkling egg file or add PySparkling module
from PyPi. If you choose to upload PySparkling egg file, don't forget to add libraries for following python modules:
request, tabulate and future. The PySparkling egg file is available in *py/dist* directory in both built Sparkling
Water project and downloaded Sparkling Water release.

	
An introduction to PySparkling
==============================

What is H2O?
------------

H2O is an opensource, in-memory, distributed, fast and scalable machine learning and predictive analytics platform that provides capability to build machine learning models on big data and allow easy productionalization of them in an enterprise environment. 

{'fraction_non_alphanumeric': 0.14919951567334858, 'fraction_numerical': 0.013588053275931657, 'mean_word_length': 3.127706829539145, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '565242', 'n_tokens_mistral': 2275, 'n_tokens_neox': 2090, 'n_words': 785}