<commit_before># H2O

[![Join the chat at https://gitter.im/h2oai/h2o-3](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/h2oai/h2o-3?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

H2O scales statistics, machine learning, and math over Big Data. 

H2O uses familiar interfaces like R, Python, Scala, the Flow notebook graphical interface, Excel, & JSON so that Big Data enthusiasts & experts can explore, munge, model, and score datasets using a range of algorithms including advanced ones like Deep Learning. H2O is extensible so that developers can add data transformations and model algorithms of their choice and access them through all of those clients.

Data collection is easy. Decision making is hard. H2O makes it fast and easy to derive insights from your data through faster and better predictive modeling. H2O allows online scoring and modeling in a single platform.

* [Downloading H2O-3](#Downloading)
* [Open Source Resources](#Resources)
    * [Issue tracking](#IssueTracking) 
* [Using H2O-3 Code Artifacts (libraries)](#Artifacts)
* [Building H2O-3](#Building)
* [Launching H2O after Building](#Launching)
* [Building H2O on Hadoop](#BuildingHadoop)
* [Sparkling Water](#Sparkling)
* [Documentation](#Documentation)
* [Community](#Community) / [Advisors](#Advisors) / [Investors](#Investors)

<a name="Downloading"></a>
## 1. Downloading H2O-3

While most of this README is written for developers who do their own builds, most H2O users just download and use a pre-built version.  If that's you, just follow these steps:

1.  Point to <http://h2o.ai>
2.  Click on Download
3.  Scroll down to find the section for H2O-3
4.  Click the version you want (generally the latest numbered release)

<a name="Resources"></a>
## 2. Open Source Resources

Most people interact with three primary open source resources:  **GitHub** (which you've already found), **JIRA** (for issue tracking), and **h2ostream** (a community discussion forum).

<a name="IssueTracking"></a>
### 2.1 Issue tracking

You can browse and create new issues in our open source **JIRA**:  <http://jira.h2o.ai>

*  You can **browse** and search for **issues** without logging in to JIRA:
    1.  Click the `Issues` menu
    1.  Click `Search for issues`
*  To **create** an **issue** (either a bug or a feature request), please create yourself an account first:
    1.  Click the `Log In` button on the top right of the screen
    1.  Click `Create an acccount` near the bottom of the login box
    1.  Once you have created an account and logged in, use the `Create` button on the menu to create an issue
    1.  Create H2O-3 issues in the [PUBDEV](https://0xdata.atlassian.net/projects/PUBDEV/issues) project

> (Note: There is only one issue tracking system for the project.  GitHub issues are not enabled, you must use JIRA.)

### 2.2 List of open source resources

*  GitHub
    * <https://github.com/h2oai/h2o-3>
*  JIRA - file issues here ([PUBDEV](https://0xdata.atlassian.net/projects/PUBDEV/issues) contains issues for the current H2O-3 project)
    * <http://jira.h2o.ai>
*  h2ostream community forum - ask your questions here
    * Web: <https://groups.google.com/d/forum/h2ostream>
    * Mail to: [<EMAIL>](mailto:<EMAIL>)
*  Documentation
    * Bleeding edge nightly build page: <http://s3.amazonaws.com/h2o-release/h2o-3/master/latest.html>
    * FAQ: <http://h2o.ai/product/faq/>
*  Download (pre-built packages)
    * <http://h2o.ai/download>
*  Jenkins
    * <http://test.h2o.ai>
*  Website
    * <http://h2o.ai>
*  Follow us on Twitter, [@h2oai](https://twitter.com/h2oai)


<a name="Artifacts"></a>
## 3. Using H2O-3 Artifacts

Every nightly build publishes R, Python, Java, and Scala artifacts to a build-specific repository.  In particular, you can find Java artifacts in the maven/repo directory.

Here is an example snippet of a gradle build file using h2o-3 as a dependency.  Replace x, y, z, and nnnn with valid numbers.

```
// h2o-3 dependency information
def h2oBranch = 'master'
def h2oBuildNumber = 'nnnn'
def h2oProjectVersion = "x.y.z.${h2oBuildNumber}"

repositories {
  // h2o-3 dependencies
  maven {
    url "https://s3.amazonaws.com/h2o-release/h2o-3/${h2oBranch}/${h2oBuildNumber}/maven/repo/"
  }
}

dependencies {
  compile "ai.h2o:h2o-core:${h2oProjectVersion}"
  compile "ai.h2o:h2o-algos:${h2oProjectVersion}"
  compile "ai.h2o:h2o-web:${h2oProjectVersion}"
  compile "ai.h2o:h2o-app:${h2oProjectVersion}"
}
```

Refer to the latest H2O-3 bleeding edge [nightly build page](http://s3.amazonaws.com/h2o-release/h2o-3/master/latest.html) for information about installing nightly build artifacts.

Refer to the [h2o-droplets GitHub repository](https://github.com/h2oai/h2o-droplets) for a working example of how to use Java artifacts with gradle.

> Note: Stable H2O-3 artifacts are periodically published to Maven Central ([click here to search](http://search.maven.org/#search%7Cga%7C1%7Cai.h2o)) but may substantially lag behind H2O-3 Bleeding Edge nightly builds.

-----

<a name="Building"></a>
## 4. Building H2O-3

Getting started with H2O development requires [JDK 1.7](http://www.oracle.com/technetwork/java/javase/downloads/), [Node.js](https://nodejs.org/), and Gradle.  We use the Gradle wrapper (called `gradlew`) to ensure up-to-date local versions of Gradle and other dependencies are installed in your development directory.

### 4.1. Building from the command line (Quick Start)

To build H2O from the repository, perform the following steps. 


#### Recipe 1: Clone fresh, build, skip tests, and run H2O

```
# Build H2O
git clone https://github.com/h2oai/h2o-3.git
cd h2o-3
./gradlew build -x test

# Start H2O
java -jar build/h2o.jar

# Point browser to http://localhost:54321

```

#### Recipe 2: Clone fresh, build, and run tests (requires a working install of R)

```
git clone https://github.com/h2oai/h2o-3.git
cd h2o-3
./gradlew syncSmalldata
./gradlew syncRPackages
./gradlew build
```

>**Notes**: 
>
> - Running tests starts five test JVMs that form an H2O cluster and requires at least 8GB of RAM (preferably 16GB of RAM).
> - Running `./gradlew syncRPackages` is supported on Windows, OS X, and Linux, and is strongly recommended but not required. `./gradlew syncRPackages` ensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using `./gradlew syncRPackages`. To set the ENV variable, use the following format (where `${WORKSPACE} can be any path):
>  
>  ```
> mkdir -p ${WORKSPACE}/Rlibrary
export R_LIBS_USER=${WORKSPACE}/Rlibrary
```

#### Recipe 3:  Pull, clean, build, and run tests

```
git pull
./gradlew syncSmalldata
./gradlew syncRPackages
./gradlew clean
./gradlew build
```

#### Notes

 - We recommend using `./gradlew clean` after each `git pull`.

- Skip tests by adding `-x test` at the end the gradle build command line.  Tests typically run for 7-10 minutes on a Macbook Pro laptop with 4 CPUs (8 hyperthreads) and 16 GB of RAM.

- Syncing smalldata is not required after each pull, but if tests fail due to missing data files, then try `./gradlew syncSmalldata` as the first troubleshooting step.  Syncing smalldata downloads data files from AWS S3 to the smalldata directory in your workspace.  The sync is incremental.  Do not check in these files.  The smalldata directory is in .gitignore.  If you do not run any tests, you do not need the smalldata directory.
- Running `./gradlew syncRPackages` is supported on Windows, OS X, and Linux, and is strongly recommended but not required. `./gradlew syncRPackages` ensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using `./gradlew syncRPackages`. To set the ENV variable, use the following format (where `${WORKSPACE} can be any path):

  ```
  mkdir -p ${WORKSPACE}/Rlibrary
  export R_LIBS_USER=${WORKSPACE}/Rlibrary
  ```

### 4.2. Setup on all Platforms
<commit_msg>Added info re: Pip & Python<commit_after># H2O

[![Join the chat at https://gitter.im/h2oai/h2o-3](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/h2oai/h2o-3?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

H2O scales statistics, machine learning, and math over Big Data. 

H2O uses familiar interfaces like R, Python, Scala, the Flow notebook graphical interface, Excel, & JSON so that Big Data enthusiasts & experts can explore, munge, model, and score datasets using a range of algorithms including advanced ones like Deep Learning. H2O is extensible so that developers can add data transformations and model algorithms of their choice and access them through all of those clients.

Data collection is easy. Decision making is hard. H2O makes it fast and easy to derive insights from your data through faster and better predictive modeling. H2O allows online scoring and modeling in a single platform.

* [Downloading H2O-3](#Downloading)
* [Open Source Resources](#Resources)
    * [Issue tracking](#IssueTracking) 
* [Using H2O-3 Code Artifacts (libraries)](#Artifacts)
* [Building H2O-3](#Building)
* [Launching H2O after Building](#Launching)
* [Building H2O on Hadoop](#BuildingHadoop)
* [Sparkling Water](#Sparkling)
* [Documentation](#Documentation)
* [Community](#Community) / [Advisors](#Advisors) / [Investors](#Investors)

<a name="Downloading"></a>
## 1. Downloading H2O-3

While most of this README is written for developers who do their own builds, most H2O users just download and use a pre-built version.  If that's you, just follow these steps:

1.  Point to <http://h2o.ai>
2.  Click on Download
3.  Scroll down to find the section for H2O-3
4.  Click the version you want (generally the latest numbered release)

<a name="Resources"></a>
## 2. Open Source Resources

Most people interact with three primary open source resources:  **GitHub** (which you've already found), **JIRA** (for issue tracking), and **h2ostream** (a community discussion forum).

<a name="IssueTracking"></a>
### 2.1 Issue tracking

You can browse and create new issues in our open source **JIRA**:  <http://jira.h2o.ai>

*  You can **browse** and search for **issues** without logging in to JIRA:
    1.  Click the `Issues` menu
    1.  Click `Search for issues`
*  To **create** an **issue** (either a bug or a feature request), please create yourself an account first:
    1.  Click the `Log In` button on the top right of the screen
    1.  Click `Create an acccount` near the bottom of the login box
    1.  Once you have created an account and logged in, use the `Create` button on the menu to create an issue
    1.  Create H2O-3 issues in the [PUBDEV](https://0xdata.atlassian.net/projects/PUBDEV/issues) project

> (Note: There is only one issue tracking system for the project.  GitHub issues are not enabled, you must use JIRA.)

### 2.2 List of open source resources

*  GitHub
    * <https://github.com/h2oai/h2o-3>
*  JIRA - file issues here ([PUBDEV](https://0xdata.atlassian.net/projects/PUBDEV/issues) contains issues for the current H2O-3 project)
    * <http://jira.h2o.ai>
*  h2ostream community forum - ask your questions here
    * Web: <https://groups.google.com/d/forum/h2ostream>
    * Mail to: [<EMAIL>](mailto:<EMAIL>)
*  Documentation
    * Bleeding edge nightly build page: <http://s3.amazonaws.com/h2o-release/h2o-3/master/latest.html>
    * FAQ: <http://h2o.ai/product/faq/>
*  Download (pre-built packages)
    * <http://h2o.ai/download>
*  Jenkins
    * <http://test.h2o.ai>
*  Website
    * <http://h2o.ai>
*  Follow us on Twitter, [@h2oai](https://twitter.com/h2oai)


<a name="Artifacts"></a>
## 3. Using H2O-3 Artifacts

Every nightly build publishes R, Python, Java, and Scala artifacts to a build-specific repository.  In particular, you can find Java artifacts in the maven/repo directory.

Here is an example snippet of a gradle build file using h2o-3 as a dependency.  Replace x, y, z, and nnnn with valid numbers.

```
// h2o-3 dependency information
def h2oBranch = 'master'
def h2oBuildNumber = 'nnnn'
def h2oProjectVersion = "x.y.z.${h2oBuildNumber}"

repositories {
  // h2o-3 dependencies
  maven {
    url "https://s3.amazonaws.com/h2o-release/h2o-3/${h2oBranch}/${h2oBuildNumber}/maven/repo/"
  }
}

dependencies {
  compile "ai.h2o:h2o-core:${h2oProjectVersion}"
  compile "ai.h2o:h2o-algos:${h2oProjectVersion}"
  compile "ai.h2o:h2o-web:${h2oProjectVersion}"
  compile "ai.h2o:h2o-app:${h2oProjectVersion}"
}
```

Refer to the latest H2O-3 bleeding edge [nightly build page](http://s3.amazonaws.com/h2o-release/h2o-3/master/latest.html) for information about installing nightly build artifacts.

Refer to the [h2o-droplets GitHub repository](https://github.com/h2oai/h2o-droplets) for a working example of how to use Java artifacts with gradle.

> Note: Stable H2O-3 artifacts are periodically published to Maven Central ([click here to search](http://search.maven.org/#search%7Cga%7C1%7Cai.h2o)) but may substantially lag behind H2O-3 Bleeding Edge nightly builds.

-----

<a name="Building"></a>
## 4. Building H2O-3

Getting started with H2O development requires [JDK 1.7](http://www.oracle.com/technetwork/java/javase/downloads/), [Node.js](https://nodejs.org/), and Gradle.  We use the Gradle wrapper (called `gradlew`) to ensure up-to-date local versions of Gradle and other dependencies are installed in your development directory.

### 4.1. Building from the command line (Quick Start)

To build H2O from the repository, perform the following steps. 


#### Recipe 1: Clone fresh, build, skip tests, and run H2O

```
# Build H2O
git clone https://github.com/h2oai/h2o-3.git
cd h2o-3
./gradlew build -x test

# Start H2O
java -jar build/h2o.jar

# Point browser to http://localhost:54321

```

#### Recipe 2: Clone fresh, build, and run tests (requires a working install of R)

```
git clone https://github.com/h2oai/h2o-3.git
cd h2o-3
./gradlew syncSmalldata
./gradlew syncRPackages
./gradlew build
```

>**Notes**: 
>
> - Running tests starts five test JVMs that form an H2O cluster and requires at least 8GB of RAM (preferably 16GB of RAM).
> - Running `./gradlew syncRPackages` is supported on Windows, OS X, and Linux, and is strongly recommended but not required. `./gradlew syncRPackages` ensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using `./gradlew syncRPackages`. To set the ENV variable, use the following format (where `${WORKSPACE} can be any path):
>  
>  ```
> mkdir -p ${WORKSPACE}/Rlibrary
export R_LIBS_USER=${WORKSPACE}/Rlibrary
```

#### Recipe 3:  Pull, clean, build, and run tests

```
git pull
./gradlew syncSmalldata
./gradlew syncRPackages
./gradlew clean
./gradlew build
```

#### Notes

 - We recommend using `./gradlew clean` after each `git pull`.

- Skip tests by adding `-x test` at the end the gradle build command line.  Tests typically run for 7-10 minutes on a Macbook Pro laptop with 4 CPUs (8 hyperthreads) and 16 GB of RAM.

- Syncing smalldata is not required after each pull, but if tests fail due to missing data files, then try `./gradlew syncSmalldata` as the first troubleshooting step.  Syncing smalldata downloads data files from AWS S3 to the smalldata directory in your workspace.  The sync is incremental.  Do not check in these files.  The smalldata directory is in .gitignore.  If you do not run any tests, you do not need the smalldata directory.
- Running `./gradlew syncRPackages` is supported on Windows, OS X, and Linux, and is strongly recommended but not required. `./gradlew syncRPackages` ensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using `./gradlew syncRPackages`. To set the ENV variable, use the following format (where `${WORKSPACE} can be any path):

  ```
  mkdir -p ${WORKSPACE}/Rlibrary
  export R_LIBS_USER=${WORKSPACE}/Rlibrary
  ```

### 4.2. Setup on all Platforms

> **Note**: The following instructions assume you have installed the latest version of [**Pip**](https://pip.pypa.io/en/latest/installing/#install-or-upgrade-pip), which is installed with the latest version of [**Python**](https://www.python.org/downloads/).  

{'fraction_non_alphanumeric': 0.1101582316704916, 'fraction_numerical': 0.01956758062567943, 'mean_word_length': 4.177923702313946, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 66, 'https://': 28, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7499187', 'n_tokens_mistral': 5460, 'n_tokens_neox': 5193, 'n_words': 2000}