<commit_before>Security
========

H2O contains security features intended for deployment inside a secure
data center.

Security Model
--------------

Below is a discussion of what the security assumptions are, and what the
H2O software does and does not do.

Terms
~~~~~

+-------------------------------------+---------------------------------+
| Term                                | Definition                      |
+=====================================+=================================+
| **H2O Cluster**                     | A collection of H2O nodes that  | 
|                                     | work together. In the H2O Flow  | 
|                                     | Web UI, the cluster status menu |
|                                     | item shows the list of nodes in | 
|                                     | an H2O cluster.                 |
+-------------------------------------+---------------------------------+
| **H2O node**                        | One VM instance running the H2O |
|                                     | main class. One H2O node        | 
|                                     | corresponds to one OS-level     | 
|                                     | process. In the YARN case, one  | 
|                                     | H2O node corresponds to one     |
|                                     | mapper instance and one YARN    |
|                                     | container.                      |
+-------------------------------------+---------------------------------+
| **H2O embedded web port**           | Each Each H2O node contains an  |
|                                     | embedded web port (by default   |
|                                     | port 54321). This web port      |
|                                     | hosts H2O Flow as well as the   |
|                                     | H2O REST API. The user interacts|
|                                     | directly with this web port.    |
+-------------------------------------+---------------------------------+
| **H2O Internal communication port** | Each Each H2O node also has an  |
|                                     | internal port (web port+1, so by| 
|                                     | default port 54322) for internal| 
|                                     | node-to-node communication. This| 
|                                     | is a proprietary binary         |
|                                     | protocol. An attacker using a   |
|                                     | tool like tcpdump or wireshark  |
|                                     | may be able to reverse engineer |
|                                     | data captured on this           |
|                                     | communication path.             |
+-------------------------------------+---------------------------------+

Assumptions (Threat Model)
~~~~~~~~~~~~~~~~~~~~~~~~~~

1. H2O lives in a secure data center.

2. Denial of service is not a concern.

   -  H2O is not designed to withstand a DOS attack.

3. HTTP traffic between the user client and H2O cluster needs to be
   encrypted.

   -  This is true for both interactive sessions (e.g the H2O Flow Web
      UI) and programmatic sessions (e.g. an R program).

4. Man-in-the-middle attacks are of low concern.

   -  Certificate checking on the client side for R/python is not yet
      implemented.

5. Internal binary H2O node-to-H2O node traffic does not need to be
   secured.

   -  The customer is responsible for the H2O cluster's perimeter
      security if this is a concern.
   -  An example would be putting the nodes for an H2O cluster in a VLAN
      and opening up one port so user clients can reach the H2O cluster
      on the embedded web port.

6. You trust the person that starts H2O to start it correctly.

   -  Enabling H2O security requires specifying the correct security
      options.

7. User client sessions do not need to expire. A session lives at most
   as long as the cluster lifetime. H2O clusters are started and stopped
   "frequently enough".

   -  All data is stored in-memory, so restarting the H2O cluster wipes
      all data from memory, and there is nothing to clean from disk.

8. Once a user is authenticated for access to H2O, they have full
   access.

   -  H2O supports authentication but not authorization or access
      control (ACLs).

9. H2O clusters are meant to be accessed by only one user.

   -  Each user starts their own H2O cluster.
   -  H2O only allows access to the embedded web port to the person that
      started the cluster.

Data Chain-of-Custody in a Hadoop Data Center Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Note**: This holds true for all versions of Hadoop (including YARN) supported by H2O.

Through this sequence, it is shown that a user is only able to access
the same data from H2O that they could already access from normal Hadoop
jobs.

1. Data lives in HDFS
2. The files in HDFS have permissions
3. An HDFS user has permissions (capabilities) to access certain files
4. Kerberos (kinit) can be used to authenticate a user in a Hadoop
   environment
5. A user's Hadoop MapReduce job inherits the permissions (capabilities)
   of the user, as well as kinit metadata
6. H2O is a Hadoop MapReduce job
7. H2O can only access the files in HDFS that the user has permission to
   access
8. Only the user that started the cluster is authenticated for access to
   the H2O cluster
9. The authenticated user can access the same data in H2O that he could
   access via HDFS

What is being Secured Today
~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Standard file permissions security is provided by the Operating
   System and by HDFS.

2. The embedded web port in each node of H2O can be secured in two ways:

 +------------------+---------------------------------------+
 | Method           | Description                           |
 +==================+=======================================+
 | HTTPS            | Encrypted socket communication between|
 |                  | the user client and the embedded H2O  |
 |                  | web port.                             |
 +------------------+---------------------------------------+
 | Authentication   | An HTTP Basic Auth username and       |
 |                  | password from the user client.        |
 +------------------+---------------------------------------+

**Note**: Embedded web port HTTPS and authentication may be used separately or together.

What is Specifically not being Secured Today
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-  Internal H2O node-to-H2O node communication.

File Security in H2O
--------------------

H2O is a normal user program. Nothing specifically needs to be done by
the user to get file security for H2O. Operating System and HDFS
permissions "just work".

Standalone H2O
~~~~~~~~~~~~~~

Since H2O is a regular Java program, the files H2O can access are
restricted by the user's Operating System permissions (capabilities).

H2O on Hadoop
~~~~~~~~~~~~~

Since H2O is a regular Hadoop MapReduce program, the files H2O can
access are restricted by the standard HDFS permissions of the user that
starts H2O.

Since H2O is a regular Hadoop MapReduce program, Kerberos (kinit) works
seamlessly. (No code was added to H2O to support Kerberos.)

Sparkling Water on YARN
~~~~~~~~~~~~~~~~~~~~~~~

Similar to H2O on Hadoop, this configuration is H2O on Spark on YARN.
The YARN job inherits the HDFS permissions of the user.
<commit_msg>Update comments that were outdated since node-to-node encryption has been present for a while.<commit_after>Security
========

H2O contains security features intended for deployment inside a secure
data center.

Security Model
--------------

Below is a discussion of what the security assumptions are, and what the
H2O software does and does not do.

Terms
~~~~~

+-------------------------------------+---------------------------------+
| Term                                | Definition                      |
+=====================================+=================================+
| **H2O Cluster**                     | A collection of H2O nodes that  | 
|                                     | work together. In the H2O Flow  | 
|                                     | Web UI, the cluster status menu |
|                                     | item shows the list of nodes in | 
|                                     | an H2O cluster.                 |
+-------------------------------------+---------------------------------+
| **H2O node**                        | One VM instance running the H2O |
|                                     | main class. One H2O node        | 
|                                     | corresponds to one OS-level     | 
|                                     | process. In the YARN case, one  | 
|                                     | H2O node corresponds to one     |
|                                     | mapper instance and one YARN    |
|                                     | container.                      |
+-------------------------------------+---------------------------------+
| **H2O embedded web port**           | Each Each H2O node contains an  |
|                                     | embedded web port (by default   |
|                                     | port 54321). This web port      |
|                                     | hosts H2O Flow as well as the   |
|                                     | H2O REST API. The user interacts|
|                                     | directly with this web port.    |
+-------------------------------------+---------------------------------+
| **H2O Internal communication port** | Each Each H2O node also has an  |
|                                     | internal port (web port+1, so by| 
|                                     | default port 54322) for internal| 
|                                     | node-to-node communication. This| 
|                                     | is a proprietary binary         |
|                                     | protocol. An attacker using a   |
|                                     | tool like tcpdump or wireshark  |
|                                     | may be able to reverse engineer |
|                                     | data captured on this           |
|                                     | communication path.             |
+-------------------------------------+---------------------------------+

Assumptions (Threat Model)
~~~~~~~~~~~~~~~~~~~~~~~~~~

1. H2O lives in a secure data center.

2. Denial of service is not a concern.

   -  H2O is not designed to withstand a DOS attack.

3. HTTP traffic between the user client and H2O cluster needs to be
   encrypted.

   -  This is true for both interactive sessions (e.g the H2O Flow Web
      UI) and programmatic sessions (e.g. an R program).

4. Man-in-the-middle attacks are of low concern.

   -  Certificate checking on the client side for R/python is not yet
      implemented.

5. You may want to secure internal binary H2O node-to-H2O node traffic
   via encryption.

6. You trust the person that starts H2O to start it correctly.

   -  Enabling H2O security requires specifying the correct security
      options.

7. User client sessions do not need to expire. A session lives at most
   as long as the cluster lifetime. H2O clusters are started and stopped
   "frequently enough".

   -  All data is stored in-memory, so restarting the H2O cluster wipes
      all data from memory, and there is nothing to clean from disk.

8. Once a user is authenticated for access to H2O, they have full
   access.

   -  H2O supports authentication but not authorization or access
      control (ACLs).

9. H2O clusters are meant to be accessed by only one user.

   -  Each user starts their own H2O cluster.
   -  H2O only allows access to the embedded web port to the person that
      started the cluster.

Data Chain-of-Custody in a Hadoop Data Center Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Note**: This holds true for all versions of Hadoop (including YARN) supported by H2O.

Through this sequence, it is shown that a user is only able to access
the same data from H2O that they could already access from normal Hadoop
jobs.

1. Data lives in HDFS
2. The files in HDFS have permissions
3. An HDFS user has permissions (capabilities) to access certain files
4. Kerberos (kinit) can be used to authenticate a user in a Hadoop
   environment
5. A user's Hadoop MapReduce job inherits the permissions (capabilities)
   of the user, as well as kinit metadata
6. H2O is a Hadoop MapReduce job
7. H2O can only access the files in HDFS that the user has permission to
   access
8. Only the user that started the cluster is authenticated for access to
   the H2O cluster
9. The authenticated user can access the same data in H2O that he could
   access via HDFS

What is being Secured Today
~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Standard file permissions security is provided by the Operating
   System and by HDFS.

2. The embedded web port in each node of H2O can be secured in two ways:

 +------------------+---------------------------------------+
 | Method           | Description                           |
 +==================+=======================================+
 | HTTPS            | Encrypted socket communication between|
 |                  | the user client and the embedded H2O  |
 |                  | web port.                             |
 +------------------+---------------------------------------+
 | Authentication   | An HTTP Basic Auth username and       |
 |                  | password from the user client.        |
 +------------------+---------------------------------------+

**Note**: Embedded web port HTTPS and authentication may be used separately or together.

3. Internal H2O node-to-H2O node communication can be encrypted.


File Security in H2O
--------------------

H2O is a normal user program. Nothing specifically needs to be done by
the user to get file security for H2O. Operating System and HDFS
permissions "just work".

Standalone H2O
~~~~~~~~~~~~~~

Since H2O is a regular Java program, the files H2O can access are
restricted by the user's Operating System permissions (capabilities).

H2O on Hadoop
~~~~~~~~~~~~~

Since H2O is a regular Hadoop MapReduce program, the files H2O can
access are restricted by the standard HDFS permissions of the user that
starts H2O.

Since H2O is a regular Hadoop MapReduce program, Kerberos (kinit) works
seamlessly. (No code was added to H2O to support Kerberos.)

Sparkling Water on YARN
~~~~~~~~~~~~~~~~~~~~~~~

Similar to H2O on Hadoop, this configuration is H2O on Spark on YARN.
The YARN job inherits the HDFS permissions of the user.

{'fraction_non_alphanumeric': 0.16010799865001688, 'fraction_numerical': 0.011879851501856227, 'mean_word_length': 1.4685104965011664, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 66, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2955692', 'n_tokens_mistral': 3593, 'n_tokens_neox': 3345, 'n_words': 1652}