<commit_before># Securing Dataproc

This initialization action installs the MIT distribution of the Kerberos packages, and configures YARN, HDFS, Hive, Spark to integrate with Kerberos, which also enables in-transit data encryption.
It will also encrypt HTTP traffic (include Web UIs and shuffle) with SSL.
Optionally, this initialization action will also enable cross-realm trust, to allow users to authenticate through a remote Kerberos/Active Directory server.

## Using this initialization action
You can use this initialization action to create a secured Dataproc cluster:
  Using the `gcloud` command to create a new cluster with this initialization action.

```bash
gcloud dataproc clusters create <CLUSTER_NAME> \
    --scopes cloud-platform \
    --initialization-actions gs://dataproc-initialization-actions/secure/secure.sh \
    --metadata "kms-key-uri=projects/<PROJECT_ID>/locations/global/keyRings/my-key-rings/cryptoKeys/my-key" \
    --metadata "root-password-uri=gs://<SECRET_BUCKET>/root-password.encrypted" \
```
The following metadata key-value pairs are the minimally required set to run this script.
1. Use **kms-key-uri** to specify the URI of the KMS key used to encrypt various password files.
2. Use **root-password-uri** to specify the GCS location of the encrypted file which contains the password to the root user principal.

By default a self-signed wildcard certificate will be created from the master and distributed to every node in the cluster. You can also provide your own certificate through the following parameters and we stronly encourage the usage of self-managed certificate instead of the default, on-the-fly-generated self-signed certificate.
```bash
    --metadata "keystore-uri=gs://<SECRET_BUCKET>/keystore.jks" \
    --metadata "truststore-uri=gs://<SECRET_BUCKET>/truststore.jks" \
    --metadata "keystore-password-uri=gs://<SECRET_BUCKET>/keystore-password.encrypted"
```

1. Use **keystore-uri** to specify the GCS location of the keystore file which contains the SSL certificate. It has to be in the Java KeyStore (JKS) format and when copied to VMs, it will be renamed (if necessary) to **keystore.jks**. The SSL certificate should be a wildcard certificate which applies to every node in the cluster.
2. Use **truststore-uri** to specify the GCS location of the truststore file. It has to be in the Java KeyStore (JKS) format and when copied to VMs, it will be renamed (if necessary) to **truststore.jks**.
3. Use **keystore-password-uri** to specify the password to the keystore and truststore files. For simplicity, the keystore password, key password, as well as the truststore password, will all be this password. The password file needs to be encrypted using KMS and stored in GCS.

A default KDC database master key (password) will be provided and can be changed later, or specify your own master key (password):
```bash
    --metadata "db-password-uri=gs://<SECRET_BUCKET>/db-password.encrypted"
```

1. Use **db-password-uri** to specify the GCS location of the encrypted file which contains the password to the KDC master database.

Optionally, enable cross-realm trust: 
```bash
    --metadata "cross-realm-trust-realm=<REMOTE.REALM>" \
    --metadata "cross-realm-trust-kdc=<REMOTE_KDC>" \<commit_msg>Update secure/README.md (#353)<commit_after># Securing Dataproc

This initialization action installs the MIT distribution of the Kerberos packages, and configures YARN, HDFS, Hive, Spark to integrate with Kerberos, which also enables in-transit data encryption.
It will also encrypt HTTP traffic (include Web UIs and shuffle) with SSL.
Optionally, this initialization action will also enable cross-realm trust, to allow users to authenticate through a remote Kerberos/Active Directory server.

## Using this initialization action
You can use this initialization action to create a secured Dataproc cluster:
  Using the `gcloud` command to create a new cluster with this initialization action.

```bash
gcloud dataproc clusters create <CLUSTER_NAME> \
    --scopes cloud-platform \
    --initialization-actions gs://dataproc-initialization-actions/secure/secure.sh \
    --metadata "kms-key-uri=projects/<PROJECT_ID>/locations/global/keyRings/my-key-rings/cryptoKeys/my-key" \
    --metadata "root-password-uri=gs://<SECRET_BUCKET>/root-password.encrypted"
```
The following metadata key-value pairs are the minimally required set to run this script.
1. Use **kms-key-uri** to specify the URI of the KMS key used to encrypt various password files.
2. Use **root-password-uri** to specify the GCS location of the encrypted file which contains the password to the root user principal.

By default a self-signed wildcard certificate will be created from the master and distributed to every node in the cluster. You can also provide your own certificate through the following parameters and we stronly encourage the usage of self-managed certificate instead of the default, on-the-fly-generated self-signed certificate.
```bash
    --metadata "keystore-uri=gs://<SECRET_BUCKET>/keystore.jks" \
    --metadata "truststore-uri=gs://<SECRET_BUCKET>/truststore.jks" \
    --metadata "keystore-password-uri=gs://<SECRET_BUCKET>/keystore-password.encrypted"
```

1. Use **keystore-uri** to specify the GCS location of the keystore file which contains the SSL certificate. It has to be in the Java KeyStore (JKS) format and when copied to VMs, it will be renamed (if necessary) to **keystore.jks**. The SSL certificate should be a wildcard certificate which applies to every node in the cluster.
2. Use **truststore-uri** to specify the GCS location of the truststore file. It has to be in the Java KeyStore (JKS) format and when copied to VMs, it will be renamed (if necessary) to **truststore.jks**.
3. Use **keystore-password-uri** to specify the password to the keystore and truststore files. For simplicity, the keystore password, key password, as well as the truststore password, will all be this password. The password file needs to be encrypted using KMS and stored in GCS.

A default KDC database master key (password) will be provided and can be changed later, or specify your own master key (password):
```bash
    --metadata "db-password-uri=gs://<SECRET_BUCKET>/db-password.encrypted"
```

1. Use **db-password-uri** to specify the GCS location of the encrypted file which contains the password to the KDC master database.

Optionally, enable cross-realm trust: 
```bash
    --metadata "cross-realm-trust-realm=<REMOTE.REALM>" \
    --metadata "cross-realm-trust-kdc=<REMOTE_KDC>" \
{'fraction_non_alphanumeric': 0.08517783544497023, 'fraction_numerical': 0.0022897267592733935, 'mean_word_length': 4.446384039900249, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3693516', 'n_tokens_mistral': 1860, 'n_tokens_neox': 1776, 'n_words': 806}