<commit_before>---
layout: global
title: Configuring Alluxio with secure HDFS
nickname: Alluxio with secure HDFS
group: Under Store
priority: 3
---

* Table of Contents
{:toc}

This guide describes the instruction to configure a
[secure HDFS](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SecureMode.html)
as Alluxio's under storage system. Alluxio supports secure HDFS as the under filesystem, with
[Kerberos](http://web.mit.edu/kerberos/) authentication.

> Note: Kerberos authentication with secure HDFS is not Alluxio internal authentication via Kerberos.

## Initial Setup

To run an Alluxio cluster on a set of machines, you must deploy Alluxio server binaries to each of
these machines. You can either
[download the precompiled binaries directly](http://www.alluxio.org/download)
with the correct Hadoop version (recommended), or
[compile the binaries from Alluxio source code](Building-Alluxio-From-Source.html)
(for advanced users).

Note that, when building Alluxio from source code, by default Alluxio server binaries is built to
work with Apache Hadoop HDFS of version `2.2.0`. To work with Hadoop distributions of other
versions, one needs to specify  the correct Hadoop profile and run the following in your Alluxio
directory:

```bash
$ mvn install -P<YOUR_HADOOP_PROFILE> -D<HADOOP_VERSION> -DskipTests
```

Alluxio provides predefined build profiles including `hadoop-1`, `hadoop-2` (enabled by default),
`hadoop-3` for the major Hadoop versions 1.x, 2.x and 3.x. If you want to build Alluxio with a specific
Hadoop release version, you can also specify the version in the command. For example,

```bash
# Build Alluxio for the Apache Hadoop version Hadoop 2.7.1
$ mvn install -Phadoop-2 -Dhadoop.version=2.7.1 -DskipTests
# Build Alluxio for the Apache Hadoop version Hadoop 2.7.1
$ mvn install -Phadoop-3 -Dhadoop.version=3.0.0 -DskipTests
```

Please visit the
[Building Alluxio Master Branch](Building-Alluxio-From-Source.html#distro-support) page for more
information about support for other distributions.

If everything succeeds, you should see
`alluxio-assembly-server-{{site.ALLUXIO_RELEASED_VERSION}}-jar-with-dependencies.jar` created in
the `${ALLUXIO_HOME}/assembly/server/target` directory.


## Configuring Alluxio

### Basic Configuration

First create the configuration file from the template.

```bash
$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
```

Then edit `conf/alluxio-site.properties` file to set the under storage address to the HDFS namenode
address and the HDFS directory you want to mount to Alluxio. For example, the under storage address
can be `hdfs://localhost:9000` if you are running the HDFS namenode locally with default port and
mapping HDFS root directory to Alluxio, or `hdfs://localhost:9000/alluxio/data` if only the HDFS
directory `/alluxio/data` is mapped to Alluxio.

```
alluxio.underfs.address=hdfs://<NAMENODE>:<PORT>
```

### HDFS configuration files

To ensure Alluxio servers pick up HDFS configurations in classpath, please copy secure HDFS
configuration xml files (`core-site.xml`, `hdfs-site.xml`, `mapred-site.xml`, `yarn-site.xml`) to
`${ALLUXIO_HOME}/conf/`

### Kerberos configuration

Optionally, you can set jvm-level system properties for customized Kerberos configurations:
`java.security.krb5.realm` and `java.security.krb5.kdc`. Those Kerberos configurations route java
libraries to specified Kerberos realm and KDC server address.
If both are set to empty, Kerberos library will respect
the default Kerberos configuration on the machine. For example:

* If you use Hadoop, you can add to `HADOOP_OPTS` in `${HADOOP_CONF_DIR}/hadoop-env.sh`.

```bash
$ export HADOOP_OPTS="$HADOOP_OPTS -Djava.security.krb5.realm=<YOUR_KERBEROS_REALM> -Djava.security.krb5.kdc=<YOUR_KERBEROS_KDC_ADDRESS>"
```

* If you use Spark, you can add to `SPARK_JAVA_OPTS` in `${SPARK_CONF_DIR}/spark-env.sh`.

```bash
SPARK_JAVA_OPTS+=" -Djava.security.krb5.realm=<YOUR_KERBEROS_REALM> -Djava.security.krb5.kdc=<YOUR_KERBEROS_KDC_ADDRESS>"
```

* If you use Alluxio shell, you can add to `ALLUXIO_JAVA_OPTS` in `conf/alluxio-env.sh`.

```bash
ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.realm=<YOUR_KERBEROS_REALM> -Djava.security.krb5.kdc=<YOUR_KERBEROS_KDC_ADDRESS>"
```
<commit_msg>[DOCFIX] Improve secure hdfs docs (#7738)<commit_after>---
layout: global
title: Configuring Alluxio with secure HDFS
nickname: Alluxio with secure HDFS
group: Under Store
priority: 3
---

* Table of Contents
{:toc}

This guide describes the instruction to configure a
[secure HDFS](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SecureMode.html)
as Alluxio's under storage system. Alluxio supports secure HDFS as the under filesystem, with
[Kerberos](http://web.mit.edu/kerberos/) authentication.

> Note: Kerberos authentication with secure HDFS is not Alluxio internal authentication via Kerberos.

## Initial Setup

To run an Alluxio cluster on a set of machines, you must deploy Alluxio server binaries to each of
these machines. You can either
[download the precompiled binaries directly](http://www.alluxio.org/download)
with the correct Hadoop version (recommended), or
[compile the binaries from Alluxio source code](Building-Alluxio-From-Source.html)
(for advanced users).

Note that, when building Alluxio from source code, by default Alluxio server binaries is built to
work with Apache Hadoop HDFS of version `2.2.0`. To work with Hadoop distributions of other
versions, one needs to specify  the correct Hadoop profile and run the following in your Alluxio
directory:

```bash
$ mvn install -P<YOUR_HADOOP_PROFILE> -D<HADOOP_VERSION> -DskipTests
```

Alluxio provides predefined build profiles including `hadoop-1`, `hadoop-2` (enabled by default),
`hadoop-3` for the major Hadoop versions 1.x, 2.x and 3.x. If you want to build Alluxio with a specific
Hadoop release version, you can also specify the version in the command. For example,

```bash
# Build Alluxio for the Apache Hadoop version Hadoop 2.7.1
$ mvn install -Phadoop-2 -Dhadoop.version=2.7.1 -DskipTests
# Build Alluxio for the Apache Hadoop version Hadoop 2.7.1
$ mvn install -Phadoop-3 -Dhadoop.version=3.0.0 -DskipTests
```

Please visit the
[Building Alluxio Master Branch](Building-Alluxio-From-Source.html#distro-support) page for more
information about support for other distributions.

If everything succeeds, you should see
`alluxio-assembly-server-{{site.ALLUXIO_RELEASED_VERSION}}-jar-with-dependencies.jar` created in
the `${ALLUXIO_HOME}/assembly/server/target` directory.


## Configuring Alluxio

### Basic Configuration

First create the configuration file from the template.

```bash
$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
```

Then edit `conf/alluxio-site.properties` file to set the under storage address to the HDFS namenode
address and the HDFS directory you want to mount to Alluxio. For example, the under storage address
can be `hdfs://localhost:9000` if you are running the HDFS namenode locally with default port and
mapping HDFS root directory to Alluxio, or `hdfs://localhost:9000/alluxio/data` if only the HDFS
directory `/alluxio/data` is mapped to Alluxio.

```
alluxio.underfs.address=hdfs://<NAMENODE>:<PORT>
```

### HDFS configuration files

To ensure Alluxio servers pick up HDFS configurations in classpath, please copy secure HDFS
configuration xml files (`core-site.xml`, `hdfs-site.xml`, `mapred-site.xml`, `yarn-site.xml`) to
`${ALLUXIO_HOME}/conf/`

### [Optional] Custom Kerberos configuration

Optionally, you can set jvm-level system properties for customized Kerberos configurations:
`java.security.krb5.realm` and `java.security.krb5.kdc`. Those Kerberos configurations route java
libraries to specified Kerberos realm and KDC server address.
If both are set to empty, Kerberos library will respect
the default Kerberos configuration on the machine. These Java system properties should be set
for the Alluxio processes. To do so, you can add to `ALLUXIO_JAVA_OPTS` in `conf/alluxio-env.sh`.

```bash
ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.realm=<YOUR_KERBEROS_REALM> -Djava.security.krb5.kdc=<YOUR_KERBEROS_KDC_ADDRESS>"
```

{'fraction_non_alphanumeric': 0.09374619041813971, 'fraction_numerical': 0.010483969279531878, 'mean_word_length': 5.122388059701493, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 10}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4731995', 'n_tokens_mistral': 2751, 'n_tokens_neox': 2570, 'n_words': 919}