<commit_before>---
layout: post
title:  "Hadoop Cluster 2.7.3 Installation on CentOS 7 in basic version"
date:   2016-12-10 09:00:00
author: Unskilled Coder
categories: hadoop
---

## Summary

### [1. Introduction](#1)

### [2. Architecture](#2)

### [3. CentOS setup](#3)

### [4. Hadoop Setup](#4)

### [5. Launch and Shutdown Hadoop Cluster Service](#5)

### [6. Verify the hadoop cluster is up and healthy](#6)

### [7. End](#7)

<br />

## <a name="1">1. Introduction</a>

This posts will give all related detail in how to setup a Hadoop cluster on CentOS linux system. Before you read this article, I will assume that you already have all basic conceptions about Hadoop and Linux operating system.
<br /><br />

## <a name="2">2. Architecture</a>

| IP Address      | Hostname    | Role                                     |
|:----------------|:------------|:-----------------------------------------|
| 192.168.171.132 | master      | NameNode, ResourceManager                |
| 192.168.171.133 | slave1      | SecondaryNameNode, DataNode, NodeManager |
| 192.168.171.134 | slave2      | DataNode, NodeManager                    |

<br />

## <a name="3">3. CentOS setup</a>

### 3.1. install necessary packages for OS

We pick up CentOS minimal ISO as our installation prototype, once the system installed, we need 2 more basic packages: <br />

```bash
sudo yum install -y net-tools
sudo yum install -y openssh-server
```

*The first line is to install ifconfig, while the second one is to be able to be ssh login by remote peer.*<br />

### 3.2. setup hostname for all nodes

This step is optional, but important for better self-identify while you use same username to walk through different nodes. 

```bash
sudo hostnamectl set-hostname master
```

*ex: at <b>master</b> node*

re-login to check the effect

### 3.3. setup jdk for all nodes

*install jdk from oracle official website*

```bash
sudo wget --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u112-b14/jdk-8u112-linux-x64.rpm
sudo yum localinstall -y jdk-8u112-linux-x64.rpm
sudo rm jdk-8u112-linux-x64.rpm
```

*add java.sh under /etc/profile.d/* 

java.sh content:

```bash
export JAVA_HOME=/usr/java/jdk1.8.0_112
export JRE_HOME=/usr/java/jdk1.8.0_112/jre
export CLASSPATH=$JAVA_HOME/lib:.
export PATH=$PATH:$JAVA_HOME/bin
```

re-login, and you'll find all environment variables, and java is well installed.

*Approach to verification:*

```bash
java -version
ls $JAVA_HOME
ls $PATH
```

if the java version goes wrong, you can 

```bash
sudo alternatives --config java
```

then choose a correct version.

### 3.4. setup user and user group on all nodes

```bash
sudo groupadd hadoop
sudo useradd -d /home/hadoop -g hadoop hadoop
passwd hadoop
```

### 3.5. modify hosts file for network inter-recognition on all nodes

```bash
echo '192.168.171.132 master' >> /etc/hosts
echo '192.168.171.133 slave1' >> /etc/hosts
echo '192.168.171.134 slave2' >> /etc/hosts
```

check the recognition works:

```bash
ping master
ping slave1
ping slave2
```

### 3.6. setup ssh no password login on all nodes

```bash
su - hadoop
ssh-keygen -t rsa
ssh-copy-id master
ssh-copy-id slave1
ssh-copy-id slave2
```

now you can ssh login to all 3 nodes without passwd, please have a try to check it out.
<br /> <br />

## <a name="4">4. Hadoop Setup</a>

P.S. the whole ***step 2*** operations happens on a single node, let's say, *master*. In addition, we'll login as user *hadoop* to finish all operations.

```bash
su - hadoop
```

### 4.1. Download and untar on the file system.

```bash
wget http://mirrors.sonic.net/apache/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz
untar -zxvf hadoop-2.7.3.tar.gz
rm hadoop-2.7.3.tar.gz
chmod 775 hadoop-2.7.3
```

### 4.2. Add environment variables for hadoop

append following content onto ~/.bashrc

```bash<commit_msg>correct hadoop article format issue<commit_after>---
layout: post
title:  "Hadoop Cluster 2.7.3 Installation on CentOS 7 in basic version"
date:   2016-12-10 09:00:00
author: <NAME>
categories: hadoop
---

## Summary

### [1. Introduction](#1)

### [2. Architecture](#2)

### [3. CentOS setup](#3)

### [4. Hadoop Setup](#4)

### [5. Launch and Shutdown Hadoop Cluster Service](#5)

### [6. Verify the hadoop cluster is up and healthy](#6)

### [7. End](#7)

<br />

## <a name="1">1. Introduction</a>

This posts will give all related detail in how to setup a Hadoop cluster on CentOS linux system. Before you read this article, I will assume that you already have all basic conceptions about Hadoop and Linux operating system.
<br /><br />

## <a name="2">2. Architecture</a>

| IP Address      | Hostname    | Role                                     |
|:----------------|:------------|:-----------------------------------------|
| 192.168.171.132 | master      | NameNode, ResourceManager                |
| 192.168.171.133 | slave1      | SecondaryNameNode, DataNode, NodeManager |
| 192.168.171.134 | slave2      | DataNode, NodeManager                    |

<br />

## <a name="3">3. CentOS setup</a>

### 3.1. install necessary packages for OS

We pick up CentOS minimal ISO as our installation prototype, once the system installed, we need 2 more basic packages: <br />

```bash
sudo yum install -y net-tools
sudo yum install -y openssh-server
```

*The first line is to install ifconfig, while the second one is to be able to be ssh login by remote peer.*<br />

### 3.2. setup hostname for all nodes

This step is optional, but important for better self-identify while you use same username to walk through different nodes. 

```bash
sudo hostnamectl set-hostname master
```

*ex: at <b>master</b> node*

re-login to check the effect

### 3.3. setup jdk for all nodes

*install jdk from oracle official website*

```bash
sudo wget --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u112-b14/jdk-8u112-linux-x64.rpm
sudo yum localinstall -y jdk-8u112-linux-x64.rpm
sudo rm jdk-8u112-linux-x64.rpm
```

*add java.sh under /etc/profile.d/* 

java.sh content:

```bash
export JAVA_HOME=/usr/java/jdk1.8.0_112
export JRE_HOME=/usr/java/jdk1.8.0_112/jre
export CLASSPATH=$JAVA_HOME/lib:.
export PATH=$PATH:$JAVA_HOME/bin
```

re-login, and you'll find all environment variables, and java is well installed.

*Approach to verification:*

```bash
java -version
ls $JAVA_HOME
ls $PATH
```

if the java version goes wrong, you can 

```bash
sudo alternatives --config java
```

then choose a correct version.

### 3.4. setup user and user group on all nodes

```bash
sudo groupadd hadoop
sudo useradd -d /home/hadoop -g hadoop hadoop
passwd hadoop
```

### 3.5. modify hosts file for network inter-recognition on all nodes

```bash
echo '192.168.171.132 master' >> /etc/hosts
echo '192.168.171.133 slave1' >> /etc/hosts
echo '192.168.171.134 slave2' >> /etc/hosts
```

check the recognition works:

```bash
ping master
ping slave1
ping slave2
```

### 3.6. setup ssh no password login on all nodes

```bash
su - hadoop
ssh-keygen -t rsa
ssh-copy-id master
ssh-copy-id slave1
ssh-copy-id slave2
```

now you can ssh login to all 3 nodes without passwd, please have a try to check it out.
<br /> <br />

## <a name="4">4. Hadoop Setup</a>

P.S. the whole **[Step 4](#4)** operations happens on a single node, let's say, *master*. In addition, we'll login as user *hadoop* to finish all operations.

```bash
su - hadoop
```

### 4.1. Download and untar on the file system.

```bash
wget http://mirrors.sonic.net/apache/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz
untar -zxvf hadoop-2.7.3.tar.gz
rm hadoop-2.7.3.tar.gz
chmod 775 hadoop-2.7.3
```

### 4.2. Add environment variables for hadoop

append following content onto ~/.bashrc

```bash
{'fraction_non_alphanumeric': 0.15086816720257235, 'fraction_numerical': 0.04977491961414791, 'mean_word_length': 3.1361702127659576, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 12, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3020785', 'n_tokens_mistral': 2948, 'n_tokens_neox': 2563, 'n_words': 928}