<commit_before># Setup Guide

This guide contains the setup instructions for *gluster4kube*.


## Infrastructure Requirements

* A running Kubernetes cluster with at least three Kubernetes worker nodes that each have an available raw block device attached (like an EBS Volume or a local disk).
* The three Kubernetes nodes intended to run the GlusterFS Pods must have the appropriate ports opened for GlusterFS communication. Run the following commands on each of the nodes.
```
iptables -N HEKETI
iptables -A HEKETI -p tcp -m state --state NEW -m tcp --dport 24007 -j ACCEPT
iptables -A HEKETI -p tcp -m state --state NEW -m tcp --dport 24008 -j ACCEPT
iptables -A HEKETI -p tcp -m state --state NEW -m tcp --dport 2222 -j ACCEPT
iptables -A HEKETI -p tcp -m state --state NEW -m multiport --dports 49152:49251 -j ACCEPT
```
**** TODO: Needs to be updated to demonstrate how to persist your new iptables configuration so that it is still the same after a reboot of the host. Also needs an update to link in the new HEKETI iptables chain so that it is active. Right now the chain is not referenced so it is just set up but not being used. ****

## Client Setup

Heketi provides a CLI that provides users with a means to administer the deployment and configuration of GlusterFS in Kubernetes. [Download and install the heketi-cli](https://github.com/heketi/heketi/releases/tag/v3.0.0) on your client machine (usually your laptop).

## Kubernetes Deployment

1) The first thing we need to do is create a Service Account so that the GlusterFS containers can communicate securely with each other. 

Create the serviceaccount file as demonstrated below

```
# vi heketi-serviceaccount.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: heketi

# kubectl create -f heketi-serviceaccount.yaml
```

Verify the serviceaccount was created successfully
```
# kubectl get serviceaccount
NAME       SECRETS   AGE
default    1         27d
heketi     1         15s
```

Verify the appropriate secrets were created with the serviceaccount
```
# kubectl get secrets
NAME                   TYPE                                  DATA      AGE
default-token-4wtpz    kubernetes.io/service-account-token   3         27d
heketi-token-dz077     kubernetes.io/service-account-token   3         25s
```
2) Deploy the GlusterFS ReplicaSets

Next we are going to deploy the Heketi Service Interface for GlusterFS as well as the GlusterFS ReplicaSets. To do this, we need to know the Kubernetes worker nodes that we are going to be deploying to. Let's start by obtaining a list of all our Kubernetes worker nodes: 

```
# kubectl get nodes
NAME                           STATUS    AGE
ip-172-20-0-217.ec2.internal   Ready     3h
ip-172-20-0-218.ec2.internal   Ready     3h
ip-172-20-0-219.ec2.internal   Ready     3h
ip-172-20-0-220.ec2.internal   Ready     3h
```
Within that node list you should see the servers you have identified to run GlusterFS (at least 3 of them). There is a  glusterfs-deployment.json file within this repo that you cloned. Run the following command for each of the 3 (or more) servers that you've set aside to run GlusterFS while making sure that the hostname or IP you provide matches the value for the node name returned from the previous command:

`# sed 's/<GLUSTERFS_NODE>/<hostname or IP>/g' glusterfs-deployment.json | kubectl create -f -`

Example:

`# sed 's/<GLUSTERFS_NODE>/ip-172-20-0-219\.ec2.internal/g' glusterfs-deployment.json | kubectl create -f -`


Verify that the deployments ran successfully by running the following command:
```
kubectl get deployments
NAME                                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
glusterfs-ip-172-20-0-217.ec2.internal   1         1         1            1           2m
glusterfs-ip-172-20-0-218.ec2.internal   1         1         1            1           2m
glusterfs-ip-172-20-0-219.ec2.internal   1         1         1            1           3m
```

3) Next we need to deploy the Pod and Service Heketi Service Interface to the GlusterFS cluster. Edit deploy-heketi-deployment.json file and change the following JSON parameters.
* Note the secret for the service account 

```
$ heketi_secret=$(kubectl get sa heketi -o="go-template" --template="{{(index .secrets 0).name}}")
```

* Deploy deploy-heketi.  Before deploying you will need to determine the Kubernetes API endpoint and namespace.

In this example, we will use `https://1.1.1.1:443` as our Kubernetes API endpoint

```
$ sed -e "s#<HEKETI_KUBE_SECRETNAME>#\"$heketi_secret\"#" \
      -e "s#<HEKETI_KUBE_APIHOST>#\"http://1.1.1.1:443\"#" deploy-heketi-deployment.json | kubectl create -f -

Once you've saved your changes, go ahead and submit the file and verify everything is running properly as demonstrated below:

```
# kubectl create -f deploy-heketi-deployment.json 
service "deploy-heketi" created
deployment "deploy-heketi" created

# kubectl get pods
NAME                                                      READY     STATUS    RESTARTS   AGE
deploy-heketi-1211581626-2jotm                            1/1       Running   0          35m
glusterfs-ip-172-20-0-217.ec2.internal-1217067810-4gsvx   1/1       Running   0          1h
glusterfs-ip-172-20-0-218.ec2.internal-2001140516-i9dw9   1/1       Running   0          1h
glusterfs-ip-172-20-0-219.ec2.internal-2785213222-q3hba   1/1       Running   0          1h<commit_msg>setup-guide: fix formatting<commit_after># Setup Guide

This guide contains the setup instructions for *gluster4kube*.


## Infrastructure Requirements

* A running Kubernetes cluster with at least three Kubernetes worker nodes that each have an available raw block device attached (like an EBS Volume or a local disk).
* The three Kubernetes nodes intended to run the GlusterFS Pods must have the appropriate ports opened for GlusterFS communication. Run the following commands on each of the nodes.
```
iptables -N HEKETI
iptables -A HEKETI -p tcp -m state --state NEW -m tcp --dport 24007 -j ACCEPT
iptables -A HEKETI -p tcp -m state --state NEW -m tcp --dport 24008 -j ACCEPT
iptables -A HEKETI -p tcp -m state --state NEW -m tcp --dport 2222 -j ACCEPT
iptables -A HEKETI -p tcp -m state --state NEW -m multiport --dports 49152:49251 -j ACCEPT
```
**** TODO: Needs to be updated to demonstrate how to persist your new iptables configuration so that it is still the same after a reboot of the host. Also needs an update to link in the new HEKETI iptables chain so that it is active. Right now the chain is not referenced so it is just set up but not being used. ****

## Client Setup

Heketi provides a CLI that provides users with a means to administer the deployment and configuration of GlusterFS in Kubernetes. [Download and install the heketi-cli](https://github.com/heketi/heketi/releases/tag/v3.0.0) on your client machine (usually your laptop).

## Kubernetes Deployment

1) The first thing we need to do is create a Service Account so that the GlusterFS containers can communicate securely with each other. 

Create the serviceaccount file as demonstrated below

```
# vi heketi-serviceaccount.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: heketi

# kubectl create -f heketi-serviceaccount.yaml
```

Verify the serviceaccount was created successfully
```
# kubectl get serviceaccount
NAME       SECRETS   AGE
default    1         27d
heketi     1         15s
```

Verify the appropriate secrets were created with the serviceaccount
```
# kubectl get secrets
NAME                   TYPE                                  DATA      AGE
default-token-4wtpz    kubernetes.io/service-account-token   <PASSWORD>
heketi-token-dz<PASSWORD>     kubernetes.io/service-account-token   3         25s
```

2) Deploy the GlusterFS ReplicaSets

Next we are going to deploy the Heketi Service Interface for GlusterFS as well as the GlusterFS ReplicaSets. To do this, we need to know the Kubernetes worker nodes that we are going to be deploying to. Let's start by obtaining a list of all our Kubernetes worker nodes: 

```
# kubectl get nodes
NAME                           STATUS    AGE
ip-172-20-0-217.ec2.internal   Ready     3h
ip-172-20-0-218.ec2.internal   Ready     3h
ip-172-20-0-219.ec2.internal   Ready     3h
ip-172-20-0-220.ec2.internal   Ready     3h
```

Within that node list you should see the servers you have identified to run GlusterFS (at least 3 of them). There is a  glusterfs-deployment.json file within this repo that you cloned. Run the following command for each of the 3 (or more) servers that you've set aside to run GlusterFS while making sure that the hostname or IP you provide matches the value for the node name returned from the previous command:

`# sed 's/<GLUSTERFS_NODE>/<hostname or IP>/g' glusterfs-deployment.json | kubectl create -f -`

Example:

`# sed 's/<GLUSTERFS_NODE>/ip-172-20-0-219\.ec2.internal/g' glusterfs-deployment.json | kubectl create -f -`


Verify that the deployments ran successfully by running the following command:
```
kubectl get deployments
NAME                                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
glusterfs-ip-172-20-0-217.ec2.internal   1         1         1            1           2m
glusterfs-ip-172-20-0-218.ec2.internal   1         1         1            1           2m
glusterfs-ip-172-20-0-219.ec2.internal   1         1         1            1           3m
```

3) Next we need to deploy the Pod and Service Heketi Service Interface to the GlusterFS cluster. Edit deploy-heketi-deployment.json file and change the following JSON parameters.
* Note the secret for the service account 

```
$ heketi_secret=$(kubectl get sa heketi -o="go-template" --template="{{(index .secrets 0).name}}")
```

* Deploy deploy-heketi.  Before deploying you will need to determine the Kubernetes API endpoint and namespace.

In this example, we will use `https://1.1.1.1:443` as our Kubernetes API endpoint

```
$ sed -e "s#<HEKETI_KUBE_SECRETNAME>#\"$heketi_secret\"#" \
      -e "s#<HEKETI_KUBE_APIHOST>#\"http://1.1.1.1:443\"#" deploy-heketi-deployment.json | kubectl create -f -
```

Once you've saved your changes, go ahead and submit the file and verify everything is running properly as demonstrated below:

```
# kubectl create -f deploy-heketi-deployment.json 
service "deploy-heketi" created
deployment "deploy-heketi" created

# kubectl get pods
NAME                                                      READY     STATUS    RESTARTS   AGE
deploy-heketi-1211581626-2jotm                            1/1       Running   0          35m
glusterfs-ip-172-20-0-217.ec2.internal-1217067810-4gsvx   1/1       Running   0          1h
glusterfs-ip-172-20-0-218.ec2.internal-2001140516-i9dw9   1/1       Running   0          1h
glusterfs-ip-172-20-0-219.ec2.internal-2785213222-q3hba   1/1       Running   0          1h
{'fraction_non_alphanumeric': 0.07542106237275588, 'fraction_numerical': 0.04682583749768647, 'mean_word_length': 2.734277816171389, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '190268', 'n_tokens_mistral': 3591, 'n_tokens_neox': 3228, 'n_words': 1294}