<commit_before>= Persistent Storage Using GlusterFS
{product-author}
{product-version}
:data-uri:
:icons:
:experimental:
:toc: macro
:toc-title:
:prewrap!:

toc::[]

== Overview
OpenShift can utilize persistent storage using Distributed File Systems (DFS)
like GlusterFS. Some familiarity with Kubernetes and Docker is assumed. It is
also assumed that there is access to an existing GlusterFS cluster and volume,
and that *glusterfs-fuse* has been installed on all OpenShift nodes in the
cluster.

The Kubernetes
link:../../architecture/additional_concepts/storage.html[persistent volume]
framework allows administrators to provision a cluster with persistent storage
and gives users a way to request those resources without having any knowledge of
the underlying infrastructure. Persistent volumes are not bound to a single
project or namespace; they can be shared across the OpenShift cluster.
link:../../architecture/additional_concepts/storage.html#persistent-volume-claims[Persistent
volume claims], however, are specific to a project or namespace and can be
requested by users.

[IMPORTANT]
====
High-availability of storage in the infrastructure is left to the underlying
storage provider.
====

[[gfs-provisioning]]

== Provisioning
Storage must exist in the underlying infrastructure before it can be mounted as
a volume in OpenShift. To provision GlusterFS volumes in OpenShift, the
following are required:

- A distinct list of servers in the Gluster cluster, to be defined as endpoints
- Existing Gluster volumes, to be defined in the persistent volume object
- The `*PersistentVolume*` API

You must also ensure *glusterfs-fuse* is installed on all OpenShift nodes in the cluster:

----
# yum install glusterfs-fuse
----

[[creating-gluster-endpoints]]

=== Creating Gluster Endpoints

In an endpoints definition, you must define the GlusterFS cluster as
`*EndPoints*` and include the IP and host name of your Gluster servers with the
port that you want to use. The port value can be any numeric value within the
accepted range of ports.

.Persistent Volume Endpoints Definition
====
[source,yaml]
----
apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
subsets:
  - addresses:
      - ip: 192.168.122.221 <1>
    ports:
      - port: 1
  - addresses:
      - ip: 192.168.122.222 <1>
    ports:
      - port: 1
----
<1> The `*ip*` values must be the actual IP addresses of a Gluster server, not
fully-qualified host names. This requirement could change in future releases.
====

Save your endpoints definition to a file, for example
*_gluster-endpoints.yaml_*, then create the endpoints:

====
----
$ oc create -f gluster-endpoints.yaml
endpoints "glusterfs-cluster" created
----
====

Verify that the endpoints were created:

====
----
$ oc get endpoints
NAME                ENDPOINTS                             AGE
docker-registry     10.1.0.3:5000                         4h
glusterfs-cluster   192.168.122.221:1,192.168.122.222:1   11s
kubernetes          172.16.35.3:8443                      4d
----
====

[[gfs-creating-persistent-volume]]

=== Creating the Persistent Volume

You must define your persistent volume in an object definition before creating
it in OpenShift:

.Persistent Volume Object Definition Using GlusterFS
====

[source,yaml]
----
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
  name: "gluster-default-volume" <1>
spec:
  capacity:
    storage: "2Gi" <2>
  accessModes:
    - "ReadWriteMany"
  glusterfs: <3>
    endpoints: "glusterfs-cluster" <4>
    path: "myVol1" <5>
    readOnly: false
  persistentVolumeReclaimPolicy: "Recycle"
----
<1> The name of the volume. This will be how it is identified via
link:../../architecture/additional_concepts/storage.html[persistent volume
claims] or from pods.
<2> The amount of storage allocated to this volume.
<3> This defines the volume type being used, in this case the *glusterfs*
plug-in.
<4> A reference to the endpoints object that defines the Gluster cluster,
created in link:#creating-gluster-endpoints[Creating Gluster Endpoints].
<5> This is the Gluster volume that will be used, as defined on your Gluster
servers.
====

Save your definition to a file, for example *_gluster-pv.yaml_*, and create the
persistent volume:

====
----
# oc create -f gluster-pv.yaml
persistentvolume "gluster-default-volume" created
----
====

Verify that the persistent volume was created:

====
----
# oc get pv
NAME                     LABELS    CAPACITY     ACCESSMODES   STATUS      CLAIM     REASON    AGE
gluster-default-volume   <none>    2147483648   RWX           Available                       2s
----
====

Users can then link:../../dev_guide/persistent_volumes.html[request storage
using persistent volume claims], which can now utilize your new persistent
volume.

[[gluster-volume-security]]

=== Volume Security
Users request storage with a `*PersistentVolumeClaim*`. This claim exists only
in the user's namespace and can only be referenced by a pod within that same
namespace. Any attempt to access a persistent volume across a namespace causes
the pod to fail.

Also, each node in the cluster should enable the following SELinux booleans by
running the following commands on each node:

====
----

[root@ose1 demo]# setsebool -P virt_use_fusefs 1
[root@ose1 demo]# setsebool -P virt_sandbox_use_fusefs 1

----
====

Additionally, permissions and ownership can be controlled as normal by the
Gluster server administrators using normal POSIX compliant security.

== SELinux and GlusterFS
By default, SELinux does not allow writing from a pod to a remote Gluster
server.

To enable writing to GlusterFS volumes with SELinux enforcing on each node, run:

----
$ sudo setsebool -P virt_sandbox_use_fusefs on
----

The `-P` option makes the bool persistent between reboots.

Gluster must be set up so that it is accessible by your pods. Either set the
export to be owned by the container's main UID, or give your pod group based
access using `*SuppplementalGroups*`. See link:pod_security_context.html[Volume
Security] for more information.
<commit_msg>Resolves #1139: Edit prompt in gluster volsec<commit_after>= Persistent Storage Using GlusterFS
{product-author}
{product-version}
:data-uri:
:icons:
:experimental:
:toc: macro
:toc-title:
:prewrap!:

toc::[]

== Overview
OpenShift can utilize persistent storage using Distributed File Systems (DFS)
like GlusterFS. Some familiarity with Kubernetes and Docker is assumed. It is
also assumed that there is access to an existing GlusterFS cluster and volume,
and that *glusterfs-fuse* has been installed on all OpenShift nodes in the
cluster.

The Kubernetes
link:../../architecture/additional_concepts/storage.html[persistent volume]
framework allows administrators to provision a cluster with persistent storage
and gives users a way to request those resources without having any knowledge of
the underlying infrastructure. Persistent volumes are not bound to a single
project or namespace; they can be shared across the OpenShift cluster.
link:../../architecture/additional_concepts/storage.html#persistent-volume-claims[Persistent
volume claims], however, are specific to a project or namespace and can be
requested by users.

[IMPORTANT]
====
High-availability of storage in the infrastructure is left to the underlying
storage provider.
====

[[gfs-provisioning]]

== Provisioning
Storage must exist in the underlying infrastructure before it can be mounted as
a volume in OpenShift. To provision GlusterFS volumes in OpenShift, the
following are required:

- A distinct list of servers in the Gluster cluster, to be defined as endpoints
- Existing Gluster volumes, to be defined in the persistent volume object
- The `*PersistentVolume*` API

You must also ensure *glusterfs-fuse* is installed on all OpenShift nodes in the cluster:

----
# yum install glusterfs-fuse
----

[[creating-gluster-endpoints]]

=== Creating Gluster Endpoints

In an endpoints definition, you must define the GlusterFS cluster as
`*EndPoints*` and include the IP and host name of your Gluster servers with the
port that you want to use. The port value can be any numeric value within the
accepted range of ports.

.Persistent Volume Endpoints Definition
====
[source,yaml]
----
apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
subsets:
  - addresses:
      - ip: 192.168.122.221 <1>
    ports:
      - port: 1
  - addresses:
      - ip: 192.168.122.222 <1>
    ports:
      - port: 1
----
<1> The `*ip*` values must be the actual IP addresses of a Gluster server, not
fully-qualified host names. This requirement could change in future releases.
====

Save your endpoints definition to a file, for example
*_gluster-endpoints.yaml_*, then create the endpoints:

====
----
$ oc create -f gluster-endpoints.yaml
endpoints "glusterfs-cluster" created
----
====

Verify that the endpoints were created:

====
----
$ oc get endpoints
NAME                ENDPOINTS                             AGE
docker-registry     10.1.0.3:5000                         4h
glusterfs-cluster   192.168.122.221:1,192.168.122.222:1   11s
kubernetes          172.16.35.3:8443                      4d
----
====

[[gfs-creating-persistent-volume]]

=== Creating the Persistent Volume

You must define your persistent volume in an object definition before creating
it in OpenShift:

.Persistent Volume Object Definition Using GlusterFS
====

[source,yaml]
----
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
  name: "gluster-default-volume" <1>
spec:
  capacity:
    storage: "2Gi" <2>
  accessModes:
    - "ReadWriteMany"
  glusterfs: <3>
    endpoints: "glusterfs-cluster" <4>
    path: "myVol1" <5>
    readOnly: false
  persistentVolumeReclaimPolicy: "Recycle"
----
<1> The name of the volume. This will be how it is identified via
link:../../architecture/additional_concepts/storage.html[persistent volume
claims] or from pods.
<2> The amount of storage allocated to this volume.
<3> This defines the volume type being used, in this case the *glusterfs*
plug-in.
<4> A reference to the endpoints object that defines the Gluster cluster,
created in link:#creating-gluster-endpoints[Creating Gluster Endpoints].
<5> This is the Gluster volume that will be used, as defined on your Gluster
servers.
====

Save your definition to a file, for example *_gluster-pv.yaml_*, and create the
persistent volume:

====
----
# oc create -f gluster-pv.yaml
persistentvolume "gluster-default-volume" created
----
====

Verify that the persistent volume was created:

====
----
# oc get pv
NAME                     LABELS    CAPACITY     ACCESSMODES   STATUS      CLAIM     REASON    AGE
gluster-default-volume   <none>    2147483648   RWX           Available                       2s
----
====

Users can then link:../../dev_guide/persistent_volumes.html[request storage
using persistent volume claims], which can now utilize your new persistent
volume.

[[gluster-volume-security]]

=== Volume Security
Users request storage with a `*PersistentVolumeClaim*`. This claim exists only
in the user's namespace and can only be referenced by a pod within that same
namespace. Any attempt to access a persistent volume across a namespace causes
the pod to fail.

Also, each node in the cluster should enable the following SELinux booleans by
running the following commands on each node:

====
----

# setsebool -P virt_use_fusefs 1
# setsebool -P virt_sandbox_use_fusefs 1

----
====

Additionally, permissions and ownership can be controlled as normal by the
Gluster server administrators using normal POSIX compliant security.

== SELinux and GlusterFS
By default, SELinux does not allow writing from a pod to a remote Gluster
server.

To enable writing to GlusterFS volumes with SELinux enforcing on each node, run:

----
$ sudo setsebool -P virt_sandbox_use_fusefs on
----

The `-P` option makes the bool persistent between reboots.

Gluster must be set up so that it is accessible by your pods. Either set the
export to be owned by the container's main UID, or give your pod group based
access using `*SuppplementalGroups*`. See link:pod_security_context.html[Volume
Security] for more information.

{'fraction_non_alphanumeric': 0.08557295960428689, 'fraction_numerical': 0.018136850783182192, 'mean_word_length': 3.6747591522157994, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 12, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2276860', 'n_tokens_mistral': 3609, 'n_tokens_neox': 3360, 'n_words': 1470}