Configure Kubernetes Dynamic Volume Provisioning With Heketi & GlusterFS

The Dynamic volume provisioning in Kubernetes allows storage volumes to be created on-demand, without manual Administrator intervention. When developers are doing deployments without dynamic provisioning, cluster administrators have to manually make calls to their cloud or storage provider to create new storage volumes, from where the PersistentVolumes are created.

This guide will discuss how you can achieve Dynamic Volume Provisioning on Kubernetes by using GlusterFS distributed storage solution and Heketi RESTful management interface. It is expected you have deployed Heketi and GlusterFS scale-out network-attached storage file system.

For Ceph, check:

• Ceph Persistent Storage for Kubernetes with Cephfs
• Persistent Storage for Kubernetes with Ceph RBD

How Dynamic Provisioning is configured in Kubernetes

In Kubernetes, dynamic volume provisioning is based on the API object StorageClass from the API group storage.k8s.io. As a cluster administrator, you’ll define as many StorageClass objects as needed, each specifying a volume plugin ( provisioner) that provisions a volume and the set of parameters to pass to that provisioner when provisioning.

So below are the steps you’ll use to configure Dynamic Volume Provisioning on Kubernetes using Gluster and Heketi API.

Setup GlusterFS and Heketi

It is expected you have a running Gluster and Heketi before you continue with configurations on the Kubernetes end. Refer to our guide below on setting them up.

• Setup GlusterFS Storage With Heketi on CentOS 8 / CentOS 7

At the moment we only have guide for CentOS, but we’re working on a deployment guide for Ubuntu/Debian systems.

For containerized setup, check: Setup Kubernetes / OpenShift Dynamic Persistent Volume Provisioning with GlusterFS and Heketi

Once the installation is done, proceed to step 2:

Create StorageClass Object on Kubernetes

We need to create a StorageClass object to enable dynamic provisioning for container platform users. The StorageClass objects define which provisioner should be used and what parameters should be passed to that provisioner when dynamic provisioning is invoked.

• Check your Heketi Cluster ID

$ heketi-cli cluster list

Clusters:

Id:b182cb76b881a0be2d44bd7f8fb07ea4 [file][block]

• Create Kubernetes Secret

Get a base64 format of your Heketi admin user password.

$ echo -n "PASSWORD" | base64

Then create a secret with the password for accessing Heketi.

$ vim gluster-secret.yaml

apiVersion: v1

kind: Secret

metadata:

  name: heketi-secret

  namespace: default

type: "kubernetes.io/glusterfs"

data:

  # echo -n "PASSWORD" | base64

  key: cGFzc3dvcmQ=

Where:

• cGFzc3dvcmQ= is the output of echo command.

Create the secret by running the command:

$ kubectl create -f gluster-secret.yaml

Confirm secret creation.

$ kubectl get secret

NAME                                               TYPE                                  DATA   AGE

heketi-secret                                      kubernetes.io/glusterfs               1      1d

• Create StorageClass

Below is a sample StorageClass for GlusterFS using Heketi.

$ cat glusterfs-sc.yaml

 kind: StorageClass

 apiVersion: storage.k8s.io/v1beta1

 metadata:

   name: gluster-heketi

 provisioner: kubernetes.io/glusterfs

 reclaimPolicy: Delete

 volumeBindingMode: Immediate

 allowVolumeExpansion: true

 parameters:

   resturl: "http://heketiserverip:8080" 

   restuser: "admin" 

   secretName: "heketi-secret"

   secretNamespace: "default"

   volumetype: "replicate:2"

   volumenameprefix: "k8s-dev"

   clusterid: "b182cb76b881a0be2d44bd7f8fb07ea4"

Where:

• gluster-heketi is the name of the StorageClass to be created.
• The valid options for reclaim policy are Retain, Delete or Recycle. The Delete policy means that a dynamically provisioned volume is automatically deleted when a user deletes the corresponding PersistentVolumeClaim.
• The volumeBindingMode field controls when volume binding and dynamic provisioning should occur. Valid options are Immediate & WaitForFirstConsumer. The Immediate mode indicates that volume binding and dynamic provisioning occurs once the PersistentVolumeClaim is created. The WaitForFirstConsumer mode delays the binding and provisioning of a PersistentVolume until a Pod using the PersistentVolumeClaim is created.
• The resturl is the URL of your heketi endpoint
• heketi-secret is the secret created for Heketi credentials.
• default is the name of namespace where secret was created
• replicate:2 indicated the default replication factor for Gluster Volumes created. For more HA, use 3.
• volumenameprefix: By default dynamically provisioned volumes have the naming schema of vol_UUID format. We have provided a desired volume name from storageclass. So the naming scheme will be: volumenameprefix_Namespace_PVCname_randomUUID
• b182cb76b881a0be2d44bd7f8fb07ea4 is the ID of the cluster obtained from the command heketi-cli cluster list

Another parameter that can be set is:

volumeoptions: "user.heketi.zone-checking strict"

The default setting/behavior is:

volumeoptions: "user.heketi.zone-checking none"

This forces Heketi to strictly place replica bricks in different zones. The required minimum number of nodes required to be present in different zones is 3 if the replica value is set to 3.

Once the file is created, run the following command to create the StorageClass object.

$ kubectl create -f gluster-sc.yaml

Confirm StorageClass creation.

$ kubectl get sc

NAME                            PROVISIONER                    AGE

glusterfs-heketi                kubernetes.io/glusterfs        1d

local-storage                   kubernetes.io/no-provisioner   30d

Step 2: Create PersistentVolumeClaim Object

When a user is requesting dynamically provisioned storage, a storage class should be included in the PersistentVolumeClaim.

Let’s create a 1GB request for storage:

$ vim glusterfs-pvc.yaml

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

 name: gluster-pvc

 annotations:

   volume.beta.kubernetes.io/storage-class: gluster-heketi

spec:

 accessModes:

  - ReadWriteMany

 resources:

   requests:

     storage: 1Gi

Create object:

$ kubectl create --save-config -f glusterfs-pvc.yaml

Confirm:

$ kubectl get  pvc

NAME             STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE

glusterfs-pvc    Bound     pvc-34b9b5e9-fbde-11e9-943f-00505692ee7e   1Gi        RWX            glusterfs-heketi  1d

After creation, you can use it in your deployments. To use the volume we reference the PVC in the YAML file of any Pod/Deployment like this for example:

apiVersion: v1

kind: Pod

metadata:

  name: gluster-pod

  labels:

    name: gluster-pod

spec:

  containers:

  - name: gluster-pod

    image: busybox

    command: ["sleep", "60000"]

    volumeMounts:

    - name: gluster-vol

      mountPath: /usr/share/busybox 

      readOnly: false

  volumes:

  - name: gluster-vol

    persistentVolumeClaim:

      claimName: glusterfs-pvc

That’s it for today. You should have a working Dynamic Volume Provisioning With Heketi & GlusterFS for your Kubernetes platform.

https://www.computingpost.com/configure-kubernetes-dynamic-volume-provisioning-with-heketi-glusterfs/?feed_id=4977&_unique_id=6332c18b14a1f