Backup Etcd data on OpenShift 4.x to AWS S3 Bucket
You have a running OpenShift Cluster powering your production microservices and worried about etcd data backup?. In this guide we show you how to easily backup etcd and push the backup data to AWS S3 object store. An etcd is a key-value store for OpenShift Container Platform, which persists the state of all resource objects.
In any OpenShift Cluster administration, it is a good and recommended practice to back up your cluster’s etcd data regularly and store it in a secure location. The ideal location for data storage is outside the OpenShift Container Platform environment. This can be an NFS server share, secondary server in your Infrastructure or in a Cloud environment.
The other recommendation is taking etcd backups during non-peak usage hours, as the action is blocking in nature. Ensure etcd backup operation is performed after any OpenShift Cluster upgrade. The importance of this is that during cluster restoration, an etcd backup taken from the same z-stream release must be used. As an example, an OpenShift Container Platform 4.6.3 cluster must use an etcd backup that was taken from 4.6.3.
Step 1: Login to one Master Node in the Cluster
The etcd cluster backup has to be performed on a single invocation of the backup script on a master host. Do not take a backup for each master host.
Login to one master node either through SSH or debug session:
# SSH Access
$ ssh core@
# Debug session
$ oc debug node/
For a debug session you need to change your root directory to the host:
sh-4.6# chroot /host
If the cluster-wide proxy is enabled, be sure that you have exported the
HTTPS_PROXY environment variables.
Step 2: Perform etcd Backup on OpenShift 4.x
An OpenShift cluster access as a user with the
cluster-admin role is required to perform this operation.
Before you proceed check to confirm if proxy is enabled:
$ oc get proxy cluster -o yaml
If you have proxy enabled, httpProxy, httpsProxy, and noProxy fields will have the values set.
Run the cluster-backup.sh script to initiate etcd backup process. You should pass a path where backup is saved.
$ mkdir /home/core/etcd_backups
$ sudo /usr/local/bin/cluster-backup.sh /home/core/etcd_backups
Here is my command execution output:
etcdctl version: 3.3.18
API version: 3.3
found latest kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-115
found latest kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-24
found latest kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-26
found latest etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-11
Snapshot saved at /home/core/etcd_backups/snapshot_2021-03-16_134036.db
snapshot db and kube resources are successfully saved to /home/core/etcd_backups
List files in the backup directory:
$ ls -1 /home/core/etcd_backups/
$ du -sh /home/core/etcd_backups/*
There will be two files in the backup:
snapshot_.db: This file is the etcd snapshot.
static_kuberesources_.tar.gz: This file contains the resources for the static pods. If etcd encryption is enabled, it also contains the encryption keys for the etcd snapshot.
Step 3: Push the Backup to AWS S3 (From Bastion Server)
Login from Bastion Server and copy backup files.
scp -r core@serverip:/home/core/etcd_backups ~/
Install AWS CLI tool:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
Install unzip tool:
sudo yum -y install unzip
Extract downloaded file:
Install AWS CLI:
$ sudo ./aws/install
You can now run: /usr/local/bin/aws --version
Confirm installation by checking the version:
$ aws --version
aws-cli/2.1.30 Python/3.8.8 Linux/3.10.0-957.el7.x86_64 exe/x86_64.rhel.7 prompt/off
Create OpenShift Backups bucket:
$ aws s3 mb s3://openshiftbackups
Create an IAM User:
$ aws iam create-user --user-name backupsonly
Create AWS Policy for Backups user — user able to write to S3 only:
Apply the policy:
aws iam create-policy --policy-name upload-only-policy --policy-document file://aws-s3-uploads-policy.json
Assign AWS Policy to IAM User:
aws iam attach-user-policy --policy-arn arn:aws:iam:::policy/upload-only-policy --user-name backupsonly
You can now create an access key for an IAM user to test:
$ aws iam create-access-key --user-name backupsonly
Take note of the Access and Secret Key IDs and use it in configuration:
$ aws configure # On OCP Bastion server
- AWS Access Key ID
- AWS Secret Access Key
- Default region
Try uploading the files to S3 bucket:
$ aws s3 cp etcd_backups/ s3://openshiftbackups/etcd --recursive
upload: etcd_backups/static_kuberesources_2021-03-16_134036.tar.gz to s3://openshiftbackups/etcd/static_kuberesources_2021-03-16_134036.tar.gz
upload: etcd_backups/snapshot_2021-03-16_134036.db to s3://openshiftbackups/etcd/snapshot_2021-03-16_134036.db
$ aws s3 ls s3://openshiftbackups/etcd/
2021-03-16 16:00:59 1549340704 snapshot_2021-03-16_134036.db
2021-03-16 16:00:59 77300 static_kuberesources_2021-03-16_134036.tar.gz
Step 4: Automated Backups to AWS S3 (From Bastion Server)
We can do a script which will perform the following:
- Login from bastion to master node
- Initiate backup of etcd
- Copy backup data from master node to the bastion server
- Delete backup data on master node
- Copy backup data to the S3 bucket
- Delete local data upon successful upload to S3
Create script file on Bastion server:
$ vim backup_etcd_s3.sh
Here is the script which can be modified further for more advanced use case.
# Create backups directory if doesn't exist
[ -d $BACKUPS_DIR ] && echo "Directory Exists" || mkdir $BACKUPS_DIR
# Login and run backup
ssh $USERNAME@$MASTER_NAME 'mkdir /home/core/etcd_backups' 2>/dev/null
ssh $USERNAME@$MASTER_NAME 'sudo /usr/local/bin/cluster-backup.sh /home/core/etcd_backups'
scp -r $USERNAME@$MASTER_NAME:/home/core/etcd_backups/* $BACKUPS_DIR/
# clean etcd backups directory on the master node
if [ $RESULT -eq 0 ]; then
ssh $USERNAME@$MASTER_NAME 'rm -rf /home/core/etcd_backups/*'
# Backup to aws s3
aws s3 cp $BACKUPS_DIR/ s3://$S3_BUCKET --recursive
# List bucket contents
aws s3 ls s3://$S3_BUCKET/
# Clean backups older than 1 day
#find $BACKUPS_DIR/ -mtime +1 -exec rm \;
find $BACKUPS_DIR/ -type f -mtime +1 -delete
Using Cron Job that runs 3am:
$ crontab -e
0 3 * * * /path/to/backup_etcd_s3.sh
In this article we’ve looked at how you can backup OpenShift etcd and push the data to an S3 bucket. In our next guide we could discuss on how you can perform a restore from the backup.