Backup Etcd data on OpenShift 4.x to AWS S3 Bucket

Step 1: Login to one Master Node in the Cluster

# SSH Access

$ ssh core@



# Debug session

$ oc debug node/
sh-4.6# chroot /host

Step 2: Perform etcd Backup on OpenShift 4.x

$ oc get proxy cluster -o yaml
$ mkdir /home/core/etcd_backups

$ sudo /usr/local/bin/cluster-backup.sh /home/core/etcd_backups
3e53f83f3c02b43dfa8d282265c1b0f9789bcda827c4e13110a9b6f6612d447c

etcdctl version: 3.3.18

API version: 3.3

found latest kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-115

found latest kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-24

found latest kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-26

found latest etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-11

Snapshot saved at /home/core/etcd_backups/snapshot_2021-03-16_134036.db

snapshot db and kube resources are successfully saved to /home/core/etcd_backups
$ ls -1 /home/core/etcd_backups/

snapshot_2021-03-16_134036.db

static_kuberesources_2021-03-16_134036.tar.gz



$ du -sh /home/core/etcd_backups/*

1.5G /home/core/etcd_backups/snapshot_2021-03-16_134036.db

76K /home/core/etcd_backups/static_kuberesources_2021-03-16_134036.tar.gz
  • snapshot_.db: This file is the etcd snapshot.
  • static_kuberesources_.tar.gz: This file contains the resources for the static pods. If etcd encryption is enabled, it also contains the encryption keys for the etcd snapshot.

Step 3: Push the Backup to AWS S3 (From Bastion Server)

scp -r core@serverip:/home/core/etcd_backups ~/
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo yum -y install unzip
unzip awscli-exe-linux-x86_64.zip
$ sudo ./aws/install

You can now run: /usr/local/bin/aws --version
$ aws --version

aws-cli/2.1.30 Python/3.8.8 Linux/3.10.0-957.el7.x86_64 exe/x86_64.rhel.7 prompt/off
$ aws s3 mb s3://openshiftbackups

make_bucket: openshiftbackups
$ aws iam create-user --user-name backupsonly
cat >aws-s3-uploads-policy.json<

"Version": "2012-10-17",

"Statement": [



"Effect": "Allow",

"Action": [

"s3:Get*",

"s3:List*",

"s3:Put*"

],

"Resource": "*"



]



EOF
aws iam create-policy --policy-name upload-only-policy --policy-document file://aws-s3-uploads-policy.json
aws iam attach-user-policy --policy-arn arn:aws:iam:::policy/upload-only-policy --user-name backupsonly
$ aws iam create-access-key --user-name backupsonly



"AccessKey":

"UserName": "backupsonly",

"AccessKeyId": "AKIATWFKCYAHF74SCFEP",

"Status": "Active",

"SecretAccessKey": "3CgPHuU+q8vzoSdJisXscgvay3Cv7nVZMjDHpWFS",

"CreateDate": "2021-03-16T12:14:39+00:00"
$ aws configure # On OCP Bastion server
  • AWS Access Key ID
  • AWS Secret Access Key
  • Default region
$ aws s3 cp etcd_backups/ s3://openshiftbackups/etcd --recursive

upload: etcd_backups/static_kuberesources_2021-03-16_134036.tar.gz to s3://openshiftbackups/etcd/static_kuberesources_2021-03-16_134036.tar.gz

upload: etcd_backups/snapshot_2021-03-16_134036.db to s3://openshiftbackups/etcd/snapshot_2021-03-16_134036.db
$ aws s3 ls s3://openshiftbackups/etcd/

2021-03-16 16:00:59 1549340704 snapshot_2021-03-16_134036.db

2021-03-16 16:00:59 77300 static_kuberesources_2021-03-16_134036.tar.gz

Step 4: Automated Backups to AWS S3 (From Bastion Server)

  1. Login from bastion to master node
  2. Initiate backup of etcd
  3. Copy backup data from master node to the bastion server
  4. Delete backup data on master node
  5. Copy backup data to the S3 bucket
  6. Delete local data upon successful upload to S3
$ vim backup_etcd_s3.sh
#!/bin/bash

MASTER_NAME="master01.example.net"

USERNAME="core"

BACKUPS_DIR=~/etcd_backups

S3_BUCKET="openshiftbackups/etcd"

RESULT="$?"



# Create backups directory if doesn't exist

[ -d $BACKUPS_DIR ] && echo "Directory Exists" || mkdir $BACKUPS_DIR



# Login and run backup

ssh $USERNAME@$MASTER_NAME 'mkdir /home/core/etcd_backups' 2>/dev/null

ssh $USERNAME@$MASTER_NAME 'sudo /usr/local/bin/cluster-backup.sh /home/core/etcd_backups'

scp -r $USERNAME@$MASTER_NAME:/home/core/etcd_backups/* $BACKUPS_DIR/



# clean etcd backups directory on the master node

if [ $RESULT -eq 0 ]; then

ssh $USERNAME@$MASTER_NAME 'rm -rf /home/core/etcd_backups/*'

fi



# Backup to aws s3

aws s3 cp $BACKUPS_DIR/ s3://$S3_BUCKET --recursive

# List bucket contents

aws s3 ls s3://$S3_BUCKET/



# Clean backups older than 1 day

#find $BACKUPS_DIR/ -mtime +1 -exec rm \;

find $BACKUPS_DIR/ -type f -mtime +1 -delete
$ crontab -e

0 3 * * * /path/to/backup_etcd_s3.sh

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
ComputingPost

ComputingPost

ComputingPost — Linux Howtos, Tutorials, Guides, News, Tips and Tricks.