Backup BOSH deployments and BOSH Director with BBR on an object storage S3

PKS Enterprise does not currently offer automatic integrated backups, we must go through the utility offered by the cloudfoundry BOSH Backup and Restore (BBR) in order to backup BOSH and/or BOSH Director deployments.

In my case, these backups were implemented on the PKS-CLI, a RHEL 8.1 VM, which serves as a gateway to administer the components of PKS Enterprise.

I created a crontab which runs a script every night at 2.30am:

  1. The script will backup BOSH Director and BOSH Deployments (clusters) locally on the PKS-CLI machine.
  2. An rclone command is then launched to copy the local backups to an S3 bucket.
  3. Then a cleaning of local backups is started.
  4. Then a check in order to keep only the backups of the last 7 days in the S3 bucket.

Backup set up with the BBR tool
https://docs.cloudfoundry.org/bbr/installing.html
https://docs.vmware.com/en/VMware-Enterprise-PKS/1.7/vmware-enterprise-pks-17/GUID-bbr-backup.html

BBR can back up the following components:

  • BOSH Director
  • Enterprise PKS control plane API VM and its ETCD database
  • Enterprise PKS control plane database VM (MySQL)
  • Enterprise PKS cluster data, from the clusters’ ETCD databases

BBR cannot back up the following components:

  • Harbor tile
  • Persistent volumes attached to nodes
  • Network resources. For example, load balancers to the cluster

Backup BOSH Director

BBR allows you to backup BOSH director and deployments.
For BOSH director, we need the private key we can find it on ops-manager/BOSH Director/Credentials -> bbr ssh credentials

To format the key:
printf — “YOUR-PRIVATE-KEY” > /home/pkscli/pks/director_bbr_ssh_private_key.pem

We need the IP of BOSH too.

We can now do the backup of BOSH Director using the private key and the BOSH IP:
[pkscli@vsissp-pkscli-p bosh-director]# cd /home/pkscli/pks/backup/bosh-director
[pkscli@vsissp-pkscli-p bosh-director]# bbr director –host 10.180.23.10 –username bbr –private-key-path /home/pkscli/pks/director_bbr_ssh_private_key.pem backup > bbr.out 2> bbr.err

Backup BOSH Deployments

In order to backups BOSH Deployments, we need the root_ca_cert.crt of ops-manager and the BOSH_CLIENT_SECRET.

We can fin the root_ca_cert.crt on the settings of “ops-manager/Advanced Options/Download ROOT CA Cert”.

The BOSH Client Secret can be find under “ops-manager/BOSH/Credentials/Bosh Commandline Credentials”.

We can now use the command to backup all the BOSH Deployments.
[pkscli@vsissp-pkscli-p pks]$ cd /home/pkscli/pks/backup/bosh-deployments
[pkscli@vsissp-pkscli-p bbr]$ BOSH_CLIENT_SECRET=YYYYYYZZZZZZXXXX nohup bbr deployment –target bosh.example.com –username ops_manager –ca-cert /home/pkscli/root_ca_cert –all-deployments backup > bbr.out 2> bbr.err

rclone copy to storage object S3

The backups are now copied locally, then we want to do a rclone copy which allows us to copy to an S3.

Installation of rclone on RHEL 8.1
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo dnf upgrade
sudo yum install snapd
sudo systemctl enable –now snapd.socket
sudo ln -s /var/lib/snapd/snap /snap
sudo yum install rclone

S3 configuration for rclone.
I’ve done it at pkscli user level because the crontab will be at a pkscli user level and not root.
In my case, we have a local S3 storage on-premise. If you have an Amazon S3 change the settings and enter your credentials.

[pkscli@vsissp-pkscli-p pks]$ rclone config
2020/05/04 12:31:25 NOTICE: Config file “/home/pkscli/.config/rclone/rclone.conf” not found – using defaults
No remotes found – make a new one

  1. n) New remote
  2. s) Set configuration password
  3. q) Quit config

n/s/q> n
name> remote
Type of storage to configure.
Enter a string value. Press Enter for the default (“”).
Choose a number from below, or type in your own value
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc) “s3”
Storage> s3

1 / Amazon Web Services (AWS) S3 “AWS”
11 / Any other S3 compatible provider “Other”
provider> 11

1 / Enter AWS credentials in the next step “false”
2 / Get AWS credentials from the environment (env vars or IAM) “true”
env_auth> 1

AWS Access Key ID.
access_key_id> XXXXXXZZZZYYYYY

AWS Secret Access Key (password)
secret_access_key> YYYYYZZZZZXXXXX

Region to connect to.  
region> local

Endpoint for S3 API.
endpoint> https://s3.example.ch

Location constraint – must be set to match the Region.
location_constraint> local

Canned ACL used when creating buckets and storing or copying objects.
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. “bucket-owner-full-control”
acl> 6

Edit advanced config? (y/n)

  1. y) Yes
  2. n) No (default)

y/n> n

  1. y) Yes this is OK (default)
  2. e) Edit this remote
  3. d) Delete this remote

y/e/d> y

Current remotes:
Name                 Type
====                 ====
remote               s3

Copy a local folder into to a S3 Bucket
[pkscli@vsissp-pkscli-p pks]$ rclone -v copy /home/pkscli/pks/bbr remote:svc1219-XXXXYYYYZZZZ/bbr-04-05-2020

See what a S3 bucket contains
[pkscli@vsissp-pkscli-p pks]$ rclone ls remote:svc1219-XXXXYYYYZZZZ

Delete a folder in a S3 bucket
[pkscli@vsissp-pkscli-p pks]$ rclone delete remote:svc1219-XXXXYYYYZZZZ/bbr-04-05-2020

One bash script to backup them all

I created a bash script to perform the requested commands.

You have to make it executable with [pkscli@vsissp-pkscli-p scripts]$ chmod u+x /home/pkscli/pks/scripts/backup.sh

#!/bin/bash
cd /home/pkscli/pks/backup
mkdir bosh-director
mkdir bosh-deployments

# Backup BOSH Deployments
cd /home/pkscli/pks/backup/bosh-deployments
BOSH_CLIENT_SECRET=YYYYYYZZZZZZXXXX nohup bbr deployment –target bosh.example.com –username ops_manager –ca-cert /home/pkscli/root_ca_cert –all-deployments backup > bbr.out 2> bbr.err

# Backup Bosh Director
cd /home/pkscli/pks/backup/bosh-director
bbr director –host 10.180.23.10 –username bbr –private-key-path /home/pkscli/pks/director_bbr_ssh_private_key.pem backup > bbr.out 2> bbr.err

# Copy to the S3 bucket adding the date
rclone -v copy /home/pkscli/pks/backup remote:svc1219-XXXXYYYYZZZZ/backup-$(date +’%d-%m-%Y’)

# Clean up local backup folder
rm -rf /home/pkscli/pks/backup/*

# Clean older backups than 7 days on the S3 bucket.
rclone delete remote:svc1219-XXXXYYYYZZZZ –min-age 7d

Crontab (scheduled script launch)

Set up a cron which execute the previously created script every morning at 2.30am.

Installation of cron RHEL 8.1. I did it as user pkscli.
[pkscli@vsissp-pkscli-p cron]$ sudo yum install cronie
[pkscli@vsissp-pkscli-p cron]$ sudo systemctl enable crond.service
[pkscli@vsissp-pkscli-p cron]$ sudo systemctl start crond.service

We can now created the crontab.
[root@vsissp-pkscli-p scripts]$ crontab -u pkscli -e
SHELL=/bin/bash
PATH=/bin:/sbin:/usr/bin:/usr/sbin
MAILTO=root
30 2 * * * /home/pkscli/pks/scripts/backup.sh >> /home/pkscli/pks/scripts/logs/backup-`date +\%d\%m\%Y`.log 2>&1

To see the crontab list for user pkscli:
[root@vsissp-pkscli-p bbr]# crontab -u pkscli -l

To see the cron logs:
[root@vsissp-pkscli-p bbr]# tail -f /var/log/cron
May  6 12:57:44 vsissp-pkscli-p crontab[1943]: (root) BEGIN EDIT (root)
May  6 12:58:31 vsissp-pkscli-p crontab[1943]: (root) REPLACE (root)
May  6 12:58:31 vsissp-pkscli-p crontab[1943]: (root) END EDIT (root)
May  6 12:59:01 vsissp-pkscli-p CROND[5038]: (pkscli) CMD (/home/pkscli/pks/scripts/backup.sh >> /home/pkscli/pks/scripts/logs/backup-`date +%d%m%Y`.log 2>&1)

Leave a Comment