PKS Enterprise does not currently offer automatic integrated backups, we must go through the utility offered by the cloudfoundry BOSH Backup and Restore (BBR) in order to backup BOSH and/or BOSH Director deployments.
In my case, these backups were implemented on the PKS-CLI, a RHEL 8.1 VM, which serves as a gateway to administer the components of PKS Enterprise.
I created a crontab which runs a script every night at 2.30am:
- The script will backup BOSH Director and BOSH Deployments (clusters) locally on the PKS-CLI machine.
- An rclone command is then launched to copy the local backups to an S3 bucket.
- Then a cleaning of local backups is started.
- Then a check in order to keep only the backups of the last 7 days in the S3 bucket.
Backup set up with the BBR tool
https://docs.cloudfoundry.org/bbr/installing.html
https://docs.vmware.com/en/VMware-Enterprise-PKS/1.7/vmware-enterprise-pks-17/GUID-bbr-backup.html
BBR can back up the following components:
- BOSH Director
- Enterprise PKS control plane API VM and its ETCD database
- Enterprise PKS control plane database VM (MySQL)
- Enterprise PKS cluster data, from the clusters’ ETCD databases
BBR cannot back up the following components:
- Harbor tile
- Persistent volumes attached to nodes
- Network resources. For example, load balancers to the cluster
Backup BOSH Director
BBR allows you to backup BOSH director and deployments.
For BOSH director, we need the private key we can find it on ops-manager/BOSH Director/Credentials -> bbr ssh credentials
To format the key:
printf — “YOUR-PRIVATE-KEY” > /home/pkscli/pks/director_bbr_ssh_private_key.pem

We need the IP of BOSH too.

We can now do the backup of BOSH Director using the private key and the BOSH IP:
[pkscli@vsissp-pkscli-p bosh-director]# cd /home/pkscli/pks/backup/bosh-director
[pkscli@vsissp-pkscli-p bosh-director]# bbr director –host 10.180.23.10 –username bbr –private-key-path /home/pkscli/pks/director_bbr_ssh_private_key.pem backup > bbr.out 2> bbr.err
Backup BOSH Deployments
In order to backups BOSH Deployments, we need the root_ca_cert.crt of ops-manager and the BOSH_CLIENT_SECRET.
We can fin the root_ca_cert.crt on the settings of “ops-manager/Advanced Options/Download ROOT CA Cert”.

The BOSH Client Secret can be find under “ops-manager/BOSH/Credentials/Bosh Commandline Credentials”.


We can now use the command to backup all the BOSH Deployments.
[pkscli@vsissp-pkscli-p pks]$ cd /home/pkscli/pks/backup/bosh-deployments
[pkscli@vsissp-pkscli-p bbr]$ BOSH_CLIENT_SECRET=YYYYYYZZZZZZXXXX nohup bbr deployment –target bosh.example.com –username ops_manager –ca-cert /home/pkscli/root_ca_cert –all-deployments backup > bbr.out 2> bbr.err
rclone copy to storage object S3
The backups are now copied locally, then we want to do a rclone copy which allows us to copy to an S3.
Installation of rclone on RHEL 8.1
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo dnf upgrade
sudo yum install snapd
sudo systemctl enable –now snapd.socket
sudo ln -s /var/lib/snapd/snap /snap
sudo yum install rclone
S3 configuration for rclone.
I’ve done it at pkscli user level because the crontab will be at a pkscli user level and not root.
In my case, we have a local S3 storage on-premise. If you have an Amazon S3 change the settings and enter your credentials.
[pkscli@vsissp-pkscli-p pks]$ rclone config
2020/05/04 12:31:25 NOTICE: Config file “/home/pkscli/.config/rclone/rclone.conf” not found – using defaults
No remotes found – make a new one
- n) New remote
- s) Set configuration password
- q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Enter a string value. Press Enter for the default (“”).
Choose a number from below, or type in your own value
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc) “s3”
Storage> s3
1 / Amazon Web Services (AWS) S3 “AWS”
11 / Any other S3 compatible provider “Other”
provider> 11
1 / Enter AWS credentials in the next step “false”
2 / Get AWS credentials from the environment (env vars or IAM) “true”
env_auth> 1
AWS Access Key ID.
access_key_id> XXXXXXZZZZYYYYY
AWS Secret Access Key (password)
secret_access_key> YYYYYZZZZZXXXXX
Region to connect to.
region> local
Endpoint for S3 API.
endpoint> https://s3.example.ch
Location constraint – must be set to match the Region.
location_constraint> local
Canned ACL used when creating buckets and storing or copying objects.
Choose a number from below, or type in your own value
1 / Owner gets FULL_CONTROL. No one else has access rights (default).
6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. “bucket-owner-full-control”
acl> 6
Edit advanced config? (y/n)
- y) Yes
- n) No (default)
y/n> n
- y) Yes this is OK (default)
- e) Edit this remote
- d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
remote s3
Copy a local folder into to a S3 Bucket
[pkscli@vsissp-pkscli-p pks]$ rclone -v copy /home/pkscli/pks/bbr remote:svc1219-XXXXYYYYZZZZ/bbr-04-05-2020
See what a S3 bucket contains
[pkscli@vsissp-pkscli-p pks]$ rclone ls remote:svc1219-XXXXYYYYZZZZ
Delete a folder in a S3 bucket
[pkscli@vsissp-pkscli-p pks]$ rclone delete remote:svc1219-XXXXYYYYZZZZ/bbr-04-05-2020
One bash script to backup them all
I created a bash script to perform the requested commands.
You have to make it executable with [pkscli@vsissp-pkscli-p scripts]$ chmod u+x /home/pkscli/pks/scripts/backup.sh
#!/bin/bash
cd /home/pkscli/pks/backup
mkdir bosh-director
mkdir bosh-deployments
# Backup BOSH Deployments
cd /home/pkscli/pks/backup/bosh-deployments
BOSH_CLIENT_SECRET=YYYYYYZZZZZZXXXX nohup bbr deployment –target bosh.example.com –username ops_manager –ca-cert /home/pkscli/root_ca_cert –all-deployments backup > bbr.out 2> bbr.err
# Backup Bosh Director
cd /home/pkscli/pks/backup/bosh-director
bbr director –host 10.180.23.10 –username bbr –private-key-path /home/pkscli/pks/director_bbr_ssh_private_key.pem backup > bbr.out 2> bbr.err
# Copy to the S3 bucket adding the date
rclone -v copy /home/pkscli/pks/backup remote:svc1219-XXXXYYYYZZZZ/backup-$(date +’%d-%m-%Y’)
# Clean up local backup folder
rm -rf /home/pkscli/pks/backup/*
# Clean older backups than 7 days on the S3 bucket.
rclone delete remote:svc1219-XXXXYYYYZZZZ –min-age 7d
Crontab (scheduled script launch)
Set up a cron which execute the previously created script every morning at 2.30am.
Installation of cron RHEL 8.1. I did it as user pkscli.
[pkscli@vsissp-pkscli-p cron]$ sudo yum install cronie
[pkscli@vsissp-pkscli-p cron]$ sudo systemctl enable crond.service
[pkscli@vsissp-pkscli-p cron]$ sudo systemctl start crond.service
We can now created the crontab.
[root@vsissp-pkscli-p scripts]$ crontab -u pkscli -e
SHELL=/bin/bash
PATH=/bin:/sbin:/usr/bin:/usr/sbin
MAILTO=root
30 2 * * * /home/pkscli/pks/scripts/backup.sh >> /home/pkscli/pks/scripts/logs/backup-`date +\%d\%m\%Y`.log 2>&1
To see the crontab list for user pkscli:
[root@vsissp-pkscli-p bbr]# crontab -u pkscli -l
To see the cron logs:
[root@vsissp-pkscli-p bbr]# tail -f /var/log/cron
May 6 12:57:44 vsissp-pkscli-p crontab[1943]: (root) BEGIN EDIT (root)
May 6 12:58:31 vsissp-pkscli-p crontab[1943]: (root) REPLACE (root)
May 6 12:58:31 vsissp-pkscli-p crontab[1943]: (root) END EDIT (root)
May 6 12:59:01 vsissp-pkscli-p CROND[5038]: (pkscli) CMD (/home/pkscli/pks/scripts/backup.sh >> /home/pkscli/pks/scripts/logs/backup-`date +%d%m%Y`.log 2>&1)