Backups

We do backups into two locations:

  • Most servers hosted on Hetzner backup into Hetzner “storage boxes” using sftp or borg.

  • Other servers backup into micrea using rsync or borg.

Each Hetzner server includes a free 100GB storage box that can be used for backups. We separately have a 2TB storage box used for backups that wouldn’t fit in the server’s corresponding backup space. This large storage box has multiple “sub-accounts” with restricted directory access, so multiple servers can do backups on it but can’t access the backups of the others.

There are two ways to connect to storage boxes. Port 22 provides access via SCP/SFTP only. We use the lftp program for this. This port requires a different ssh key format, but we’re currently using passwords to authenticate instead.

Port 23 provides SSH with more flexibility. It supports SCP, SFTP, rsync, and BorgBackup. It must be enabled explicitly on a given storage box on the Hetzner web control panel. This connection uses standard OpenSSH public keys, which must be copied to the storage box’s .ssh/authorized_keys.

Adding backups

Most configuration for backups is handled with Ansible, so most steps to add backups to a new server involve making changes to the kde-ansible repo and running the playbook; but a few manual steps are required too.

The instructions differ depending on whether the backup destination is Hetzner or our own server.

Adding backups to Hetzner storage

First, if you’re adding a new Hetzner backup box, or a new sub-account on the shared storage box, add the credentials for it to production/group_vars/all/backups-vault.yml.

Add hetzner_backup_host: xxxx to the server’s hostvars, replacing xxxx with the ID of the backup credentials as listed in backups-vault.yml. If using Borg, also add backup_apt_dependencies: [borgbackup].

If this is the first server using this storage box, also add backup_size_logging: yes. This will make the server log the used space in the storage box into the monitoring system. For each storage box or sub-account there should be only one server with this enabled.

Add the backup script to roles/kde-backup/templates/backup-{host}.sh. Usually it involves copying and pasting things from the backup scripts for other hosts. Don’t put any passwords directly here; use Ansible variables.

Finally, add the server to backups group in the inventory (production/hosts).

Run python3 scripts/validate-backup-config.py. This will do some sanity checks, and for example warn if a backup script uses borg but it’s not listed in backup_apt_dependencies.

Now you can deploy the changes by running the playbooks:

  • backups.yml: installs the needed packages (lftp and borgbackup if enabled), installs the backup script into bin/run-backup.sh, and configures the backup cronjob.

  • monitoring-agent.yml: modifies telegraf config to monitor when the backup script runs.

  • backup-size-log.yml: installs script and cronjob to log the storage box disk usage (if backup_size_logging was set).

If not using Borg, that should be all. Run the backup script manually once with bash -x bin/run-backup.sh to ensure everything is working, and you’re done.

If using Borg, we need several more manual steps: we need to setup public key authentication, and create the Borg repositories. Ensure there is an ssh key in /root/.ssh (use ssh-keygen if not). Run lftp and copy the key to the storage box’s authorized_keys:

lftp :~> connect sftp://u12345.your-backup.de -u u12345,aBcDeFgHiJkLmN
lftp :~> mkdir .ssh
lftp :~> put .ssh/id_rsa.pub -o .ssh/authorized_keys

# also create the root directory for borg
lftp :~> mkdir borg-backups

Now to create the repositories, set the BORG_PASSPHRASE and BORG_REPO environment variables and run borg init --encryption=repokey for each. When setting BORG_PASSPHRASE you may want to prefix the command with a space so that it’s not stored in the bash history. For example, if the backup script contains this:

# Prepare to run Borg backups
export BORG_PASSPHRASE='AbCdEfGhIjKl'

BORG_SERVER="ssh://u12345@u12345.your-backup.de:23"

# Backup databases
export BORG_REPO="$BORG_SERVER/./borg-backups/mimi-databases"
mysqldump ... | borg create ...

# Backup home directories
export BORG_REPO="$BORG_SERVER/./borg-backups/mimi-homes"
borg create ...

you use this to initialize them:

# prefix with space to keep out of bash history
 export BORG_PASSPHRASE='AbCdEfGhIjKl'

BORG_SERVER="ssh://u12345@u12345.your-backup.de:23"

export BORG_REPO="$BORG_SERVER/./borg-backups/mimi-databases"
borg init --encryption=repokey

export BORG_REPO="$BORG_SERVER/./borg-backups/mimi-homes"
borg init --encryption=repokey

Finally, run the backup script manually once with bash -x bin/run-backup.sh to ensure everything is working.

Adding backups to micrea

Add gohma_backup_user: foobackup and gohma_backup_home: BACKUP.foo.kde.org to the server’s hostvars, replacing foo with the server hostname. (Yes, it’s actually “gohma”, we never renamed the variables after moving to micrea.) If using Borg, also add backup_apt_dependencies: [borgbackup].

Add the backup script to roles/kde-backup/templates/backup-{host}.sh. Usually it involves copying and pasting things from the backup scripts for other hosts. Don’t put any passwords directly here; use Ansible variables.

Finally, add the server to backups group in the inventory (production/hosts).

Run python3 scripts/validate-backup-config.py. This will do some sanity checks, and for example warn if a backup script uses borg but it’s not listed in backup_apt_dependencies.

Login to the server you’re backing up and run ssh micrea.kde.org. This is just to ensure the fingerprint is added to known_hosts. You should do this before running the playbook, because there is a task that checks if the fingerprint is trusted and will fail if it’s not.

Now you can deploy the changes by running the playbooks:

  • backups.yml: creates the user and directory on the backup server, configures ssh keys, installs the needed packages (borgbackup if enabled) on the server to be backed up, installs the backup script into bin/run-backup.sh, and configures the backup cronjob.

  • monitoring-agent.yml: modifies telegraf config to monitor when the backup script runs.

If not using Borg, that should be all. Run the backup script manually once with bash -x bin/run-backup.sh to ensure everything is working, and you’re done.

If using Borg, we need more manual steps to create the Borg repositories. First, login to micrea and ensure the root directory for borg backups (like ~/borg-backups or ~/backups/borg) exists and is owned by the xxxbackup user.

Now to create the repositories, login again to the server you’re backing up. You need to set the BORG_PASSPHRASE and BORG_REPO environment variables and run borg init --encryption=repokey for each repository. When setting BORG_PASSPHRASE you may want to prefix the command with a space so that it’s not stored in the bash history. For example, if the backup script contains this:

# Prepare to run Borg backups
export BORG_PASSPHRASE='AbCdEfGhIjKl'
BORG_SERVER="ssh://platnabackup@micrea.kde.org"

# Backup Prosody
export BORG_REPO="$BORG_SERVER/./borg-backups/kdetalk-prosody"
borg create ...

# Backup homes
export BORG_REPO="$BORG_SERVER/./borg-backups/homes"
borg create ...

you use this to initialize them:

# prefix with space to keep out of bash history
 export BORG_PASSPHRASE='AbCdEfGhIjKl'

BORG_SERVER="ssh://platnabackup@micrea.kde.org"

export BORG_REPO="$BORG_SERVER/./borg-backups/kdetalk-prosody"
borg init --encryption=repokey

export BORG_REPO="$BORG_SERVER/./borg-backups/homes"
borg init --encryption=repokey

Finally, run the backup script manually once with bash -x bin/run-backup.sh to ensure everything is working.