SystemAdministration/Backups

Oxygen Backup Server

Oxygen is our backup server. It is a Ramnode 1024MB CVZ - OpenVZ SSD-Cached system with 200GB of hard drive space, that is located in the New York data center. It connects to the Linode servers over IPv6 and looks to the other systems like it is just another Linode VM. Backup are stored in /home/backups. There are two types of backups. /home/backups/tarballs stores the full tar backups of the various servers at certain points in time with the date of the archive in the archive name. /home/backups/rsnapshot stores full system snapshots of the servers as regular files that can easily be browsed and copied.

rsnapshot Backups

rsnapshot is a filesystem backup utility based on rsync. Using rsnapshot, it is possible to take snapshots of filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus differences.

Currently rsnapshot is configured to keep 7 daily backups, 4 weekly backups, and 3 monthly backups. The backups are scheduled using cron with the script /etc/cron.d/backups. Since rsnapshot uses rsync over ssh and needs to run as root to copy all files easily, a root Kerberos principal has been created with a corresponding keytab. A wrapper script, /home/backups/rsnapshot/krsnapshot, setups the kinit before running rsnapshot. The backup cron script also rsyncs /etc on Oxygen to /root/oxygen-backups/ on Boron as the only live backup of the Oxygen system (static copies of the different scripts in /home/backups/ was also made).

MySQL Backups

Service VMs

Full MySQL backups on each service vm (Boron and Beryllium) is handled by a simple cron job the does a mysqldump of all databases on the server. The file is gziped and saved to /root/db_dumps/. It runs daily and overwrites the backup each day. The backup file will be stored by rsnapshot on Oxygen, so no old backups on local server needs to be kept. The cron job is /etc/cron.d/mysql_backup and calls /root/db_dumps/mysql_backup.sh.

Production Server

<<<This will be updated soon when the new MySQL Cluster takes over as the database server for both Production and Dev.>>>

The primary MySQL database for the production slash server is now backed up to Oxygen via a script that runs from a crontab job on Neon. The jobs are run by the user slash from the /srv/soylentnews.org directory. The scripts is run daily. A local copy is generated, gziped, then stored in /srv/soylentnews.org/db_dumps. Old copies are kept for two weeks and deleted by the script.

Files are permanently stored in /home/backups/db_dumps on Oxygen. These files are not automatically deleted so there may be a need in the future to delete the old files.

Here is the mysql_backup.sh script for Neon:

#! /bin/bash
HOME=/srv/soylentnews.org

backup_dir=$HOME/db_dumps
today=`date +%F`
week=`date --date="-14 days" +%F`

if [ -e $backup_dir/soylentnews-$week.sql.gz ]
  then
    rm -rf $backup_dir/soylentnews-$week.sql.gz
fi

/opt/mysql-5.6.17-linux-glibc2.5-x86_64/bin/mysqldump -h 127.0.0.1 --single-transaction soylentnews > $backup_dir/soylentnews-$today.sql

gzip $backup_dir/soylentnews-$today.sql

kinit -k -t $HOME/slash.keytab slash

scp $backup_dir/soylentnews-$today.sql.gz oxygen:/home/backups/db_dumps/soylentnews-$today.sql.gz

Here is the assoicated crontab entry:

4 5 * * * /srv/soylentnews.org/mysql_backup.sh

For now production mysqldump takes only a few seconds and the transfer to oxygen takes actually more time. Current dump size is approximately 100Mbi compressed for the production database.