Author: | Liraz Siri <liraz@turnkeylinux.org> |
---|---|
Date: | 2013-11-07 |
Manual section: | 7 |
Manual group: | backup |
Yes, TKLBAM is licensed under the GPL3. You don't have to care about free software ideology to appreciate the advantages. Any code running on your server doing something as critical as encrypted backups should be available for peer review and modification.
Using TKLBAM on TurnKey Linux provides the best experience but it will also work with any Debian or Ubuntu derived system and even with other Linux distributions if you use the --skip-packages option to disable integration with APT, the Debian package manager.
TKLBAM was originally designed to work with TurnKey Linux, where it takes advantage of an appliance's fixed installation state to make the smallest possible backup and make migration easier. The design has since been extended to work well with any Debian derived system, including Ubuntu.
In principle it should backup/restore files and databases on any Linux system (e.g., you can always install it and its dependencies from source), it just doesn't know how to talk with non-APT package management systems (yet).
Pretty much anything, though storing backups to Amazon S3 is easiest because authentication and key management are automatic. You just need to run:
tklbam-backup
But you can also backup to any storage target supported by TKLBAM's back-end Duplicity including the local filesystem, NFS, Rsync, SSH, FTP, WebDAV, Rackspace CloudFiles and even IMAP.
The local filesystem is one of the easier storage targets to use because you don't need to mess around with authentication credentials.
So assuming you want to store your backup at /mnt/otherdisk:
tklbam-backup --address file:///mnt/otherdisk/tklbam/backup tklbam-escrow /mnt/otherdisk/tklbam/key
And restore like this:
tklbam-restore --address file:///mnt/otherdisk/tklbam/backup \ --keyfile=/mnt/otherdisk/tklbam/key
Not as easy as the Hub-enabled "automatic" mode, but still vastly easier than your conventional backup process. The disadvantage is that you won't be able to restore/test your backup in the cloud, or from a VM running in another office branch (for example). Also keep in mind that a physical hard disk, even a RAID array, provides much lower data reliability compared with Amazon S3.
For this reason we recommend users use local backups to supplement cloud backups (e.g., providing fast local access).
Currently, only MySQL and PostgreSQL have built-in support but TKLBAM can work with other databases so long as you configure custom serialization/unserialization procedures in a hook script.
One of my favorite ways to do this:
# step 1: generate a backup dump tklbam-backup --dump=/tmp/mybackup # step 2: interactively review the dump's file contents & disk usage cd /tmp/mybackup apt-get install ncdu ncdu # step 3: add includes or excludes, go back to step 1, rinse, repeat vim /etc/tklbam/overrides # Everything perfect? tklbam-backup --upload-raw=/tmp/mybackup
By default, TKLBAM will automatically determine what paths and databases need to be backed up on a given TurnKey system according to the backup profile it gets from the Hub. The default profile tracks changes to the user-servicable, customizable parts of the filesystem (e.g., /etc /root /home /var /usr/local /var /opt /srv) while ignoring changes in areas maintained by the package management system.
You can "override" the default backup profile configuration by specifying overrides, either on the command line, or preferably by editing the /etc/tklbam/overrides configuration file.
Every TurnKey appliance that TKLBAM supports has a corresponding backup profile, which is downloaded from the Hub when you initialize TKLBAM. The profile is used to calculate the list of system changes we need to backup. It usually describes the installation state of a TurnKey appliance and contains a list of packages, filesystem paths to scan for changes and an index of the contents of those paths which records timestamps, ownership and permissions.
You can also generate your own custom profiles with the following command:
tklbam-internal create-profile
The backup profile is stored in /var/lib/tklbam/profile and contains the following text files:
Users can override which files and directories are checked for changes by configuring overrides (See below).
Because it "overrides" the default backup configuration in the appliance profile.
By adding a negative override to /etc/tklbam/overrides:
echo -/var/www/*/logs >> /etc/tklbam/overrides
You can also specify overrides on the command line at backup time:
tklbam-backup -- -/var/www/*/logs
By adding an override to /etc/tklbam/overrides:
echo /mnt/images >> /etc/tklbam/overrides
Or on the command line:
tklbam-backup /var/www/*/logs
Make sure you understand the implications of doing this. For example, if you add a directory handled by package management this may break package management on the system you restore to.
By adding a negative database override to /etc/tklbam/overrides:
# exclude drupal5 database echo -mysql:drupal5 >> /etc/tklbam/overrides # exclude sessions table in drupal6 database echo -mysql:drupal6/sessions >> /etc/tklbam/overrides
Or on the command line:
tklbam-backup -- -mysql:drupal6/page_cache
By default ALL databases are backed up so adding a negative database override excludes only that database or table from the backup.
Excluding a table only excludes its data. The schema is still backed up as long as the database is included.
Specifying a positive database override changes the default behavior so that only the database or table specified in the override is included in the backup.
You can mix positive overrides with negative overrides.
Yes. For example, let's say your default TKLBAM backup is several gigabytes in size and you'd like to create a lighter 100 MB backup that will be updated more frequently and take less time to update/restore.
cp -a /etc/tklbam /etc/tklbam.light
echo -/var/www/*/logs >> /etc/tklbam.light/overrides echo -/home/liraz/bigfiles >> /etc/tklbam.light/overrides echo -mysql:mydatabase/bigtable >> /etc/tklbam.light/overrides
export TKLBAM_CONF=/etc/tklbam.light
mkdir /var/lib/tklbam.light export TKLBAM_REGISTRY=/var/lib/tklbam.light
tklbam-init tklbam-backup
For convenience you may want to create a script that sets the TKLBAM_REGISTRY and TKLBAM_CONF environment variables:
cat > /usr/local/bin/tklbam-backup-light << EOF #!/bin/bash export TKLBAM_CONF=/etc/tklbam.light export TKLBAM_REGISTRY=/var/lib/tklbam.light tklbam-backup EOF chmod +x /usr/local/bin/tklbam-backup-light
Yes. Here are a couple of recommended ways to do this:
Create a separate backup with an empty backup profile:
export TKLBAM_REGISTRY=/var/lib/tklbam.srv export TKLBAM_CONF=/etc/tklbam.srv tklbam-init --force-profile=empty tklbam-backup --skip-packages --skip-database -- /srv
Use the --raw-upload option
This lobotomizes TKLBAM so instead of creating a system level backup it just backs up the directory you specify. In other words, --raw-upload turns TKLBAM into a directory-level backup tool rather than a system-level backup tool.
For example, let's say you have a collection of big files at /srv that you don't want to include in your system backup (e.g., because you don't want to bloat your backup).
So you configure an overrides to exclude the /srv directory from your default backup and create another TKLBAM backup just for the big files:
echo -/srv >> /etc/tklbam/overrides export TKLBAM_REGISTRY=/var/lib/tklbam.srv-raw tklbam-backup --raw-upload=/srv
Later, you'll need to use the --raw-download option to restore:
tklbam-restore --raw-download=/srv <your-backup-id>
If you don't use the raw-download option, TKLBAM will assume you are trying to restore a system-level backup and you'll get an error.
A full backup is a backup that can be restored independently of any other backup. An incremental backup links with the last backup before it and only includes changes made since.
Backup chains are links of backup sessions which start with a full backup, and then a series of incremental backups each recording only the changes made since the backup before it. Incremental backups are useful because they are fast and efficient.
Restoring an incremental backup requires retrieving the volumes of all backup sessions made before it, up to and including the full backup that started the chain. The longer the backup chain, the more time it will take to restore.
By default, a full backup will happen if the last full backup is older than 30 days. Between full backups, all backup sessions are incremental.
We recommend enabling the daily backup cron job so that daily incremental backups happen automatically:
chmod +x /etc/cron.daily/tklbam-backup
You can override the default by setting the full-backup parameter in the tklbam configuration:
# create a full backup every 14 days echo full-backup 14D >> /etc/tklbam/conf
Sorry, if your server is gone (e.g., terminated EC2 instance) nobody can help you. Next time either save an escrow key somewhere[s] safe or don't set a passphrase.
The encryption key for your backup was generated locally on your server not ours. Passphrase protection uses special cryptographic countermeasures to make typical cracking techniques (e.g., dictionary attacks) very difficult even with access to massive computer resources.
Note, if the system you backed up is still available, just log into it as root and change the passphrase (you don't need to know the old passphrase):
tklbam-passphrase
We recommend the following package:
apt-get install iftop iftop
Here's one way to do it:
apt-get install trickle trickle -u 100 -d 100 tklbam-backup
No! TKLBAM stores backups in the cloud for convenience, but it also supports local / custom backup storage targets.
There are two main alternatives to letting TKLBAM store a backup in the cloud:
Low-level tklbam-backup --dump option: lets you dump the raw TKLBAM backup extract to a directory, which you can then store anyway you like.
For example here's how we'd a system backup into a simple unencrypted tarball:
cd /tmp mkdir mybackup tklbam-backup --dump=mybackup/ tar jcvf mybackup.tar.bz2 mybackup/
And later restore it like this:
cd /tmp tar jxvf mybackup.tar.bz2 tklbam-restore mybackup/
The --dump option bypasses Duplicity, which usually create a series of encrypted archive files that can be incrementally updated. These archive files are stored by default in the Amazon S3 storage cloud but you can override this with the --address option and specify any storage back-end supported by Duplicity (e.g., local directory, rsync over ssh, ftp, sftp, etc).
High-level tklbam-backup --address option: lets you specify a custom backup target URL that is passed on to Duplicity.
It is highly recommended to rehearse a trial restore. Testing your backups is always a good idea, and even more so with a custom --address as this may complicate usage.
The Hub normally helps you manage your backup's metadata when it auto-configures the storage address. If you specify a manual address you need to manage storage locations, encryption keys and authentication credentials by hand.
TKLBAM doesn't store its data in generic S3 buckets, but in an isolated TKLBAM-specific area on S3. This means generic S3 tools such as the AWS management console, or S3Fox will not be able to access the storage buckets in which TKLBAM backup volumes reside.
Amazon supports payment by credit card and bank account. We recommend heavy users add a bank account as their payment method, as it's usually more permanent than a credit card.
In any case, if your payment method is invalidated (e.g., cancelled or expired credit card), billing will fail and Amazon will attempt to contact you (e.g., by e-mail) to provide a new, valid payment method.
Amazon S3 cloud storage fees are around $0.15/GB per month.
You can use simulation mode to calculate how much uncompressed data TKLBAM is going to store in a full backup:
$ tklbam-backup --simulate CREATING /TKLBAM FULL UNCOMPRESSED FOOTPRINT: 148.30 MB in 6186 files
In practice, the actual footprint of a full backup will usually be smaller due to compression, but this depends on the type of data being compressed (e.g., text compresses very well, video very poorly).
By default, a full backup is performed if one month has passed since the last full backup. In between, incremental backups will be performed which only record changes since the last backup. The full backup frequency can be customized. See this manual page for details.
If you notice $0.00 in the backups console, there's no need to open a support request. It's not a bug. At 15 cents per gigabyte, if you have just a few megabytes of data Amazon doesn't charge you anything.
Backups start from around 100KB for a freshly installed TurnKey appliance. Remember, TKLBAM only saves changes you've made since the appliance was installed.
No, for a couple of reasons:
Yes. Backups which have already been configured will continue to work normally. If TKLBAM can't reach the Hub it just uses the locally cached profile and S3 address.
Yes - manually. It just won't be as easy. You'll need to do a couple of steps by hand:
transfer the escrow key to the restore target.
This means you'll need to have stored the escrow key somewhere safe or be able to create it on the backed up machine.
specify the S3 address and the key manually when you restore.
For more details see the tklbam-restore documentation.
Yes - but you'll have to manage it manually. The Hub won't know anything about these backups so you'll have manage keys and authentication credentials by hand.
tklbam (8)