What Each Number Protects Against
- 3 copies: any one copy can fail, be deleted by mistake, or turn out to be corrupt. With three, losing one still leaves two.
- 2 kinds of storage: the same fault, such as a bug in one storage system or one bad batch of drives, cannot take out every copy.
- 1 off-site: fire, theft, flooding or a whole provider having a bad day takes out everything in one place. The off-site copy is the one that survives.
What It Looks Like for a Server
- The live server. Copy one.
- A nightly backup to separate storage, such as a storage box with Borg, or object storage with restic. Copy two, on a different kind of storage.
- A third copy somewhere else: another provider, another region, or a NAS at your office. Copy three, off-site from the other two.
Our storage boxes and our object storage are in the same data centre. Using both gives you two kinds of storage, but for the off-site copy, choose a different location or provider.
A Copy the Server Cannot Delete
Ransomware and attackers go after backups first. If the server can delete its own backups, so can whoever takes over the server. Make sure at least one copy is out of its reach:
- Storage box snapshots are read-only copies made on our side. Nothing connecting over SSH, including a compromised server with the box's key, can change or delete them. Protect your dashboard account with two-factor authentication, since that is where they can be removed.
- Pull rather than push: a backup server that logs in and fetches data holds credentials the production server never sees.
- Separate keys: give each server its own storage credentials, so one compromised server exposes only its own backups.
Test the Restore
A backup is only as good as the last time you restored from it. Restore a few files every month, and once a year rebuild a whole server from backups alone, using only what you would have if the original were gone: the backup, its password and its key. Note how long it took; that is your real recovery time.
Choosing the Tools
Borg vs restic vs Kopia vs rclone compares the main open-source backup tools, and storage box vs object storage compares where to keep the backups. For databases, add nightly dumps, so a restore does not depend on copying a running database's files.