4.2. Backup, My new years resolution¶
Well no not really, I’ve been using backups at home for many years now, but my solution was getting very complex and no longer fitting my requirements.
A long time ago I sat and thought what would happen if I lost all my data? I have photos and videos of the kids growing up, can’t replace those! If I lost them I think my wife would kill me. In total I have around 11Tb of data I need to maintain and backup in the house. So how to backup 11Tb of data without costing the earth or exposing my data other people?
I’ve been using FreeNAS as my file server, however I made a basic mistake at the start and used a 3ware RAID card in RAID-5 to allow access to the drives. This caused very poor performance on the NAS to the point where I needed to build something faster. I ran some tests on other hardware and saw the much better performance I could get and decided now was the time to look at all my file server needs.
4.2.1. New FreeNAS Server¶
I bought a new motherboard, “ASRock C2750D4I, Intel Octa Core Avoton C2750, DDR3, SATA III 6Gb/s, VGA, 2x GbE LAN, 1x IPMI LAN, 2x USB 2.0, mITX” and 32Gb RAM. I kept the 2U rack case and drives and just replaced the motherboard. I used 8 SATA ports for my 3Tb drives, 1 for an SSD Read cache, 2 for a SSD write cache and 1 for a boot drive.
Installed FreeNAS and let it set up the ZFS system, I wanted performance and reliability so ended up with ZRAID-2. Under advanced settings I made sure to “Enable autotune” which make a real difference and bonded using LCAP the two 1Gb NIC interfaces.
This gave me around 21Tb of storage space. Next I created “Datasets” for all the different shares I wanted. Previously I’d just had one primary dataset and create sub dir from there for sharing. As we use MacOS in the house I also created a TimeMachine Dataset and I’ve specified a quota of 2Tb, that should be enough for most of the computers in the house to back up to.
I copied all my data back over, which now means I have around 11Tb used. I have to find away to back up 11Tb of data! ouch!
4.2.2. How to Backup my 11Tb¶
Online backups like “crash plan”, “SugarSync”, “Carbonite”, “openDrive”, etc. Well they might work for some/many people, but I have a few issues with them:
Sending 11Tb of data is really not going to work out well. Ok maybe the first backup would take ages but how about others? Well it turns out as I’m using Proxmox for virtual systems and have many large (200Gb+) files which are the filesystems. Each backup would require sending all 200Gb+ for each file over the wire. That’s really not going to fly.
Encryption. Many sites offer encryption, but do they have a back door to the data? I don’t know, so I’d always want to add my own on top. Sure I can do that, but it’s getting complex again.
Retention policies, how long are they holding to the data and have they really deleted it when they say they do?
Who else can see the data?
So I need to backup myself. Tapes, they are expensive, well the drives and autoloaders are, Hard drives on the other hand, they are cheap. £80 for 3Tb drive, now we’re getting close, but how to copy large files to drives making sure I have everything and making the best use of the space?
4.2.3. Snapshots¶
FreeNAS and ZFS supports snapshots, so first thing is to setup snapshots on the FreeNAS server so I can easily roll back to a previous version of data within seconds.
create hourly snapshot and keep for 24 hours
create daily snapshot and keep for 1 week
create weekly snapshot and keep for 1 month.
This way I have a few history snapshots so I can go back 1 month and recover files and more detailed backups closer to current.
As the TimeMachine dataset is already it’s self a backup I’m not snapshotting it.
4.2.4. Replicate to another FreeNAS server¶
I had my old motherboard still around so I decided to make use of that, as well as a some older 1 and 2TB drives. I created another FreeNAS server which is elsewhere in the house, i.e. not next to the primary. This time I set up ZRAID-1, so only 1 drive can fail, but that’s ok. It’s already a backup system so I don’t really care too much. Using the snapshots from the primary I set up replication between the FreeNAS servers. Whenever a new snapshot is created on the primary that snapshot is sent to the backup server. As it’s only small delta, it’s just sending maybe 1-5Gb data depending on what’s changed. That’s much nicer.. I fact I could likely move this backup FreeNAS to another location. Now I have my data on multiple spindles on the primary and multiple spindles on the secondary. A lot of bad things need to happen to lose my data.
4.2.5. Bacula¶
There’s a wonderful backup solution called Bacula, which can backup using well, anything including treating hard drives as if they were tapes. I have been using Bacula for many years and it works well. You can install clients onto windows, Mac, linux etc. Have separate backup destinations, schedules, the complete package. Downside it’s editing lots of config files and while there are GUIs most of the time you revert to command line.
vchanger, allows me to treat a drive as a magazine of tapes and then files on the drive as the tapes. I make each virtual tape 45Gb, that way on a 1Tb drive I get around 19 tapes, with a little space left over.
By Also using “autofs” I’m able to auto mount the drive if I know the UUID of the partition, which I do as I’m the one creating them. So I just need to format the drive, in my case ext3, grab the UUID and give that to vchanger as a new magazine. I can then ask vchanger to create the virtual tapes, and bacula to add those tapes to it’s set.
My Bacula server is an old Intel NUC PC, which has USB 3 interface connected to a USB3 to SATA 4 drive Dock. I can then hot swap drives as the backups need. Having 4 at a time, I’m not sitting around baby sitting it during it’s backup cycle.
I’ve labelled the drives with a QR code with all it’s details, when it was created, it’s UUID etc, and then just the last part of the UUID so I can quickly identify the correct drive to remove when it’s full.
Those drives are then recycled in the backup system after 1 month ready for use again. I maintain two sets of drives, one off site and the other in the dock ready for their backup run.
4.2.6. The complete solution¶
I have my FreeNAS primary fileserver handing all the shares etc. with ZRAID-2, so I can support two drives failing at the same time. The FreeNAS is creating snapshots regularly depending on the “dataset”. Those snapshots are replicated to my backup FreeNAS server with it’s ZRAID-1 drives, giving high capacity but reduced redundancy.
From this backup FreeNAS I’m then exporting (NFS) the shares (READ ONLY), to my bacula server, Now I run a bacula Backup from the Backup FeeNAS to these virtual magazine drives once a month. As I’m running from the backup FreeNAS I’m not effecting user performance and it’s free to work away.
When the backups are complete I’m putting the drives in a flight case and taking them to my parents-in-laws house to look after. This then gives me my off-site backup.
