• Welcome to the Two Wheeled Texans community! Feel free to hang out and lurk as long as you like. However, we would like to encourage you to register so that you can join the community and use the numerous features on the site. After registering, don't forget to post up an introduction!

New automated backup system in place for the TWT server!

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
47,003
Location
Huntsville
First Name
Scott
Last Name
Friday
Okay, so after the snafu a few days ago, I got serious about the backup thing.

Some of you might not know, but when we made the switch from vB to XF, we also moved the site from a server that I have owned for 11 years onto a Virtual Private Server (VPS) at HostGator.com (HG). After some issues with HostGator on New Years day that took the entire site down without warning, I moved to a different service provider, Knownhost.com (KH). I also got a VPS with them. These guys are awesome and they are VERY XF friendly. They even use XF for their own community platform. Their tech support is just outstanding! :thumb:

One thing that I ran up against with the recent data loss issue is that for the VPS systems, KH only does a full server backup every other day. That is still far better than HG which basically did nothing. So I have been looking into options for backing up the current file system of the server and the MySQL database data for the forum on a daily basis.

The first option was to try the backup feature in cPanel, a domain specific management tool. When I did a full backup of the site last weekend using cPanel, the compressed file was still 60GB, which means the full site is even larger than that. The only reason that worked is because it makes you download the backup file as it is created so that it does not take up space on the server. Even with a 45Mbps connection, that took a pretty long time to download and it would have to be done daily! Worse, if I actually needed to use the backup file, I would have to upload it back to the server. On a good day, my upload speeds peak around 12-15Mbps. That would take FOREVER! Not really a practical solution.

The next option was to use the WHM server manager software's backup utility. This allows scheduling and transferring of the backups to off-site locations. However, it is kind of an all or nothing method and it also requires that I have enough space on my VPS to store a copy of the backup before it can be moved off-site. That was a problem. My VPS only has 100GB of space. If the compressed backup was 60GB, then there was no way the uncompressed data and the backup would fit on the server. So again, not practical.

I headed on over to the XF support forums, which are much better than the vB support forums, and looked for solutions. What I eventually found was a paid add-on that has been around for a while and has a good reputation. It was $10 for a life time license. I figured it was worth risking $10 to see if it worked.


This is a pretty cool add-on. It lets me do a backup of just the MySQL database, the XF filesystem, or both. It stores them locally, but can also transfer them to a remote destination. Because it let me do the database without having to do the filesystem, I can use it to only do the database. It zips the file, stores it locally, then transfers it over to my HG VPS. It was super easy to setup. The current uncompressed database size is only 1.5GB and that compresses down to about 500MB, so I can store plenty of copies of that locally without any issue, and they are NOT a big hit on the bandwidth of the server when they are transferred to the HG VPS. I have it set to retain seven copies of the backups before it starts over writing the first backup. These I can even download to my home computer just to have a third copy. The add-on DOES close the site for a few minutes while the contents of the database are dumped. However, I checked the transfer from last night and it completed the process from closing the site to having the remote file saved on the HG VPS in about five minutes. The site is only closed while the database is being dumped, not while the file is being moved off-site, so the forum is only closed for a minute or so. I have it set to run every night at 3:00am CST.

So that takes care of backing up the database, which leaves about 58.5GB of data to backup and move off site every night!! The VAST bulk of this is due to the user attachments. I looked into doing incremental backups, where only new content gets backed up and I'd still have to do an occasional full backup (like once a week). So still a good bit of FTP loads when moving that 58.5GB. Then I got to thinking... :ponder:

Here at home, I have a Synology network storage box sitting up on my shelf. It has software that monitors specified directories on my computer for any new files. When it sees any, it instantly starts copying them to the network box. This way, when I transfer a load of RAW files from my camera to my computer, they immediately start getting backed up to the box, which has two 3 TB drives in it that are set up in a mirrored raid. If one dies, I can hot swap it with a new one, and it immediately starts rebuilding the data on the new drive. The end result is that I have three copies of all my files. I occasionally have to load files on a big thumb drive to take them to my office computer so I have an off-site backup because again, my upload speeds suck so bad at home that it would take forever to move them somewhere like an Amazon cloud account. The key though is that the box doesn't require things to be compressed into a single BIG file before backing them up. It just syncs instantly with whatever shows up on my computer. Surely something like this has to exist for servers!?

So I started Googling and found a command called "lsyncd". What this does is run in the background on the server and monitors specified source directories for any new files. It then securely links to a remote location and transfers the files, basically acting just like my network storage box. I contacted the tech support peeps at KH and asked if they could install it if it wasn't already installed. They had that done in about two minutes. Then I had to figure out how to get it all setup. Back to Google and I soon had step by step instructions. When I tried to fire it up, nothing was happening. So back to tech support and in a few minutes they figured out why. I was editing a conf file in one directory based on the instructions, but the program was actually looking for the conf file in a different directory. That solved, it still didn't work. So the tech guy went off to do some reading because he was not real familiar with lsyncd. A short while later I got an email back from him telling me that there was a slight difference in syntax between the instructions I had and what the program expected, and he had it up and running!!

So this program is now scanning the root directory and all subdirectories of the site for ANY new files. It has almost completed a full copy of the entire site over to my HG VPS. Once that is done, there will be no need to do any more BIG transfers. The only things that will get transferred will be just whatever new files showup throughout the day, which imposes no worse a load on the site than regular users using the site. This gives me an instantaneous backup of all system files. Any time a user uploads a new attachment, a copy is made to the HG VPS. It is totally automated.

Now I have a daily backup of the forum database and an instant real time backup of the file system running around the clock. I don't have to mess with trying to download and upload massive files between the servers and my home computer. If I need a backup, I can just transfer it directly from the HG VPS to the KH VPS at MUCH higher speeds. When the HG VPS contract expires in June of 2021, I'll have to look for a different remote storage destination. However, I was talking with one of the guys from KH and they are actively working on adding better backup options for their VPS packages and forums. So maybe by then they will have something similar.

So ideally, IF something else were to happen, at most we might lose 24 hours of content.
 

Tourmeister

Keeper of the Asylum
Admin
Joined
Feb 28, 2003
Messages
47,003
Location
Huntsville
First Name
Scott
Last Name
Friday
Great... :argh:

So the backup program closed the site and never reopened it :doh:

One more problem to figure out :roll:
 

StromXTc

Forum Supporter
Joined
Dec 29, 2017
Messages
2,895
Location
George West
First Name
Brian
Im mostly following you. I know you are probably even simplifying it too. Sounds like you (we) have a great partner (kh) in keeping the lights on!
 
Top