Backer Upper is a custom backup solution I made to automate monthly backups of my servers for long-term storage.
Since about 2016, I’ve backed up my servers at the top of each month onto a NTFS-formatted hard drive. This process usually includes downloading all files from specific directories across servers, in addition to making a copy of any SQL databases on my server.
Unfortunately, running these backups manually took about 2 hours of manual labor and supervision. To solve this problem, I made a 2-stage backup process that does exactly what I’d usually do each month, then sends all those files to a Windows PC for storage. Along the way, live updates are provided in Slack to monitor progress (which is what you’ll see here)
—
Images are coming soon!
—
Languages: Python
Frameworks: Slack API, TBD
Lines of code: ~500
Timeframe: October 2022 - present
Here’s the general flowchart of Backer Upper and the process it goes through for backups.
It starts with Stage 1 on a Linux server where files are downloaded from my web & mail VPSes, along with any local servers on the network. I’m able to configure which directories it backs up from, and also which directories not to get files from.
SQL backups are pretty self explanatory, I get a SQL file from the server so that it can be restored at a later date if needed.
Once that’s done, Stage 1 zips up all the files and hands it off to Stage 2, which is a Windows PC with my NTFS backup drive attached. This drive contains many years of backups & work, so out of caution I only plug it into Windows machines.
Stage 2 then unzips the files, dumps it onto the backup drive, and then updates the OctoCam video archive that I save locally. Once that’s done, the backup process is complete.
The entire thing takes about 1.5-2 hours to complete, limited mostly by how fast my VPSes can upload data (500 Mbps for the web VPS, 100 Mbps for the mail VPS) and the unzip/zip process. I’m hoping to move the entire thing onto the Windows machine to avoid this process and save about 30-45 minutes of time.
Throughout the entire journey, slack updates are provided for each step.
For any file transfers, the number of folders indexed, the number of files transferred, and the actual amount of data transferred is shown, and this updates every 3 seconds.
For SQL/stage 2 handoff, live progress is shown on each part of the S2 handoff and for each database being downloaded.
And that’s really all there is to this system, it’s a nifty little experiment and helps me save some time with monthly backups.