Backups
Over the last few weeks I’ve been working on upgrading my home desktop computer, and I’ve documented much of the process here. New RAM, new CPU and cooling fan, several missteps along the way, but I got it all working nicely. Which of course meant that something new had to go wrong.
First, let me tell you about my current backup system for this box. For a number of years, every time I switched computers or OSes, I would back up the entire disk (or at least my home/user directory) to an external hard disk. And then I’d copy back all the stuff I needed, but kept the old copy around. Eventually, all of these copies got consolidated into my desktop computer. There’s all kinds of good stuff in there. Old documents, old code, old programs and downloads, photos, plans, projects, on and on.
In addition to that, I have a well-curated collection of audio, video, and ebooks. All told, this is about 800 GB of data. It all currently lives in my Dropbox folder, so it’s backed up to the cloud that way. And in addition to that, I do an rsync backup of my Dropbox folder to another 2TB drive.
The rsync job runs via crontab every night at midnight, and I’ve been using healthchecks.io to monitor that job. It’s a great service. You just add a ping to healthchecks when your backup (or whatever) job is done. If healthchecks gets the ping, all is good. If it doesn’t get a ping within a specified period, it alerts you. I have mine set up to alert me via email, but there are a ton of other integrations. It’s free to use if you just have a few personal jobs like this.
Generally, when I’ve gotten a ping on this, it turns out to be the Internet was down, or I dual booted into Windows and forgot to switch back before midnight, or the computer was just turned off (because, say, I broke my cooling fan mount). But the other week I started getting pings and my logs showed that rsync was failing to copy files to the backup drive. A couple of times I just ran the job again and it worked, but some files still had problems. Then, the other day I booted back into Manjaro after doing playing some Halo on Windows, and… well, actually I did NOT boot back into Manjaro. Because fstab was failing to mount my backup disk. I got into recovery and edited fstab and was able to boot fine. Mounted the disk manually and found 1,900 something errors. Luckily, it was just the local backup disk, so no data loss. Wiped it and repaired it, and it seemed to work fine. The errors are still there, but marked as areas of the disk to not use. But … yeah. This is the 2TB disk that I bought back in 2011 when I built this machine. It’s had a good life. But I’m not going to trust it for anything vital at this point.
I was going to buy a 2TB SSD for around $200. But decided on a new HDD for about a quarter of that price. The way I have my Linux system set up is with a 256 GB SSD on root /. Then my /home directory mounted on a separate 2TB hard disk. So I copied my whole home directory over to the new disk, wiped the old one and now that’s the backup disk. A bit of trial and error editing fstab, but once I got it right, it all booted up like nothing ever happened.
But it got me thinking about my backup strategy. Dropbox has served me pretty well, but it’s not a really professional backup system. They do keep copies of deleted files for a while, but you have no control of that. If I corrupt or change some file incorrectly, once it gets backed up, that’s that.
[Edit: Just discovered that Dropbox Pro, which is what I’m on, also gives you 30 days of file versioning, which is not too shabby. Other plans offer up to 180 days of versioning.]
My rsync strategy wasn’t much better. It was just overwriting and deleting copies to make a mirror of my source data.
I will note that any non-trivial code I write gets checked into github or gitlab or bitbucket on any major changes.
So I started looking into better tools. My first attempt was using restic and Backblaze B2. This will allow me to do incremental snapshots, so I can go back and get a previous good version of a file that might have been deleted or corrupted. Backblaze B2’s storage is pretty cheap. But they also charge for downloads. And I discovered that when you run a restic check or restic prune it makes a bunch of api calls to get file data and Backblaze counts these as downloads. This seems to have the potential of inflating the cost. But I’m not really sure how serious of an issue it will be until I try it and see the real world usage numbers and charges. It may wind up just being a few cents per month.
The other service I looked at is Wasabi. This service caters more towards enterprise accounts, but there’s nothing to stop you from using it personally. On the plus side, they only charge for storage, not downloads or any kind of data resulting from api calls. On the minus side, you have to pay for a minimum of 1TB of storage, and they have a 90-day retention policy, which means that if you upload a large file and delete it the next day, you still pay for 90 days worth of storage for that file. Not a big deal for a backup solution that is going to be a pretty stable size. And it’s only $5.99 for that 1TB, with $0.0059 per GB beyond that. Since I’ll be backing up close to 800GB, this is not bad.
So, I’m still figuring out what I want to do about all of this. For now, I’m at least switching over to restic for my local backup to give me some more security there. I need to clean up all my old archived files because there is a ton of duplicate stuff there. Then I’ll probably go with Backblaze for a month or two and see how that pans out.
XFCE
This is just a fun little thing I have been playing with the last couple of days.
As you may have gathered, I run Linux on my personal machined and after spending plenty of time on many different desktop environments, I keep coming back to XFCE as my favorite. It hardly changes year after year, so there are plenty of XFCE memes making fun of its nonexistent “exciting new features”, but it’s stable as hell, quite customizable and very minimal, which I like.
I use multiple workspaces (virtual desktops) and I use the workspace switcher panel plugin to indicate which workspace I’m in. By default, it looks like this:
I run four workspaces. The switcher shows a preview of what’s on each screen. The config is minimal:
I turn off the miniature view, so it just shows the number of the workspace, like so:
That’s fine, but rather boring. But it turns out that just about everything in XFCE is visually configurable via CSS. You just edit (or create) the file: ~/.config/gtk-3.0/gtk.css. Then you need to reload the thing that you’ve changed. The workspace switcher is part of the XFCE panel, so you can reload it with xfce4-panel -r. The tough part is knowing what the element names, ids and classes are. To help you inspect the “DOM”, there’s a GTK debugger tool. First, you quit the panel: xfce4-panel -q, then you restart it in debug mode by running GTK_DEBUG=interactive xfce4-panel.
This restarts the panel and opens up a bunch of windows – one for each gtk element running on the system. Here’s the one where my workspace switcher is:
You can see the XfcePanelWindow, the PanelItemBar, and under that the workspace switcher is the XfcePagerPlugin with XfcePagerButtons. There’s all kinds of views into what’s going on. This one I found most useful:
Now you can see each element, its ID and class, if any, and all the CSS properties for that element. Armed with this tool, some helpful tips from some posts on reddit, and a lot of hacking and slashing, I wound up with this in my gtk.css file:
#xfce-panel-toggle-button:checked {
background-color: #cccccc;
color: black;
}
#xfce-panel-toggle-button {
font-size: 12px;
padding: 0 2px;
margin: 6px;
background-color: #444444;
border-radius: 12px;
box-shadow: 0 0;
}
And when I reload my panel now, my workspace switcher looks like this:
This is life changing… not. But I like it and it was fun to do. Maybe I’ll mess with some other elements, but right now I’m pretty happy with how everything else already looks.
Bit-Box v2
No real significant updates here, but a few ideas I’ve been kicking around.
For one, I learned about how you light up keyboard keys. I assumed you had to buy switches that had LEDs in them. Maybe you can, but there are two other methods that are more prevalent. One is to just have a surface mounted LED under a clear switch or a switch that has an opening to let the light shine through. The other method is to insert the right sized LED into your switch. Little did I know that the switches I bought do actually have a little LED-sized cutout in the top, with two holes that go right through the switch to the back.
Some research led me to buying 100 3mm flangeless 3 volt LEDs for something like $5-6. They pop right into the switches like this:
Since they are 3 volt, they can run right off the Arduino micro board.
I think I have this mounted upside down, but whatever. I also think it might be too bright, so if I ever get to the point of adding these to all my switches, I’ll have to see how bright it looks and might need to add a resistor, or maybe even a knob to control the brightness. Or… I just thought, another control to have the Arduino to it via PWM. So many things to try.
I’ve also been thinking about alternate build ideas to get around printing switch plates each time. I printed out the switch plate template I have on paper, used it to mark up a piece of 1/4″ oak, and chiseled out a cherry-mx-sized square hole.
This worked surprisingly well and was surprisingly non-difficult. It fits snugly and securely. This opens up all kinds of other build ideas, which are mostly just floating around in my mind for now until I get back down to the workshop and actually try something.