TLDR: I learned the hard way that, at some point, my nightly backups were switched to being encrypted. As far as I can tell, it seems like it was part of an update and not a lot of users noticed it. Thankfully (and concerningly?), the key is stored in plain text if you can recover data from your SD card.
A storm came through my area yesterday and caused the power to drop multiple times. The first time, my HA Pi rebooted fine as it always does, but the second time it failed to boot because it couldn’t mount the HassOS boot partition.
After work, I pulled the SD card and connected it to a linux VM on my laptop to see if I could recover the backups from it and just restore from a clean install because I didn’t want to spend my night chasing down boot issues. The first thing I did was dd to dump the card to an image before I tried to mess with anything. I found the nightly backups, copied them over to my system, briefly tried to fix the boot issues, gave up, and then wiped and reinstalled Home Assistant OS with Raspberry Pi Imager.
Imagine my surprise when I go through the setup process, get to the “Restore from Backup” screen, and I am then prompted to enter the encryption key for my backup. My backup through Nabu Casa also conveniently had an error when I tried to load it, I assume that one is encrypted as well, but I didn’t even get that far.
I have always been pretty good about copying encryption keys into my password manager when I get them, so I was really confused when that prompt came up. I tried my account password and a few others that I might have used if it was a key that I set manually, no luck. After some research, the keys are in the form of XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX, which I definitely did not have, and the SD card had already been wiped, so any hope of finding it died.
At that point, it was getting late and I was getting frustrated, so I threw in the towel and gave into doing a fresh Home Assistant, not looking forward to rebuilding my automations and all that fun stuff.
When I woke up this morning, I had the epiphany that I had dumped the SD card as my very first step, so I grabbed the dump, mounted it, did a few find and greps, and found my encryption key in a json file called <mount path>/supervisor/homeassistant/.storage/backup. I copied the key, pulled up the restore page on my blank Home Assistant, put everything in, and it immediately began the restore process and has been happy all day.
Lessons Learned:
- Save your encryption key and emergency kit at
<server>/config/backup/settings - Look into remote backups because next time the SD card might be totally dead and I had bad luck with my Nabu Casa backup, I am going to install this Google Drive add on as a simple start
- You can always trust that someone, somewhere will leave an encryption key in plain text
I also learned the hard way: beware of SD cards! They have limited write endurance and will end up corrupting. With the amount of logging a system does and that the SD cards writes in pages, those writes add up soon enough.
Using a SSD boot drive over USB has been reliable and faster for my use. Alternatively, get high endurance SD cards or higher capacity to spread out the writes. At some point though, with the higher price of high endurance/capacity SD cards, you might as well get an SSD.
Thank you so much for sharing this, you may have just saved me a miserable future if my system were to crash.
I try to read the updates they share every month with detail, but this one must have slipped through the cracks. This feels like another thing they should have been way louder about considering the problems it could cause someone who didn’t know it flipped on without their interactive decision.
Oof. Yeah when storing encryption keys for backup media it’s also important to remember to store the key in a place that is accessible when your services are down (like they could be at the time you’re needing to restore a backup).
I’ve definitely screwed up and realized that a key I needed was stored in a credential database that was stored on the encrypted backup I was trying to restore. I was able to access another copy but it slowed me down.
I just learned about this yesterday. Haven’t found out if my previous backup solution also has become encrypted. The cloud backup, I understand. But… I don’t have much going on there that’s sensitive. Family locations? I guess…
For me the biggest problem would be door locks, outside of that, all they’d really be able to do is turn lights on and off menacingly
Nice save, and a fantastic PSA.
Also I’m a big fan of sleeping on a problem as a path to a solution. I’m not sure how exactly that skill develops, but it’s definitely something that I’ve done a few times over the years.
Hmmmm. Wonky events like this are part of why I’m glad I’ve always kept plain text hard copies of every key and password I’ve used.
Shit’s a nightmare when you get shut out of your stuff…
Good on you that you sorted it all.
