🚨 Yesterday, I discovered that 100% of my main document server... was just gone. No files. Nothing. This was HUNDREDS of gigs of data in thousands of directories, and it just vanished. In addition, numerous server-side directories had permissions changed, and file ownership changed. It is unclear why, and I've never seen this in the history running web servers for The Black Vault.
I am not sure when exactly this happened, but I discovered it yesterday. I have server monitors up, but never got notified of anything down. The server was just throwing "Forbidden" errors which is why it technically didn't show as down, so I never got a notification.
Let me be clear, I do not fully suspect foul play, but the main web hosting provider for that server (I run 4) had no idea what happened, and on their side, they said it was a deletion, not corruption.
In my honest opinion, I feel it was a very oddly timed server maintenance done by the hosting provider, that went awry. They didn't catch it, and when I did, they didn't take blame and there was no way to fully prove what happened, and by whom. Could I be wrong? Yes. Could it have been foul play? I can't rule it out.
Yes, I have numerous backups, and yes, it appears I got everything restored already (as of early evening yesterday). But, please let me know if you find any errors when downloading documents (or anything, for that matter).
I remain the sole person who runs The Black Vault, and this will be the start of my 30th year running the website, come this September. I began hand typing documents in high school to a 5 megabyte free space server back in 1996 (Primenet!). I have now grown to nearly 4 MILLION pages of records online, running 4 dedicated servers just for the site, and no data deletion, accidental or otherwise, will ever keep me down for good.
But it is a stark reminder to us all, me included. Keep backups. Keep them in multiple places. And never be intimidated by anything that comes our way, no matter what we expect may have happened.
Stay the course. I am. And I'm not going anywhere.