Jeff Attwood has lost the content from two popular blogs that he runs:
http://blog.stackoverflow.com
http://www.codinghorror.com
thanks to:
100% data loss at our hosting provider, CrystalTech.
He gives a little more detail here. He is now trying to recover data from search engine caches such as Google’s – a painful business, apparently; Google banned his IP.
Backup is a complex problem. I’d been meaning to post on the subject following another recent incident. Here’s a quote from an email a friend received from his ISP after asking whether the SQL Server database was backed up:
Needless to say, we do back the databases up every 12 hours to a remote location automatically.
Just 11 days later “a crucial disk” failed on that SQL Server; following which the ISP discovered that its recent back-ups were also “corrupt” and data was lost. In the end a data recovery specialist was enlisted and most, but not all data recovered.
No doubt the post-mortem will reveal multiple issues; but it shows that knowing backups are being done is insufficient. You have to do test restores as well, because the backup might not be working as well as you think.
In addition, as Attwood is now tweeting:
don’t trust the hosting provider, make your OWN offsite backups, too!
Good advice for those of us using commodity ISPs. But it also gives me pause for thought following the CloudForce event I attended earlier this week. A specialist like Salesforce.com has more resources to put into data resilience than any of its users. So if Salesforce.com (or Amazon, or Google, or Microsoft) is your ISP, is it then OK to leave backup to them?
Doing a salesforce backup which doesn’t depend on salesforce is no trivial matter. There are products which do it, sort of, but they seem to be more o” an “export” than a backup per se.
I don’t think you should be leaving backup to anyone else. It’s as important to ensure the integrity of your backups as your primary database (or other servers). If you can automate your backup process to make offsite backups, then why not automatically restore those backup in another environment? With the cost of ‘machines’ going down with the virtualisation and the cloud, surely this is going to offer further protection in a lot of cases.
This has been a consistent problem of IT in general through the ages, even back to the days of the old tape backups. Fact is most companies do regular backups. Fact is also that most companies never actually try and restore any of those backups to check the process, only in the event of a catastrophe which is when they find there is either a problem with the process, the media or the backup job.