Wednesday, May 04, 2011
First of all, my condolences to everyone who has lost people. The sheer randomness of it all makes it hard to attach any sort of sense to it.
But for the rest of us, it’s a reminder that we need to plan for disaster recovery. What would your college do if the campus were suddenly struck by an earthquake, tornado, or flood? (Insert whichever natural disaster makes sense in your location.)
Academia isn’t terribly well suited to handle this sort of thing. Culturally, we like to work slowly, with committees as the preferred structure. But in a disaster, that isn’t always an option. These events typically happen without much (if any) warning, and depending on severity, they can disable the infrastructure to even call meetings. They can throw even the most carefully crafted budgets into disarray. Depending on who is around in the moment, things like “chain of command” can be murky.
Some kinds of preparation make sense. On campus, I’ve been a consistent and vocal proponent of “cloud” based solutions for things like the email system and record-backup. Even with the recent Amazon hiccup, I still hold to that, just because the on-campus server room could easily flood. If that happened, and we didn’t have some sort of offsite backup or storage, the damage to continued orderly operation would be devastating. We’ve also done site drills based on a few scenarios, and we expect to do more.
Some disasters are harder to plan for than others. We have plenty of practice with blizzards, so we pretty much know the drill. Yes, sometimes a given storm acts differently than predicted, but the basic protocol is in place. But an earthquake could strike at any moment, at any level of severity, completely without warning. We’re not in tornado alley, but tornadoes have been known to happen; so far it has just been dumb luck that none of them has happened on campus. And although I hate to even bring it up, with thousands of mostly-young people around, it’s always possible that we could have a shooter. God knows I hope that never happens, but there are no guarantees.
As with life insurance policies, doing disaster recovery scenarios amounts to spending time that you hope will ultimately prove useless. They’re grim, and flawed, and incredibly unpleasant. I don’t usually like to waste time, but I hope I’m wasting time with these.
Whenever we have a minor incident, we get an e-mail from the president's office detailing the incident, the campus's immediate and ongoing response, and typically reminding us what the response to a similar major incident would be. For example, we had a tiny earthquake last year, just enough to get all the non-earthquake-having midwesterners excited. We got an e-mail saying no damage, no injuries, everything inspected, blah blah blah, "But in the event of a major earthquake, here's what you would do ..."
Off site backup is hardly a new idea. My dad's company made a duplicate of their critical financial records on a removable disk pack (think stack of 33 rpm record albums about one foot high) and put it in a nearby bank vault. Every week. This was more than 40 years ago. Today I would expect this to be done with a single flash drive, even for a largish campus, and a safety deposit box is a lot more secure than any "cloud" solution.
But it is also new: One place I worked kept ALL of the backups in the same room as the computer system that would get destroyed in a fire ... until I pointed this out to them.
At least the students have been safe. The flooding was prepared for as we had a similar event in '08.