Wednesday, May 04, 2011


Disaster Recovery

I’ve followed with interest the outbreak of tornadoes in the South and Midwest, especially in Arkansas. Several colleges there have had to truncate their semesters, just because the local area was so devastated. (As far as I know, direct damage to campuses themselves was not the driving reason.)

First of all, my condolences to everyone who has lost people. The sheer randomness of it all makes it hard to attach any sort of sense to it.

But for the rest of us, it’s a reminder that we need to plan for disaster recovery. What would your college do if the campus were suddenly struck by an earthquake, tornado, or flood? (Insert whichever natural disaster makes sense in your location.)

Academia isn’t terribly well suited to handle this sort of thing. Culturally, we like to work slowly, with committees as the preferred structure. But in a disaster, that isn’t always an option. These events typically happen without much (if any) warning, and depending on severity, they can disable the infrastructure to even call meetings. They can throw even the most carefully crafted budgets into disarray. Depending on who is around in the moment, things like “chain of command” can be murky.

Some kinds of preparation make sense. On campus, I’ve been a consistent and vocal proponent of “cloud” based solutions for things like the email system and record-backup. Even with the recent Amazon hiccup, I still hold to that, just because the on-campus server room could easily flood. If that happened, and we didn’t have some sort of offsite backup or storage, the damage to continued orderly operation would be devastating. We’ve also done site drills based on a few scenarios, and we expect to do more.

Some disasters are harder to plan for than others. We have plenty of practice with blizzards, so we pretty much know the drill. Yes, sometimes a given storm acts differently than predicted, but the basic protocol is in place. But an earthquake could strike at any moment, at any level of severity, completely without warning. We’re not in tornado alley, but tornadoes have been known to happen; so far it has just been dumb luck that none of them has happened on campus. And although I hate to even bring it up, with thousands of mostly-young people around, it’s always possible that we could have a shooter. God knows I hope that never happens, but there are no guarantees.

As with life insurance policies, doing disaster recovery scenarios amounts to spending time that you hope will ultimately prove useless. They’re grim, and flawed, and incredibly unpleasant. I don’t usually like to waste time, but I hope I’m wasting time with these.

You don't already have this? We have a 33-page pamphlet distributed to everyone at the start of each year, plus copies in each classroom, with information on immediate disaster response as well as where to get more information for disaster recovery (website). The pamphlet includes instructions for earthquakes, fires, tornadoes, blizzards, shooters, medical emergencies, etc. It's pretty comprehensive. Professors are reminded at the start of each semester to review the pamphlet, and changes to the pamphlet are typically covered during pre-semester faculty meetings. They usually also review one or two of the types of disasters that have recently been in the news to fill out their half an hour and "retrain" people on that particularly.

Whenever we have a minor incident, we get an e-mail from the president's office detailing the incident, the campus's immediate and ongoing response, and typically reminding us what the response to a similar major incident would be. For example, we had a tiny earthquake last year, just enough to get all the non-earthquake-having midwesterners excited. We got an e-mail saying no damage, no injuries, everything inspected, blah blah blah, "But in the event of a major earthquake, here's what you would do ..."
Good show, Anonymous @ 4:45. My school has a similar program, and I consider my own leaders and colleagues well prepared to handle emergencies of just about any stripe. Remember the Scout motto: Be prepared. I’ve yet to hear that anyone came to grief while following it.
Actually, Shaw University (in Raleigh, NC) sustained enough damage to their student housing and services center that they were forced to truncate the semester (after getting approval from SACS). Thankfully, no-one was killed.
Your on-campus server room could "easily" flood? That is poor planning from the get go. I'd expect it, and all of the UPS electrical systems, to be above the 200 year flood elevation.

Off site backup is hardly a new idea. My dad's company made a duplicate of their critical financial records on a removable disk pack (think stack of 33 rpm record albums about one foot high) and put it in a nearby bank vault. Every week. This was more than 40 years ago. Today I would expect this to be done with a single flash drive, even for a largish campus, and a safety deposit box is a lot more secure than any "cloud" solution.

But it is also new: One place I worked kept ALL of the backups in the same room as the computer system that would get destroyed in a fire ... until I pointed this out to them.
I'm at one of those school (Missouri) that missed a number of days owing to flooding in the surrounding areas. The school is fine but the student body is a mess. We are having an "special" faculty meeting with three hours notice to discuss the arising issues with students finishing the semester. It's a mess. I don't know how much planning you can do to prepare for the 'afterwards' factor when these incidents can occur at any time during the semester. Losing a week and half at the front side of the semester is a lot different than losing that same amount of time just two weeks before finals.

At least the students have been safe. The flooding was prepared for as we had a similar event in '08.
The school I attend requires professors to be prepared to go to web only, such as by taping lectures from the previous year, in case campus access is lost.
The school I attend is in earthquake country, and the number of "soft first story" buildings we have is frightening! ("Soft first stories" are those held up with widely-spaced columns -- which collapse with only moderate shaking.) As far as I know, there's no game plan for a devastating earthquake. Most distressing. Fortunately for me, all I have to do is finish my thesis.
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?