Although I was never a hardcore stat-head, I had to endure the requisite statistical methods courses in both college and grad school as part of my social science training. (The ugliest course requirements repeat themselves, the first time as tragedy, the second time as farce.) Although they were billed as the 'empirical' part of the discipline, I regarded them as the purest mysticism, and, with the benefit of hindsight, still do.
The graduate course was particularly absurd. It was a disciplinary requirement, so we hated it. The professor was working out some psychological issues before us in real time, which made for some amusing moments but little real education. My defense mechanism of choice was ironic distance; my final paper, an examination of the relationship between two variables (I don't even remember which ones) controlling for sex, found nothing, so I titled it “But Sex Always Affects a Relationship!” Not much of a paper, but a damn fine title, if I do say so myself.
The true highlight of the course, though, occurred when one student – either batshit crazy or a comic genius, history will decide – strolled in about ten minutes late one day brandishing a box of Crunch Berry cereal, offering the nutrition information on the side as a real-world example of statistics. (He was also the one who brought his guitar to a feminist theory seminar to favor the group with – I'm not kidding, and neither was he -- “I'm a lesbian, too.” Andy Kaufman had nothing on this guy.)
I think of him sometimes when we use stats on campus to try to make decisions.
Anybody with even the faintest whiff of mathematical training, or intuition, or common sense, can spot flaws in most of the statistics used to make decisions. That's not a shot at the folks generating the statistics – they/we know perfectly well that most of the information is partial, somewhat corrupted by flawed collection methods, and extremely hard to isolate from other variables. (“We changed the course requirements last year, and this year the enrollments went up 5%. Clearly, the new curriculum is responsible.” Um, not really...) The problem, other than the fatal combination of small samples and sheer complexity, is that most of what we want to know derives from problems we didn't anticipate, so we didn't think to collect the data at the time that would address the question we hadn't thought of yet.
Faculty tend to be pretty bright people, as a group, so any data-driven argument for a policy change they don't like immediately brings out the “your methods are flawed” crowd. Well, duh. Of course they're flawed – even a relative non-data-head like me can see that. They're flawed in any direction, but the flaws only seem to draw attention when the policies themselves are unpopular.
(Exception: sometimes, statistics can disprove certain things. That's real, but it's limited. It's hard to sustain the “my program is thriving” illusion when its enrollments are down twenty percent in two years. That said, the statistic doesn't tell you what to do about it.)
In the rare cases when it's possible to get good information, I'm a fan of evidence-based management. The catch is that really solid evidence is remarkably hard to come by, especially in the brief moments when decisions are actually possible. Scholars of higher ed can do national studies on foundation dimes and issue reports that say things like “student engagement in campus activities leads to higher graduation rates,” but even they can't really disentangle causation; do campus activities spark the less-driven students to step up, or do type A personalities naturally gravitate to organized activities?
Most of the decisions, though, are much more mundane than that. I once asked one of my chairs to add a weekend section of a popular intro course, since we're trying to reach out to adult students. He didn't want to, so he suggested that we do a study to see if this would work before actually committing resources to it. The academic in me is so used to seeing this line of argument as reasonable that I almost didn't catch what he was doing. The only way to see if it would work is to try it. We could ask a random sample of residents, but what people say and what they do are only vaguely correlated. The only way to do the study is just to run the damn class and see what happens. (We did, and it worked.) Calling for more data is much more respectable than just saying “I don't wanna,” but it often carries the same meaning.
One of the real shocks of moving from the classroom into administration was growing comfortable with making decisions based on far, far less (and less clear) information. Numbers bounce around, depending on when and how they're gathered, and the possible number of intervening variables in determining why program X is down this year is infinite. I can concede all of that, but still need to make a decision. Standards of evidence that even Andy Kaufman Guy would have found laughable sometimes carry the day in administration, simply by default. The owl of Minerva spreads its wings at dusk, but I don't have that long. Semesters start when they start, and we need to make decisions on the fly to make that happen. If you wait for the statistical dust to settle, you'll miss the moment. In faculty ranks, that would be called 'reductionist,' and it is. It has to be. Part of being in administration is being okay with that, and developing the intuition to focus on the two or three facts that actually tell you something. The rest is mysticism.
