Thursday, April 22, 2010
If only it were that easy...
Over the last couple of years, I've been pushing a much stronger adherence to data-driven decision-making. In the process, I've seen firsthand many of the obstacles to its adoption. I put these out there not in the spirit of opposing data-driven decisions -- I still think it's a damn good idea -- but in the spirit of explaining why it's so hard to move from idea to implementation.
First, there's the availability of personnel. Although most colleges have Institutional Research (IR) departments, they're typically understaffed and overwhelmed with federal and philanthropic reporting requirements. It's harder to argue internally for resources for an IR staffer than for a direct service provider, since most of what IR does is at at least one remove from working with students. If you don't get that Nursing professor, you'll have to shrink the program next year; if you don't get that IR staffer, well, what will happen, exactly? Since it's harder to argue for short-term direct benefits, it tends to lose battles to positions that are easier to explain. While that makes short-term sense, over time it means that the quantity and quality of data at hand will be limited.
Second, there's the data itself. Looking backwards, we can know only what we thought to track at the time. Sometimes we can rejigger old data to tell us new things, of course, but if certain key questions weren't asked -- it usually takes the form of "we didn't flag that in the system" -- then it's just not there. Colleges weren't designed as research projects, so a great deal of what has gone on over time was done without any kind of eye towards future research. Even when something is done self-consciously as a 'pilot' -- meaning as a research project -- it's often difficult to isolate the relevant variables. Did the small project succeed because it was well-designed, or because it was small?
Third, there's the clash between the need to plan ahead and the need to wait for the data. The owl of minerva spreads its wings at dusk, but we can't always wait that long. When you have to set up next year's schedule before the results from this year's experiment are in, you have to default to hunches. If the program in question uses the gap time to tweak its own delivery, it can always explain away the first, lousy results with "yes, but that's before we did such and such." Worse, in any given case, that could be true.
Then, there's the clash between the drive to innovate and the deference required to "past practices." This can sound trivial, but it's actually a major issue.
For example, one of the Gates foundation programs contemplates setting up dedicated classes for at-risk students in which the program manager serves as the primary contact person for the students, including being their academic advisor. The idea is to give students a trusted person to go to when issues arise. But the union here has taken the position that academic advisement is unit work, and can only be done by unit members. Since management is not unit work by definition, we can't follow the Gates guidelines even if we wanted to. It's a shame, too, since the program seems to have good early results where it has been tried.
The 'past practice' issues become hairier when you look at 'modular' or 'self-paced' alternatives to the traditional semester schedule. By contract, faculty workloads are based on credit hours and the semester calendar. (Similar expectations hold in the realm of financial aid.) If you break away from those models, you have to address workarounds for financial aid -- which have serious workload impacts for the financial aid staffers -- and unit concerns about workload equity. Maintaining workload equity while experimenting with different workload formats is no easy task, and some unit members are just looking for an excuse to grieve, for reasons of their own. It's not impossible, but the process of 'impact bargaining' and its attendant concessions amounts to an unacknowledged deadweight cost. That's before even mentioning the time and effort involved in dealing with grievances.
Then, of course, there's the tension between fact and opinion. When there's a long history of decisions being made based on group processes that have been dominated by a few longstanding personalities, they'll read anything data-driven as a threat to their power. Which, in a way, it is. I saw this a couple of years ago in a discussion of a course prerequisite. A particular department argued passionately that adding a prereq to a particular course would result in much higher student performance. The college tried it, and student performance didn't budge. After two years of no movement at all in the data, I asked the chair if he had changed his mind. He hadn't. Facts are fine, but dammit, he wanted his prereq, and that was that. Facts were fine in theory, but when they got in the way of preference, preference was assumed to be the "democratic" option. Since facts aren't subject to majority rule, reliance on facts is taken as anti-democratic.
Finally, there's the tension between the culture of experimentation and the culture of "gotcha!" The whole point of the data-driven movement is to get colleges to try lots of different things, to keep and improve what works, and to junk what doesn't. But when a college has an entrenched culture of "gotcha!," you can't get past that first failure. If something didn't work, you'll get the self-satisfied "I told you so" that shuts down further discussion. A culture of experimentation, by definition, has to allow for some level of failure, but that presumes a cultural maturity that may or may not be present.
None of these, or even all of them in combination, is enough to convince me that we should stop testing assumptions and trying new things. But they do help explain why it isn't nearly as easy as it sounds.
I did assessment projects for Residence Life at Penn State, and can say that your comments are dead-on.
The difficulty of navigating institutional politics can be especially frustrating. It isn't just about what to justify or what to change or get rid of. Many projects stumble coming out of the gate, due to disputers about who should be doing the assessment in the first place...the department, the division's assessment officer, or the college's formal "institutional assessment" office.
Such political wrangling gets in the way of common sense, because the higher up (or furhter out) the chain you go, the less connected you are to the issues which led to the reason for assessment in the first place.
It's encouraging that so many master's programs (from which we draw new staff) are emphasizing assessment skills. Hopefully, we'll work to get over some of our hangups and learn how to harness these skills and properlu direct them. Until then, I guess we have no choice but to keep "coralling the cats" until some of them come home.
Some organizations have caught on to this danger and not only encourage risk taking but actually celebrate some of their failures. They've found that the best way to counter the fear of failure when trying new things is to celebrate some of the great failures. It's a way to relieve some of the pressure and fear of failure and also use the failure as an opportunity to say, "Even though this didn't work out, it was a great idea. What can we learn from the failure and what might we try differently next time?"
IR is important and when it gets screwed up, it causes real problems.
I've seen other people get excited over numbers that mean nothing. My department chair is excited because we're now keeping records of attendance at seminars. That would be great if he wanted to use this data to justify things related to seminars, but he thinks we can use it to justify all sorts of things only very, very peripherally related to the seminar (if at all) because "It's numbers! It's assessment!"
Well, it's definitely something with "ass" in the name...
Although most colleges have Institutional Research (IR) departments, they're typically understaffed and overwhelmed...
Spoken like an ex-literature prof. :) Let your IR people stick to record-keeping, then take a stroll over to your math and social science departments and you'll find more than enough expertise for your analysis. If you turn over enough rocks you might even find someone like me that is already doing their own assessment and would help you with college-wide data for little more than kudos and a beer. After all, you can tell him, curricular decisions are faculty governance, right?
There is also more to the time-frame issue you raise. Your desire to frame a question, test it, and implement the answer "next fall" strikes me as horribly out of place in an academic setting. Maybe welding schools work that way, but not people who teach from 2500 year-old texts and wear 500 year-old costumes even when it's not Halloween. Try something like "have an idea one semester, talk about it the next, try the test the following semester, chuck those results out because you later realized that your tool was flawed (it always is), try the revised tool again for a few semesters so you have enough data to be reliable, and then a semester to let the results percolate through the relevant committees." You'll have a lot more luck convincing people that way. There is more to this than a vague stodginess, though.
Putting my research methods instructor's hat on, I'm pretty confident that any data you gather in a short-term (one-year) project is subject to enough unknowable measurement error to warrant a great deal of skepticism. Personally, I wish that "data-driven" advocates would think about their results like a Bayesian, in cycles of "start with a belief, compare it to a piece of the world, modify the belief a little, compare it to the world, again, and again." Not only does that produce more robust results, it also makes a much more convincing argument than "our quick study last semester said A so we're going to re-write our curriculum to feature A."
Finally, I've listened to numbers for years, but you know, I've never heard them speak for themselves. The notion that "facts are facts" and there is only one implication thereof is a superficial way of looking at data. While the example you cite may be more cut and dried than most, I would suspect even that one is more nuanced than you describe. Some of the negative reactions you encounter may be from people who resist your assertions about the "facts" (especially from short-term research).
I like collecting data on teaching and I do think it can be useful to shape curriculum development. It even helps me improve my teaching, but it has become painfully clear to me that is not the data that counts.
Due to budget issues, the four year comprehensive where I work has decided to double the size of our introductory surveys. The idea is that we should be a 'cash cow' for the college of liberal arts. And we need to do this with fewer faculty members.
One of the department learning outcomes is writing proficiency demonstrated by a senior thesis. Well, all of our majors take six survey classes and we expect them to write essays in those classes. Doubling the size of the survey means scantron tests instead of essays.
So a curriculum decision that should have been made on the basis of assessment data was actually made on the basis of budget data. That decision effectively blows up our assessment plan too. Next time I will not spend as much time working on that assessment plan, because frankly, I know that the budget makes it irrelevant.
Then get some stats and social science people involved. Please note that local universities have some of these, and are frequently willing to help. Especially if publications and dissertations are involved.
If things are going well at that level, *and* the funding is secure for years, then hire a stats guy yourself.