Monday, January 08, 2007


Rubrics and Templates and Bears, Oh My!

If you really want to get faculty backs up, start dropping terms like 'rubric' or 'template.'

In a recent meeting with some other deans, we discussed the ways in which teaching effectiveness is evaluated. (As a cc, teaching effectiveness is the single most important component of a professor's evaluation.) It quickly emerged that there's no single way to do it. We do class observations, but there's no set list of criteria to use in judging whether a given class worked or not. We have end-of-semester student evaluations, which are easily quantified but of limited value. For reasons I still don't fully understand, we don't even have access to some fairly basic data, like grade distributions and drop/fail rates.

One of my colleagues opined that, in the absence of some pretty clear templates, we're setting ourselves up for lawsuits if some professor takes issue with her evaluation. Since class observations now are basically holistic writeups of what went on, along with the 'expert opinion' of the observer as to whether it worked, it might be hard to defend any given opinion in court. Playing anticipatory defense (which is a pretty good description of administration generally), he suggested codifying as much as possible what a 'good class' looks like, distributing the template/rubric ahead of time, and just checking off the various items (“delivered material to multiple learning styles”) as we go.

I could see his point, but something about the idea makes my skin crawl.

It's easy to come up with a list of teaching practices that, all else being equal, would generally be considered positive. It's harder to engage every single one of those in any given class period, or in every single discipline. When a given observation results in checking off, say, only 50 percent of the boxes, is that good or bad? This strikes me as the wrong question.

For example, our lab science classes are typically split into two components: lecture and lab. If you observe the lecture, you'd come away criticizing the professor for being abstract. If you observe the lab, you'd come away criticizing the professor for not explaining very much. If you observe both, you're violating the faculty contract.

In other disciplines, other issues crop up. It's easier to do 'hands-on' in a ceramics class than in a philosophy class – does that make the ceramics professor good and the philosophy teacher bad? While we generally think that incorporating technology is a good idea, is it automatically true that PowerPoint makes a class stronger? (I've seen both effective and ineffective uses of it.) For that matter, is it automatically true that lectures are bad? Is ambiguity always bad? (In my perfect world, “learning to tolerate ambiguity” would rank as one of the most important goals of higher education. How that's supposed to happen without ever encountering ambiguity is beyond me.)

Anticipatory defense can be prudent, but it can also lead to valuing all the wrong things. Just as faculty resent teaching to a test, I'd expect them to resent teaching to a template, and rightly so. Some of the best classes I've seen – hell, some of the best classes I've taught – have been at least partially serendipitous, the result of an unanticipated student comment that sent the discussion reeling in exciting ways. The ability of the professor to react to the teachable moment, to call an audible when the defense lines up in an unexpected way, is part of the professor's professional toolkit. The ability to recognize when that's done well or badly is part of the evaluator's professional toolkit. That's not to deny that evaluators can be obtuse or that class discussions can go horribly wrong, obviously, but to say that those risks are part of the cost of doing business.

From a liability standpoint, rubrics are appealing. They lay out in easy-to-digest form some nice, neutral criteria by which you could defend any particular adverse finding. They place a 'check' on any given observer's power to notice only what he wants to notice. They ought to be able to prevent the worst abuses. But they do so at the cost of conveying the overall success or failure of the class. I've seen professors break rules and succeed, and I've seen classes in which I couldn't quite specify why, but it just didn't work. Rubrics are based on the fiction that the whole is exactly the sum of its parts.

Have you seen a rubric that really worked well in capturing whether or not a given class was successful? If there's a way to play anticipatory defense and still capture what I need to capture, I'm all for it.

I think the rubric would work just fine. The problem is that your sample size is too small. I'm sure you dont' have time to check out each professor multiple times throughout the course but that's probably the only way. The the science class for instance needs two reviews, one for lecture and one for lab, at a minimum.
I suppose the question I'd have is this: what makes evaluation by rubric any more "authoritative" or "objective" than the current system? If a faculty member wanted to make a legal case against his/her evaluation, he/she could still make a claim that the evaluation process was unfair. This is my problem with rubrics generally: they give the appearance of objectivity, but at the end of the day, what you'd have is still a subjective evaluation of one class throughout the course of a semester, right? To me, in order for the "rubric" thing to have support among faculty:

1) The rubric would need to be designed by a committee across disciplines (which, of course, could take years).
2) I agree with Joe that you'd need to have multiple evaluations of a professor per class per semester, also not an incredibly efficient use of time.

Wouldn't an easier solution to this problem be to have the evaluator and the professor have some sort of communication about the evaluation immediately after it and to have the professor sign off on the evaluation before it's placed into the file? If the professor has a problem with the evaluation, couldn't there then be a process in place where the professor could request another evaluation of his/her teaching? To me this seems like a more reasonable approach, but then, I'm a humanities type who doesn't really have much faith in "objective" measures for evaluation.
I agree with Dr. Crazy here about the perception of objectivity.

I've had some really great observation experiences, and some not so helpful ones. One of the problems with yours is that it sounds purely evaluative rather than even partly developmental. Shouldn't the idea be to help the instructor teach better, at least in part?

The most valuable observation experiences (being observed and doing observations) have involved a good meeting ahead of time, where the instructor talked about the context, goals of the course and session, difficulties and strengths of the class and so forth. And then after the class, bother observer and instructor talked about what happened in the class, what went well and what didn't.

Those meetings helped prevent the problematic situation of an observer trying to fit an instructor into the observer's stylistic expectations when those are inappropriate.
At my CC there is a standard form used for classroom evaluations, then a conversation with the observing dean -- at which the faculty either signs off or records objections.

I'm not sure that teaching effectiveness can be quantified. To improve instruction, the best thing that can happen is to provide many means by which instructors can improve their classroom interactions.

It is also reasonable to assess student outcomes.
I worked for a big teacher-training and placement organization. (You've probably heard of them, but let's not bring that into it just yet) They had incredibly good evaluations, but they would be impossible for you to implement. Teachers worked in groups ("collabs"), and evaluated themselves and each other. They met in larger groups that were also evaluated by their group leaders. There was a team of people who floated from room to room, observing, evaluating, and giving feedback. Any of the admin staff or random passers-by could also leave observation and evaluation data for the teachers to read, which was cosidered (sort of) in the teachers' overall feedback. Each student was tracked on multiple levels, and the data was aggregated and presented on a week-by-week, teacher-by-teacher, collab-by-collab (and overall institute) level.

That's obviously not doable, nor truly desirable at the level of teachers already in the classroom. (Our teachers were in the classroom, but they were also being trained simultaneously, so there was a direct link between 'what they're doing' and 'what they're learning')

There were some things that were really useful, though. The observation rubrics (the most basic evaluatory ones, that anyone could do, not the extensive, point-by-point ones that took hours) had some concrete questions and three (depending on the form used) broad categories for notes. Those were either plus/minus/delta, or plus/delta/questionmark. The latter is the one I prefer. (Plus = positive feedback, minus = negative feedback, delta = suggestions for change, question mark = questions)

A rubric that would preserve flexibility and allow thoroughness would follow a similar model. A bunch of easily seen and quantified sections at the top (checkboxes, yes/no/short answer questions, whatever); "Is class attendence 80% or more?" "Is there a clear topic of discussion?," followed by a few more subjective, but still direct questions ("Do students seem engaged with professor?" "Is information presented clearly?" "Are multiple learning styles being engaged?") with some space for short answers. Then on the bottom, a plus/delta/questionmark section with some guidelines for feedback (keep it concrete, give an explanation where needed; "Very clear handwriting on board; students seemed to be able to take notes easily." "You seem to pay more attention to the right side of the room when calling on students. Perhaps moving around the classroom would allow you to engage other students." "Why did you use Power Point for this lecture? It seems like it would have lent itself more to discussion.")

In the interest of getting people to use it, portioning this out as a one or two page evaluation would work fairly well. If you can get a shared language going (hard, if your institutional culture isn't there, but very useful), you're ahead of the game.

My instincts for this would be a three-page packet. The first page giving an overview of the new rubric (and being not-so-covvertly an opportunity to a) get people in the right mindset and b) add another layer of preventative defense, so everyone knows what is being looked for, generally), and giving some concrete examples and suggestions for things to look for. The second page has your checkboxes on the front, short-answer questions on the back (or checkboxes on top, short-answer on the bottom). Third page has plus/delta/questionmark on the front, and a big blank space on the back labeled: "Notes and observations." (Or, again, the top/bottom deal) Professors are, of course, encouraged to add as many pages as they need for additional observations. Debriefing afterward between the observer and the oberved is reccomended, but not mandatory.

"This rubric is designed to aid you in your observation of this class. While it is by no means exhaustive, it is intended to help guide your thoughts when observing someone in a discipline other than your own. The first page contains questions about the setting and overall feel of the classroom. The second page contains space for observations. Be concrete and specific, and try to phrase your feedback constructively. If you have a question, be sure to record that, as well."

... heh. I'm sure I just overthought that by quite a bit. The simple answer is that it's perfectly possible to come up with good rubrics that don't stifle, and don't put the emphasis too far into the realm of simple quantifiability. Getting it approved and accepted, now that's the hard part.

(I must say, however, that I find the idea pretty exciting. I am such a dork.)
The standard form we use has very vague categories (e.g. "content mastery," "rapport with students) that the individual evaluator then rates with a number and writes a brief description of the individual teacher's approach to that category. Both the instructor and the evaluator later meet and sign off on the form (as Dr. Crazy suggests), and there's a process for disagreement. It's not a perfect system (vagueness of categories? Here's where Grandiloquence's packet is vastly superior), but it is a way of assessing everyone in a relatively transparent manner, and you know the criteria before you step into the classroom.
This seems a place where it would be useful to collaborate with the local school district/system...

They have been doing this for years and have likely developed some decent thoughts...

And, if your CC is like mine then making connections to the district was something we were "supposed" to do.

If you approach them in the spirit of "this seems like something we could work on together" it would probably go over quite well.
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?