Tuesday, October 27, 2015


Why Good Student Course Evaluations Are So Hard to Find

What are student course evaluations for?

If the answer to that were simple, it would be easier to design them.  But student course evaluations -- and administrator’s observations, for that matter -- serve multiple purposes.

They serve as feedback for the professor.  While much of what students write in evaluations is contaminated by one form or another of a halo effect, it’s occasionally possible to discern something useful.  Sometimes they liked the class as a whole, but really disliked a particular assignment or reading.  Maybe an exercise intended to show one thing was taken to show another.  This kind of feedback is intended to be formative for the next semester or year.

They serve as safety valves for student opinions.  It’s harder for students to complain that nobody cares what they think when they get asked directly, over and over again.  Translating those opinions into observable action on a student’s timeframe is another issue, but they can’t say they weren’t asked.  That matters.

They frequently play into promotion or tenure decisions.  That’s probably untrue at research universities and only theoretically true at high-profile national colleges, but it’s typically true at community colleges with tenure systems.  In this sense, they’re summative.  They offer either confirmation of or counterevidence to professors’ claims of wonderfulness.

On the flip side, they can serve as ammunition for negative personnel decisions.  When a dozen students from the same class write variations on “good professor when she bothers to show up,” that’s a red flag.  Even the numerical part can be instructive.  At a previous college, I received the numerical rankings every year.  After a few years, I noticed that the same few names kept bringing up the rear, usually by a significant margin.  If the same person scores multiple standard deviations below the mean year after year, well, I have some questions.

The tricky part is that what makes for constructive feedback may not make for useful fodder for promotion decisions.  Formative assessments are great and humane, and I’m all for them when people are basically competent and actually trying.  But when those conditions don’t hold, for whatever reason, formative assessments aren’t terribly helpful in making adverse decisions.  If the decision gets challenged -- and it will -- you’ll need strong and unambiguous language condemning the performance.  When one form has to serve both purposes, it’s little wonder that it does neither well.

Even the ‘theater’ function is flawed at best.  The degree to which the opinions have an effect varies from case to case.  For a full professor with tenure, the external impact of the difference between “above average” and “meh” is approximately zero.  In that case, some student cynicism is hard to dismiss.  In the case of someone coming up for tenure, or an adjunct hoping to make the leap to full-time, the same difference could matter.  And the degree to which any given professor takes feedback and uses it constructively varies from person to person, even within ranks.

Clearly, student course evaluations shouldn’t be dispositive on their own.  Students respond to cues both appropriate (clarity, respect) and inappropriate (attractiveness, accents).  I read once that students tend to reward gender-conforming behavior: they prefer men who are authoritative and women who are nurturing.  Women who are authoritative and men who are nurturing get downgraded.  New and unexpected teaching approaches can be polarizing, so faculty who are not in a position to risk it may avoid innovation.  And if years of blogging have taught me anything, it’s that the caliber of anonymous comments can be, uh, let’s go with “uneven.”  Too much faith in any one source is a problem, even assuming that the source is clear and valid.  

I’ve seen and heard proposals to do away with student course evaluations altogether, but I’ve never seen a convincing alternative.  Theoretically, one could do pre- and post-tests to measure “value added,” instead of asking opinions, but how you’d get students to take pre-tests seriously isn’t clear.  I’d also hate to see higher education repeat the mistakes of K-12. You could measure performance in subsequent courses, though in small programs or departments you’d run into an issue of circularity, and in any size department you’d have trouble controlling for inputs.  Colleague and supervisor observations can help round out the picture, but they’re often so limited -- and even pre-announced -- that they come closer to measuring potential than performance.  If I observe a class for a day, I may see a terrific discussion, but I won’t notice that the professor takes a month to return papers.  The students are the only ones in a position to see that.  Shutting off that source means giving up on some pretty important information.

Wise and worldly readers, have you seen a particularly good version of student course evaluations?  Is there a reasonably elegant way to serve so many disparate purposes at once?

Our biggest problem with course evaluations is getting students to actually bother doing them. Years ago we transitioned from in-class paper-and-pencil evaluations to online ones, and the response rate is so low that they are practically useless. I have had classes of 24 students where not a single one actually completes the evaluation.

I do know of an institution where students who complete their course evaluations by the last day of classes get a special privilege that makes it worth it: they get access to a database where they can search composite data from previous evaluations by course and professor in enough time before the next semester that they can do a little rearranging of their schedules based on the results (which is limited, of course, because sections in prime time and with popular professors are already full).
Like every word you've ever written, Matt, this post is utterly without any merit. It's boring, tendentious, tedious, simplistic, and ultimately moronic and worthless drivel. Supervisor, pull the plug on this horrible blog.
^^^ LOL, Anonymous.

We had a really good system, but replaced it with an on-line system that was often worthless. (Some mediocre faculty were never evaluated because the response stats were too low to be statistically significant. This was easy to achieve if the prof didn't push the response in some way, because then the only notice was yet another spam-like e-mail from the college.) The only flaw in our earlier system is they used scantron forms that did not provide an option to write comments on the back. So they rarely got any comments unless you supplied your own blank paper. More expensive forms with a directive to put comments on the back (at a different institution) generated a lot of feedback. One of its other advantages is the response rate told the boss how many people were present on the day it was done. That alone would be a course evaluation.

Regarding your point about identifying problems, you could look for effusive praise of a certain type on Rate My Professor as a warning sign. I have no idea if our regular student evaluations provided any evidence of a huge (firing offense) underlying problem, because I have never been in a position to see those ratings, but I doubt it. No survey I've ever seen has asked some of the kinds of questions they have on RMP. What showed up on RMP was a really high score for "easiness" and comments about how students were told what was going to be on the exam so you didn't even have to attend class and, most importantly, that class regularly got out a half-hour early.
There is a giant body of scholarship on student evaluations. The short answer to your two questions is: 1) yes, there are good forms, and 2) no, there is no form that will serve all those disparate purposes adequately.

If a professor wants feedback on how the class is going and what the students perceive as serving their learning or not, it's way better to ask that in, say, week 4 or 5. Do a simple "stop-start-continue" questionnaire. Or, have your friendly local teaching & learning center consultant come to observe and interview the students. Then, TELL the students what you will and won't do, and why.

For the summative purpose, if I were king, I would make an iron-clad rule that the forms have to be created in accordance with best survey-authoring practices for reliability and as much validity as possible, AND that the data have to be analyzed and interpreted in statistically defensible ways!

Beyond that, it would be nice to see patterns acted upon -- professors who have a pattern of excellent student responses get asked to share their practices; professors who have a pattern of problems get held accountable for changing.

If I -- were King -- of the Foreeeeeeesssst!
I think that a lot of faculty members fear that course evaluations are only used negatively by the administration. A good course evaluation probably doesn’t matter for very much (especially at a research-oriented university), but a bad or mediocre course evaluation could be used against a faculty member when they come up for tenure or if retrenchment has to be done or if staff has to be cut. Adjuncts recognize that they are particularly vulnerable when it comes to course evaluations—just a few lukewarm course evaluations or even a couple of bad reviews could result in their contracts not being renewed for the next term. You definitely don’t want anything bad said about you on Rate_My_Professors.com.:-)

Consequently, in order to avoid negative reviews, there is pressure on faculty members to try and “game” the system, by easing up on standards, by giving simple or easy assignments, by giving lots of good grades, and by not challenging their students too much. Some faculty members even resort to handing out free pizza to their students at course evaluation time, hoping to drive up their scores.

Ideally, course evaluations should be for the benefit of the individual faculty member—they should be used to determine if what they are doing in the classroom really works well, or if something needs to be fixed or done differently. They should not be used by the administration as tools to reward or punish faculty—if they are used in this manner, this can lead to all sorts of perverse incentives.

My favorite part of our evaluation forms asks the students the following gem (quoted verbatim):

Length and difficulty of homework assignments were:
Excellent, Very Good, Good, Fair, Poor, Very Poor.

So if the students choose Very Poor, am I giving not enough homework? Too much? Too easy? Too hard?

You can guess, but you can't be sure.
While it's difficult to know exactly what you're getting with CTEs, some are better than others. At my former institution, in the school of business, the faculty had adopted a set of objectives for the program and (presumably) intended courses to do something to achieve those objectives. *Yet the CTE asked nothing about them.* Specifically, one of the major objectives was to foster teamwork. Yet the CTE did not even ask if the class included any team-based assignments.

Badly designed assessments will get less-than-useful results.

And I agree with all the comments about moving the CTEs from in-class to on-line. We also tried that, but the response rate was less than 10% across all courses. So we went back to in-class.
As with others here, we used to have a good system (Scantron forms with specific open-ended questions on the back—student filled out the forms during 15 minutes of class time during the last week of instruction). The administration decided to save money by moving the forms on line. The response rates dropped from around 85% to around 20% and the comments disappeared almost entirely. Now only a handful of disgruntled (or hero-worshipping) students fill out the forms, so the results are uninterpretable. The administration saved a tiny amount of money, at the cost of an enormous loss of information.
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?