Wednesday, April 02, 2008
Ask the Administrator: They're Spying on Us!
A frustrated correspondent writes:
I work in a service department. Another department, which is a large client of ours, has scoured the records of grades to determine which instructors in our department are easy graders and which are hard graders. They are now advising their students to try to get into the sections taught by the easy graders.
I feel that this is about three kinds of wrong. For one thing, I think it's a violation of professional courtesy. You don't do a study like this on another department without the consent of the department. For another, if you do the study, you take the results to the department, not to the students. And frankly, I don't think that advisers should be taking the professional stand that the best thing to look for in an instructor is easy grading.
How would you, as dean, react to this issue? I'd like to see the offending department censured, but I'm not holding my breath. How should we, as a department, react to this issue? Should we take a stand (either publicly or privately), or should we simply ignore the issue? To what extent should we pursue policies that would make our grade distribution more uniform? That is, how important is it to deal with the underlying issue?
Juuuuust a few issues here...
In the age of the internet, it's much tougher to keep secrets than it once was. Unfortunately, some folks who aren't schooled in, say, statistics, will sometimes misinterpret facts and derive false patterns. (For example, some folks will confidently assert global patterns after hearing of two or three cases.) That's annoying, but it's increasingly a fact of life. Even if you manage to censure the department, which I wouldn't bother trying to do, you couldn't shut down the student grapevine. Word gets out. It often gets out in distorted, inaccurate, and unhelpful ways, but it gets out. Over at ratemyprofessor, one of the categories on which folks are rated is easiness of grading. Hell, back in the Stone Age when I was in college, we had a pretty well-developed (analog) grapevine about which professors graded easier or harder than most.
I'll admit wondering how they got their hands on the data, though. Since FERPA prohibits the time-honored practice of taping grade rosters to office doors, it should be considerably harder to get this kind of information in a systematic way than it once was. If there was some sort of skulduggery involved, then that might be cause for some sort of censure.
But let's say that, by whatever method, they've unearthed some non-trivial disparities. It's fine to make the point about airing dirty laundry in front of students, but that doesn't get the laundry clean. You need to address those disparities in a serious way.
This may be a blessing in disguise. Experience suggests that wild disparities in grading standards often reflect wildly disparate ideas of the point and/or proper level of the course. If you have faculty effectively defining the class in tremendously different ways, then you really aren't doing right by the students. A professor who passes everybody in Composition I is setting up his students to tank Composition II, and that's not fair to the students.
I've seen English departments convene 'norming' workshops, in which various composition instructors discuss how they'd grade some sample essays. This strikes me as a fantastic idea, since it manages to combine consistency with input. A standard announced ex cathedra, but without faculty buy-in, is meaningless. A standard the faculty develop for themselves, on the other hand, could actually work. (In a really high-functioning department, they'd road-test the standard for Comp I by having the local Institutional Research folk track success rates in Comp II. If the courses are understood as a sequence, rather than as stand-alone classes, then the students will actually have a realistic shot at success.)
Along these lines, one of my prouder moments at my current college has involved finally getting the ESL exit standards to align with the English entrance standards, so students who complete the ESL sequence are actually ready to take the English courses. That involved a certain amount of inter-departmental diplomacy, but I honestly believe it was the right thing to do for the students.
For courses that don't feed directly into sequences, the issue is slightly less clear-cut, but students should still have reasonable confidence that Professor X's General Psych class isn't significantly easier or harder than Professor Y's. There will always be judgment calls on the margins, which is to be expected. But large, sustained differences shouldn't happen.
Paradoxically, those norming workshops are obviously harder, yet also obviously more important, when you have a significant adjunct population with considerable turnover. Locally, we've found that paying adjuncts to attend, and providing a little food, makes a huge difference in their participation levels. A little time and money upfront saves plenty of misunderstandings later. No, it doesn't compare to an all-permanent faculty, but within what the taxpayers are willing to pony up, it helps.
In terms of what the other department is doing, you may or may not be able to prevent 'opportunity,' but you can deflate the 'motive.' If they find, over time, that the grading levels have become fairly consistent, they won't have much reason to spy, and everybody wins.
Wise and worldly readers – what would you suggest?
Have a question? Ask the Administrator at deandad (at) gmail (dot) com.
For example, PickaProf.com has had some success in submitting FOIA requests for grade information, aggregated by instructor, which they then publish online.
In general, I think this sort of information probably ought to be available, and that departments need to get out in front of how it's presented.
I am worried about the correlation between grades and learning -- as well as the practice of guiding students to profs whose grades are higher... but, it doesn't mean that the profs with higher grades aren't just better at teaching....
(1) At my grad school, the grade distributions for each class were posted online, provided that the class was larger than some certain number of students.
(2) I have to third Joyce's comment about the assumption of easy grading. A professor of mine was accused of grading too easy because his freshman chem average was higher than that of the overall department. This was partway through the semester. The department had a policy of using the same final exam across all sections and grading it jointly. Guess the result. The professor's students did better than the other students by about the same factor as they had all along. I'm sure that's not always the explanation, and it may even be rare, but the question and post seem to assume that it's not.
I know that when the time comes, I'll be identified as a killer prof, because I primarily teach freshman-level and remedial courses with higher-than-average W rates. For the most part, the students who earn Ds and Fs in my courses are those who either: 1) don't come to class, 2) don't do the work, or 3)Plagiarize the work they do.
1. If it happens to be the athletic department doing this, be proud that your school is emulating the practice the Ann Arbor News documented at the University of Michigan. Surprisingly, some psych profs were upset that one psych prof had 45 independent study students in a semester (the norm there is 2 or 3) for what seemed to be an upper division study skills class with fewer hours than the norm.
2. It is fair to ask if this was done in a way that violated college rules, but I doubt it. They could just use pick-a-prof.
3. Like DD said, maybe there is something you need to look at. (See the second part of item 1.) Are there large and persistent differences? Do they correlate with p-t or untenured status? Do low scores reflect abandonment of a bad instructor, or do high ones indicate unearned grades to make up for poor instruction?
4. If sequential, can you see consequences in outcomes? Are the students in my class who don't do any homework a product of a "student centered" grading policy in earlier math classes? That might explain the inability to solve an equation.
5. What happens if the grade distribution is high because one year's group of students did the homework, and this year's group thinks it is easy ... and fails because they don't take homework seriously? I have some sample data on my blog showing an interesting correlation.
--Shane in Utah
The same experimental approach and material was offered again, and the new batch of first years struggled a lot more. They were much younger on average and had a different prof.
I don't know what made the difference, but I know it wasn't slacking. My classmates and I spent a lot of long hours holed up together studying. Periodically one of us would break under the mental effort and wax quasi-hysterical, but fortunately only one of us ever snapped at a time.
Not fer nothin we just got done with the annual brouhaha over "how to fairly measure teaching performance."
Critical issues like:
- degree of difficulty (big freshman sections vs. small master's level sections etc.)
- role of student evaluations (Yes they do! No they don't!)
- role of grade inflation (easy graders get inflated evaluations!)
all seem to get stirred into a toxic stew of jealousy (professional and otherwise), envy, and greed.
A couple of us have low grade distributions *and* high student evals ("Tough but fair!"); this really seems to confound/confuse/reduce to a sputtering heap a lot of the more experienced (older) already tenured professors.
The *really* intersting question in all of this is:
Are the students our CUSTOMERS or are they our PRODUCTS?
Good question. My first year calc prof stands out in my mind because he was utterly devoted to producing as much understanding of the material in as many students as possible.
We complained A LOT. Exams wrung us out badly. But we got attached to and protective of him all the same, probably because of his utter faith that we could do whatever insane thing he asked.
This is annoying enough. It's even MORE annoying if you end up in vastly disparate classes without getting to choose them; assigned classes sting more than elective when it comes to low-wealth classes.
Given that knowledge, what would YOU tell someone who wanted to go to grad school? Would you ignore the effect of a given professor on their GPA? If so, you'd be failing in your duty.
Grades are like money. If you assign them unequally, you deserve all the workarounds you can get. The difference betwee dean's list and not, between summa and not, between job offer and grad school and not.. it often comes down to 0.1 of a GPA, or less. Yet we all buy the right calculators and try to improve our study habits--it's a lot easier to raise your GPA by an easy professor. Only a fool would ignore that.
The obvious solution is simply to mandate that professors give out an equal amount of "grade wealth" for each identical class. This is especially important for large classes: if 1/2 your freshman class ends up with a 0.1 lower GPA average at the end of first semester because they got ms. reallyhard, how can you claim that's fair? Assign a mandatory average and let them curve how they please (this has its own set of problems but it eliminates the inequality issues.)
Teachers will complain. So what? They still have the freedom to reward the stars and screw the rest. Or they can go with a more average class. But if they think that they (as opposed to everyone else) have a magic ability to determine what a "fair" grade is I don't tink that's correct.
"The obvious solution is simply to mandate that professors give out an equal amount of 'grade wealth' for each identical class."
Doesn't that assume that each chunk of students is inherently the same? There's got to be some room for random variation. I know that I've had two sections of the same class where one performed significantly better than the other. Why? Because I'm an easier grader at 2:00 than at 1:00? Don't be ridiculous.
If you are gathering data over the long term, you might well find that some professors give more high grades than others. Maybe that's a problem to be fixed, and maybe its not. But I don't think that the appropriate fix is mandating an artificial uniformity of grading.
How well do you think that you know the exact performance and academic indicators of the students who have taken everyone else's classes BUT yours? Everyone else, in the whole school? Don't forget to go across grade levels, up and down a year at least.
Not very well, right?
OK: Given that you don't know, and given the GPA-wealth analogy, how do you possibly justify giving less or more 'wealth' per student than any other teacher?
Again: Not very well, right? It's just that everyone else does it.
This is bad.
However, if you are teaching identical sections of the EXACT SAME CLASS, then I would agree that yes, you have some ability to distinguish which classes are more or less deserving of higher or lower wealth. (though even there, I'd want to be careful.)
Where I work, we do our best to make the coverage and difficulty of exams uniform across different sections of the same class. There are also common grading standards and a standard way of converting numeric grades into letter grades. Still, GPA varies from one section to another.
The procedures are up to us, and we make them as uniform as possible. Outcomes are up to the students.
That said, I made two fairly specific points in the above post relating specifically to your ability to judge your student's performance relative to other sections. Can you answer them with any accuracy?