Wednesday, April 02, 2008

 

Ask the Administrator: They're Spying on Us!

A frustrated correspondent writes:

I work in a service department. Another department, which is a large client of ours, has scoured the records of grades to determine which instructors in our department are easy graders and which are hard graders. They are now advising their students to try to get into the sections taught by the easy graders.

I feel that this is about three kinds of wrong. For one thing, I think it's a violation of professional courtesy. You don't do a study like this on another department without the consent of the department. For another, if you do the study, you take the results to the department, not to the students. And frankly, I don't think that advisers should be taking the professional stand that the best thing to look for in an instructor is easy grading.

How would you, as dean, react to this issue? I'd like to see the offending department censured, but I'm not holding my breath. How should we, as a department, react to this issue? Should we take a stand (either publicly or privately), or should we simply ignore the issue? To what extent should we pursue policies that would make our grade distribution more uniform? That is, how important is it to deal with the underlying issue?

Juuuuust a few issues here...

In the age of the internet, it's much tougher to keep secrets than it once was. Unfortunately, some folks who aren't schooled in, say, statistics, will sometimes misinterpret facts and derive false patterns. (For example, some folks will confidently assert global patterns after hearing of two or three cases.) That's annoying, but it's increasingly a fact of life. Even if you manage to censure the department, which I wouldn't bother trying to do, you couldn't shut down the student grapevine. Word gets out. It often gets out in distorted, inaccurate, and unhelpful ways, but it gets out. Over at ratemyprofessor, one of the categories on which folks are rated is easiness of grading. Hell, back in the Stone Age when I was in college, we had a pretty well-developed (analog) grapevine about which professors graded easier or harder than most.

I'll admit wondering how they got their hands on the data, though. Since FERPA prohibits the time-honored practice of taping grade rosters to office doors, it should be considerably harder to get this kind of information in a systematic way than it once was. If there was some sort of skulduggery involved, then that might be cause for some sort of censure.

But let's say that, by whatever method, they've unearthed some non-trivial disparities. It's fine to make the point about airing dirty laundry in front of students, but that doesn't get the laundry clean. You need to address those disparities in a serious way.

This may be a blessing in disguise. Experience suggests that wild disparities in grading standards often reflect wildly disparate ideas of the point and/or proper level of the course. If you have faculty effectively defining the class in tremendously different ways, then you really aren't doing right by the students. A professor who passes everybody in Composition I is setting up his students to tank Composition II, and that's not fair to the students.

I've seen English departments convene 'norming' workshops, in which various composition instructors discuss how they'd grade some sample essays. This strikes me as a fantastic idea, since it manages to combine consistency with input. A standard announced ex cathedra, but without faculty buy-in, is meaningless. A standard the faculty develop for themselves, on the other hand, could actually work. (In a really high-functioning department, they'd road-test the standard for Comp I by having the local Institutional Research folk track success rates in Comp II. If the courses are understood as a sequence, rather than as stand-alone classes, then the students will actually have a realistic shot at success.)

Along these lines, one of my prouder moments at my current college has involved finally getting the ESL exit standards to align with the English entrance standards, so students who complete the ESL sequence are actually ready to take the English courses. That involved a certain amount of inter-departmental diplomacy, but I honestly believe it was the right thing to do for the students.

For courses that don't feed directly into sequences, the issue is slightly less clear-cut, but students should still have reasonable confidence that Professor X's General Psych class isn't significantly easier or harder than Professor Y's. There will always be judgment calls on the margins, which is to be expected. But large, sustained differences shouldn't happen.

Paradoxically, those norming workshops are obviously harder, yet also obviously more important, when you have a significant adjunct population with considerable turnover. Locally, we've found that paying adjuncts to attend, and providing a little food, makes a huge difference in their participation levels. A little time and money upfront saves plenty of misunderstandings later. No, it doesn't compare to an all-permanent faculty, but within what the taxpayers are willing to pony up, it helps.

In terms of what the other department is doing, you may or may not be able to prevent 'opportunity,' but you can deflate the 'motive.' If they find, over time, that the grading levels have become fairly consistent, they won't have much reason to spy, and everybody wins.

Good luck!

Wise and worldly readers – what would you suggest?

Have a question? Ask the Administrator at deandad (at) gmail (dot) com.


Comments:
If the correspondent writes from a public school, it might not be *that* hard.

For example, PickaProf.com has had some success in submitting FOIA requests for grade information, aggregated by instructor, which they then publish online.

In general, I think this sort of information probably ought to be available, and that departments need to get out in front of how it's presented.
 
I don't have a suggestion, but I just wanted to comment that I resent the implication by academia (heck, I do it sometimes, though, even though I hate it) that a grade distribution favorable to students means an easy instructor. Sometimes, it actually means an instructor who knows what they're doing and has set up a fair grading system, something we don't seem to hear about very much.
 
I agree with Joyce -- especially in a small department, a couple of profs with innovative teaching techniques and/or personalities that welcome student questions may result in those profs having statistically higher grades and/or completion rates. Heck, even the difference in textbooks for technical subjects might make a statistical difference.

I am worried about the correlation between grades and learning -- as well as the practice of guiding students to profs whose grades are higher... but, it doesn't mean that the profs with higher grades aren't just better at teaching....
 
I went to grad school at a university where the prof's eval scores and grade distributions and average GPAs (cumulative) for each class were made public on the web. I think it was allowed because grades were published in aggregate form and no one student was singled out. And yes, students did check that out to figure out whose section they wanted to take. I'll admit to looking to see if it was true that all doctoral students have 4.0s (checked out the average GPA for students enrolled in dissertation hours -- and the answer is no, the average was more of an A-).
 
Two things:

(1) At my grad school, the grade distributions for each class were posted online, provided that the class was larger than some certain number of students.

(2) I have to third Joyce's comment about the assumption of easy grading. A professor of mine was accused of grading too easy because his freshman chem average was higher than that of the overall department. This was partway through the semester. The department had a policy of using the same final exam across all sections and grading it jointly. Guess the result. The professor's students did better than the other students by about the same factor as they had all along. I'm sure that's not always the explanation, and it may even be rare, but the question and post seem to assume that it's not.
 
My Proprietary U. has identified "killer courses" and is on the way to identifying "killer profs" in much the same way the correspondent outlines. "Killer course" means any course with fewer than 80% of the students finishing with a C or better over a rolling 12-month period. Or, to look in the negative: more than 20% of the students have a course grade of F, D, or W (yes, withdrawals count).

I know that when the time comes, I'll be identified as a killer prof, because I primarily teach freshman-level and remedial courses with higher-than-average W rates. For the most part, the students who earn Ds and Fs in my courses are those who either: 1) don't come to class, 2) don't do the work, or 3)Plagiarize the work they do.
 
0. FERPA only prohibits posting grades in a way that might identify a particular student (e.g. with anon numbers but still alphabetical).

1. If it happens to be the athletic department doing this, be proud that your school is emulating the practice the Ann Arbor News documented at the University of Michigan. Surprisingly, some psych profs were upset that one psych prof had 45 independent study students in a semester (the norm there is 2 or 3) for what seemed to be an upper division study skills class with fewer hours than the norm.

2. It is fair to ask if this was done in a way that violated college rules, but I doubt it. They could just use pick-a-prof.

3. Like DD said, maybe there is something you need to look at. (See the second part of item 1.) Are there large and persistent differences? Do they correlate with p-t or untenured status? Do low scores reflect abandonment of a bad instructor, or do high ones indicate unearned grades to make up for poor instruction?

4. If sequential, can you see consequences in outcomes? Are the students in my class who don't do any homework a product of a "student centered" grading policy in earlier math classes? That might explain the inability to solve an equation.

5. What happens if the grade distribution is high because one year's group of students did the homework, and this year's group thinks it is easy ... and fails because they don't take homework seriously? I have some sample data on my blog showing an interesting correlation.
 
What amuses me about this is that the department that is rewarding grade inflation in the correspondent's service will no doubt be the first and loudest to complain that their students are learning nothing and have really poor writing skills, or whatever it is that the service courses are supposed to be teaching students. And in response to Joyce and others: yes, sometimes an effective teacher can raise the overall grade spread in a class, but it's probably more common for "easy" grades to reflect a lazy teacher...

--Shane in Utah
 
Huh. My undergrad first year calc sequence had unusually high retention and grade, but that was a combination of an experimental teaching method, a great prof, and a class dominated by older students in career transition. When we hit the second year, our new calc prof was staggered by what we knew.

The same experimental approach and material was offered again, and the new batch of first years struggled a lot more. They were much younger on average and had a different prof.

I don't know what made the difference, but I know it wasn't slacking. My classmates and I spent a lot of long hours holed up together studying. Periodically one of us would break under the mental effort and wax quasi-hysterical, but fortunately only one of us ever snapped at a time.
 
Well, it's "merit evaluation" time again at Compass heading U. We get peer and chair eval'ed on Research, Teaching, Service, etc.

Not fer nothin we just got done with the annual brouhaha over "how to fairly measure teaching performance."

Critical issues like:
- degree of difficulty (big freshman sections vs. small master's level sections etc.)
- role of student evaluations (Yes they do! No they don't!)
- role of grade inflation (easy graders get inflated evaluations!)

all seem to get stirred into a toxic stew of jealousy (professional and otherwise), envy, and greed.

Anyhow

A couple of us have low grade distributions *and* high student evals ("Tough but fair!"); this really seems to confound/confuse/reduce to a sputtering heap a lot of the more experienced (older) already tenured professors.

Sigh.

The *really* intersting question in all of this is:

Are the students our CUSTOMERS or are they our PRODUCTS?

Yikes.
 
"Are the students our CUSTOMERS or are they our PRODUCTS?"

Good question. My first year calc prof stands out in my mind because he was utterly devoted to producing as much understanding of the material in as many students as possible.

We complained A LOT. Exams wrung us out badly. But we got attached to and protective of him all the same, probably because of his utter faith that we could do whatever insane thing he asked.
 
Grades matter. They matter in many cases MORE than the content, I'm sorry to say: Six years down the line when you're applying to a PhD program in economics, nobody gives a shit that your sophomore psych 101 teacher was Ms. Reallyhard and your biology prof was Mr. Failem; they would rather admit the guy down the hall who got A grades in both from Ms. Easyclass and Mr. Donowork.

This is annoying enough. It's even MORE annoying if you end up in vastly disparate classes without getting to choose them; assigned classes sting more than elective when it comes to low-wealth classes.

Given that knowledge, what would YOU tell someone who wanted to go to grad school? Would you ignore the effect of a given professor on their GPA? If so, you'd be failing in your duty.

Grades are like money. If you assign them unequally, you deserve all the workarounds you can get. The difference betwee dean's list and not, between summa and not, between job offer and grad school and not.. it often comes down to 0.1 of a GPA, or less. Yet we all buy the right calculators and try to improve our study habits--it's a lot easier to raise your GPA by an easy professor. Only a fool would ignore that.

The obvious solution is simply to mandate that professors give out an equal amount of "grade wealth" for each identical class. This is especially important for large classes: if 1/2 your freshman class ends up with a 0.1 lower GPA average at the end of first semester because they got ms. reallyhard, how can you claim that's fair? Assign a mandatory average and let them curve how they please (this has its own set of problems but it eliminates the inequality issues.)

Teachers will complain. So what? They still have the freedom to reward the stars and screw the rest. Or they can go with a more average class. But if they think that they (as opposed to everyone else) have a magic ability to determine what a "fair" grade is I don't tink that's correct.
 
My classes fill soon after registration opens. These are usually the most motivated and disciplined students. My class averages are higher than those of my colleagues, in part, because my students walk in the door as more capable learners. I like to think that many of the elements that I bring to the classroom contribute to their success, but I know that any of my colleagues would see higher averages with my students than with the ones who end up in their classes.
 
Depends on how big your class is, and some random variation as to students. I don't think a class of Pre-Algebra will likely end up with the same grade distribution as a class of the same size of Honors Calc 1. Even in 30-person classes of the same size, I've seen vast differences in exam means -- one of my friends and I compared means for the same remedial math class. The mean score of my students was 15 points higher. We then double-checked grading, and found that while there were some things where we would have differed, overall having them regraded by someone else resulted in no significant difference. She just got a low-performing bunch of students, and probably >50% are going to fail. Why should she be forced to pass them, if they can't do the work? Or maybe everyone who DOES pass gets an A, just to force them to conform to some arbitrary mean?
 
To sailorman: when I teach two sections of the same class, they sometimes end up with noticeably different grade point averages, even though I teach and grade them in the same way. Should I add x points to one of two sections to make averages equal? I seriously doubt that the students in the other section would see that as "fair".
 
Quoth sailorman:

"The obvious solution is simply to mandate that professors give out an equal amount of 'grade wealth' for each identical class."

Doesn't that assume that each chunk of students is inherently the same? There's got to be some room for random variation. I know that I've had two sections of the same class where one performed significantly better than the other. Why? Because I'm an easier grader at 2:00 than at 1:00? Don't be ridiculous.

If you are gathering data over the long term, you might well find that some professors give more high grades than others. Maybe that's a problem to be fixed, and maybe its not. But I don't think that the appropriate fix is mandating an artificial uniformity of grading.
 
Based on my personal experience classes offered at certain time of the day attract certain kind of students. I used to teach a section of a freshmen calculus class, where the sections were offered every hour of the day from 8am to 4pm. We have found that consistently student would have a better grade average in sections offered at certain time (for example 10am), irrespective of who would teach it. The instructors were a mixture of tenured faculty members and graduate students, both American and foreign.
 
Change all the instructors in the catalog to TBA. Then students won't be able to pick an "easy grader" prof. Originally I thought of switching the profs at the last minute, or publishing incorrect instructors. But this is more honest and elegant.
 
Shortwoman: that's covering up the (alleged) problem instead of solving it.
 
Let's start with the different-teacher issue first:

How well do you think that you know the exact performance and academic indicators of the students who have taken everyone else's classes BUT yours? Everyone else, in the whole school? Don't forget to go across grade levels, up and down a year at least.

Not very well, right?

OK: Given that you don't know, and given the GPA-wealth analogy, how do you possibly justify giving less or more 'wealth' per student than any other teacher?

Again: Not very well, right? It's just that everyone else does it.

This is bad.

However, if you are teaching identical sections of the EXACT SAME CLASS, then I would agree that yes, you have some ability to distinguish which classes are more or less deserving of higher or lower wealth. (though even there, I'd want to be careful.)
 
sailorman: I don't give grades. Students earn them. And some groups of students earn more than others. You may have noticed that it happens with wealth, too.

Where I work, we do our best to make the coverage and difficulty of exams uniform across different sections of the same class. There are also common grading standards and a standard way of converting numeric grades into letter grades. Still, GPA varies from one section to another.

The procedures are up to us, and we make them as uniform as possible. Outcomes are up to the students.
 
Yes, I've heard the "earn" platitude before.

That said, I made two fairly specific points in the above post relating specifically to your ability to judge your student's performance relative to other sections. Can you answer them with any accuracy?
 
Students in other sections are evaluated by their own instructors. And as long as instructors in different sections follow similar grading procedures, I sleep just fine at night. Thanks for your concern.
 
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?