Thursday, November 15, 2007

 

Death Valley

Although the very thought of it makes some academics blanch, I'm beginning to think that 'evidence-based management' could be really useful in solving some nagging academic problems.

As I understand it -- and I'm no expert -- 'evidence-based management' takes as a given that it's appropriate to look at statistical patterns that have emerged over time, and to use those as reality checks for future decisions. It's particularly helpful in testing long-held assumptions for which we somehow keep noticing exceptions.

Most colleges of any size have someone in the back corners of the administration whose title is something like "Institutional Researcher." (The really big places have entire offices devoted to IR.) It's a sort of locally-applied social science. I think I've found something to ask my local IR guru to check out.

It's an article of faith among certain faculty that certain time slots for class meetings generate wildly higher attrition than others. ("Death Valley" is the local term.) The late afternoon weekday sections of required Gen Ed courses -- English Composition, say -- are held to be the last to fill, and the first to shrink as students just vanish. The "Prime Time" sections -- basically late morning to early afternoon, Monday through Thursday -- fill first, and usually finish the semester with almost as many students as they started.

So my questions for the IR guru, and for faithful readers who may know of research in this area:

1. Do Death Valley classes actually have higher drop rates? If so, does the effect disappear for upper-level courses? (If it does, I could see a pragmatic argument for running Gen Eds in prime time, and upper-level courses in Death Valley. That's pretty close to what Proprietary U used to do.)

2. If the drop rates are actually higher, is that really a function of the well-documented "last in, first out" rule of registration? (That is, the students who decide to sign up for classes the day before the semester starts have much higher drop/fail rates than the students who sign up months in advance. This is true, presumably, for many reasons.) Or do the students who sign up for Death Valley early also have higher drop rates?

3. Can we predict, with some level of specificity, the degree to which we could expect higher drop rates? Do those students usually come back, or are they usually lost for good? (Do they disappear at higher rates than other students who drop?)

4. If the answer to 3 is an actual number -- or at least a relatively modest numerical range -- could we do a little cost-benefit action to see if adding 'spillover' sections in Death Valley actually makes fiscal sense over time? (If that strikes you as cold, do the analysis in terms of future academic success.) Would we be better off, in the aggregate, simply abandoning Death Valley and going to waitlists and/or more online and/or hybrid offerings, or is Death Valley the least bad option?

5. Other than registering later, do Death Valley students differ in meaningful ways from Prime Time students? Are any higher dropout numbers actually consistent with those factors, rather than the time slot per se? (If so, then moving them around wouldn't really solve anything.) Or are these really the same students, just with funny schedules?

6. Do the instructors who typically teach the Death Valley sections -- and it's usually the same cohort from year to year -- have higher drop rates in Prime Time? Is this an instructor effect written onto a timeslot?

Wise and worldly readers -- I'm asking for your help in a couple of ways.

1. Do you know of any good research on this? It isn't the area of my scholarly training, and I'm just a wee bit busy.

2. Can you think of other questions I should add to the list?

Thanks for your help! My IR guru will likely have me killed, but evidence-based answers to these questions might actually be useful, which is sort of the point.


Comments:
Oh, wow, awesome that you want to do this.

My department frequently pesters the Powers-that-Be asking for the data so that we can do these types of analyses on our own (for free!), but the PTB claim that they can not get the data out of the computer system. (Not a FERPA or policy objection -- a technical one.)

If your IR guru can't devote time to your project, you can probably get someone in math or econ to do it in exchange for a very small bribe.
 
This is mainly for RH: From your blog, I gather that your system is similar in many respects to ours. It is hard to get stuff out of that system (pick-a-prof had to pay for the programming needed to scarf up their grade distribution data), but that is a one-time deal.

We, as a faculty, learned recently that all sorts of institutional data we thought were lost in the entrails of the system are actually available for internal use by planners (albeit maybe only recentlly) and will shortly be available to all faculty on our portal, and mining the full data set for cross-course correlations (passing as a function of grades in pre-req courses) will be supported.

It can be done. What has been done is already interesting.

DD: That is a great idea. Please let us know what you learn. One thing we know is that the true Death Valley is a section created to handle demand (opened up on the last day of registration), regardless of the quality of the instructor. Predicting enrollment so the right number of sections are available is a big effort here.
 
If the "Death Valley" effect disappears in upper level courses, I wouldn't necessarily say that it's a good idea to schedule them all in those time slots.

Since the difference between your gen ed courses and your upper level courses is that there are a bazillion sections of gen ed courses whereas with upper level ones, you have to take them when they're available... student behavior in the gen ed courses would seem to be a good indicator of student preferences in scheduling.

If you take the courses that there are few of, that students have to take when they are offered, and put them in the least convenient time slot for your students, it is entirely possible you are creating serious barriers to completing the degree for some students.

After all it's unlikely that all your students are saying to themselves, "I really hate 3:30 in the afternoon," but it may be things more like having to pick up their children from day care, work schedules, etc. that make those time slots unworkable for them. It seems unkind then to take advantage of the fact that some courses are ONLY offered in one time slot and therefore are a take-it-or-leave-it deal.
 
I used to do some class scheduling, and I bet you almost anything that the "Death Valley" effect has at least as much to do with PROFESSORS not wanting whatever timeslot that is as it does with students.


That said, there are a few things that influence enrollment against Death Valley Syndrome (DVS).

1. the professor is a star/has a cult of personality on campus.
2. the class is required of all majors.
3. the topic is sexy enough and/or is so infrequently offered, you take it when you can get it.

Can you schedule your "stars" at certain times?
 
I've been a database person at a number of colleges, and I can say with some certainty that the data is more often available than is admitted to. The trick is that more than likely your IT people have 'better things to do than some silly project for some darned professor'. Unfortunately the data pulls are rather complex, and probably not that many people can do them, so you'll probably be fighting for a very scarce resource. Also, sooner or later your registrar will notice someone poking around in student records, and that might cause some difficulty too. Sadly, I don't have many solutions to offer - if you can push the issue high enough to actually make it so that's one way - otherwise you might find you need to find a side door.
 
That sounds like a fairly simple multivariate regression; if you end up hitting a wall, send me the data and I'll run it for you.
 
As a PROFESSOR, I have taken on two death valley courses in the last two semesters. Both backed up to two sections of the same class earlier in the day. I am not a morning person. The death valley class would get the best of the three lectures (the most polished) with the most energy because I hit my stride around 3pm. Both classes had a nearly 50% drop compared the the 15-25% in the other two sections each semester. I do not think the problem is me.

Not that it is relevant to your potential data research Dead Dad (of which I would love to hear the results), but those death valley classes were both last minute additions.
 
Another great topic. And I'm glad to hear (or interpret in my favor) that it's not me!
I'm teaching two sections in Death Valley of, you guessed it, English Comp. It's a required class and the most basic of the writing classes in the English Department.
My drop rate this semester has been insane. Out of 50 students total I'm left with 22. I definitely think there is (in general) something different about these students. The good students are great and the rest are struggling. So I, too, would be interested to know the answers to your questions. I'd love to get some facts about drop rates at my institution for different times. I've taught the class earlier, and less people drop, but the pass rate isn't much better.
Thanks for asking those questions.
 
Since I am an adjunct, I get those classes every semester because the full timers usually get the earlier classes. I originally chose the DV classes because I am better in the aft., and now I seem to automatically get them.
This semester I taught a class in the morning after the professor had a schedule conflict and couldn't do it. I think the morning students are more focused or partly asleep. The drop out rate was about 10%. I am talking 8:00 AM. The aft. ones seem to be more sociable. Drop out rate-50%.
I think the aft. classes are usually overflows, so many of the students in them enroll at the last moment. To me that means they may not have well developed organizational or planning skills, which may be the cause of the high drop out rate. They just do not plan their assignment working times well or something along those lines and pretty soon they are overwhelmed. Maybe you could print a little booklet on organizing time.
I would like to see the results of the data, too.
 
Well, maybe if I have some time next term I can play around in our datamart and see what comes out. As department techie, I've been granted full access to the IR datamart (front-end, only, mind you, but you can customize the heck out of your calls). I'm just too swamped this term to even contemplate logging in anytime soon.

From departmental experience, I've seen more effect from the "last scheduled, first dropped" sections AND students than it being the timeslot. But it would be nice to see more data and figure out if that's more than a few odd one-offs of my own experience talking.
 
Sort of off topic, but I wanted to add that back in spring we happened to keep a list of all the "crashers" we overloaded to our sections or added to empty spots and the overload section --- this was a 150-spot large GE lecture. We happened to check these names against the grades at the end and noticed that, while not every single drop, fail or disappearance was a crasher, 90% of the crashers were drops, fails, or just vanished partway through.

I made sure to point that out to the prof as we really didn't like overloading each section by ten students to "do these kids a favor." Now I'm not really likely to let students crash full sections at all.
 
I think you've mentioned having at least one large research school near by. At my alma matter (which I think was R1) the statistics department ran a free service offering statistical consulting. For more in depth work, you could get a grad student or even an undergrad to do the analysis as a project. What you're trying to extract doesn't sound like it would be that hard to do, if you had enough data.
 
I had an interesting situation that can act as a test...

I was scheduled to teach an honors section which didn't fill with honors students (another problem, another post..). So, in the last few weeks of registration, it opened up -- in prime time (T/Th 9:30).

This is the WORST class I've ever taught. Their drop rate is pretty high and their failure rate will be much higher than any class I've taught in 'Death Valley'.

I'm doing the same material on the same schedule as my evening section, and the difference between classes is shocking.

I've also taught amazing sections of this course in the 9:30 time slot. Those sections fill up quickly, so I doubt the difference is me.
 
We're just beginning to do this analysis at our CC and what I have pulled from the numbers so far is this...

1) Classes scheduled in DV will fill last.

2) The "golden" classes will fill and get hefty wait lists

3) Drop rates seem no different unless...

4) As someone noted above, these classes are added at the last minute.

The DV idea is one faculty are fond of, as it lets them go home early. I've just started looking at Saturday courses and they seem to *not* be DV classes - our low level transfer ones all fill and overflow. Hard to say what this means, since we've offered so few of them, but I'd like to see a graduated expansion to figure it out..
 
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?