Friday, November 18, 2011


Black Boxes and Bumping

A few years ago, I read a piece about airlines “bumping” passengers who had legitimate tickets. (I’ve never understood how it’s legal to sell the same seat twice, but that’s another post.) The article made the point that some central computer makes decisions that result in bumping, but that agents at counters have to deal with angry passengers, so over time, agents at counters started entering ‘dummy’ passengers with names like Mickey Mouse so they could outsmart the computer. Mickey Mouse wouldn’t yell at them if he got bumped. Over time, the computer compensated by overbooking even more.

The flaw there was that the folks who designed the central system never thought about the needs of the folks on the ground.

I’m watching a few state/national higher ed initiatives -- well-intentioned ones -- come to grief, and they all seem to be flailing for the same reason. They treat colleges as black boxes. They fail to grasp the motivations of the various actors.

Take transfer, for example. It’s one thing for a state to declare that its entire public higher ed system should be a coherent whole, with seamless transfer from each college to every other. And on paper, many of them have that.

But that doesn’t mean students escape having to re-take (and pay again for) courses they’ve already taken and passed.

That’s because while broad policy decisions may be made at the top, actual implementation occurs in the departments. And departments often have very different interests.

In the world of transfer, the usual evasion involves giving a course “free elective” status. The chair of the receiving school’s art department, for example, typically won’t raise an issue with accepting English Comp or Intro to Psych, since her own department doesn’t teach those anyway; the nits she’ll pick will be among the art classes. I’ve had chairs say, to my face, that they don’t want to “give away” too many credits. But if she’s under a mandate from above to accept credits in transfer, she can simply allow the transferring art credits as “free electives.” If her major doesn’t happen to have any free electives in it, well, tough luck. That way, her department gets to re-teach whatever it wants, while she still gets to claim compliance with the mandate. Her interest -- keeping the enrollment and funding levels of her own department high -- are at odds with the larger systemic interest in seamless transfer.

Now that states are starting to define college “performance” in terms of graduation rates, I can see a similar -- considerably more sinister -- version of the same thing on the horizon.

Graduation rates reflect any number of variables, including quality of curriculum and instruction. But those variables also include things like the student profile. To take an easy example, students who arrive at college with strong academic preparation in high school graduate at much higher rates than students who arrive with serious skill gaps. Nobody seriously disputes that. So the quickest and easiest way for a college to nudge its graduation rates upward is to become exclusionary. If you don’t let the higher-risk students in, they can’t drop out.

Some colleges build that into their missions, and that’s fine. If you need developmental math, MIT won’t take you. It’s a private university -- albeit a land-grant, oddly enough -- and it can choose its own path. But to compare graduation rates of places that can cherry-pick with places that take all comers is simply to load the dice.

My concern here is that unthinkingly adopting a single bottom-line standard will push the more accessible colleges to become much less so. They won’t necessarily want to, but if funding depends on it, they’ll do what they’ll have to do. If we assume the same kind of self-interest as in the case of the department chairs, it isn’t hard to predict either evasive or perverse maneuvers.

Those maneuvers could be overt -- admissions requirements, say -- or they could be sub rosa. Moving ESL and developmental classes onto a separate set of books, for example, would immediately elevate the graduation rate. Discreetly reducing outreach into the most disadvantaged communities would elevate the graduation rate. It isn’t hard to come up with ways to game the measure.

As with the airlines, I’d expect the people on the front lines to engage in evasive maneuvers to meet their own needs. The folks who would get bumped would be the most vulnerable students. Bumping is one thing if it’s Mickey Mouse, but something else altogether if it’s a kid from a shaky high school who’s trying to escape poverty. Colleges aren’t black boxes or agents of a single mind; they’re complicated operations with self-aware moving parts. Policies need to reflect that. If they don’t, entire generations will be left sitting on the tarmac.

Congratulations. You've just discovered what happens when you rewards and/or punish schools and teachers based on how well their students do on an external measure.
Like with the Airlines, the trick would seem to be to make the cost of selling the seat twice too large to make it worth it. Today, with TSA screening and the need to appear at an airport an hour early, even a free ticket isn't usually worth the expense of a flight delay. So regulate that the airline has to pay damages (real costs plus punitive) for overbooking and suddenly planes will no longer be over-bookable.

In the same sense, creating a single metric to evaluate schools (like graduation rates) will lead to serious homogenization of the schools. It creates a single optimal strategy. In a decentralized educational system (like the US), it seems that you;d want diversity in approaches. Otherwise you might as well have a national education system.

With the airline thing (long background in it...) There's some subtle nuances in there, as the airline doesn't really sell seats, it sells access to inventory. Don't believe me? You can book a seat 330 days in advance of your flight, but the airline won't actually assign a particular aircraft type until a few months in advance.

That said, overbooking allows the public access to cheaper seats. If my aircraft holds 100 seats, and I know from my data that there is a 90% chance that 5 people won't show up, then I can safely sell 105 seats. That means I either make more money, or lower the price of your ticket. From an economic perspective, this just increases the supply of seats, which lowers prices.

Despite the frustrations that some people have, the amount of people who are involuntarily denied boarding is really small. Airlines will give out $300-$500 (or a free ticket) to take a later flight. I'll take $300 any day of the week to take a later flight on the same day. (And these people really aren't angry, most of them are quite thrilled.)

But I honestly can't think of a system that would incentivize the ground agents to play nicely. If you give them a bonus for this situation, they can actually increase the number of IDBs (which is really bad), and the airline will never know, and the passenger will be disadvantaged.

So, to your larger point, I think in some situations, it's impossible to create a structure where everybody behaves exactly the way you want them to.
Do you ever watch the show "Airline"? Not sure if it's still taping but it comes on LMN or something like that. Hysterical look at working for Southwest and the crazy customers. If I paid for my ticket, showed up two hours in advance and got bumped, I'd be steamed. But if I rolled up about 10 minutes before the plane was supposed to depart and got bumped. I really have no room to whine.

On the transfer side, my state has worked out the transfer thing pretty well. If you graduate from one of the state CC's then all of your credits transfer as a block. Even LL's into your UL core program courses. You still have to take your 45 UL credits and your 30 credits at our school but you don't have the free elective issue. Most courses have been articulated and if they aren't, we will on a course by course basis for each student.
Then there's the fact that graduation rates can also be raised by lowering standards in the classes that students find it hard to pass. This certainly happens at the 4-year colleges and also at the 2-years, except when the students also have to pass certification or licensure exams in order to enter their chosen occupation.
So true... We're "eagerly" anticipating several changes that have been dreamed up by people at the top, who probably have never set foot in the classroom and so have no idea what chaos these changes are likely to create on the ground.
I'd be interested in knowing what state-mandated transfer policies look like in other states. In Indiana, there's a master list of courses that all state-supported universities are required to accept for transfer. This "transfer library" has all the equivalent courses at all the institutions. So while there are still games that can be played, it gets harder...
As with the airlines, I’d expect the people on the front lines to engage in evasive maneuvers to meet their own needs. The folks who would get bumped would be the most vulnerable students. Bumping is one thing if it’s Mickey Mouse, but something else altogether if it’s a kid from a shaky high school who’s trying to escape poverty. Colleges aren’t black boxes or agents of a single mind; they’re complicated operations with self-aware moving parts. Policies need to reflect that. If they don’t, entire generations will be left sitting on the tarmac.
Folks trying to pay for quality in healthcare are facing a similar difficulty. For instance, we'd like to pay providers for successfully managing their patients' diabetes rather than paying them for multiple visits from patients with complications from poorly controlled diabetes. One metric for judging the quality of diabetes management is the percentage of a provider's diabetic patients have A1c results under 7% at annual exams. Using such a metric, though, providers whose patient populations are generally well insured, financially stable, and well educated are much more likely to get a good score than providers whose patient populations don't fit that description. So, if providers are being paid based on their patients' A1c results, they have an incentive to cherry-pick the patients who are best equipped to control their diabetes and avoid those who aren't.

In the health policy field, there's a sense that it's feasible and worthwhile to resolve the cherrypicking problem. It'll take a lot of research and testing to figure out how to adjust providers' quality scores to account for differences in their patient populations, but there are researchers committed to figuring this out (and some funding for them). Is the same true in higher education?
Come to think of it, moving ESL and developmental classes onto a different set of books WOULD improve graduation rates. Would this necessarily be a bad thing? CC's vary in the proportion of students they have who are ELL's, and the proporation they have that are academically unprepared. Would it be a bad thing if they were compared based on their non-ELL, non-remedial grad rates AND also rewarded for improving the grad rates for ELL's and developmental students?
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?