Tuesday, September 30, 2014


A Question for the Statisticians

In a recent exchange with one of my favorite higher education peeps, I realized that I’ve been stubbing my toe on the same issue for years now.  I’m hoping that some very smart readers have found an elegant way around it, the better to spare my theoretical feet.

Let’s say that you’re trying, for whatever reason, to measure the average earnings of graduates from a particular college or set of colleges.  And let’s pretend that actually getting the data is relatively easy, just for the sake of argument.  (“In theory, there’s data.”)  

What do you do with the students who transfer to the next level of education, instead of getting a job?  

For community colleges, the issue is students who move on to four-year colleges.  If you measure earnings a year after graduation, they’ll look markedly low, but that’s artificial.  If you count those students as they are, you’ll get a misleadingly low average.  If you exclude them from your count, you’ll essentially exclude the highest-achieving echelon of graduates.  We have grads who have gone on to medical school and done quite well for themselves, but they don’t show up in our numbers because the gap between graduating community college and making big money as a physician is too long.  

Four-year colleges face a related issue.  How do you count the students who went on to law school, med school, or, heaven help us, grad school?  In the first year out, or even the second, they’re making peanuts.  But they’re on a track that historically has done quite well, other than certain flavors of grad school.  The future cardiologist will do fine, eventually, but by then it’ll be too late for your numbers.

Again, factoring them out means discarding the cream of the crop, which hardly seems fair.  But leaving them in builds a systemic distortion.

Many of these issues go away if you assume a long enough timeframe.  Look at earnings twenty years out, and get a much clearer picture.  But if the point is to hold colleges’ feet to the fire, using data from twenty years ago isn’t likely to work.  The lag is too long.

I could understand the “just don’t count them” solution for, say, grads who drop out of the workforce to have children.  They’re playing a different game.  It isn’t necessarily as clean as that, of course; dropping out may be more appealing if your job is low-paying and dead-end.  But for the sake of simplicity, I could understand assuming choice.  

Graduates who join the military may also bring some low numbers initially, which I would argue are also misleading.  

For the sake of simplicity, let’s leave out students who “swirl” among several different institutions.  If a student transfers three times before graduating, which college should get credit for the earnings?  

It’s tempting to just throw up one’s hands and declare the entire enterprise a fool’s errand, but politically, that’s a non-starter.  If there will be “accountability,” how should we count?

Arguably you should be predicting life-time earnings from the data that you have, and if you have enough historical data then this could be manageable. For example, if a student transfers into major X at university Y, and historically transfers into this program have long-term earnings of Z, then you can make some sort of prediction of the long-term outcome for this student. But do you really have data that good?
Good thinking, Morgan Price.

...but all of this kind of IS a fool's errand. Living and dying by the numbers is silly when the numbers will never capture the full picture.

And how sad it is that college's only purpose is now to get a job. What ever happened to learning for the sake of learning and being a better citizen. John Dewey must be rolling in his grave.
I would un-ask the question, because education is not about next week. A society that looks at education as something that needs to pay off immediately, and eliminates all options that don't do so, is going to die. (One can make the same argument about a similar approach to business. Notice where that concept took us in 1929 and 2008?)

Should we punish an elementary school because its grads don't make much money in 6th grade? A HS because its grads make very little while in college?

An AA degree (and dual-enrollment credits) should be measured by looking at the odds of succeeding at the transfer institution and earnings 5 and 15 years after the final degree is earned. Those metrics (success and income) should also count for/against the transfer school, which should be penalized for the failure of AA transfer students just as it should be for any student who has to transfer to a CC before returning to graduate.

PS to comment above:
IMO, there is a difference between learning for its own sake (a life of the mind) and Dewey's view of educating the mind so everyone can help run the nation. Education still has practical ends, but ones that require more than trade school just as they require more than being merely an "educated" voter who can't hold a job.
There is value in data on students moving on to other education; I have definitely advised students to look for those numbers. Why not let them stand for themselves? It could be something like XX% of graduates move on to higher education from our program (with whatever relevant breakdown, because of course, there are data!). Other students opt to go directly to the work force, and earnings average [some amount of money, broken down as you choose].
@CCPhysicist - I see your point, but the political environment needs answers now because so much public money is involved. Politicians have voters to worry about, and voters (in my area very much so) are worried about finding jobs. Politicians need to be seen as doing something about it, and college/uni grad data is one of the data sources they need. I agree with the spirit of your point, but don't see its practicality.

@Morgan Price - Agreed, especially your question at the end. Perhaps this is what DD means when he quotes "In theory, there's data".

If different graduate streams make different amounts of money, then separate the streams and show them separately. In other words, show what transfer students ultimately make, what direct graduates/workforce entry make, and what "life happens" people make 2 years out. Some of those numbers won't be glamorous, but then your work changes to explaining why some grads make more than others. If including transfers 2 years out drags your ratings down, while including them 20 years out lifts your ratings, then you either have to accept the distortion (good and bad), or present them as their own dataset.

An alternate way of presenting the data is to present the data as X years into the workforce, rather than X years post-graduation. That would get around your transfer-issue quite cleanly. Is that politically do-able?
This points out some of the negative effects that are introduced by a college or university trying to manage itself strictly by the numbers. First, one has to consider whether the numbers that are being gathered and used to manage the organization actually mean anything. Next, as Matt says, is the means by which the numbers are gathered actually reliable and honest? Is there something important that is being missed, or are there things that are being overcounted? If not, this might mean that the organization is being managed by bad data.

When management develops a nearly Pythagorean obsession with numbers, this can lead to some very perverse incentives. Lest they get the Feds or their accreditors angry with them, or if they let their ratings slip, management now needs to make sure that the college or university “meet its numbers”. College and university managers become as narrowly focused on short-term results as corporate executives trying to satisfy analysts’ expectations for the next quarter’s balance sheet. Any faculty member or employee who does something (or fails to do something) that is perceived as preventing the university from meeting its numbers is in danger of being dinged on their next performance review. Will the dean get canned if the college has too low a graduation rate? Will the college president be in danger of being dumped by the trustees if the college slides downward in the ratings? If the gainful employment numbers look bad, will the Feds come down hard? If an assistant professor fails too many students, could they be in danger of failing to get tenure? Will a part-timer who fails too many students get thrown under the bus? Since meeting their numbers is now so important, there is a temptation for management to fake the numbers or make up the results, just to keep them out of danger. The system will encourage everyone to lie to everyone else, all the way up the chain.

Perhaps most important is the danger that a college or university that is too strictly managed by a set of numbers can lose sight of its reason for being there in the first place. The mission on the college or university is now no longer to increase knowledge or to educate the next generation of citizens in a democracy, it is now strictly to meet its numbers.

I'd add some other metrics, like % working in the field of their choice, % admitted to graduate school, etc.

I think it's fair to limit your income reporting to people who are full-time employed, as long as you're also including a metric that tells your data consumers how big a chunk of your data is being excluded.

You could argue against that of course. People in grad school earn less, but so do people doing internships or taking jobs with significant "dues-paying" periods.
Some thoughts about the purpose of a college education.

Under the college ratings system being proposed by the Obama administration (as well as possibly upcoming gainful employment regulations), the true purpose of a college education seems now to be for graduating students to be able to obtain good jobs which provide enough income, presumably to ensure that college loans can be repaid.
Now , it has certainly been true that the ability to obtain a good, well-paying job upon graduation has always been an important part of a college education. But until very recently, this was never the most important reason for going to college. More important was the opportunity to obtain a liberal education so that one gained exposure to the most important aspects of our culture so that one could become a more productive and better-informed citizen in a democracy. An important product of a successful college education was the development of intellectual curiosity, along with the ability to learn new things, so that one could make accommodations as the economy and society in general undergo changes. Best to teach a graduate how to learn—a curriculum too narrowly focused on teaching skills that are now in demand on the current job market runs the risk of teaching skills that may well be obsolete in only a couple of years.

But back in the day, a graduate could usually count on getting a decent job upon completing their education, one which would a sufficiently high salary so that they could live a middle-class lifestyle, to perhaps buy a house, start a family, and take a vacation now and then. But things are now quite different—the job market has tightened considerably and a college education is no longer a guarantee of being to live a middle-class lifestyle. Employers are now a lot more fussy, and demand a lot of experience in the particular projects that they are working on before they hire anyone. The want an employee to be able to jump right in, without needing a lot of training.

So the pressure for colleges to train their graduates to be able to survive in the current job market has increased greatly in recent years. It now appears that colleges will be judged on how much money their graduates make immediately upon graduation. But how does one measure this? Does a liberal arts graduate who is now flipping burgers at Wendy’s count the same as a STEM graduate who is now in medical school? How do you compare a collage graduate who immediately obtains a job at a Wall Street hedge fund with a graduate who enrolls in Harvard Medical School—does the first one count favorably in the rankings, whereas the second one counts against the college? Does a community college get punished when too many of their students transfer to four-year colleges? Will a graduate who is taking an unpaid internship count against the college in the rankings system? Will the college get dinged if too many of their graduates enroll in graduate school and who are currently making very little money? A meaningful rankings system based on graduate employment needs to able to look further down the road—more significant would be for a college to look at how well their graduates are doing 10 to 15 years down the road.

As I mentioned earlier, a college rankings or ratings system based on the amount of money that their graduates make could lead to a whole bunch of very perverse incentives. Colleges will be pressured to eliminate programs and majors that graduate a whole bunch of students who do not immediately make a lot of money. There will be pressure to introduce courses and majors designed to teach skills that are now in demand on the current job market. College presidents, deans, and vice presidents will rise or fall based on how well the college does on these gainful employment scores. More seriously, colleges may be tempted to fake the statistics or turn in fake numbers, just to avoid getting into serious trouble.

Seems like a 3-4 year delay would solve most of your questions, especially at the CC level.

Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?