Thursday, June 09, 2005
Peer Review, Small Audiences, and The Incredible Shrinking Guilds
From what I’m told by people in those cognate disciplines, though, the same problems hold true there, as well; I’m just reading the exceptions.
Without taking my personal taste as normative, I still have to wonder how a field with such wonderfully interesting subject matter became mired in such turgid irrelevance. When I try to think of contributions to the public discourse to emerge from my field in the last thirty years, I can think of exactly one, and that one is iffy.
What the hell? How can so many bright people, applying rigorous (and very, very extended) study to compelling issues, produce such unreadable crap?
I’m beginning to suspect ‘peer review.’ In order to be taken seriously, a scholarly article needs to be ‘peer reviewed,’ or given the stamp of approval by people already in the field. The idea is to prevent quackery, or faddism, and to keep a premium on academic rigor.
Well, okay, but has it worked?
Freakonomics and Moneyball, two of the more interesting popular books of the last few years, are both premised on the (empirically-tested) idea that groupthink can trump evidence. They both tell stories of people using empirical proof to show that articles of faith among ‘experts’ in a given field are either false or badly limited. In both books, the empirical evidence had to become overwhelming before the groupthink was broken.
What would a Freakonomics or Moneyball of academia look like?
Some fields lend themselves easily to reality checks. An engineer friend used to like to tell the story of the first-day lecture by a prof in her first semester of grad school: You Build Bridge. Bridge Fall Down. No Partial Credit! If the bridge falls down, peer review ain’t gonna save it. Other fields, like literature, are impervious almost by definition. If the very subject matter is, by definition, fiction, then empirical evidence can only be of glancing relevance.
(Before the inevitable flaming, let me just admit that I’m using broad strokes here. Certainly, it’s possible to discover new facts about old texts. Although I’m using Manichean language here, I’ll admit that there’s more of a continuum in reality. Polemical license.)
In the social sciences, we’re somewhere in between. There are some basic facts we can use to check interpretations – I recall explaining to one incredulous undergrad that Hamilton couldn’t have been influenced by Marx, because Hamilton died before Marx was born — but there’s also considerable space for interpreting the facts that exist. More basically, there is considerable disagreement over what constitutes an interesting problem.
In theory, I guess, peer review could help make up the difference between what we can verify objectively and what makes for a good interpretation. In practice, though, it seems to fall prey to a really dreary version of groupthink. A typical journal submission will be sent to three ‘peers’ to review. If one of those three doesn’t find the problem interesting, that’s that.
Interesting-ness was never supposed to be a criterion. Peer review is the worst possible mechanism for determining interesting-ness. In a sample size that small, interesting-ness is somewhere between groupthink and whim.
Accordingly, we get journal articles about ever-smaller and more specialized subjects (such as would appeal to, say, three reviewers), using ever more arcane methods (each carefully footnoted), while the world blissfully ignores everything we say.
I think this is part of why so many younger academics have taken to blogging. The gatekeepers are so caught up in the internal fetishes of their respective guilds that they’ve lost sight of the big picture. Blogging gets rid of the gatekeepers, and lets authors find their own peers. If the people who find my stuff interesting come from literature, or engineering, or chemistry, that’s cool.
As a biochemist correspondent noted, guilds were built to ensure that production took place in the interests of the producers. When consumers found other options, the guilds dissolved rapidly. I see the guild system of publishing (and all that goes with publishing) starting to break down. The growth of the University of Phoenix (and of business majors generally) is a signal from the public that we’ve stopped talking about what they care about. The academic job market, as brutal and perverse as it is, is a kind of reality check; the guilds are losing their ability to reproduce themselves. Maybe they’re producing what nobody wants?
Sorry for such a shaggy rant. I haven’t nailed this one down yet. Any thoughts?
I don't know what the better system is, but I still think there's something deeply flawed with peer review.
And DO NOT EVEN get me started on how long it takes to get a paper published...
Also, I think you get at another big problem, which I see among junior faculty all the time: writing for the three people who know more about the subject than you do.
If you're going to write something, you need to believe that you are the authority (and you are, technically, the authority on what you think about any given thing) and that your reader doesn't know as much about it as you do. That way your audience is automatically larger.
Plus, it just can't go on the way it is. Libraries are cutting way back on journal subscriptions; presses can't afford to do books with print runs of only 300; and tenure goes to the published. The only real, viable solution is for people to start writing better books for larger audiences.
I'm convinced a practicle solution would be to utterly do away with the dissertation and instead, use that time to help grad students learn how to write engaging books. Let the "dissertation" actually be a real first draft of a book, instead of making folks write something no publisher wants without substantial re-envisioning.
Some highly-rated journals (Public Choice is one) are not peer-reviewed--the editor decides, and the turnaround time is very short. But then you face the issue of writing to the editor's tastes, don't you? At least with (blind) peer review, the review is done by someone who does not who the author is. And we know that this tends to increase acceptance rates of articles by junior scholars, by people from less-prestigious institutions, by women, by...everyone except the established names.
(And don't get me started about how badly written some of our submissions are...)
And Moneyball is, of course, written by a non-academic, Michael Lewis, who is a first-rate observor.
Without the tenure system, I'm not under massive pressure to publish. I just like to...but I really don't try to write more than one or two papers a year. I have too much responsibility in the classroom. At my last university, I knew an absolutely lovely man in the philosophy department. He was nearing retirement age, and had been at the uni for 40 years. He taught four courses every term, as well as several graduate seminars. He'd graduated over 50 PhDs. He held senior positions on every departmental board, and quite a few uni-wide ones. And in his career, he published exactly one article, 30 years ago. He was a brilliant teacher, but had no truck with research, or the writing process. I would rather have a dozen like him, than a hundred scholars eager to publish a paper on any old bit of rubbish,
Could I publish in peer-reviewed journals? No. There was always some old geezer who felt threatened by the new approach, and who tried to make me toe the obscure-what-you-have-to-say-until-no-one-can-understand-it line. But why would I want to write a book, or a few articles, to gain tenure, which would be read by, at most, half a dozen people? And where would I find the supportive senior scholars IN my field to write the necessary letters?
Blogging is much more democratic. If you write well, people will read what you have to say, whether they are familiar with your field or not. They'll also be free with constructive criticism. Even if the criticism is not so constructive, the exchange of views still seems a so much more breathable air, than the over-rarified air that the academy is currently suffocating itself upon.