Thursday, June 09, 2005

 

Peer Review, Small Audiences, and The Incredible Shrinking Guilds

A friend recently asked me who I’ve been reading. I had to admit that most of the folks I’ve been reading for the last several years have been from outside my home discipline. (I don’t just mean the management lit, but the stuff I read to keep my ‘scholarly’ side alive.) When asked why, I could only say that the stuff from cognate disciplines is simply better – better focused, less caught up in the fetish of methodology, less imprisoned by footnotes, more willing to take risks.

From what I’m told by people in those cognate disciplines, though, the same problems hold true there, as well; I’m just reading the exceptions.

Without taking my personal taste as normative, I still have to wonder how a field with such wonderfully interesting subject matter became mired in such turgid irrelevance. When I try to think of contributions to the public discourse to emerge from my field in the last thirty years, I can think of exactly one, and that one is iffy.

What the hell? How can so many bright people, applying rigorous (and very, very extended) study to compelling issues, produce such unreadable crap?

I’m beginning to suspect ‘peer review.’ In order to be taken seriously, a scholarly article needs to be ‘peer reviewed,’ or given the stamp of approval by people already in the field. The idea is to prevent quackery, or faddism, and to keep a premium on academic rigor.

Well, okay, but has it worked?

Freakonomics and Moneyball, two of the more interesting popular books of the last few years, are both premised on the (empirically-tested) idea that groupthink can trump evidence. They both tell stories of people using empirical proof to show that articles of faith among ‘experts’ in a given field are either false or badly limited. In both books, the empirical evidence had to become overwhelming before the groupthink was broken.

What would a Freakonomics or Moneyball of academia look like?

Some fields lend themselves easily to reality checks. An engineer friend used to like to tell the story of the first-day lecture by a prof in her first semester of grad school: You Build Bridge. Bridge Fall Down. No Partial Credit! If the bridge falls down, peer review ain’t gonna save it. Other fields, like literature, are impervious almost by definition. If the very subject matter is, by definition, fiction, then empirical evidence can only be of glancing relevance.

(Before the inevitable flaming, let me just admit that I’m using broad strokes here. Certainly, it’s possible to discover new facts about old texts. Although I’m using Manichean language here, I’ll admit that there’s more of a continuum in reality. Polemical license.)

In the social sciences, we’re somewhere in between. There are some basic facts we can use to check interpretations – I recall explaining to one incredulous undergrad that Hamilton couldn’t have been influenced by Marx, because Hamilton died before Marx was born — but there’s also considerable space for interpreting the facts that exist. More basically, there is considerable disagreement over what constitutes an interesting problem.

In theory, I guess, peer review could help make up the difference between what we can verify objectively and what makes for a good interpretation. In practice, though, it seems to fall prey to a really dreary version of groupthink. A typical journal submission will be sent to three ‘peers’ to review. If one of those three doesn’t find the problem interesting, that’s that.

Interesting-ness was never supposed to be a criterion. Peer review is the worst possible mechanism for determining interesting-ness. In a sample size that small, interesting-ness is somewhere between groupthink and whim.

Accordingly, we get journal articles about ever-smaller and more specialized subjects (such as would appeal to, say, three reviewers), using ever more arcane methods (each carefully footnoted), while the world blissfully ignores everything we say.

I think this is part of why so many younger academics have taken to blogging. The gatekeepers are so caught up in the internal fetishes of their respective guilds that they’ve lost sight of the big picture. Blogging gets rid of the gatekeepers, and lets authors find their own peers. If the people who find my stuff interesting come from literature, or engineering, or chemistry, that’s cool.

As a biochemist correspondent noted, guilds were built to ensure that production took place in the interests of the producers. When consumers found other options, the guilds dissolved rapidly. I see the guild system of publishing (and all that goes with publishing) starting to break down. The growth of the University of Phoenix (and of business majors generally) is a signal from the public that we’ve stopped talking about what they care about. The academic job market, as brutal and perverse as it is, is a kind of reality check; the guilds are losing their ability to reproduce themselves. Maybe they’re producing what nobody wants?

Sorry for such a shaggy rant. I haven’t nailed this one down yet. Any thoughts?

Comments:
I'm still very junior, but I've been ranting about peer review ever since I started. It's very clear in my field, at least, that people are turf-protectors. If you don't mention their work (if it's even minorly related), they think you haven't considered enough ideas. Or, if you're too interdisciplinary, people reject your paper because they only know how to evaluate half of it.

I don't know what the better system is, but I still think there's something deeply flawed with peer review.

And DO NOT EVEN get me started on how long it takes to get a paper published...
 
I think it has to do with the tenure process as well as peer review. Doing what others are doing in the same boring, stuffy way that they did it is a pretty good way to ensure that they'll grant you tenure.

Also, I think you get at another big problem, which I see among junior faculty all the time: writing for the three people who know more about the subject than you do.

If you're going to write something, you need to believe that you are the authority (and you are, technically, the authority on what you think about any given thing) and that your reader doesn't know as much about it as you do. That way your audience is automatically larger.

Plus, it just can't go on the way it is. Libraries are cutting way back on journal subscriptions; presses can't afford to do books with print runs of only 300; and tenure goes to the published. The only real, viable solution is for people to start writing better books for larger audiences.

I'm convinced a practicle solution would be to utterly do away with the dissertation and instead, use that time to help grad students learn how to write engaging books. Let the "dissertation" actually be a real first draft of a book, instead of making folks write something no publisher wants without substantial re-envisioning.
 
I have the responsibility of managing the manuscript process for an on-line journal dealing with teaching and learning, which involves managing our peer-review process. It's an inter-disciplinary journal, so people frequently review submissions outside their own disciplines. And it's frustrating for me as well, not so much because the reviewers wear disciplinary blinkers, but because of the difficulty of getting rapid and thoughtful reviews done.

Some highly-rated journals (Public Choice is one) are not peer-reviewed--the editor decides, and the turnaround time is very short. But then you face the issue of writing to the editor's tastes, don't you? At least with (blind) peer review, the review is done by someone who does not who the author is. And we know that this tends to increase acceptance rates of articles by junior scholars, by people from less-prestigious institutions, by women, by...everyone except the established names.

(And don't get me started about how badly written some of our submissions are...)
 
I forgot to mention that virtually everything that shows up in Freakonomics began life as an article in a refereed journal. Whether that's proof that the review process works, or that Leavitt is just an exceptional researcher and interesting writer, I leave for someone else to decide.

And Moneyball is, of course, written by a non-academic, Michael Lewis, who is a first-rate observor.
 
In my field, I'm always stunned by the disparity in writing ability. Some authors write clearly and convincingly on whatever topic they turn their pen to. The data are always well presented, the arugments clear. Other authors can make even compelling evidence dull, while some seem to view the writing process as a way to keep an axe ground for every perceived wrong or slight.

Without the tenure system, I'm not under massive pressure to publish. I just like to...but I really don't try to write more than one or two papers a year. I have too much responsibility in the classroom. At my last university, I knew an absolutely lovely man in the philosophy department. He was nearing retirement age, and had been at the uni for 40 years. He taught four courses every term, as well as several graduate seminars. He'd graduated over 50 PhDs. He held senior positions on every departmental board, and quite a few uni-wide ones. And in his career, he published exactly one article, 30 years ago. He was a brilliant teacher, but had no truck with research, or the writing process. I would rather have a dozen like him, than a hundred scholars eager to publish a paper on any old bit of rubbish,
 
And this is exactly why I left the academy. As a junior scholar, I was doing exciting field work, pulling in the grants, but working in such an arcane field with such a new take on things that....there was no-one to be a peer reviewer. Plus, I wanted to make my work accessible to a wider audience because I knew there was an audience interested in what I was doing. (Gave a lot of lectures to the "general public.")

Could I publish in peer-reviewed journals? No. There was always some old geezer who felt threatened by the new approach, and who tried to make me toe the obscure-what-you-have-to-say-until-no-one-can-understand-it line. But why would I want to write a book, or a few articles, to gain tenure, which would be read by, at most, half a dozen people? And where would I find the supportive senior scholars IN my field to write the necessary letters?

Blogging is much more democratic. If you write well, people will read what you have to say, whether they are familiar with your field or not. They'll also be free with constructive criticism. Even if the criticism is not so constructive, the exchange of views still seems a so much more breathable air, than the over-rarified air that the academy is currently suffocating itself upon.
 
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?