Thursday, February 03, 2011


Thoughts on "Academically Adrift"

Still marooned by snow -- seriously, guys, the bloom is off the rose -- I had the chance to devour Academically Adrift, by Richard Arum and Josipa Roksa. It’s a study of student performance on the Collegiate Learning Assessment exam, focusing particularly on demonstrated critical thinking skills. It’s the book that made headlines with its claim that most students don’t learn anything during their first two years of college. As someone who works at a two-year college, I considered the gauntlet thrown.

My first observation, which was largely ignored in the initial wave of reports, is that the sample they used only included four-year colleges. Community colleges were not included in the sample. Based on the rest of the findings, I doubt that we’ve cracked the secret code, but it’s certainly a glaring oversight for a study of the first two years of college.

That said, I’m pretty conflicted in my responses overall. It’s an impressive piece of analysis, certainly. The data work took some doing, and the prose is evenhanded and relatively clear by social science standards. (Social scientists aren’t known for our limpid prose, as a breed.) I’m just not sure how useful it is.

My takeaways:

- I wasn’t shocked to see that math, science, humanities, and social science majors tend to show the greatest gains in critical thinking, as compared to students in business, health, computer science, or communications. The authors were gracious enough not to go into too much detail there, and it would be politically unwise for me not to heed their example, but it was hard not to notice.

- Also not shocking: students who are assigned more reading and writing get better at reading and writing. In a related story, bears tend to crap in the woods.

- Group study was no more conducive to improved critical thinking skills than socializing. Individual study correlated strongly to improved skills; group study did not. This certainly fits my own personal preference and intuition, but it’s nice to see empirical confirmation.

- “Student engagement,” as measured by NSSE (and presumably CCSSE), correlates to retention, but not to increased learning. Fraternities and sororities lead to higher student satisfaction, but lower learning. Again, not shocking, but nice to see confirmed.

- Instructor expectations matter. Students who have more professors with high expectations learn more than students who don’t. Given that students will often go out of their way to seek out ‘easy’ professors and avoid ‘hard’ ones, this suggests a dilemma.

- Federal research funding dwarfs federal funding for improving instruction -- say, FIPSE -- by several orders of magnitude. Incentives matter.

- What’s good for retention may or may not be good for learning. Students can stick around for years without learning much, depending on what they’re doing. Arum and Roksa note that learning communities are positively correlated with retention in the national literature, for example, but there is no evidence one way or the other of their effects on learning.

- Reflecting on my time at Snooty Liberal Arts College, I could see why its students would do markedly well on tests of critical thinking. It had no ‘business’ or ‘communications’ majors; it had very selective admissions and therefore a strong ‘peer culture,’ and it lacked frats. My cc also lacks frats, but the other components don’t really carry over.

- ‘Peer culture’ is huge. If you run with a crowd of high achievers, you will adapt to it; if you run with a crowd of hard partiers, you will adapt to that. In an open-admissions institution, this presents a substantial challenge. (Some peer cultures are trickier than others. Coming from a public high school in a middle-class suburb, it took me a semester to raise my game when I got to SLAC. I didn’t know that the prep school kids affect insouciance in public while studying like crazy behind closed doors.)

- On-campus employment helps, if it’s up to ten hours a week. Off-campus employment hurts. Interestingly, grants help but loans hurt.

- Many students see college (and here the fact that it’s a sample of four-year colleges may matter) as primarily a social experience. It’s a chance to get away from Mom and Dad, to make new friends, to explore lifestyle options, and to get a credential. If that’s your orientation, then ‘learning’ is fine, as long as it doesn’t require time and effort. In that climate, lone instructors who raise academic expectations may pay a price in student anger.

At my cc and at most that I’ve seen, dorms don’t exist, and the whole “college experience” is pretty attenuated. (We don’t have climbing walls, a football team, fraternities, or even a quad.) If football Saturdays are your idea of college, you don’t come here. That said, though, it’s still very much the case that academics are often only one priority among many in students’ lives. As our student body gets progressively younger and more ‘traditional,’ some of the quirks of 18 year olds will probably become more relevant here.

- To their credit, Arum and Roksa note that making sustained and significant progress on student critical thinking skills would require fundamental realignments of incentives across the entire structure of higher ed. They seem a little too quick, in my estimation, to assume that “employers” want critical thinking skills -- at the entry level, in my observation, they’re much more focused on enthusiasm than on analytical prowess -- but that just makes matters worse.

And the incentives point is what’s ultimately so frustrating about the book. Yes, it would be lovely if students naturally clustered into the liberal arts, where virtuous and civic-minded professors larded their plates with ample helpings of robust reading and writing assignments. In the settings where that actually happens, measured learning outcomes are strong. But when you have open-door admissions and low per-student funding, getting there from here would require changes of staggering magnitude. Funding mechanisms would have to change; national markets would have to change; collective bargaining agreements would have to change; longtime readers can guess the rest...

Still, it’s a reminder of some of the right questions, and it sheds useful light in some corners. Maybe expanding the “individual quiet study” area in the library should take precedence over the “group study” section; I can do that. Maybe a little more skepticism towards “student support” offices, as against direct instruction, is in order; that may work.

Now if I could just get the voters to do something about that funding...

"Most students see college as a social experience" and link that with the peer group comment. When I was in my undergrad at a SLAC, I also worked in admissions and gave tours. One of my lines was that we weren't a party school. We were here to learn. We'll have fun when we're done. And for at least the first 2 years as a student, this was entirely true. Then a new crop of students came in and expectations changed. Positions were hired for the freshman experience. More activities. More learning communities. More social less learning.
You could also send a memo to your professors about group learning!

Back when I was a CC student (I transferred to a 4-year this year, so it wasn't long ago), it seemed like all the teachers got the OPPOSITE memo. Every science teacher I had stressed the importance of study groups (one prof even forced us to form study groups! I'm pretty sure that a lot of those groups never actually met...). At least 1/3 of the time in my humanities and social science classes were spent discussing things in small groups.

And you know why it's no better than socializing? Because it mostly IS socializing!

In lab groups, group projects, and group discussions, one of two things happens:

1) students pick their own groups and they self-select. This is great for Team Competent but you wind up with A LOT of groups where the blind are leading the blind. It seems like the majority of study groups end up this way because most students only join study groups because they're doing poorly on their own.

2) students are put into groups by the teacher and/or just group with those in closest proximity.
I'm happy to help my peers and I'll tutor anyone or edit any any paper if the student approaches me independently. But I'm not going to be the annoying nag in a discussion group who says "Sorry! You guys are disinterested and want to socialize but I'm going to force you to discuss the readings that you didn't read!"

When professors try to socially engineer groups so that they contain students of mixed competence or dedication, it does not make the poor students improve. It's just the opposite. The good students notice that nobody else is invested in doing a good job and they wind up doing all the work! In groups that are fixed for the course of a semester (e.g. lab groups), the poor students quickly realize that they can completely slack off and the work still gets done. Then they slack off even more. If the professor isn't paying attention, it can SEEM like the poor students do better in a group, but that's just because the good students do 90% of their work for them. When students are evaluated as a group, slackers get grades they don't earn and dedicated students often have little choice but to let their slacker peers take advantage of them. Poor students who AREN'T slackers don't benefit because usually they're too worried about 'dragging the group down' to make much contribution.

Plus, teachers often make these groups way too big to be effective. Two-person groups can work; 6-person groups can't (this was one of the nastier effects of budget cuts on my California CC last year -- bio and chem profs suddenly couldn't afford sufficient supplies and we were forced to work in HUGE lab groups. My physiology lab had 8-10 people in a group! In o-chem, where we were supposed to do most experiments alone, we sometimes had to work as 6-12 people as a bench because the supplies were expensive. In a group of 10, 7 or 8 people's roles are going to be trivial or nonexistent)

I'm sorry for posting something of a rant, but I know a lot of professors and adjuncts read your blog and it's nice to finally have confirmation on something that I've observed and experienced so many times!
Ps -
Dean Dad, does your CC have a tutoring center over which you hold any authority? I worked at my CC's tutoring center & I have a lot of ideas about how to make them more effective. I'm just a student, not a professor or anything, but if you're interested, I'd be happy to email you. -Ashley
@Ashley and in response to the group work issue: There are strong pedagogical reasons for using group work, but not as a stand-alone. I can tell you that I use group activities in my classes for a few reasons:

1) It improves participation in discussion and attentiveness when we're NOT doing small group activities.

2) It allows students to work together to learn how to do difficult things, giving them a practice run for when they will be expected to do those things on their own.

When I assign small group stuff, I give students very specific directions so that all of them have to participate, and I typically connect that work to independent writing and then full-class discussion. Sure, if you say "get into a group of six and discuss the reading assignment" that is likely not to produce anything of value, but there is group work and then there is group work. And learning to work productively with other people is an important - and necessary - skill.

As with all pedagogical techniques, I'd say that there isn't a one-size-fits-all model, and I'd say that any technique in excess is not going to have a positive outcome. But to say "group work is bad," just as to say "all lecturing is bad," is painting with too broad a brush, I think.

@DD - Thanks for this review! I've been interested in the book but I have no time for reading right now that isn't for research or teaching.
Hey, we agree!

1. Is that book really a one-day snow-day read?

2. Funding interests me. How does your per-student funding for lower division classes like English comp, history, and "college" algebra compare to the per-student funding for lower division classes at a middling State uni in Your State?

3. How do you create incentives for new (not yet tenured) and adjunct faculty to hold students to high standards when they live in fear of student evaluations? (And I write from a unit where our Dean clearly places an emphasis on learning over popularity.)
Forty-five percent of students made no significant improvement in their critical thinking, reasoning or writing skills during the first two years of college, according to the study. After four years, 36 percent showed no significant gains in these so-called “higher order” thinking skills.

Maybe they were measuring the wrong thing with these students. Might the students have needed to develop other more basic skills first, before they were ready to develop those higher order skills? If this measure was looking at the move from level 3 to level 4 and these kids spent their first year of college going from zero to 3 that would show up as no improvement but it would still be the foundational work that needed to be done before the student could develop higher order skills.

Students who took courses that required lots of reading (more than 40 pages a week) and writing (more than 20 pages in a semester) showed higher rates of learning.

This could be because of self selection – the potential for improvement might be better with these students and they may (correctly) sense that they are ready for challenges that other students avoid in an effort to preserve their GPA.

Interesting - I'll have to find this and read it.
There is at least one more thing you could do, DD: clamp down on any notoriously undemanding courses you have.

It shouldn't be hard to find them. The students have already sniffed them out. Ask around, or take a peek at the course selections of the students that are just barely getting by, and not because of any external problems such as poor preparation. You'll find the really easy courses there.
Since you wisely wouldn't touch communications, I will. It's my field. I've taught at 5 different institutions, one of them being a CC. At 2 of them (the CC is included here), I'm actually encouraged to push the students as hard as I want to--that goes with the whole high expectations thing. My students KNOW I demand a lot from them. They deliver.

However, overwhelmingly, I'm discouraged by my communication colleagues to push the students. "Giving them exams makes the class too hard. Why are you pushing theory on them? Theory isn't important." (yeah, ok.) I realize 5 schools isn't a huge sample size, but I know I'm not alone in this experience. Because communication(s) is supposed to "be easy," those of us who actually try to demand more, incorporate critical thinking, theory and application all together, get told we're doing it wrong. The big problem here is that it's other professors who are discouraging me. My students? Have yet to complain. In fact several have gone so far as to say "you're the only one I had to work for."
DD's and others' comments are interesting and thoughtful (except for the whining about group work. OMG, man up!). However, the study seems to indicate that the majority of students do learn more about studying, critical thinking and higher-order learning by the time they graduate. Maybe that number should be higher, but it is still a majority.

Also, while higher-order learning is important, the acquisition of a knowledge base is important too. As DD pointed out (but was loathe to pursue), disciplines such as computer science and business did not show as much of an increase in higher-order learning, but I bet those students learned a lot about their field of interest. Maybe that should not be considered enough, but I also wouldn't simply discount it as, 'not learning'.
As a footnote to the disciplines that "don't learn as much" as stated by the study, I'd be interested to see how they define "computer science" as a discipline. There's probably a huge difference between prestigious STEM universities where a CS degree is basically an applied math/physics/engineering degree with lots of programming and smaller programs were a CS diploma means that a graduate can successfully install and use Excel. Distinguishing between math and computer science is an interesting decision.
'Forty-five percent of students made no significant improvement in their critical thinking, reasoning or writing skills during the first two years of college, according to the study. After four years, 36 percent showed no significant gains in these so-called “higher order” thinking skills.'

The tests are measuring the wrong thing. Focusing on academics, students do not sign up to learn critical thinking skills in college, students sign up to learn whatever is in their major. I didn't improve my critical thinking skills in my years at Flagship Small State U. Instead, I learned a lot about economics and mathematics, which is what I went there to do.
cody, if you didn't improve your critical thinking skills while learning about economics, your profs are doin' it wrong...
I don't view the comments about group activities as a "rant". I am well aware of the issues raised in that comment. In lab, I use lab exams to isolate the Lab Partner effect. In lecture, although I encourage study groups -- and students generally recommend them as effective -- I know that there are good groups and bad ones. Some students improve a lot when they leave a group where someone merely explained how to do a problem so they never learned how to do it themselves, or the group was more social than studious.

I've already blogged about this book based on reports and other analysis (see link from my name), where one issue is the specific measurement used. I'd guess that I would not have improved on it because my HS comp class was so strong on critical thinking that I should have hit the top score as a freshman. The instrument used is also not at all like the kind of vague problem that engineers, algorithm designers, or business professionals have to deal with. Whether it separates a Frito-eating code monkey from a skilled software designer is an interesting question!
"Now if I could just get the voters to do something about that funding..."

It's probably hard for voters to see that your budget is tight. What do the local senior citizens see when they go to the CC for a basketball game, a play, a speaker that is in town, or to vote? Everything looks fine: the lights are on, the building is heated and nothing seems amiss. The painful cuts are probably happening in the classrooms.

So, I wonder, why not turn the thermostat down to 60 degrees? It's an obvious and immediate sign of the unfortunate austerity measures that have been forced upon the college.
I agree with cody - for a lot of students, they're not going to college to learn "critical thinking." Especially for more vocational, career-oriented majors like education and business, they're going to (a) learn skills that are specific to their intended career and (b) gain a credential to signal their competence and help move them ahead in their career.

Critical thinking is a useful measure for liberal arts students, and not for others. Not surprising, then, that those are the students for whom you saw improved results in critical thinking.
"Critical thinking is a useful measure for liberal arts students, and not for others."

Which explains much about why union voters or hunters vote a straight-party ticket?
I think Cody makes a really good point about using "reasoning" and "critical thinking" not being a good measure except for liberal arts and social science grads.

If I'm at the hospital, I don't care about the nurse anesthetist's ability to think critically; I care about her ability to administer the anesthesia without killing me. CS grads aren't hired for critical thinking abilities; they are hired for the ability to write code or do whatever other CS related skill their job requires.

Health, science, CS, ... and to some extent foreign languages ... are skills-based areas, where you spend much of your undergrad career trying to master and apply a certain skillset. Learning calculus is hard for a lot of students...and having done so will probably not increase your reasoning ability much. But you will know calculus.

My undergrad degree was liberal arts, and while I'm sure I learned some critical thinking/reasoning skills in undergrad, it wasn't really until grad school that I felt that I *really* had good critical thinking skills.

So I'm not sure that measuring the reasoning skills of 20-year-old sophomores really tells us very much.
Seriously, Peter, you didn't just suggest that careers in health and science don't have much to do with critical thinking -- did you? Aside from very low level jobs in those fields (e.g., "Take this tube of blood to the lab, but you're not the one responsible for doing anything with it"), I wouldn't want anyone working in those fields -- or going on for a four-year degree in those fields -- who did NOT have strong critical thinking skills. Like you, I kind of like the idea of staying alive when folks in those professions are making decisions that affect me. ;)
I just had a good laugh with my husband, software engineer and former computer science minor turned philosopher, over the idea that computer science doesn't require reasoning skills. It is logical reasoning, man.
@ Steve -

No, that's not the point I was trying to make...and I hope that my health workers have reasoning ability, too.

But the reality of health education, especially at the beginning, is that there is a *lot* of memorization and a lot of very practical hands-on exercises in giving shots, taking blood, and dealing with other bodily functions. Essentially, nursing students are crammed with facts and skills, but there's not a lot of *analysis* involved. The students have to understand a lot of basic facts before they know enough to do any kind of real analysis in their field. (And even for doctors, real analysis, I'm told, doesn't start until they've graduated from med school and started doing their residency and working with real patients.)

Compare this to an English class. The students already (one hopes) know how to read, and have already (again one hopes) been exposed to literature and even some interpretation in high school. The class is *all about* reasoning and analysis because (much more so than nursing students, at any rate) the students already know the basic facts necessary to do an analysis.
Peter W captures nicely the difference between training and education, the point I made in distinguishing between the software designer and what a friend refers to as the Frito-eaters who can only implement the design. One would expect that a four year CS degree incorporated the critical thinking challenges to prepare you to be the former rather than just producing certified code monkeys.

But much of this misses a broader point: selection bias.

The authors used a one-dimensional test and the reports only talk about IMPROVEMENT rather than some "absolute" (subjective) level of critical thinking broken down by major, so we have no way of knowing if students in one area are already skilled at the specific skill they were measuring while others developed that skill in their college classes.

Further, they weren't asking you to apply your knowledge of chemistry and math to decide if the dose prescribed by the doctor might kill a child. One group of majors would not improve at all on that sort of test in 4 years of college, while another would improve a lot.
As DD said, these results really aren't surprising. I, as a student, am glad to see some form of confirmation on the group work thesis. I agree with Ashley and think that many (though not all, of course) tutors use group work as a class "filler". As one of the more precocious students, I mostly learn how to translate journal articles for people who haven't read them or tried but didn't understand them when put into a group. Like Ashley, I'm happy to help students (and to receive it!), but I don't enjoy operating as a teaching assistant for a class I attend to learn from.

Mind you, I have found online discussion groups (in hybrid classes - so tied to assessment) extremely valuable. I'm assuming the transparency of online postings makes us all accountable and thus raises the stakes.
I wasn’t shocked to see that math, science, humanities, and social science majors tend to show the greatest gains in critical thinking, as compared to students in business, health, computer science, or communications
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?