Thursday, January 23, 2014


Thoughts on the Experimental Site Authority Concept Paper

Sometimes, it’s worth reading the whole thing.  As they say on the Supreme Court: concur in part, dissent in part.

A consortium of seventeen colleges and universities has submitted a concept paper to the Department of Education, petitioning for “experimental site authority” for their campuses to keep financial aid eligibility while moving to competency-based education.  (Hat-tip to Amy Laitinen, from the New America Foundation, for calling attention to it on Twitter.)  

Yes, the buzzwords were strong in that one.  Essentially, the colleges are asking for special permission to waive certain rules and regulations that normally govern financial aid eligibility.  As written, the rules are largely about time: hours per week in class, semesters or quarters with quotas of credits earned in each (“satisfactory academic progress”), and the like.  The colleges would rather move to “direct assessment” of student learning.  The idea is to measure what they’ve learned, rather than how long it took them to learn it.  Doing that requires new definitions of, or approaches to, “satisfactory academic progress,” continuous attendance, and so forth.

The idea behind moving to competency-based measurements makes sense.  On an intuitive level, measuring learning is more to the point than measuring time spent trying.  If you’re a quick study, it’s hard to justify making you sit and wait for others to catch up.  And if we’re ever going to get a serious handle on budgets, we’re going to have to get Baumol’s Cost Disease under control.  By definition, that cannot happen as long as we measure our product in units of time.  Competency-based measures allow the possibility of finally achieving actual productivity gains, using the term ‘productivity’ in the Econ 101 sense.

All of that said, though, the concept paper is well worth a close read for the odd little asides and compromises it includes.  It’s claiming both more and less than it should.  Some highlights:

- The report correctly notes that while colleges now are theoretically allowed to use “direct assessment,” many of the regulations in Title IV are written in ways that prevent it for all practical purposes.  Moreover, colleges are required to be either entirely time-based or entirely competency-based; they can’t mix and match.  In practical terms, that means that a college that wanted to try a competency-based approach would have to redo every single back-office function before starting.  (This is probably why SNHU had to establish an entirely separate division, College for America, before making the leap.)  That’s a hell of a commitment for an institution with students currently enrolled.  The report asks for the rules to be rewritten to allow colleges to try a competency-based approach in individual programs, rather than across the board.  This strikes me as an excellent idea.  Get some proof of concept before trying to renegotiate faculty union contracts, for example.

- Since it’s difficult to measure “satisfactory academic progress” in a given semester if semesters don’t exist, the report offers alternatives.  The first, which seems reasonable to me, is to look at time in the context of overall degree completion, and to pro-rate the number of competencies (with some wiggle room) so that a student needs to be on pace to finish within 150 percent of normative degree time.  The second is radical, and I can’t decide if it’s brilliant or preposterous.  It’s to pay out financial aid only after competencies have been demonstrated.  The idea there is that if aid is disbursed only when students are actually learning something, then the amount of time it takes them to learn it becomes irrelevant.

Yes and no.  Yes, it solves the SAP problem with admirable elegance.  But no, it doesn’t solve the cost problem.  If aid isn’t disbursed until after the fact, how are the upfront costs covered?  Colleges incur costs from day one, and asking them to “eat the cost” of that first semester-equivalent isn’t terribly realistic.  Many colleges are running close to the bone as it is.

The “learn now, pay later” model is also analytically distinct from a competency-based approach.  In theory, we could apply “learn now, pay later” to traditional semesters.  But we don’t, and there’s an obvious reason for that.  That obvious reason would still hold under the alternative approach.  

The paper further suggests allowing Title IV student aid only for “direct costs,” such as tuition and books, in the name of reducing student borrowing.  (Current rules allow for some estimation of living expenses, as well.)  The merits of that suggestion are arguable -- I’d argue that moving living expenses off-books doesn’t make them go away, and would instead just drive students to private lenders -- but more importantly, it’s separable from the competency-based vision.  The only reason I can imagine that it’s included is to run up the score.

Of course, “direct costs” are harder to figure when each student moves at a different pace.  If a college charges by the competency -- similar to charging by the credit, now -- then there’s a question to be raised about students who show up already testing out of some.  Do you charge them for testing out?  (To be fair, CLEP exam fees aren’t covered by financial aid now.)  I prefer the subscription model that CfA and some others use, in which you pay x dollars for y amount of time, and use an “all you can eat” approach during that time.  That gets around the “where did you learn that?” problem cleanly, although it reintroduces time-based measures through the back door.  

The weirdest moment in the report came on page 20.  I can’t really do it justice in paraphrase, so here it is in all its glory:

Federal aid policies discourage students in competency-based degree programs from taking courses offered in the traditional credit-hour format.  Some students in competency-based programs might do better in traditional credit-based courses in certain subject areas.  For example, students who struggle with mathematics might thrive in credit-based courses in which there is significantly more direct contact with instructors.  Allowing students to choose the instructional or learning modality that best suits their learning styles could reduce the amount of repeated course-taking and also shorten time to degree and save money. (emphasis added)

Wow.  That one contains multitudes.

From a community college perspective, my first thought is that students who struggle with math aren’t the exception.  They’re the majority.  If a competency-based approach disadvantages those students, then it’s inappropriate for us.

Why would it disadvantage them?  Apparently, a competency-based approach features “significantly” less direct student contact with instructors.

I’m not sure why that has to be true.  But if it does, it’s pretty damning.

The assertion flies in the face of what I’ve seen on the ground.  This semester we’re rolling out our new self-paced developmental math course, which features both faculty and on-site tutors to help students as they confront new (or persistent) hurdles.  We’re hoping to recoup the extra cost through improved student completion over time.  It may or may not work, but it’s worth trying.  (We’re working with developmental because the “transfer” issue is off the table at that level.)  It’s not a pure competency-based approach, since the course is still scheduled into semesters, but a student could conceivably get through two or three semesters’ worth of material in one.  

The report mentions in passing some dramatic changes to the faculty role.  On page 11, it notes matter-of-factly that “[s]ome institutions separate subject matter-expert faculty who design programs and assessments from student-mentor faculty, who serve as the primary contacts with students.  In addition, some programs have additional student supports and faculty who solely handle grading and assessments.”  

The faculty role gets unbundled, with a production model that winds up looking very much like my graduate program did back in the 90’s. And that model didn’t require a competency-based approach.  In this model, t.a.’s have a different name, but are otherwise recognizable.  In the new model, most faculty would regress (or be regressed) to the t.a. duties they performed in grad school.  

I understand the usefulness of presenting a concept in its purest form; you can always take less than you’re granted, but you can’t take more.  Might as well ask for it all upfront, and make the necessary compromises later.  I get that.  And if that’s what this is -- an opening salvo -- then many of my concerns are moot.

But it also raises hackles that don’t necessarily need to be raised.  A competency-based approach could conceivably work in a number of different ways.  

The paper is also quiet on some of the concerns that led to a new focus on seat time over the last ten years or so.  In the 2000’s, a number of for-profits took some pretty implausible liberties with seat time and credits in order to maximize student loan income while minimizing labor costs.  Requiring a minimum number of hours of seat time may be asinine in certain ways, but it’s also, at least in part, a response to some very real abuses.  It’s not clear from this paper what would prevent a recrudescence of those abuses.  

I offer these thoughts as a sympathetic critic, as one who wants the idea to work.  It will work best if it’s less theoretically stringent and more cognizant of facts on the ground.  I hope the experimental site authority gets approved, precisely so the idea could be refined through contact with those facts.

Am I understanding correctly (w/o reading the paper) that some faculty would have their responsibilities change so they spend all their time grading and assessing, rather than having those activities interrupted occasionally for activities such as planning a class, teaching a class, meeting with students, or conducting research?

Grading is bad enough when it is far less than 100% of job responsibility. I know this is far from the main point of the paper, but still ...
The idea is to measure what they’ve learned, rather than how long it took them to learn it.
You never measure what they have learned. You measure what they demonstrate given the assessment tools you have to develop a model. It is an important distinction. Low level on Bloom's taxonomy is relatively easy to assess and model. It is the higher order that is more challenging.

What wil the false positive and false negative rates be? I don't see that discussed enough.

A semester's worth of assessments, my models on students demonstrated learning is fairly robust.

On a student level, I do see few high stakes assessments don't model an individual student's learning all that well. On average for the course if n is large enough they do a reasonable job overall.

Also how are labs done under this model?
Ah yes, Baumol's cost disease.....
Does Dean Dad teach in Lake Wobegon (I think not - Holyoke is something completely different).
When people talk about breaking the credit hour, they always throw out the example of the student who could do 2-3 semesters of content in 1 semester of time. To which I say (as Amy Poehler might say) "really...really". Having taught many hundreds of students over the past decade, there certainly were some who could have moved through the material faster. But there were many more who would have benefitted from moving more slowly. Is there data to support the optimistic assertion that there are more students for whom competency-based systems would allow faster completion than there are student for whom it would lead to slower completion. Because (and of course this is just my personal experience), if I set up my courses in a competency-based fashion the average time to "competency" would be > 1 semester, not less. Now to be idealistic, maybe that would be good to let students learn at their own pace but if the motivation is efficiency and cost savings (as it is always touted to be) this would lead the opposite of the desired effect. Or, and my suspicion is this is more likely to be the case, "competency" will be defined downwards until the desired time and cost savings are achieved.

Does Dean Dad have any comment about this? Is there data or personal experience to suggest that there are more students are capable of accelerating than there are in need of deceleration?
My question for the auditors is how they plan to identify those who just want the aid, not the grades or the competency.

I see that a few others have picked up on my past remarks about such arguments, so I'll keep this short (for now) by just saying "Insert previous comments here".

Have you ever taught or taken or assessed such a course?

I have taken one (about 4 decades ago, showing this is a "new" idea) where you had to get a perfect score on an assessment to pass that competency. I liked it. I did the entire course in about 20% of the CALENDAR time for the usual course and retained it really well. It was not cheap or efficient.

I also teach one. The comment above about grading is on point. Getting test 4 from one student on the same day you get test 1 from another does not make for an efficient grading system, even if you use m.c. scanned exams. (It is faster to do them by hand when they come in one at a time.)

The fun part there was filling out the accreditation paperwork, all the while thinking about what the answers would be for a face-to-face class. I'm sure the students who pass the self-paced version of the class spend MUCH more time studying than those in the classroom.

It would be fun to log the hours spent by a typical student on different types of classes. The US DofEd should fund something like that. It could be amusing.
One wrinkle of this that Dean Dad touches on -- but that probably won't register with a lot of folks in academia -- is the proposal to disallow financial aid for living expenses. I was a financial aid advisor at a community college for a while. There's a substantial portion of our population that just. Can't. Attend. School. unless they get assistance with living costs. A lot of these students also have life circumstances that make it harder to reach academic success as we commonly define it. So, if this proposal goes through, I expect we'll see the "private school" effect on some scale -- grades and completion will improve, because economic circumstances will drive out students on the margins.
What Uncle Matt says. I see this at my R1 with students on "need-based" scholarships (usually target at underrepresented minorities, first-generation college students, and low-SES students). Many of these scholarships cover tuition, some include books, and one (just one that I know of) includes a stipend that was probably adequate in 1963. This li'l college town is damn expensive, so my students end up working 1-3 part time jobs to pay for food and housing IN ADDITION to their scholarship money. The time they need for studying goes into earning a few dollars for food and heat, which does not exactly help their grades; when their grades drop low enough, they lose their scholarship. In some ways, these partial scholarships are just an exercise in cruelty.
I'd like to see them try this with labs. Do students get one crack at a lab, or can they do it over and over until then demonstrate 'competency'? Who is going to prep all the chemicals, or specimens, or keep the equipment clean, functioning, and in order? Or are they planning on using 'virtual labs' on a computer, which is to a lab what a video game is to a sport.
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?