Sunday, April 22, 2012

Measuring Attendance and Knowledge Transfer

My theory is that if you look confident you can pull off anything - even if you have no clue what you're doing.
Jessica Alba

By a quirk that I don't understand, I have only eight students in the course I'm teaching this semester, where in advance of the offering the expectation was that there would be about sixty students. So at the start of the semester I converted the class from a lecture to a seminar. I don't require attendance when I teach a lecture class and I frequently provide sufficient online materials that a diligent and confident student could learn the stuff without coming to class. Nonetheless, my experience from when I last taught a lecture class in this manner is that the better students do come to class and among the few students who ask questions, most of those are the good students.

In the course this semester I did require attendance as well as weekly writing via posting in a blog. The requirements notwithstanding, only one student has a perfect attendance record. A few others have missed only a couple of classes.  Some have missed quite a few, and likewise for the written work.  We've had only one session the entire semester where everyone has shown up.  In some cases the students bring me a note from the McKinley Health Center on campus.  In other cases I get an email explanation about a job interview or some personal matter that requires attention.  But quite often I get nothing.  I chalk it up to immaturity in those students who miss without keeping me in the loop, though it might be that alienation is the better explanation --- the class is soooooo borrrrriiing.

There is another meaning of attendance, however.  I associate it with my dad's pigeon French.

Attendez-vous à vos affaires.

Google Translate says it means, Expect your business, which doesn't make any sense.  My dad would say this, and get a smile on his face while doing it.  He meant it as an alternative to saying "watch out," or "pay attention."  I believe he thought it translated as, "attend to your business."  For students, schoolwork is their business.  They have assignments to do, presentations to prepare, studying for exams, etc.  An obvious question arises.  How long does it take for students to attend to their business?  The equally obvious answer, though some might find it unsettling, is that it takes as long as it takes.  One doesn't know in advance.  One can only know afterward.

Prudence, or "rational" decision making, or a love of the activity itself will encourage getting started early with the schoolwork.  In that way there is additional time, if needed, to do it in a way where the student herself feels she has learned something from the activity.   Getting started early is a way to let the schoolwork take as long as it takes.  But we know getting started early is the exception.   When there is a deadline, as there almost always is, most students initiate quite near the time when the work is due.

Last year I taught a class on Behavioral Economics.   One topic we covered was procrastination, a departure from economic rationality that all of us have experienced.  I had students read the James Surowiecki essay, Later.   Students blogged in that class too.  They surprised me with what they wrote.   Quite a few of them argued that leaving things to the last moment was efficient.  They could concentrate then.  Since they weren't distracted, they got things done.  Some gave responses in the vein I had expected - talking about writer's block and the like.  But many were entirely unrepentant about their approach.  It "worked" for them.

* * * * *

Sixteen years ago I wrote an essay about how online learning (we called it ALN back then) could help students do better in their courses.  Ever the economist, I cast the issue in the frame of the principal-agent model.  In my mental conception of the ideal, the professor would be a little Tinkerbell for the student as the student went about doing the outside-of-class schoolwork, directing the student to go about that work in the "right way."  Absent the Tinkerbell, the student might go astray or stop doing the work altogether.  Online learning wasn't quite as good as the professor qua Tinkerbell, but it provided a reasonable proxy.   

In the essay I cast the situation as if there were three distinct types of students, Eager Beavers, Drones, and Sluggos.  I think that classification is useful and will employ it here, though my focus is different.  I'll get to my focus in a bit.  First let me describe the types.  Eager Beavers are the ones who enjoy school for itself and are most inclined to take up purely optional suggestions the professor offers, simply to pursue the interest, even when doing so has no implication for the grade.  Eager Beavers might very well initiate work well before deadlines and then get a kick from figuring things out for themselves.  Drones put in substantial time on task, but frequently that time is not very productive in generating insight in what is supposed to be learned.  They may very well have taken a lot of notes on the subject, but they frequently will not get the implications of what they are studying, more often getting only a surface understanding of the subject.  They do anticipate getting good grades, however, as reward for their effort.  Sluggos get by on their wits alone.  They don't put in much time at all.  They too expect a decent grade, maybe not an A but certainly a B, though they can live with getting a C if the professor is a tough guy.

In that earlier essay I treated the type of a student as fixed, a reasonable assumption for any given course, and asked whether through our Tinkerbell proxy we might get the students to behave as if they were one type better, Drones like Eager Beavers, Sluggos like Drones.  Here I want to ask a couple of different questions.  Does school overall, meaning the entire experience including K-12, impact what type the student is?  And if it does so, is school encouraging or discouraging the student in this respect? 

These questions have been rolling around in my head for quite a while, but the reason I'm writing this post has to do with some specific events that are quite recent.  On a listserv I follow regularly though post in only once in a great while, there was a discussion about whether a professor could opt out of faculty development programs with learning technology because "pedagogy is so personal."  The person posing the question was implicitly arguing that some faculty are shirking in their responsibility as teachers.  His solution was to impose a "standard" for high caliber instruction.  I reacted negatively to use of the word standard in this context.  I hope to make clear why later in this post.  The thread got participation from a variety of folks.  One mentioned the National Academy of Sciences volume, How People Learn, which is what got me to think about knowledge transfer and put it into the title of this post.   Another wrote about being fine with personal pedagogy, as long as there are appropriate and predetermined levels of student learning that can be demonstrated.  This too set me off.  Then on Friday David Brooks had a column about learning in college and value added assessment.  It's great to see Higher Ed being the focus of popular attention, but to me it is quite disturbing that people already think they have the solution even when they can't or don't analyze the underlying issues.  Yesterday, there was a piece in the Times about standardized testing using a nonsense story to measure reading comprehension, and getting the students upset about it.  The implication was that the testers didn't understand what it was they were measuring.

* * * * *

There are some topics students are exposed to that many have a tough time understanding and ultimately don't really learn.  A former colleague from Accountancy, Dave Ziebart, had a wonderfully descriptive expression to describe the phenomenon.  "The knowledge vanishes through the students' fingertips as they write the final exam."  Somehow, most of them pass the exam.  Yet the knowledge doesn't stick. 

Dave was talking specifically about present discounted value, a concept that Business students see in many courses, but one that can remain uncomfortable all the way through.  There are perhaps several reasons why students are unable to internalize this concept.  I'll focus on two of those reasons.  One is that doing so requires being comfortable with how to sum a geometric series.  Students should have learned this in high school math.  Clearly they were taught it, but many didn't learn it.  When the foundation is shaky, building upon the foundation doesn't work very well.  This is a frequent issue in teaching at the college level - what can the instructor rightly assume that the students already know?  Must the instructor teach the prerequisites too?   The other issue has to do with taking multiple perspectives.  For present discounted value, the student has to understand how the future looks like from the perspective of the present.  While getting the student to think this way the instructor also has the student think about how the present looks from the perspective of the future. That you can look at the same thing from multiple perspectives and as a result get different measurements of that same thing is an alien concept for many students, including some of those who can do the math.  How does one get students to internalize alien ways of thinking?  I'm not sure but certainly spending a significant amount to time getting familiar with the approach is necessary.  Students who do their schoolwork only as the deadlines approach often don't get sufficiently familiar with their subject.  They learn they can get by, test-wise, while remaining in an intellectual muddle.

The ability to transfer knowledge to a novel setting is the way to verify that the knowledge is real.  That's what the National Academy volume tells us.  One might suppose, therefore, that all of us learn new things by repeatedly trying to transfer knowledge, with self-assessment built into our learning process.  I believe that Eager Beavers do this, but Drones do not.  Here's a metaphor to illustrate.

Children learn via play.  The learning is an indirect consequence.  The play is the focus.  Play itself is absorbing so the children concentrate while doing it.  Is there an adult equivalent?  Consider having a new software application to which you need to acquire a functional use.  You can read the manual.  Or you can simply try this and that and see if you achieve the functions that you hope the software will let you do.  You don't throw the manual away.  If you get stuck it is there as a reference.  Otherwise you simply keep proceeding by trying new things to accomplish.  Over time, you become familiar with the software and proficient in its use.  When I do this sort of thing I like to say I'm futzing around with the software.  I try a lot of stuff that no training session with the software would have you do, just to amuse myself and make sure I can achieve with it what I want.  A couple of more formal names for this is Discovery Learning or the Inquiry Approach to learning.  If you follow an inquiry approach, you'll surely be able to transfer the knowledge.  But it will take time to get there.  Also, you as the learner need enough prior knowledge to know what things to try to allow you to make progress in your inquiry.  If you have that, the learning can be totally absorbing, like a child's game.  If you don't, it's dull as dishwater.  

Now take this metaphor and apply it to college courses.  It's pretty straightforward to treat the textbook like a software manual.  The question is whether the student has the wherewithal to futz with the content.  One possible approach is to rely on the end of chapter problems that many textbooks have.  The student can work those, even if they are not assigned, to see if they feel comfortable thinking them through.  Another possible approach is for the student to generate scenarios on her own to which she can apply the lessons from the chapter.  If she's not able to do this, she has a reason to ask about the problem at the appropriate juncture during the class session, or if not there then during office hours, or in the online forum for the class set up to handle such queries. 

Put this way, the question about Drones is whether they don't proceed in this manner because they don't know how or is it because they lack confidence in taking an inquiry approach to their learning.  I don't know the answer to this.  What does seem clear is that given how much time most students put into their studies, these students don't expect to learn in an inquiry manner most of the time.   One would like to know whether an inquiry approach is necessary for the ability to transfer the knowledge.  I believe it is.  But it may be vast overkill for students if their goal is only to pass the tests in a course.  They often don't need the inquiry approach to do that.

* * * * *

There is some advantage in thinking about teaching and learning by looking at psychological research on cognitive function, research which is entirely outside the course setting.  One can then learn about cognitive limitations of people, on the one hand, as well as about the large failure rates of individuals who should know better, to apply what they have been taught in certain situations.  They go with instead what seems intuitively plausible, though it is logically untrue.  Daniel Kahneman offers up repeated examples of this sort in his book, Thinking Fast and Slow.   In Chapter 15, Linda: Less Is More, Kahneman provides some very disturbing yet highly revealing results.  First, participants in the experiment are provided with a description of a fictitious character, Linda, whose attributes are chosen to set off certain stereotypical thoughts in the the participants' minds.

Linda is thirty-one years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in antinuclear demonstrations.

Then the participants are asked to respond to the following question

Which alternative is more probable? 
Linda is a bank teller.
Linda is a bank teller and is active in the feminist movement.

Regardless of the character description, the first alternative has to be more probable than the second since in the Venn Diagram sense the second is a proper subset of the first (note the conjunction "and").  That's the logical answer.  If participants transferred knowledge after having some basics in probability, all the participants had those basics, they would opt for the first alternative.  Yet Kahneman reports the following.

This stark version of the problem made Linda famous in some circles, and it earned us years of controversy. About 85% to 90% of undergraduates at several major universities chose the second option, contrary to logic. Remarkably, the sinners seemed to have no shame. When I asked my large undergraduate class in some indignation, “Do you realize that you have violated an elementary logical rule?” someone in the back row shouted, “So what?” and a graduate student who made the same error explained herself by saying, “I thought you just asked for my opinion.”

Kahneman is writing about something called the Availability Heuristic.   The description of Linda conjures up an intuition that she is a feminist.  The second alternative makes explicit mention of her feminism, so participants deem it more likely.  In some sense Kahneman has tricked them with the description to focus on their intuition and not consult their logic.  In that way the description acts like the misdirection a magician uses in pulling off a trick.

When using in class exams to test students ability to transfer knowledge I don't believe the instructor is trying to trick the students.  But the instructor does need to create a scenario that is sufficiently novel that applying the knowledge supposedly learned has to be a mindful activity, not something immediate.  When I've done that in the past teaching intermediate microeconomics, I'd get low scores on the exams (not as low as what Kahneman gets but far lower than what our Lake Woebegone culture demands).  Then I'd get low course evaluations in return and comments to the effect:

Test us on what we know, not on what we don't know.  

In this circumstance, there is enormous pressure on the instructor to cave in on how to test the students.  Get the average test scores up to a tolerable level.  This is done by making the exam questions only minor tweaks from what the students have seen before, either in homework problems or in practice exams.  Transfer isn't measured that way.  But the Drones get what they're after, a decent grade in the class.  If I'm at all typical, the instructor doesn't like to teach this way.  But the course evaluations are in the tolerable range so the instructor grins and bears it.

* * * * *

I can only conjecture about the impact on students from taking class after class in a mode where there is surface learning only.  Were I one of them, I'd become a Sluggo, no doubt.  I'd also be entirely disillusioned about my own education, a paper chase farce.  I could maintain my sanity as an Eager Beaver, because then I'd be going about things for myself in my own way.   I would go crazy as a Drone.

In this closing section of the post,  I want to briefly argue that we in Higher Ed contribute to this vicious cycle.  It has been an issue for quite a while, at least since I started teaching 30+ years ago.  NCLB exacerbated the problem by narrowing the K-12 curriculum.  Less commented on, but I think just as damning, is the movement away of tenured and tenure track faculty from teaching the core subjects and instead relying on adjuncts and advanced graduate students for this purpose.

Let me note two separate but related issues that matter.  First, inexperienced instructors are apt to emphasize technique over intuition in their teaching, as a way to establish their bona fides with the students.  This is especially true for newly minted PhDs or ABDs.    Then, when the results from doing so appear less than promising, the instructor is apt to adjust by moving to a cookbook approach, spoon feeding the course material to the students.  Second, the adjuncts need decent course evaluation scores to secure their employment.  So they can't grade too harshly or push the students too hard for fear that the students will rebel from that and then they'd lose their jobs.

In other words, in the core courses the setup encourages the students to be Drones.  Even if they were Eager Beavers in high school, the adjustment to college encourages Drone like behavior thereafter.  The students may feel they are adjusting because they had been a big fish in a small pond when in high school or because the pace of the new material is more rapid than what they've been used to.  But it's also because GPA matters in its own right and if students get slammed by the very first midterms they take they look to find a way past the shell shock in a hurry.  Getting good grades may then trump their personal learning needs.  In developing habits that produce a tolerable GPA they may then lock themselves into an approach to learning that doesn't serve themselves well in the long term.

This is not a pretty picture.  But it does demonstrate systematically what's at issue.  If we want different outcomes, let's agree it is the system that must change.    Let's not make the mistake that we can apply a quick and dirty fix and then wash our hands of the matter.  Doing that we'll only get more of the same.

No comments: