A recap/The ALN Research Web and Evaluation

Following up on yesterday’s discussion about the Japanese retirement system, I started to imagine retiring at 60 and starting another job. What job would that be? I thought I might become a “sloganeer” for and ad agency. I began coming up with a few.

Next to the welcome sign at an aviary
One good tern deserves another.

Outside a fish restaurant
In cod we trust.

A new slogan for an exercise/diet company
A waist is a terrible thing to mind.

Sometimes Google can be terribly ego deflating. I checked each of these in Google and sure enough there were multiple entries for each of them. With head drooping, I was determined to come up with one more that wasn’t in Google.

After a fashion, and thinking of both my wife’s mom and my mom who are in various stages of dementia, I came up with
A word on the tongue is worth two in the mush (of the brain).

Sure enough, it isn’t in Google. But it’s not very catchy either. The lesson, I think, is what the Japanese already concluded. When you switch jobs your productivity drops. And at age 60 you may not learn fast enough about the new work for your productivity to return to its level, say 10 years earlier.

-----

Yesterday on the Sloan-C listserv Roxanne Hiltz made a post about the ALN Research Web. There are a lot of good resources there and I’d encourage folks to check it out. I found this interesting tutorial on evaluation. I’m including the full url because you have to establish a login first before you can access the content.

http://www.alnresearch.org/JSP/Tutorials/AssessmentTutorial/index.jsp

What I liked about the tutorial is first that on the introductory page it emphasized the scientific method of proof so that we don’t go around half cocked about claims of the benefits of technology interventions and we do try to control for things like Hawthorne effects and excellent-teacher effects.

But at the same time, to the extent that practice is still emerging, requiring scientific method may be too stern a way to evaluate the activity and some more crafty approach might be preferable. Certainly, in my own teaching I get a sense of whether something has worked or not at a level to ask whether I’ll continue to do that same something the next time I teach the course. And since I don’t teach that frequently, I might very well encourage a colleague to try something similar, without having the scientific proof.

A second thing I liked about the tutorial was the way it classified different learning issues and that the very first item in the classification was “argumentative reasoning.” That pleased me enormously as it is my belief that is the most important single area to cultivate in our students. The tutorial authors came up with an interesting heuristic for thinking about argumentative reasoning.

• logical argument structure
• justification of claims based on evidence
• justification of claims based on course content vs. outside resources vs. personal experience
• consideration of alternative claims (i.e., the extent to which students' weigh the arguments on all sides)
• cognitive procedures (e.g., elaborations, clarifications, questions, syntheses)
  rhetorical moves with a given group (e.g, agreements, disagreements, questions, rebuttals)
• group participants' opinion or belief change as a result of interaction
  argument types (e.g., functional, non-functional, non-justificatory)

Perhaps others are well acquainted with this breakdown of argumentative reasoning. But it was new to me as an explicit listing of the desiderata. So, as I’m prone to do when I confront something new, I start to tweak it for my own purposes. The issue in the tutorial is assessment. But first and foremost I’m interested in teaching strategies that might promote these outcomes.

I zeroed in on consideration of alternative claims. This is definitely an area where students have trouble, even the better students, because of their prior disposition that there must be a “right” way to view something and, of course, that the professor was there to give them that view. (Incidentally, I recall a book we read in graduate school called The Foundation of Statistics by Leonard Savage and Milton Friedman that in the introduction talks about controversy in fields of study and usually the most controversial are the fundamental assumptions. This important point is typically not made at all to undergraduates or if made is not done in a way that resonates with them.)

My own conclusion on the alternative claims issue and what instructors might do is that students don’t immediately see what is in their own realm of experience that can tie into consideration of the alternative claim. The teacher can “give the students perspective” by finding some common element of experience that would provide such tie in. I don’t think this is particularly hard for the instructor, if one is alerted to the need for doing it. But it probably does require some agility because the instructor must assess what more narrow belief the students are clinging too as well as why some alternative view is being discarded.

The great thing about the Web in this regard is that by having the students do brief writes online, either in a content survey or in a discussion forum, the instructor can get a sense of the class quite quickly so this type of quick assessment is readily facilitated.

It is my belief, really my hope (I haven’t come close to scientifically testing the hypothesis) that if the instructor does this tie in activity several times, the students will start reframing the issues on their own and make their own tie-ins.

As for the other bullet points on the list, it seems to me an interesting agenda to come up with teaching strategies that have a chance to promote each of these. My friend and colleague, Peggy Lant, has repeatedly made the point that one of the most important things we do as teachers is to model for our students what we want them to achieve. It may not be flattering to express it as “monkey see monkey do” but that surely is what we believe should happen in the classroom.

Yet it is worth asking if we can go further than that by designing activities for the students that will elicit the desired behaviors en passant as they do the work. And what we should try to avoid is having them treat these things as a bunch of arbitrary rules they must follow to get a good grade in the course but which otherwise doesn’t resonate with them at all. As I’ve mentioned in a post a while back, experiential learning is very powerful. The trick is planning for it to happen.

Here is one last thing about the tutorial that is worth mentioning. There are examples provided of successful assessment in each area within the classification scheme. Monkey see monkey do doesn’t just apply to our students. We should go for it too and do more evaluation work.