Revisiting The Umpire Theory Of Technology

I came up with this metaphor in late January of 2007, in a post called Learning Technology and "The Vision Thing". This was soon after the annual ELI conference and in my prior post I had been quite critical of the conference and the supposed vision it was attempting to convey to attendees.  So, I provided an alternative in this post.  The metaphor itself is explained early on as follows:

Let me begin with a little personal philosophy; I subscribe to the Umpire Theory of Technology. According to that, in a baseball game the umpire is absolutely critical to make the calls in an unbiased way. But if you watch a game the only time the umpire gets noticed is when he makes a bad call. If the umpire does his job very well, he becomes invisible. Learning technology well employed should be invisible too.

It then follows that the visioning itself can't be about the technology. The technology only plays, at best, a supporting role.  The visioning then must either be about aspirations for teaching and learning, in this post I articulated my aspirations about Humanism Across The Curriculum, a next generation view of Writing Across The Curriculum, or it must reflect current issues/problems with teaching and learning and then offer up problem solving solutions.  (On the issues/problems I will offer one possible look in the discussion that follows.)  Learning technologists might benefit themselves by an embrace of the Umpire Theory, at least temporarily, to give them a perspective where they can be self-critical about their beliefs and their practices with the technology.  My belief is that such an exercise would be quite useful to learning technologists prior to a complete embrace of AI.  

But in this piece I will have a historical look only, where I know something about what happened, and not discuss the present at all, since I've been out of it for some time.  Yet I think the history tells us something about the present as there are persistent patterns with new technology adoption.  Considering those patterns is where I hope there is benefit in this post.  First, there is a tendency for learning technologists to become highly enthusiastic about the next big technology innovation.  That enthusiasm served as one of the drivers for learning technologists to enter the field.  And a certain fraction of instructors will likewise be so enthused, call them innovators or early adopters.  This feeling that technology itself will drive major change is fueled by what the innovators and early adopters do with their teaching as a consequence of the technology. 

I felt this way in the late 1990s, when the online components of instruction seemed to give courses a new vitality and in my campus role I got to talk with these early instructors who were doing wonderful things with their teaching, even if by current standards the technology itself was quite primitive. The drive and and creativity of these instructors infected me.  For a while I became a true believer.  Then, my little Center for Educational Technology, which had just come into being, had as its implicit mission to get as many courses as possible onto one of the learning management systems we then supported.  I embraced that mission, at least for a while. 

A variety of political economy issues got in the way of fulfilling that mission in a satisfying manner.  We were underfunded from the get go and remained that way.  There were other campus systems in support of instruction that had developed prior to the forming of my center, and those would have to be abandoned eventually.  The transition to the learning management systems was painful.  While we did offer small grants for adoption, followed by attendance at a multi-day workshop that would give instructors ideas about how to implement, we couldn't afford to make such workshops a pre-condition for adoption and many other instructors simply adopted the LMS, perhaps after getting some training on its functionality from one of my staff, though perhaps not.  After a couple of years of this, we stopped giving out the small grants and providing the multi-day workshops, with the funding redirected into my Center's budget.  Fast forward now several years to where scaling considerations forced us to embrace an "Enterprise Learning Management System" and convert courses from the previous systems or have the courses start anew, it was determined that upwards of 90% of the class sites used the LMS in such a lightweight way that conversion was unnecessary.  Those facts were in the background when I wrote the visioning post. 

Let me fast forward again another 5+ years and consider a different technology entirely, so as to begin to ask whether the prior experience generalizes.  The following is from an email I sent to the Educause CIO listserv, which at the time was discussing the pros and cons of lecture capture technology.

It is interesting to read all these testimonials about lecture capture and how popular it is.  However, given the recent NY Times piece about the value (or lack) of laptop initiatives, http://goo.gl/mzVge, it behooves us to remain skeptical about the value of lecture capture on learning, in spite of the admitted popularity.  The argument against must be something like this – lecture capture encourages student effort to focus on the ability to reproduce the lecture.  But we know from the How People Learn volume of a dozen years ago, http://goo.gl/ZZ8vQ, that learning happens primarily via “transfer.” At issue then is the impact of lecture capture on student efforts aimed at transfer.  If students find transfer difficult but mastering the lecture within grasp, their preference for the technology is understandable. And instructors who want students to like their classes have reason to feed that preference.  So it seems possible that the technology can be popular but the impact on learning might be nil.  It would be good to gather data on this parse to try to relate lecture capture to transfer activities.

I want to note that my hypothesis at the end of this paragraph is not novel at all.  It follows from what George Kuh called the Disengagement Compact, in this piece, What We're Learning About Student Engagement From NSSE.  The operative paragraph is here:

And this brings us to the un-
seemly bargain, what I call the
"disengagement compact": "I'll
leave you alone if you leave me
alone." That is, I won't make you
work too hard (read a lot, write a
lot) so that I won't have to grade as
many papers or explain why you
are not performing well. The existence
of this bargain is suggested by the fact
that at a relatively low level of effort,
many students get decent grades - B's
and sometimes better. There seems to
be a breakdown of shared responsibility
for learning - on the part of faculty
members who allow students to get by
with far less than maximal effort, and
on the part of students who are not tak-
ing full advantage of the resources in-
stitutions provide.

Real learning is labor intensive.  Technology can't change that, though perhaps it can save some time the student spends in activities that support the learning.  I would argue that real teaching is labor intensive too.  When I first started as an assistant professor in economics, back in 1980, almost all the faculty were tenured or on the tenure track.  Then the reason to shortchange the time devoted to instruction, particularly at the undergraduate level, was that the added effort beyond some minimal level didn't count for promotion, tenure, or salary review.  Nowadays, most of the undergraduate instruction on campus is done by adjunct (specialized) faculty.  They have a different reason for participating in the Disengagement Compact.  Lacking tenure, their job security hinges on students being satisfied, as indicated by the course evaluations administered near the end of the semester.  If students are entirely instrumental in their approach, motivated by grades alone, not at all by learning, then by giving them good grades for comparatively little effort the instructor gets decent evaluations and to teach the course yet again.  

Kuh's essay is from 20 years ago.  Apart from the move to adjunct instructors, what has changed in that time and are things better or worse than they were then?  My guess is that if you surveyed instructors who have been teaching undergraduates for the past 20 years or longer, many if not most would say that things are worse now.  I would point to three factors to explain this.  One is the decline in reading, particularly as a recreational vehicle and to stay informed about the world (in other words, non-course reading).  Another is the mediation of communication via the smartphone, so face-to-face interpersonal skills don't have a chance to develop well.  The third, something we've been talking about only comparatively recently (but this talk began well before Covid) is that so many students suffer from emotional health problems.  In turn, this can be explained by the artificial nature of the game that is school which students play coupled with the high tuition they or their families must bear.  

I want to make a few more points before giving some very broad stoke thoughts on what a fix might look like.   One is about having an open conversation regarding these issues.  Sometime after I became aware of Kuh's essay, I learned that the U of I participated in the National Survey of Student Engagement, but it didn't release the results publicly.  We were strong in some areas, not so strong in others.  Having the areas of weakness be considered in public media might have damaged the university's reputation.  Why risk that?  

The university did make some internal changes to address the findings.  One of those was to develop a formal program of undergraduates engaged in research.  But only a fraction students were involved in that, with the more elite students much more likely to be involved.  This is another issue with a focus on student engagement, one I've found over the years in my own teaching.  The better students liked my course a lot, though they were only 10-15% of the class.  For much of the rest, it was a struggle.  It is reaching the rest that is the real challenge.

There is a lot of inertia in how most instruction occurs.  If a particular instructor tries to innovate in teaching, his students might find it a refreshing alternative to what they are getting in their other classes.  But it can go the other way, with the students so accustomed by how their other courses are conducted that they are put off by the instructor's attempt at innovation.  I've taught one course a year in retirement, ending in fall 2019.  My experience is that it was the first way from 2012-2014, but became the second way during 2015-2017 and was even worse in 2019.  (I didn't teach in 2018.)  My current view is that for innovation to be successful it must happen across the board, not just in one course at a time.

On the other hand, the Center for Teaching, which most campuses have, tends to make their offerings opt-in for instructors.  Unless there are mandates for innovation from individual colleges or departments, the opt-in approach is what's available to them because they don't have the leverage to do otherwise.  But then those who do opt in are the usual suspects and only a small fraction of instructors while those who don't end up teaching courses that become very static. 

Now to the suggestions.  When I was still teaching, the discussion about student mental health focused on the lack of mental health professionals on campus, so the difficulty that students had in getting appointments with such professionals.  There was essentially no discussion about whether the overall academic environment was creating many of these mental health problems.  Now, with Covid in the rear-view mirror (we hope), it seems time to ask that question in a systematic way.  This will require a serious evaluation effort of students, their friends and families, their instructors, and possibly the people they worked with during internships.  In other words, lead with an evaluation, one where student mental health is the driver, but where the Disengagement Compact might linger in the background as an explanatory factor. 

Campuses are unlikely to do this on their own, for reasons I've mentioned above.  So outside foundations need to step up here to encourage this, with implied funding for remedies after the evaluation has produced its findings.  Implicit in how I've written this post is that the issues I've identified will be found as significant factors in explaining the findings.  The next few suggestions are based on that assumption. 

There needs to be a major effort to move instruction from credentialing to learning. I've written about this a lot, for example in this post called Excise The Textbook.  One should anticipate substantial resistance in making such a move, from both instructors and students.  So there needs to be a plan to overcome that resistance.  As I'm an economist, I'd look to incentives for doing this.  Grant funding might very well be deployed to provide such incentives.  Assuming that the initial resistance is overcome, at least for some subset of students and instructors, one question is whether this move to learning has legs that will endure beyond the grant funding.  Another question arises if the initial group provides promising results.  Will other groups then form that follow a similar path, even if there is no grant funding to support that?

Then there needs to be drill down on what a move from credentialing to learning looks like.  My view is that high-stakes assessment, via exams and term papers, needs to be diminished if not totally eliminated.  Regular low-stakes assignments must be done with much more attention to the student getting good feedback on those and that transfer of some sort is required to complete the assignments.  The hope is that after the initial resistance students come to appreciate this alternative approach, want even more feedback, and begin to understand why the alternative has been put in place.  Beyond that, one might hope that students learn to create their own low-stakes assessment as they begin along the path to self-teaching, one of the meta goals from moving to the alternative approach

There is the matter of who will write these new low-stake assessments and who will provide the feedback.  On the latter, during the first year of writing blog posts I had a series on Inward Looking Service Learning, seven posts in all, based on my experience with peer-mentors who had previously taken my course as the ones providing the feedback, mainly via online office hours held in the evening.  I thought it the biggest innovation in my teaching at the time, with the technology as an enabler, an early example of that.  The INSL posts tried to generalize from that experience.  But I want to note that in the course I was teaching, there hadn't been graduate student TAs.  The undergraduate peer-mentors weren't substitutes for those.  They could be afforded because my lecture went from about 60 students to about 180 students.  So the approach de-emphasized the lecture and made the online office hours held by peer-mentors a feature.  

Now there is a big deal with graduate student TAs unionizing because of feelings that they are being exploited.  In courses where there are graduate student TAs, it would have to be worked out in advance that the use of undergraduate peer-mentors would be an addition to the overall labor in instruction.  I'm afraid that under the current circumstances, where everyone seems to be feeling a budget pinch, that's unlikely to happen.  But again, grant funders might come to the rescue, at least at first, so that pilot projects can demonstrate the feasibility (or not) of this suggestion. 

As to the writing of the assessments, I've done that in my own teaching, so tend to minimize the effort entailed as well as the sense of competence in the subject matter that such authorship requires.  But there is also the matter that it might seem to be a lot of work, so individual instructors will balk at it for that reason.  This might be addressed by creating a team of instructors who teach the same course but at different institutions, to divide up the assessments to be written, with an agreement that the products will be part of an OER and that they will use the assessments written by other members of the team in their own teaching.  This means they will also serve as peer-reviewers of these assessments as those are still in draft form. The authorship and the peer-review function will go hand-in-hand. 

Sometime later, the students who do the work to complete these assessments can themselves be considered as reviewers, so their feedback on the assessments can be used in revising them.  Further, students can be given extra-credit projects to write additional assessments and/or to revise the extra-credit project that was completed in a previous offering of the course, under the proviso that these extra-credit projects would also find their way into an OER.  This would provide a mechanism for the assessments to stay current and students might then come to see themselves as authors, another step in the direction of self-teaching.

Let me wrap up.  There may be a lot of wishful thinking in this visioning exercise.  I have no doubt about it.  I hope that readers aren't put off by it.  The point is that it's been done without making technology the driver of the change.  That makes as much sense to me now as it did when I originally came up with the Umpire Theory of Technology.  And for learning technologists who typically assume technology should be the driver, it might be useful to think through for themselves an alternative view.