One way to understand Drezner's hypothesis is to consider how research universities gets funded. We live in an era where much funding is solicited from rich donors. It stands to reason that those donors have some expressed interest in the type of research that their money supports. The work of a thought leader (in the proper disciplinary area and with the appropriate slant on the issues in that field) is more apt to please the donor than the work of a public intellectual, who may produce ideas that raise the donor's ire. Similarly to the extent that there are corporate sponsors of university research, they want good ROI. They hunger for new ideas from which they can make a handsome profit. They are far less interested in new ideas as things in themselves, even if those ideas might eventually provider the fodder for other ideas that they can profit from down the road.
This gives an external driver theory, based on the rise of private funding of the research activity. I think there is also an internal driver theory. This is based on two different ideas. One is that departments which teach large numbers of undergraduates in lecture courses need TAs to run discussion sections. This creates a demand for graduate students as inexpensive teaching labor, quite apart from how those same graduate students will fair after they get their PhDs. The other idea is that research faculty prefer to teach graduate classes as compared to undergraduate classes. The former complements their research. The latter is service work only. In order to satisfy this research faculty preference, there must be enough graduate students around so that the demand for graduate classes in a given discipline matches the faculty desire to offer those classes. I taught graduate classes in Economics through the mid 1990s. At the time the rule was you could get credit for teaching a graduate course as long as enrollments were 5 or greater. A class of 2 or 3 would have to be offered as an independent study and wouldn't give the instructor a teaching credit. We had a two-course load per semester then and the norm was one graduate course and one undergraduate course per term.
I should point out here that even if the TA function is absolutely necessary, having graduate students to staff that function is not. Indeed as a Freshman at MIT in spring 1973, taking the second semester course in physics on electricity and magnetism, I had a professor as my TA/instructor. That faculty run discussion sections instead of graduate students, or faculty teach small class versions of a course that had previously been offered in large lecture mode, is surely possible. Most faculty, however, would prefer to teach upper level undergraduate courses or to teach graduate courses exclusively. We have a tendency to think something is necessary because that's the way we've always done things.
There are similar issues in STEM disciplines, particularly in those where the research is done in a laboratory. The director of the lab needs staff to do the work. These staff members are typically graduate students or postdocs. They are not fellow faculty members. Getting grants provides a measure of faculty productivity. Working under the grant of another faculty member may not, particularly if the other faculty member is not a big shot.
The undergraduate teaching, graduate teaching, and research demands for graduate students are myopic, in the sense of being in the here and now. Those demands exist in a universe that largely is unconcerned about what the graduate students will end up doing after they have earned their PhDs. The graduate students themselves, however, are greatly concerned with that matter, perhaps not when they enter the doctoral program and are young and idealistic, but surely at the dissertation stage when they want to graduate, not just from their academic studies but also from the frugal graduate student existence they have been living.
If the graduate student interests were put front and center and if the number of academic jobs that graduate students could obtain are in decline then either fewer graduate students would be admitted to PhD programs or there must be nonacademic jobs that new PhDs in the field deem worthy of all the effort they put in during graduate school.
In the Humanities there has been chronic excess supply of new PhDs for quite some time. Rationally, a person would enter a doctoral program in the humanities only if the person believed he or she were much better than her classmates, so thought there was a reasonable chance to land one of the few academic jobs still out there, or had already come to terms that there'd be a paying job for which the PhD didn't provide preparation (driving a cab or teaching junior high school, for example) while working on the manuscript at night, during so-called leisure time. Otherwise, it would seem a certain amount of zealotry is required to persist under these sort of conditions. Many have noted the consequences of this zealotry, most recently here, a snipped from which is below.
Rather, a kind of intellectual intolerance, a political one-sidedness, that is the antithesis of what universities should stand for. It manifests itself in many ways: in the intellectual monocultures that have taken over certain disciplines; in the demands to disinvite speakers and outlaw groups whose views we find offensive; in constant calls for the university itself to take political stands. We decry certain news outlets as echo chambers, while we fail to notice the echo chamber we’ve built around ourselves.
What I've not yet seen others do is connect this situation of chronic excess supply of PhDs in these fields to the increase in zealotry. There appears to be a self-enforcing feedback loop between the two, at least as I perceive it from afar. This gives another driver.