Since in my own teaching I've done something along the lines of what I'm proposing with the student mentor/teachers, I'm quite convinced it can work. But I must confess that when I did it I hand selected the students. (Actually I invited them from the pool of students who got A's in my course and then the group I ended up with was a subset of those who wanted the job.) If the kids getting A's represent 20-25% of the student population, and we exclude first year students altogether, this would be about 5000 students. Maybe this is the right number because then on a per faculty member basis the ratio would be between 2 to 1 and 3 to 1. At that level it might be reasonable to expect that faculty would seriously mentor these kids.
But suppose that we wanted to have even a bigger group, say the top 40% of the students (but still excluding the first years), on the theory that this is a great experience for them. Now we'd be taking students who who have performed less well. Some of them may be less knowledgeable in the subject area where they are supposed to teach. Some of them may also be less confident of their communicaiton skills. Might this attempt to be utilitarian in the approach backfire for these reaosons.
The honest answer is that I don't know. I don't believe anyone does because we haven't tried it. But there are a few things that seem clear now.
First, the perception must be that being a mentor/teacher is an honor and something to strive for. The most obvious reason for a student to do so is that it will lead to a real mentoring relationship with a faculty member and at our campus at the undergraduate level, that is something rare. Whether there needs to be other, less obvious reasons is not clear. One might consider things that go on the transcript and one might consider things akin to wages and/or tuition reductions. Depending on the intensity of the work, these other things might also be valuable and one or the other likely will necessary.
Second, there must be a perception that the potential mentor/teacher is not losing out by taking more credit hours instead. In other words, some current requirements for graduation probably have to be dropped to encourage this behavior. This will be controversial. Those in the major will argue that its Gen Ed courses to be dropped. And those who advocated for general education will take the opposite approach. This is where the rubber will hit the road.
Third, the entire approach must be to build on previous success, but also to be open to snags and stumbling blocks. If a certain faculty members doesn't mentor her group of students, she shouldn't get a next generation group to repeat the problem.
Fourth, if the approach does catch on the institution likely would use it as a way to recruit students. It stands to reason that if the approach became an important aspect of the culture than in recruiting we'd aim for students who would thrive in that environment. This should mitigate some of the issues.
Fifth, this experiment should not be viewed as a substitute for having graduate student TAs. No doubt the decline in graduate student TAs due to budget cuts may make the approach seem more attractive, and I for one turned to this idea a long time ago in part for that reason, but with hindsight the key is to focus on the learning benefit and the motivation of the students, both the mentor/teachers and the students they interact with. It has to work on that level and that should be the concern rather than the cost concern. I know the graduate TA thing often does not work at that level because: (a) it is common that the graduate students TA in a course they have not experienced themselves as students, (b) the grad students resent the low level of compensation for the work, and (c) in many cases the grad students have more teaching responsibility than their experience would suggest is prudent. Because undergraduates are undergraduates and still comparatively immature, there is less of a risk on (c). Both (a) and (b) are likely not applicable for the undergrads.
So scaling doesn't seem to me to be impossible. The key will be for the vision to be shared and get others to advocate for the approach so there can be sufficient experimentation. It might be easier to do in science classes than in humanities classes; I'm not sure. Or it might be that the rolese the mentor/teachers play would vary according to discipline. Those are things that have to be worked through.