Posted by sethbannon 1/2/2026
This is to be expected. The big commercial LLMs generally respond with text that agrees with the user.
> But here's what's interesting: the disagreement wasn't random. Problem Framing and Metrics had 100% agreement within 1 point. Experimentation? Only 57%.
> Why? When students give clear, specific answers, graders agree. When students give vague hand-wavy answers, graders (human or AI) disagree on how much partial credit to give. The low agreement on experimentation reflects genuine ambiguity in student responses, not grader noise.
The disagreement between the LLMs is interesting. I would hesitate to conclude that "low agreement on experimentation reflects genuine ambiguity in student responses." It could be that it reflects genuine ambiguity on the part of the graders/LLMs as to how a response should be graded.
Part 2 is that when you are ready, an examiner sits with you, looks over your stuff and asks questions about it, like clarifications, errors to see if you can fix them, fake errors to see if you can defend your solution, sometimes even variations or unrelated questions if they are on the fence as to the grade. Typically that takes 3-10 minutes per person.
Works great to catch cheating between students, textbook copying and such.
Given that people finish asynchronously you don't need that many examiners.
As to being more stressful for students I never understood this argument. So is real life.. being free from challenge based stress is for kindergarteners
In a presentation, you are in control. You decide how you will present the information and what is relevant to the theme. Even if you get questions, they will be related to the matter at hand that you need to dominate in order to present.
In oral exams, the pressure is just too great. I doubt it translates to a proper job. When I'm doing my job, I don't need to come up with answers right there on the spot. If I don't remember something, I have time to think it through, or to go and check it out. I think most jobs are like this.
I don't mind the pressure when something goes wrong in the job and needs a quick fix. But being right there, in an oral exam, in front of an antagonistic judge (even if they have good intentions) is not really the way to show knowledge, I think.
(I invented some kind of metric based on a centered gaussian around a country ahaha)
One big issue that I had is that the system asked for a number in dollars, but if I answer $2000,2000,2000 per agent per month, the answer was always the same, I cannot accept a number, give it in words, after many tries I stopped playing, it wasn't clear what it wanted.
I could see myself using the system. With another voice as it was kind of agressive. More guidelines would be needed to know exactly how to pass a question or specify numbers.
I don't know my grade, so I don't know how much we can bullshit the system and pass
'This next thing is the best idea ever and you will agree! Recruiters want to sell bananas '
'OK, good, what is the... '
I hope this is catched by the grading system afterward.
The student is supposed to submit a whole conversation with an LLMs.
The LLM is prompted to answer a question or resolve a problem, and the LLM is there to assist. The LLM is instructed to never reveal the answer.
More interesting is the concept that the whole conversation is available to the instructor for grading. So if the LLMs makes mistake, or give away the solution, or if the student prompt engineer around it. It is all there and the instructor can take the necessary corrective measures.
87% of the students quite liked it, and we are looking forward to doubling the students that will be using it next quarter.
Overall, we are looking for more instructor to use it. So if you are interested in it please get in touch.
More info on: https://llteacher.blogspot.com/
I'm still somewhat concerned about exposing kids to this level of sycophancy, but I guess it will be done with or without using it in education directly.
Students are very simply NOT doing the work that is require to learn.
Before LLMs, homeworks were a great way to force students to approach the material. Students did not have any other way to get an answer, so they were forced to study and come up with an answer to the homeworks. They could always copy from classmates, but that was considered quite negatively.
LLMs change this completely. Any kind of homework you could assign undergraduates classes are now completed in less than 1 second, for free, by LLMs.
We start to see PERFECT homeworks submitted by students who could not get a 50% grade in classes. Overall grades went down.
This is a common pattern with all the educators I have been talking with. Not a single one has a different experience.
And, I do understand students. They are busy, they may not feel engaged by all the classes, and LLMs are a way too fast solution for getting homeworks done and free up some time.
But it is not helping them.
Solutions like this are to force students to put the correct amount of work in their education.
And I would love if all of this would not be necessary. But it is.
I come from an engineering school in Europe - we simply did not have homework. We had frontal classes and one big final exams. Courses in which only 10% of the class would pass were not uncommon.
But today education, especially in the US, is different.
This is not forcing student to use LLMs. We are trying to force student to think and do the right thing for them.
And I know it sounds very paternalistic - but if you have better ideas, I am open.
- The stuff being covered in high school is indeed pretty useless for most people. Not all, but most, and it is not that irrational for many to actually ignore it.
- The reduction in social mobility decreasing the motivation for people to work hard for anything in general, as they get disillusioned.
- The assessment mechanisms being easily gamed through cheating doesn't help.
It's probably time to re-evaluate what's taught in school, and what really matters. I'm not that anti-school but a lot of the homework I've experienced simply did not have to be done in the first place, and LLM is exposing that reality. Switching to in-person oral/written exams and only viewing written works as supplementary, I think, is a fair solution for the time being.
The key implementation detail to me is that the whole class is sitting in on your exam (not super scalable, sure) so you are literally proving to your friends you aren’t full of shit when doing an exam.
I wonder: with a structure like this, it seems feasible to make the LLM exam itself available ahead of time, in its full authentic form.
They say the topic randomization is happening in code, and that this whole thing costs 42¢ per student. Would there be drawbacks to offering more-or-less unlimited practice runs until the student decides they’re ready for the round that counts?
I guess the extra opportunities might allow an enterprising student to find a way to game the exam, but vulnerabilities are something you’d want to fix anyway…
To the extent of wondering what value the human instructors add.
Absolutely the easiest solution would have been to have a written exam on the cases and concepts that we discussed in class. It would take a few hours to create and grade the exam.
But at a university you should experiment and learn. What better class to experiment and learn than the “AI Product Management”. Students were actually intrigued by the idea themselves.
The key goal: we wanted to ensure that the projects that students submitted was actually their own work, not “outsourced” (in a general sense) to teammates or to an LLM.
Gemini 3 and NotebookLM with slide generation were released in the middle of the class, and we realized that it is feasible for a student to have a flaweless presentation in front of the class, without understanding deeply what they are presenting.
We could schedule oral exams during the finals week, which would be a major disruption for the students, or schedule exams during the break, violating university rules and ruining students vacation.
But as I said, we learned that AI-driven interviews are more structured and better than human-driven ones, because humans do get tired, and they do have biases based on who is the person they are interviewing. That’s why we decided to experiment with voice AI for running the oral exam.
https://sibylline.dev/articles/2025-12-31-how-agent-evals-ca...
if we want to educate people 'how people work', companies should be hiring interns and teaching them how people work. university education should be about education (duh) and deep diving into a few specialized topics, not job preparedness. AI makes this disconnect that much more obvious.
This is why we need to continue to educate humans for now and assess their knowledge without use of AI tools.