Top
Best
New

Posted by sethbannon 1/2/2026

Fighting Fire with Fire: Scalable Oral Exams(www.behind-the-enemy-lines.com)
221 points | 279 commentspage 4
latexr 1/3/2026|
I’m doubtful of most of the “fixes”. Putting more instructions in the prompt can maybe make the LLM more likely to follow them, but it’s by no means guaranteed.
phren0logy 1/2/2026||
I had plenty of oral exams throughout my education and training. It's interesting to see their resurgence, and easy to understand the appeal. If they can be done rigorously and fairly (no easy thing), then they go much further than multiple can in demonstrating understanding of concepts. But, they are inherently more stressful. I agree with the article that the increased pressure is a feature, not a bug. It's much more real-world for many kinds of knowledge.
dvh 1/2/2026||
Students cheat when grades are more valuable than knowledge.
viccis 1/2/2026||
And then they complain when they gain no knowledge, can't pass the simplest of coding interviews despite their near 4.0 GPA, and blame it all on AI or whatever.

In reality, they cheat when a culture of cheating makes it no longer humiliating to admit you do it, and when the punishments are so lax that it becomes a risk assessment rather than an ethical judgment. Same reason companies decide to break the law when the expected cost of any law enforcement is low enough to be worth it. When I was in college, overt cheating would be expulsion with 2 (and sometimes even 1 if it was bad enough) offenses. Absolutely not worth even giving the impression of any misconduct. Now there are colleges that let student tribunals decide how to punish their classmates who cheat (with the absolutely predictable outcome)

semilin 1/2/2026|||
I think this points to the only real sustainable solution: make it so that students would prefer to do real work. We have seen for ages the distinction between seeming and being in regards to verbal understanding blurred. LLMs are only an acceleration of the blurring. Therefore it will at some point become essentially impossible to determine whether one really understands something.

The two solutions to this are (1) as some commenters here are suggesting, give up entirely and focus only on quality of output, or (2) teach students to care about being more than appearance. Make students want to write essays. It is for their personal edification and intellectual flourishing. The benefits of this far surpass output.

Obviously this is an enormously difficult task, but let us not suppose it an unworthy one.

j_w 1/2/2026||
Or you just make in person exams the majority of the work and make the exams brutal. If you can't pass the exams you don't pass the class, so you need to learn enough to pass the exams.
Aurornis 1/2/2026|||
I knew some hardcore, dedicated cheaters in college. All of them hit a wall where their cheating tricks stopped working. Most of them couldn't get back on track.

I suppose there are other fields where the degree might be used mostly as a filtering mechanism, where cheating through graduation might get you a job doing work different than your classes anyway. However, even in those cases it's hard to break the habit of cheating your way around every difficult problem that comes your way.

Arodex 1/2/2026|||
So, what is your solution to turn teenagers and 20-somethings into wise men and women?
margalabargala 1/2/2026|||
Identifying a problem is the first step towards solving it. Coming up with a solution is a later step.
senko 1/2/2026||
Very insightful!

Here, I'll identify another: There is much pain and suffering in this world.

Coming up with a solution is left as an excercise for the reader.

margalabargala 1/2/2026||
Thank you for your input!

Perhaps we as humans should stop making choices which cause pain.

Why do you make choices that cause pain in yourself and others?

jimbokun 1/2/2026||||
Written exams at a set time and location hand graded by a human grader.
baq 1/3/2026|||
Making knowledge valuable for getting passing grades would be a start
beezlebroxxxxxx 1/2/2026||
This is not hitting the problem. Most students in universities are completely fine with awful grades or expect comical levels of grade inflation. Ask a professor or TA and you'll hear about an insane level of entitlement from students after they hand in extremely shoddy work. Failing students is actually quite hard or extremely discouraged by admins.

The real problem is students and universities have collectively bought into a "customer mindset". When they do poorly, it's always the school's fault. They're "paying customers" after-all, they're (in their mind) entitled to the degree as if it is a seamless transaction. Getting in was the hardest part for most students, so now they believe they have already proven themselves and should as a matter of routine after 3-4 years be handed their degree because they exchanged some funds. Most students would gladly accept no grades if it was possible.

Unfortunately, rather than having spines, most schools have also adopted a "the customer is always right" approach, and endlessly chase graduation numbers as a goal in and of itself and are terrified of "bad reviews."

There has been lots of handwringing around AI and cheating and what solutions are possible. Mine is actually relatively simple. University and college should get really hard again (I'm aware it was a finishing school a century ago, but the grade inflation compared to just 50 years ago is insane). Across all disciplines. Students aren't "paying for a degree", they're paying to prove that they can learn, and the only way to really prove that is to make it hard as hell and to make them care about learning in order to get to the degree - to earn it. Otherwise, as we've seen, the value of the degree becomes suspect leading to the university to become suspect as a whole.

Schools are terrified of this, but they have to start failing students and committing to it.

themantalope 1/2/2026|||
There is a lot in this comment I agree with, however I think may universities have backed themselves into a corner with the degree of tuition inflation that has taken place over the last 20+ years.

I graduated from a SUNY school in 2012. At the time, you could still actually go to school and work part time and get through it. Not saying it was easy by any stretch but it was possible. Tuition + living expenses were about $17/year on campus , less expensive housing was available off campus.

Now, even state schools have tuition which is only affordable through family wealth or loans. Going to university is no longer a low stakes choice - if you flunk you’re stuck with that debt forever. Not to say students aren’t responsible for understanding that when signing up, but the stakes are just a lot higher than what it used to be.

jimbokun 1/2/2026|||
Universities are in for a rude awakening when employers realize their degrees mean nothing, stop hiring their graduates, and then students stop enrolling.
mirrir 1/3/2026||
My university had a great policy for this. For every major assignment you went through interview grading. if you failed it you lost 60% of that grade.
latexr 1/3/2026|
> interview grading

Would you mind expanding on what exactly that entails?

mirrir 1/6/2026||
Yes absolutely. For large code writing assignments and projects grad students would be tasked with writing the code. After submission they had to schedule a 15-20 minute chat with a grad student in the course for Interview Grading. Through that process the grad would ask why the student made specific choices in their code, where improvements could have been made, and the process they took to solve the task. It wound up being a pretty effective and kind way to get people to not really care as much about referencing things like StackOverflow (no GPT at the time), and helped make a lot of students need to care much more aabout the why of their code.
gyulai 1/3/2026||
It is quite telling, regarding the state of higher education, if actual teachers actually talking to students 1:1 (which is all that an oral exam really needs to be) is brushed away as a non-starter. I can highly empathise with students who feel like the whole enterprise is a farce, and trying to game and cheat that system at every possible turn is the only appropriate response.
fcatalan 1/3/2026||
I'm always somewhat uncomfortable with any solutions that can be summed up as "AI for me but not for thee".
TehShrike 1/2/2026||
My ability to recall and express things that I have learned is different when writing versus speaking. I suspect this is true for others as well.

I would prefer to write responses to textual questions rather than respond verbally to spoken questions in most cases.

Wowfunhappy 1/2/2026||
...if I was a student, I just fundamentally don't think I'd want to be tested by an AI. I understand the author's reasoning, but it just doesn't feel respectful for something that is so high-stakes for the student.

Wouldn't a written exam--or even a digital one, taken in class on school-provided machines--be almost as good?

As long as it's not a hundred person class or something, you can also have an oral component taken in small groups.

jimbokun 1/2/2026||
I would be annoyed that I can’t use AI to do my work but the instructor can have AI do his job.
semilin 1/2/2026||
Too bad. The premise should be that the instructor, by nature of having the position, already has understanding of the subject. As a student, you do not, and your goal is to gain it. Prompting an LLM to write a response for you does not build understanding. Therefore you should write unhindered by sophistry machines.
jimbokun 1/2/2026||
But the instructor is not applying their understanding in any way. By delegating the evaluation to AI, there is zero value add vs just asking ChatGPT to evaluate your knowledge and not paying $1000s or $10000s in tuition.

And universities wonder why enrollment is dropping.

semilin 1/2/2026||
I'm not intending to say it's acceptable for professors to use AI entirely in their grading. They obviously ought to contribute. I realize I actually misread your original comment, thinking of "instructor can have AI do his job" as "instructor can have AI to help do his job." Sorry about that. Point being, I think the expectation for real human thought ought to hold for both teacher and student.
ted_dunning 1/2/2026|||
A written exam is problematic if you want the students to demonstrate mastery of the the content of their own project. It's also problematic if the course is essentially about using tools well. Bringing those tools into the exam without letting in LLMs is very hard.
Wowfunhappy 1/2/2026||
I don't entirely disagree but all exams are problematic. We don't have the technology to look into a person's mind and see what they know. An exam is an imperfect data point.

Ask the student to come to the exam and write something new, which is similar to what they've been working on at home but not the same. You can even let them bring what they've done at home for reference, which will help if they actually understand what they've produced to date.

throwaway7783 1/2/2026|||
Why is it disrespectful? It is just a task. And it is almost an arms race b/w students and profs. Has always been (smuggling written notes into the exam etc)
Wowfunhappy 1/2/2026|||
The student has a lot riding on the outcome of their exam. The teacher is making a black box of nondeterministic matrix multiplication at least partially responsible for that outcome. Sure, the AI isn't the one grading, but it is deciding which questions and follow up questions to ask.

Let me ask, how do you generally feel when you contact customer service about something and you get an AI chatbot? Now imagine the chatbot is responsible for whether you pass the course.

jimbokun 1/2/2026||||
Talking to a disembodied inhuman voice can be disconcerting and produce anxiety in a way that wouldn’t be true communicating to a live human instructor.

Adding this as an additional optional tool, though, is an excellent idea.

viccis 1/2/2026|||
Unless class sizes are astronomical, it's absurd to pay US tuition all to have a lazy professor who automates even the most human components of the education you're getting for that price.

If the class cost me $50? Then sure, use Dr. Slop to examine my knowledge. But this professor's school charges them $90,000 a year and over $200k to get an MBA? Hell no!

jimbokun 1/2/2026||
Yes.

At that point what’s the value add over using YouTube videos and ChatGPT on your own?

baq 1/3/2026||
The certificate is the value as long as everyone trust it actually certifies what it says is certified. If a diploma can be had for promoting ChatGPT or Gemini a couple dozen times a year, trust in what it certifies should be rapidly eroding and universities should be scared because what you suggest is actually rational.
jimbokun 1/3/2026||
I suspect it’s already started with the declining enrollment numbers in recent years.
kelseyfrog 1/2/2026||
If I was a professor, I don't think I'd want students submitting AI generated work. Yet, here we are.

Students had and still have the option to collectively choose not to use AI to cheat. We can go back to written work at any time. And yet they continue to use it. Curious.

Wowfunhappy 1/2/2026|||
> Students had and still have the option to collectively choose not to use AI to cheat.

Individuals can't "collectively" choose anything.

This test is given to the entire class, including people who never touched AI.

kelseyfrog 1/2/2026||
What are you talking about?

Students could absolutely organize a consensus decision to not use AI. People do this all the time. How do you think human organizations continue to exist?

ted_dunning 1/2/2026||||
So what if the students used and AI not to cheat, but to produce good content that the student understood well.

Wouldn't that be a fine outcome?

anonymous908213 1/2/2026||||
Ah yes, collective punishment. Exactly what we should be endeavouring for our professors to do: see the student as an enemy to be disciplined, not a mind to be nurtured.

I know we've had historical record of people saying this for 2000 years and counting, but I suspect the future is well and truly bleak. Not because of the next generation of students, but because of the current generation of educators unable to successfully adapt to new challenges in a way that is actually beneficial to the student that it is supposed to be their duty to teach.

throwaway7783 1/2/2026||
Since when did exams become punishment? Aren't they a reflection of what you have learnt as imperfect as they are?
anonymous908213 1/2/2026||
The subject is "AI exams", not "exams". GGP expressed that they believe that AI exams would be an extremely unpleasant experience to have your future determined by, something I find myself in agreement with. GP implied that students deserve this even though it's unpleasant because of their actions, in other words they agree that this is unpleasant but are okay with it because this is punishment for AI cheating. (And which is being applied to all students regardless of whether they cheated, hence the "collective" aspect of the punishment.)
jimbokun 1/2/2026|||
And instructors also have the option to not have AI do their work.
amelius 1/3/2026|
What makes me so sad about LLMs is that I used to get questions about math, physics all the time from cousins, nephews, etc. but that seems to be a thing from the past :(
More comments...