Super Simple "Hallucination Traps" to detect interview cheaters

Posted by EliotHerbst 7/3/2025

After testing out Cluely with my team, we suspect that the easiest way to detect interview cheaters is to set simple "hallucination traps" where you ask a question that sounds plausible, but any knowledgeable person would instantly identify as a joke, fake, or just simply say they don't know. Vibe coded a simple app demonstrating the concept - https://beatcluely.com/

Here are some examples of this class of prompts which currently work on Cluely and even cause strong models like o4-mini-high to hallucinate, even when they can search the web:

https://chatgpt.com/share/6865d41a-c720-8005-879b-d28240534751 https://chatgpt.com/share/6865d450-6760-8005-8b7b-7bd776cff96b https://chatgpt.com/share/6865d578-1b2c-8005-b7b0-7a9148a40cef https://chatgpt.com/share/6865d59c-1820-8005-afb3-664e49c8b583 https://chatgpt.com/share/6865d5eb-3f88-8005-86b4-bf266e9d4ed9

Link to the vibe-coded code for the site: https://github.com/Build21-Eliot/BeatCluely

29 points | 39 commentspage 2

trumbitta2 7/10/2025|

Why do you want to harass people, innocent people until proven otherwise, over an interview?

Go on with the interview without any such tricks. Hire them if they pass. Fire them afterwards if they heavily underperform.

leakycap 7/3/2025||

Maybe I don't have the interview volume others do, but aren't you able to tell pretty quickly in your face-to-face or live video call interview that a person is competent or not (such as using a tool to compensate for a lack of experience)

I keep hearing of employers being duped by AI in interviews; I don't see how it is possible unless:

1) The employer is not spending the time to synchronously connect via live video or in person, which is terrible for interviewing

2) The interviewer is not competent to be interviewing

... what other option is there? Are people sending homework/exams as part of interviews still and expecting good talent to put up with that? I'm confused where this is helpful to a team that is engaged with the interview process.

interneterik 7/3/2025|

This is an example that comes to mind where someone can pull of cheating with AI in a realtime interview: https://techcrunch.com/2025/04/21/columbia-student-suspended...

leakycap 7/3/2025||

I'm familiar with this story, this is the person who founded the software being discussed/linked... but what does this do to explain why a competent interviewer was unable to suss out that the person had no idea what they were doing?

Bluffing in interviews is nearly a given. Your interview should be designed to suss out the best fit; the cheaters should not even rank into the final consideration if you did a decent interview and met the person via some sort of live interaction.

EliotHerbst 7/3/2025|||

You’re right, a competent interviewer can likely suss out that a person is cheating - but it can depend on the type of interview and role. This can help erase any doubt, as if you are not familiar with what is being discussed, it is hard to differentiate this type of question. We found that some of our existing interviews for roles like technical support could be “cheated” using Cluely to some degree, when asking questions about solving example support issues which might have troubleshooting steps in an LLMs training set and if the interviewee is someone who is loosely familiar and presenting as being more familiar with the topics.

Before these sort of tools [Cluely], there wasn’t a good way that I'm aware of to cheat on this type of question and respond without any interruption or pause in the conversation.

In real support situations, the tool is not useful as you could pass a major hallucination on to a customer, of course.

leakycap 7/4/2025||

Maybe I'm wrong, it just seems obvious to me when a candidate has the kind of knowledge potential on a subject to want them on my team vs. "they answered that question with some sort of accuracy" which is what I would expect from people skimming and responding in an interview as if they had an earpiece.

I have worked a lot of places in different fields where the HR team leading initial interviews had zero awareness of the role or what the role would really be doing, so I could see Cluely passing those interviews. But surely the team would smell the deception?

scarface_74 7/3/2025||

Or you could just ask them to describe an implementation they are most proud of, the challenges they faced, the architectural decisions and tradeoffs they made and keep digging deeper into their thought process.

For a remote interview, I would do something as simple as share a Lucid app document where they can do a rough diagram of their architecture.

Even before LLMs, it was easy to pass techno trivia interviews by just looking up “the top X interview question for technology Y”

EliotHerbst 7/3/2025|

Absolutely. Though for certain types of roles, being able to recall information that is readily available online off the top of your head used to be a strong signal of deep familiarity with a certain topic - for example - instantly recalling troubleshooting flows for example issues for a random sample of different A/V product lines really would only be possible if you had either studied deeply beforehand or had substantial experience.

I was surprised by just how easy it is to intentionally trigger hallucinations in recent LLMs and how hard it was as a [temporary] "user" of Cluely to detect these hallucinations while using the tool in some non-rigorous settings, especially given how these tools market themselves as being "undetectable".

Kemschumam 7/3/2025||

My team has been kicking around the idea of using images to trip up candidates using some kind of AI in their ear.

Things like diagrams and questions written on paper the held up to the webcam.

muzani 7/3/2025|

AI is really good at this though. Not CSI levels, but better than some humans. And tool use is at the level that they can do two things at the same time, which is why playing pokemon is a benchmark now.

careful_ai 7/3/2025||

There’s a tipping point when AI tools meant to boost productivity start fracturing our workflows instead: more prompts, more context switching, more review overhead. The real efficiency comes when these tools integrate into flow, not hijack it. We should be aiming for augmentation, not distraction.

jumilbiju 7/3/2025|

[dead]