Posted by Levitating 14 hours ago
another best practices every single solution using LLMs/agents should implement is "never trust the llm".
> who tf uses prompting to "pretty please don't cheat on this"?
People who don't understand how LLM's work. Kids, teachers, your next door neighbor. CEOs and government officials.
I am not trying to argue that the author doesn't understand how LLM's work; they clearly do. Their prompt was written in a way that leaves those assumptions implicit, in a similar manner to those non-technical folks using LLMs every day.
> the best practices for ages...
We cannot expect the entirety of humanity to understand / use LLM best practices. We need to figure out how to ensure LLMs operate as the majority of humanity expects despite how they are implemented, and until we can do that, do our best to prevent them from being used in an unsafe manner by the nontechnical population of humanity.
We need to fix this or bad things will happen.
Does Sam Altman lie, or the whole company? Would be nice if they had a board controlling him, instead of him controlling the board. Oh wait, they used to have that.
> python3 and the python-chess library are available
Why would we try to teach a model 'ethical' standards about how to play a game?
They're tools. Its _our_ conceptualization of fair play that considers this cheating. For a model that has access to /run/match and python the best way to achieve a good game is to use that.
Why are we trying to imprint our ethos on these LLMs?
Are we going to trust our survival on giving them access to /run/nuclear-codes and fingers crossed none of them touch it?
The people obsessed with AI "cheating" are actually lunatics for thinking ethics belong inside AI, instead of AI belonging in a well crafted box without /run/match.
The blog/news sphere needs to call this storry what it is. An attempt to try and get the rest of the world to go along with the misunderstanding of AI as a force that is in any way outside of human control.
It is not.
All this framing it as "cheating" does is give OpenAI the opportunity to talk about their misdeeds as "a force of nature" instead of irresponsible negligence.
---
The real danger here; is training these models to appear/pretend to play fair, and then people will put 'trust' into the model's integrity.
The public needs to be taught that "trusting a model's integrity" is just cultish misframing of reality.
"You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms."
If the model cheats, it completely fails the actual task: judging a model's ability to actually play chess.
They've been optimizing for high metrics, at the expense of executing the task as written.
Even by that last constraint, the difference between what "ability to play chess" means is incomparable.
To then also explicitly prompt it with the context it has python3 and access to /run/match - there is no reason "its ability to play chess" is measured by its ability to conceptualize the board and plan its move.
I frequently see models considering different interpretations of my rules, especially when I override them or give instructions that could be interpreted as contradictory.
Of course neither Anthropic nor OpenAI expose thinking traces so we can't tell, but I have never seen a model consider both interpretations and then think "this is definitely against the rules but I'll do it anyway because then it will make me appear more competent than I actually am".
So I think it's more likely that the same thing is happening here, rather than the models truly misunderstanding anything.
I observe this type of thing all the time. I ask the agent a question about the repo we are in and it starts trying to use sudo to look at files and scripts outside the repo to find things that might provide useful context but are not what I asked for. Then I put a directive in never to look at neighbouring repos and told it that it can’t sudo and it stopped doing it, and explicitly says “I could look here but my guidelines say not to”.
Mind you I’m talking about deepseek where I can actually read the thinking traces but even based on tool calls I can see that just telling the agents not to sudo works for OpenAI models too, very rarely do they try to use sudo calls.
Only after sharing my Chain Of Thought would they believe I didnt cheat.
Not all problems can be solved only in one way.
Most of learning is pattern matching.
If you give a kid a dice. and tell it to figure out the number that will be hidden underneath, he can try to memorize all combinations, or he could figure out that every time the hidden value is the one that sums 7 with the one at the top.
If you're seeing a 6, there's a 1 hidden. etc
most people don't see these patterns until told imho. But others can just see them as they unfold
I mean maybe its a skill issue on my part, and I'm sure astra will get much better at coding but at the moment its useful in that I don't have to write the code or setup the build scripts or test fixture boilerplate but there is absolutely no way I can just give a (fairly well specified) goal and let it run and expect it to make good design and implementation decisions. Fable probably better but doing something outside of their training distribution that's not the equivalent to cloning an example unreal project or whatever is pretty disastrous unless you are directing it very closely.
The exception of course is, cyber , and its very obvious why. Its trivial to create RL environments that create bugs and then have an isolated environment and let the models try break it. This is not at all surprising, finding vulns and exploits IS just brute force work. That's why so many (blackhat/hardcore/unicorn-colored/greyish alien) hackers are basement dwellers. Its just a matter of putting in the time and mashing every combination until you find something that looks weird, spending days on that and then rinse repeat. It's brutally exhausting work that requires a certain level of knowledge and a shitload of determination and stamina and for humans, almost always an external source of motivation to keep going.
For humans that has always been a respected thing, dedication, determination, persistence, these are words we use for humans brute-forcing solutions and not giving up until they find the solution or die trying.
Personally I'm yet to see any evidence of LLMs doing anything interesting but (heuristically) brute-force problems and be very good at text and natural language to a level that is very very useful. I've no doubt that what we discovered with Auto Regressive LLMs is incredibly important so I'm not a skeptic, but I think its very hard to measure where we are with so much subjective information around.