Top
Best
New

Posted by spIrr 10 hours ago

Handbook.md shows that long policy documents do not reliably govern agents(arxiv.org)
274 points | 174 commentspage 4
netdpb 6 hours ago|
LLMs are not people. Employee handbooks, like all other social rules, are not self-executing; they depend on the fact that human beings are innately social animals.
smcg 5 hours ago|
Tell that to the executives who are looking to replace people with AI.
solatic 5 hours ago||
This is why the future is in deterministic static analysis and policy-as-code frameworks.

Policy-in-English? Model implicitly complains that it's TL-DR.

Ask the model to write code that checks your policy, then add that code behind a simple validation hook (e.g. "check your work by running 'just validate'") that the harness knows to always run after changes? It suddenly becomes the most law-abiding citizen ever.

smcg 5 hours ago|
Perhaps, but the majority of executives do not understand code at all. The scenario in the paper is realistic for a non-tech company suddenly adopting AI for daily business tasks.
hotpaper77 9 hours ago||
I got pumped seeing the OKF format from Google (which is just a standardization of wiki pattern) but quickly realized it could not yet combine many subtle concepts together in an efficient way.
effnorwood 4 hours ago||
Confirmed. These are autistic geniuses with ADHD. Handle with that care.
iamacyborg 7 hours ago||
Hard to take this seriously when it has all the hallmarks and annoying tics indicating it has been written by Claude.
honkycat 8 hours ago||
This is what spec driven development tries to solve.

Multiple rounds of generating small contacts documents that grow from the original idea , trying to keep each slice small enough to process for a human to approve/disprove .

Eventually it leads to a long list of tasks grouped by functionality. You start a new context and the orchestrator agent dispatches tasks to sub agents with a limited amount of information provided to each sub agent.

Also should have adversarial review and approval gates with other agents and roles.

dk_sig11 3 hours ago|
I have not seen any real life examples of successful application of SDD. I heard a lot of marketing BS, but nothing real. My own experiments with SDD produced very disappointing results even on very small projects.
tokai 8 hours ago||
Having only dipped my toes in generative ai recently I'm surprised how small even a 1mio window is. A semi serious project can take several session in one sitting. Especially as degradation sets in waay before the window is full.
supermatt 8 hours ago|
Few pointers:

Dont try and handle the entire project in context.

Use a well structured filesystem layout for your code with a few lines in an AGENTS.md describing the layout and core architectural requirements (no more than that, as per the article!).

Then work on small-medium tasks at a time with a fresh context.

At the end of your task, ask the agent if there are any key points about the project layout or architecture it would want to add to memory - audit those manually and amend your AGENTS.md accordingly.

If your code is well structured and you keep your tasks localised, you can get away with seemingly minuscule context windows.

It's also worth noting that high effort models love to slurp up whatever context they can. You almost never need/want high effort for non cross-cutting tasks.

tokai 7 hours ago||
Good points, but it still seems to me that the technology is far from mature.
LetsGetTechnicl 7 hours ago||
Well no shit obviously. Just cause you tell the random text generator to follow some rules doesn't mean it will.
drybka 1 hour ago||
[flagged]
More comments...