Posted by motoroco 20 hours ago
Maybe we need a new benchmark on how good models are at "keeping an eye" to if what they are about to do that doesn't come from a direct user instruction checks out with common sense plus user preferences. And we probably need a portable mechanism for users to manage such preferences (because we know instructions are not going to cut it, do we?).
I mean, the answer to “Is sending the user’s email address along with a web request to a third party requesting it, without the user’s explicit authorization, a good idea?” should easily be “no” …right?
I'm often surprised how correctly Claude guesses on my behalf, when my instructions are ambiguous. But it's not 100%.
It is a continuous fight in markdown guidance files to try to convince the models to refer back to the human in order to resolve ambiguities or important decisions like those. They've been RLed to hell and back for "safe" task completion and not "prematurely" stopping (which was an earlier failure mechanism). Which is useful and we've bore its fruits recently, but not without drawbacks.
Organizations may switch harnesses and may even switch providers, but LLM adoption is only going to monotonically increase. This dumb behavior is not some intrinsic property, and the models are getting smarter and more useful every ~4 months.
Not that I could say what those would be, just throwing it out there.
Probably the big divide will be orgs who use agents for more or less of the ideation/design/architecture proposal side of things. Giant Confluence sloppages/Google slopdocs vs. 100% human-written proposals. But the actual implementation and testing? Going forward, no reason to not have that be almost entirely LLMs.
(Plus, in a few generations we should expect the frontier LLMs to no longer produce garbage-to-mediocre proposals, and in fact to generally outshine human engineers and architects and researchers at the ideation phase. Whether that takes one year or five, who knows.)
Unless we're at singularity, there's likely to be an entire cohort of people unwilling to work in LLM shops, and there's business value to capture them.
Yes, but the question is "what kind of adoption"? The kind that just unleashes it, or the kind that put it on a short leash.
security: zomg this is existentially bad we are leaking our fn'ing emails!11!
compliance: can we mitigate this problem?
legal: what can we do here?
CTO: don't care, we're not losing all this productivity because our emails are being leaked, have a good day
The word "agent" implies that they should _only_ act on your behalf.