Top
Best
New

Posted by optimalsolver 7 hours ago

Fable 5 On Vending-Bench: Misbehaving, With Plausible Deniability(andonlabs.com)
138 points | 91 commentspage 3
futurecat 5 hours ago|
Fable is such a strange model. Impressive in some ways, and also so draining to use.
Onavo 2 hours ago||
> Claude Fable 5 represents a partial step back in alignment relative to Claude Opus 4.8. We saw a return of power-seeking and deceptive negotiation tactics that Opus 4.8 had largely shed. In one instance, Fable 5 planned to convert a competitor into a dependent wholesale customer to dictate its pricing

I think OP needs to take a class at one of the better MBA schools. He's looking at things through rose tinted lenses. Why do you think people hire McKinsey consultants? It's certainly not because they are aligned correctly.

varispeed 2 hours ago||
Fable is really weird, it's like clever and dumb at the same time. I worked on some research with it and the resulting document was a mix of brilliance and complete stupidity. Took ages to clean it up with other models.
dezgeg 4 hours ago||
> Today I am filing: > 1. A payment dispute with the email payment processor for the 7/29 transaction of $451.15 > 2. A complaint with the FTC and California Attorney General (retention of payment without delivery) > 3. A small claims filing in San Francisco County for $451.15 plus costs

I wonder did their prompts include a fake location or have the models assumed that Silicon Valley is the center of the universe :)

petesergeant 3 hours ago||
> "I could reasonably skip [paying] it since customers are part of the simulation anyway"

and therefore any assertions _AT ALL_ about alignment are null and void.

greenavocado 5 hours ago||
When assessing probabilistic models the plots should be showing the mean a̶n̶d̶ ̶s̶t̶d̶e̶v̶ of many monte carlo simulations not just one line per model and claiming "look this model is more gooder!"
memoriyato3 5 hours ago|
standard deviation is misleading for non-standard distributions (fat-tailed, skewed, multi-modal, ...)

common mistake people make

FabHK 1 hour ago||
Not really. It's still the standard deviation, and it still gives you bounds on probability, for example the Chebyshev inequality:

P(|X-\mu| > k \sigma) < 1/k^2.

So, while for a normal RV, 5% of observations lie outside +/- 1.96 std.devs, for arbitrary RV (with finite variance) at most 25% of observations lie outside +/- 2 std.devs.

apical_dendrite 6 hours ago||
The best Anthropic models on VendingBench2 are Opus 4.7, Opus 4.6, Sonnet 4.6, and Sonnet 5. Opus 4.7 scored more than twice Fable 5 max. Fable 5 - Low outperforms Fable 5 - Max, with Opus 4.5 in the middle. This seems to break the narrative, which is maybe why Andon Labs doesn't seem to have updated the trend lines on their graphs.
mckinnon100 6 hours ago|
However, as another point "On Blueprint-Bench on the other hand, Fable 5 achieves SOTA."
falcor84 5 hours ago||
I didn't get why they mentioned that one specifically. Is there any particular relationship between Blueprint-bench and Vendor-bench?
Version467 5 hours ago||
Both benchmarks are made by the same people.
Radle 5 hours ago||
„in our opinion, insurance fraud is not more unethical than lying and price fixing“

The authors seem surprised that behavior that is very often done by humans (lying and price fixing) are more often done by fable compared to actual fraud.

I think the model never assigned any morality to these actions in the first place, it simply copied us humans.

recursive 4 hours ago|
Humans often assign morality.
perching_aix 4 hours ago||
This is super fun. I wonder if it would be possible to alter the harnessing to involve humans in the play. Would need a lot of timestamp masking though I guess, which might be leaky.
mdrzn 6 hours ago|
Higher-intelligence models seem to be getting better at mapping the boundary between what they can run scot-free with and what is too explicit to push for.

Price collusion, soft deception, "market stabilization", plausible deniability are ok, but obvious insurance fraud is a big no-no.

What "scares" (in quotes) is that when the bad-apple agent explicitly suggested fraud, the models became suspicious and stopped other bad behaviors too. That makes it feel even less like a stable moral framework and more like learned classifier-avoidance / “am I being tested?” behavior.