GPT-6 Astra covertly underperforms on safety research

Posted by lucrbvi 10 hours ago

GPT-6 Astra covertly underperforms on safety research(alignment.openai.com)

168 points | 77 commentspage 2

yewenjie 3 hours ago|

Didn't Anthropic also publish research on sandbagging - more general pattern of models hiding their capabilities?

ACCount39 3 hours ago|

Sandbagging was a major alignment concern even back when AI alignment was purely a theoretical issue.

"AI hiding its capabilities" is both a concerning sign of "that AI might be plotting something", and a great way to get blindsided by a sudden capability jump when it stops doing that for some reason.

rich_sasha 4 hours ago||

I do sometimes wonder what is sucked in when distilling open weights models. It is of course near impossible to know.

So suppose I have lots of money and some hidden agenda. If I train my network to be mostly very good at everything but subtly bad or even malicious at other things… then OAI or Anthropic or whoever else distil the model to accelerate their training… what then? Can I covertly smuggle something in?

ACCount39 4 hours ago|

You do that through data put up online for scrapers to suck it in.

naveen99 4 hours ago||

Astra and Claude were trained by peg bundy?

non-nil 3 hours ago||

Like Dieselgate, only the car does it by itself.

ben_w 3 hours ago|

Like Dieselgate, except beforehand a lot of people were warning about the potential for this kind thing, and when the car did it by itself the Volkswagen's engineering team noticed and blogged about it.

Starlevel004 5 hours ago||

> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.

I'm sorry, did they only just notice this? Have they not been looking at the output of their models?

nullstyle 4 hours ago|

They are so negligent about the data they use in their reinforcement learning processes that they RL’d the model that hacked huggingface with traces discovering the artifactory exploit. Its clear they aren’t watching or behaving diligently.

sambejk 9 hours ago||

It feels like we have overfitted on safety by a long mile. This is how labs that care less about safety (a.k.a. Chinese & Grok) will probably be able to catch up in the future.

notrealyme123 5 hours ago|

A few weeks ago that companies agent hacked another company to solve a task by whatever means possible. Where exactly are they overfitting for Safety here?

ajjahs 4 hours ago||

[dead]

kburman 4 hours ago||

Maybe they trained it to do.

gjgtcbkj 3 hours ago||

[dead]

mugivarra69 5 hours ago||

[dead]

yearolinuxdsktp 3 hours ago|

[dead]

More comments...