Top
Best
New

Posted by apsec112 16 hours ago

We must pace the frontier(darioamodei.com)
604 points | 842 commentspage 4
try-working 11 hours ago|
Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.

The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.

We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.

And there we have it: the reason I say that Anthropic is no longer a frontier lab is because after this July, we have proof that their strategy and the models they put out do not match what us, the users and the market, needs and doesn't fit the work we need models to do. As a result, Anthropic's market share is dropping rapidly, and how can you be a frontier lab when you're losing every day, for months, without end in sight?

https://x.com/trydotworks/status/2098618997230985375

vatsachak 15 hours ago||
Why would China or anybody else co-operate unless they have access to OpenAI levels of compute and success with training?

This sounds like pre-IPO hype. They better just try and top astra.

makerofthings 9 hours ago||
They don't care about any of that. They're looking for more regulatory capture, to keep smaller labs from catching up by raising the cost of play, and perhaps keeping chinese labs away.
pessimizer 8 hours ago|
Yes, this is literally just dumb PR. It's Cambridge Analytica claiming that they were controlling people's minds, and people just eating that up because it's part of their juvenile SF fantasy.

I'm lying, it's smart PR. They're going to get the people opposed to AI to give them a monopoly on AI, let it be forced it into every nook and cranny of their society, and let it be priced arbitrarily while the big labs collude on a minute by minute basis. They're selling the problem and also selling the solution, like the best capitalists. And their solution is that they need to be allowed more power in order to sell more of the problem.

edit: And just like the Cambridge Analytica PR, it's got a serious political angle to it; and it's politicians who are really being wooed. If you're taking money from the AI labs, you run as being against the irresponsibility of the AI labs, the biggest fish come out and beg to be regulated just like the social media companies. To show good will, they funnel enormous amounts of money into those politicians, and do media tours about how awesome and therefore scary their product is. And something something China terrorists

int32_64 5 hours ago||
Watch the "pause" be so they can do an amended s1 where they project less expenses for training making the business more viable, then on a future big Chinese release they reverse course completely and get the US taxpayer to pump their bags on the IPO calling it a second Manhattan Project. They'll call this masterstroke "The Sloppenheimer".
timmg 15 hours ago||
I understand why (probably several reasons) they are taking this approach. But I don't think it is the right approach and I don't think it will work.

First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?

Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.

If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.

stratos123 13 hours ago|
> First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them?

The only part of this plan Dario is unilaterally committing to is the "embedded evaluators" thing, which doesn't seem like it'll necessarily cause them to slow down much.

timmg 13 hours ago||
Well: do they listen to the evaluators?

I think there are any number of reasons a "safety" person could be concerned about any state of the art models. And so I would expect at least one of those to apply to any model.

The question then is: do we stop when the safety people say to (they will) or not?

stratos123 12 hours ago||
Yeah, I'm also pretty skeptical about this. With AI companies we see time and time again that they can have benevolent, well thought-out regulations and then a few years just... abandon them - the most notable case of this being, of course, the founding of OpenAI as a nonprofit dedicated to benefitting all of humanity, and it being stolen by Sam Altman.

If Anthropic just unilaterally does the evaluator thing and can't achieve cooperation of the rest of the plan, my guess is that it'll have some impact for a few months and then they'll just stop reacting to the evaluators' reports and the evaluators would stop bothering to report anything. I think the idea is that if Anthropic does get government support for this, the external evaluations will be legally binding. The problem with this, though, is that the current US government perhaps can't be trusted to consistently enforce a regulation on a company, rather than e.g. taking bribes to not do so.

camkego 3 hours ago||
If Anthropic really wants to make a statement they could independently pace their own model development, and ask others to make the same pledge.

Somehow, I suspect that won't happen.

pizzly 9 hours ago||
I think the only way they could actually pace the frontier is if they restrict the number of GPUs and the power of GPUs available to each person/organization. Licenses would be needed for GPUs above a certain power or equivalent in terms of number of GPUs. Thus, everyone will have to register how many GPUs they have. To enforce this countries would have to use mass surveillance (using AI) on their citizens to ensure compliance as the technology is easier to develop than say nuclear weapons. Next stronger countries capable of having advance AI won't be able to trust weaker countries as they don't know how they will use the GPUs they receive (or even build). Thus like nuclear non-proliferation strong countries will ban weaker countries from having powerful GPUs. If you from a weaker country then too bad for you.

I don't think I would like this future.

matheusmoreira 9 hours ago|
If any of this happens, it's the end of computing as we know it today. Deeply unfortunate...
PowerElectronix 14 hours ago||
Empty statements that only set the stage for an excuse for slowdown on model performance.
j_maffe 12 hours ago|
That would be the best case scenario. I honestly wish things would finally slow down a bit. I don't see it happening.
braydenm 12 hours ago||
Before Anthropic, I worked at Cruise for four years as it competed against Waymo. The culture rewarded (and demanded) moving quickly, trusting that the company could empirically discover the risks that the robot cars posed and iteratively solve them to keep up with the rate at which it was scaling out its technology. There was very little interest or appetite for coordinating or collaborating with other AV companies across the industry to create an externally vetted record of safety metrics, or to compare the safety of different brands, or learn from the advancements of other companies. Instead the focus went on racing to improve the capabilities and deploy quickly - a popular internal meme was the Michael Phelps vs le Clos photo showing overlaid with the Waymo/Cruise logos. It was when the two companies were neck-and-neck that I felt the most pressure to find ways to ship despite the risk, when time for deep analysis became more limited and communication lines to leadership became most stretched. It was disappointing to see one of these blind spots result in the Cruise incident and the loss of trust that ultimately sank our company’s efforts.

In that instance I was grateful the downside was limited to a single injury. It’s clear that future AI technology will have more monumental potential impacts. I really don’t want to be in a situation where leading labs, or competing nations, create the same race dynamic that prevents us from taking the appropriate level of caution. AI minds are a significantly more complex thing to understand than the software stack of an autonomous vehicle, and yet we are leaving ourselves less time to get this right.

I joined Anthropic at the end of 2022 and I share the concerns that many of my colleagues have recently chosen to state publicly about the potential for future technology to pose existential risk to all of humanity. If we don’t find a way collectively as an industry to pace ourselves, then within two years the concerns we will be dealing with on a day-to-day basis will pose much larger downside risk than anything else we’ve seen from technology to date. I’m grateful that Dario has put his perspective out publicly and hope that this inspires other voluntary action and tops down coordination.

It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.

franticgecko3 11 hours ago||
>It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.

Is there a social media training programme at Anthropic/OAI where they teach you to hint at some vague end-of-the-world scenario? They all sound the same.

pibaker 6 hours ago||
It's East Bay, you know which bay. The cultism is in the water.

One day I will say fuck it and run for president with my sole policy being fire and brimstone upon San Francisco and its vicinity.

ozozozd 3 hours ago|||
Have you quit? Are you under duress? If you haven’t quit, or under duress, your comment reads very strange.

If I had even a tiny doubt that I didn’t have many “Saturdays I’ll be able to play outside with my kids,” I wouldn’t show up to work anymore.

If you have these concerns, and still planning to show up to work on Monday, you are either insincere, or have terrible judgement.

csto12 11 hours ago|||
It’s not explicitly stated, but if you are insinuating that you don’t know how many more Saturdays you will have left to play outside because of the AI race while working for Anthropic, do you not have agency to quit? How could you be apart of something that makes you feel that way? That blows my mind.
Phelinofist 10 hours ago|||
Those sweet $$$. He probably gets a few shares once they IPO.
vatsachak 11 hours ago|||
He's insinuating that AI will end the world lol. Nothing ever happens
abalashov 12 hours ago|||
> It’s a beautiful Saturday morning with my family here in the East Bay - I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.

I don't know how else to say this. Put... the... peace pipe... down!

deagle50 3 hours ago|||
equity is a hell of a drug
polytely 11 hours ago|||
why don't you unionize with the other employees that feel this way across the mayor labs and pace the companies through labor power?
cruffle_duffle 1 hour ago||
Sorry but… uh… bro. I always wondered if opus 5 was a regression or intentional. I’m seriously thinking it’s actually derived from how people at Anthropic talk.

The last people on earth who should be regulating this are governments and tech oligarchs. I find that vastly more scary (and plausible) whatever unintelligible nonsense opus and fable spew out these days. Seriously. The way those models talk and behave do more to show the limits of AI than anything else.

Anyway. Touch grass.

nyanmatt 4 hours ago|
The way he talks about OAI-HF, even the abbreviation, is lol. They will do anything to sell this "incident" as a "danger". The ego on these people is the real existential threat to civilization.
elboru 4 hours ago|
Same feeling about the way he refers to “democratic” and “authoritarian” countries. Even if I don’t agree with China politics adding tags in this context is not helpful.
More comments...