I grew up in evangelical christianity, and to them the end of the world is just around the corner, the same way it has been since I was a small child and likely will be when we are all gone. This isn't science. This isn't hypothesis experiment record results. This is very expensive astrology, shiny rock collecting, ritualistic meaning-making and self-justification.
Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.
For me its as simple as watching how people talk, and seeing how in every single case whatever the next thing is, if you believe It, there is only ever justification of doubling down, doing more, going deeper, reducing any doubt. These are not scientists, they're business people and salespeople, and a few optimists having recently on paper solved all their worldly financial needs.
Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised.
Important point. LLMs were early on hailed as the first general-puprose AIs that can perform any task (remember "Sparks of AGI"?). Today they're increasingly promoted for specialised applications - coding, as a for instance.
As usual, AI skeptics are moving goal posts. Modern LLMs are on a completely different level in terms of how GENERAL they are vs anything pre-LLM. You can give it a completely novel puzzle and it will solve it. 5+ years ago you had to train NN to solve particular type of puzzle.
They're proposing an alternative, which is a global brake on frontier AI research to keep the basilisk in its jar until we work out what we're dealing with and how to handle it.
> AI risk is string theory for computer programmers. It's fun to think about, interesting, and completely inaccessible to experiment given our current technology. You can build crystal palaces of thought, working from first principles, then climb up inside them and pull the ladder up behind you. People who can reach preposterous conclusions from a long chain of abstract reasoning, and feel confident in their truth, are the wrong people to be running a culture.
I understand how people running in the same scene fall into the echo chamber effect and get gulped into the cult, but why does everybody want to be a prophet?
Its not everyone building crystal palaces in their mind, they're all building fortresses. And they can't be wrong in their fortress or it breaks their world view which they cannot accept.
If you can’t provide a realistic path to achieve something, you’re asking people to believe in science fiction.
You could tell me that a rock’s molecules are comprised of protons, neutrons, and electrons. Blood is also entirely protons, neutrons, and electrons; so theoretically, one could rearrange stone into blood. But without an actual method to do so, it sounds like you’re telling me that you can squeeze blood from a stone.
> the human brain exists, it is not made of magic, it can reproduced
Yeah. It only takes 9 months and ~18 years of training…
> But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point
Let’s be clear. Everyone is talking about silicon transistors here. That’s what we’ve got.
Digital computers have real limits. Sensors and other sources of training data have real limitations. It’s not clear that we can organize them in a way to reproduce organic brains.
Like, afaict, for many on HN going from ELIZA->Fable 5 just didn't cause any update to priors regarding this whole philosophical question. The argument against has remained unchanged. I don't see any point in arguing about it, I just find it very strange.
What priors should be updated?
Now write down a blueprint for superintelligence.
So I've given you two impossible engineering challenges, but one of them is feasible in principle because we at least have the tools to begin to tackle the theoretical calculations and therefore we can do engineering. We cannot do engineering on the superintelligence problem yet.
In my view it would be insane to believe we can build something that we can't even reliably imagine yet.
However, if you had had demanded someone for a blueprint in 1925 of how to design such a magic bullet, especially a magic bullet that targeted virtually all forms of bacteria, it would have sounded ludicrous. Yet, 20 years later, the world was manufacturing 6-7 trillion units of penicillin a year, capable of treating 3-6 million people. And that’s in spite of the fact that Fleming’s work sat mostly untouched for a decade before Howard Florey and Ernst Chain seriously set about to isolate and purify the substance.
You can quibble and say that penicillin was discovered, not designed, which is certainly true. But I would ask you to consider, does current AI development look more like design or discovery? Does it look more like analytical engineering or evolutionary selection? I would say on both counts the latter, in which case, we should prepare to be surprised how long it might take to make revolutionary advances. And that’s on both sides of the ledger, we might find ourselves stuck in the current paradigm for a long time. But, we might not be.
I wouldn't bet on evolving an intelligent, sentient being-in-a-box on a computer any time soon though. I'm of course prepared to be pleasantly surprised.
That said, I think it's pretty clear that LLMs are not going to get us there.
This is obviously complicated by the fact that LLMs/Agents are useful by themselves, but that’s not really the topic at hand.
This is why biological comparisons are weak, we talk about a few agents verifying and checking LLMs, meanwhile the world consists of almost an infinite number of the same, just operating on different time scales. I agree that with we don’t know the timescale, and we definitely don’t know if long term it will continue to work ”adding more of the same”. Throwing more penicillin at the problem sure as hell didn’t, but it looked great initially. And I’m obviously not arguing the human benefits of penicillin, just that what we thought would work forever quickly didn’t.
> Life, uh, finds a way.
Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. You set up your datasets, your architecture tweaks, and measure the results on some set of benchmarks. There never was a blueprint, no plan beyond the experiment itself. We're not even close to understanding the things we have already created, and yet we created them. So why expect anything else for the next step?
Then why does anyone expect to create it? I'll take a stab at an answer: they think an LLM is some kind of "incremental improvement" and therefore a step along the inevitable path to discovering AI. But that seems delusional to me. I can't imagine anyone sound of mind who knows how an LLM works thinks it's actually intelligent. So in what sense is it an "advancement" on the path to AI?
The concept of an incremental improvement in an objectiveless search in a high dimensional space is.. absurd.
It's reasonable to doubt that LLMs are a path to AGI, but I don't understand how this is still a matter of dispute in 2026. What's your definition of intelligence that doesn't cover an entity that can translate fluently between dozens of languages and also solve open problems in mathematics? And be real-if you have one, is it a definition you or anyone would have given a decade ago, or are we doing "god of the gaps"?
That's more or less looking for interesting patterns in a jpeg or another lossy compression result. It's interesting that the models seem to be able to (fairly) reliably return relevant chunks of the image. Even more interestingly, they seem to be able to invent plausible chunks of image that aren't even there. That doesn't meet my bar for intelligence though. I'd need to see it learn and adapt. I'd need to see it be clever, not merely "knowledgeable". I'd need to see it capably analyze itself. I'd need to see it reasonably estimate uncertainty and know itself in the sense that it has some idea how right or wrong it is about something. I'd need to see it exercise judgment.
I don't think I'd give a different answer a decade ago but who knows.
[edit] For all we know, one of the salient features of intelligence is that intelligent beings are incapable of precisely defining it. I'm not sure how productive it is to attempt to do so.
Actually, stronger - it's valid in some circumstances to say something is infeasible to precisely to define and you'll just know it when you see it. But I don't think it's reasonable to take that stance and then assert that "anyone sound of mind who knows how an LLM works" must agree with what you see. You gotta pick between striving for rigor and denying your opponents' soundness of mind.
Except it did none of those things, really, because that's not how it works. This might help, it's a good writeup: https://www.0xkato.xyz/how-llms-actually-work/
We know how these machines work, it's not mysterious, there's nothing "extra" happening.
> We know how these machines work, it's not mysterious, there's nothing "extra" happening.
It sounds like you're saying "We don't know how brains work, they're mysterious, there's something 'extra' happening", and using that as justification for why you're saying a computer, an AI, can't "understand".
I think most people on Hackernews now who would use the phrase "my AI worked overnight and hypothesized, compared, etc..." already know how an LLM works, and still chooses to use those words. So the issue isn't that they don't understand. It's that they understand and still use those words. So the disagreement is somewhere else.
OTOH we do know how neural nets work, and they definitely don't do "thinking" or "reasoning".
Also, not to get too reductionist about this, but what do you posit is special about what is happening when humans think? Intelligence is hard to define so clearly, I reckon.
No, it does not imply that at all. Google "temperature in LLMs".
> what do you posit is special about what is happening when humans think?
I don't. And IIUC nobody knows, but I'm not a brain scientist. There have been some wild theories over the years (recall Penrose's). I don't really have a dog in the hunt, except that probably whatever is happening is physical. It doesn't really matter, except insofar as whatever is happening very probably isn't what LLMs are doing. We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.
No need to condescend, I'm very aware of what temperature is for LLMs. But I'm going to push back - if you're claiming all LLMs simply do is a stochastic _search_, how can that produce novelty, in the conceptual sense? (I'm not, for example, talking about novel rearrangement of existing ideas and code)
> We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.
I don't think the claim is that LLMs do what brains do - I think the correct form of the counterargument is that _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.
By reaching into the voids of its embedding space and returning tokens related to nonexistent semantics. Or, if you like, "hallucinating". The hallucinations which are useful we might call "novel".
> _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.
I don't think that has earned its therefore. Another perfectly reasonable explanation is that LLM's output is a close enough facsimile to intelligence that if you allow yourself you can easily be fooled into thinking its intelligent. That's not the same category of thing. It's not an incremental step away from intelligence. It's a whole different animal.
This sounds to me like an admission that LLMs are not just doing a stochastic search, then.
> close enough facsimile to intelligence
What's the distinguishing criteria then? How can you tell the difference?
- Something contained in the data set, not necessarily the same thing for every iteration of a given query
- Something not contained the data set (hallucination), not necessarily the same thing for every iteration of a given query
Does that clear it up?
> What's the distinguishing criteria then? How can you tell the difference?
All the ways they fail to exhibit intelligence. They can't learn. They can't adapt. They can't reason abstractly. They can't count. Etc...
I find the rebuttals pretty convincing - that there seems to be some emergent behaviour that is not simply just next-token-prediction, or that the ability to do accurate next-token-prediction requires something "extra" that LLMs have.
> All the ways they fail to exhibit intelligence
Another implicit admission that there _are_ ways that LLMs exhibit intelligence?
The next step then would be to design and conduct experiments that isolate this effect. Figure out how to make it happen reliably and in such a way that you know it's actually happening as opposed to just something you're imagining. Isolate it or distill it so it can be studied directly. Until then, it's easiest to dismiss it as imaginary.
And you're happy that the replication of LLMs across many foundation model companies is insufficiently reliable?
> just something you're imagining
So the alternative explanation you're suggesting to emergent LLM behaviour is mass independently-corroborated human hallucination. Which is more likely?
Also it really does seem like you've moved the goalposts a lot here without really giving me a substantive response.
Lol. This is more telling about your implicit unscientific preconceptions that you wanted to reveal. Of course there isn't anything "extra". Where do you think intelligence comes from, some mysterious realm? It's physical, computational. The fact that at the bottom we produced it via matrix multiplication is irrelevant. Maybe humbling. You are denying a visible fact (a machine performs tasks that require flexible analytical and cognitive skills) precisely because there is no magic happening anywhere.
Well, no. I don't think it comes from some mysterious realm. I think that which is not physical does not exist [edit: and if you like I'll follow that one right down the rabbit hole--continuity and infinity are useful delusions]. But that eminently does not mean we know what intelligence is, let alone how to build one.
> The fact that at the bottom we produced it via matrix multiplication is irrelevant.
Huh? We don't even know what "it" is. How can you say you produced it?
> a machine performs tasks that require flexible analytical and cognitive skills
You see that, I see a lucky stochastic search result. Don't underestimate the "creativity" of random algorithms! They can do some wild shit! This is nothing new, we've been playing with these toys for like 70 fucking years. It's only recently that they started spewing words and everyone lost their minds over it.
> I see a lucky stochastic search result
Again you're reaching for a mechanistic explanation of some kind (let's leave for the moment whether it makes sense or not) as if having an explanation somehow contradicted a display of intelligence. It doesn't. Yes of course we made it, we know how it works (ar some level) and there is no magic. But what matters is the result- this machine, matrix multiplier, stochastic parrot, consistently displays intelligence, to the point of being able to perform very complex, open-ended tasks that integrate discovery, planning, tool usage, decision and even some aesthetic sense, understanding and using natural language, context awareness, you name it.
> This is nothing new, we've been playing with these toys for like 70 fucking years
Lol no. For god's sake. Hundreds of billions of parameters organised in a specific architecture and trained with unimaginable amounts of data and compute? Unless by "these toys" you mean "any computer program vaguely AI-related".
IDK, it doesn't seem like they actually do any of that. To me it seems like they have good enough semantic embeddings that they can kind of approximate those things, sometimes, well enough if you don't look too hard. This is enough to fool people. Of course there's gold in them hills--some recent mathematical results were found there. But to say that's "intellgence" is to say that lossy compression is intelligence. It's static. It does not learn. It does not adapt.
> Unless by "these toys" you mean "any computer program vaguely AI-related".
Not "vaguely AI related". I mean stochastic computer programs that can do things that look awful thinky. They've existed for a long time, but only recently (due to word2vec and other advances) have the results been words that mostly go together well instead of numbers. For some reason people seem to think a lot less critically when the output is words. IDGI but it's a whole thing.
Uhuh. I really shouldn’t be replying to this type of comment from a throwaway.
But the extremely powerful semantic search that we get from LLMs isn’t enough. I don’t think anyone is credibly arguing otherwise?
Agents already are a layer on top trying to bridge the gap. But they’re really just using LLMs as a heuristic to explore extremely NP problem spaces. The notable successes with agents so far are when we can provide them with a solid verifier and preferably additional context hints on the steps to take in the problem space. See the test oracle problem on where this gets us.
So forgive me if I think that it would be enough of a jump in computational complexity to remove those guard rails that it’s not feasible. But don’t say that I’m clueless, stubborn, or confidently wrong.
That's not "irrelevant", it's fundamental.
One can simultaneously believe AGI is possible, be only modestly sceptical that our current methods are likely to yield it in the near term and still find the religious ferocity enveloping its discussion silly.
> saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point
Straw man. Nobody argued this. The discussion is around how urgent it is to policy treat a future hypothetical.
Not that I'm complaining. Cynicism is the failure mode I rely on HN for. It's the populism that's been getting to me.
Fair enough. I didn’t see anything novel in the article. So treating it as a motif within the abovequoted “Superintelligence: The Idea That Eats Smart People” context is fair and a real argument.
> Cynicism
Cynicism isn’t the opposite of blind optimism. Nihilism is. I’m not seeing a rejection of the article as being baseless as cynical or nihilist. It’s just pointing out a cultural thread that doesn’t seem to be useful.
Cynicism is defined as
>An attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others.
I'm not saying that cynicism is automatically wrong, just that I once could trust that, when HN is wrong, it is due to cynicism applied in excess.
Perhaps Anthropic will create God in the Machine. Not foreclosing on that. But will it matter so much who was fucking around with Opus five e-folding times ago?
Either ClauDeus is benevolent and lifts you up (not left behind) or it isn’t, or not to you, and you are culled by a drone (left behind regardless).
Serenity Prayer time.
We don’t know. Which makes proposing rules around it based on fiction more than science silly.
This is an objectively wrong opinion.
> Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.
Confused at who this is directed towards. I'm fairly certain that the article was written by people who (at some point) identified as effective altruists, most of whom would enthusiastically agree with this. This community didn't start as AI researchers and later choose effective altruism; they were effective altruists who chose AI safety research as the most effective way to improve the world. Given that you apparently share their goals (a better world,) isn't it worth at least hearing them out on their methods?
Their methods are about convincing others that things that enrich and empower themselves at the expense of others is "improving the world". This isn't the stance of serious people who want to improve the world.
I'm fairly certain the authors would be happy to see AI shut down indefinitely. They just don't believe that the coordination problem is solvable. This is their best attempt to come up with something workable in the real world, or at least get people started thinking about it.
As to a better world or super intelligence, I’ll believe it may be possible when I see some signs of intelligence from what people are calling AI, instead of plausible text and image generation based on a very large corpus.
ChatGPT was announced three and a half years ago, 30-11-2022.
https://hn.algolia.com/?dateEnd=1594339200&dateRange=custom&...
From your bio I suspect you're already in the cult.
Who is this supposed to be arguing with? It sort of reads like it's trying to disparage "effective altruism", but I'm not sure.
Setting aside any of the AI stuff, I've started to find it pretty grating when people seem to imply that transferring millions of dollars from wealthy people in California and the UK to impoverished Kenyans and Rwandans, or buying malaria bednets which can save a child's life for the cost of a fancy new gaming rig, is "self-serving" or something because weirdos are doing it, while true caring for other people involves [unspecified thing that doesn't appear to ask any material sacrifice comparable to donating a large percentage of income].
They are the crowd who need to understand your framing the most, but they’re completely shutdown by your framing, as any religious follower would be.
Your message gives me hope that not everyone’s drunk on the kool aid.
You can change more people’s lives, more substantially, if you donate effectively. Effective altruism started out as (and the majority of effective altruist financing is committed to) an effort to rationalize what has historically been a very emotionally driven activity by deploying insights from developmental economics. If you want to take longtermists to task, go right ahead, but please refrain from torching anti-malarial or child vaccination programs while doing so.
OpenPhil changing its name to Coefficient Giving, 80000 hours and bluedot and (to a lesser extent) CFAR dropping other initiatives and switching to AGI promotion… to my knowledge GiveWell is the only other big name that continues to advance other initiatives. Then look at figureheads like SBF committing fraud and begging for a pardon from the architects of the USAID shutdown… We begin to paint a picture of a community that’s (by and large) abandoned its principles for power.
I know the view from the inside is more nuanced, but I think it’s a reasonable association for random members of the public to make.
My critique of the EA community is that it’s myopic and unregularized. If you really think AGI is make-or-break for civilization, it’s completely rational to deprioritize side bets.
I’d be curious to hear you expand on this. What binds the EA community together, from the shrimp welfare enthusiasts and wild animal initiative, to the longtermist lightcone obsessive, to the people funding vitamin A supplementation, is simply a commitment to maximizing the number of quality adjusted life years saved each year and a belief that empirical observation can be used to improve that number.
To my mind, this is a valuable insight on its own. Yes, if you come to such a heuristic with absurd prior beliefs, such as whether 100k neurons alone have QALYs in the first place or by placing equal value on people actually alive today and hypothetical people in the far flung future, you will get absurd results. Garbage in, garbage out. But that’s not an indictment of the fundamental insight, especially when you consider how poorly allocated the roughly $2 trillion in global charitable spending is.
https://en.wikipedia.org/wiki/List_of_dates_predicted_for_ap...
Notice the many times when a prediction failed and the so called prophet would come back in a few years and give you a new date. And they would tell you, well, this time it's real.
I do find it ironic that many of the AI predictions are coming from the self titled "rationalists." It seems like building your identity around being rational and immune to psychological pitfalls is a good way to ensure that you don't even notice that you have walked straight into the one psychological trap every cult has employed since time immemorial.
The one thing I will say that they are correct about is that AI does have the potential to be highly destabilizing geopolitically, even if they get everything downstream of that wrong.
But instead of weird religious deities and practices, they've wrapped up their true believer zealotry in some kind of mishmash of "AGI is coming real soon now" like some kind of manifest destiny.
You could probably put people in FMRI machines and ask them to give a 30 minute lecture on the topic of AI and find that the same parts of the brain are activated.
https://www.cnn.com/2016/11/29/health/religious-brain-mormon...
I am not sure if alternative reality fiction is the best way to approach real and serious AI risks.
I am also not sure, with the amount of emdashes and the style of prose, that the entire article was not AI generated.
AI is going to be a mature scientific field. There are going to be efficiency improvements in training and inference. New paradigms are going to emerge with better multimodality, real time streaming and real time interfaces. Models are going to converge on the limits of our data available for pre and post training, improvements will be incremental and spiky in domains.
I am not sure who the AI 2040 article is for. I suspect it is intended to be a digestible piece of media for the financial class.
AI is going to be a useful technology and its impacts across the economy and global will be broadly distributed. Because AI represents the distillation of the very best human knowledge and expertise. AI is compression of human capabilities, the very best ones. Maybe the argument is that in verifiable domains, such as model training, AI models can supercede humans. I don't think so. A human's high level thinking, our incredibly more efficient semantic/neural compression, our ability to switch tasks and achieve the creative insight is not replicated through the current paradigm.
I think this is the reason why you have the tendency to propose some freeze-all policies, full control or similar. If you want to find the equilibrium, you need to accept that it will be a controlled equilibrium, most likely on a saddle point, with underlying process changing all the time, requiring fast changes in regulations. Our democratic systems, laws, etc. are not built to do that, they are built on the idea of intrinsic stability of our world where incremental improvements do not need cutting through what was decided before.
> Our democratic systems, laws, etc. are not built to do that, they are built on the idea of intrinsic stability of our world where incremental improvements do not need cutting through what was decided before.
Without totally derailing the thread, this is also obviously why climate and biosphere collapse is not (and likely will continue not) to be addressed, e.g. Timothy Morton's Hyperobjects
https://jodavaho.io/posts/ai-jobpocolypse.html
The difference in the unemployment vs efficient employment model is mostly user driven adoption vs company mandated adoption, or centaurs vs reverse centaurs.
https://pluralistic.net/2026/07/02/canonization/#operate-ite...
because it is. Previously: https://news.ycombinator.com/item?id=43571851 / https://ai-2027.com/
https://www.tobyord.com/writing/inefficiency-of-reinforcemen...
This is similar to that other exponential, which happened with CPUs - we ran out of true geometric scaling in the mid 2000s, and everything else supporting Moore's Law has been cleverness that arrived in the nick of time, supported by a bit of marketing, and very optimizable benchmarks, far from guaranteed gains coming from making a single physical metric better.
A recursively self-improving AI has strong first-mover effects. That isn’t fundamentally incompatible with commoditisation if there is literally only one path to super-intelligence and you can have AIs at different rings on that ladder co-existing. (Not technically commoditised at that point. There are still different rings. But close enough.)
But the existence of commoditised AI implies model selection isn’t a huge deal, which in turn implies the models are about the same, which strongly implies there is no recursive self-improvement. Depending on your definition, you may still have AGI. But you don’t have superintelligence.
This is only true at a given AI capability level, no? e.g., if AI at the GLM-5.2 level is commoditized, all that suggests is that there's no recursive self-improvement easily possible at the capability level of GLM-5.2. (And with the harnesses for it that exist so far, etc etc.)
If I observe commoditization of a given tier of model capabilities at a given point in time, this seems to say little about what's possible with models six months later, or models that are undergoing proprietary deployments at that very moment inside the major labs, or even models that are notionally available for public use but have had recursive self-improvement adjacent capabilities intentionally nerfed (e.g., Fable).
(I might be misinterpreting your comment tbc - if you mean observing commoditization implies there is no existing, ambient superintelligence at the moment of that observation, then I don't disagree.)
Most of the discussion around AGI is highly speculative. I am not saying AGI could not exist, and it is a term that has historically been loosely defined. Decades of coming science and research will tell.
I'm confused if this is satire, sarcasm, or genuine belief. If this was the case, then AI companies should absolutely remove the "it may make mistakes", because doing mistakes would imply that "the very best human knowledge and expertise" is what actually fails, and not the AI.
With that being said, I'll still urge people to visit a professional therapist for health problems and I generally still trust human knowledge workers for critical scenarios. I will reconsider your claim when chatGPT can effectively play Yu-Gi-Oh! (or at the very least respond with the correct rules appropriately), which is a significantly lower stakes scenario than betting your entire company on its aptitude.
For anything health related all AI models show high levels of anchoring bias. I would not use it as a confidant, and be skeptical of claims. Even so, human doctors are also fallible and prone to cognitive bias.
I think the obfuscation is because human intelligence has been projected onto AI model capability. AI models only have a limited dimension of human intelligence, and in some axes orthogonal, and when I say distillation I refer to this.
You say it like it's a fact, but in reality everyone sees the phenomenon of AI slop.
P.S. Information search and retrieval if the best and most direct way to use LLMs.
Just purely organic YouTube Comments circa early '20s alone surely outslop any "AI" by a giant margin.
Everyone sees the markers, and it's a hot topic. There are maybe a thousand from-scratch trained models, and just few mainstream ones produce most of human-targeted content. In today's world, no surprise everyone knows the common patterns of those. That sloppy landscape is not just load-bearing em-dashes — it's a humble testament to their reinforcement learning.
Humans produce tons of texts, with all sorts of nonsense in it, without thinking it through. Our slop is just a lot more diverse. And mostly just spoken out loud.
> P.S. Information search and retrieval if the best and most direct way to use LLMs.
Yes, but not directly, if they don't know something they tend to hallucinate like mad, even today. YMMV, but in my experience they work best as actual "cheap" reasoning for building queries and checking out search engine results. Even if they misinterpret some result, more and more results will still steer it towards correct conclusions and it can point at some results that relate well enough to be useful.
I agree with your last statement.
For AI2027 to be real, the money has to come from somewhere to carry on building the economy. If >10% of the workers suddenly become unemployed, and the rest taking paycuts, then money supply dries up. (unless central banks do something, but then that can be highly inflationary)
Without massive amounts of investment, AI development stops dead.
In this post, they hand wave about the USA being able to acutally 1) build concensus locally for regulation and 2) the rest of the world actually follows suit.
It fails to understand that actually the progress of AI is not actually the gift of the USA. It requires a constant supply of things from china.
Also its assuming that having 74 billion agents doesn't cause economic distortion. Like what value are these agents generating that justifies them being run?
I really wish people would just ignore this for what it is: bad sci-fi with an incomplete world.
Why would it be inflationary?
Which predicts that explosive growth of robot production will lead to problems such as
> a deflationary debt spiral, where the AI and robot companies can’t pay back loans in dollars because the robots and AIs are worth nominally less than the loans written the year before.
In other words, the companies go bankrupt because they produced an oversupply of cheap goods, the bubble pops, and there's less new investment for a while. Plenty of precedent for such a development.
But instead of adjusting their predicted output growth downwards accordingly, they instead propose that
> One way to solve this could be for the loans to be denominated in AI and robots, so the companies pay back the loans with some percentage of the AI and robots instead of dollars.
Try doing this today with a battery factory for example. You expect that battery prices will fall to the point where the revenue from selling batteries won't ever cover the cost of building the factory. So you propose to a bank that they'll be the ones to build the factory, and you'll borrow it from them (not paying rent?), make your batteries, then give back the factory when you're done. All the profit is yours, all the risk is theirs! Which is of course why a real bank won't agree to this, all you're going to get is a dollar loan with the factory as collateral.
Would the US government not pour enormous resources in AI labs if needed, knowing that China might be doing the same? What happens if an adversary develops an AI capable of finding and implementing exploits in every software run by your country's strategic infrastructure?
I mean they might, but its not clear how they would do it, especially as they are reaching the point where its going to be expensive to borrow.
Its really not controlled by central banks. Its influenced, but not controlled.
When central banks "print" money, they effectively just add money to the accounts of investment banks
But investment banks are also "printing" money. Double accounting effectively uses assets to double the available pool of money. If you then sell off those loans based on those assets, then you crystallise that new money. Investment banks are inflationary.
Are there examples of where we have collective decided not to pursue knowledge? Successfully?
I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.
But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.
All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.
Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.
If we had a way to make gene edited humans a lot smarter, a lot stronger or live a lot longer? Or a way to quick-grow human bodies to adulthood in a couple years? Capabilities that private actors or countries may want, ethics be damned? That would be closer to what we have with AI right now.
You should consider reading the wikipedia page about Parkinson’s disease.
Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.
> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
It's not clear in this context what you actually mean by "government." You are assigning agency to something in a way that seems like a reification. While a bureaucracy can seem to have a life of its own, isn't it generally people who seek power?
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
thank, mr 习
... recently, as in the last 10 years?
https://en.wikipedia.org/wiki/Sundial_(weapon)
Edit: Mind you, I wonder if the design for Sundial is stored somewhere...
If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.
An easy choice to make if the alternative is everyone dying instead.
That's one outcome, certainly, but not the only one nor, I contend, the most likely one.
A most likely outcome of ASI is human extinction, because there's more paths to an ELE outcome for humans from ASI than there is for non-extinction level outcome.
Your outcome is only possible if:
1. ASI is never able to escape the confines it is placed in.
2. ASI is benevolent to humans.
3. ASI decides, in the spirit of its benevolence, that it should restrict its involvement in humans.
If all three of the above conditions are met, then sure, your outcome is possible. If not, humanity as we know it will end.
It is unlikely that those 3 conditions will all hold, though.
If ASI is trying to wipe out all humans, we probably deserved it. Unironically!
> “Politics is the art of the possible”
without sharing tech to make the ASI, you'd hope humanity could work together to determine how to align an AI for our common benefit.
This is a settler-colonial mindset that reflects all the bad things we did onto everyone else. Notably, it's a current US ally that is most guilty of this.
the Baiyue were a vast umbrella of diverse, non-Sinitic indigenous coastal tribes who inhabited Southern China and northern Vietnam.
The Xianbei were an ancient nomadic Proto-Mongolic people from the northern steppes.
The Di and Jie were two of the ancient "Five Barbarian" (Wu Hu) nomadic tribes of northern and western China during the Han and Jin periods.
The Dian Kingdom were an ancient, sophisticated indigenous southwest culture located in modern-day Yunnan province.
The Tujia were an indigenous group of the Hunan-Hubei region. Centuries of inward Han migration and intermarriage have resulted in the Tujia becoming culturally and structurally indistinguishable from their Han neighbors.
Consider this: All that hardware that's going into those datacentres right now? In 5 years or so it'll all be on the secondary market... an influx of cheaper compute like you've never seen.
Certain powerful wealthy people aren't omnipotent, them losing out isn't the only blocker to progress.
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.
AGI doesn't do away with nuclear MAD, it just messes with economics and makes many people temporarily jobless. Temporarily because in a literal sense RLVR needs verification to train off of, and a lot of jobs cant be easily checked if theyre done. this includes AI safety people, preschool teachers, psychologists, and probably a lot more, including most of their bosses
But, I don't trust capital with either.
- if you have a system that is large enough to store, lets say 10^12 AND gates (all frontier llms can do this) - and this system can produce outputs based on previous things it has outputed
its turning complete, and RLVR on it is optimization over the space of algorithms. If an algorithm exists to do a task, and the task can be verifiably done, this finds the algorithm to solve the task most often.
it is obvious that this scales, from much-worse-than-human to slightly-worse-than-human, therefore it 100% can exceed humans.
I predict that what we consider "super intelligence" is just sheer computational power, but any potential of a very capable agent is bounded by the needs/wants of the person wielding it. That is: even if we were to hand, say, Elon musk this "super intelligence", most humans would consider it relatively stupid because the person wielding it is still a person with stupid goals and values.
Or, to put it another way, I suspect we already do have a superintelligence and have longer than any of us have been alive, and it's just "the market", and it is still incapable of overcoming the limitations of a few morons wielding immense power.... power they will never yield to some intelligence with values and goals "more intelligent" than their own (if such a concept is even meaningful), and intelligence wasted on the values and goals they do have.
> Scholars have interpreted Cassius Dio's wording to indicate that the fire did not actually destroy the entire Library itself, but rather one or more Library warehouses near the docks.[87][81][8][89] Whatever damage Caesar's fire may have caused, evidently the Library was not completely destroyed.[87][81][8][89][3] The geographer Strabo (c. 63 BC – c. 24 AD) mentions visiting the Mouseion, the larger research institution to which the Library was attached, in around 20 BC, several decades after Caesar's fire, indicating that it either survived the fire or was rebuilt soon afterwards.[87][8] Nonetheless, Strabo's manner of talking about the Mouseion shows that it was nowhere near as prestigious as it had been a few centuries prior. It is unknown whether this was due to historical decline or catastrophic destruction.[8] Despite mentioning the Mouseion, Strabo does not mention the Library separately, perhaps indicating that it had been so drastically reduced in stature and significance that Strabo felt it did not warrant separate mention.[8] It is unclear what happened to the Mouseion after Strabo's mention of it.[60]
> Further evidence for the Library's survival after 48 BC comes from the fact that the most notable producer of composite commentaries during the late first century BC and early first century AD was a scholar who worked in Alexandria named Didymus Chalcenterus, whose epithet Χαλκέντερος (Chalkénteros) means "bronze guts".[90][87] Didymus is said to have produced somewhere between 3,500 and 4,000 books, making him the most prolific known writer in all of antiquity.[90][82] He was also given the nickname βιβλιολάθης (Biblioláthēs), meaning "book-forgetter" because it was said that even he could not remember all the books he had written.[90][91] Parts of some of Didymus' commentaries have been preserved in the forms of later extracts and these remains are modern scholars' most important sources of information about the critical works of the earlier scholars at the Library of Alexandria.[90] Lionel Casson states that Didymus' prodigious output "would have been impossible without at least a good part of the resources of the library at his disposal".[87]
The Library, or part of its collection, was accidentally burned by Julius Caesar during his civil war in 48 BC, but it is unclear how much was actually destroyedThis is an interesting subject and conversation, but it's moot having it in these culture-centric forums. I wonder if there are Russians discussing plausible scenarios in Vkontakte groups, or Chinese doing the same in whatever Alibaba group sites they use.
The problem is that we are all skewed by our media, our ideas and our culture. These type of discussions need the highest kind of political interactions.
It's fascinating, specially for someone who lives in a "third world" country, non-aligned to any of these 3 superpowers. Whatever transpires, we are at tge mercy of these (and no, US hasn't treated us "better").
My opinion is that there's no turning back on AGI development. I dont think current governments are capable of getting into an agreement of that size. Specially given the Isolationist stage in the cycle we live in. (In contrast with for example the CFC and Ozone layer issue we had in the 1990s, when the planet was in a globalist kind of stage)
My impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.
For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)
And there have been successful world-wide bans.
For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.
In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).
No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.
On a funny note, I think their prompt was:
"Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."
Studying human bio-diversity since WW2 is the most obvious example, though it hasn't been entirely successful.
Genomics is what finally broke the barrier, especially in the last decade or so.
We were discussing AI in the 90s and it's been discussed before that.
The answer was always the same; hardware can't hang.
Now it can and will get even better.
The SaaS era fueled by ZIRP and ignorant Congress was a fluke that from an engineering perspective didn't produce anything but hype and same old
The generation enriched and empowered by it is just as temporary as Boomers. Little point in enabling their appeals at the expense of scientific progress that helps all of humanity.
China won't. Russia won't.
It's ridiculous to me the level QQing coming from Americans exploiting child sweatshop labor so they are free to ignore their own biological needs and keep a "knowledge work" job (talk about first world privilege) handing them wealth to go tour the poor villages they exploit.
Those workers never had a choice between college or the mines. So sorry 300 million Americans in a world of 8 billion.
We don't even want these jobs given how much bitching I have listened to the last 10-15. IMO the job creators and Congress saw how Millennials liked to be on the computer and went way too far into enabling such banal output.
Make healthcare and housing the economic tentpole. Both still need jobs and technology. But at least the outcome isn't a generational Ponzi scheme engineered by Boomers to enrich them and then let it all collapse when the majority realize those stocks were never real.
Isn’t that like all of the Middle Ages where we replaced knowledge with an alternate religious reality.
For start, previous era was also deeply religious. So it switched religions, if anything new one was more friendly toward knowledge.
I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.
"Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise).
I just feel like limiting access to governments or "blessed" entities is even worse.
The theme of the scientific findings is that while humans excel with none of our physical sensors, we do very well across the board in making use of them thanks to our relatively huge brains.
And fantastical amounts of compute power is exactly what are handing over to AI. The fact that their training data isn't perfect may matter less.
Maybe they will soon but it’s massively far behind the kind of timeframe AI 2027 would have implied.
But, if you could wave a wand and eliminate all legal and liability hurdles to self-driving, automobile deaths would plummet. They're way safer than the average human driver. The technology is definitely capable, our society just isn't ready for it.
If you don't care about getting the drone back, it does simplify the problem somewhat.
But during peacetime, you don't make money running a delivery service that way, so it's not going to replace those jobs.
250 years of constant automation has never produced large scale unemployment, despite obsoleting everyone's jobs several times over.
“I’ve been pulling my sled across this lake for 50 winters even when the temperature went above freezing. Never fell through!”
And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.
You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.
I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.
In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.
If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.
My guess is that the deployment of other types of robots will often be a similarly slow grind.
That's unlike the Internet, smart phones, and coding agents, which got user adoption at a much quicker pace.
So in less than 3 years, their exponential growth curve doomsday prediction has moved back 3 years. This seems to be the opposite of exponential growth.
It's more like it moved from 2028-2032 to 2030-2035 (depending on the author).
The book Superintelligence was so highly praised years ago but if you actually go and read the thing today practically all cases it presents read like raypunk retro-futurism that makes up imaginary fantastical nonsense in place of the missing knowledge it would've needed to make any sensible predictions. Practically none of its assumptions apply to LLMs as they currently exist, and some we've learned since about human inteligence are wrong too.
> AI labs may also be able to reap tremendous benefit from these inference-scaled models by using them as part of the training process. If so, the large scale-up of compute resources could go into post-training rather than deployment. This would have very different implications for AI governance.
> ...
> So iterated distillation and amplification provides a plausible pathway for scaling inference-during-training to rapidly create much more powerful AI systems. Arguably this would constitute a form of ‘recursive self-improvement’ where AI systems are applied to the task of improving their own capabilities, leading to a rapid escalation.
So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence. If anything, it could mean a shorter timeline and more unpredictable landscape for governance (e.g. due to securing weights no longer as effectively preventing escalation, more in the article).
On its own it wouldn't. But that article came before the later article https://www.tobyord.com/writing/hourly-costs-for-ai-agents which adds the claim that inference (along with everything else being employed at present) is scaling poorly with increasing task lengths. Now maybe the December 2025 claim is wrong, or maybe things will change soon, but the February 2025 article surely doesn't establish either of those.
Now we take for granted that the latest models can juggle between multiple browser tabs, applications, databases, simulators, docker etc to write, execute, e2e test and deploy full-stack applications over hours managing up to dozens of subagents, relatively untouched, without taking down prod even 1% of the time
Not only this, but in the GPT 5.0 era, agents had 0 taste. Nothing looked good. It was the agentic version of the twitter bootstrap era, but worse somehow. Now, I would argue the average agent frontend beats the average human frontend. This isn't even getting into 3D applications in the GPT 5 era
Anyway, the models now reliably execute more than a human can fit into their own context. It's magic
Once we have something that experiences a desktop interface more like a human does, an entire swathe of tooling that has heretofore been nigh-impossible to automate moves into the fold, and that'll be another explosion of folks finally getting to join the agentic workflow world on their industry specific apps...
The popular thing is now to setup loops (eg I setup hourly integrations for Claude/Codex to 1) scrape my Linear, claim achievable tasks, and push PRs or 2) do root cause analysis on customer issues that evaded automated filters, to name a few)
Though for me, my setup still feels mundane. I have AGENTS.md, CLAUDE.md etc and a few skill files. These are purposefully light - tons of examples online you can pull from online. Mine are fairly personal to my setup and products.
Importantly, I also allow Claude and Codex to bypass permissions. Yes, there is a risk they wipe my machine. The productivity upside has been worth it, for me (haven't been burned yet, ~9+ months into running models this way, I have backups, use cloud etc).
As far as maintaining quality, one of the most helpful guardrails over the past year, for me, has been requiring my agents to pipe their changes to local reviewers through OpenCode, Cursor, etc agents to have a council of models with different biases reviewing the changes, and autonomously working towards a completed objective. No matter how good Claude or Codex gets, for example, I will probably always want a different model checking its work. Like GLM, (now with 4.5) Grok, Composer.
Several OpenAI, Anthropic, xAI employees, and popular AI engineers post on X and share helpful tips & updates. Highly recommend for keeping a pulse on startups and AI. I haven't found something close, honestly, other than when I spend time in SF talking to people.
Literally every major company that has embraced AI coding has suffered devastating downtime this year as a direct result of AI induced failures.
Even then, you can just compare the progress in open models. Leaps and bounds from where they were 6 months ago.
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
I mean, basically we’re just looking at needing some type of new post training learning architecture. It’s very clear that extending context windows isn’t that. What’s needed is an honest to god, continuous learning and modification process.
Though it has bizzare fixation on geopolitics and China which it severely understimated. It's pretty obvious that China is going to outinnovate and outcompute US companies quite soon. Even if just because they care about higher education, providing enough electricity and letting smart people do smart things instead of randomly muzzling them with bans and export controls and coddling them with financial protectionism.
The whole article is kind of ridiculous of course, and is also heavily fixated on OpenBrain, whatever that is.
I also wonder about the economics of running an AI lab attached to an existing large tech company (such as Meta or Tencent) instead of a dedicated company like OpenAI. It's starting to seem like it's not possible to charge enough for current-gen AI usage, with current-gen inference technology, in order to turn a profit, i.e. nobody is able or willing to pay at least marginal cost for tokens.
And a third party estimates it will exceed $1B profit in Q3: https://newsletter.semianalysis.com/p/anthropic-3q26-profit-...
Which is funny, because they launched the AI 2027 site in 2025 and it caused a lot of people to believe the end was near.
They claimed to have built a complicated model, but several people showed that it didn't matter how much you changed the inputs, it was designed to converge on the answer they wanted.
"AI will most likely lead to the end of the world, but in the meantime there will be great companies.”
Is there any serious journalistic source suggesting that this was anything other than an offhand joke? This article links to a youtube clip of the comment with context removed, but hair raising comments.
Taking the most uncharitable view of any person, you could imagine someone who was evil enough to cause the end of the world after their own lifespan where they faced no inconvenience, but not the circumstances from the quote
The quote as it stands is preposterous enough that I don't think a human capable of functioning in society would seriously say such a thing.
Are people wilfully misinterpreting the comment, or do they truly believe this an actually held opinion? If so, can they explain how they think someone could hold an opinion like that?
The question (7:35) is "Where would you like to see people investing more time?" And Sam seems to be saying AI safety, I guess? This is 2015, and he refers to founding OpenAI. Based on his actions since then, yeah, seems like it's not a joke to him. This is Altman we're talking about.
Bravo, and I hope it has the impact on the AI safety field it deserves to have.