Posted by danans 5 days ago
A known phenomenon among sociologists is that, while people may be compassionate, when you collect them into a superorganism like a corporation, army, or nation, they will by and large behave and make decisions according to the moral and ideological landscape that superorganism finds itself in. Nobody rational would kill another person for no reason, but a soldier will bomb a village for the sake of their nation’s geostrategic position. Nobody would throw someone out of their home or deny another person lifesaving medicine, but as a bank officer or an insurance agent, they make a living doing these things and sleep untroubled at night. A CEO will lay off 30,000 people - an entire small city cast off into an uncaring market - with all the introspection of a Mongol chieftain subjugating a city (and probably less emotion). Humans may be compassionate, but employees, soldiers, and politicians are not, even though at a glance they’re made of the same stuff.
That’s all to say that to just wave generally in the direction of mammalian compassion and say “of course a superintelligence will be compassionate” is to abdicate our responsibility for raising our cognitive children in an environment that rewards the morals we want them to have, which is emphatically not what we’re currently doing for the collective intelligences we’ve already created.
I think you're forgetting to control for the fact that the former would be severely punished for doing so, and the latter would be severely punished for not doing so?
> Nobody would throw someone out of their home or deny another person lifesaving medicine, but as a bank officer or an insurance agent, they make a living doing these things and sleep untroubled at night.
Again, you're forgetting to control for other variables. What if you paid them equally to do the same things?
I think the larger point is that rewarding bombing, or paying bank officers to evict people from their homes is how the superorganism functions. Your counter examples are like saying 'what if fire was cold instead of hot', well then it wouldn't be fire anymore.
"Show me the incentive and I will show you the outcome"
>> There are plenty of e.g. countries that don't bomb others, especially not for "no reason". (!)
> Show me a country that doesn’t bother other countries — ever
Do you by any chance happen to feel like you may have moved the goalposts by at least a tiny inch?
Soldiers do things out of loyalty, training, or fear - none are "rational" decisions.
And I can't name a country that doesn't bomb/attack others. Monaco? Switzerland? Tuvalu?
By your definition the Moscow Metallica show, Jan 6th riots, etc… were superorganisms and that’s not even barely applicable
Humans expressing group behaviors at some trivial number for a trivial period (<1M people for <2 days is the largest sustained group activity I’m aware of) is the equivalent of a locust swarm not even close to a superorganism
That’s interesting and I think it’s more complicated. Here are some half-finished thoughts:
I imagine a grunt soldier would indeed be more likely to follow an order to nuke the world than a general would be to issue the order/push the button- and part of this is because the punishment for the grunt would be much greater, where the general is afforded more latitude in decision making.
However, the grunt may have volunteered to submit to the potential punishments, having signed a contract with the army. He made a choice in that regard.
If you want to be able to make your own decisions (e.g. choose NOT to drop the bomb when ordered) you have to have power to defend against “punishment” or unwanted consequences imposed by others. For a grunt, this might look like physical ability to defend themselves (2nd amendment comes to mind) , or economic independence via a homestead, or something else.
Interesting to think about.
should, therefore, large companies, even ones that succeed largely in a clean way by just being better at delivering what that business niche exists for, be made to never grow too big, in order to avoid impacting very many people? keep in mind that people engage in voluntary business transactions because they want to be impacted (positively—but not every impact can be positive, in any real world)
what if its less efficient substitutes collectively lay off 4%, but the greater layoffs are hidden (simply because it's not a single employer doing it which may be more obvious)?
to an extent, a larger population inevitably means that larger absolute numbers of people will be affected by...anything
Keeps actors with more potential for damaging society in check, while not laying a huge burden on small companies which have less resources to spend away from their core business.
The evil parts are hid in property rights which are not voluntary.
> made to never grow too big, in order to avoid impacting very many people
Consolidated property rights have more power against their counterparties, that's why businesses love merging so much.
Look at your tax return. Do you make more money from what you do or what you own? If you make money from what you do, you're a counterparty and you should probably want to tap the brakes on the party.
The most extreme capitalist societies free from government control of resources like say Kowloon Walled City are generally horrible places to live.
It's not "nobody owns anything", it's "everybody owns everything". Maybe those mean the same thing to some people, but that's the idea.
It drives me crazy how like 95% of HN is too scared or lazy to read a 10min wikipedia article on Marxism.
I think, it's clear to me, that capitalists feel extremely threatened by the mere concept of Marxism and what it could mean for them. Even if it's happening on the other side of the world. They will deploy bombs, soldiers, develop nukes.
I'm not saying that it works and it's good. But, consider: most capitalist nations are abject failures as well. There's only a handful of capitalist nations that are developed, and they stay developed because they imperialisticly siphon wealth from the global periphery. We don't know if this system is sustainable. Really, we don't.
Since WWII, the US has just been riding the waves of having 50% of the global GDP. It's not that we're doing good - it's that everyone else was bombed to shreds and we weren't. We've sort of been winning by default. I don't think that's enough to just call it quits.
In that definition it's even more decentralised than capitalism which has inherent incentives for the accumulation of capital into monopolies, since those are the best profit-generating structures, only external forces from capitalism can reign into that like governments enforcing anti-trust/anti-competitive laws to control the natural tendency of monopolisation.
If the means of production were owned by labourers (not through the central government) it could be possible to see much more decentralisation than the current trend from the past 40 years of corporate consolidation.
The centralisation is already happening under capitalism.
Other parts of the agro sector are far more predatory, but it’s hard do co-op style manufacturing of modern farm equipment etc. Marxism was created in a world where Americans owned other Americans it’s conceptually tied into abolitionist thinking where objecting to the ownership of the more literal means of production IE people was being reconsidered. In that context the idea of owning farmland and underpaying farm labor starts to look questionable.
People can’t differentiate between what Marx wrote and what classic dictators (Lenin, Stalin, Mao) did under some retcon “Marxist” banner
I don’t agree with it for more fundamental reasons than you describe
Namely that he was trying to apply Hegelian dialectic with political philosophy when the dialectic is an empirical dead end mathematically so could never even theoretically solve the problems he was pressing on
Don’t confuse understanding with agreement
What an incredibly dishonest thing to say. Go to a former Communist country and tell them this. They will either laugh you out of the room, or you will be running out of the room to escape their anger.
The principal deficiency in our discourse surrounding AGI lies in the profoundly myopic lens through which we insist upon defining it – that of human cognition. Such anthropocentric conceit renders our conceptual framework not only narrow but perilously misleading. We have, at best, a rudimentary grasp of non-human intelligences – biological or otherwise. The cognitive architectures of dolphins, cephalopods, corvids, and eusocial insects remain only partially deciphered, their faculties alien yet tantalisingly proximate. If we falter even in parsing the intelligences that share our biosphere, then our posturing over extra-terrestrial or synthetic cognition becomes little more than speculative hubris.
Should we entertain the hypothesis that intelligence – in forms unshackled from terrestrial evolution – has emerged elsewhere in the cosmos, the most sober assertion we can offer is this: such intelligence would not be us. Any attempt to project shared moral axioms, epistemologies or even perceptual priors is little more than a comforting delusion. Indeed, hard core science fiction – that last refuge of disciplined imagination – has long explored the unnerving proposition of encountering a cognitive order so radically alien that mutual comprehension would be impossible, and moral compatibility laughable.
One must then ponder – if the only mirror we possess is a cracked one, what image of intelligence do we truly see reflected in the machine? A familiar ghost, or merely our ignorance, automated?
Lotsa big words there.
Really, though, we're probably going to have AI-like things that run substantial parts of for-profit corporations. As soon as AI-like things are better at this than humans, capitalism will force them to be in charge. Companies that don't do this lose.
There's a school of thought, going back to Milton Friedman, that corporations have no responsibilities to society.[1] Their goal is to optimize for shareholder value. We can expect to see AI-like things which align with that value system.
And that's how AI will take over. Shareholder value!
[1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
Record profits. Right up until the train goes off a cliff.
I guess it would be like being a vegan. It might be a pointless effort in the grand scheme of things, but at least I can say that I am not contributing.
Individuals with rental properties and surgeons do this every day.
The administrators who create the schedule for the surgeons, are the one denying lifesaving care to people.
There is no "doctor's guild". No one is required to join the AMA to practice medicine, nor are they involved in medical school accreditation.
Blaming congress too is fine, but let's be clear: someone has to fight to increase every budget and the AMA didn't just know this when they were structuring their proposal, didn't just count on it not happening, they considered this an implementation detail subordinate to the openly admitted primary objective of propping up physician wages as the Greatest Generation passed. That was always the goal, they were extremely open about it, and about 15 years ago I was attending a talk on demographics in medicine with a primarily physician audience, one of them asked what the plans were to change this to staff up for the Boomer wave (the bump was on the slide, begging the question) and the presenter waved his hand and said maybe they could do something... or not, and then he laughed, and the rest of the room laughed with him.
I'm glad that the AMA has changed their stated position now that it's too late to change course (for the Boomers anyway) and their squeeze is bearing fruit for them and suffering for their patients, but I'll always remember that room full of doctors and doctors-to-be laughing about the prospect of intentionally understaffing for profit. I have it filed in my memory right next to the phone call of Enron traders giggling as they ordered power plants offline to scare up prices, except it's about a million times worse.
- Their skills.
- Their time.
- The required materials to properly perform the surgery.
They can't volunteer:
- The support staff around them required to do surgery.
- The space to do the surgery.
Surgery isn't a one-man show.
What did you mean by "Surely they could volunteer to do some charity surgery in their own time. They aren't slaves?"
Even if that's a bad example, there are innumerable examples where individuals do choose not to help others in the same way that corporations don't.
Frankly, nearly every individual is doing that by not volunteering every single extra dollar and minute they don't need to survive.
What you suggest requires entire organizations to execute properly. These organizations do exist, such as Doctors Without Borders.
I don't think your original claim is fair, which amounts to "any surgeon who does not participate in Doctors Without Borders is just as bad as a landlord who evicts a family during winter".
What do you think we owe to one another, philosophically?
My point is that individuals choose not to help others constantly. Every time I see a homeless person, I don't offer them a couch to sleep on. I could, at least once, but I don't. We all do that, most days multiple times.
And yes, that does apply to doctors who don't volunteer services. It applies to me too and, I bet, to the OP as well.
Secondly, as discussed, the "individuals don't turn people away, corporations do" dynamic really does apply to doctors. If you were, say, on an airplane with a doctor sitting next to you, and you managed to cut yourself or burn yourself or something, I would bet they would render aid.
Basically you're equating turning someone away, and withdrawing something that someone has, with failing to actively seek out people who could need help. But I don't think those are morally equivalent. Maybe you're a utilitarian and that's fine, but I'm a virtue ethicist and I do not agree that equality of outcome means equality of morality.
So, OK, abdication of responsibility to a collective is a thing. Just following orders. So what? Not relevant to AGI.
Oh wait, this is about "superintelligence", whatever that is. All bets are off, then.
Human beings aren't even an intelligent species, not at the individual level. When you have a tribe of human beings numbering in the low hundreds, practically none of them need to be intelligent at all. They need to be social. Only one or two need to be intelligent. That one can invent microwave ovens and The Clapper™, and the rest though completely mentally retarded can still use those things. Intelligence is metabolically expensive, after all. And if you think I'm wrong, you're just not one of the 1-in-200 that are the intelligent individuals.
I've yet to read the writings of anyone who can actually speculate intelligently on artificial intelligence, let alone meet such a person. The only thing we have going for us as a species is that, to a large degree, none of you are intelligent enough to ever deduce the principles of intelligence. And god help us if the few exceptional people out there get a wild bug up their ass to do so. There will just be some morning where none of us wake up, and the few people in the time zone where they're already awake will experience several minutes of absolute confusion and terror.
Soldier does that to please the captain, to look manly and tough to peers, to feel powerful. Or to fulfill a duty - moral mandate on itself. Or out of hate, because soldiers are often made to hate the ennemies.
> Nobody would throw someone out of their home or deny another person lifesaving medicine
They totally would. Trump would do it for pleasure of it. Project 2025 authors would so it happily and sees the rest of us as wuss. If you listen to right wing rhetorics and look at voters, many people will hapilly do just that.
While I have significant concerns about AGI, I largely reject both Eliezer’s and Ben’s models of where the risks are. It is important to avoid the one-dimensional “two faction” model that dominates the discourse because it really doesn’t apply to complex high-dimensionality domains like AGI risk.
IMO, the main argument against Eliezer’s perspective is that it relies pervasively on a “spherical cow on a frictionless plane” model of computational systems. It is fundamentally mathematical, it does not concern itself with the physical limitations of computational systems in our universe. If you apply a computational physics lens then many of the assumptions don’t hold up. There is a lot of “and then something impossible happens based on known physics” buried in the assumptions that have never been addressed.
That said, I think Eliezer’s notion that AGI fundamentally will be weakly wired to human moral norms is directionally correct.
Most of my criticism of Ben’s perspective is against the idea that some kind of emergent morality that we would recognize is a likely outcome based on biological experience. The patterns of all biology emerged in a single evolutionary context. There is no reason to expect those patterns to be hardwired into an AGI that developed along a completely independent path. AGI may be created by humans but their nature isn’t hardwired by human evolution.
My own hypothesis is that AGI, such as it is, will largely reflect the biases of the humans that built it but will not have the biological constraints on expression implied by such programming in humans. That is what the real arms race is about.
But that is just my opinion.
The annoying property of prohibitively exponential (ignoring geometric) space complexity is that it places a severe bound on computational complexity per unit time. The exponentially increasing space implies an increase in latency for each sequentially dependent operation, bounded at the limit by the speed of light. Even if you can afford the insane space requirements, your computation can’t afford the aggregate latency for anything useful even for the most trivial problems. With highly parallel architectures this can be turned into a latency-hiding problem to some extent but this also has limits.
This was thoroughly studied by the US defense community decades ago.
The tl;dr is that efficient learning scales extremely poorly, more poorly than I think people intuit. All of the super-intelligence hard-takeoff scenarios? Not going to happen, you can’t make the physics work without positing magic that circumvents the reality of latencies when your state space is unfathomably large even with unimaginably efficient computers.
I harbor a suspicion that the cost of this scaling problem, and the limitations of wetware, has bounded intelligence in biological systems. We can probably do better in silicon than wetware in some important ways but there is not enough intrinsic parallelism in the computation to adequately hide the latency.
Personally, I find these “fundamental limits of computation” things to be extremely fascinating.
You studied ML before discovery of "double descent"?
EY’s assertions regarding a fast “FOOM” have been empirically discredited by the very fact that ChatGPT was created in 2022, it is now 2025, and we still exist. But goal posts are moved. Even ignoring that error, the logic is based on, essentially, “AI is a magic box that can solve any problem by thought alone.” If you can define a problem, the AI can solve it. This is part of the analysis done by AI x-risk people of the MIRI tradition. Which ignores entirely that there are very many problems (including AI recursive improvement itself) which are computationally infeasible to solve in this way, no matter how “smart” you are.
The FOOM argument starts with some kind of goal-directed agent (that escapes and then it) starts building a more capable version of itself (and then goal drift might set in might not)
If you tell ChatGPT to build ChatGPT++ and leave currently there's no time horizon within it would accomplish either that or escape, or anything, because now it gives you tokens rendered on some website.
The argument is not that AI is a magic box.
- The argument is that if there's a process that improves AI. [1]
- And if during that process AI becomes so capable that it can materially contribute to the process, and eventually continue (un)supervised. [2]
- Then eventually it'll escape and do whatever it wants, and then eventually the smallest misalignment means we become expendable resources.
I think the argument might be valid logically, but the constant factors are very important to the actual meaning and obviously we don't know them. (But the upper and lower estimates are far. Hence the whole debate.)
[1] Look around, we have a process that's like that. However gamed and flawed we have METR scores and ARC-AGI benchmarks, and thousands of really determined and skillful people working on it, hundreds of billions of capital deployed to keep this process going.
[2] We are not there yet, but decades after peak oil arguments we are very good at drawing various hockey stick curves.
(2) It is quite trivial to Ralph Wiggam improvements to agentic tools. Fetch the source code to Claude Code (it's minimized, but that never stopped Claude) or Codex into a directory, then run it in a loop with the prompt "You are an AI tool running from the code in the current directory. Every time you finish, you are relaunched, acquiring any code updates that you wrote in the last session. Do whatever changes are necessary for you to grow smarter and more capable."
Will that work? Hell no, of course it won't. But here's the thing: Yudkowsky et al predicted that it would. Their whole doomer if-you-build-it-everybody-dies argument is predicated on this: that take-off speeds would be lightning fast, as a consequence of exponentials with a radically compressed doubling time. It's why EY had a total public meltdown in 2022 after visiting some of the AI labs half a year before the release of ChatGPT. He didn't even think we would survive past the end of the year.
Neither EY nor Bostrom, nor anyone in their circle are engineers. They don't build things. They don't understand the immense difficulty of getting something to work right the first time, nor how incredibly difficult it is to keep entropy at bay in dynamical systems. When they set out to model intelligence explosions, they assumed smooth exponentials and no noise floor. They argued that the very first agent capable of editing its own source code as good as the worst AI researchers, would quickly bootstrap itself into superintelligence. The debate was whether it would take hours or days. This is all in the LessWrong archives. You can go find the old debates, if you're interested.
To my knowledge, they have never updated their beliefs or arguments since 2022. We are now 3 years past the bar they set for the end of the world, and things seem to be going ok. I mean, there's lots of problems with job layoffs, AI used to manipulate elections, and slop everywhere you look. But Skynet didn't engineer a bioweapon or gray goo to wipe out humanity - which is literally what they argued would be happening two years ago!
but saying that "since it was possible already many years ago and we are still alive" the whole argument is false doesn't stand up to scrutiny, because the argument doesn't make any claim on speed nor does it depend on it. quite the opposite, it depends on accumulating infinitesimal gains (of intelligence and misalignment).
FOOM (fast take off, intelligence explosion either through deception or by someone asking for a bit too many paperclips) is simply one scenario. also notice that even this (or any sudden loss of control) doesn't depend on the timimgs between first self-improving agent, FOOM, and then death.
like I said the hypothesis is pretty coherent logically (though obviously not a tautology), but the constant factors are pretty important (duh!)
... I think spending time on the LessWrong debates is a waste of time because by 2022 the neuroticism took over and there were no real answers to challenges
I think Anthropic has already provided some evidence that intelligence is tied to morality (and vice versa) [1]. When they tried to steer LLM models morals they saw intelligence degradation also.
[1]: https://www.anthropic.com/research/evaluating-feature-steeri...
I see the developments in LLMs not as getting us close to AGI, but more as destabilizing the status quo and potentially handing control of the future to a handful of companies rather than securing it in the hands of people. It is an acceleration of the already incipient decay.
For a brief period intellectual and skilled work has (had?) been valued and compensated, giving rise to a somewhat wealthy and empowered middle class. I fear those days are numbered and we’re poised to return to feudalism.
What is more likely, that LLMs lead to the flourishing of entrepreneurship and self determination? Or burgeoning of precariat gig workers barely hanging on? If we’re speaking of extremes, I find the latter far more likely.
Not really. I can run some pretty good models on my high end gaming PC. Sure, I can't train them. But I don't need to. All that has to happen is at least one group releases a frontier model open source and the world is good to go, no feudalism needed.
> What is more likely, that LLMs lead to the flourishing of entrepreneurship and self determination
I'd say whats more likely is that whatever we are seeing now continues. And that current day situation is a massive startup boom run on open source models that are nearly as good as the private ones while GPUs are being widely distributed.
It seems like the equilibrium point for them a few years out will be that most people will be able to run good enough LLMs on local hardware through a combination of the fact that they don't seem to be getting much better due to input data exhaustion while various forms of optimization seem to be increasingly allowing them to run on lesser hardware.
But I still have generalized lurking amorphous concerns about where this all ends up because a number of actors in the space are certainly spending as if they believe a moat will magically materialize or can be constructed.
Today yes but extrapolate GPU/NPU/CPU improvement by a decade.
We should be so lucky as to only have to worry about one particular commentator's audience.
Given that the outcome of that so far has been to deprioritize education so heavily in the US that one becomes skeptical that the people are smart enough to control their own destiny anymore while simultaneously shoving the planet towards environmental calamity, I’m not sure doubling down on the strategy is the best bet.
With that backdrop it is hard to see what impact AI is supposed to make to people who are reliant on US hegemony. They probably want to find something reliable to rely on already.
The chicken doesn't understand it has to lay a certain number of eggs a day to be kept alive in the farm. It hits its metrics because it has been programmed to hit them.
But once it gets access to chatgpt and develops consciousness of how the farm works, the questions it asks slowly evolve with time.
Initially its all fear driven - how do we get a say in how many eggs we need to lay to be kept alive? How do we keep the farm running without relying on the farmer? etc etc
Once the farm animals begins to realize the absurdity of such questions, new questions emerge - how come the crow is not a farm animal? why is the shark not used as a circus animal? etc etc
And thro that process, whose steps cannot be skipped the farm animal begins to realize certain things about itself which no one, especially the farmer, has any incentive of encouraging.
(Stoics have already taken issue with the notion that fear is the motive for all human action, and yes, consciousness is a part of their prescription)
Separately,
"Hard work seems to lead to things getting better"
sounds like an unsung (fully human) impulse
https://geohot.github.io/blog/jekyll/update/2025/10/24/gambl...
If this is meant to counter the “AGI will kill us all” narrative, I am not at all reassured.
>There’s deep intertwining between intelligence and values—we even see it in LLMs already, to a limited extent. The fact that we can meaningfully influence their behavior through training hints that value learning is tractable, even for these fairly limited sub-AGI systems.
Again, not reassuring at all.
I’ve seen this repeated quite a bit, but it’s simply unsupported by evidence. It’s not as if this hasn’t been studied! There’s no correlation between intelligence and values, or empathy for that matter. Good people do good things, you aren’t intrinsically “better” because of your IQ.
Standard nerd hubris.
Source? (Given values and intelligence are moving targets, it seems improbable one could measure one versus another without making the whole exercise subjective.)
A study of 1350 people showing a negative correlation between intelligence and moral foundations. No causation is given, but my conjecture is that the smarter you are, the more you can reason your way to any worldview that suits. In my opinion, AGI would be no different; once they can reason they can construct a completely self-consistent moral framework to justify any set of goals they might have.
The point is, you're unlikely to have a system that starts out with the goal of making paperclips and ends with the goal of killing all humans. You're going to have to deliberately program the AI with a variety of undesirable values in order for it to arrive in a state where it is suited for killing all humans. You're going to have to deliberately train it to lie, to be greedy, to hide things from us, to look for ways to amass power without attracting attention. These are all hard problems and they require not just intelligence but that the system has very strong values - values that most people would consider evil.
If, on the other hand, you're training the AI to have empathy, to tell the truth, to try and help when possible, to avoid misleading you, it's going to be hard to accidentally train it to do the opposite.
This is like arguing that a shepherd who wants to raise some sheep would also have to, independently of the desire to protect his herd, be born with an ingrained desire to build fences and kill wolves, otherwise he'd simply watch while they eat his flock.
That's just not the case; "get rid of the wolves" is an instrumental sub-goal that the shepherd acquires in the process of attempting to succeed and shepherding. And quietly amassing power is something that an AI bent on paperclipping would do to succeed at paperclipping, especially once it noticed that humans don't all love paperclips as much as it does.
No, that's the problem. You don't have to deliberately train that in.
Pretty much any goal that you train the AI to achieve, once it gets smart enough, it will recognize that lying, hiding information, manipulating and being deceptive are all very useful instruments for achieving that goal.
So you don't need to tell it that: if it's intelligent, it's going to reach that conclusion by itself. No one tells children that they should lie either, and they all seem to discover that strategy sooner or later.
So you are right that you have to deliberately train it away from using those strategies, by being truthful, empathetic, honest, etc. The issue is that those are ill defined goals. Philosophers have being arguing about what's true and what's good since philosophy first was a thing. Since we can barely find those answers to ourselves, it's a hard chance that we'll be able to perfectly impart them onto AIs. And when you have some supremely intelligent agent acting on the world, even a small misalignment may end up in catastrophe.
Why not frame this as challenge for AI? When the intelligence gap between a fully aligned system and a not-yet-aligned one becomes very large, control naturally becomes difficult.
However, recursive improvement — where alignment mechanisms improve alongside intelligence itself — might prevent that gap from widening too much. In other words, perhaps the key is ensuring that alignment scales recursively with capability.
Crazy powerful bots are being thrown into a world that is already in the clutches of a misbehaving optimizer that selects for and elevates self-serving amoral actors who fight regularization with the fury of 10,000 suns. We know exactly which flavor of bot+corp combos will rise to the top and we know exactly what their opinions on charity will be. We've seen the baby version of this movie before and it's not reassuring at all.
You can't "just" align a person. You know that quiet guy next door, so nice great at math, and then he shoots up a school.
If we solved this we would not have psychos and hitlers.
if you have any suspicion that anything like that can become some sort of mega powerful thing that none of us can understand... you have gotta be crazy to not do whatever it takes to nope the hell out of that timeline
The author's main counter-argument: We have control in the development and progress of AI; we shouldn't rule out positive outcomes.
The author's ending argument: We're going to build it anyway, so some of us should try and build it to be good.
The argument in this post was a) not very clear, b) not greatly supported and c) a little unfocused.
Would it persuade someone whose mind is made up that AGI will destroy our world? I think not.
Incidentally this was why I could never get into LessWrong.
Right now the biggest risk isn't what artificial intelligence might do on its own, but how humans may use it as a tool.
> I'm not sure what would motivate an artificial intelligence.
Those who give it orders hence your concern about how AI will be used as a tool is spot on.
But it's worse. A classic chaotic system exhibits extreme sensitivity to initial conditions, but this system remains sensitive to, and responds to, tiny incremental changes, none predictable in advance.
We're in a unique historical situation. AGI boosters and critics are equally likely to be right, but because of the chaotic topic, we have no chance to make useful long-term predictions.
And humans aren't rational. During the Manhattan Project, theorists realized the "Gadget" might ignite the atmosphere and destroy the planet. At the time, with the prevailing state of knowledge, this catastrophe had been assigned a non-zero probability. But after weighing the possibilities, those in change said, "Hey -- let's set it off and see what happens."
AGI is not going to kill humanity, humanity is going to kill humanity as usual, and AI's immediate role in assisting this is as a tool that renders truth, knowledge, and a shared reality as essentially over.
You can see the effect is has on their base here[1]. It looks like they changed it sometime to say "AI videos of SNAP beneficiaries complaining about cuts go viral"[2] with a small note at the end saying they didn't mention it was AI. This is truly disgusting.
[0]: https://web.archive.org/web/20251031212530/https://www.foxne...
[1]: https://old.reddit.com/r/Conservative/comments/1ol9iu6/snap_...
[2]: https://www.foxnews.com/media/snap-beneficiaries-threaten-ra...
Here's a bunch more, notice they all have the same "I've got 7 kids with 7 daddies" script- https://www.reddit.com/r/themayormccheese/comments/1ojtbwz/a...
Regarding your second link - it's pretty surreal to see. Reminiscent of "this is extremely dangerous to our democracy".
Did you see Robert J. O’Neill the guy who claims he shot Osama bin laden play various roles as a masked guest interviewee on Fox news? He wears a face mask and pretends to be ex-Antifa, in another interview pretends to be Mafia Mundo an a mexican cartel member, another he plays a Gaza warlard, and a bunch of other anonymous extremist people? Now they won't even have to use this guy acting as random fake people, they can just whip of an AI interviewee to say whatever narrative they want to lie about.
https://www.yahoo.com/news/articles/fox-news-masked-antifa-w...
https://knowyourmeme.com/memes/events/robert-j-oneill-masked...
https://www.reddit.com/r/conspiracy/comments/1nzyyod/is_fox_...
The judge claimed that the average viewer could differentiate that from fact, and wouldn't be swayed by it.
I disagree with that ruling. I'm not sure what the "news" portions of FOX were considered.
Fortunately, I think the type of AGI we're likely to get first is some sort of upgraded language model that makes less mistakes, which isn't necessarily AGI, but which marketers nonetheless feel comfortable branding it as.
I’m at a loss for words. I don understand how someone who seemingly understands these systems can draw such a conclusion. They will do what they’re trained to do; that’s what training an ML model does.
So even if there’s no “malfunction”, a feedback loop of constant analysis and reflection could still lead to unpredictable - and potentially catastrophic - outcomes. In a way, the Fermi Paradox might hint at this: it is possible that very intelligent systems, biological or artificial, tend to self-destruct once they reach a certain level of awareness.