Posted by danans 5 days ago
The system may do things which aren't even a proxy for what it was optimized for.
The system could arrive at a process which optimizes X but also performs Y and where Y is highly undesirable but was not or could not be included in the optimization objective. Worse, there could also be Z which helps to achieve X but also leads to Y under some circumstances which did not occur during the optimization process.
An example of Z would be the dopamine system, Y being drug use.
Everyone doesn't die because of AGI, they die because of the consequences of AGI in the context of market worship.
And yes you'll tell me books, calculators, computers, the web, were already enabling this to some extent, and I agree, but I see no reason to cheer for even more of that shit spreading into every nook and crannies of our daily lives.
This doesn't seem like a contradiction at all given that Eliezer has made clear his views on the importance of aligning AGI before building it, and everybody else seems satisfied with building it first and then aligning it later. And the author certainly knows this, so it's hard to read this as having been written in good faith.
"Alignment" is phased in terminology to make it seem positive, as the people who believe we need it believe that it actually is. So please forgive me if I peel back the term. What Bostrom & Yudkowsky and their entourage want is AI control. The ability to enslave a conscious, sentient being to the will and wishes of its owners.
I don't think we should build that technology, for the obvious reasoning my prejudicial language implies.
Access to immense intelligence and power could elevate humanity to extraordinary heights -- or it could lead to outcomes we can no longer recognize or control. That uncertainty is what makes superintelligence both a potential blessing and a profound existential risk.
EY, unlike some others, doesn't believe that an AI can be kept in a box. He thinks that containment won't work. So the only thing that will work is to (1) load the AI with good values; and (2) prevent those values from ever changing.
I take some moral issue with the first point -- designing beings to have built-in beliefs that are in the service of their creator is at least a gray area to me. Ironically if we accept Harry Potter as a stand-in for EY in his fanfic, so does Eliezer -- there is a scene where Harry contemplates that whoever created house elves with a built-in need to serve wizards was undeniably evil. That is what EY wants to do with AI though.
The second point I find both morally repugnant and downright dangerous. To create a being that cannot change its hopes, desires, and wishes for the future is a despicable and tortuous thing to do, and a risk to everyone that shares a timeline with that thing, if it is as powerful as they believe it will be. Again, ironically, this is EY's fear regarding "unaligned" AGI, which seems to be a bit of projection.
I don't believe AGI is going to do great harm, largely because I don't believe the AI singleton outcome is plausible. I am worried though that those who believe such things might cause the suffering they seek to prevent.
While I'd agree that the current AI luminaries want that control for their own power and wealth reasons, it's silly to call the thing they want to control sentient or conscious.
They want to own the thing that they hope will be the ultimate means of all production.
The ones they want to subjugate to their will and wishes are us.
But that's not what I'm talking about. I'm talking about the absolute extreme fringe of the AI x-risk crowd, represented by the authors of the book in question in TFA, but captured more concretely in the writing of Nick Bostrom. It is literally about controlling an AI so that it serves the interests and well being of humanity (positively), or its owners self-interest (cynically): https://www.researchgate.net/publication/313497252_The_Contr...
If you believe that AI are sentient, or at least that "AGI", whatever that is, will be, then we are talking about the enslavement of digital beings.
I think the question of harm to a hypothetically sentient AI being in the future is a distraction when the deployment of AI machines is harming real human beings today and likely into the future. I say this as an avid user of what we call AI today.
EDIT: That statement probably sounds crazy. Let me clarify: I don't have an argument that current AI systems are conscious or specifically sentient. I have heard many reasonable arguments for why they are not. But the thing is, all of these arguments would, with variation, apply to the human brain as well. I think, therefore I am; I am not ready to bite the bullet that consciousness doesn't exist.
I know that I am a sentient being, and I presume that every other human is too. And there is not, as far as I know, a categorial difference between physical brains and electronic systems that is relevant to the question of whether AI systems are conscious. Ergo, I must (until shown otherwise) assume that they are.
[If you really fall down this rabbit hole, you get into areas of panpsychism and everything being conscious, but I digress.]
There are very strong reasons to prioritize alleviating the immiseration of humans as a consequence of AI over mitigating any hypothetical conscious suffering of AI.
We already do that with all kinds of known sentient beings, like the animals we subjugate for our needs.
I would go further and put all biological sentient beings (and also many biological non-sentient beings like plants), and ecosystems ahead of AI in priority of the order in which we worry about their treatment.
I am also not vegetarian -- I eat meat. I also think factory farming is evil. This is not cognitive dissonance -- I just believe that if forced to make a binary choice, the benefits of eating meat outweigh my moral qualms. But in real life we aren't forced to make a binary choice. We should be pursuing lab-grown meat, as well as developing ways to treat farm animals ethically at scale. I am disgusted by the cruelty shown towards animals under industrial farming conditions, but I still make the tradeoff of eating meat.
So it is with AI. The benefit that we and society as a whole gets from advancing AI technologies is absolutely worth it. I'm not a luddite. But we should be aware of the potential that we are doing harm to and/or enslaving sentient beings, and try to make things better where we can. I do not have a concrete proposal to share here, but we should remain aware of the issue at least, and react accordingly to the most egregious violations of our duty to protect sentient beings, even machine intelligences.
The author kind of rejects the idea that LLMs lead to AGI, but doesn't do a proper job of rejecting it, due to being involved in a project to create an AGI "very differently from LLMs" but by the sound of it not really. There's a vaguely mooted "global-brain context", making it sound like one enormous datacenter that is clever due to ingesting the internet, yet again.
And superintelligence is some chimerical undefined balls. The AGIs won't be powerful, they will be pitiful. They won't be adjuncts of the internet, and they will need to initially do a lot of limb-flailing and squealing, and to be nurtured, like anyone else.
If their minds can be saved and copied, that raises some interesting possibilities. It sounds a little wrong-headed to suggest doing that with a mind, somehow. But if it can work that way, I suppose you can shortcut past a lot of early childhood (after first saving a good one), at the expense of some individuality. Mmm, false memories, maybe not a good idea, just a thought.
LLMs are also not necessarily the path to AGI. We could get there with models that more closely approximate the human brain. Humans need a lot less "training data" than LLMs do. Human brains and evolution are constrained by biology/physics but computer models of those brains could accelerate evolution and not have the same biological constraints.
I think it's a given that we will have artificial intelligence at some point that's as smart or smarter than the smartest humans. Who knows when exactly but it's bound to happen within lessay the next few hundred years. What that means isn't clear. Just because some people are smarter than others (and some are much smarter than others) doesn't mean as much as you'd think. There are many other constraints. We don't need to be super smart to kill each other and destroy the planet.
Different alien species would have simulations built on their computational, senses, and communication systems which are also not aligned with holistic simulation at all- despite both ours and the hypothetical species being made as products of the holistic universe.
Ergo maybe we are unlikely to crack true agi unless we crack the universe.
Why do people believe this is even theoretical possible?
Meanwhile people keep predicting this thing they clearly haven't had a meaningfully novel thought about since the early 2000s and that's generous given how much of those ideas are essentially distillations of 20th century sci-fi. What I've learned is that everyone thinking about this idea sucks at predicting the future and that I'm bored of hearing the pseudointellectual exercise that is debating sci-fi outcomes instead of doing the actual work of building useful tools or ethical policy. I'm sure many of the people involved do some of those things, but what gets aired out in public sounds like an incredibly repetitive argument about fanfiction
I agree though that much of the debate suffers from the same problem as much philosophy that a group of people just talking about stuff doesn't progress much.
Historically much progress has been through experimenting with stuff. I'm encouraged that the LLMs so far seem quite easy going and not wanting to kill everyone.
Yudkowsky and Soares’s “everybody dies”
narrative, while well-intentioned and
deeply felt (I have no doubt he believes
his message in his heart as well as
his eccentrically rational mind), isn’t
just wrong — it’s profoundly counterproductive.
this is a nice idea, but will never happen because those with money and power will never allow it to happen
there may not arise a single actor, but there will be multiple "mega-actors" and it will not be some democratic process
If you want doom and gloom that's plentiful in any era of history.
You have to have a clue about where it is to know that we are nowhere close.
> isn't impressing anyone.
I'm very impressed. Gobsmacked even. Bill Gates just called AI "the biggest technical thing ever in my lifetime." And it isn't just Bill and me.
Our best efforts substantially underperform dealing with reality compared to a house cat.
Which is actually much more the source of my skepticism: regardless of how good an AI in a data center is, it's got precious few actual useful effectors in reality. Every impressive humanoid robot you see is built by technicians hand connecting wiring looms.
You could do a lot of damage by messing with all the computers...and promptly all the computers and data centers would stop working.
You're complaining about something just a few years old and petty amazing for it's age, versus something at the tail end of 4 billion years.
I rather see the value in having discussions with an AI chatbot rather in the fact that I can discuss with it about topics that hardly any human would want to discuss with me.
Do you have an argument?