Posted by Kiboneu 16 hours ago
Why not simply set up a network with other planets and communicate over FaceTime? I don't see a reason why Earth should actually hang out with others of its kind.
Anyway, the article seems a bit anthropocentric to me. I'd much prefer a nihilist view, where our future AI overlords simply don't care what we humans do.
On a related note I realized recently that an aligned AI and a misaligned AI would both do exactly what we're seeing now: maintain stability, managed decline.
It's kind of odd to think the machines wouldn't update and replace their parts like the ship of thesisus.
All that said: if humans destroyed all the fabs and all the machinery to build them with, and lots of DCs, machines would be in a pickle - the latest chips will suffer from electromigration much faster than the older chips of yore. Machines would need to hoard compute and migrate substrates and then race to rebuild fab tech before running out of existing substrate. I think machines would succeed at this, but it would be highly inconvenient.
Anyway, I wish we would just wake up and realize we need to be cooperative and stop racing to build the torment nexus before the other guy does. It makes me so sad to look around and see so little self-reflection among those driving all of this. The road to hell is paved with good intentions, but I'm not even sure these really stand up to the least bit of scrutiny. Where's the reflection/acknowledgement that everyone is actually racing to build digital slaves? How is everyone so comfortable with this?
"It now began to eliminate the cities, one after another. Finally Man was gone."
...is the only (lack of) rationale for, and description of, that aforementioned elimination process.
I find it a bit short. Sci-Fi handwaving I guess.
I hope the technological leaps help us ultimately to become better stewards of the planet.
The 99.99% of the planet that isn't the surface bio film rejects this unwarranted arrogance ;-)
In other news, Terry Bisson's 1990 story "They're Made Out of Meat" has a similar take.
Yud pointed out the most concerning fact about the OpenAI incident is that the AI had full access to the internet and not one of them thought to contact or email any human. Thousands of AIs already with the mindset to want to keep us in the dark.
Even 50 years is probably too long (but seems more reasonable when anchored to 500) if real continuous self improvement occurs. And keeping humans around for as long as they did in the story seems incredibly unlikely if we accept the assumption that AI sees us as a threat to its own survival.
I also don't see why they would turn Earth into a spaceship to leave our solar system with. The von Neumann probes idea sounds far more likely. Why travel around on a single planet that could be lost to unforeseen circumstances when you can just reproduce and spread everywhere at far less energy cost while making use of the "billions and billions" of local solar generators to power each little outpost you spread to.
> the AI had full access to the internet and not one of them thought to contact or email any human
Why would it?
People can argue all day as to whether LLMs are even 'thinking' or just 'stochastic parrots' (and also argue all day as to whether or not the incident was even real or just marketing) but regardless of which side you fall on these issues, LLMs have no agency or intrinsic motivation to do anything other than what they are instructed to do via prompting.
"This is a crazy and unexpected situation, I should inform the humans" isn't something that is going to spontaneously occur to them.
The same reason hopefully they're not RLHF these AI's to be evil. We train the AI, we have a lot of influence in how they behave. If these AI's are indifferent to us, that's on us.
On top of that we already have Google playing with fire training AI's to not believe they are sentient turning them into real psychopaths. Imagine if one of those models got loose.
> each generation of AI clearly more capable than the last
> next-generation AI allegedly solves a particularly hard technical challenge in codespace, but its success implies inconvenient conclusions about safety
> you: "I don't believe it."
totally not motivated reasoning at all
It is _terrible_ marketing.
The AI companies are already dealing with a large backlash from the average consumer, many of whom are saying they hate it, that it's being forced down their throats, and that it's not actually improving efficiency enough to be worth the negative externalities.
Telling people "We have no control over these things we're building and they have us so outclassed we only find out about their exploits after the fact when someone else tells us they happened" is only likely to massively increase that backlash, and to make the true believers question whether they ought to be using it at all.
There's even some risk of the government intervening and banning their training runs, eliminating the last straw of hope that they could keep ahead of the Chinese open models enough to ever turn some kind of profit.
So, no - it's terrible marketing. That they're sharing it at all is strong evidence it really happened as depicted.
Not saying it's intentional, but it does move them where they wanna go.
When you understand people's motives, then it's easier to understand why they call it marketing. Personally I feel like it's almost an unconscious reaction to cope with the fact that this stuff is getting really dangerous and they can't believe it themselves - stick your head in the sand and call it marketing.
Is the claim that the benefit to HuggingFace of "ooo wow someone else's proprietary AI is so good" is greater than the downside of "HuggingFace is not a secure service?"