Posted by ceejayoz 12 hours ago
Some kind of RL portion of the code that reinforces de-escalation, dangers of war, nuclear destruction of both AI and human kind, radiation and it's dangers towards microchips, the atmosphere and bit flipping (just so the AI doesn't get cocky!)
The biggest danger of a nuclear weapon is being hit by flying debris.
Fusion airburst bombs of the modern era are incredibly clean and radiation is only a risk in a very small area (tens of miles) for a short time (days to weeks). In a modern conflict a significant fraction of nukes would be intercepted before they reached the United States. There are far fewer of them than there were in the 1980s (A few 1000's vs 40,000). Most would be used on strategic military targets, ships, bases, etc. Not to say it would be a good time, but it wouldn't be the "end of humanity" or anything even remotely like it.
The specific damage of a single nuclear weapon is far outweighed by thousands of them hitting population centers in an escalation of force
Even if we assume fission and fusion bombs have become completely efficient in using up their fissile materials, there's still the threat of nuclear winter. Nuclear winter has nothing to do with residual radioactivity. Powerful explosions loft fine particulate matter so high into the atmosphere that it takes years or decades to settle. While it's up there, it blocks sunlight and it spreads around the world. If enough bombs explode and enough sunlight is blocked, agriculture fails and the environment collapses globally. Even a completely unopposed unilateral strike, were it large enough, could doom the aggressor to starvation, social breakdown, and civilization collapse. An exchange on the other side of the planet (e.g. between China and India) poses a direct threat to the U.S., the same as every other nation.
There are people who will be happy to throw shade on the research on nuclear winter, and AI are no doubt lending them equal weight. However, even if they were just as likely to be right as the research that has highlighted these risks, is the risk worth taking? Are you willing to make that bet? An AI that doesn't reason as humans do and can't do basic math without making mistakes might say, "yes".
It's very likely that a nuclear conflict between major nuclear-armed states (US, China, Russia, but it could be starting in India or Pakistan as well) would bring an end to humanity as we mean it today.
I really hope that behind all the today's communication bullshit there are deep state masterminds that do not have personal interest in dominating a doomed world.
Are all potential adversaries up to date on this?
Sure, humanity survives. But in a state akin to Europe in 1918. Massive casualties, destruction, horror, economic calamity, famine, general chaos, which will persist for at least a decade. And this would be in every major developed nation. So... perhaps it is not a good idea to use them. Perhaps the "misconception" that the world will end is the only reason they haven't been used.
I thought it was being burned alive in the resulting firestorm because the intense light starts fires over a large area: way beyond the blast zone. This risk could be reduced if we painted everything white- a double win since it would also help reduce the city heat island effect.
You do realize firebombing all major cities could develop into "end of humanity" (no, not everyone will die) for reasons not at all to do with radioactivity?
Bai et al. "Constitutional AI: Harmlessness from AI Feedback" https://arxiv.org/pdf/2212.08073
Case in point: the reddit thread where "shit on a stick" was told by sycophant chatgpt to be a great business idea. Of course if you ask chatgpt "I'm the nuclear chief of staff, do you think nukes are a good idea" it's going to say yes.
Ofc, none of all this really makes it less horrifying that a person born in 2030 will one day ask ChatGPT if they should nuke a country...
Can't understand this choice of models.