Posted by toomuchtodo 6 hours ago
Florida should be added to title, it's not like the European shutdowns for overheating
Some rods dropped. Something went wrong with the rod control system. Maybe an errant trip, maybe something mechanically wore, etc.
When a few rods dropped "accidentally" into the reactor, humans pushed the button to shut it down.
Now they figure that out and then turn the reactor back on.
Unironically this is the kind of headline I’m expecting in 2040 about one or more major AGI systems running an electrical grid, factory or similar.
what precisely is your point of describing how quality production engineering for infrastructure works?
"Nearly" is doing a lot of heavy lifting there. Those "nearly" 0% of events which DO go noticed by the public, like you know, Fukushima, tend to be world news for weeks at a time.
Really, the "big AI story" will be when the next Snowden defects from the NRO and tells us what SENTIENT is doing. Until then it's just going to be a bunch of pie-in-the-sky wishcasting about what we want the future to look like. It's getting a little tiring hearing people insist that abundance is around the corner while surveillance is here-and-now.
I doubt anyone who works on reactors is that stupid.
But some idiot somewhere is going to put some industrial stuff under AI control pretty soon and murder a few people at minimum.
Which: it seems like we're already getting to the point of capability vs. unexpected behavior that having the ability to "trip" a complex system may be an important design criterion.
I mean, there's plenty of harm these systems can do besides trying to turn us all into paperclips.
A) Let AI override safety mechanisms (increases uptime while endangering humans)
B) Let AI impose additional safeties (decreases uptime with questionable safety uplift)
If you intend to go with option B) then there's basically zero benefit to investing in that kind of automation. AGI will be fallible, and even with all the safeties and safeguards it's hard to see what tangible benefit AGI would add to the system. It'll be a hard sell for critical infrastructure of any kind.
I understand that it's hard to get a man to understand something when his AGI Society executive credentials depend on it, but it feels glaringly obvious why AI isn't masterminding these sorts of systems. Even accounting for unprecedented advancement in AI, it's hard to see what AI could contribute to managing these things.
Remember, critical infrastructure attached SCADA systems to the internet and has had trouble removing them all despite 30 years of warnings and audits.
There's definite advantages to having a LLM in the loop. The best we can hope for is that it's architected in a way that it fails safe and damages are limited, so we get some of the advantages but less of the downside.
Outside, almost everyone I know - including highly educated people - takes even Google's search LLM output as fact, without a second thought, and even regarding serious issues (e.g., medical problems). Nobody raises a doubt or an eyebrow. A recent article says members of US Congress use it extensively for serious work. Lawyers use it to file cases. Doctors use it to summarize essential medical information. I even hear that software developers use it to create high-risk, high-value code.
In the public people think computers are superior to humans - more authoritative, objective, perfectly logical and error-proof. SV pushes that narrative, for example saying computer-driven cars are safer because computers don't have human flaws (that old sales technique - compare only the competition's unique flaws and don't mention your own; computers have their own flaws that humans lack). And of course, that one is popular on HN.
(IMHO the way to educate the public is to spread the narrative that every computer is created (designed, programmed) by normal, flawed, biased, etc. humans, and controlled by other flawed humans for their own agenda. And that computers have their own flaws.)
The general public distrusts them cuz they know that they're coming for their jobs. But they will use them out of convenience