Posted by devonnull 3 hours ago
Reading the details, "AI" doesn't really seem like the culprit -it's a scapegoat.
The intelligence that it was no longer a military target never entered the target database, the team that was responsible for vetting the target list was gutted, and said team was never even consulted.
The White House wanted 1000 targets and pulled from their database without any due diligence. Whether it was an AI call or an SQL query - this was from pure human maliciousness and incompetence.
But don't worry, the other P AI system will ensure that the first one doesn't do mistakes...
UnitedHealth uses AI model with 90% error rate to deny care, lawsuit alleges - https://news.ycombinator.com/item?id=38299921 - November 2023 (58 comments)
https://litigationtracker.law.georgetown.edu/wp-content/uplo...
> Some US intelligence analysts took notice, the officials said. Bloomberg reported in June that one analyst detected changes to the site as early as 2019, and logged remarks about the modifications in a system that wasn’t connected to the primary military intelligence database, which informs targeting
At the time a news outlet put out commercially purchased images of the target site before and after -- even a layman could see that it had a children's playground next to the building, and that it was separated from the compound.
https://www.channel4.com/news/factcheck/factcheck-iran-schoo... has the info from around the time, but doesn't show the detailed images in the news report (video) I seem to remember seeing.
I think it was maybe some earlier coverage from NYT, they published this retrospectively - https://www.nytimes.com/video/world/middleeast/1000000107503... (which is also on YouTube, https://www.youtube.com/watch?v=hDcim48W4Io). It shows that even a cursory analysis - without the benefit of enhanced military grade satellite/flyover imagery - indicates the building targeted with pin-point accuracy is a school (eg https://youtu.be/hDcim48W4Io?t=98).
In short, this seems entirely avoidable. I don't think you need any connected system, you just need an image of the target and eyes.
If they trained a model to find targets and didn't train it to avoid civilian targets, and couldn't be bothered to have anyone check them, that's still a war crime. The analyst could probably see the kids playing there with the imaging tech they have available.
This war was unlawfully started by the military at Trump's command, knowing it was not a lawful order; just as they knew firing on shipwrecked civilian sailors was murder. Seemingly it was started to win money betting on fuel prices.
Now if you ACTUALLY used AI, as opposed to what happened here (Trump stooge getting request for 1000 targets and doing SELECT * FROM targets LIMIT 1000, except getting AI to write the query) could could drive the odds of this happening much closer to zero.
-------------------
Maven is an untested tool for rapidly selecting targets under time pressure during a war. It's reason for existing is speed. If you're in a shooting war, the ability to select good military targets faster than your enemy, but not necessarily with zero error, is an advantage.
Planning a first strike is a different situation. Time pressure should not be a significant factor. Humans should have been doing their due diligence on every single target. If an untested tool for rapid mid-battle target selection was relied upon completely because of a hasty executive order, then the responsibility falls entirely on the executive.
Unfortunately the military all turned their backs on their oath to uphold the constitution (Again! Deploying against USA cities and kidnapping foreign heads-of-state at the whim of Trump is also not lawful). At that point it is hard to see how your democracy can be resurrected.
Just like clowns in charge were caught off guard by the fact that Iran has a drone program, resulting in the loss of dozens of air frames and much of our strategic munition stockpiles. Iran literally jump started Russia's drone program in a high profile way, and yet trump/hegseth somehow completely failed to plan for how to counter those drones and had to resort to lobbing scarce multi-million interceptor missiles at them and watching our bases take heavy damage.
These are just some of the failures we know about. The administration has been extremely tight lipped about these failures and there are surely many more that aren't common knowledge. Who knows how many hundreds of other children have been needlessly killed outside of the school strike.
Except laundering our biases & discriminations, the unaccountability seems to be the purpose of a huge huge swarth of AI usage. To eliminate accountability. That sure seems like A Feature Not A Bug to the worlds worst people.
...
> Hegseth and the commander in charge of the Iran operation, Adm. Brad Cooper, were questioned about the strike before Congress. Hegseth said the U.S. does not target civilians and would investigate
So the administration does not intentionally target civilians, they have just stopped caring about who a civilian is.
https://skeptics.stackexchange.com/questions/56556/does-the-...
> fight in civilian clothing
(Fake) physician, heal thyself.
https://www.pbs.org/newshour/world/israeli-undercover-forces...
> Article 4 applies to soldiers (that e.g. fight in uniform)
Israel openly forbids the importation of uniforms.
https://www.jpost.com/arab-israeli-conflict/israel-seizes-la...
What a fucking moron
Basically they stopped caring those woke UN rules about war crimes.
I cannot believe this has come to pass.
> The Pentagon chief last year slashed offices that didn’t contribute to his goal of “lethality,” including the group that assists in limiting risk to civilians, known as the Civilian Protection Center of Excellence. Around 200 employees who worked on the issue, including at that office, have been reduced by about 90 percent, according to two current and former officials and a person familiar with the effort. The team that handles civilian casualties at Central Command, which oversees the Middle East, has dropped from 10 to one.
Negligent homicide in pursuit of their DEI goals, basically.
Negligence was involved.
DEI for drunk white men who look good on TV got us Hegseth, who's now rather hilariously requiring gender-affirming treatment for the troops. (https://www.pbs.org/newshour/world/pentagon-moves-forward-wi...)
(setting aside when civilians are purposefully harmed which is a conversation about morals rather than the capability to make correct decisions)
What I am getting at is, has AI really made the problem worse or has the mechanism just changed?
When it was just called a "database query" or "algorithmic estimation", it was more implied that the actual decision is left to be made and that no analysis had actually been done yet. But when AI preemptively says "I've analyzed every source of data and come to the conclusion that this ship is carrying nuclear bomb material"; it really sounds urgent doesn't it? It probably doubled down when asked about it.
https://edition.cnn.com/2026/09/18/politics/us-military-ai-f...
And I hedged that. Anything short of 100% responsibility with humans is a failure. I get investigating how it occurred, but that needs be very careful to separate ought from is.
The obvious questions is when the intelligence needed to be updated (ie, when Iran decided to put a girls school on a military base) and when Iran made that fact public, or at least when the US intel should have figured it out with satellites and other sources.
If it's years old then definitely on human. Intelligence will always be flaky in these hostile countries.
> An AI can be tried in a court.
An AI can't be tried in a court.
This is what happens when people who don't understand AI and its limitations embrace it wholeheartedly, thinking it's going to behave like The ultimate analyst. Except it's not. Its got blind spots in more places than anyone knows. And not to mention the system itself has multiple areas of failure.
This was a totally expected outcome, similar to how CV cameras arrest the wrong minorities in neighboring states 100s of miles away because it told the officer it was them; or how health insurance companies use ML algorithms to deny Americans care.
It's going to happen in other sectors of life and it will continue killing innocent civilians.
you can also develop pro-social coping mechanisms like open-mindedly talking to someone who opposes you and trying to understand their worldview because you trust that it might expand your own
but, yeah, I doubt pro-social traits are common at the top of any hyperstrict hierarchical organization like the USAF
Really good time to do various crime.
Replace "AI" with any other tool or software in your scenario, and the outcome is the same. The same people and organizations are accountable, it's just that we currently live in a world without accountability in the highest places where it's most needed, and that's unrelated to AI.
Or with that the EU AI Act has been redundant and postponed in 2023 because it was drafted on ResNets, and then redundant and postponed in 2025 because it was drafted on ChatGPT 3.5, and then this year probably again because it didn't have agents included?
Or with that other countries that getting their companies hacked by AI models are ignoring any kind of prosecution of said companies and delegating this to the US?
I feel like we're optimizing the wrong metric here.
These missiles cost a couple million each.
https://gizmodo.com/almost-started-a-war-us-military-nearly-...
> That didn't happen.
> And if it did, it wasn't that bad.
> And if it was, it was AI's fault, not mine.
It's gonna say something like "they were all military-aged males". https://en.wikipedia.org/wiki/Military-age_male
There needs to be some protection for first responders and others helping the injured.
Mind you Israel is currently funding a lobbyist organization in the UK to flag Médecins Sans Frontières (Doctors Without Borders) as a terrorist group (because many years ago a couple of people who had affiliations with other anti-Israel groups were found to be also affiliated with MSF).
No-one mentioned "using ambulances as cover". That is separate from "enemy combatants are acting in a first responder capacity, but are still enemy combatants, therefore the entire first response effort is a viable target". That is a war crime that Israel regularly likes to commit.
> These are the same people who view women as property, homosexuality as a reason for execution, and 13 years olds as viable brides.
Halacha? Acquiring your wife through kiddushin? Only a husband can issue a get (divorce document)? Women in agunahs (dead marriages)? Sounds like property to me.
Rabbinic law makes capital punishment effectively impossible, but the Orthodox sects do everything possible to punish: disownment, both family and social and actual exile from physical communities.
Don't girls reach halachic adulthood at 12?
And lets not throw stones here. Even in the US, you can get married with parental consent or a judge's consent at any age in Mississippi, New Mexico and Oklahoma.
You can get married at 15 in Hawaii and Kansas.
You are "grandfathered" into sexual consent (not as in "marital rape" but "no longer statutory rape") when you do so.
So we may not want to be lording it too much on some guise of moral superiority, particularly given that most of the marriages that happen at those ages are not "young couple got pregnant, want to do the right thing" but "older man in conservative/religious community saw young vulnerable girl and decided he had to have her".
"You're absolutely right."
> A Palantir spokesperson told Bloomberg the company “is not responsible for the underlying data nor identifying intelligence deficiencies” and that there is no evidence its software was at fault.
If you work for Palantir, you are writing software that kills people. Software that, in this case, killed 123 children. What salary is worth that?
I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth [0]. I think about it nearly every day.
My brother in Cthulu, they're the ones making the life or death decision that killed people, who else should be to blame?
>But more relevant to HN, this couldn't have happened without technology professionals working for Palantir.
If you wanna go the historical tech-tree scapegoat route then none of this would have happened if DARPA wouldn't have invented the internet, or if Thomas Edison wouldn't have invented the lightbulb, or if the fish haven't ventured onto land and evolve into us land mammal apes who invented sharpened sticks as murder objects.
This isn't the case where we are talking about a company doing anything socially good. It's simply there to help authoritarian governments.
>It's simply there to help authoritarian governments.
The fact that you're so close yet so far away to figuring out who's to blame still boggles my mind.
It’s an evil service and no matter who requests it or supplies it, doing so is an evil act.
No, you are. I'm implying evil are (primarily) those who use it, not (just) those who make it. You can try to make whatever you want, if nobody will buy it, then you won't have money to make it. Palantir only exists because governments are evil and willing to buy it. If you want Palantir to not exist you need to hold your government accountable.
Are gun makers evil and responsible whenever someone gets killed by a murderer, or are those who pull the trigger responsible?
Yes. There's a reason Remington decided to settle for $73M dollars with the sandyhook parents even though they didn't "pull the trigger" [1]. That's not a number you pick because you think you'll win the case.
[1] https://abcnews.com/US/sandy-hook-families-settle-remington-...
It's only because of how the legal system works in the US, with ambulance chasing lawyer and where cases are decided on the emotions and societal moral panics of the people in the jury and negative image such cases can cause to a brand, not if someone is objectively right or wrong.
Meaning it's cheaper to pay people off even if they're legally wrong, than try to prove you are right but then deal with the bad publicity of the mob with pitchforks and torches who blames you for their kids dying and looks bad in the media.
Keep in mind parents of that time also blamed violent videogames like Doom for it, so it's not like they were right about who was to blame, they were just looking for retribution from anyone they could aided by ambulance chasing lawyers looking to cash in.
Palantir were in no way obliged or coerced into making an evil thing, they just decided being evil for money was ok.
> I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth.
I know a guy who works for well-known hardware company. He isn't a software dev, but close enough, works with Verilog on hardware design and verification. One day he tells me how he is working on video processing module and he doesn't know if that module will be part of surveillance system, missile guidance or nanny cam. It's possible that even middle management doesn't know more than him, and very likely the module will find uses in multiple products. He is a small cogs in a large machine, maybe his work will save a baby's life, maybe it will kill a person, he cannot know.
So yeah, the military and whoever decided that a war was a good idea are to blame.
People working at Palantir are as much responsible for the killing of these 123 children than seatbelt manufacturers are responsible for the people who died because their product failed. That is, because they didn't do their job correctly, people died, definitely worth feeling bad about. However, should people stop working in seatbelt manufacturing because of it? Absolutely not, we need seatbelts, because without them, things would be worse.
Palantir, or whoever does target identification, is like the seatbelt of the reckless drivers of the militaries, who, for the lack of precise information, would just carpet bomb the place. It is not a hypothetical, it is how the US has worked before they had precision ammunition.
This is essentially a trolley problem. The war is the trolley, people who work for Palantir are the ones pulling levers. And if there is no one to pull the lever, the trolley is gong on a track with a lot of people on it.
Would you rather not have someone make good software that does not misidentify schools at sites? Or should we outsource this to worse engineers who don't care about morals and don't find this metric meaningful?
If so, I'd say yes, you share some of the responsibility. Legally, I'm sure you'd be fine, but if that were me, I'd feel responsible.
The doctor of course is responsible too; they're the one who didn't double-check that it was an appropriate drug. But multiple people can share responsibility for a bad thing.
Whether you may/should feel responsible depends on how the software works, I guess. Did it automatically recommend some drug, somewhat confidently, with gray text on gray background saying "AI sometimes makes mistakes"?
So I think we should behave responsibly when making software, even more now with AI