Top
Best
New

Posted by 12_throw_away 18 hours ago

Law360 mandates reporters use AI "bias" detection on all stories(www.niemanlab.org)
58 points | 77 comments
JoshTriplett 16 hours ago|
Timeless quote:

> If Jimmy Allred says it’s raining, and W. Lee O’Daniel says it isn’t raining, Texas newspapermen quote them both, and don’t look out the window to see which is lying, and to tell the readers what the truth is at the moment.

https://quoteinvestigator.com/2023/11/14/rain-look/

If an AI tool, or for that matter a meddling editor, says a headline is "framing the action as unprecedented in a way that might subtly critique the administration", the correct response is "yes, that was the idea".

zaphar 16 hours ago||
I don't think most reporting bias takes the form of incorrect facts. It takes the form of picking which facts to share driven by the facts that matter the most to the particular reporter sharing them. This results in slanted coverage even if technically it's "factual".
lcnPylGDnU4H9OF 16 hours ago|||
That as well as the words and tone that are used to describe things and the context surrounding those things. A headline that is neither negative nor positive is not strictly unbiased because it implies that the thing being reported is not very unusual. If it is unusual, then the neutral tone is communicating a bias that this unusual thing should not be regarded as such.
bmelton 15 hours ago||
“... by definition, news is something that almost never happens.”

- Bruce Schneier

mr_toad 15 hours ago||
Except for the annual “health system in crisis” story.

Edit: substitute any issue you care to name for “health system”, there’s always a crisis that can be used to grab eyeballs.

JoshTriplett 16 hours ago||||
Take a look at the headline flagged in the article, which the tool flagged as "framing the action as unprecedented in a way that might subtly critique the administration". The headline is factually accurate, and the facts are not cherry-picked.

Yes, some reporting is biased. But some reporting is simply accurately reporting damaging information, and "biased" is a way of attacking that without addressing the substance of the problem.

zaphar 15 hours ago|||
I agree that article's headline is pretty factual nor are the facts cherry-picked. But "some reporting is biased" is heavily understating the problem. And I hypothesize that the reason there is a leap to "This is biased" today on reporting is because the news media organizations have participated in the cherry-picked facts case to enough of a degree that trust has eroded.

Which is perhaps why organizations are searching for ways to not appear biased and begin to restore that trust. I think the reason the LLM approach is likely to fail is because it is unable to detect the "missing facts" case and can only really advise about sentiment and phrasing. Which is not I think the actual problem that needs fixing.

JoshTriplett 15 hours ago||
I agree that an AI is unlikely to help. But also, I doubt that the primary reason is "restoring trust". I think the primary reason for many of them is that some of their readers react strongly to things they don't want to hear, and they're afraid of losing customers, so they're watering down their reporting to avoid being inconveniently right.

Or, even less charitably, management and employees have different politics, and management are the ones who find the articles inconvenient.

To be clear, there are absolutely biased news sources out there. For many of them, the bias is the point, and they have no particular desire to "restore trust" because they're already trusted by people who only want to read things supportive of their party. But a politician who finds the truth inconvenient will decry everything accurate as biased.

tempodox 15 hours ago||||
How could an LLM even decide whether facts are presented in a balanced way? Someone at Law360 seems to believe in a magical oracle.
sneak 15 hours ago||||
The factual accuracy of a statement does not have any bearing on the bias or agenda of the person making the statement.

I have seen many documentaries that contain only facts and real events, but nevertheless are pushing a heavily biased agenda. Which facts we report and which we highlight and how we frame them tells a story.

mike_hearn 2 hours ago|||
Most of the examples are correct decisions by the model. The headline is the most arguable, the rest are pretty clearly improvements, at least to people who actually want unbiased journalism. And as the article is itself highly biased, written in an antagonistic tone that takes the side of the journalists, we must assume they picked the worst examples they could find. So nearly all of them being clearly correct implies the model is doing a great job overall. Given the evidence presented here, it's not surprising that Law360 is mandating journalists accept the edits.

For example: "the bias indicator flagged a line that said the suit spotlighted challenges with ableism and sexism in the healthcare industry. This copy was flagged because it “frames the lawsuit as a representative example of systemic issues.” Instead, the bias indicator said the story should “state the facts of the lawsuit without suggesting its broader implications.”

The model made the right decision. That's the kind of language that makes people distrust journalists: a passive voice assertion of a radical belief, which adds nothing to the story and isn't supported by any evidence. It doesn't belong in a professionally written report.

And yet we read, "that edit is at odds with any attempt to deliver legal analysis. In most cases, reporters chose not to accept these edits, but they were still required to go through the motions."

Claiming the healthcare system is challenged by "ableism" isn't legal analysis, that's pure politics. But more importantly it's also just absurd. Of course people who work in healthcare want people to be "able". Why would you become a surgeon if you thought disability and sickness was fine? No surprise their bosses are placing the LLM's judgement over that of their own employees, when the journalists have gone so far off the rails. The union should be careful of picking a fight over this because frankly the next step is to just automate away the journalists entirely.

o11c 13 hours ago||||
That's why "the truth, the whole truth, and nothing but the truth" is so important. Yet we don't even enforce the limited perjury laws we have.

The common "must present both sides" approach can fail the "nothing but the truth" criterion. But even with its many errors, it's better than not trying anything at all.

egberts1 12 hours ago|||
"slanted", isn't that a form of biasness? #headduck
IshKebab 15 hours ago|||
Yeah the BBC suffers badly from this problem, because they are required by Ofcom to have fair and "balanced" reporting, and they interpret that as meaning they always have to get one view from each side of a story. Doesn't matter how batshit or fringe a side is, they'll present them equally.
hitekker 14 hours ago|||
You might have seen different BBC stories than I have.

Like when the BBC said all converts to Islam are "reverts"; https://www.spectator.co.uk/article/why-did-the-bbc-say-musl... ; the nasty implication being that one religion is the status quo of humanity

Or when the BBC framed an Ad as racist, because it called out a politician's sectarian & anti-LGBTQ appeals https://www.bbc.com/news/articles/c5yg0g18989o https://www.reddit.com/r/unitedkingdom/comments/1ksv0y7/refo...

The other day actually, I saw a newscaster describe the horrors Apartheid in South Africa as though it were happening today. She then closed with "... [Apartheid] is now no longer enforced" and transitioned quickly to the next topic. Not ended or abolished, but merely "not enforced".

Their bias is rather pernicious.

rafram 13 hours ago|||
> Like when the BBC said all converts to Islam are "reverts"

They didn't. They used their interviewees' own preferred terminology to refer to them in the story, which is fine. "Reverts" is the most common term among Muslims. Sort of like how capitalizing God in a story about Christians doesn't invalidate the beliefs of people who worship multiple gods.

> Or when the BBC framed an Ad as racist

The first line: "Scottish Labour has described [...]".

IshKebab 6 hours ago|||
> Or when the BBC framed an Ad as racist, because it called out a politician's sectarian & anti-LGBTQ appeals

This is an insane characterisation of that article, which seems fair to me and not at all what I was talking about. Wanting your race to be represented in politics is a perfectly reasonable view and it doesn't need "calling out".

ars 15 hours ago|||
Is that why they come across as so antisemitic?

Obviously there are biased antisemitic news organizations, but of the high-profile ostensibly neutral ones BBC stands head and shoulders above the rest in the level of antisemitism.

(You don't have to take my word for it - a quick Google will find huge numbers of examples. Usually they'll get criticized and then post a correction, so in some sense they themselves acknowledge the problem - yet it keeps happening.)

bigyabai 15 hours ago|||
Which definition of "semite" are we using, today? It tends to get thrown around to mean a lot of different things on HN.
mhb 15 hours ago||
You're happy to redefine genocide in a ludicrously expansive fashion but pretend to need clarification about what antisemitic means in this context? I know, you're just asking questions.
adrian_b 8 hours ago|||
"Semitic" is a word that includes both hebrews and arabs, because both are classified as descendants of Shem in the Bible.

Despite that, most people who use the word "antisemitic" apply it only to something that is done against hebrews, and not to something done against arabs.

Therefore it makes sense to request clarification about what someone means by that word, i.e. if they meant that BBC is anti-Israel or it is anti-arabs.

It would be much better if everyone who means that something is anti-Israeli, would say it clearly, instead of using the ambiguous word "antisemitic".

The word "Semitic" has been created due to a misunderstanding of the Bible, because there the classification of the people was not based on real descendance from common ancestors, but it was based on the political dependence of those people at the time when the Book of Genesis was written. Unfortunately, nobody has found a suitable replacement for this word.

(In the Bible, the descendants of Shem were those dominated by Assyro-Babylonia, while the descendants of Ham were those dominated by Egypt, regardless of their true ancestors. For instance the Phoenicians were classified as descendants of Ham and the Hebrews as descendants of Shem, despite being 2 extremely closely related populations, separated by little else except their different religions.)

mhb 6 hours ago||
Yes, of course. But in this context there was no confusion and no ambiguity.
bigyabai 10 hours ago|||
I don't believe I ever did that. Sounds like you're deflecting.
mhb 6 hours ago||
You never did what? Deflecting what?
mhb 15 hours ago|||
> Is that why they come across as so antisemitic?

If you're inclined to offer the most charitable interpretation in the universe, perhaps.

sandwichsphinx 16 hours ago|||
reminds me of the essay "Politics and the English Language" George Orwell wrote in 1946, it's a good read
lazyeye 14 hours ago|||
This really is not it.

Outright lies are very ineffective in manipulating public opinion because they can be easily disproven.

It's much more about pretending its only a little rain when its raining alot. Ignoring the rain when it doesnt support your narrative. Pretending the rain is really important when its not important at all. Pretending it only rains where you are and much less everywhere else etc etc etc.

JoshTriplett 14 hours ago|||
> Outright lies are very ineffective in manipulating public opinion because they can be easily disproven.

I sincerely wish this were true. "A lie can run around the world before the truth has got its boots on."

lazyeye 12 hours ago||
Convenient half-truths, distortions, mis-characterizations (to support a desired narrative) go alot farther, faster.
thrance 13 hours ago|||
Have you looked at American politics recently? The current administration creates huge lies daily, and most of the media relays them uncritically. It's a level of shamelessness previously reserved for countries like Russia.
lazyeye 13 hours ago||
The previous administration lied alot too but because of your own personal bias you are unable to see that.
thrance 5 hours ago||
I don't hold them in my heart, but no comparison is possible here.
lazyeye 5 hours ago||
Of course, that's as expected.
thrance 5 hours ago||
What is? Can you seriously claim that Trump and his administration are acting more truthfully than Biden's? Don't be stupid.

Just look at what happened in Iran, it's a shitfest of constantly changing narratives, going from warmongerism to peace claims. Lying about what they did and wether it was successful. Lying about wether Israel is on board with making peace.

xp84 16 hours ago||
If “the administration” is The Good Party, the action is “groundbreaking” and “landmark.” If the administration is The Bad Man, the action is “unprecedented.” This is how you frame things to maximize the propaganda effect and scare/please the audience, while maintaining plausible deniability that you’re definitely not pushing an agenda.

(If my comment offends you, I assure you, don’t worry, your party is definitely the Good Party in this scenario.)

JoshTriplett 15 hours ago|||
I'm well aware of how spin works. However, that doesn't mean that such characterizarions are always wrong. One of the severe problems in current politics is that reporting that looks bad for a particular party will always be characterized as biased, whether the reporting is accurate or not.
JoshTriplett 12 hours ago||
(off-topic)

> characterizarions

Sigh, mobile keyboard. *characterizations

rrr_oh_man 15 hours ago|||
"It says here in this history book that, luckily, the good guys have won every single time"
jerf 17 hours ago||
Today, somewhere in the world, some people made an event occur. Some people approved and some people disapproved. It impacted some people more than others, which some people think may be a bad thing, although others think that it may have been a good thing. The event stemmed from many past events, which many people have various strong opinions about. Politicians made various strong statements about the event, as well as a few celebrities. The future impact of the event is under debate, though experts agree that the situation is complicated and people should probably think about it very carefully before coming to conclusions. It is unclear what the market will do in response to this event.
Herring 16 hours ago||
> Party A: wants to abolish slavery

> Party B: wants way more slaves

> Media: record low percentage of Americans satisfied with agriculture today!

Impartiality is very important.

bilbo0s 14 hours ago||
In unrelated news:

General Sherman Begins Demolition in Georgia For Urgently Needed Infrastructure Projects

I mean, you could ask ChatGPT or something to make some pretty braindead impartial newspaper headlines about, like, the Holocaust or slavery. I guess you're right, in some cases, impartiality will come off as a bit ridiculous to the common sense segment of society.

nxobject 15 hours ago|||
Welp, I’ve read every newspaper article - at least I have an excuse not to read the news anymore for my sanity.
encom 15 hours ago||
Breaking News: Some Bullshit Happening Somewhere: <https://www.youtube.com/watch?v=9U4Ha9HQvMo>
owisd 16 hours ago||
The problem with trying too hard to neutrally report both sides of a story is that's not unbiased either, it's just biased in favour of the side that can lie the most convincingly.
wredcoll 16 hours ago||
I am so, so tired, of "both sides" reporting. How about just report the facts instead?
schmidtleonard 16 hours ago|||
It's a rare issue that you can't swing both ways by selecting the subset of facts to focus on. Attention doesn't just matter, it's all that matters. This is why adversarial forums are so important even if they so often degenerate into shouting past one another: each side has the ability and motivation to bring its strongest facts, so they are at least present in the discussion, even if debate tactics add tons of noise and confirmation bias dampens the effects.
wredcoll 15 hours ago||
Yes, in principle, but I object to "reporters" merely quoting people instead of attempting to actually verify if what they said has any basis in reality.

That being said, I'm not sure how much this actually matters, trump seems to pretty comprehensively prove that people would rather hear lies, even if they know they are lies.

kube-system 15 hours ago||||
Nearly all political issues are rooted in philosophical differences of opinion. "The facts" are merely the icing on the cake that various factions use to lure people to their ideology. Factual reporting is better than direct opinion reporting, but both are inherently interpreted by people in the context of the Overton window.
wredcoll 15 hours ago||
You're not wrong, but very often those philosophical differences are being masked by lies.

I don't want to get into the gory details in this tiny text box, but any number of current fear based political campaigns use wildly "distorted" facts to bolster their arguments, and most of the time reporters just blindly repeat whatever the person says without attempting to verify it.

MangoToupe 13 hours ago||||
First you'd have to get people to agree on what a fact is, let alone what the facts are.
an0malous 16 hours ago||||
Because there’s basically no such thing and choosing which facts you report is a huge source of bias anyway
const_cast 6 hours ago||
It's also naturally biased in favor of whoever is more extreme. It's like trying to balance a scale so you place both items on the scale... but one is a rock and the other is a feather.
jauntywundrkind 14 hours ago||
> Several sentences in the story were flagged as biased, including this one: "It’s the first time in 60 years that a president has mobilized a state’s National Guard without receiving a request to do so from the state’s governor." According to the bias indicator, this sentence is "framing the action as unprecedented in a way that might subtly critique the administration." It was best to give more context to “balance the tone."

"Might subtly critique"?! This isn't protecting versus bias, this isn't even charitably regardable as cowardice.

It's an attack on reason & reporting. To shield the worst cry bullies & offenders. It's ghastly how basic simple reporting so offends, that the public interest is sabotaged by these foes. We face such a need to improve the media, and this is the opposite path, of craven blunt un-intelligently intermediated corporatism.

saaaaaam 16 hours ago||
I said to a colleague the other day that Law360’s reporting seems to be more tortuous and weird than usual. I wonder if this is why.

There was one piece on a case I’ve been following quite closely where I was genuinely unable to make sense of what had happened from the Law360 report without sitting down and spelling my way through it.

wredcoll 16 hours ago||
Remember kids, if you don't like the truth, just accuse it of being biased and force it to change!
bilbo0s 14 hours ago|
Truth is biased. That's what makes it truth.

If you wanted unbiased information, you should have asked for facts..

and then collected as many of them as you possibly could.

MangoToupe 13 hours ago||
More people should really read Baudrillard. Objectivity and language don't combine very well. Show me a person who wants the facts and I'll show you someone who is easily manipulated.
jasonthorsness 16 hours ago||
LLMs and ML algorithms are beginning to influence the entire lifecycle of articles: researching, writing, editing, publication, discovery (TikTok), and consumption (ChatGPT summarize this). With the few big players, it could be the same model involved at every step. It's scary how a small change to a system prompt could subtly influence things across the board and guide popular opinion.
exiguus 12 hours ago||
I have so many question.

Let's say a media outlet, like a newspaper, commits to common journalism best practices, such as neutrality. In my opinion, this does not mean avoiding certain words; it means presenting different perspectives on a topic based on evidence.

Besides that, if someone writes an Op-Ed or commentary, the author expresses their personal opinion. How does this work with this AI thingy? Currently it looks like plain censorship, isn't it?

And why should I buy a newspaper that does not represent my political or personal opinion? I thought that was how it worked.

eddythompson80 12 hours ago|
well, in this particular example I think the answer is straightforward: it's just an ad so it doesn't matter.

This is a marketing/sales piece for LexisNexis by their marketing agency as they try to sell AI services to their customers (law firms and other legal professions). It gives them a news article to share with their customers and tell them something like "see, we're taking the AI plunge. You should too. We offer a bunch of legal AI services, lets talk". Now their CEO and sales people can keep saying "I'm sure you've seen the news about Law360. We think AI is finally ready for legal tasks. You know, as an assistant."

In reality, they are probably to run the article, look at what the AI says and shrug and move on.

BrenBarn 16 hours ago||
Who checks the bias detector for bias?
ethan_smith 15 hours ago||
This creates an infinite regress problem - bias detection systems are themselves trained on data reflecting human judgments about what constitutes "bias."
kelseyfrog 12 hours ago||
That's the nice thing. You don't. The conceit of centrism/Liberalism is that extreme voices cancel out. Just ingest a large enough corpus and the central limit theorum takes care of the rest.
MangoToupe 13 hours ago|
To me, a newsroom's bias IS its value. If you let me know your values, I can actually interpret what you're saying. It's the pretense of impartiality that makes me distrust someone.
More comments...