Top
Best
New

Posted by Betelbuddy 2 hours ago

AI agents ask better questions than KPMG accountants in audit trial(europeanbusinessmagazine.com)
92 points | 45 comments
prometheus1992 8 minutes ago|
READ TFA- The experiment is conducted by KPMG itself. The result of this experiment is the equivalent of 'most of the code in faang today is written by AI'.

The task of auditing a business is extremely complicated by virtue of data/information being all over the place. AI had not gotten to a point of where it could outperform humans but in this experiment they find out a group of agents are good enough for this task. I still think there are so many poorly defined tasks that auditors just understand implicitly, eg data discovery; before the audit even starts, that humans will always be needed to initiate and conclude audits just like software projects.

mannyv 3 minutes ago||
At some point an audit is literally a checklist of things to look over. If there are anomalies then they get dug into.

I expect for 80% of the audits AI will be 'good enough.' And those are the bread and butter for the public accounting firms.

echelon 1 minute ago||
> 'most of the code in faang today is written by AI'.

And it's true.

We're getting to a point where you don't even have to look at the code anymore, and I'm serious.

> AI had not gotten to a point of where it could outperform humans but in this experiment they find out a group of agents are good enough for this task

The more this is done, the better it gets. Especially as more resources are poured into making it better.

It's only a matter of time before AI auditing is the common practice and humans are the backup for the AI.

trentnix 19 minutes ago||
KMPG accountants are generally inexperienced, overworked, and poorly incentivized. The entire big firm public accounting model is exploitative at multiple levels.
origini 1 hour ago||
Even in normal jobs we have such a HUGE difference between two people. The only organization which is working against are pilots with procedures and checklists.

And every year a new human comes along and has to learn all of it again.

Of course ONE AI agent and teaching ONE AI is massivly in an advantage as soon as its as good as the avg person in the job and it blows everyone else out of the water as soon as it gets a tag better.

joshAg 29 minutes ago||
There are other professions with procedures and checklists. Just off the top of my head there's safe surgical checklists to minimize errors during surgeries [1] and checklists for A+P procedures on planes, checklists for operating/working at nuclear plants, and for factories/manufacturing plants making precision parts or operating within a good safety regulatory regime (like semiconductor fabs or auto manufacturing), to say nothing of checklists in the military, which designs equipment to have easily disableable safety mechanisms if it has any at all because the operator might need to operate it outside normal, safe limits [2].

[1]: https://www.who.int/teams/integrated-health-services/patient... [2]: https://en.wikipedia.org/wiki/Battleshort

KolmogorovComp 11 minutes ago||
But all these procedures stem from pilot checklists.
lixtra 36 seconds ago||
[delayed]
lazide 1 hour ago||
Except that ‘one AI’ will have it’s own blindspots which can be exploited universally and all at once, until the magic fix goes into it.

It’s literally a monoculture.

For example, if an iteration of the singular AI ends up being crashable with a magic visual pattern or QR code (highly likely at some point), that is a major national security risk if that AI is piloting aircraft, as the entire nations aircraft could be taken down by anyone projecting said image in it’s visual plane until the fix gets pushed.

And these systems are far too complicated to exhaustively test all possible inputs before release.

There is a reason why nature rarely has such monocultures - that live for very long, anyway.

origini 36 minutes ago||
With RL it will learn from all misstakes once while with humans after a generation, information might just be gone.

Its a possible as an attack vector for sure.

But in comparision to nature, compute can speedrun all of this with RL.

disgruntledphd2 5 minutes ago||
RL data is really really really expensive to collect and train on, and only works for verifiable tasks, which is not most tasks.

Even a lot of what software professionals do isn't verifiable.

chilmers 55 minutes ago||
I think properly-trained AI agents will have a huge advantage against humans in such contexts: They will have no intrinsic tendency towards laziness. I think most humans have a natural inclination to optimise the amount of work we do in a professional setting, unless we really happen to enjoy it doing it for the sake of it. We can fight against this by setting up processes and incentives, but we'll always be going against a tendency deeply ingrained by evolution to minimise our mental energy expenditure and to avoid repetitive, tedious or boring tasks.

On the other hand, with AI we'll use reinforcement learning to create agents without such evolved-in tendencies. Where (so long as we can solve alignment and cheating) they are trained to do the job, no matter how much effort it takes, or what's involved.

wongarsu 25 minutes ago|
LLMs absolutely suffer from lazyness. Both reinforcement- training environments and benchmarks have token limits, which makes giving 100% on every subtask disadvantageous

They are less lazy than humans though

disgruntledphd2 3 minutes ago|||
They are differently lazy from humans. They'll happily bang their heads against a wall to make something work, but they'll almost never step back to consider a different approach.
jtmarl1n 8 minutes ago|||
Is that laziness or just resource constraints
chirau 35 minutes ago||
I am not surprised at all.

AI labs were working with companies like Mercor for more than a year paying the consultants and accountants themselves from these very firms a good amount per typical task they work on everyday, some even providing actual documents.

The expectations were more rigorous, and the rubrics pretty expansive. so yeah, the agent trained from tens of thousands of consultants would eventually ask better questions than an individual consultant.

marconey 1 hour ago||
https://www.ft.com/content/5168b3e9-75d0-4be0-9ccd-d327d6b16...

I think this is the source FT article

joe_the_user 25 minutes ago||
I think AIs can ask good questions and lead a person to important insights.

But they can also fail to ask "obvious" questions and get stuck at points that an average person wouldn't be stuck in.

Which makes them for adversarial processes - like auditing or hacking. But no so good at producing defect-free software or planning without guidance. With guidance, they're great aids in this too. But when people AI processes to "turn off their brains" or act autonomously, then you get problems. And you get a lot of pressure to very lightly if at all scan AI output when AI investment is supposed to show productivity gains.

redwood 40 minutes ago||
What this means is we Face a future in which instead of a sample of accounts auditing moves into being a comprehensive assessment. That's what near free at scale Brute Force enables that humans simply could never do.

There are many downsides to this in that business cannot ever be perfect and a sample-based regime is useful because it allows for a few mistakes here and there it's not waste everyone's time at audit time

So similar to cybersecurity where you're going to need to apply AI offensively and defensively I suspect we'll see the same in accounting

jsty 14 minutes ago|
> It allows for a few mistakes here and there it's not waste everyone's time at audit time

But even moving from sampling to an exhaustive analysis, the time-wasting mistakes could still be rejected as non-material (i.e. too small to care about). If it wasn't possible to automate this, I doubt an exhaustive approach would take off.

jordanb 51 minutes ago||
KPMG's nickname in Germany is Kinder Prüfen Meine Gesellschaft (children auditing my company). Not exactly a stamp of excellence. The KPMG folks were probably also asking questions they got from gpt.
TrackerFF 13 minutes ago|
TBH the same could be said about pretty much any of the big professional white collar firms. Accounting, consulting, etc. these industries are built on a never-ending stream of hungry graduates that will pull long hours for a couple of years.

The very first time I ever worked with a McKinsey consultant, he turned out to be a jr. consultant half year into his gig at McKinsey. In all likelihood a really sharp and ambitious person, but extremely junior when compared to the domain experts (with decades of experience) in the same room.

shye 1 hour ago|
I suspect this says more about KPMG accountants than AI.
johndhi 1 hour ago|
this is a cute comment, but the reality is a similar analysis could probably apply across literally every profession
munk-a 1 hour ago|||
Certainly within large corporations where the best survival skill to have is not asking inconvenient questions.
devindotcom 1 hour ago|||
literally every profession
hall0ween 18 minutes ago||
literally. every. profession.
More comments...