Top
Best
New

Posted by landonxi 7 hours ago

Dayjob: Tax – evaluating AI agents on 50 professional tax assignments(surgehq.ai)
167 points | 97 commentspage 2
gamblor956 1 hour ago|
As a counterpoint, I ask the big llms every month the same set of tax questions about the last US tax bill and they still can't give me the correct answers.

I started my career fixing the mistakes that my clients' former tax advisers made. It looks like the next generation of tax lawyers will get to do the same except this time the former advisers will be llms.

andriy_koval 6 hours ago||
What is human baseline? How it stacks against 10% LLM accuracy?..
andrewdong 5 hours ago||
wonder if we should bring back https://taxheaven3000.com/ in the age of ai
matkoniecz 6 hours ago||
Sharing all my financial info with LLM companies is not something that I like.

EDIT: good point that paying accountant does not ensure avoiding it

wonderful, you made a good point that make me dislike situation even more

cornholio 6 hours ago||
Your accountant sure will.
grey-area 5 hours ago||
Most accountants will not in fact share client data with an LLM company.
koliber 1 hour ago||
Curious where you're getting this confidence from.
inanutshellus 4 hours ago|||
I'm for sure reinventing the wheel here but I've had an LLM generate an app that walks me through my existing tax routine I've been running with for years. Thus no real numbers go to any LLM I just give it my situation and it fills out the form.

The plan now is to retrieve the current year's tax forms then have an LLM compare them to stored versions from last year, and look for variances (e.g. new field, new form requirements, etc) all of which an LLM would excel at (whilst basic math it historically did not).

vatsachak 6 hours ago|||
Some inference providers have a Zero Data Retention policy
oklahomasports 6 hours ago|||
Worst case scenario what does OpenAI do with your tax return?
matkoniecz 2 hours ago||
- their data will be partially leaked in general, including my financial info

- someone who hates me personally will get access to it (see case where someone put their ex-spouse on no-fly-list etc)

- leak my private info via new version of LLM was trained on it

- sell it to everyone interested (maybe sold after their bankruptcy)

- give it to NSA and whoever else

from obvious ones

ChucklsTheBeard 6 hours ago||
Oh look, yet another reason to use local inference.
ilmalan 4 hours ago||
Useless benchmark as it seems to be evaluating literally only the weights.
ffwang2 6 hours ago||
i want to always have a fast way of reviewing my tax returns without having to hassle my accountant or hire another one.
aquafox 6 hours ago|
In Germany, there are companies like Buhl data that let you do your tax return very easily and give a live estimate of your return based on the data you provided. From my experience over the past 5 years, its prediction error is less than 5%.
edoceo 5 hours ago||
They exist in USA - eg: freetaxusa.com
FLeXMurphy 3 hours ago||
Article I hoped to read: Evaluating AI agents on 50 professional tax evasions
ilmalan 4 hours ago||
Ah, yes, the 'dude trust us' bench.
jimbokun 2 hours ago||
So are the accountants completely fucked yet?
koliber 1 hour ago||
Not yet, and not completely.

Some accountants need to make judgement calls. LLMs are not great at many judgement calls yet.

crisnoble 1 hour ago||
10% of the time it works every time
sbseitz 4 hours ago|
Kinda points to another thing... that our tax code is insanely bad.
More comments...