So do a lot of financial planners / advisors

Is this just because LLMs can't do math directly? If so, they can certainly write scripts to do math though and those will be a lot more reliable for queries.

I would prefer to use agent-assisted python scripts that chatbot.

From the official report:

> Since LLMs can give different answers to the same question, each question was run five times. That means, each LLM was tested 600 times, and in total over 10,000 questions and answers were assessed.

> All models were given the same zero-shot format. They were not given worked examples, previous conversations, hints or an opportunity to correct their answers. This is to make it as similar as possible to a response to a question from consumers.

As for the evaluation itself:

> Responses were checked against this (using an LLM-as-a-judge), and was only given a pass if every element was met; otherwise it was assessed as a fail. This all-pass approach was intentionally strict, so that the score measures whether an answer is complete enough to meet the expert legal standard, rather than how many individual points it gets right.

It's just AI slop and it should be taken with a mountain of salt.

Now, compare this to a recent story that seemed to claim the opposite:

https://news.ycombinator.com/item?id=49139102

I don't have the time to review the underlying research and decide which one is more correct. My personal biases make me want to believe the current one. Your personal biases may be pulling you in the other direction. How do we make the conversation more intelligent than that?

A company selling combined human + AI financial advice finds that AI advice alone is unreliable? Color me surprised.

> Younger investors are also more likely to trust AI than financial influencers or TV shows

I mean, you absolutely should not trust any of those things on financial matters, bloody hell.

This seems like undisclosed paid stealth-advertising for Thomson Reuters' new model.

These models do pretty well in benchmarks and real world so I'm highly suspicious of this article. Further more, in the original report, the examples of bad answers are from Haiku - at least 7 out of 10. Anyone who knows anything about LLMs know that haiku shouldn't be used for anything pretty much.

There's no reproducible set either. I'm not gonna trust this report.

Well duh! If it's not using tools to look up the state of the market empirically it's not likely to be accurate financially.