Posted by vincent_s 11 hours ago
Kimi K3 Intelligence, Performance & Price Analysis: https://artificialanalysis.ai/models/kimi-k3
From all the models available to me I'm most happy with Kimi K2.7 (given the cost/performance).
The question it came up with, "which U.S. state is closest to Africa?" is a pretty standard trivia question without any reason to believe other AIs would get confused. https://pellmell.ai/s/dccdeca69f929f79bc89317035610049
Even GPT-OSS-120b gets this right: https://pellmell.ai/s/1a43dfc7a3baa214aa0fa1b95d2c536a
IMHO an Ai is the llm plus it's harness.
A good harness would allow the llm to investigate on a map.
Just like the llm can use a python script to figure out how many r's there are in strawberry.
These tests are simply not that predictable of performance of the llm.
The other AIs don't see the question until they are asked to react.
Is them pricing at Sonnet level actually give us any information at all at how big Sonnet is or is there too much opacity around inference margins?
If that's true, then the price makes sense