Top
Best
New

Posted by droidjj 9 hours ago

Kimi K3, and what we can still learn from the pelican benchmark(simonwillison.net)
230 points | 131 commentspage 3
Xx_crazy420_xX 7 hours ago|
I would be surprised if pelican svgs are not part of the training corpus rn
skeledrew 7 hours ago||
If that were the case then it'd do a way better job. Think experienced artist level.
teravor 5 hours ago||
how would great pelicans make their way into the training set?

what they do have are many different pelicans and people helpfully rating them in the comments.

dgellow 5 hours ago|||
That’s covered in the article
seventeengivens 7 hours ago||
[dead]
somelamer567 1 hour ago||
I'm wondering what the grift here is.

Usually, the pattern is that we see a tsunami of planted "China number one" stories boosted by hordes of Chinese "internet commentators", and then the world trembles for a few days until the scam mechanics are revealed.

My would be either: crippling limitations on the model, vast, unfair, and/or illegal subsidies by the CCP regime as a mercantilist attack on Western capabilities (as we've already seen in iron smelting and clean energy), sanctions-busting, gamed benchmarks, outright theft -- or a combination of the above.

sneurlax 1 hour ago|
Invest in energy, manufacturing, and education (ie. your own people) for 75 years and people will look for a trick card up your sleeve and accuse you of cheating when your 7th of the world population has a 7th of the world's genius
csomar 6 hours ago||
If anyone wants to try SVG generation from different models, I made this: https://codeinput.com/svg (here is an older generation: https://codeinput.com/s/5KEGl1e3rB3)

You still need an OpenRouter API Key and be careful this can burn quite a bit of money.

danielrmay 2 hours ago||
[dead]
hdjdjdjdjdjdjd 7 hours ago||
[flagged]
brcmthrowaway 6 hours ago||
Imagine shilling some CLI tools no one uses in this post.
dghlsakjg 6 hours ago|
Lighten up.

You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.

Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.

Zsfe510asG 5 hours ago||
Kimi is right out since they use classical music branding to sell their slop. At least McDonalds does not sell Verdi or Allegro burgers.

Why does Kimi not use a "Double Cheese Whammy" branding for "their" butchered and stolen IP?

mrcwinn 6 hours ago||
K3 is as expensive as Sonnet, not great at writing English, is handing IP back to the Chinese, and once open source will be difficult to run at scale without the compute that OpenAI and Anthropic have largely grabbed.

Sorry, how again is this the end of the frontier labs?

rootlocus 6 hours ago||
According to some benchmarks has the coding capability of Opus at the price of Sonnet, supposedly will be open weights and is not subject to random trade wars with allied states.

Competition is always good.

olig15 6 hours ago||
You mean the scale that AWS provides with Bedrock?
nickthegreek 6 hours ago||
Bedrock needs to actually update their chinese models to the newest versions for this to matter.
BugsJustFindMe 7 hours ago|
> This is expensive—the pelican cost 25 cents!

Engineers get unbelievably silly about evaluating costs of things.

"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.

Yiin 7 hours ago||
they're comparing to similar capability llm models, not humans. If one dishwasher does job at similar quality as another dishwasher, but using 30% more water and energy, you wouldn't compare to how much it costs human to do the same work, it would make no sense.
BugsJustFindMe 7 hours ago||
> they're comparing to similar capability llm models, not humans

25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.

jchw 7 hours ago||
Well first of all, any non-trivial use of LLMs is going to be orders of magnitude more tokens than this, usually multiple millions at minimum. Benchmarks are just benchmarks after all.

Secondly, humans vs LLMs are apples vs oranges. It makes no more sense to compare human costs vs LLM costs as it would have to compare human costs vs calculator costs. LLMs are faster and cheaper but extremely different beasts with different limitations. Humans do not one-shot SVGs of pelicans riding bicycles, and they do not charge in tokens.

Comparing LLM cost efficiency is not something that should need to be defended. It's quite straightforward and reasonable...

bakugo 7 hours ago||
Would anyone pay a human to create an SVG of a pelican riding a bike?
BugsJustFindMe 7 hours ago|||
In fact humans get paid to create SVGs of all kinds of things.
dgellow 5 hours ago||
Well, not anymore
codezero 6 hours ago|||
Well, no, not now they won’t.