Top
Best
New

Posted by delichon 17 hours ago

Karpathy’s Pelican(twitter.com)
https://xcancel.com/karpathy/status/2083749667410727319
229 points | 179 commentspage 2
toolslive 2 hours ago|
Reading the title, I was thinking "Karpathy? I don't know this chess player." (The Pelikan is a well known chess opening, and famous chess players often have book titles like "X's Y" where X is the player, and Y is the opening)
siliconc0w 1 hour ago||
There is a tipping point between procedurally generating everything in SVG to maybe giving them tool access to something like 3dsmax (or having them build and then use a tool to do the thing vs doing the thing).
gordonhart 1 hour ago|
I’d love to see this benchmark using Blender. Asking a model to animate a scene in Three.js is a square peg/round hole; it doesn’t convey much when the model can’t get it to fit. With Blender the human expert ceiling for this task has been proven to be very high
trentor 3 hours ago||
I always thought of the Pelican more of like a gimmicky quick test. There are people who took it as a serious benchmark for overall model performance?
NitpickLawyer 3 hours ago|
"Draw a pelican on a bicycle" is not a serious benchmark.

"Draw an animation of this long ass scene from a movie, and only call me when everything works e2e" can be.

ActionHank 1 hour ago||
I feel like we are going to look back on this era of llm usage like we do at the period of time when we thought radiation was magic.

Consuming radium and using uranium glass, that’s what we’re doing.

sinaatalay 1 hour ago||
On consumer devices, AI communicates with us through speakers and screens. Screens are the richer medium, so most consumer AI innovation will happen there.

Computer graphics will have enormous applications because they are directly controllable by LLM-generated code. Video models are probabilistic and less suitable when precision matters. In education, for example, we need exact visuals. If an AI wants to plot y = sin(x), it should generate the precise graph through computer graphics rather than approximate it with a video model.

duxup 1 hour ago|
Coding, graphics, all seem to have very defined background data for an LLM to base their decisions on, test, and even when I correct it we're all on the same page. Seems like there's lots of room for AI productivity there.

What I find funny is that approximate to computer graphics are video games. When I ask AI about a decision available to me in a video game AI completely fails, OFTEN. I assume all the forums and changes made to a game over time might be quite confusing for AI. But I've also seen it completely make up characters and decisions and weapons and so on about some very clearly defined games and paths. It's an interesting dynamic.

baron816 4 hours ago||
IMO, the area where AI is going to be most useful over the next couple years is in developing manufacturing processes top to bottom. Maybe a million token budget is too small, but something like "design me a sneaker and all the equipment to manufacture it autonomously".
knollimar 2 hours ago||
They can't even run a vending machine. I don't want to do a shallow dismissal, but I think there's a gulf between my understanding and yours. I hope it's me so I learn something.

I think LLMs will be excellent glue of "find the right function/button and run/push it" but design without constraints and they just explode immediately

toasty228 7 minutes ago|||
Is anyone genuinely excited about these kind of scenarios?
root_axis 24 minutes ago|||
Physical reality is not software or math, LLMs can't design "equipment to manufacture autonomously".
lelanthran 1 hour ago|||
> IMO, the area where AI is going to be most useful over the next couple years is in developing manufacturing processes top to bottom.

Why do you say this? I have some experience of manufacturing processes and am not seeing where AI would be useful other than to drive the robots which we can already do quite well without AI (see all the dark/lights-out factories that already exist).

Where do you see it being useful? An example would be nice.

hgoel 3 hours ago|||
I think embodied AI or autonomous experimentation will need to make a lot more progress before that kind of thing is possible.

Asking AI to design real world objects doesn't work very well because all of its tests involve proxies and thus miss things that are glaringly obvious when the object is actually built.

bagels 3 hours ago|||
You think the bottleneck in creating a sneaker factory is not having an LLM to tell you what machines to buy?
aabhay 4 hours ago|||
Yes but the second order effect of this is that the cost of the tooling goes up since it is now the bottleneck, and therefore the shoemakers that survive do it off of technical complexity, branding, and regulatory capture.
billyp-rva 3 hours ago|||
> "design me a sneaker and all the equipment to manufacture it autonomously"

We already have sneaker designs and the equipment to manufacture them. Whatever it spits out is going to be, at best, a mediocre clone of something that already exists. What exactly is the point?

baron816 2 hours ago||
Sneaker manufacturing is a very manual process. Nike famously tried to automate it and failed (https://www.wsj.com/economy/trade/why-its-so-difficult-for-r...). Designing and constantly changing all the processes is more expensive than just having people do it by hand. If you can have AI design the process, then the equation switches.
Invictus0 4 hours ago||
Gotta love when a techbro just says some complete nonsense like this with total confidence
etdznots 3 hours ago||
Grok build me a spaceship to mars, make no mistakes
xyzsparetimexyz 3 hours ago||
How much are the hobbit houses described in the book? The ones here look exactly like the movie
singron 21 minutes ago|
In the first paragraph (either of chapter 1 or the prologue), not at all. It's pulling everything from pre-training, so it is likely relying just as much on all the visual mediums like the Peter Jackson films, the animated The Hobbit (1977), and all kinds of random depictions of Tolkien's works.

Bilbo's house is actually described in The Hobbit and the exterior isn't really described at all in Fellowship. The prologue of Fellowship (Concerning Hobbits) mentions hobbits like round doors and windows and the fact some hobbit homes are underground, but the turf-dome design here is not mentioned. It actually mentions hobbit homes typically have bulging walls, so unless you've read the The Hobbit, you might not picture this entirely-underground style.

In The Hobbit, his home is described as a (nice) hole in "The Hill" with a perfectly round front door and round windows, which could imply the design here.

xyzsparetimexyz 18 minutes ago||
Right, yeah I asked because I assumed it copied the look from the movie. I think it'd be more interesting to do this for something that hadn't been adapted (yet). The first chapter of Neuromancer perhaps
matsemann 3 hours ago||
I'm pretty tired of the "Y made this game in Z tokens" all over the internet last week. They look impressive, and it's cool that it's even possible, but they're useless as games. None of them are any fun. They're like the most boring variant of basic controllers you can imagine. None have any cool mechanics. None have any tweaks made from hours and hours of testing. All have the same cel-shader.
singron 13 minutes ago||
I've heard this called a "dancing bear" before. It's interesting because it's a bear dancing, not because the dancing is any good.

These conversations get frustrating since one group is saying the dancing is bad, a second group is saying it's good (for a bear), and a third group is saying we are a few years from a bear-only dancing industry.

Aurornis 3 hours ago|||
I’ll never not be amazed that we can type some words and get those results back out.

However I do agree that the results are much worse when you try to use them. They look great in screenshots and video clips which makes them perfect for content farmers.

All of the LLM generated games I’ve played have been really bad to play, though. I even tried my hand at a simple game, thinking I could iterate on it with prompts to fix some rough edges. After the initial productivity burst every change turned into a slog of tokens with one thing changing and something else breaking it. I would try to use my remaining weekly token budget across Anthropic and OpenAI to refine it at the end of every week but after a couple weeks it felt like I wouldn’t be getting anywhere without scrapping it and going back to having the LLM build it one step at a time with my careful instruction.

Which, in retrospect, is the only way I can get usable output of an LLM for anything complicated, so it’s not surprising. It’s a fun reality check project though.

techblueberry 2 hours ago||
I think it’s interesting to watch that we all have to sort of fine tune our own expectations and build the mental model for how impressive this is.

On the one hand I think most of us are incredibly impressed because we know, that quick demo would have taken us months of work to build in the before times.

On the other hand the promise is a cure for cancer and the end of all work.

So when everyone is telling you “skill issue is why you can’t one shot WoW”. It’s hard to know how you’re supposed to feel about Karpathy advertising one shot custom virtual worlds but giving you slop. Incredibly impressive slop when compared to how long it would take to create it just 3 years ago, not so much compared to Elon saying - “by the end of this year, grok will create a version of the odyssey that competes with Nolan’s”

revel 3 hours ago|||
LLMs are bad at creative work and I don’t see them improving any time soon. Try asking an agent to write a story about raccoons. It will almost certainly involve either stealing food or raiding trash with a 50% chance of having a character named Pip. If a location is mentioned, it’ll be Elm Street.

I guess this is the average story and, similarly, the average game is boring and predictable

CuriouslyC 3 hours ago|||
Ironically, they're getting worse at creative work because all the RL is collapsing their distributions.
dofm 2 hours ago||
Worse in a very specific, bland, uninteresting way, too.

Early generative AI at least had the virtue of relentless, unsettling weirdness, in the same way that generative art from the late 90s and early 2000s did. A handful of people made creative use of that spooky weirdness.

Now it turns out "Airspace" art.

MasterScrat 2 hours ago||||
I don't think this is bad. I want tools that deliver what I envision. The fact that given the same input, I get similar output is something I would rather consider a positive. The imagination should come from the human at the wheel.
superdisk 2 hours ago||||
Heh, tried it and got a story about raiding trash starring Pip. But yep, I'm well aware of this problem as well. /r/sillytavernAI is a sub dedicated to trying to coax LLMs into good creative writing but it's kinda impossible to do consistently.
ben_w 2 hours ago||||
While they are bad at creative work*, this reasoning isn't going to show it.

If you took the best, most creative, human writer in the world, and for thought experiment reasons they had amnesia (to mimic AI blank context windows) specifically while you asked them for a story idea 100 times in a row, my expectation is that this human would also give you the same idea at least 80 times out of that 100.

* still better than the mean human, but even the top 0.1% of humans aren't all professional authors.

emp17344 2 hours ago||
Better than the mean human? The mean human creates far better stories while daydreaming. AI enthusiasts have such a distorted view of human capability, it’s bizarre.
Supermancho 1 hour ago||
The qualification "better" is doing some hefty lifting in your assumption.

What you think is better is not what I think is better. Imagination is not storytelling. You ask the "mean human" to write, it's going to be worse than an LLM in spelling and grammar, if you get anything at all.

emp17344 1 hour ago||
Daydreaming is absolutely storytelling. The average person is able to craft elaborate stories on a whim. Writing is used, in part, for conveying stories, but writing is a separate skill. The good news is the average person can learn to write well, but LLMs, so far, have not demonstrated the ability to craft interesting stories and it’s pretty unlikely they’ll be able to learn, as this is an unverifiable domain.
jmole 3 hours ago||||
Seriously, what is the deal with Pip! I was experimenting with childrens' fiction more than a year ago and 9 times out of 10 you'd get a character named Pip unless you were specific in the prompt not to do so.
cautiouscat 3 hours ago||
I recently saw a study on this actually. https://arxiv.org/abs/2605.26492
dofm 2 hours ago||
I saw that, tested it in Qwen. Elias Thorne, on the first run, as a clockmaker too.
josh-wrale 3 hours ago||||
I think the trick is to use LLMs like scalpels instead of hammers. That said, you can place a scalpel in the hands of a deterministic (or not) robot.
dofm 3 hours ago|||
Elias Thorne!
Jtarii 3 hours ago|||
LLMs currently have 0 imagination, I have strong doubts that will be solved any time soon even if they continue to become superhuman at everything else. They just have the creative instincts of a 50 year old accountant.
egeozcan 3 hours ago||
> They just have the creative instincts of a 50 year old accountant

That's oddly specific and it'd really hurt my cousins feelings :)

dofm 2 hours ago||
Especially since the whole AI IPO concept seems at least somewhat predicated on creative accounting! ;-)
8n4vidtmkvmk 3 hours ago|||
We're all just impressed that it's even possible. No one is saying these are fun.

I guess the next question is if they can be made fun without too much additional work with a human guiding the AI.

Jtarii 3 hours ago|||
How well encoded is "good game feel" in the weights of an LLM. I am guessing not very well.
tripleee 3 hours ago||||
> the next question is if they can be made fun without too much additional work

They can't. If you think about how these things are trained it's blatantly obvious fun is an impossible metric to optimize them for

egeozcan 3 hours ago||
Training impossibilities aside, how would you even make an optimization loop for "fun"?
tripleee 3 hours ago||
Brain-computer interface, maybe. Hook a million play-testers up to the output and have it iterate.

..feels like there was a black mirror episode about that though

minimaxir 3 hours ago|||
Unfortunately people are saying they are fun for the purposes of ragebait that goes viral, which is half the reason people intentionally post AI slop on social media.
eterm 2 hours ago|||
Right, but the new phenomena is that we've lost a signal.

Used to be if a game looked that good it probably had time spent on the game part too.

dofm 2 hours ago|||
And you could almost always tell a "game construction kit" game from a real one, couldn't you? Whatever was missing from those is always missing from these, and it seems just as rare that the game will reach much beyond its origins.

Whenever I think about AI games I find myself thinking about Tiny Wings, Flappy Bird and Angry Birds. Three simple, elegant games.

It is easy to see what makes Tiny Wings so completely loveable — it has a sculpted, adorable, perfect charm with a cleverly inverted game mechanic that has a calibrated level of exasperation and reward.

But why were Flappy Bird and Angry Birds, very basic games with very old game mechanics, so charming?

It seems equally impossible to imagine an AI coming up with a game with the quality of any of them, even with maximised creativity. But explaining why for Flappy Bird seems quite difficult, especially when you consider it uses some stolen visuals!

knollimar 2 hours ago||
it's crazy because I've seen "implement flappy bird on an FPGA with VGA output" as a 2 week project to a human as something immediately understandable.

Maybe it's the smoothness of motion that makes these games understandable and LLMs seem to consistently fail at that. Ask them to do something snowboarding and they go really hard on the physics since it seems like they don't know what kind of approxmations feel good.

ben_w 2 hours ago|||
I've been hearing similar from a while back, but the blame wasn't being directed at AI, it was directed at Unreal's defaults now making random indies look like AAA in the screenshots.

(It's been a while since I was in the game industry, so IDK quite how accurate this is).

throwatdem12311 3 hours ago|||
It’s called “demo porn” and it’s really obnoxious and borderline offensive to people that take the craft and art form of video games seriously.

If I see one more “one shot MMO” where you just walk around and do absolutey nothing or another menu slop idle battler or rogulike deck builder I’m going to go Postal in Minecraft.

tripleee 3 hours ago|||
It's the flashy game/movie trailer equivalent for LLMs. Completely unrelated to the real experience, but good for marketing.
upmostly 3 hours ago|||
Unfortunately the Phaser framework has decided to go all-in on this route, and it's really disappointing.

These one-shot products aren't games. They're barely even demos. I don't even know what to call them. For a mature framework like Phaser to sell-out like this and create a vibecoded platform for vibecoded games is shocking.

moron4hire 3 hours ago|||
It's blockchain for gaming all over again, putting the cart before the horse.
segmondy 3 hours ago||
you're tired because you lack imagination.
throwatdem12311 3 hours ago||
It’s the people spamming these shitty games that lack imagination.
segmondy 2 hours ago||
people are excited, let them be excited! but more than the excitement is realizing the implications of what this means for building software, first it looks like a toy and then it doesn't. the OP is talking about how the games are not playable and fun. who cares? that's not the point.
matsemann 35 minutes ago||
What does it mean for building software? These games cannot become fun. There is no way to prompt it fun, and any change you try to prompt it to make will inevitably blow up something else. The code is a mess and impossible to build upon.
eichin 2 hours ago||
Is anyone else getting "mongodb is webscale" vibes? (Except 16 years ago that was a lot smoother, because it used some sort of "render this conversation" engine...)
barrenko 2 hours ago||
This has started to feel a bit like the beginning of railroads and then the steampunk fiction of "let's just build railroads to everywhere". We don't need it and there's no use for it.

As with painting, after a while there's nothing really new to paint, we genuinely need 0 new software. We need to fix our broken physical world, our social lives, our kids and what's left of our democracies.

This software crap is done, leave it to the nerds.

fasterik 1 hour ago||
Agreed that software isn't going to fix our institutions. But we need 0 new software? AlphaFold won its creators a Nobel prize in chemistry and solved a research problem that feeds into every area of biology, medical research, and drug discovery. We still have an untold number of unsolved problems related to human health, food production, energy production, infrastructure, transportation, education... The list is practically infinite.
barrenko 49 minutes ago||
Agree for stuff like alphafold, for the likes of education, imho, it's been solved for at least a couple of centuries.

Textbook and blackboard > ipad.

fasterik 43 minutes ago||
For childhood education sure, we don't really need high tech solutions there. But what about training the next generation of mathematicians, physicists, and engineers? Even decades ago, computer algebra systems and numerical solvers started to become indispensable, at least in many subfields. Now we're moving into the territory of automated proofs of mathematical conjectures. The state of the art is going to keep improving, and education is going to have to adapt to keep pace.
morbicer 2 hours ago||
Beautifully put. Software is ok. It's not going to fix our broken world. If AI could at least find cure for some illnesses that would be great. Or decrease social injustice and help with global warming. But it's likely going to do the exact opposite.
cocoa19 3 hours ago|
We must not be using the same opus 5, because if I tried to generate this it would refuse based on copyright grounds.
throwatdem12311 2 hours ago||
Karpathy works for Anthropic so he obviously has full access to everything, and doesn’t have to pay for token burn either.
knollimar 2 hours ago||
I saw that "~free" and laughed.
More comments...