Posted by Bluestein 1 day ago
Related, more recent post (today) from the author:
Everything around LLMs is still magical and wishful thinking
First, an example I saw online
https://www.reddit.com/r/WebVR/comments/1lh9d3h/webvr_game_c...
I've seen others. They seemed impressive.
In particular, I thought it might be fun to ask Gemini to make a fireworks display for the 4th of July. What I imagined was Fantavision
https://www.youtube.com/watch?v=pD2AnDhyynM
I didn't expect it to get there but I did hope one a scale of 1 to 10 (10 = Fantavision) it would get to 6 ish. I'd say it got to 3 of 10.
What I got was some cubes with yellow planes on the side for buildings. This means things are massively inefficient since instead of a simple textured cube for each building, each building is 1000s of polygons, 2 for each window. Lots of Z fighting. I asked for a river with reflections. The reflections were practically invisible. I think it was generating reflection fireworks under the water plane, not actually "reflecting" the ones above. Because they were under the plane they were faded out because of blending which is not how reflections of fireworks work with water. Then, the fireworks themsevles were horrible, made with 1 pixel particles, no glow, no trails, no substance. They were further very un-random in their velocities and looked nothing like real fireworks.
I tried for 90 minutes to get it to make something better but mostly failed. I did manage to get it to stop using pixels for particles and use textured quads.
It's first version started with "click to launch fireworks". That didn't work at all, blank screen. I told it I didn't want to launch firework, I just wanted them to auto-launch. It made a new one, this time it had a caption "with auto launching fireworks" (and still blank). I told it there was no point in such a caption as it would be obvious from seeing the fireworks that they were auto-launching. It got rid of that caption but still a blank screen. I started over, this time adding "auto-launching, auto camera" type elements to my intial prompt.
I tried several times to get it to make an interesting city around both sides of a river (because this was supposed to be an NYC fireworks display). It failed in all kinds of ways. I'd asked it to fly the camera around the fireworks and never have it go behind any buildings. I put the camera in the buildings so you couldn't see any fireworks. I told it "not behind the builings". It moved the camera further way, twice. I finally got it to put the camera only above the river between the two shorelines (if you can call a plane with some cubes on both sides a river with shorelines). The other way it failed was it just got too slow. The way it was inefficiently making buildings mean it started running at 3fps. I started over.
At least twice it started with code that had an error. I'd click "Fix Error" and it would fail to fix it. I'd click "Fix Error", repeat 5-6 times. Eventually give up and start over with a slightly different prompt.
I tried asking Gemini to write the prompt as someone suggestion. First use gemini to write the prompt. Then paste that into to Gemini Canvas mode. It didn't help nor generate anything noticable different than my hand written prompt.
The issue I spent the most time on, it would "launch a firework" (which it referred to as a rocket). The "rocket" (a particle? a sphere? no idea) would go straight up verically (as in y += velocity * deltaTime). As it did it would leave a trail of static exhaust particles just hanging in the air motionless. The firework would then explode and all of the static hanging in the air exhaust particles from the rocket trail would unfreeze fall a little and fade out.
My goal was to get those particles to fall and fade as they were emitted, not wait for the rocket to hit it's peak. I tried 15 or so prompts but it utterly failed to understand the problem. It kept tweaking the velocity of the rocket trail particles after the firework explodes. It never got that them hanging static in the air was the issue no matter how many times or ways I explained the issue. Ever time it would just adjust the velocity of the trail particles, after the firefox exploded and make up nonsense about how it "solved the issue now".
If I'd gotten that working I'd have asked it to stop launching the rockets straight vertically but give them a parabolic simple physics based arc.
I could have done the entire thing myself in 15-30 minutes and got exactly what I wanted. Instead I faught with non-understanding LLM right out of the a Sci-Fi comedy movie.
I know this will work some day and I've seen examples of it working. Still ... ugh for now.
I know, I know. California changed their law so coders could puff themselves up and put on airs and make their resumes look more impressive. But this is a sad anomaly.
But perhaps not in CA.