Very nice comparison but constrained to greenfield.

Would love to see a similar article that uses LLMs to add a feature to Gimp, or Blender.

They nerfed it as of sunday March 30, a lot of people noticed performance drop and it rambling.

https://x.com/nisten/status/1906141823631769983

Would be nice if this review actually wrote exactly when they conducted their test.

I still can't get any LLM to use my niche API and build out API REST requests for all the endpoints. It just makes stuff up even though it knows the api documentation. As soon as one can do that, I'll be sold. until then I feel like its all coding problems its seen in github or source code somewhere.

The amazing bit about claude code is it's ability to read code, and fit into the existing code base. I tried visual studio code w/ roo, and it blew up my 50 daily request limit immediately. Any suggestions on better tooling for a claude code like experience with Gemeni 2.5 pro?

I tried the exact prompt and model from the blog post, but my outputs were way off—anyone else see this? This is the best of 3 output of flight simulator prompt (gemini 2.5 pro (experimental)):

https://imgur.com/0uwRbMp

>Minecraft-styled block buildings

The buildings weren't minecraft style in either case. They weren't formed on a voxel grid and the textures weren't 16x16, but rather a rectangle or at least stretched to one. Also buildings typically are not just built as a cuboid.

Just a moment ago I tried to use Gemini 2.5 (in Cursor) to use Python Gemini SDK. It failed, even with a few iterations.

Then run Claude 3.7 - it worked fine.

So yeah, depends on the case. But I am surprised that model creators don't put extra effort into dealing with setting their own tools.

Sorry for the noob question, but claude has claudecode, does Gemini Pro work with any software in the same way "claudecode" works? If so what software would I use with it? Thank you.

This is interesting but too greenfield, someone should do one with an existing OSS project and try to add a feature or fix a bug.

Model is insane but the RPM limit is insane too.