Wonder what would happen if we unleashed Karpathy’s autoresearch on the pelican bicycle test. And had it read back the image to judge it.

Oh maybe it might continue to iterate on the existing drawing?

This is a useless benchmark now a days, every model provider trains their models on making good pelicans. Some have even trained every combination of animal/mode of transportation

I'm really curious about what competes with Claude Code to drive a local LLM like Qwen 3.6?

I've been using Qwen3.5-35B-A3B for a bit via open code and oMLX on M5 Max with 128Gb of RAM and I have to say it's impressively good for a model of that size. I've seen a huge jump in the quality of the tool calls and how well it handles the agentic workflow.

I really wish they spent some time training for computer use. This model is incapable of finding anywhere near the correct x,y coordinate of a simple object in a picture.

Between the legs and the beak I'd still rate the opus pelican higher

I love this benchmark!

That Qwen flamingo on the unicycle is actually quite good. A work of art.

looks like opus have been nerfed from day1

All those models that were just at version 1.x in 2024

That’s so wild