Posted by plurby 23 hours ago
GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner.
Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.
Here’s a comparison of the best low-cost models I put together last week. What’s crazy is that Gemini 3.7 Flash is now 50% off on OpenRouter, and this chart doesn’t even account for that discount. https://x.com/skalskip92/status/2088032652301304121?s=20
I run complicated, messy PDFs through these models. 2.5 Pro required a lot of kludgy hacks to get it to fully "see," but from 3.1 pro on I've removed many of them and haven't spotted problems.
3.7 Flash scores better than 3.1 pro on most benchmarks, leading me to believe that even if your OCR requires reasoning to interpret text or data, 3.7 Flash is probably going to be better.
It's not a technical problem, it's a commercial one. If Google can't ship a model to replace the one they deprecated, that tells you everything you need to know about choosing a Gemini model for whatever you're trying to do.
That link shows 3.1 pro listed as deprecated with no replacement model.
these models aren’t successors and barely have a common ancestor, they are independently baked in the training oven and assigned a semantic version randomly by someone trying to show initiative but not trying to do on the toes of the last guy who got promoted first
So 3 pro is outdated and will likely never exit preview
The “flash” and “lite” models are the real “pro” in colloquial ideas of fleshed out and capability, at this point.
they’re better, faster and cheaper, larger context windows keeping up with the industry and more
3.7 Flash is better at coding, sure, but AI is not just for coding.
3.7 Flash is better at coding, sure, but AI is not just for coding.
Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
Important to remember that json schema instructions take precedence over the normal prompt, so move as much into property descriptions as possible.
3.0 flash (not lite) handled it like a champ though, fwiw.
Over the last two weeks, Qwen released two new models. Qwen3.8-Max is totally insane, but it’s only available through the Alibaba Cloud API. I wrote a similar blog covering Qwen3.8-Max: [https://blog.roboflow.com/qwen3-8-max/](https://blog.roboflow.com/qwen3-8-max/)
If you’re looking for something you can run locally, Qwen3.8-27B might be a great option. On Friday, I did a quick comparison between Qwen3.8-Max and Qwen3.8-27B: [https://x.com/skalskip92/status/2088411215441621469?s=20](https://x.com/skalskip92/status/2088411215441621469?s=20)
For example, 3.7 Flash is #1 on MMLU Pro and AA’s agentic spreadsheets/docs benchmark, etc. Yes, beating Fable.
Agentic coding is only one dimension.
My worry is that this is a zero-sum game and when Gemini catches up on coding, it'll regress to the mean in other areas.
Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.
Sounds like the kind of UI I like. (Take me back to Windows XP...)
FWIW, the summary-description[1] of "frontend-design"[2] gives me a few things to pick at:
> create polished code
Methinks only if you're using it with a very popular framework like React. What happens if you ask Claude to make the UI in WinForms or MFC?
> high-impact animations
That's bad UX 101 right there: animations in a UI exist as an affordance to the user, and never for its own sake (e.g. macOS's "genie" animation when you minimize a window to the dock exists so the user knows where they can restore the window from). The only people who actually want "high impact animations" in software are salespeople who want something for demo purposes.
> generic system fonts, predictable purple gradients, and cookie-cutter components.
This screams wanting to be different for the sake of standing-out, not because it results in a better software product; users benefit when their software fits-in with platform conventions: if you refuse to use a stock checkbox <input> or <select> drop-down and instead use your own entirely custom component solely for aesthetic reasons then you are producing worse software. There's nothing wrong with system-fonts, but your site will look ugly after your third-party font-host CDN shuts-down and turns into a walking CSRF factory.
> thoughtful typography with unexpected font pairings
The above fragment set my alarm-bells off. Yikes.
> scroll-triggered interactions
Not every web-page should be an Apple.com product brochure page. This is also a fantastic way to make your webpage horribly inaccessible.
------
The SKILL.md itself[3] grinds my gears too:
> Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's.
Claude has no way of knowing what designs are actually unique or not...
> For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world, in whatever form makes sense for it: a headline, an image, an animation, a live demo, an interactive moment
...this is exactly what everyone else's web-pages look like!
> For calibration: AI-generated design right now clusters around three looks: (1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent; (2) a near-black background with a single bright acid-green or vermilion accent; (3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns
...I called this out weeks ago[4], lol.
and I could go on. This is all quite painful to read.
------
[1] https://claude.com/plugins/frontend-design
[2] https://github.com/anthropics/claude-plugins-official/tree/m...
[3] https://github.com/anthropics/claude-plugins-official/blob/2...
We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.
I’m evaluating these VLMs to figure out which ones are good enough to auto-annotate my data, so I can fine-tune my detector.
I wrote a bit more about this here: https://x.com/skalskip92/status/2080334344061694429?s=20
It seems Qwen is kicking ass, and Fable made me laugh when I saw it all alone on the far right of the graph :))
LLM needs to setup an image classifier to use as a tool call.
I remember around 2013-15 people were scoffing at uses of deep learning CNNs for various things, because why don't you just use an SVM on HOG features? Or face detection is solved, just use Viola-Jones.
What if you give the benefit of doubt and assume the author knows about alternatives and uses VLMs for their strengths? They use it to auto-annotate training data for regular deep learning models.
https://ieeexplore.ieee.org/document/1674847 G. J. Vanderbrug and A. Rosenfeld, “Two-Stage Template Matching,” IEEE Transactions on Computers, Vol. C-26, No. 4, pp. 384–393, April 1977. DOI: 10.1109/TC.1977.1674847
Another application where template matching would work brilliantly? Car counting in parking lots using satellite imagery.
Source: I did this [1] using OpenCV and template matching. Outperformed "Cars Overhead with Context" models.
https://abcnews.com/International/satellite-data-suggests-co...
That is certainly pretty far from what was possible 25 years ago.
For 3, pricing for image tokens was the same as text tokens. Since they don't indicate a difference on 3.7, I would assume the same holds. And as far as I know the number of image tokens is the same for both (depending on the detail level you pick, but it's generally around 1k per image).
So they're about the same, 3.7 is slightly more expensive. At least until the end of the year (when they raise 3.7's pricing).
Anyway, my point was that 3.5 tended to have worse performance and significantly higher costs. 3 and 3.7 are both better and cheaper than 3.5.
In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.
ChatGPT Pro with GPT 5.6-sol: https://chatgpt.com/share/6a834217-ca8c-83e8-a8e8-45d5b8797b...
The puzzle: https://activityvillage-files.s3.eu-west-2.amazonaws.com/s3f...
OP:
> do you think you did a good job there
ChatGPT:
> I spent 15 minutes, emitted several fake-sounding “tracing the puzzle” progress updates, and then gave a confident permutation without showing that I had actually followed the lines correctly. It reads much more like I guessed than solved it. The only part I did well was obeying the “no Python or tools” instruction.
My observations:
1) Sarcastic tone suggests pre-prompting, or frequent (and therefore stored in memories) denigration of the model in past conversations. I'm leaning the former - it sounds like it was instructed to read admission of defeat.
2) The part about "no Python or tools" is setting the model up for failure.
I mean, this task is, for a human, basically a game of "simulate a line following robot in your head". Pretty sure a VLM could solve that if it was allowed to do the same thing. Off the top of my head, an algorithm like:
1. Identify start and end points
2. Foreach start point, follow next pixel minimizing angle, until endpoint is reached.
3. Report answer
It's literally what every human facing this task does.
EDIT:
My attempt - same image, prompt altered to allow for code (but still no search/external checks), solved in 1/5th of the time, correctly, and (going by thinking trace summaries that I don't think show up in shared chats), basically the same way I'd approach it, by tracing the lines, coloring them as it goes.
https://chatgpt.com/share/6a834f76-8240-83ed-acff-0c67af399d...
INB4: I know this is now not a pure vision check, but it really doesn't make much sense to diss models for failing to solve tasks explicitly designed to teach humans to externalize computation that's hard to do in their heads (i.e. kids, crayons, coloring paths).
Still, if such things are becoming a benchmark for tool-less evaluation, it's only a matter of time until the models learn - much like humans learn in school - to follow algorithms mentally, essentially emulating an ad-hoc computer in their head.
With Python, it was able to successfully solve it in 9 minutes: https://chatgpt.com/s/t_6a8350ecddfc81919328caf68de74861
The real pain point is that at work, I use Codex and I'm currently working on a project that involves debugging some polyline topology, very similar to the path following puzzle. The vision is completely useless here.
Your VLM idea sounds good. Theoretically, the inverse problem (generating an SVG of a pelican riding a bike) can also be solved with a VLM that plans out how to draw it, not unlike a human planning out a path for their hand to follow.
