I run the free service https://countrx.app/ so i have some idea what goes into counting.

The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models.

Their feedback loop of training on user data is incredibly strong. I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set.

Or the future will develop in llms using fine-tuned models as tools? Inference cost and speed does still seem to be below user expectations.

But for being able to one shot with this accuracy... IMPRESSIVE

I really like the combo 5.6 Luna & Sol for price and performance and would be perfectly happy if they stayed here for a moment without mucking about with sidegrades that I think AI evolution has often felt like lately.

It’s gotten so good that I now have infrastructure to render all mermaid/plantuml in my project to png and have AI’s always load both text and image versions. And they are instructed to review the rendering as part of the diagramming cycle (for layout, salience, usefulness, etc). They can now produce useful diagrams that help reach shared architecture understanding.

All of your use cases are very advanced.

I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.

I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.

I will add that all frontier models were fine with such tasks from the early 2024's.

I wonder what about UI review, which model is the best?

"Best iPhone ever" vibes.

I would love more vision benchmarks! Once I asked the model to inspect a completely black picture and it hallucinated a nice wooden kitchen wall. Took me some time to figure out where the kitchen came from...

I usually go to https://arena.ai/leaderboard/vision/pareto for a nice overview of current models.

5.6 Sol looks nice, but the Gemini 3.5 Flash comparison is interesting. It’s cheaper and still came out ahead on detection and counting, which doesn't really give me much of a reason to use Sol since Flash is much cheaper and hence much easier to scale. Not to mention we now have 3.6 Flash too

It's vision capabilities poisoned my cucumber bed, misidentifying the malaise and having me spray them down with water, which only spread the fungus that gemini later informed me was actual cause, which I went and checked myself.

I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.

One of the use cases I've wondered about for AI is giving it a picture of the "spice wall" in a grocery store and asking it to find all jars of e.g. cardamom. This takes me an annoyingly long time to do when I'm shopping, so it would actually be useful.