Posted by plurby 1 day ago
When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning (Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.
The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).
Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.
Anecdotal but I've seen it use python to crop, zoom, and "enhance" (fiddle with sharpness and brightness) images to read sections of handwritten census data from the 1800s. Feels like that there might just be a mismatch of capabilities when it comes to straight outputting coordinates but I bet the model is better at actually finding the answer given any tools available. Which I get is a bit of an apples and oranges situation.
I've also tried to use it to identify an old pair of glasses and it didn't stand a chance, so I do think it's not quite there yet when it comes to some vision tasks.
Overall I've been hooked on using agents from different companies for what they are best at (Thanks to Theo). Fable is expensive, but unmatched for planning and top level organization of other agents. Sol is fast, will persistantly go after goals (sometimes to its detriment), and does well with computer use.
Also it seems to be more capable, need to test more, but I think it's at least getting on par and it's fully open-source and open-weights.
Here's some benchmarks:
https://benchlm.ai/compare/gpt-5-6-sol-vs-qwen3-8-max
https://qwen.ai/blog?id=qwen3.8#full-benchmark-table (incredible UI/UX demos)
https://venturebeat.com/technology/qwen3-8-max-arrives-with-...
EDIT: Am I early to the discussion, or is none else using Qwen3.8-max?
Does YC have similar problems like those at wikipedia/reddit? (wikipedia-editor-wars, or reddit-mod-wars)
I concur with your conclusion, Qwen 3.8 has exceptional Vision Capabilities. The other commenter mentioned "I thought Qwen 3.8 didn't have Vision", it does, just not on the open weights version, only via official API.
In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.
To use an LLM, you just prompt it with an image + text saying "count the pills in this image".
To use OpenCV, ... you just prompt an LLM with an image + text saying "count the pills in this image, using OpenCV instead of eyeballing it".
(I like to throw in "produce intermediary artifacts so I can see the process" for more difficult tasks; this helps the model avoiding making hallucination-prone leaps and gives more opportunities to self-correct. At a cost of extra time and tokens, of course.)
Using OpenCV without an LLM? Nah, not touching that, I don't have free weekends to waste anymore.
It's not an LLM, it's a custom thing we built. Here's a comprehensive list of support for various notation glyphs: https://www.soundslice.com/help/en/creating/pdf-import/294/s...
For example, a UI / UX professional being asked to appraise a website screenshot may determine that the image in question has "desirable" traits which are inherently not deterministically measurable. Such as, if the interface elements have strong information hierarchy, or if they are deemed to be "fashionable" with current UI trends.
...but that's an example of a UX/usability matter that can be assessed objectively and non-subjectively.
Is 16 px or 14 px a better font-size value for a subheading, in a hypothetical layout? Immediately that kind of decision, where both options are objectively good for 12 px paragraph text, suddenly becomes an issue of taste that cannot be evaluated crudely by an algorithm.
But, if you have a proper well documented design system and you tell the LLM to use the DS and to avoid styling hacks they can generally do it. Even the dumber ones than Sol 5.6.
Of course only if the design is achievable in the design system.