If the highest tier model makes even one or two fewer mistakes that require human intervantion, that almost makes it a no-brainer choice for production.
At least for my use case of evaluations of a set of features.
I'm building stuff at home with my work focused on managing infra. If anyone wants to sponsor a SOTA model subscription I will check it out. Until then I'll see how flexible a Strix Halo setup and my homelab can be.
I think many people still expect their tasks to need something more fancy, but could do with a reset of expectations and try to go much lower.
On the AA benchmark it kinda makes Sonnet and Terra look weak. Hopefully that translates to real world usage. Can’t wait to try it out.
Comparisons:
Hydrogen: https://html.non.io/lcars-gemini-4-hydrogen
Sonnet 5.5: https://html.non.io/lcars-sonnet-5.5/
Original designs: https://diffui.ai/app/canvas/5093e689-1e74-4f26-b632-2a4500f...
IMO it sits between sonnet and opus in terms of quality, and is equivalent with gpt sol (for this particular benchmark). Google did a great job with this.
For pro sub, Argon is still very limited and Carbon is only available on Ultra. This is not competitive at all. To say nothing of also being locked into Antigravity.
My use case benchmarks 37% more expensive per task but 40-60% of the total time per task (median and p90 had the same improvement).
Both models were smart enough to complete all eval tasks, speed was the main thing we were struggling with. Both models on medium reasoning.
Overall feels like a good upgrade.
My TL;DR is: seems like a better choice than Sonnet in areas outside of coding, and at high reasoning it's smarter than Terra but somewhat more expensive.