This benchmark is shitty because it puts Gemini in 3rd position.

I tried gemini on simple code base and it invoked 210 tool calls just to update 3 lines of code.

real software engineering benchmark is how much stress you can take at work. Everyone knows this, bakka.

Wake me up when September ends or when I can do this locally.

I think these benchmarks are not that useful, e.g. this suggests Fable is better than Astra, but in practice Astra is waaaaaay faster (like 5x; it's not even close), and also waaaay less annoying to talk to.

There's only two or three sane options here - you can easily try them all and pick yourself.

[flagged]

[flagged]

[flagged]

[dead]

[dead]