Posted by screm 8 hours ago
Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked.
To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or senior engineers) in different sizes of companies.
All the results are now public and we'd love to know what findings surprise you the most, here are a few we found interesting: - Claude Code rarely searches the web while Codex almost always does it and Cursor sits in the middle. - Coding agents disagree more frequently than they agree. - Some players (LangChain, Supabase, Netlify, Paypal, Adyen) are almost always mentioned in their categories but never chosen. - Modifying repository context can change the pick entirely.
If you feel like digging, all the traces are there and we probably missed interesting learnings so let us know what you find!
I’m sure you - or Claude - built something you’re proud of. But I left your website frustrated.
I’m trying to understand why they are opposite. I think it is true, I find myself giving a secondary prompt to Claude to “research this” and only then will it fetch. Codex is bang on fetching already.
By tool I mean mcp server, cli, etc.
You are perpetuating shitty practices that have hurt developers for years now. Part of the reason people use AI is because of how useless search is due to the previous generation doing the same kind of thing you propose.
Please do something else with your life.
A large portion of the utility of AI is the barren ad-driven growth-hacked hellscape search has become. Don't encourage the next generation of these businesses, I beg of everyone.
What can man do against such reckless ~hate~ money?
If we felt like we couldn't trust AI because of slop, soon we won't be able to trust it because it'll push whatever pays them to do it.