Posted by gumby 6 hours ago
Did anyone solve this problem? Any good approaches?
Something like continuously delivery and validation through the whole stack so you can develop an incremental change, validate it as much as you can locally, then validate it as much as you can in the cloud using integration tests with synthetic, then with replayed production, and then finally allow it become production. If stuff starts breaking, you have your AI swam go in, identify the culprit, bounce it, and the train never stops.
In any non-trivial integration test scenario, it's easy to end up several times the LoC in tests compared to actual shipped code.
Second problem: different models, agents, and harnesses seem to have different tendencies and they tend to avoid touching code that is not their own unless explicitly instructed. What I mean by this: it will happily duplicate some helper method from another test file rather than pulling it out and reusing it, seemingly avoiding coupling (good kind in this case since model level changes should be propagated atomically).
Tests are where I see agents just spew absolute mountains of code.
I think this approach will continue to work for some people and orgs, but others are going to hit walls where their agents can't manage the mountains anymore. It's going to be interesting to see how people and orgs respond when that happens.
> I've asked them what their plan is when 500 of their 1000 test suddenly stop passing
Bigger problems to me are more practical: 1) large tests waste context both in and out, 2) large volume of tests waste CI cycles which are becoming the bottleneck.
But more specifically, I usually see the explosion of test code in integration tests where there is heavy setup of entity graphs.
https://github.com/zeeq-ai/zeeq-biome
Was initially hand written as part of our codebase before AI, just extracted it as MIT, OSS since there's no secret sauce here. But really cuts down on test verbosity and makes it much, much easier to read, review, and evaluate tests.
The reality is that new apps and new companies are being built without the bottleneck, constantly now.
Look at the shipping speed of the frontier labs themselves. It’s all the evidence you need to know that this isn’t true.
I certainly look at how their software sucks, and wonder why their fans think the rest of us should adopt such a bad approach to making software.