evals are glorified integration tests, would you invest in an integration test startup? absolutely not. I don't get why we are making all of this fuzz around evals

Everything eventually fails. Nothing is constant, not even evals.

> Not enough eval customers

Aha.

Because they operate on untrusted input

I found this pretty hard to read as the author has a very specific understanding of what an eval startup means but it is only implied rather than explicitly described. I would have thought that they were referring to the companies that provide a technology platform to enable you to do evals in an AI application context for example companies like Comet/Opik and Braintrust.

But it sounds like the author does not mean those companies at all since those are actually important in enabling the very Venn diagram he/she describes.

Based on what I assume the author's referring to they are referring to something more like a public benchmark report provider... I would say but yes that's a relatively small total addressable Market space no matter how you look at it

Wtf is an eval startup?

[flagged]

[dead]