This has to be an intentional echoing of the "pace the frontier" language?
I tried to figure out if I could reproduce Carbon's aggregate score using yesterday's (v4.3.2's) benchmark weights—but apparently you can't actually do that, without paying AAI. The raw numbers going into that aggregate aren't available on their public website.
Again, no access to open source developers. The AI companies want bugs for dramatic public relations! Also, of course there aren't enough bugs so they have to be rationed for the "defenders" and future models.
https://blog.google/innovation-and-ai/models-and-research/ge...
https://blog.google/innovation-and-ai/models-and-research/ge...