This might be the most annoying habit of corporarte AI that it might be one of the few industries that goes around demanding everyone else provides clear use cases and proof of efficacy for it.

1. If the benchmarks are just testing the ability to get the answers from history then something is clearly wrong with the benchmark.

2. If that's even a possibility then that's going to lower confidence in the ability to deal with the vast majority of problems where you don't already have the answer written down.

3. That's not the customers problem to solve on behalf of the vendor.

So instead of calling out the cheaters we victim blame the benchmarks for leaving traces of exploits?

The linked ticket isn’t suggesting the commit is in the training data. It’s demonstrating that models run ‘git log’, find the exact code to fix the issue against which they’ll be scored, and then they implement that code as-is.

The test environment contains the answers to the questions.

Why does this matter if these models are a super intelligence with reasoning etc and don't need the answers sucked off the internet?

Well, we're dealing with (near) superintelligence here, according to the companies that created the models. Not only would I expect them to regurgitate the answers they were trained on, which includes practically the entire internet, but I would expect them to answer questions they weren't trained on. Maybe not with 100% accuracy, but certainly much higher than they do now.

It's perfectly reasonable to expect a level of performance concordant with the marketing of these tools. Claiming this is superintelligence, while also excusing its poor performance is dishonest and false advertising.

No because that's the baseline. It's what you do when you have no other choice. Railing against that would be pointless.

Lol...That contract has Oracle offering licenses at a discount of 75% and is estimated to make them not more than one 1 Billion. The other big contract on Cloud services the DoD JWCC is $8B to 9B but shared by four vendors (AWS, Microsoft, Google, Oracle) and Oracle orders under it are in the hundreds of millions not even 1 Billion...

Wall Street is currently heavily punishing any company who misses their quarter, including NVIDIA!, after beating on their quarter.

Oracle had a earnings miss in the current quarter!

Their current REALITY is ~$15B quarterly revenue (with cloud infra ~$3B) and only ~$12B in near-term deferred backlog and deferred backlog is NOT revenue. To justify the valuation, this would imply OCI going from ~$18B in FY26 to ~$140B by FY30 that is an insane promise of +$120B in 4 years but back-loaded into the year 3 or year 4. :-))

Capex needs ~$35B next year just to chase GPUs/power and if they miss one quarter the story implodes. The supposed rational, efficient market, is paying near $1T today for back-loaded hopes.

Is completely bubble math. Like anybody, including Oracle AND their Customers, have ANY idea of their Capex in 4 years.

Complete and total bubble.