Posted by __cayenne__ 8 hours ago
I think with the abilities of AI improving, playing games would be a good benchmark as long as they're not trained on them.
It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.
They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)
But right now the response time would be so slow that the bot would not be close to competitive except with the most low level players. Especially because its hard to click on things that are moving at the same time as well.
Sometime soon, I'd like to see an LLM be competitive using only computer vision, mouse movement, and locked into a human possible number of actions per minute.
TLDR they suck and many default to doing probe rushes.
You can train a specialized model for this, but a frontier model is ill-fitting.