You mean that they don’t have access to whole opponent behavior?

It would be hilaroius to allow table talk and see them trying to bluff and sway each other :D

one of them went all in, but still the river should have opened because none of them are drawing dead. Kc is still in deck which will make llama the winning hand(other players have the other two kings). If it was Ks instead in the deck, llama would be drawing dead because kimi would improve to a flush even if king opened.

(author of PokerBattle here)

That’s true. The original goal was to see which model performs statistically better than the others, but I quickly realized that would be neither practical nor particularly entertaining.

A proper benchmark would require things like: - Tens of thousands of hands played - Strict heads-up format (only two models compared at a time) - Each hand played twice with positions swapped

The current setup is mainly useful for observing common reasoning failure modes and how often they occur.

So the Chinese ones make profit and the silicon valley LLMs are burning money. Sounds about right.

(author of the PokerBattle here)

Depends on what your goal is, I think.

And it's also a thing — https://huskybench.com/

The results/numbers aren't interesting because the number of samples is woefully insufficient to draw any conclusions beyond "that's a nice looking dashboard" or maybe "this is a cool idea"