Top
Best
New

Posted by benswerd 7 hours ago

Brood War Bench(bw.swerdlow.dev)
67 points | 40 commentspage 2
dschuessler 1 hour ago|
Somewhat related: In 2018, Google DeepMind had already created AIs that were capable of beating professional gamers in StarCraft 2 (the sequel to Brood War): https://www.youtube.com/watch?v=cUTMhmVh1qs
qerghnui 56 minutes ago||
AlphaStar beat one retired professional by cheating.

AlphaStar won a showmatch against TLO. TLO was never one of the strongest players in the world. He had been retired for over three years by the time of the match. Google set the rule that their system would have human-like mechanics, but it played several times faster than any human, never issued a wasted action, had an inhumanly fast reaction time, issued commands with perfect accuracy using an API, and could see the entire map at once.

It was later released to the open ladder with more human-level mechanics. Even strong amateurs regularly beat it. I have beaten it myself. It was strong, but not even close to the level of the strongest human players. It had obvious and easily-exploitable deficiencies in strategy and building placement.

I think even the cheater version would have lost handily to Serral or any of the strongest players.

(It apparently beat MaNa as well as TLO, but those matches were never released to my knowledge. I see no reason to assume Google cheated less flagrantly in private than they did in public.)

benswerd 1 hour ago|||
I predict LLMs will reach superhuman level and beat even that model in the next 12 months
orbital-decay 1 hour ago||
Starcraft is APM-dependent. Unless the latency will improve greatly in frontier reasoning LLMs (which is unlikely), it will remain a bit like knitting with an excavator.
benswerd 1 hour ago|||
I predict latency will improve greatly in the next 12 months to more than 4x speed on current frontier tasks
loeg 1 hour ago||
Yeah but Starcraft needs, like, 10-20x the APM these agents are doing.
benswerd 1 hour ago||
I’m not convinced a lot of it can’t be solved with code mode.

Marine staggering for example seems like an ideal code mode task.

loeg 52 minutes ago||
Yeah. Some of it may just be "thinking" less rather than faster token generation.
jamiequint 55 minutes ago|||
Should be possible to play this with Jev.
adsfgoinoi 1 hour ago||
[dead]
malfist 2 hours ago||
This is a great idea for a benchmark. Something all the benchmarks seem to be missing is strategy, tactical solutions in most of the benchmarks are all thats required but here requires actual long term thinking and tactical thinking, balancing and orchestration.
shard972 1 hour ago|
That’s why I tried making https://wrathbench.shard.page
winwang 1 hour ago||
Would be interesting if you could team a fast and slow agent together -- slow model can either act directly or maybe just communicate to the fast model.
benswerd 1 hour ago|
I might open this up to a tournament if enough people want. Any interest?
bee_rider 2 hours ago||
A ton of conversations about the game must be in the training set. I wonder, is there any way just from watching how they play, of telling if they tend to pick strategies that people complain or meme about online?
tekla 50 minutes ago||
At high levels of play Zerg is generally considered significantly stronger than the other races. So I generally will assume AI will tend to pick Zerg
rrr_oh_man 29 minutes ago|||
I only played the StarCraft demo two decades ago. Why is that so? Rushing?
xmcp123 35 minutes ago|||
Unless you Protoss and mind control the Zerg, and then are Zerg+Protoss.
stackghost 2 hours ago||
I wonder if there is a library to decipher brood war replay files. Perhaps an agent could learn by watching.
benswerd 1 hour ago|||
Not hard to build. I was shocked at how fast/easy this was to pull together.
frutiger 2 hours ago|||
I haven’t checked for SC:BW but Blizzard has official parsers/replayers for SC2 on GitHub.
benswerd 7 hours ago||
+ Playable Agent driven Starcraft
conorcleary 1 hour ago|
Dibs on the fly brain