It's hard to try and compare these benchmarks against real world use cases. As an experiment I let Meta Muse make all major decisions on a weekend trip to Seattle ~2 weeks ago. I think this would have been using Muse Spark 1.3, which is equivalent to GPT 6 - Sol.

On the plus side it told me taking the light rail from SEA into the University district was faster and cheaper than Uber.

On the downside it would recommend restaurants that were out of business, checking out Salmon swimming through locks out of season, got the opening of Beecher's Cheese in Pikes Place wrong, and fumble it's way through most simple planning decisions. I think my favorite was when it asked me what flight I have tomorrow so it could help plan when to head to SEA. It sent me a very panicked message that the flight was leaving today! Very soon! I had to remind it this is a daily flight and I am booked on the next day before it checked and apologized.

For things that are verifiable, and code tends to be that way, these models can do all sorts of things. But I find it hard to reconcile Meta's goal of having a personal assistant bot with something that is unable to do the simplest tasks.

I trust Meta with my codes as much as I trust it with my nudes

This is still an entire generation behind in real-world performance. I'd say it's even behind ds4.1f in the limited testing I have done so far. It actually reminds me of 5.5 haiku more than anything, but the prompt intelligence is as bad as mimo models.

OpenAI halved the value of their pro subscription, literally. It kicks off on Oct 28. And simultaneously is now two steps back on the frontier? It's a confusing moment for that company as they try to IPO.

I have a feeling "intelligence too cheap to meter" is going to haunt/mock this industry. Recall that only a year and a half ago Llama 4 flopped so hard Meta rebooted their development efforts. Desperately wish more information was available publicly about proprietary model architectures and training regimes. Are they running Manhattan project style compartmentalization? Otherwise, are these details, and the economic implications, an open secret in their circles given the amount of crossover between companies? It'd be of great interest to us mere mortals that have to deal with the consequences of how this turns out.

Apparently “the models were trained at Meta’s Prometheus data center in central Ohio, which came online earlier this year”

https://www.nbc4i.com/news/local-news/new-albany/meet-promet...

The benchmarks I saw posted to X are impressive, which makes them hard to believe.

I recall that Meta made massive investments into compute infrastructure that put their capacity near the top of all competitors. If all of this is true, it looks like ownership of compute is a significant differentiator in performance.

I wish I could benchmark Muse Wonder Full, because Muse Wonder isn't much better than Muse Spark 1.3[1]. I created a brand new programming language designed to confuse LLMs[2] and created a model benchmark out of it.

A lot of the tasks were failed because they exceeded the $5 budget though, so maybe it would have gone on to complete them, it just required a lot of funds to do so. Still, that's the benchmark, and it's not better than MiMo V2.6 Pro which is pretty disappointing.

1 - https://bench.killswitch-lang.org/

2 - https://killswitch-lang.org/

Hopefully it's benchmaxxing. I am terrified of the idea of Zuck leading the frontier of intelligence.