I’ve been experimenting with two things on this:

- multi-model consensus, with multiple cross-review rounds. Obviously, the number of inference tasks explodes with the number of models. Led to some interesting results [^0].

- giving an agent "stray thoughts" produced by the same model, or another, giving the second model a selection of the agent’s context, with different triggers (random, loop detection,…)[^1]. So far has proven very helpful and much cheaper than the first.

[0]: https://github.com/lightless-labs/refinery

[1]: https://github.com/Lightless-Labs/skunkworks/tree/main/flux

I wonder if regenerating the same prompt with the same model multiple times at a higher temperature would be equivalent to running different models. I suspect the perceived variance among different frontier models may be largely due to randomness associated with non-zero temperatures.

Models seem to be trained to return nice, round numbers of items, like 5, 10, or 15 (because of interference from training on marketing materials?) Plus, recall is far from 100% on large contexts. So if your code has 27 bugs, each run may find a different set of 10 issues out of the 27, whether you use several models or call the same one repeatedly.

I'm sure many have made something like this, I've done a few. I've found simply submitting one's prompt to multiple models to be kind of pointless. You're just going to get statistical noise from the variances in their training methods, as they are all training on pretty much the same data.

I get significantly better results by pre-prompting each LLM (they can be the same LLM too, just another instance), I pre-prompt them to approach from a different perspective. Basically, I create expert personas that each believe they are someone of a different career, different intellectual perspectives, and then that generates a real debate between experts.

It should be called something else, maybe Ensemble? It doesn't fuse anything.

I have been experimenting with multi-agent llms for last month, as I put in the writeup for my repo and in the video the biggest value I have found is when you run a bunch of different agentic strategies in parallel then have a judge review the variance of them. So far that has uncovered interesting insights. The rest of it is so-so. Been fun but also expensive!

Repo with video: https://github.com/monkeydust/rightmind

I opened the page and prompted it `Which 3d printer is the best`. I mean this is a stupid question but I was looking at some 3d printers so it popped into my mind.

Seeing this log is interesting: https://link.ekin.dev/6RzYGGX7

It came up with a decent response but I guess Opus or GPT 5.5 would do fine anyway. Gotta try it on different stuff. But this feels like it would work great on some situations.

I tried OpenRouter Fusion with the budget model option but swapped out DeepSeek v3.2 for DeepSeek V4 Pro. The results weren't that bad. An interesting take on quorums for sure. However I did notice a tool call to Claude Opus 4.8 for 1168 - 237 tokens, and $0.0118 cost, which I cannot account for because Opus was not in my selection and only revealed in logs. Strange.

Everyone is focused on model ensembles, but the bigger bottleneck for me is state.

A panel of models can debate for 5 minutes. The harder problem is making an agent remember why a decision was made 5 weeks ago.

Interestingly I've had a similar experience with agent teams/swarms, albeit they can get much more expensive depending on the workflow.

I found that Fable didn't have as much of an impact when put in a team.

But it was/is a very pleasant model to work with 1:1. And was the first time I didn't use my primary team based workhorse in months, across 10s of sessions last week.

Is there any formal research in this space? I too have tried flavors of this approach, but I can't confidently say my results were better.

I worry its kind of like asking 2-3 different consultants what the optimal strategy is for your business...and I'm not sure merging the answers produces anything material better.