Posted by adchurch 1 hour ago
At Weave, we write most of our code with AI, and it's been getting more expensive. This came to a head when Opus 4.7 was released and, thanks to its tokenizer changes, our costs shot up. We knew we didn't need Opus for everything but we didn't want to lose out on the intelligence for the cases where you really need it. So we decided to build a model router to handle this for us.
The Weave Router acts as an Anthropic/OpenAI endpoint specifically for coding agents. It looks at every inference request and intelligently (more on that in a sec) decides what model to send it to, handling all the translations required along the way. So it can use faster/cheaper models (e.g. DeepSeek v4, GLM 5.2, Kimi K2.6) when possible, and frontier models (Opus 4.8 & GPT 5.5 (& Fable whenever it's back)) when necessary.
How do we know what model to route to? We trained an RL model on tens of thousands (so far!) of agent traces. We reward the routing model when it selects an LLM that successfully completes the given task.
Here's an example: if you ask the router to plan a complex change, it will (probably) route that request to Opus 4.8. Subagents exploring the codebase to gather context will be routed to more suitable models (e.g. DeepSeek V4 Flash). Then when you have the plan ready to implement, it will be (most likely) be handed to a quicker model (e.g. GLM 5.2) to carry it out.
We've been using this internally for the last month or so. We've saved 40% on tokens vs. what we otherwise would have paid, with no noticeable differences in quality or velocity.
The router is source-available under Elastic License 2.0, so you can self-host it. Or if you prefer, you can also use our hosted version: weaverouter.com.
I'll be here to answer any questions you may have!
Hard to quantify this ofc but that's what I've felt vibes wise from using this for the last month.
I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .
How does this router translate to $$$ when developing?
This is the key thing that other routers we've seen miss: they're stateless so for a coding agent use case you end up spending more money due to all the cache misses.
In practice you just pick one and stick with it until the API stops or you hit performance issues.
As prices increase we will see more of these tools to optimise and make the best use of token budget
We haven't yet set up local model routing though, that's really interesting - have you had any success using local models for coding tasks? Tbh I haven't heard many success stories from using local models yet
Happy to talk about this in some more depth if there's anything specific you're curious about!
Will this use my Claude Pro/Max subscription? Or will it always use the API billing "pay as you go"?
1. https://github.com/instavm/murmur - Murmur
Also, small LLMs are prone to stop before completion, throw errors and produce loops. Is this factored in the design of the tool? I am not sure.
edit: spellcheck
Totally right about small LLMs btw, that's why we trained this on real agent sessions where we forced it to use different models. If the routing model sees small models can't handle a certain type of task then they won't be assigned. (Also as a fallback we have some guardrails that will have a bigger model come in to "rescue" a smaller model if it gets stuck)