We're going to do a followup blog detailing our routing approach soon! In short, the router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before (this is the advantage of having a meta-harness that is actually planning all the sub-agents centrally)

Maintaining the cache is extremely, extremely important, so we're iterating fast but that's a major factor we track in the router's development. Couple things I'd look at:

1. The cache is generally reset after a compaction - this is the best time to make a switch if you want.

2. In many cases, the max duration of a cache is 1h, so if a session is being resumed after a long time, that's also a good time to re-assess the complexity.

We're iterating fast here and learning a lot! Definitely a lot to think about it in this area.

The kv cache is wiped as soon as you get your answer, cloud hosts are not going to hold the GPU memory for your entire session. You're probably referring to some agent level cache

It seems Databricks open-sourced it a while ago:

https://www.databricks.com/blog/introducing-omnigent-meta-ha...

https://github.com/omnigent-ai/omnigent

Ultimately, Databricks wants your enterprise on their platform. I dont think they particularly care about open source or the little guy.

Well yes, first hit is free.

What did you move to and what type of workload, if I may ask?

I think you’ve misunderstood the article. It’s about how Databricks reduced their own costs, not about how adopting Databricks will reduce anyone else’s costs.