Posted by rochansinha 4 days ago
If multiple models can use cache representations for this kind of enrichment, the KV cache representations of different models must be somewhat compatible.
What stops us then from going a step further, and producing a model family where all models are "KV aligned", and each model can utilize the KV cache of other models directly?
So, an "expensive" reasoning model can use its full faculties to plan, but "delegate" simple subgoals to a smaller model. That smaller model can access the large model's intent directly, as rich KV cache representations - with no prefill recompute and no associated "handover" latency. Or, likewise, a "cheap" small model can generate a diminished but highly compact KV cache that the "expensive" model can then operate on - for example, for skimming a large file for shallow patterns.
Curious to know if anyone is aware of research trying what parent suggested?
A "local 8B VLA" that runs at high frequency can be sufficient for things like sensor fusion, plan execution and manipulation dynamics, but insufficient for advanced problem solving and in-depth "exception handling". So being able to ask a larger model for steering cues is going to be valuable for AI robots. And that does introduce latency and interface boundaries.
Not sure if the exact approach described there is the way to go, but I do expect something similar to show up in deployments eventually. If nothing else, having this one described does provide a point of comparison.
https://news.ycombinator.com/item?id=47195212
I wouldn't have thought to use it for LLM-to-LLM communication, though
Which is not necessarily something the humans training a model would want re/ alignment.
I think this is similar to how Gemma 4 12B is implemented, but even then I don't think the single layer image embedding is "aware" of the context.
In general an embedding doesn’t have intent or awareness in the way you’re looking for. “Embedding” just means one mathematical structure stuffed inside another. So for example the real number line is embedded into the Cartesian plane as each axis- that’s an embedding.
Now in this case specifically, the embeddings in any kind of transformer model encode the meaning of the thing they represent into vectors (which is what the model itself actually operates on). You can train the embedding to be more useful for a particular task at inference time, which already happens.
So embeddings are used in vision models to convert problems which are about the content and meaning of images into problems which are about multiplying matrices. The model doesn’t want to work with pixels (that’s what very basic vision models do, but it tends to be limited to special purpose applications) it wants to work with concepts in the image. That’s what the embedding gives it.
I still don’t really know what you mean about giving the embedding model some context. It embeds whatever you want to embed. So if you want to give it just a jpeg, fine. If you want to embed a jpeg and a json blob with some additional metadata/“context”/whatever, that’s also fine. That’s already how embeddings work.
I'm not sure how practical it is to train that architecture though or whether there would be performance issues.
Seems like it could still help but also feels like one of those things where it becomes vastly more complex and difficult to debug.