Posted by albelfio 20 hours ago
it is a structured data model, but technically not a language model (it doesn't generate language)
1. Do you provide any kind of largest common subtree caching for cheaper input?
2. Have you tried auto-generating Lisp programs structurally?
3. Have you tried augmenting a Lisp language with a `choice` function that makes choices given a prompt, the environment, and the continuation stack?
(2-3) No, but that's kind of a sick cook ... Want to get access and try it? nathan@typesafe.ai
Maybe I'm just not doing a very good job at prompting Jev, but I think right now it's not quite capable enough to generate Lisp code.
Link: https://console.typesafe.ai/playground?share=shr_148e1248984...
I’m guessing it might be able to replace maybe 40-70% of LLM calls for a given pipeline depending on the business task, cutting the API costs on those calls by an order of magnitude.
1. Start with an LLM, so that your model understands natural language.
2. Replace RoPE with a tree embedding scheme, and causal attention with a sparse attention on the graph structure. (You could use full attention... but it's cheaper to use graph attention.)
3. Chop off the final unembedding layer, replacing it with a projection down to two scalars, one for logits and one for confidence.
4. Each option of a choice is represented by a number of tokens in leaf position; average these tokens' logit outputs to get the option's logit. Average all of the confidences from all of the options to get the choice's confidence.
5. Train the logits by KL divergence from a true distribution (or NLL on samples from a true distribution).
6. Train the confidences on a subset of the data in which you know the entire true distribution.
The hardest part is getting real world data for workflows, but I wildly speculate that you can get by with only ~50,000 documents if you first adapt domains using synthetic data.
Except this, this thing looks like revolution.
lol, I bet they would publish them if their score on those benchmarks were good.