Posted by geoffbp 15 hours ago
am I using it wrong?
This is a better research direction I feel
https://research.google/blog/the-sleep-framework-a-new-way-f...
So if you're hoping for that you should probably just look at the models that can be ran locally... And then you prolly realize that that's still several years into the future.
AI that remembers through continuous experience instead of documents would be significant and lucrative enough to change the business model.
Long term I think it's probably more interesting to explore continuously updating weights. But (and this may be yet another opinion that falls victim to the bitter lesson) even if it's _theoretically_ possible to encode everything in weights alone, I'm not sure it's not the most optimal. Today it's too expensive & finnicky to get better on your own tasks without overfitting (ie [0])
Practically (like next few months) getting the right instructions & context would be a huge improvement in capabilities.
[0] https://thinkingmachines.ai/blog/on-policy-distillation/#tra...
So the main improvement to memories I'd like to see is better judgment on which ones to create.
The parts about "dreaming" make me think of how agent transcripts contain a ton of information, which I think is currently under-utilized. I'll repeat advice I've seen here a few times: ingest your own transcripts into duckdb (claude code will build that pipeline in about 10 minutes) and do some data analysis on them. Some ideas if you don't know where to start: look at tool call failures to see if you can prevent common ones, check out if you have large cache misses and what causes them, look at various efficiency metrics over time (tokens per $, lines of code per tokens, etc.) with main thread vs sub-agents, etc. I started with this sort of things, and ended up following some threads to interesting insights. For example, I found indications that I tended to make Opus 5.5 / Sol 6.1 divide their work into units that were too small. They have the capacity to work on more stuff at once than older models, and I wasn't really taking advantage of that.
However, I have it on for ChatGPT (use it mostly for on the go queries via mobile) and their memory experience is significantly better. I haven't enabled Claude's memory in a long time so things may have changed though.
And in the Claude Desktop UI I manually curate the memories.
Works well enough, I don’t like the concept of dreaming as it quickly leads to memory bloat and/or outdated stuff.
I often see tool call failures because the agent invokes the tool with wrong parameter names, e.g. for file creation it provides "file_content", but the tool expects "file_text" instead. My colleague told me to put this in a pre-prompt file, but I don't understand why the models make these kinds of mistakes - the model should have the available tools with their parameter names in context, should it not?
It would be useful if they could scope them properly. Then it would be useful to keep around. But I haven’t seen memory systems be able to do that.
How did you determine this?
Do you mean that you were giving them tasks that were too small or you were telling them to split the tasks up or something?
When analyzing my session logs, I tried to look at how story size relates to how much work the AI had to do to fully implement it (made easier by my sessions always having epic#.story# - like 2.6 - in their name). Neither "story size" nor "AI work" have fully obvious metrics, so I looked at a variety of things, like length of story plan, number of lines of codes written for the story, number of different files read/written (and for AI work: number of tool calls, number of tokens for main thread / implementer / reviewer subagents, etc.).
The pattern is always pretty much the same: smaller stories are less token-efficient. One major factor for this is that in bigger stories, the average number of lines of code written per tool call is significantly higher (and not just a little bit, if I remember correctly it's a factor of 1 to 3 between my smallest and largest stories).
None of this is scientifically rigorous, but it's consistent over a fairly significant sample (about 60 stories up to this point). I'm also only measuring amount of work and not quality, but when I find things wrong with a story, I generally ask the AI to fix the issues in the same session, so the two should be correlated (sometimes the issues are large enough that they require new, separate stories to fix, but this is normally caused by problems in ideation and/or planning, not implementation). This is enough to encourages me to ask for more rather than less per story. It might also not apply to everyone, I'm sure this is dependent on everyone's workflow and way to work with the AI. But yeah, that's the sort of stuff you can explore when analyzing your session logs.
The slices are usually a bit too wide. I terminate subagents at 200k and get them to handoff to lead. It works pretty well but again, it's hard to get the orchestrator to size the slices a bit smaller.
The thing you're asking it for might be intent, which is something the technology fundamentally lacks.
Which.. is something that has been discussed very extensively, which makes one wonder why Anthropic still keeps trying to push that square peg through a round hole.
Curios to learn more, you do have some reference discussion or posts about that.
I also never use memories. I don't see the usefulness above project-specific prompting and a good knowledge of your own codebase. Am I missing something?
So the main improvement to memories I'd like to see is better judgment on which ones to create.
I started with [proactive post-hoc analysis], and ended up following some threads to interesting insights.
I partly say this because I also finally pulled the trigger on deprecating the last ClaudeCode memories on my system in favor of a personal system[1] recently, and the issues I noticed were things a casual user/'vibecoder' might appreciate, but any engineer working fulltime on multiple projects via multiple harnesses would find counterintuitive. E.g.:
- The memories were hidden away and clearly not written for human eyes; useful for low-stakes ledgers, frustrating when the right kind of errant note can derail a longrun 2 weeks later.
- The memories seemed to err towards the side of remembering everything and recording incidental details; useful for remembering the user is gluten-free, but not for tracking spec changes over time.
- The memories could be specific to a subagent or the project or the whole system; an obvious & elegant system for toy examples, but a terribly unweildy one for anyone working at scale over the span of months.
[1]: if curious, a somewhat-outdated version of it is here: https://gitlab.com/doering-ai/libs/fleet TL;DR: Friendship ended with ClaudeCode memory files; my new best friends are `BULLETIN.md` for transitory alerts, `campaigns/[slug]/LEDGER.md` for basic episodic memory, and `reports/[date]-[slug].md` for writeups polished specifically for posterity.
I'm assuming search is much cheaper than dreaming (though maybe it depends on the time scale?) so that sounds pretty important.
That’s like saying walking is space flight because both happen to change the location of an individual.
Remember, any analogies fail at some point.
Say, given your space flight example, astronauts do "space walk" spacecraft to perate. They are not "walking" but still makes sense.
I am exploring an alternative dialectic-based approach: https://particulars.fyi/
At some point I stopped noticing it doing it.
The biggest thing I want to avoid with a long running agent is volatility. Anything that pushes it into one direction or the other that quickly is not desirable in the average case. At least not to me.
Curiously this seems to apply to humans as well. The "dreaming" is making it question its own assumptions about difficulties it has had in the past. That sounds like a very helpful therapeutic process.
(Not sure of the word dreaming is entirely suitable though: it was cross referencing its notes against the actual history! Very rigorous "dreamer" lol)