Posted by AmazingTurtle 1 day ago
Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine?
The lack of long context is the main reason that I still end up using Anthropic.
The worst is when you need it to hold for example a number of papers in its head, or large and complex materials that it needs full resolution on and your context window ends up being perennially at 16%. You have about five minutes of conversation and it compacts and then you have to wait for it to read that again, get to 16%... and repeat.
372 was not perfect, but it was so much better and a godsend. It turned that 12 to 20% into more like 40%.
This is really a killer feature in my opinion. I'm currently working on pi brains extension that is designed to solve the compaction issue I have, which is, I don't know what the AI knows after compaction.
https://gitsense.com/screenshots/pi-inspect-overview.png
The screenshot shot above contains what I call a compaction capsule which contains all the messages/events at the point of compaction. The "Files" section in the right side panel shows me how many files were read/modified since compaction.
After compacting, I can easily tell what Pi use to know and use `/tree` to go back in time.
I still wouldn't compact more than 5 times before starting a new session since compaction messages persists in Pi.
config.toml
model_context_window = 27000
model_auto_compact_token_limit = 17000
model_auto_compact_token_limit_scope = "total"Same. I've told @tibo many times about these issues but apparently OpenAI's focus is on features that buy them the most users ASAP instead of increasing harness quality.
npm uninstall -g @openai/codex && curl -fsSL https://claude.ai/install.sh | bash
OpenAI devs, if you're reading, this is how you lost a customer.
It asks you to create new chats (Claude Design) and Claude Code nudges you to start over too, I think `/compact` is optional though?
That used to be one of the major issues why clearing or compacting context when the context nears 150 to 170k is usually preferred.
It really depends on the session, and what you are doing how far you can push it.
I don't think I've ever pushed past 500k though, it may well fall apart at the higher end of what is officially supported.
^ this is literally what the op did. Don’t flag facts.
Start with the big idea. Pin down the product manager-level description, and maybe some details about which features are in or out, how the phased design/roadmap will work, maybe what the basic tech stack will be.
That document gets saved, and has the big picture context.
Next, we go a little deeper, flesh out schemas, APIs, a little more about what the code should look like, the overall testing strategy. This document gets saved, and the AI can reference it for context.
The next level is a very specific implementation plan. Think individual small JIRA tickets in an epic. The AI writes this based on previous context. I make sure each individual step has very explicit instructions about branch names, which branch is branched from where, the testing and documentation that will be done for that step. It should be obvious from the higher level documents, but that helps keep the AI to only worrying about the local document in its context window.
Then have a DAG of stacked pull requests designed for subagents to work through.
All of this keeps the context window down (as well as costs down, and accuracy up) and works super well, and I can get a tremendous amount of code that is easy to review through stacked PRs. Invariably I will want to fix earlier ones then percolate that down the stack, so I have some skills for that.
Every large feature has at least one of each file created for it in that order. I start by analysing stuff and making reports that might also contain design thinking and decisions. Once I'm satisfied we have properly analysed and thought though whatever it is we're working on then I might make a plan or just jump into implementation depending on how complex the issue is. And then I might have zero or multiple reviews of different kinds by different agents depending on how much verification I think the issue warrants.
This is my super simple worse-is-better development process. I make up what is needed on the spot.
It's hard to believe that sort of confusion doesn't hurt performance a bit, the question is if that degradation is worse than the performance fall off from long-context (which is highly task specific)
Poking around in the repo the whole implementation is an unsupervised LLM fever-dream.
However at this point it can completely maintain itself. When a new version of Claude or OpenCode is released, it updates itself to work on the latest version. It can also add new implementations for harnesses pretty reliably. It's actually pretty fun to watch it at this point. "Make this work on Hermes agent and message me when you're done" and an hour later or so, I can go play with it in Hermes.
P.S. Love the "bonsai" name btw!
i couldnt figure out a nice enough way to reload the conversation, and didnt want to have to worry about the cache rebuilds so its a bit languishing
So... at least as of a week ago or so, I don't believe so.
Multiple (slow) conversations seems to be the efficient path for me the past week or so
Think: A VP of engineering has some super high level goal: "build a new product to beat our competitor in this new market". It then filters down and various levels of reports build more and more detailed yet more narrow pieces of planning/orchestration. They don't make it work by keeping all of the context in their head and micromanaging.
You can also ask it to read your past sessions, find places where it was wrong after compaction, and figure out a strategy to persist the most important summary information.
I have several Hermes threads that have each had >10 compactions with a 200k context limit, and with the right instructions on note-taking, they require at most a "double-check that against our past decision records" to be put back on track after a slip-up.
it's hard to modularize a monolith without keeping large chunks in context at first pre-delegation; the orchestrator -- however you implement it -- needs to carry as much of the original thing as possible into context and big monoliths make this task heavier.
so, tl;dr : I use a lot of tokens re-writing legacy codebases that were constructed with very little CS training in some huge spaghetti fashion by random people around the world.
With Anthropic I run out of quota very fast on a Pro plan with long contexts.
Is long context primarily for Max tier or for API usage? Or are there special ways of working with long context on a Pro plan?
I didn't quite get it tuned up enough to be a daily driver but I'm basically sure it can be hotrodded however you want. It comes out of the box with like four different vendor hostile compact strategies, including compacting into images at the smallest size Claude can read, which the "coding is basically solved" geniuses conveniently leak the resolution heuristic out of their website along with all the other side channels they print for the MSS.
I use GLM5.2 and gpt-5.5 and 5.6 and never noticed any quality degradation near the limits of the context or due to compaction.
that's because you and your workloads probably fall into a bucket of users where compaction is tuned well.
but the reality is that those processes need to pick what to carry over, and that's a tough thing to get right for everyone, so for me compaction is a signal to restart the session or to tell the LLM to reference existing design docs lest we go off the rails.
large context limits is one of the things that make the open model competitors very attractive to me.
Anthropic's "long context" is mostly marketing bullshit. It semi-reliably holds about the same amount of info, and then starts suffering the same issues.
I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is. I never let claude get about 300k and in general I don't compact, I just divide up the work in chunks where I can fit it into 300k and I try to keep really clean and slim docs and modular code bases.
It's still kind of annoying, sometimes the larger context is useful for one time tasks, but I think if you are regularly going beyond 300k you are losing a lot and probably have a poorly designed code base.
It does not match my experience that the model gets significantly dumber. It does get slower and more expensive, yes, but that's a sacrifice that needs to be made when working on anything complex.
My process involves having the main agent use subagents to explore what is needed for the given task. Then it writes a plan. Then it has the plan adversarially reviewed by more subagents and hardens it. After all is said and done, the 1M token window is 30-40% full. This flow would never work with 272k context, and in fact I've had to tone it down significantly for 5.6 Sol. Which, now that I think about it, probably explains why the results I get with it are inferior.
The linked tweet is an unofficial reply to Tibo's official info and Tibo makes a correction in a reply.
Why do the woke lefties on HN get so upset and downvote and flag.
[1] https://arxiv.org/abs/2605.12366 [2] see F1 of GraphWalks 256K vs 1M in Opus 4.8 System Card: https://www-cdn.anthropic.com/0b4915911bb0d19eca5b5ee635c80f... [3] https://context-folding.github.io/
I view Pi as similar to NeoVim, both programs have kinda crappy defaults that nobody should really use but are incredibly extensible and customizable for power users willing to invest their time into them.
[0]: helix: A post-modern text editor. https://helix-editor.com/
When context compaction introduces a gap, I use the /total-recall skill to pull prior turns back into context and off it goes.
The tool is free for personal use and has a source available local cloud option to sync convo histories across multiple machines.
I swear there is a executive convention out there sharing worst practices.
So did many NLP experts who worked on LSTM, GRU, etc. But transformers empirically won out and here we are.
It will be interesting to see if future model architectures need to take this into account again.
If humans are any indication, one thing we're still missing is efficient movement of information from short term to long term memory. Fine tuning in principle does something akin to this, but not efficiently.
I'm guessing you just started using codex, it suffered greatly from "model context size exceeded" errors early on, where compaction couldn't even recover from it, these errors just stopped happening maybe some month(s) ago.
Nowadays it's a lot better though and I've don't get stuck in those anymore. Although I don't like the idea that they don't surface what goes into the "concise summary" afterwards, make it really hard to know if it actually got everything important or not.
In general, codex seems to be moving into the way of hiding as much from the user as possible, and it wouldn't surprise me if they eventually start encrypting the entire session logs just like they did with agent>sub-agent prompts recently. Sad though as it's easily the best harness+model combo available today out of the ones I've tried.
Pretty much every single AI firm had the same issue when the new model came out, it's the result of everyone jumping onboard and consuming all available compute capacity to break it for days.
Claude had the same issues when they released their newer models.
Quite literally a harness feature/bug though, that it's unable to compact when reaching the max context (which seemingly both codex and Claude Code struggled with in the beginning), not something that will happen at every model release forever.
Hasn't happen for at least the two last model releases in Codex, not sure why you'd imply that.
But definitely earlier GPT models suffered greatly as context got large, and the compaction itself in Codex was really crappy. That changed around January.
I try to keep it under 200k but my DeepSeek and MiMo sessions can sometimes grow to 350k tokens when I try to squeeze one last iteration I compact.
Can't OpenAI copy DeepSeek K/V cache tech (from published papers) to make it super cheap?
If you use Reasonix with DeepSeek it gets silly, as it is append-only to work with how caching works. It gets something like 97-98% cached tokens in a long session. It makes an already cheap model even cheaper.
I've also tailored an agent system prompt with llamacpp reasoning budget and message that asks the agent to spawn subagents then compress to reduce context bloat. It's in opencode using the dynamic context pruning, so it's mostly keeping a vector&direction without having to bloat up.
It mostly works to allow iteration into multiple sub components.