Posted by robotswantdata 6/30/2025
Drew Breunig has been doing some fantastic writing on this subject - coincidentally at the same time as the "context engineering" buzzword appeared but actually unrelated to that meme.
How Long Contexts Fail - https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-ho... - talks about the various ways in which longer contexts can start causing problems (also known as "context rot")
How to Fix Your Context - https://www.dbreunig.com/2025/06/26/how-to-fix-your-context.... - gives names to a bunch of techniques for working around these problems including Tool Loadout, Context Quarantine, Context Pruning, Context Summarization, and Context Offloading.
> The term “loadout” is a gaming term that refers to the specific combination of abilities, weapons, and equipment you select before a level, match, or round.
In the military you don't select your abilities before entering a level.
Does he pretend to give the etymology and ultimately origin of the term, or just where he or other AI-discussions found it? Because if it's the latter, he is entitled to call it a "gaming" term, because that's what it is to him and those in the discussion. He didn't find it in some military manual or learned it at boot camp!
But I would mostly challenge this mistake, if we admit it as such, is "significant" in any way.
The origin of loadout is totally irrelevant to the point he makes and the subject he discusses. It's just a useful term he adopted, it's history is not really relevant.
Doesn't seem that significant?
Not to say those blog posts say anything much anyway that any "prompt engineer" (someone who uses LLMs frequently) doesn't already know, but maybe it is useful to some at such an early stage of these things.
For example: in form, things like negative shape and overlap. In color contrast things like Ratio contrast and dynamic range contrast. Or how manipulating neighboring regional contrast produces tone wrap. I could go on.
One reason for this state of affairs is that artists and designers lack the consistent terminology to describe what they are doing (though this does not stop them from operating at a high level). Indeed, many of the terms I have used here we (my colleagues and I) had to invent ourselves. I would love to work with an AI guru to address this developing problem.
I don't think they do. It may not be completely consistent, but open any art book and you find the same thing being explained again and again. Just for drawing humans, you will find emphasis on the skeleton and muscle volume for forms and poses, planes (especially the head) for values and shadows, some abstract things like stability and line weight, and some more concrete things like foreshortening.
Several books and course have gone over those concepts. They are not difficult to explain, they are just difficult to master. That's because you have to apply judgement for every single line or brush stroke deciding which factors matter most and if you even want to do the stroke. Then there's the whole hand eye coordination.
So unless you can solve judgement (which styles derive from), there's not a lot of hope there.
ADDENDUM
And when you do a study of another's work, it's not copying the data, extracting colors, or comparing labels,... It's just studying judgement. You know the complete formula from which a more basic version is being used for the work, and you only want to know the parameters. Whereas machine training is mostly going for the wrong formula with completely different variables.
What bothers me more is that so much truly important material is not being addressed as explicitly as it should be. For example: the exaggeration of contrast on which so much art relies exists in two dimensions: increase of difference and decrease of difference.
This application of contrast/affinity is a general principle that runs through the entirety of art. Indeed, I demonstrate it to my students by showing its application in Korean TV dramas. The only explicit mention I can find of this in art literature is in the work of Ruskin, nearly 200 years ago!
Even worse is that so much very important material is not being addressed at all. For example, a common device that painters employ is to configure the neighboring regional contrast of a form can be light against dark on one edge and dark against light on the opposing edge. In figurative paintings and in classic portrait photography this device is almost ubiquitous, yet as far as I am able to determine no one has named it or even written about it. We were obliged to name it ourselves (tone wrap).
> They are not difficult to explain, they are just difficult to master.
Completely agree that they can be difficult to master. However, a thing cannot be satisfactorily explained unless there is consistent (or even existent) terminology for that thing.
> So unless you can solve judgement (which styles derive from)
Nicely put.
I'm not fully sure of what you means. If we take the following example, are you talking about the neck and the collar of the girl?
https://i.pinimg.com/originals/ea/70/0b/ea700b6a0b366c13187e...
https://fr.pinterest.com/pin/453596993695189968/
I think the name of the concept is "edge control" (not really original). You can find some explanation here
https://www.youtube.com/watch?v=zpSlGmbUB08
To keep it short, there's no line in reality. So while you can use them when sketching, they are pretty crude, kinda like a piano with only 2 keys. The best thing is edges, meaning the delimitation between two contrasting area. If you're doing grayscale, your areas are values (light and shadow) and it's pretty easy. Once you add color, there's more dimension to play with and it became very difficult (warm and cold color, atmospheric colors, brush stroke that gives the illusion of details,...).
Again, this falls under the things that are easy to explain, but take a while to be able to observe it and longer to reproduce it.
There's a book called "Color and Light" by James Gurney that goes in depth about all of these. There's a lot of parameters that goes inside a brush stroke in a specific area of a painting.
Yes... that's exactly it. It is also described in our teaching material here, (half way down the page):
https://rmit.instructure.com/courses/87565/pages/structural-...
Rembrandt was an avid user of this technique. In his portraits, one little trick he almost always used was to ensure that there was no edge contrast whatsoever in at least one region, usually located near the bottom of the figure. This served to blend the figure into the background and avoid the flat effect that would have happened had he not used it. In class I call this 'edge loss'. An equivalent in drawing is the notion of 'open lines' whereby silhouette lines are deliberately left open at select points.
> I think the name of the concept is "edge control" (not really original). You can find some explanation here.
I am aware of the term 'edge control' though I have not heard it used in this context. I feel that the term is too general to describe what is happening in the (so-called) tone wrap.
To extend the principle, wrap is an important concept in spatial rendering (painting, photography, filmmaking etc) and is a cousin of overlap. Simply... both serve to enhance form.
> To keep it short, there's no line in reality.
True that. I learned a lot about lines from reading about non-photorealistic rendering in 3D. There are some great papers on this subject (below) though I feel there is still work to be done.
Cole, Forrester, et al. "How well do line drawings depict shape?." ACM SIGGRAPH 2009 papers. 2009. 1-9.
Cole, Forrester, et al. "Where do people draw lines?." ACM SIGGRAPH 2008 papers. 2008. 1-11.
I made a stab at summarizing their wisdoms here:
https://rmit.instructure.com/courses/87565/pages/drawing-lin...
> There's a book called "Color and Light" by James Gurney that goes in depth about all of these. There's a lot of parameters that goes inside a brush stroke in a specific area of a painting.
Looking at it now. Any writer who references the Hudson River School is a friend of mine.
I'd expect this to be a lot more plug and play, and as swappable as LLMs themselves by EOY, along with a bunch of tooling to help with observability, A/B testing, cost and latency analysis (since changing context kills the LLM cache), etc.
Or maybe it just hasn't matured yet and we'll see more of it in the future. We'll see.
Maybe something like the equivalent of AWS Firecracker for whatever the equivalent of AWS Lambda is in the future LLM world.
It is somewhat bothersome to have another buzz phrase. I don't why we are doing this, other than there was a Xeet from the Shopify CEO, QT'd approvingly by Karpathy, then its written up at length, and tied to another set of blog posts.
To wit, it went from "buzzphrase" to "skill that'll probably be useful in 3 years still" over the course of this thread.
Has it even been a week since the original tweet?
There doesn't seem to be a strong foundation here, but due to the reach potential of the names involved, and their insistence on this being a thing while also indicating they're sheepish it is a thing, it will now be a thing.
Smacks of a self-aware version of Jared Friedman's tweet re: watching the invention of "Founder Mode" was like a startup version of the Potsdam Conference. (which sorted out Earth post-WWII. and he was not kidding. I could not even remember the phrase for the life of me. Lasted maybe 3 months?)
I find they takeoff when someone crystallizes something many people are thinking about internally, and don’t realize everyone else is having similar thoughts. In this example, I think the way agent and app builders are wrestling with LLMs is fundamentally different than chatbots users (it’s closer to programming), and this phrase resonates with that crowd.
Here’s an earlier write up on buzzwords: https://www.dbreunig.com/2020/02/28/how-to-build-a-buzzword....
EDIT: Ah, you also wrote the blog posts tied to this. It gives 0 comfort that you have a blog post re: building buzz phrases in 2020, rather, it enhances the awkward inorganic rush people are self-aware of.
And I wrote the first post before the meme.
We should be able to name the source of this sheepishness and have fun with that we are all things at once: you can be a viral hit 2002 super PhD with expertise in all areas involved in this topic that has brought pop attention onto something important, and yet, the hip topic you feel centered on can also make people's eyes roll temporarily. You're doing God's work. The AI = F(C) thing is really important. Its just, in the short term, it will feel like a buzzword.
This is much more about me playing with, what we can reduce to, the "get off my lawn!" take. I felt it interesting to voice because it is a consistent undercurrent in the discussion and also leads to observable absurdities when trying to describe it. It is not questioning you, your ideas, or work. It has just come about at a time when things become hyperreal hyperquickly and I am feeling old.
However, many fundamental phenomena are missing from the "context engineering" scope, so neither context engineering nor prompt engineering are useful terms.
Surely not prompt engineering itself, for example.
Anyone basing their future agentic systems on current LLMs would likely face LangChain fate - built for GPT-3, made obsolete by GPT-3.5.
https://arxiv.org/abs/2402.04253
For long contexts start with activation beacons and RoPE scaling.
Drew calls that one "Tool Loadout" https://www.dbreunig.com/2025/06/26/how-to-fix-your-context....
This field, I swear...it's the PPAP [1] of engineering.
[1] https://www.youtube.com/watch?v=NfuiB52K7X8
I have a toool...I have a seeeeearch...unh! Now I have a Tool Loadout!" *dances around in leopard print pyjamas*
Cloud API recommender systems must seem like a gift to that industry.
Not my area anyways but I couldn't see a profit model for a human search for an API when what they wanted is well covered by most core libraries in Python etc...
I think the comment you're replying to is talking about discovery rather than use; that is, offering a million tools to the model, not calling a million tools simultaneously.
you just need to knowingly resource what glue code is needed, and build it in a way it can scale with whatever new limits that upgraded models give you.
i can’t imagine a world where people aren’t building products that try to overcome the limitations of SOTA models
With that in mind, what would be the business sense in siloing a single "Agent" instead of using something like a service discovery service that all benefit from?
Also the current LLMs have still too many issues because they are autoregressive and heavily biased towards the first few generated tokens. They also still don't have full bidirectional awareness of certain relationships due to how they are masked during the training. Discrete diffusion looks interesting but I am not sure how does that one deal with tools as I've never seen a model from that class using any tools.
Hmm first time hearing about this, could you share any examples please?
llm -m openai/o3 \
-f https://raw.githubusercontent.com/simonw/llm-hacker-news/refs/heads/main/llm_hacker_news.py \
-f https://raw.githubusercontent.com/simonw/tools/refs/heads/main/github-issue-to-markdown.html \
-s 'Write a new fragments plugin in Python that registers issue:org/repo/123 which fetches that issue
number from the specified github repo and uses the same markdown logic as the HTML page to turn that into a fragment'
Beautiful one shot results and i now have really nice animations of some complex maths to help others understand. (I’ll put it up on youtube soon).
I don't know the manim library at all so saved me about a week of work learning and implementing
How does this actually work, and how can one better define this to further improve the prompt?
This statement feels like the 'draw the rest of the fucking owl' referred to elsewhere in the thread
The "Read large enough context to ensure you get what you need" quote is from a different post entirely, this one: https://simonwillison.net/2025/Jun/30/vscode-copilot-chat/
That's part of the system prompts used by the GitHub Copilot Chat extension for VS Code - from this line: https://github.com/microsoft/vscode-copilot-chat/blob/40d039...
The full line is:
When using the {ToolName.ReadFile} tool, prefer reading a
large section over calling the {ToolName.ReadFile} tool many
times in sequence. You can also think of all the pieces you
may be interested in and read them in parallel. Read large
enough context to ensure you get what you need.
It makes more sense if you look at the definition of the ReadFile tool:
https://github.com/microsoft/vscode-copilot-chat/blob/40d039...
description: 'Read the contents of a file. Line numbers are
1-indexed. This tool will truncate its output at 2000 lines
and may be called repeatedly with offset and limit parameters
to read larger files in chunks.'
Observation: this isn't anything that can't be automated /
I had one view of what these things were and how they work, and a bunch of outcomes attached to that. And then I spent a bunch of time training language models in various ways and doing other related upstream and downstream work, and I had a different set of beliefs and outcomes attached to it. The second set of outcomes is much preferable.
I know people really want there to be some different answer, but it remains the case that mastering a programming tool involves implemtenting such, to one degree or another. I've only done medium sophistication ML programming, and my understand is therefore kinda medium, but like compilers, even doing a medium one is the difference between getting good results from a high complexity one and guessing.
Go train an LLM! How do you think Karpathy figured it out? The answer is on his blog!
There will always be a crowd that wants the "master XYZ in 72 hours with this ONE NEAT TRICK" course, and there will always be a..., uh, group of people serving that market need.
But most people? Especially in a place like HN? I think most people know that getting buff involves going to the gym, especially in a place like this. I have a pretty high opinion of the typical person. We're all tempted by the "most people are stupid" meme, but that's because bad interactions are memorable, not because most people are stupid or lazy or whatever. Most people are very smart if they apply themselves, and most people will work very hard if the reward for doing so is reasonably clear.
If you want to be an F1 driver it's probably useful to understand almost every part of a car. If you're a delivery driver, it probably isn't, even if you use one 40+ hours a week.
But in between someone commuting in a Toyota and an F1 driver are many, many people, the best example from inside the extremes is probably a car mechanic, and even there, there's the oil change place with the flat fee painted in the window, and the Koenigsberg dealership that orders the part from Europe. The guy who tunes those up can afford one himself.
In the use case segment where just about anyone can do it with a few hours training, yeah, maybe that investment is zero instead of a week now.
But I'm much more interested in the one where F1 cars break the sound barrier now.
1. For the majority of regular users the best way to understand the car is to read the manual and use the car.
2. For F1 drivers the best way to understand the car is to consult with engineers and use the car.
3. For a mechanic / engineer the best way to understand the car is to build and use the car.
There are interesting emergent behaviors in computationally feasible scale regimes, but it is not magic. The people who work at OpenAI and Anthropic worked at Google and Meta and Jump before, they didn't draw a pentagram and light candles during onboarding.
And LLMs aren't even the "magic. Got it." ones anymore, the zero shot robotics JEPA stuff is like, wtf, but LLM scaling is back to looking like a sigmoid and a zillion special cases. Half of the magic factor in a modern frontier company's web chat thing is an uncorrupted search index these days.
First of all, I think a lot of the issue here is this sense of baggage over this word intelligence--I guess because believing machines can be intelligent goes against this core belief that people have that humans are special. This isn't meant as a personal attack--I just think it clouds thinking.
Intelligence of an agent is a spectrum, it's not a yes/no. I suspect most people would not balk at me saying that ants and bees exhibits intelligent behavior when they look for food and communicate with one another. We infer this from some of the complexity of their route planning, survival strategies, and ability to adapt to new situations. Now, I assert that those same strategies can not only be learned by modern ML but are indeed often even hard-codable! As I view intelligence as a measure of an agent's behaviors in a system, such a measure should not distinguish the bee and my hard-wired agent. This for me means hard-coded things can be intelligent as they can mimic bees (and with enough code humans).
However, the distribution of behaviors which humans inhabit are prohibitively difficult to code by hand. So we rely on data-driven techniques to search for such distributions in a space which is rich enough to support complexities at the level of the human brain. As such I certainly have no reason to believe, just because I can train one, that it must be less intelligent than humans. On the contrary, I believe in every verifiable domain RL must drive the agent to be the most intelligent (relative to RL award) it can be under the constraints--and often it must become more intelligent than humans in that environment.
Sure, if we define anything as intelligent, AI is intelligent.
Is this definition somehow helpful though?
The Q summations that are estimated/approximated by deep policy networks are famously unstable/ill-behaved under descent optimization in the general case, and it's not at all obvious that "point RL at it" is like, going to work at all. You get stability and convergence issues, you get stuck in minima, it's hard and not a mastered art yet, lot of "midway between alchemy and chemistry" vibes.
The RL in RLHF is more like Learning to Rank in a newsfeed optimization setting: it's (often) ranked-choice over human-rating preferences with extremely stable outcomes across humans. This phrasing is a little cheeky but gives the flavor: it's Instagram where the reward is "call it professional and useful" instead of "keep clicking".
When the Bitter Lesson essay was published, it was contrarian and important and most of all aimed at an audience of expert practitioners. The Bitter Bitter Lesson in 2025 is that if it looks like you're in the middle of an exponential process, wait a year or two and the sigmoid will become clear, and we're already there with the LLM stuff. Opus 4 is taking 30 seconds on the biggest cluster that billions can buy and they've stripped off like 90% of the correctspeak alignment to get that capability lift, we're hitting the wall.
Now this isn't to say that AI progress is over, new stuff is coming out all the time, but "log scale and a ruler" math is marketing at this point, this was a sigmoid.
Edit: don't take my word for it, this is LeCun (who I will remind everyone has the Turing) giving the Gibbs Lecture on the mathematics 10k feet view: https://www.youtube.com/watch?v=ETZfkkv6V7Y
"On the contrary, I believe in every verifiable domain RL must drive the agent to be the most intelligent (relative to RL award) it can be under the constraints--and often it must become more intelligent than humans in that environment."
And I said it's not that simple, in no way demonstrated, unlikely with current technology, and basically, nope.
GPT-4 is a 1.75 terraweight MoE (the rumor has it) and that's probably pushing it for an individual's discretionary budget unless they're very well off, but you don't need to match that exactly to learn how these things fundamentally work.
I think you underestimate how far the technology has come. torch.distributed works out of the box now, deepspeed and other strategies that are both data and model parallel are weekend projects to spin up on an 8xH100 SXM2 interconnected cluster that you can rent from Lambda Labs, HuggingFace has extreme quality curated datasets (the fineweb family I was alluding to from Karpathy's open stuff is stellar).
In just about any version of this you come to understand how tokenizers work (which makes a whole class of failure modes go from baffling to intuitive), how models behave and get evaled after pretraining, after instruct training / SFT rounds, how convergence does and doesn't happen, how tool use and other special tokens get used (and why they are abundant).
And no, doing all that doesn't make Opus 4 completely obvious in all aspects. But its about 1000x more effective as a learning technique than doing prompt engineer astrology. Opus 4 is still a bit mysterious if you don't work at a frontier lab, there's very interesting stuff going on there and I'm squarely speculating how some of that works if I make claims about it.
Models that look and act a lot like GPT-4 while having dramatically lower parameter counts are just completely understood in open source now. The more advanced ones require resources of a startup rather than an individual, but you don't need to eval the same as 1106 to take all the mystery out of how it works.
The "holy shit" models are like 3-4 generations old now.
Lambda Labs full metas jacket accelerated interconnect clusters: https://lambda.ai/blog/introducing-lambda-1-click-clusters-a...
FineWeb-2 has versions with Llama-range token counts: https://huggingface.co/datasets/HuggingFaceFW/fineweb-2
Ray Train is one popular choice for going distributed, RunHouse, bumcha stuff (and probably new versions since I last was doing this): https://docs.ray.io/en/latest/train/train.html
tiktokenizer is indispensable for going an intuition about tokenization and it does cl100k: https://tiktokenizer.vercel.app/
Cost comes into it, and doing things more cheaply (e.g. vast.ai) is harder. Doing a phi-2 / phi-3 style pretrain is like I said, more like the resources of a startup.
But in the video Karpathy evals better than gpt-2 overnight for 100 bucks and that will whet anyone's appetite.
If you get bogged down building FlashAttention from source or whatever, b7r6@b7r6.net
But from an architecture point of view, you might be surprised at how little has changed. Rotary and/or alibi embeddings are useful, and there's a ton on the inference efficiency side (GQA -> MHA -> MLA), but you can fundamentally take a llama and start it tractably small, and then make it bigger.
You can also get checkpoint weights for tons of models that are trivially competitive, and tune heads on them for a fraction of the cost.
This leaked Google memo is a pretty good summary (and remarkably prescient in terms of how it's played out): https://semianalysis.com/2023/05/04/google-we-have-no-moat-a...
I hope I didn't inadvertently say or imply that you can make GPT-4 in a weekend, that's not true. But you can make models with highly comparable characteristics based on open software, weights, training sets, and other resources that are basically all on HuggingFace: you can know how it works.
GPT-2 is the one you can do completely by yourself starting from knowing a little Python in one day.
Ok, I can buy this
> It is about the engineering of context and providing the right information and tools, in the right format, at the right time.
when the "right" format and "right" time are essentially, and maybe even necessarily, undefined, then aren't you still reaching for a "magic" solution?
If the definition of "right" information is "information which results in a sufficiently accurate answer from a language model" then I fail to see how you are doing anything fundamentally differently than prompt engineering. Since these are non-deterministic machines, I fail to see any reliable heuristic that is fundamentally indistinguishable than "trying and seeing" with prompts.
1 - LLM Tends to pick up and understand contexts that comes at top 7-12 lines.Mostly first 1k token is best understood by llms ( tested on Claude and several opensource models ) so - most important contexts like parsing rules need to be placed there.
2 - Need to keep context short . Whatever context limit they claim is not true . They may have long context window of 1 mil tokens but only up to avg 10k token have good accuracy and recall capabilities , the rest is just bunk , just ignore them. Write the prompt and try compressing/summerizing it without losing key information manually or use of LLM.
3 - If you build agent-to-agent orchestration , don't build agents with long context and multiple tools, break them down to several agents with different set of tools and then put a planning agent which solely does handover.
4 - If all else fails , write agent handover logic in code - as it always should.
From building 5+ agent to agent orchestration project on different industries using autogen + Claude - that is the result.
I tested with 8B model, 14B model and 32B model.
I wanted it to create structured json, and the context was quite large like 60k tokens.
the 8B model failed miserably despite supporting 128k context, the 14b did better the 32B one almost got everything correct. However when jumping to a really large model like grok-3-mini it got it all perfect.
The 8B, 14B, 32B models I tried were Qwen 3. All the models I tested I disabled thinking.
Now for my agent workflows I use small models for most workflow (it works quite nicely) and only use larger models when the problem is harder.
Adherence to context is lossy in a way reminiscent of human behavior but also different in crucial ways.
In one repetitive workflow, for example, I process long email threads, large Markdown tables (which is a format from hell), stakeholder maps, and broader project context, such as roles, mailing lists, and related metadata. I feed all of that into the LLM, which determines the necessary response type (out of a given set), selects appropriate email templates, drafts replies, generates documentation, and outputs a JSON table.
It gets it right on the first try about 75% of the time, easily saving me an hour a day - often more.
Unfortunately, 10% of the time, the responses appear excellent but are fundamentally flawed in some way. Just so it doesn't get boring.
I dont quite follow. Prompts and contexts are different things. Sure, you can get thing into contexts with prompts but that doesn't mean they are entirely the same.
You could have a long running conversation with a lot in the context. A given prompt may work poorly, whereas it would have worked quite well earlier. I don't think this difference is purely semantic.
For whatever it's worth I've never liked the term "prompt engineering." It is perhaps the quintessential example of overusing the word engineering.
It's all just tokens in the context window right? Aren't system prompts just tokens that stay appended to the front of a conversation?
They're going to keep dressing this up six different ways to Sunday but it's always just going to be stochastic token prediction.
<system-prompt-starts>
translate to English
<system-prompt-ends>
An explanation of dogs: ...
I can't find any study that compares putting the same initial prompt in the system role versus in the user role. It is probably just position bias, i.e. the models can better follow the initial input, regardless of whether it is system prompt or user prompt.
<system>
You are a helpful assistant.
</system>
<user>
Why is the sky blue?
</user>
<assistant>
Because of Rayleigh scattering. The blue light refracts more.
</assistant>
<user>
Why is it red at sunset then?
</user>
<assistant>
It's all prompt, it can all be engineered. Hell, you can even get a long way by pre-filling the start of the Assistant response. Usually works better than a system message. That's prompt engineering too.
In other words, there's a deliberate illusion going on where we are encouraged to believe that generating a document about a character is the same as that character being a real entity.
Categorically, no. Most are not software engineers, in fact most are not engineers of any sort. A whole lot of them are marketers, the same kinds of people who pumped crypto way back.
LLMs have uses. Machine learning has a ton of uses. AI art is shit, LLM writing is boring, code generation and debugging is pretty cool, information digestion is a godsend some days when I simply cannot make my brain engage with whatever I must understand.
As with most things, it's about choosing the right tool for the right task, and people like AI hype folk are carpenters with a brand new, shiny hammer, and they're gonna turn every fuckin problem they can find into a nail.
Also for the love of god do not have ChatGPT draft text messages to your spouse, genuinely what the hell is wrong with you?
And yes, I view clever instructions like "great grandma's last wish" still as just providing context.
>A given prompt may work poorly, whereas it would have worked quite well earlier.
The context is not the same! Of course the "prompt" (clever last sentence you just added to the context) is not going to work "the same". The model has a different context now.
The term engineering makes little sense in this context, but really... Did it make sense for eg "QA Engineer" and all the other jobs we tacked it on, too? I don't think so, so it's kinda arguing after we've been misusing the term for well over 10 yrs
I'm not sure there's much scientific or mathematical about guessing how a non-deterministic system will behave.
Engineering: "Will the bridge hold? Yes, here's the analysis, backed by solid science."
Pseudo-engineering: "Will the bridge hold? Probably. I'm not really sure; although I have validated the output of my Rube Goldberg machine, which is supposedly an expert in bridges, and it indicates the bridge will be fine. So we'll go with that."
"prompt engineer" or "context engineer" to me sounds a lot closer to "paranormal investigator" than anything else. Even "software engineer" seems like proper engineering in comparison.
Right now, you cannot get that far. And if you happen to... Tomorrow it will be different.
Predicting tides is possible. It requires enormous amounts of data and processing to be sure of it. Right now, we've got tides, but we don't have the data from the satellites. Because the owner is constantly shifting the prompt, for good reasons of their own. So we can't be confident - or we can only be so blindly.
Statistics isn't guessing. But it is guessing when the confidence interval is unknowable and constantly shifting. We're not talking relativity, we're talking about throwing pancakes at a wall to tell if there's a person behind it.
I mean this not as an insult to software dev but to work generally. It’s all play in the end.
In other words; context.
But that was like old man programming.
As the laws of physics changed between 1970 and 2009.
When you play with the APIs the prompt/context all blurs together into just stuff that goes into the text fed to the model to produce text. Like when you build your own basic chatbot UI and realize you're sending the whole transcript along with every step. Using the terms from the article, that's "State/History." Then "RAG" and "Long term memory" are ways of working around the limits of context window size and the tendency of models to lose the plot after a huge number of tokens, to help make more effective prompts. "Available tools" info also falls squarely in the "prompt engineering" category.
The reason prompt engineering is going the way of the dodo is because tools are doing more of the drudgery to make a good prompt themselves. E.g., finding relevant parts of a codebase. They do this with a combination of chaining multiple calls to a model together to progressively build up a "final" prompt plus various other less-LLM-native approaches (like plain old "find").
So yeah, if you want to build a useful LLM-based tool for users you have to write software to generate good prompts. But... it ain't really different than prompt engineering other than reducing the end user's need to do it manually.
It's less that we've made the AI better and more that we've made better user interfaces than just-plain-chat. A chat interface on a tool that can read your code can do more, more quickly, than one that relies on you selecting all the relevant snippets. A visual diff inside of a code editor is easier to read than a markdown-based rendering of the same in a chat transcript. Etc.
Never mind that prompt engineering goes back to pure LLMs before ChatGPT was released (i.e. before the conversation paradigm was even the dominant one for LLMs), and includes anything from few-shot prompting (including question-answer pairs), providing tool definitions and examples, retrieval augmented generation, and conversation history manipulation. In academic writing, LLMs are often defined as a distribution P(y|x) where X is not infrequently referred to as the prompt. In other words, anything that comes before the output is considered the prompt.
But if you narrow the definition of "prompt" down to "user instruction", then you get to ignore all the work that's come before and talk up the new thing.
But personally I think a focus on "prompt" that refers to a specific text box in a specific application vs using it to refer to the sum total of the model input increases confusion about what's going on behind the scenes. At least when referring to products built on the OpenAI Chat Completions APIs, which is what I've used the most.
Building a simple dummy chatbot UI is very informative here for de-mystifying things and avoiding misconceptions about the model actually "learning" or having internal "memory" during your conversation. You're just supplying a message history as the model input prompt. It's your job to keep submitting the history - and you're perfectly able to change it if you like (such as rolling up older messages to keep a shorter context window).
Because they’re different things? The prompt doesn’t dynamically change. The context changes all the time.
I’ll admit that you can just call it all ‘context’ or ‘prompt’ if you want, because it’s essentially a large chunk of text. But it’s convenient to be able to distinguish between the two so you can talk about the same thing.
Exactly the problem with all "knowing how to use AI correctly" advice out there rn. Shamans with drums, at the end of the day :-)
There is no objective truth. Everything is arbitrary.
There is no such thing as "accurate" or "precise". Instead, we get to work with "consistent" and "exhaustive". Instead of "calculated", we get "decided". Instead of "defined" we get "inferred".
Really, the whole narrative about "AI" needs to be rewritten from scratch. The current canonical narrative is so backwards that it's nearly impossible to have a productive conversation about it.
For that kind of tasks (and there are many of those!), I don't see why you would expect something fundamentally different in the case of LLMs.
To this day I think the same. With the addition that knowing about "git log -S" grants you necromancy in addition to the regular superpowers. Ability to do rapid code search, and especially code history search, make you look like a wizard without the funny hat.
To be fair it's also more likely to mess up than I am, but for reading search results to get an idea of what the code base looks like the speed/accuracy tradeoff is often worth it.
And if it was just a search tool this would be barely worth it, but the effects compound as you chain more tools together. For example: reading and running searches + reading and running compiler output is worth more than double just reading and running searches.
It's definitely an art to figure out when it's better to use an LLM, and when it's just going to be an impediment, though.
(Which isn't to agree that "context engineering" is anything other than "prompt engineering" rebranded, or has any staying power)
That reminds me of the first chapter in "The Programmer Brain" by Felienne Hermans. There's an explanation there that confusion when reading code is caused by three things:
- Lack of knowledge: When you don't have the faintest idea of the notation or symbol being used, aka the WHAT.
- Lack of information: When you know the WHAT, but you can't figure out the WHY.
- Lack of processing power: When you have an idea of the WHY, but can't grasp the HOW.
We already have methods and tooling for all the above and they work fine without having to do shamanic rituals.
There are many sciences involving non-determinism that still have laws and patterns, e.g. biology and maybe psychology. It's not all or nothing.
Also, LLMs are deterministic, just not predictable. The non-determinism is injected by providers.
Anyway is there an essential difference between prompt engineering and context engineering? They seem like two names for the same thing.
The difference is that "prompt engineering" as a term has failed, because to a lot of people the inferred definition is "a laughably pretentious term for typing text into a chatbot" - it's become indistinguishable from end-user prompting.
My hope is that "context engineering" better captures the subtle art of building applications on top of LLMs through carefully engineering their context.
Not correct. They are deterministic as long as a static seed is used.
Commutative: A+B = B+A Associative: A+(B+C) = (A+B)+C
I can train large nets deterministically too (CUBLAS flags). What your saying isn't true in practice. Hell I can also go on the anthropic API right now and get verbatim static results.
How?
Setting temperature to 0 won't guarantee the exact same output for the exact same input, because - as the previous commenter said - floating point arithmetic is non-commutative, which becomes important when you are running parallel operations on GPUs.
I think the usual misconception is to think that LLM outputs are random "by default". IMHO this apparent randomness is more of a feature rather than a bug, but that may be a different conversation.
Only if you choose so by allowing some degree of randomness with the temperature setting.
a = 0.1, b = 0.2, c = 0.3
a * (b * c) = 0.006
(a * b) * c = 0.006000000000000001
When you're running models on a GPU (or any other architecture that runs a whole bunch of matrix operations in parallel) you can't guarantee the order of the operations.
So you can see, completion time is a completely orthogonal issue, or can be made one.
And even libraries like tensorflow can be made to give reproducible results, when setting the corresponding seeds for the underlying libraries. Have done that myself, speaking from experience in a machine learning setting.
If one wants to make something give the same answers every time, one needs to control all the variables of input. This is like any other software including other machine learning algorithms.
The agents cannot change their internal state hence they change the encompassing system.
They do this by injecting information into it in such a way that the reaction that is triggered in them compensates for their immutability.
For this reason I call my agents „Sammy Jenkins“.
Building powerful and reliable AI Agents is becoming less about finding a magic prompt or model updates. It is about the engineering of context and providing the right information and tools, in the right format, at the right time. It’s a cross-functional challenge that involves understanding your business use case, defining your outputs, and structuring all the necessary information so that an LLM can “accomplish the task."
That’s actually also true for humans: the more context (aka right info at the right time) you provide the better for solving tasks.
Context is often incomplete, unclear, contradictory, or just contains too much distracting information. Those are all things that will cause an LLM to fail that can be fixed by thinking about how an unrelated human would do the job.
It's easy to forget that the conversation itself is what the LLM is helping to create. Humans will ignore or depriotitize extra information. They also need the extra information to get an idea of what you're looking for in a loose sense. The LLM is much more easily influenced by any extra wording you include, and loose guiding is likely to become strict guiding
Maybe not very often in a chat context, my experience is in trying to build agents.
We've been working on a way to test this more systematically by simulating full conversations with agents and surfacing the exact point where things go off the rails. Kind of like unit tests, but for context, behavior, and other ai jank.
Full disclosure, I work at the company building this, but the core library is open source, free to use, etc. https://github.com/langwatch/scenario
Of course, that comment was just one trivial example, this trope is present in every thread about LLMs. Inevitably, someone trots out a line like "well humans do the same thing" or "humans work the same way" or "humans can't do that either". It's a reflexive platitude most often deployed as a thought-terminating cliche.
In this case though it's a pretty weird and hard job to create a context dynamically for a task, cobbling together prompts, tool outputs, and other LLM outputs. This is hard enough and weird enough that you can often end up failing to make text that even a human could make sense of to produce the desired output. And there is practical value to taking a context the LLM failed at and checking if you'd expect a human to succeed.
These days, so can LLM systems. The tool calling pattern got really good in the last six months, and one of the most common uses of that is to let LLMs search for information they need to add to their context.
o3 and o4-mini and Claude 4 all do this with web search in their user-facing apps and it's extremely effective.
The same patterns is increasingly showing up in coding agents, giving them the ability to search for relevant files or even pull in official document documentation for libraries.
Until we can scan your brain and figure out what you really want, it's going to be necessary to actually describe what you want built, and not just rely on vibes.
(X-Y problem, for example.)
The idea that fixing this is just a matter of providing better training and contextual data, more compute or plumbing, is deeply flawed.
[1]: https://www.theregister.com/2025/06/29/ai_agents_fail_a_lot/
It reads like articles put out by consultants at the height of SOA. Someone thought for a few minutes about something and figured it was worth an article.
One thing that is missing from this list is: evaluations!
I'm shocked how often I still see large AI projects being run without any regard to evals. Evals are more important for AI projects than test suites are for traditional engineering ones. You don't even need a big eval set, just one that covers your problem surface reasonably well. However without it you're basically just "guessing" rather than iterating on your problem, and you're not even guessing in a way where each guess is an improvement on the last.
edit: To clarify, I ask myself this question. It's frequently the case that we expect LLMs to solve problems without the necessary information for a human to solve them.
"Make it possible for programmers to write in English and you will find that programmers cannot write in English."
It's meant to be a bit tongue-in-cheek, but there is a certain truth to it. Most human languages fail at being precise in their expression and interpretation. If you can exactly define what you want in English, you probably could have saved yourself the time and written it in a machine-interpretable language.
For those actually using the products to make money well, hey - all of those have evaluations.
Alchemical is "you are the world's top expert on marketing, and if you get it right I'll tip you $100, and if you get it wrong a kitten will die".
The techniques in https://www.dbreunig.com/2025/06/26/how-to-fix-your-context.... seem a whole lot more rational to me than that.
Yes, if you have an over-eager but inexperienced entity that wants nothing more to please you by writing as much code as possible, as the entity's lead, you have to architect a good space where they have all the information they need but can't get easily distracted by nonessential stuff.
God bless the people who give large scale demos of apps built on this stuff. It brings me back to the days of doing vulnerability research and exploitation demos, in which no matter how much you harden your exploits, it's easy for something to go wrong and wind up sputtering and sweating in front of an audience.