Posted by mooreds 7 hours ago
I suspect a lot of the training set for this sort of thing is people online speculating about cache performance incorrectly.
At a low enough level, every performance tweak becomes unique and bespoke.
Of course, you could find people online talking about how to write high-performance code, but beyond a few basic techniques, their advice may not work for you — nobody can write a generalist article about performance engineering that will definitely solve the problem you have right now.
Arguably, there are fewer patterns for an LLM to infer as highly optimised code tends to become more and more opaque in the search for a nanosecond here or there.
Getting and using tools to find hotpaths is generally the most important performance tweaking skill.
There are plenty of real-world reasons why you'd want to get knee deep in this stuff. I wasn't suggesting not using tools (I've literally spent the day buried in JetBrains' memory and tracing tools!), but those tools can only tell you what is happening now, not what to do to improve it.
Profiling is, of course, essential. But performance tweaking can be quite a laborious process: if you're judging things by big-O notation, then that's a different level above the real low-level tweaking (imho of course). Picking the correct data-structures is all in the 101 of performance engineering. That's in the literature. But it's all too basic and simplistic. Most performance minded engineers wouldn't need a profiling tool to know which data-structure to use.
At the smallest level there's a lot of mental theory building and experimentation as you try out different approaches, which is where the instinct and intuition starts to build. I never see any of that in discussions about performance engineering.
They tell you what's happening now, but they also tell you if what you've done has had a positive impact.
> If you're judging things by big-O notation, then that's a different level above the real low-level tweaking (imho of course).
I completely agree. My point isn't that Big-Oh is low level, but rather that Big-Oh is often enough for most programming problems. Even in some of your examples like a compiler, game engine, or collection library, the big oh matters and if it's wrong, that can be a lot more important than shaving 0.1% on writing a function in a low level fashion. Big-Oh is gotten wrong a surprising amount of time even though it's 101 level stuff.
> At the smallest level there's a lot of theory building and experimentation as you try out different approaches, which is where the instinct and intuition starts to build. I never see any of that in discussions about performance engineering.
Oh because people get these things wrong all the time. That's why performance engineering stresses that you test, test, test and know what your testing and know why your testing could be wrong or corrupted. You should not trust your intuition because things change and it isn't always correct.
A good example of how easy it is to get measuring wrong. Imagine you start tweaking a function and you measure that your application became 1% faster. Was it the work you did on that function? Surprisingly, not always (at least not directly). Sometimes, it's the case that when you work on a function you re-align other functions as the machine code has to go it memory. It's possible that an undiscovered misaligned while loops was actually causing a large portion of your performance spill and by tweaking the function here, you aligned the while loop (or maybe a few of them). And, importantly, a new change somewhere else might re-unalign that same while loop.
You walk away thinking you've learn some low level lesson when in actuality your bit twiddling simply accidentally fixed something somewhere else.
This is why measuring is so important but also good measuring is even more important.
For what I work on, the hidden constant is often more important than big O. For example, a hash map has better complexity than just searching through a vector. But if the vector is small enough it will best the hash nap for actual time. Just plain searching until you find the element will even beat binary search on a sorted vector for small enough vectors. The reasons are complex, to do with cache, prefetch, branch prediction and also just how many instructions your tight inner loop has. (And the specific reasons will vary between desktop class CPUs and microcontrollers. But both exhibit this pattern.)
You could argue that at that point why bother optimising at all (there aren't a lot of elements in the collection after all). But there are two distinct cases I have come across over the years where it still matters (and for what I work with, they represent the common cases):
* You need to look up in a small collection a lot (either lots of lookups into a few small collections or a few lookups each into lots of different small collections, I have seen both cases).
* Hard realtime code where predictable latency matters. Hashmap has a bad worst case, binary trees and binary searching has badly predictable memory access patterns. And in this case the collections are usually small anyway (there are only so many actuators and sensors your equipment has, and/or the embedded microcontroller doesn't have a lot of memory anyway).
Sure, but for example - today - I am literally building and optimising high-performance collections for my open-source library. Big-O is irrelevant, because I have built pretty much all of the fundamental collection types, what I care about is lower than that: what happens when enumerating any one of those collection types. Big-O tells you the scale of the problem, but not the per-element cost, which is still important when you're building core data-structures.
I am concerned about cache-friendly memory-layouts, how to do collection compositions without unnecessary memory allocations, keeping enough guards in place to make the types safe whilst removing as many branches as possible, reducing memory copying as much as possible, and catching stupid shit the compiler or JIT does and try to work around it. Literally what happens per-instruction, per-iteration, not how to pick a big-O based data-structure: trying to make all data-structures as fast as possible.
Anyway, we seem to be talking past each other. You're talking about the basics, I'm talking about the original source of this thread which was that (apparently) LLMs are surprisingly bad at optimisation. Which, I am trying to highlight becomes almost voodoo at a low-enough level and that highly-optimised code looks progressively more strange and opaque (in the hunt for a few nanoseconds here and there), which for an LLM wouldn't look statistically significant. I think the basics of data-structure choice should be easily within the realms of an LLM's current capabilities.
Surprisingly, it isn't. I catch the output of LLMs breaking these rules all the time. Just as it is pretty common in general programmer code.
The greatest sin I often find isn't necessarily Big-Oh related but rather multiple traversal problems. Much like regular programmers, LLMs love to do multiple passes over the same list to extract data. For example
let cats = pets.filter((p)->p.isCat());
let dogs = pets.filter((p)->p.isDog());
If you give an LLM a proper testing harness and feedback loop to actually generate hypotheses, test them and revise them, I suspect it will do much better.
Some humans carry detailed models of CPU microarchitectures in their heads, against which they can design code from first principles that will be nearly ideal on the first try. It is repeatable and verifiable. The best people can accurately predict the measured performance before writing a line of code.
Measurement is useful in cases where the model of software and hardware interaction is materially incomplete. In most cases this is because the people writing the software have insufficient understanding of the hardware. Having a limited understanding of the hardware is a choice.
It would be surprising if this wasn't possible.
The great thing about LLM is that it seems to have the checklist for everything. If I rattle off a few things like "don't allocate on the hot path" and "remember to pin the cores" it will come up with a few items of its own that I might have forgotten.
Eventually, it will have gone through the whole list with me, while having documented all the measurements along the way.
But it's still guided by experience. If I see unusual numbers, I might say "hey did you forget to compile it in release mode?" and it will apologize and fix that. If I don't, it may just continue exploring without realising everything is wrong.
Once I had repo commands that could dump `sample` results and a cpu profiler/trace and then a benchmark tool that let me A/A + ABBA/BAAB-test the current modified git workspace against HEAD or any commit, the LLMs could just do their thing.
And that's how my homemade terminal uses much less memory than ghostty/kitty/iterm yet has more throughput.
AI is going to increasingly unmask people and companies who don't care about correct and performant software now that it's become so trivial to guarantee both. It used to at least be expensive and time-consuming and expertise-demanding to do those things.
This isn't a new problem by any means, but now that code is cheap, it means instead of getting frustrated with engineering and their pesky unimportant details, people will get frustrated with the AI and it's pesky unimportant details.
I think it's one reason why ADRs are an important of a software project, especially with LLMs. You need a place were you can document invariants, why you have them + the rejected ideas and acceptable risks.
It helps smart agents like Fable help you decide on trade-offs and it's kind of incredible to witness that happening.
This is why I'm not worried about being replaced for now or the forseeable future. For all of the improvements they've made, this part just never seems to change. They could slap another heuristic prompt for the edge case, but eventually it'll revert to the mean again.
I think there is a way to use LLMs to help with programming, but not when I'm not the driver in the seat writing the tests and deciding the architecture. Also I would never ship code written by them as the final product for anything I care about. Since I, like most people, find reading code to be arduous. The more fun thing to do is to force yourself to rewrite it all, treating the LLM's work as a rough draft.
They can, in fact, generate plausible performance optimization ideas on their own.
Make sure you process doesn't depend on anyone reading your mind.
When I run into things like this, it becomes a one-liner in my instructions/harness or in the canned prompt/skill I use that sets off a process.
In this case, I instruct agents to proactively build/improve diagnostic tooling if it would help them with their task + if it meets a bar of generalization/reusability (else it should be an ephemeral probe that gets abandoned at the end of the solution).
Then they can start attempting to optimize it. They can also spin round and round making the numbers worse because they don't actually know what to do.
You need a measurement that can falsify hypotheses and reject branches that won't work.
Also, if all you have left in your project are performance issues that are hard to identify without flailing around (even with Fable/Astra) despite sampler/profiler reports, then you're doing really well and I wouldn't assume you're going to fare much better than the sota models in terms of stabs in the dark.
In one case I used a made-up metric (since I didn't know the exact name or if it existed) and it somehow optimized that too.
"Claude, if this idea doesn't measure as an improvement (use X benchmark and a T-test), discard it and try the next idea."
One of the keys for me was the use of types. Typestate when functions mint witnesses that can only come from it and are required to proceed and newtypes where you use custom types instead of strings so the agent can't forget. You can also use it to force the agent to use the implementation rather than reinvent the wheel by simulating linear types. Types are a much smaller target to optimize and provide constraints that fail loudly at compile time.
Unless its an easy memory/parallel/algorithmic win, its not worth it.
I also found this is really nice for quality evals. Have one agent with no context on how something is made do a quality review, with lots of detailed feedback. Then pass back the review notes to the implementor for feedback. It works a lot better than having an agent self-evaluate its own quality.
I wish it could run interpreted to massively speed up development, and that it had a Mull-like mutant testing framework so that mutants didn't take hours and tens-to-hundreds of gigabytes of space.