> How much of an impact can this have? > Reading is:alive (1 byte) Across 1M Monsters
You aren't reading one byte here, you are reading 1M bytes! Of course, optimizing the access to 1M bytes is something to consider. Optimizing the access to one byte isn't.
The article is definitely worth reading IMHO, but it really needs a better headline!
Now think about random access to single struct instances instead: the CPU loads a cache line worth of data for each field and uses only one element out of the whole cache line. This is much worse than a compact structure representation of the same data.
SoA is not universally better.
In both cases you want to think about locality of the next read and structure the data accordingly.
Great for this access pattern, but I wouldn't make a general statement like that. This is the same thing as row-oriented vs column-oriented databases, OLTP vs OLAP. SoA is weak if you are adding/removing monsters more often than accessing a single "hot" field.
Why is that? Genuinely curious. Does "weak" mean that it performs worse than AoS, or that the gains aren't as significant versus AoS?
Or swap it with the last monster, and keeping an index for the last monster alive.
SoA is only useful when you don't read multiple fields for most operations.
$ sysctl -a | grep "l.*cachesize" | gnumfmt --field=2 --to=si
hw.perflevel1.l1icachesize: 132k
hw.perflevel1.l1dcachesize: 66k
hw.perflevel1.l2cachesize: 4,2M
hw.perflevel0.l1icachesize: 197k
hw.perflevel0.l1dcachesize: 132k
hw.perflevel0.l2cachesize: 13M
hw.l1icachesize: 132k
hw.l1dcachesize: 66k
hw.l2cachesize: 4,2M
$ sysctl -a | grep hw.cachelinesize
hw.cachelinesize: 128The JVM is an odd place where it requires too much heap to compete with the AOT compiled languages, but its startup time is too slow compared to interpreted languages. I think these enhancements are essential to keep the platform relevant.
Since JDK 25 it's already 64 bits with the `-XX:+UseCompactObjectHeaders` flag [1], but in JDK 27 it will be the default [2].
> where it requires too much heap to compete with the AOT compiled languages
Not to compete but to beat, and not too much, but the right amount. Low level languages are optimised for control, not performance (that control translates to better performance in smaller programs, and to worse performance in larger programs), and their particular constraints prevent them from enjoying certain important optimisations, especially those offered by JIT compilation and moving collectors, which remove some overheads that AOT compilers and free-list allocators incur. Their memory management is forced (by their constraints) to optimise for footprint rather than speed.
There are common misunderstandings about memory management and why moving collectors were created to reduce the CPU overheads of malloc/free, especially in large programs, in exchange for what is effectively free RAM. This is why moving collectors are chosen by the languages that are unconstrained enough to use them and have the resources to implement them (Java, .NET, V8). With the exception of Zig (and even there it requires some effort), it's hard for low level languages to use the basic optimisation that's behind moving collectors. I gave a talk about how moving collectors optimise memory management at the last Java One, and it should be available on YouTube soonish [3].
> but its startup time is too slow compared to interpreted languages
That hasn't been the case for some time. You are right, though, that startup/warmup time is worse than in AOT compiled languages, and that is the tradeoff of optimising JITs: reduce the overheads associated with AOT compilation in large program in exchange for warmup.
Both startup and warmup have already been improved thanks to Project Leyden's "AOT cache" [4], but it will never be as low as C.
In general, the tradeoff is between optimisations that help large programs vs optimisations that help small programs.
[1]: https://openjdk.org/jeps/519
[2]: https://openjdk.org/jeps/534
[3]: I can't reproduce the full talk (which goes into the maths of memory management) here but what happened with moving collectors was that until very recently (open source low-latency moving collectors are newer than ChatGPT), they required pauses and so weren't suitable for programs requiring low latencies. As a result, many developers either forgot or never learnt just how incredibly efficient moving collectors are. But the key is that because accessing RAM by necessity requires CPU, using CPU effectively captures RAM even it's not used by the program. Bringing the CPU and RAM usage into a good balance is more efficient than trying to minimise one or the other. This is also the reason why hardware (physical or virtual) is packaged within a very narrow band of RAM/core ratio.
In general, the tradeoff is between optimisations that help large programs vs optimisations that help small programs.
There are of course plenty of optimizations the JVM does that aren't possible AOT, but that that doesn't imply an automatic win at large scales, as Rust demonstrates.
Yes. I was working in a place that made large sensor-fusion applications, air-traffic control applications, and logistical planning, each in the 2-8MLOC range. Over time, we ported all of them from C++ to Java because C++'s performance overheads were too annoying to work around.
Of course, in principle it's always possible to match and perhaps even exceed Java's performance in a low-level language, but in practice it becomes ever more difficult as the program grows (and the cost remains with maintenance forever). The reason is that as programs grow, patterns become less regular (e.g. the variance in object lifetimes grows), the need for concurrency grows (and so the need for sharing objects among threads and for lock free data structures), and more general constructs are used (e.g. more dynamic dispatch). Improvements in modern allocators, as well as LTO and PGO have helped, but not enough to match the extent of optimisations you can do once you're free of the design constraints of low-level control and the focus on the worst case.
Java's thesis (not initially, but from very early on) was to rely on optimisations that can't be effectively employed by low-level languages because of their constraints, such as efficient memory management that benefits from being able to move most pointers in a program, and highly aggressive speculative optimisations (that are nondeterministic and can fail, resulting in deoptimisation). These optimisations tend to be global, and so they don't restrict program structure much, keeping maintenance costs lower, but they do help the average case at the cost of harming the worst case, which is a tradeoff that programs written in low-level languages don't want, and of course, it doesn't give the low-level control that's the entire point of low-level languages. Proving that thesis took a while, and longer in some aspects than others (moving collectors that don't pause were first released to a wide audience three years ago).
Of course, the differences aren't huge because the hot paths are typically small enough that they can be improved without adding too much cost (and hot paths require some manual optimisation in all languages), but gaining some performance as a side effect of significantly lowering costs is nice.
> There are of course plenty of optimizations the JVM does that aren't possible AOT, but that that doesn't imply an automatic win at large scales, as Rust demonstrates.
I don't know what it is that Rust demonstrates given how few large scale projects have chosen it, but I've seen nothing to indicate that it doesn't suffer from the same performance issues as C++ compared to Java. In fact, someone I know who works at one of the world's largest tech companies told me that his team lead really wanted to do something in Rust, so they ported a small-to-medium service from Java to Rust. The result was such a huge performance drop that it wouldn't meet their minimum requirements. They were then forced to spend an additional 6 to 12 months carefully hand-optimising their Rust code until it matches Java's performance, but the result is such that all future maintenance will be more expensive. This is the exact same pattern I've seen with C++.
It's interesting that 20 years ago the people who said Java can't beat C++ on performance were experienced low-level programmers who had little or no experience with Java (and they were also right on several axes at the time). Today the people who say that are those with little experience with low-level languages (and are under the impression that low level languages are universally fast), but they will eventually learn about their fundamental performance issues just as we did decades ago.
I think that Rust in particular has made people without much experience in low-level programming (among which Rust has made much more inroads than among those with a lot of experience in low-level programming) believe a certain story, namely that the problem with low level languages was memory safety and that that was the reason so many large programs switched to Java despite the performance sacrifices they had to make. Now that Rust fixes that problem, they can have their cake and eat it too! In reality, memory safety was indeed one of the several significant problems with low level languages that Java sought to fix, but another was the performance issues low level languages suffer from as they get large (making good performance ever more costly). The tradeoff isn't performance (in large programs there might even be a performance gain) but low-level control, as that is what low-level languages are about. That was what they offered back then, and it's still what they offer now. Rust was first designed twenty years ago, back when things still looked a certain way (which is why, IMO, it repeated most of C++'s design mistakes), but these days I think that a better, more modern design of low-level languages is more focused on control, leaving large programs to high-level languages. Lack of memory safety has, without a doubt, been one of the things that made low-level languages less palatable to "ordinary" applications, but it was far from the only one.
Anyway, I'm sure the debate of which is faster, C++ (/Rust/Zig) or Java, will continue, and frankly, due to the nature of modern hardware, compiler, and runtime optimisations these days (when the question of the cost of some individual operation is all but meaningless and out ability to extrapolate from the performance of one program to another is close to nil), it largely comes down to empirical questions such as which program patterns are more or less common in the field and in which domains, as there are code and workload patterns that could give an advantage to either one.
That result would say less about performance of languages than it would about competency of developers with a language.
I just to don’t buy that could assign a task to two teams with comparable expertise and domain knowledge in Rust and Java, and have the Rust result be at a “huge” performance deficit.
No, don’t believe that was an apples to apples comparison.
I don't know what it is that Rust demonstrates given that few large scale projects have chosen it, but I've seen nothing to indicate that it doesn't suffer from the same performance issues as C++ compared to Java.
Of course, in principle it's always possible to match and perhaps even exceed Java's performance in a low-level language, but in practice it becomes ever more difficult as the program grows (and the cost remains with maintenance forever).
These aren't the biggest advantages. I would say that the biggest ones are aggressive speculative optimisations that allow inlining of virtual calls (by default, up to a depth of 15 calls) and the ability to freely move pointers, which allows alternatives to free-list-based memory management. Low-level languages can't afford pervasive speculative optimisation (as they're focused on the worst case) and can't allow most of their pointers to be moved (because they often share them directly with the hardware and/or device drivers).
> and the wiki page on Java performance [0] is repeating what I understood.
That may be because the information on that page seems to be up to date to 2011-2. Java is now on version 26, BTW.
I have even tried removing/rewriting some of the questionable sentences but my edits weren't accepted.
I am not sure how it compares with C++, Rust and Zig, but we made a benchmark with a similar Go binary, Java native version performance (load tests) is similar to Go binary. Only RAM usage of Java native binary is 3 times to Go binary (and JVM app took almost 10 times more RAM than Go version).
I gave a talk on the subject that I hope will be published soon, and while I can't reproduce it here, let me give an example that offers some basic intuition. Imagine needing to do some computation in two ways on a machine with 1GB of free RAM. You could run for 10s, taking up 100% CPU and consuming 80MB of RAM, or for 9s, taking up 100% CPU and consuming 800MB of RAM. The second is more efficient, despite taking up 10x more RAM and saving "only" 10% of CPU, regardless of the relative cost of RAM and CPU. This is because taking up 100% of the CPU effectively captures 100% of RAM (as no other program can use it), so both programs capture the entire 1GB only the second one captures it for a second less. This scales to non extreme situations because accessing RAM requires CPU, so using CPU means capturing RAM whether you use it or not. So HotSpot uses it if it can use it to balance the CPU utilisation.
In some situations it may not matter, and I assume that if Native Image and Go work just as well for you, then the workload isn't very high, but under high workloads, this can matter a lot.
Memory isn’t free. CPU isn’t free.
But there is a semi-fundamental tradeoff here, you either use more CPU to use less memory or the reverse. Java can be dynamically configured for either end (though defaults to less CPU by not running the GC unnecessarily).
Most developers, in Java and in most other languages, do not consider the cost of every field, but I can tell you that people who need micro-optimisations certainly do care, and in Java's standard library, a layout is very much a concern (except, as always, you want to optimise what really matters; there's no point in optimising something that is unlikely to be a hot spot in a real program). Sometimes, though, you want to intentionally spread out the layout to avoid cache line sharing when concurrency is involved. You will find such examples in the standard library, too.
Are you saying most developers are bad? It’s the equivalent of most employees don’t consider the cost of every action to the employer and is how company spend blows up.
And speaking about costs, knowing what to optimise is the key to software performance. Improving the performance of an operation by 10000x will improve the performance of your program by less than 1% if the operation is only 1% of the profile to begin with. So I'm only saying that most developers don't work on code where the layout is very significant, but some certainly do.
I've heard this theory before. This isn't just about performance and I don't buy it.
I've seen too many examples of this is just a temporary solution so it doesn't matter. >3 years later that "temporary solution" was still there and at the heart of many operations yet it's now to hard and too costly to fix.
I've also seen the this is a quick hack. No 1 uses it. It doesn't go through any hot paths. All good. You know what happens? Years later, every service literally goes through it. Again, it's too hard to fix.
In the real world these "theories" are really loose. The only fix is every should be aware of what they are doing and do it properly. The it might not happen, etc mindset is dangerous.
And that's the problem. Who decides that? How do you know and that's my problem with it. Things always change. It's always temporary, not in the hot path, doesn't matter etc until it does.
So what is considered "doesn't impact" often comes back to bite.
If your profile shows you a hot path that's responsible for 90% of the time your program spends, any second optimising anything outside of it harms your performance, as it's a second spent on low ROI instead of high ROI.
It's much more niche to work on stuff where such changes actually matter, like much much more people write boring CRUD backends than those who write physics simulators and audio processing pipelines combined.
Understand the language, the memory model, etc. Don't do "it works on my machine". Understand the architecture, layout, implications etc.
E.g. if you need an int and not a long you should clearly use an int. Wait until you do this every time and things blow up and it's too "hard" to change.
It's called be aware of your actions. Take responsibility of what you do.
> It's much more niche to work on stuff where such changes actually matter,
Not true and that's why there's so much wastage.
A lot of things matter. I've seen more times than the other way that simple awareness and changes can pay for my salary, e.g. not updating to newer EC2 instances when they get released in AWS. Even in a mid size company that was hundreds to thousands in savings.
I've seen CI/CD pipelines where the developers never considered caching and it takes hours to run. It's not free. When every PR and update (hundreds a day) triggers a run it's a cost and a cost not just on machines but developer time waiting.
I can list a lot more examples and everyone in the chain can contribute.
This runs counter to most modern software performance principles. Thanks to modern hardware optimisations (cache hierarchy, ILP, branch prediction), modern compiler optimisations (aggressive inlining that leads to a much wider view), and increased concurrency, the notion of some action having a cost lost most meaning about 20 years ago, and increasingly since. Because how fast some action is now depends on a much broader context of what else is going on in the program (and the machine), action X can be faster than Y in one program and the same or slower than Y in another.
Because it's nearly impossible to generalise (and so what was true in your previous program may not be true in your current one unless they're nearly identical), the advice is to first profile your program so that you know how fast or slow different parts are in the context of your particular program and then to focus the optimisation efforts on the hot paths in your program. Otherwise, you may end up spending effort where it makes no difference, and this comes at the cost of optimising what matters, overall harming performance.
Taking responsibility means being smart about directing your resources to where they can have the most impact.
We often used bit (not byte) fields, to convey information.
Made life challenging.
However, being able to be sloppy has its definite advantages. It takes a long time to design highly-optimized stuff. If just declaring a couple of new properties takes thirty seconds, and designing a bitfield takes an hour, then we have some real cost-savings, there.
That said, it's easy to get crazy, these days. I just spent a couple of days, chasing down greedy memory hogs. These were operations that ate gigabytes of memory. I determined that the real culprit was actually Apple MapKit, and figured out a simple workaround, but it took a long time to get there. If I suspect the OS, then it's usually my fault, and trying everything before going back to the OS takes time.
In this one case, allocating a MapView via storyboard, caused some kind of cascading strong reference stuff.
Simply allocating it programmatically, fixed it.
Took awhile to get there, though.
Most of the bottlenecks I see are not due to the organization of data. Unnecessary communication of data is the #1 offender.
mild /s
I am sorry, I only know Odin. Jai is this cult on reddit/discord, right? You get access if you socialize enough or something? Not my thing. Not for a language.
I was just throwing out an idea. I had no idea there were already implementations! Because, to my knowledge, conventional popular languages like C/C++/C#/Java/JS/Python don't do that, and automatically doing that feels like an easy performance win.
When you are developing most other applications every byte does not matter. What matters much more is overall system architecture, collapsing unnecessary abstraction layers that some developers (especially java developers) seem to love and optimizing your datastore access.
As always, profile profile profile.
A company I worked for spent a violent couple of man-decades flipping our proprietary scripting language from interpeted to bytecode generation, obviously with tons of bugs and subtle semantic changes, and it ended up boosting overall system performance by about 30%. We could have done nothing over that period of time and hardware advances would have made a bigger impact.
To get the speed up, you can't just abstract it as an access pattern because it's tied to the specific way the memory is laid out.
If you were trying to make some kind of collection type that could be queried by both row and column, you would need to store it both ways at all times and also keep both representations in sync, which also defeats the purpose, somewhat.
I feel like if you're trying to do this pattern then it doesn't make sense to also keep the objects.
Profiling important workloads matters. Without that everything else is guesswork.