Posted by Anon84 5 days ago
This is the key line right here:
> Negotiating, agreeing, communicating the shared picture of what we are building has become the work. And it’s just as hard as it was.
But if software (via code) is what we ultimately produce and sell, how did we get here? The main reason is the following lemma:
Lemma A: "The loss of fidelity of what can fit in any one person's head scales superlinearly (exponentially?) as the scope of work scales up." Or more colloquially: "It is impossible to fit a large scope of work in any one person's head." This is largely because any non-trivial task is a fractal of smaller dependencies.
The chain of logic to today's situation is then obvious:
1. Writing code requires humans who are slow and expensive.
2. To do large things we need large groups of humans.
3. As the number of humans grows (like beyond 5? 10?) it becomes impossible to keep them aligned, largely because Lemma A.
4. We need to coordinate these humans, so: enter managers!
5. But even a manager can't manage too many people and coordinate with all other managers because, again, Lemma A. Enter hierarchy!
6. As the size of the organization grows, so does the coordination overhead (exponentially, if Google AI overview is to be believed) until as,that quote surmises, the majority of the work is just that.
7. Coordination costs (or "Conway Overhead" as I call them) are very well understood in the literature, but this also brings in undesirable dynamics like bureaucracy, politics, organizational metrics (also due to Lemma A, but now triggering GoodHart's law!) and eventually territorial disputes and empire-building. Lots of friction and subtle mis-alignments.
As you can see the overhead scales superlinearly with the number of leaf workers added. And for the same reason, once the leaf workers are decimated because one worker can now do the work of a whole team, the entire organizational overhead above that is gone, which is also a superlinear change! Assume a conservative 2:1 reduction in ICs and a 1:5 manager:reportee ratio, a simplistic hierarchy that was:
1 CEO -> 5 VPs -> 25 Dirs -> 125 Managers -> 625 ICs
now becomes something like:
1 CEO -> 12 SVPs -> 60 Sr. Managers -> 310 Sr. ICs.
Not only did that eliminate 300 ICs (mostly junior I suspect) it took out 60 managers and removed an entire layer of Directors from the hierarchy! Worse, the leaf-layer will probably get decimated 5:1 not 2:1, and this will also eliminate coordination-specific roles like Program Managers. The rest of the hierarchy is much fewer but mostly more experienced (or politically savvy) people. They will be paid more, but not superlinearly more, of course, what do you think this is, socialism?
It's very much a pyramid scheme of cards built on that one bottleneck. And this bottleneck applies for pretty much all knowledge work. Once that bottleneck opens up, everything collapses. This is why I fear that the coming job changes are going to be much more disruptive that people realize, something I'm extra concerned about as a parent of high-schoolers.
Insane amount of bureaucracy, paperworks, and how we are missing deadlines so we write shit code that the quick and dirty solutions were never replaced.
Algorithms and data structures therefore are more like helping you utilize the machine economy better, but it doesn't have any meaningful impact on the social aspect of it. That's a hard lesson I had to learn from my two previous job, though now I'm considering starting my own small business just to make a little bit of living enough to survive.
But now my ADHD kicked in and is still lazy and I had so many concerns whether the market validation is great, how to deal with situations if I broke customers stuff, how to gain (and hopefully not regain) trust if any bad things ever happen, what if I want to go vacation and suddenly the server broke and got code zero (the highest level of alert I termed internally, when you had alertmanager flashing everything red, network storage is down, corruption happened) during a trip to Bahamas.
I'm still in the watershed of thinking really to do this or not, but the job market is filled with ghost jobs that are not worth my time either, I'm basically "dead locked" right now and had to make a decision quick.
Either choice is fucked for me, as I started to notice after going to work, despite I got some really interesting ideas in tech, but I'm not a charismatic person so I can't really make those idea to fruition, because no one wants to listen to me and implement it together, so I'm pretty sure it is impossible for me to be a great leader (tech lead probably, but CEO level of leadership and coordinator and manipulate the grand scheme of thing, nah, I pretty much can't do).
Now the problem is, even if I'm pretty sure to get fucked, you should choose the one that inflicts minimum pain to you. So far having my own business seems like a less painful to die and bankrupt, and I'm preparing to sell off some of my stuff to get a last dip of my fortunes and have fun. Will see how it looks. Bankruptcy is nothingburger in this modern society perhaps.
Now you see how the bottlenecks can't even be the code anymore and even goes beyond code, despite having the same core template: I don't even have to code, to repeat the same "quick and dirty" kind of mindset in another domain, in another instance. That's something LLM, heck not even AGI can solve: decision-making based on situations with limited time and resources, and it can be personal or organizational or even structural.
This is very much not going to be solvable by a bunch of lines and statements and expressions, but it really need some time to dig in and compromise. Pick your kool-aid and drink it