Posted by var0xyz 2 days ago
That is incorrect when dealing in the real world with many different humans using a complex system. I’ve heard other people say this, and it always confuses me how it is not self evident. Human interaction is not a maths equation. There are too many random variables.
Imagine that one has to implement a function to multiply two large integers. A junior programmer will likely use the default multiplication operator. A mid-level programmer will add a couple of strategies with a weak heuristic for routing, and perhaps with basic caching. A senior will sample the incoming requests to identify which strategy might empirically be good, and ensure that it is being hit, with optimal fallbacks. A superintelligent AI might design custom kernels, and design a custom model for routing. A god will... we don't know what a god will do. Perfection, like beauty, is in the eyes of the beholder.
> Over-engineering is solving the wrong problem. That's the whole definition.
Wholeheartedly disagree. Over-engineering already seems pretty well defined just based on the words alone without tacking on a strange arbitrary definition to suit your blog post, but I think most people most people would define it more as "spending too much time and/or resources to solve a problem more-thoroughly than necessary".
Suppose I was given the task to convert all the JPEG images in a folder to PNG.
Solution A) I could write a highly optimized program writing custom CUDA decoders and encoders and using Open MPI for CPU-based optimization, and create an elaborate plugin framework on top of this to make this modular and cool.
Solution B) I could just run `find -name "*.jpg" | parallel magick {} {.}.png`.
Both solve the "right" problem; the ask was to convert these images, and both approaches can absolutely do that, so they didn't solve the "wrong problem", but I would argue that for most people Solution A would be "over engineered". You could argue that by building custom encoders and decoders and the like we're solving the "wrong problem" by going beyond what the problem asked for, but I feel like if you're criticizing the engineering methodology then that kind of proves my point.
We generally think someone is "over engineering" primarily when we are happy with the low-effort result.
There are times when Solution A is the right way to go about it. Maybe you want something that takes advantage of custom hardware, or maybe you can make an assumption about these images that Imagemagick can't, or maybe you really just understand image encoding significantly better than the Imagemagick people. in those cases, we generally don't consider it "over-engineering".
No one starts a project for perfectly objective reasons. Your choice of careers was not perfectly objective, you decided you liked a field, and it seemed you had enough aptitude for it, and then you did the work to get into it, learning as you went along and adjusting.
So it is with all kinds of decisions that go along with starting or changing a software project. Of course are your architectural and engineering acumen increases, you get a goo feel for how to choose good constraints and requirements, and then within the those constraints it is now up to you to find optimal solution for them, not the most over-engineered one.
Like I said before, sometimes going from A-B-C-D is much less efficient (and can actually be counter-productive) in the aggregate than going form A-D directly, even though YAGNI would dictate A-B-C-D. So long as you generally know that D would be needed sooner or later, A-B-C-D can be counter-productive because it causes to you to miss the forest for the trees. Even if you are not doing D immediately, some thought given to D can improve greatly your understanding of the context of A and the roadmap, leading to better design.
Note that string design instincts honed over time make this process work, not building out stuff just because of fads (e.g., microservices) or for some functionality you do not actually need. So here is my point of agreement with the article: the reason teams might waste time over-engineering is not because they are exploring a design space and objectives that is larger than what is strictly required for now; it is because they lack good architectural skill and experience and instincts. Again, considering D upfront can actually simplify the A you build now.
So, yes, if you have a purpose in mind, you can build something that can be used for that purpose, more or less effectively. The more general the purpose, the less specific the implementation; the more specific the purpose, the less general the implementation. Over-engineering here is something that indeed goes beyond the purpose of something, with feature creep as an example of moving goalposts and a slippery grasp of the limit.
However, production is only one layer of concern in engineering. You can talk about ideal artifacts whose forms perfectly enable the realization of some purpose, but there is the practicality of constructing one that must also be considered. Think of diminishing returns. Does it makes sense to invest a certain amount of time and resources into producing a more perfect artifact? If not, then pursing such perfection is itself an imperfection on the part of human judgement. That is, a more perfect engineer might know the limit of perfection for an idealized artifact, but he also knows what the perfect practical limit of justifiable effort is. Perfectionism is a sign of an imperfect engineer.
Returning to feature creep, the author might argue that the moving goalposts are a sign that the purpose is being refined, but feature creep can also mean that the shifting purpose is detached from real needs. So, in that case, the purpose served is no longer the purpose that ought to be served, and in that sense, while one may realize an end perfectly in a relative sense, the end pursued is itself the result of imperfect understanding, which is effectively what the author is gesturing at when discussing requirements.