Posted by canto 1 day ago
Like, sorry, no, not to a point. Yes, if you have a small app without a lot of scale, and it doesn't need to be uber reliable and have very little if almost 0 downtime, okay, sure. Most use cases are like that! This is correct but applying it as a generality is just plain wrong and displays the type of arrogance people accuse kubernetes users of having.
What happens if a container in the VM goes down or the app inside of it crashes, how do you recover? Now you need some self-recovery mechanism via systemD or whatever, which will grow in complexity and fickleness over time. Congrats, you are now doing your own version of kubernetes.
What happens when you need to upgrade/restart your VM? Ok, make a standby VM as backup that will mostly sit idle, or require a full-app redeploy any time you need to do anything to the first VM. Now you need to design a blue/green mechanism between them, and probably some networking layer work. Congrats, you are now doing your own version of kubernetes.
What happens, if running in cloud, you have a regional outage or degradation? Stand up another VM in another region and manage the networking layer between them. Or, if running locally, your ISP has an outage because of a backhoe or something. Ok, we'll rent rack space in another data center as backup. Own all the mechanisms between cutting between those two now. Congrats, you are now doing your own version of kubernetes.
What happens if your app gets huge volume during peak times, and very little volume during non-peak, and you find yourself overprovisioning to the point your CFO/CTO freaks out about the bill? Well, we'll make our own dynamic scaling mechanism. Congrats, you are now doing your own version of kubernetes.
What happens when your app traffic gets so large you start running into OS limitations, like file descriptor limits? Start trying some of the aforementioned solutions. Congrats, you are now doing your own version of kubernetes.
What happens if you need service discovery, monitoring, or ensure network isolation between various services? Different VM's + your own hacked together service mesh, or wire something in the VM. Congrats, you are now doing your own version of kubernetes.
What happens when you need to guarantee secret isolation between containers? Congrats, you are now doing your own version of kubernetes.
Let's say you don't actually need any of this or think you never will. Fine! That's valid. But what you don't want, is to suddenly hit some scale and any of these things (I could list way more but I feel I am belaboring the point), migrating off these setups can become a year+ project, if not way longer. I know because I have had to do this twice now. I cannot possibly overstate how painful it is.
So, people usually just go with kubernetes because 1) it is operationally not that hard to deal with compared to the things I just mentioned, and has a massive ecosystem and 2) the risk of the VM + container spiraling into complexity is perceived as way more than going more complex at the start.
You just used a load-balancer (an AWS one preferably) with a few machines behind it, optionally an auto-scale group. AWS has multi-region support built in.
Re: file descriptors -- that's not something handled by kubernetes, if anything you just have more layers now.
You don't need service discovery, you never did, Host your services at a private zone {service-name}.{env}.company.com zones which points to the loadbalancer.
You need monitoring and should use an observability solution for that, has 0 to do with kubernetes.
You're just taking a bunch of random entirely solved problems and for some reason suggesting kubernetes is somehow helpful for them.
>What happens if a container in the VM goes down or the app inside of it crashes, how do you recover?
Docker will restart container automatically. You don't have to do anything. Docker-compose will restart after VM restart. You don't have to do anything. If a VM goes down - I do have a HA (another VM at another provider) and DNS load balancing.
>Now you need some self-recovery mechanism via systemD or whatever, which will grow in complexity and fickleness over time. Congrats, you are now doing your own version of kubernetes.
While I don't like systemd, it does this automatically, while, it's not really used here.
> What happens when you need to upgrade/restart your VM? Ok, make a standby VM as backup that will mostly sit idle, or require a full-app redeploy any time you need to do anything to the first VM. Now you need to design a blue/green mechanism between them, and probably some networking layer work. Congrats, you are now doing your own version of kubernetes.
This has been pretty much answered already but, upgrades does not affect containers (unless docker engine upgrade). Restarts - docker will handle these automatically - nothing to do here.
> What happens, if running in cloud, you have a regional outage or degradation? Stand up another VM in another region and manage the networking layer between them. Or, if running locally, your ISP has an outage because of a backhoe or something. Ok, we'll rent rack space in another data center as backup. Own all the mechanisms between cutting between those two now. Congrats, you are now doing your own version of kubernetes.
This actually handles way better w/o managed kubernetes, as it's usually a single region and your cluster and workloads would simply be completely down, while mine would work, because of provider redundancy.
> What happens if your app gets huge volume during peak times, and very little volume during non-peak, and you find yourself overprovisioning to the point your CFO/CTO freaks out about the bill? Well, we'll make our own dynamic scaling mechanism. Congrats, you are now doing your own version of kubernetes.
Kubernetes with autoscaling wins hands down here, but, it's not automatic, nor hassle free. You are also assuming overprovisiong which is usually not the case for traffic spikes.
> What happens when your app traffic gets so large you start running into OS limitations, like file descriptor limits? Start trying some of the aforementioned solutions. Congrats, you are now doing your own version of kubernetes.
This also affects k8s, exactly the same way.
> What happens if you need service discovery, monitoring, or ensure network isolation between various services? Different VM's + your own hacked together service mesh, or wire something in the VM. Congrats, you are now doing your own version of kubernetes.
I do have service discovery and network isolation built into docker, thanks.
> What happens when you need to guarantee secret isolation between containers? Congrats, you are now doing your own version of kubernetes.
Believe it or not, it's the default with docker.
> Let's say you don't actually need any of this or think you never will. Fine! That's valid. But what you don't want, is to suddenly hit some scale and any of these things (I could list way more but I feel I am belaboring the point), migrating off these setups can become a year+ project, if not way longer. I know because I have had to do this twice now. I cannot possibly understate how painful it is.
All my workloads are containerized and I can just move them to a k8s cluster whenever I want, if needed.
2) the risk of the VM + container spiraling into complexity is perceived. as way more than going more complex at the start.
The risk of your k8s ecosystem spiraling into operators madness and argoapps over helmfiles all while trying to accommodate for ci/cd and costs offing the chart is - IMHO - way higher.
Rolling your own zero-downtime deployments is as about as a good idea as rolling your own security... it's not a good idea.
I run our small system on a single EC2 instance with K3s. It runs a half-dozen or so services and does it quite well. I don't think it is particularly complex or over engineered. I like how easy it is to maintain the configuration in a helm package and quickly deploy it in different environments.
There is a learning curve for doing K8s well, but that's true for any non-trivial system.
I think this line summarizes better than anything. Perfect example of how move fast and break things begins gloriously at first then inevitably, the breaking you thought you were doing hasn't even started and you find out what that part means.
I've always been the one saying "this is going to be a problem in a couple months" then I get shot down for "being negative." Then in a couple of of months when it fails I start getting aggression thrown at me "oh i know you want to say I told you so" and such even though I've never said such a thing when something fails. No. I would just like you to hear out my thoughts even when they may not be what you want to hear. We are all working towards the same goal.
sounds like ghetto engineering
But if someone wrote a blog to brag about not using k8s, they can't stop people from wanting to compare their work against k8s. If there's any arrogance in the air, it feels stronger on the other side.
EKS is literally one click of a button away and you dont need to handroll this.
even if you dont know AWS console nor terraform, claude code with aws mcp can do that for you
Out came Docker, dnsmasq, miles of duct tape and a whole lot of swearing. Just to come nowhere close to reinventing something better folks were doing years prior.
Just because you can (or think you can) doesn't mean you should. I sure do hope no one is maintaining that NIH monstrosity now!
(on iOS with dark mode enabled system wide)
I use it to keep from getting flashbanged by my monitor. In this case it also fixes the above site, however some websites need the color filter mode changed to work better, so realistically I’m not ending up with less fixing of websites, just easier fixing.