Posted by Torq_boi 1 day ago
No, low isn't good perse. I worked in a datacenter which in winters had less than 40%, ram was failing all over the place. Low humidity causes static electricity.
Things would be different in a colder climate where humidity goes --> 0% in the winter
It is much cheaper to use external air for cooling if you can.
Also this is where cutting corner indeed results in lower cost, which was the reason for the OP to begin with. It just means you won't get as good a datacenter as people who are actually tuning this whole day and have decades of experience.
RAM that is plugged in and operating isn't subject to external ESD, unless you count lightning strikes. Where are you getting this?
Old hardware is _plenty_ powerful for a lot of tasks today.
Pains I faced running BIG clusters on-prem.
1. Supply chain Management -- everything from power supplies all the way to GPUs and storage has to be procured, shipped, disassembled and installed. You need labor pool and dedicated management.
2. Inventory Management -- You also need to manage inventory on hand for parts that WILL fail. You can expect 20% of your cluster to have some degree of issues on an ongoing basis
3. Networking and security -- You are on your own defending your network or have to pay a ton of money to vendors to come in and help you. Even with the simplest of storage clusters, we've had to deal with pretty sophisticated attacks.
When I ran massive clusters, I had a large team dealing with these. Obviously, with PaaS, you dont need anyone.
I have had a similar transformation. I still host non-critical services on-prem. They are exceptionally cheap to run. Everything else, I host it on Hetzner.
[1] https://www.techradar.com/news/remember-the-ovhcloud-data-ce...
[2] https://blocksandfiles.com/wp-content/uploads/2023/03/ovhclo...
When someone point out how safe are cloud providers, as if they have multiple levels of redundancy and are fully protected against even an alien invasion, I remember the OVH fire.
It's their "Compute" under "Public Cloud" that is competing against AWS EC2. https://us.ovhcloud.com/public-cloud/compute/
They handled the fire terribly and after that they improved a bit, but an OVH VPS is just a VM running on a single piece of hardware. Quite not the same thing as the "Compute" which is running on clusters.
Something very similar happened at work. Water valve monitoring wasn’t up yet. Fire didn’t respond because reasons. Huge amount of water flooded over a 3 day weekend. Total loss.
why build one when you can have two at twice the price?
But, if you're building a datacenter for $5M, spending $10-15M for redundant datacenters (even with extra networking costs), would still be cheaper than their estimated $25M cloud costs.
You need however plan for 1MM+ pa in OPEX because good SREs ain’t cheap (or hardware guys building and maintaining machines)
https://intellectia.ai/news/stock/ibm-mainframe-business-ach...
60% YoY growth is pretty excellent for an "outdated" technology.
It's easy to inspire people when you have great engineers in the first place. That's a given at a place like comma.ai, but there are many companies out there where administering a datacenter is far beyond their core competencies.
I feel like skilled engineers have a hard time understanding the trade-offs from cloud companies. The same way that comma.ai employees likely don't have an in-house canteen, it can make sense to focus on what you are good at and outsource the rest.
They spend too much time on yet another cloud native support group call, learning for ThatOneCloudProvider certificates, figuring out that single implementation caveats, standardizing security procedures between cloud teams, and so on.
Yet people in the article just throw a 1000 lines of code KV store mkv [0] on a huge raw storage server and call it a day. And it's a legit choice, they did actual study beforehand and concluded: we don't need redundancy in most cases. At all. I respect that.
The cloud requires expertise in company-specific APIs and billing systems. A data center requires knowledge of Watts, bits, and FLOPs. I know which one I rather think about.
This company sounds more like a hobby interest than a business focused on solving genuine problems.
Re: the "hobby" part is where I agree with you the most. Where you say it's not solving genuine problems is where I differ the most.
It really feels to me like Comma is staffed by people who recognize that they never stopped enjoying playing with Lego -- their bricks just grew up, and they realized they can:
1) solve real-world problems
2) not be jerks about it
3) get paid to do it
Not everything has to be about optimizing for #3.
I'm a happy paying customer of Comma.ai (Comma four, baby!) -- their product is awesome, extremely consumer-friendly, and I hope they can grow in their success!
glad you're enjoying it :)
This is becoming increasingly common as far as I can tell.
There are benefits either direction, and I think that each company needs to evaluate the pros and cons themselves. Emotional pros/cons are something companies need to evaluate as employee morale can make or break a company. If the company is super technical in culture and they gain something intangible that is boosting the bottom line, having a datacenter as a "cool" factor is probably worth it.
Cost and lock-in are obvious factors, but "sovereignty" has also become a key factor in the sales cycle, at least in Europe.
Handing health data, Juvoly is happy to run AI work loads on premise.
