Top
Best
New

Posted by Torq_boi 2 days ago

Don't rent the cloud, own instead(blog.comma.ai)
1178 points | 493 commentspage 6
yawnxyz 12 hours ago|
running your own ai inference is quite stressful, and reading this article definitely makes me feel stressed
ex-aws-dude 1 day ago||
I can see how this would work fine if the primary purpose is for training rather than serving large volumes of customer traffic in multiple regions

It would probably even make sense for some companies to still use cloud for their API but do the training on prem as that may be the expensive part.

dh2022 1 day ago||
LOL’ed IRL at “ In a future blog post I hope I can tell you about how we produce our own power and you should too.” Producing own power as a pre-requisite for running on-prem is a non-starter for many.
asdfman123 1 day ago|
"It's really not hard to create your own coal power. Our engineers have built a small coal power generator and simply get coal from our mines (which I'll describe in a future blog post)."
dh2022 1 day ago||
LOL'ed again IRL :).
bovermyer 1 day ago||
I'm thinking about doing a research project at my university looking into distributed "data centers" hosted by communities instead of centralized cloud providers.

The trick is in how to create mostly self-maintaining deployable/swappable data centers at low cost...

RT_max 2 days ago||
The observation about incentives is underappreciated here. When your compute is fixed, engineers optimize code. When compute is a budget line, engineers optimize slide decks. That's not really a cloud vs on-prem argument, it's a psychology-of-engineering argument.
dagi3d 2 days ago||
> San Diego power cost is over 40c/kWh, ~3x the global average. It’s a ripoff, and overpriced simply due to political dysfunction.

Mind anyone elaborate? Always thought this is was a direct cause of the free market. Not sure if by dysfunction the op means lack of intervention.

amluto 2 days ago||
Did you say “free market”? There is one provider. There is a lot of regulation, mostly incompetent. It’s a mess.
omoikane 1 day ago|||
Electricity cost in California is generally more expensive than most other US states, except Hawaii. Not sure why.

Perhaps Comma needed the datacenter to be in San Diego for latency or other reasons, but if they need it mostly for compute, it would have been cheaper to operate their datacenter elsewhere... but if we keep going down that path, maybe it actually becomes cheaper to rent a cloud after all.

throwawaypath 1 day ago||
>Mind anyone elaborate? Always thought this is was a direct cause of the free market. Not sure if by dysfunction the op means lack of intervention.

The majority of Californians have no say and cannot choose their utilities provider. This is the polar opposite of the "free market".

sakopov 1 day ago||
Does anyone remember how cloud prices used to trend down? That was about 6 years ago and then seemingly after the pandemic everything started going the other way.
throwaway-aws9 1 day ago||
The cloud is a psyop, a scam. Except at the tiniest free-tier / near free-tier use cases, or true scale to zero setups.

I've helped a startup with 2.5M revenue reduce their cloud spend from close to 2M/yr to below 1M/yr. They could have reached 250k/yr renting bare-metal servers. Probably 100k/yr in colos by spending 250k once on hardware. They had the staff to do it but the CEO was too scared.

Cloud evangelism (is it advocacy now?) messed up the minds of swaths of software engineers. Suddenly costs didn't matter and scaling was the answer to poor designs. Sizing your resource requirements became a lost art, and getting into reaction mode became law.

Welcome to "move fast and get out of business", all enabled by cloud architecture blogs that recommend tight integration with vendor lock-in mechanisms.

Use the cloud to move fast, but stick to cloud-agnostic tooling so that it doesn't suck you in forever.

I've seen how much cloud vendors are willing to spend to get business. That's when you realize just how massive their margins are.

re-thc 1 day ago|
> The cloud is a psyop, a scam.

You're just young.

> Suddenly costs didn't matter and scaling was the answer to poor designs.

It did.

Did you know that cloud cost less than what the internal IT team at a company would charge you?

Let's say you worked on product A for a company and needed additional VM. Besides paperwork, the cost to you (for your cost center) would be more than using the company credit card for the cloud.

> Sizing your resource requirements became a lost art

In what way? We used to size for 2-4x since getting additional resources (for the in-house team) would be weeks to months. Same old - just cloud edition.

throwaway-aws9 1 day ago||
> You're just young.

And I feel great!

> Did you know that cloud cost less than what the internal IT team at a company would charge you?

Yes. Internal IT teams ran old-school are inefficient. And that's what the vendor tells you while they create shadow IT inside your company. Skip ITSM and ITIL... do it the SRE way.

Until the cloud economist (real role) comes in and finds a way to extract more rent out of their customer base (like GCP's upcoming doubling rates on CDN Interconnect). And until internal IT kills shadow IT and regains management of cloud deployments. Cybersecurity and stuff...

Back to square one. ITIL with cloud deployments. Some use cases will be way cheaper... but for your 100s of PBs of enterprise data, that's another story. And data gravity will kill many initiatives just based on bit movement costs.

> Besides paperwork, the cost to you (for your cost center) would be more than using the company credit card for the cloud.

To some extent. One is hard dollars the other is funny money. But I thought paying for cloud with the company credit card was a 2016 thing. Now it's paid through your internal IT cost center, with internal IT markup.

I've seen petabytes of data move to the cloud and then we couldn't perform some queries on it anymore as that store wouldn't support it, and we'd need to spend 7 figures to move to another cloud database to query it. And that's hard dollars.

Yes, during early cloud days it was lean and aimed at startups. Now it's aimed at enterprise, and for some reason lots of startups still think it's optimized for them. It's not and it hasn't been for a long time.

re-thc 1 day ago||
> Yes. Internal IT teams ran old-school are inefficient.

They aren't. It's politics. They want to protect and improve their own headcount and resources.

> One is hard dollars the other is funny money.

All the same to a team / department. It's not like people run it like their own wallet.

> finds a way to extract more rent out of their customer base

I find you just have a grudge against the cloud and hence too young. For every example you have the so-called "internal" IT team can and will do just the same. Go back to 90s, 00s - it was the same. The infra team wanted some fancy new storage arrays and charge everyone 2x for the new service etc.

> and for some reason lots of startups still think it's optimized for them. It's not and it hasn't been for a long time.

The problem isn't the cloud. Startups have always worked like this even 10-20 years ago. It's about wastage. They can raise and grow faster. So they think. The problem, if any is recently money isn't as cheap. Nothing new.

throwaway-aws9 1 day ago||
> the so-called "internal" IT team can and will do just the same.

but how is shadow IT gonna solve anything? it'll get kudos from the junior VP, chuckles from the SVP, but the CIO will laugh you out of the room at how poor you are at getting shit done internally.

> The problem isn't the cloud.

The problem is how gullible folks are at cloud advocacy, or any vendor advocacy in general. It's all lies but cloud lies are better than others! Your 3-year commitment won't scale down to low figures. Oh you wanna have that many nodes come black Friday? Gotta reserve! Yup, infinite scale actually means infinite lies.

Above all, the cloud is not cheap. 11B profit on 33B revenue per quarter at AWS. If your local IT spend is inefficient, I bet it won't be more efficient in the cloud.

faust201 2 days ago||
Look the bottom of that page:

An error occurred: API rate limit already exceeded for installation ID 73591946.

Error from https://giscus.app/

Fellow says one thing and uses another.

nickorlow 1 day ago|
Even at the personal blog level, I'd argue it's worth it to run your own server (even if it's just an old PC in a closet). Gets you on the path to running a home lab.
drnick1 1 day ago|
Absolutely. I don't have a blog but run my own email, several game servers, Matrix instance, Nextcloud and other internal services on a retired gaming PC. The total cost of my cloud subscriptions is $0, and no one is snooping on me. It's a great setup when combined with Linux machines and GrapheneOS phones, completely private and free of Big Tech.
More comments...