Posted by Torq_boi 2 days ago
I have a feeling AI is going to be similar in the future. Sure, you can "rent" access to LLM's and have agents doing all your code. And in the future, it'll likely be as good as most engineers today. But the tradeoff is that you are effectively renting your labor from a single source instead of having a distributed workforce. I don't know what the long-term ramifications are here, if any, but I thought it was an interesting parallel.
On the other hand, there's significant vendor lockin, complexity, etc. And I'm not really sure we actually end up with less people over time, headcount always expands over time, and there's always cool new projects like monitoring, observability, AI, etc.
My feeling is, if we rented 20-30 chunky machines and ran Linux on them, with k8s, we'd be 80% there. For specific things I'd still use AWS, like infinite S3 storage, or RDS instances for super-important data.
If I were to do a startup, I would almost certainly not base it off AWS (or other cloud), I'd do what I write above: run chunky servers on OVH (initially just 1-2), and use specific AWS services like S3 and RDS.
A bit unrelated to the above, but I'd also try to keep away from expensive SaaS like Jira, Slack, etc. I'd use the best self-hosted open source version, and be done with it. I'd try Gitea for git hosting, Mattermost for team chat, etc.
And actually, given the geo-political situation as an EU citizen, maybe I wouldn't even put my data on AWS at all and self-host that as well...
If you’re at the scale of hundreds of instances, that math changes significantly.
And a lot of it depends on what type of business you have and what percent of your budget hosting accounts for.
That said from the risk perspective I assume for what their doing in the data center there is low risk if downtime happens.
The opposite is also true: one is risking being banned by exascalers.
IIRC, Slurm came out of LLNL, and it finally made both usage and management of a cluster of nodes really easy and fun.
Compare Slurm to something like AWS Batch or Google Batch and just laugh at what the cloud has created...
(And it would be fun too.)
It was fun to build - especially Infiniband - but my next iteration is going to be a single beefy server, maybe with storage attached externally. What I had had outstanding uptime, but ultimately it was massively overkill, noisy, hot, and sucked power down.
While I have no intention to scale up low spec hardware like this, it at least seems to beat the Azure VMs we use at work with "4 CPUs", which corresponds to two physical cores on an AMD EPYC CPU.
And that super slow machine I understand costs more than $100 per month, and that's without charges for disk space slower than the SSD, or network traffic.
Renting at Azure seems to be a terrible decision, particularly for desktop use.
Ps... bx cable instead of conduit for electrical looks cringe.
I'm imagining a setup that can work like this:
- I can purchase/lease from some vendor (maybe even a used dell 1U) and have it sent directly to them and they construct and install (same with ssd replacements, ram upgrade, etc.).
- They can setup remote KVM over IP access if needed.
- I never have to drive to their facility, but based in the US.
I'm willing to trade off some control and turnaround time here. The idea is to have something like a $500/month VPS but with a higher upfront cost and lower monthly cost for space, power, and bandwidth.
Budget hosts such as Hetzner/OVH have been known to suddenly pull the plug for no reason.
My kit is old, second hand old (Cisco UCS 220 M5, 2xDell somethings) and last night I just discovered I can throw in two NVIDIA T4's and turn it in to a personal LLM.
I'm quite excited having my own colocated server with basic LLM abilities. My own hardware with my own data and my own cables. Just need my own IP's now.
The same would apply for any number of hosts. Hetzner/OVH are cheap, but as your own numbers show the location price gap is more than sufficient to cover the costs of servers.
In fact you can colocate with Hetzner too, and you'd get a similar price gap - the lower cost of real-estate is a large part of the reason why they can be as cheap as they are.
Data centre operations is a real estate play - to the point that at least one UK data centre operator is owned by a real estate investment company.
Where I feel that data has become a commodity in that I can sell your username and email for a few pence, I would rather prefer to have my own hardware in my own possession and that any request of it has to go to me, nor some server provider.
This is one reason I hate dealing with AWS. It feels like a waste of time in some ways. Like learning a fly-by-night javascript library - maybe I'm better off spending that time writing the functionality on my own, to increase my knowledge and familiarity?