Posted by paweladamczuk 1 day ago
Why are banks allowed to have their services dependent on third party clouds.
Back in the old days the banks - with their vast amounts of money - took responsibility for running computer systems.
We were all sold the cloud on its reliability.
Now we are all stuck in it and everyone shrugs when its down "well it's the cloud, you know, its not very reliable".
Let me tell you a story about an idiot politician who woke up one day and thought Brexit would be a fantastic idea, and then the stupid dim-witted "ultra-hard brexit" manner in which it was implemented by a certain Mr Johnson.
The same reason UK people can't open EU bank accounts is the same reason EU banks have separate operating companies and infrastructure in the UK.
Given they operate in a heavily-regulated environment it almost certainly made life easier for them to keep UK stuff in the UK because they had no idea how (or if) the regulatory environment would ever progress beyond ultra-hard brexit.
Afaik, even Russia and Ukraine aren't having such issues while dealing with data centres getting droned, lol.
It's really quite poor performance.
Not having issues is overstatment, but Ukraine has a cheat of having a backup DC on NATO soil, which helps a lot.
They've had ~5yr to work on that.
The government should be resilience testing everything. Did Covid teach no lessons? Is the defence budget just for shits and giggles?
Will be a great day when Europe can finally say goodbye to the America taxes on most digital payments.
In terms of upfront costs, cloud is always cheaper, initially. There's sort of a weird sweet spot for cloud. If you're to small it makes no sense operationally, if you're to large, it just way to expensive.
Even for large companies, if you have somewhat stable usage most of the year, but periodically have usage spikes, cloud providers still remain super beneficial.
That's the whole selling point of cloud. Spins up the resources when you need them. Spin them down when you don't. Cloud thrives off companies spinning up resources and never spinning them down, even when they don't need them anymore.
If you're leasing in a building and put your equipment in it, you're usually outsourcing at least some of the facility management duties. What if the AC fails? What if there's a leak? Who manages the building's power contracts? Who handles on-site security? Do they own the generator and its maintenance or do you?
If you're at a colo, you're outsourcing even more management of the actual datacenter work inside the building on top of the rest of the facility management. Who is managing access control? Are they providing network connectivity? Are they providing backup power?
When the colo we were at lost power due to employees not following procedure while doing a power failover test (didn't match the phase), I still got the "you need to do something!" talk. Other than being ready to validate everything came up properly, there was nothing I could really do in the end though. We didn't have the budget to operate at multiple facilities, this was just an accepted risk. What, was I going to go out, negotitate a new datacenter contract, haul all the gear, rack it all up, and get it all deployed in the half hour or so they spent to validate things and get power restored?
Sounds like most of these companies were not designed to handle outages in a single availability zone, and so they went down.
If you put everything into configuration management, make scalable, redundant and interconnected servers and services built on top of it, you unlock amazing possibility of one error bringing hundreds if not thousands machines down.
Nice lil reminder that adding more things for redundancy only increases it if they are entirely unrelated to eachother and independent.
Which INCLUDES not running same software. You can have redundant switches, doesn't matter if they have software bug that triggers on all of them (or just one in a way that prevents network from working correctly).
You can have database cluster that can tolerate node dying just fine but still be brought down by a bug despise redundancy.
Can someone ask Claude, Codex, Copilot, Antigravity, Devin to resolve this outage?
No Googling or help from humans this time.