Posted by paweladamczuk 1 day ago
We are experiencing an issue with multiple services in
europe-west2 beginning on Monday, 2026-10-19 23:08 PDT.
Due to a power event we had suffered an outage within
europe-west2-b zone. The power event is mitigated. Partial
mitigation to infrastructure services will be observed within the
next 30 minutes and the remaining services will be mitigated there
after.
(Also reporting the times in PDT is quite unhelpful for users of this zone who are likely based in the UK or EU)
For example, AWS us-east-1 outages have caused global outages on multiple occasions, such as https://www.networkworld.com/article/4168878/aws-hit-by-us-e...
Disclaimer: I work for Google Cloud.
When did that change? I'm glad it did, but it seems like that was long in coming.
If you mean Google: Yes, extensive. One of the techniques they use is DiRT (Disaster Recovery Testing) which is a scheduled time-frame in which Google purposefully breaks systems that are exceeding their SLAs to verify that automated recovery executes as intended. In general, a DiRT test would catch something like "This service has a key dependency on us-east-1."
Now, in the past (also a Xoogler)... Some pieces of Cloud were moving at a cadence that they got away with a bit of a relaxed DiRT awhile back, so if someone tells me "I remember when a us-east-1 outage took down a key Cloud service" I would believe it. DiRT is also planned testing, it's not slapping a chaos monkey on the cluster fabric and crossing one's fingers, so it's entirely possible for key outage modalities to get missed (engineers are human and even though Google aims for a "no haunted closets" engineering practice, people develop a sixth sense of where the weak parts are of your infra and it takes courage to commit to pushing on those until they break).
Also keep in mind GCP has a global network so it's possible some other regions are seeing disconnects/packet drops, though the source of the issue is contained to a single datacenter.
Most things are at a regional level which is multiple zones. Your AWS us-east-1 is one such region. A lot of datacenters spread across a decently large physical area have to have an issue for a whole region to start having failures.
For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)
If AWS themselves can't handle cross region isolation and availability well, my expectations for most companies running on top of AWS are quite modest.
IAM is usually the culprit when us-east-1 issues affect other regions, but Lambda runs independently in every region.
here’s one third party, it was pretty well documented as a control plane failure in us-east-1
https://aws.amazon.com/message/101925/
This, too, was contained to us-east-1:
"While the event started at 11:48 PM PDT on October 19 and ended at 2:20 PM PDT on October 20, there were three distinct periods of impact to customer applications. First, between 11:48 PM on October 19 and 2:40 AM on October 20, Amazon DynamoDB experienced increased API error rates in the N. Virginia (us-east-1) Region. Second, between 5:30 AM and 2:09 PM on October 20, Network Load Balancer (NLB) experienced increased connection errors for some load balancers in the N. Virginia (us-east-1) Region. This was caused by health check failures in the NLB fleet, which resulted in increased connection errors on some NLBs. Third, between 2:25 AM and 10:36 AM on October 20, new EC2 instance launches failed and, while instance launches began to succeed from 10:37 AM, some newly launched instances experienced connectivity issues which were resolved by 1:50 PM."
my original claim was that their control plane runs in us-east-1. this is a fact.
The claim I'm testing is whether the impact spread to AWS services in other regions.
> At this point we’re playing semantic nittery.
It's not "semantic nittery." It's important as an engineer to be precise about the impact of incidents and how systems are architected, as opposed to making broadly sweeping (and false) claims that "[an entire] control plane runs in us-east-1" and "the whole fucking internet went down that day."
As someone who worked for AWS, I have insider knowledge that what you said is not true as a blanket statement. They generally take great care to isolate both control and data planes regionally, with rare exceptions where service data has to be globally consistent. Moreover, they have always recommended not to rely on control planes to respond to incidents: https://docs.aws.amazon.com/wellarchitected/latest/framework...
(who would be motivated to check if anything is restricted to only us-east-1)
Why do you say it's not supposed to happen? It's the predictable outcome of not having unlimited scale. The hyperscalers are big, but to be able to accommodate a tectonic shift in demand in a very compressed amount of time is very costly--costs that will be passed to their customers--if it can be done at all given current constraints. Today's hardware crunch, even for hyperscalers, is very real; customers are hoovering up spare capacity like never before.
As a former guidance counselor (specialist SA) for AWS, I regularly told customers that they shouldn't expect to be able to pack up and relocate into a new AZ once an incident commenced. They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.
Do you blame them? You not only have to pay double for the infra but also be charged for inter-region data transfers. And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
Well, yeah. Redundancy isn't free. I don't see how this is controversial.
> And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
What do you mean, "IAM doesn't work"? IAM has always worked, even during control plane impairments. You cannot depend on control plane operations to work during incidents; you should plan in advance for failure and have your orchestration already set up so that you don't have to touch a thing when you have to weigh away from an AZ or region.
https://docs.aws.amazon.com/wellarchitected/latest/framework...
That said, that's how Microsoft handles it, I'm not sure how GCP compares.
It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events.
AWS has always guaranteed that different AZs are physically separate sites, but that hasn’t always been the case with GCP.
im only familiar with microsoft in this sense, but they have "geo-zone-redundant storage" which copies data across 3+ availability zones in the primary region, and also a secondary geographic region (i think at least 100 miles away if i remember, but maybe further). any incident would have to be massive to affect multiple geo-zones
2. That article you linked to does not support your claim. That is a report of a single-zone incident impacting us-east-1 only. Five other AZs in the same region were still fully operational.
[1] These are not the same as control plane dependencies: you might not be able to make certain mutations, but everything should still work in the steady state.
One more than one occasion there are AWS URLs, including redirects inside the console, that include US East 1 in the URL that were only spotted during an outage. Manually changing it to an different region allowed us to get around it but to wholely blame customers when you don't know the context is bad form.
But there are also always exceptions. I've seen US East 1 outages cause autoscaling issues in other regions including failing to start new EC2s. A real problem if things have scaled down overnight.
Yeah, that tracks. The Console has often been wonky like that.
> I've seen US East 1 outages cause autoscaling issues in other regions including failing to start new EC2s
Do you mean scaling failed due to insufficient capacity, or something else? ICE errors I get (thundering herd of attempted capacity shifting); but otherwise, that sounds implausible, and I respectfully believe you are mistaken about cause and effect. Happy to evaluate a causal chain of evidence that proves you correct, though.
You are allowed to be wrong.
How do you know for certain that they are causally connected? Take me through it.
I will do better at saving exact details for internet strangers in future.
Much as with backups - if you don't test them, they don't exist.
Everywhere I've worked has had multi-AZ infrastructure. I have never worked anywhere where the failure of an AZ has been simulated regularly, with second-order effects mapped out.
It's like our on-prem systems were ordered to plan out and exercise DR plans to account for a data center going offline, but we can't exercise a DR plan fully if part of the plan is 'Adapt to how application X, which is hosted in the same datacenter, also chose to fail over.' We do not own application X. There is no appetite in the business to co-ordinate the DR exercise across the entire company at the same time.
We can't even do it all in a lower environment because there are dependencies on systems that are shared across environments. Some massive, redundant filesystem that is sync'd across our data centers, but the sync is a boolean. One datacenter instance of it is the primary and the others are read-only until they get promoted. We have one of these and lower environments are just a different root folder on the system with different access patterns, but they can't make DataCenter A the primary in prod and DataCenter B the primary in UAT.
Since most companies would have to do it in production and they don't want to have customer impacting outage, they tend to do it during an actual outage.
It seems like a lot of work, and it seems that as architectures evolve it would be hard to ensure that the multi-AZ failover works well.
Extra fun if there is a key vault involved, then you have to manually delete every key in it, otherwise you can't create new keys with the same names in the new vault.
I had a client that was contracting with a very large bank and there were only something like four zones globally that we could possibly deploy in to.
Google also is unlike AWS in that their zone abstraction is very virtual, and two zones might be in the same physical building. If you have requirements that your deployment zones be physically separated by some minimal distance then it can make things pretty challenging.
How do you to do that ?
> To protect against the loss of an entire region due to natural disaster, have a disaster recovery plan and know how to bring up your application in the unlikely event that your primary region is lost.
> The regions Stockholm, Osaka, and Montreal have three zones housed in one or two physical data centers.
See here: https://docs.cloud.google.com/docs/geography-and-regions
2023 europe-west9 incident: https://www.datacenterdynamics.com/en/news/water-leak-at-par...
If they at least included “(UTC -7)” after PDT, while still a bit annoying, at least people could work out what time that is without having to lookup PDT from somewhere first (as I just had to).
We just keep a clock on the wall set to Google time, like they do in newsrooms and NASA mission control. It's convenient enough.
It doesn't sound like it.
Southwest airlines does it.
Rackspace used to do it, to the point of needing to ask them pretty please stop provisioning hardware on central time. Obnoxious.
The argument for Florence Mean Time is that when you spread the map out, you get a nice split of the Pacific on either side. It's quite beautiful actually.
- Chuktoka isn't split and stays all together on the right side of the map - Zealandia all stays together on the right side, along with Samoa and Tokelau
It's also kinda poetic having the 'centre of time' being basically equidistant from Rome and Carthage.
So, being in Arizona where it's never DST, I called them out on it, and I continually bothered them when they used this "EST" notation. I told them that they were simply wrong to use it during Daylight Savings Time. I told them that MST and Eastern Time did not have a year-round relationship. They didn't care.
It turns out that a lot of very large corporations use "EST" (and other "standard time" notation) all year round. Including FedEx and UPS, for some reason.
It is obviously ridiculous, when you could simply use "ET" instead, and always be right, no matter what your DST status is. But they insisted on using this "EST" thing.
I had a conversation or two with some LLMs about it, and they rationalized it. But it is crazy to subject people to this "corporate fiction" that is obviously wrong/inaccurate for an entire half of the year, and it leads to a lot of problems.
The plan might've just been "if it gets bombed, we restore backups into other zone". Just waiting for it to fix itself for few hours mightv'e been acceptable risk.
1) the ability to view everyone's bank data, in secret
2) the ability to change EU bank data at will, in secret
(see US Cloud act)
There are much worse things you can do to data than bomb it.
Well then you should know that a Google Zone is not the same as an AWS AZ and many of Google regions are just firewalls across the same data center.... Despite all their SREs and Books and Phds...they dont know how to do infrastructure.
The overall point is accurate. Google expects you to spread things more across zones. Most users would rather zones not share buildings.
"The regions Stockholm, Osaka, and Montreal have three zones housed in one or two physical data centers." - https://docs.cloud.google.com/docs/geography-and-regions
"This means the multiple zones within each of these regions may not be located in physically separate data center campuses." - https://docs.cloud.google.com/backup-disaster-recovery/docs/...
"A region is a set of buildings operated by Google in a geographic location, such as a data center campus." - https://cloud.google.com/vpc/network-pricing
Yeah well, they drank the cloud kool-aid. The promise was that they'd go "cloud", and that everything would be taken care of.
Turns out: availability is worse than when those companies used to self-host and major cloud outages are happening all the time (at this point it's basically a meme).
And now the promise that "the cloud is simpler than doing it yourself" turns out to be not true?
I'm not surprised at all: these companies have been sold a lie.
You will probably have issues logging into Llyods even when the cloud is back up.
The usability of their mobile app is atrocious though. Good thing they have a website.
They have a stand-in system that uses a separate codebase and separate cloud provider. They have a blog post about it here: https://monzo.com/blog/tolerating-full-cloud-outages-with-mo...
The stand-in system has a limited subset of features, i.e. no bells and whistles, but enough that you can get any actual banking done.
(Not an ad / I have no connection to Monzo, I just had some appreciation for a bank doing this properly as normally banking IT is a disaster. Plus the idea of keeping a separate codebase for almost an entire bank's functionality seems very invested.)
I have always got bad vibes from Revolut...
Multi-cloud is really tough, tbh. It's easier said than done, especially if you care about data consistency at all.
It sounds like that story could make a great blog post
Help me understand, is the bad vibes you get from stories from the company culture? Or rather that the media put a target on their back (at at least in my opinion this is because they have a Russian born founder? Revolut is one of the best products I've ever used, I've honestly nothing but good things to say about them.
I opened up a Revolut account a couple of years ago with the intention of switching my current account to it (i.e. wages going into it etc), but I got spooked after hearing that they weren't registered as a UK bank and thus subject to the various regulations and protections. I tried using Chase instead and have been happy with it so far.
They are reigistered now
- The vast majority of their infra (at least as of ~3 years ago) is self-hosted in data centers that they own, and a lot of their other key cloud infra is/was multi-cloud (i.e. operations could fail over to GCP in the event of an AWS outage)
- Financially they have the healthiest balance sheet of any non-state-owned bank (like, in the entire world). Over the last ~20 years JPMC has acted as a sort of lender of last resort several times, most recently when they bought out First Republic.
- Politically, after navigating the aforementioned instability Jamie Dimon has almost carte blanche support from the shareholders, and everyone knows it. As a result, he can afford to be a lot less... compliant with the current US regime than other major players have been recently.
Another reminder that we need redundancy for all essential services that we rely on.
Perhaps this "bank" knows its phones would be down too.
> ssh root@eu-cloud
We value your privacy. We and our 847 partners track your network packets.
[Accept all] [Manage preferences]Happily hosting a site that will cross 4 billion queries per year there, and I didn’t face that banner you mention :-)
1) 4b queries a year is nothing. 4b queries is less than an hour for some of the software I have written.
2) The banner was a joke referencing the fact that the EU has a terrible track record with privacy.
There is literally no sensible reason that a bank should use another companies computers.
A lot of the information people are leveraging here comes from Downdetector which is mostly users reporting that stuff is down, not whether they are actually down. It's a really bad source.
If I had a nickel for every solid-green status page I shook my fist at while the actual site faceplanted on every request...
I've seen it happen over and over again when a wireless provider like Verizon or a docsis provider like Spectrum has a widespread outage that is isolated to their service: People push the wrong button.
Verizon issue, other providers fine? Verizon spikes on downdetector, but so do reports of AT&T and T-Mobile problems.
Same with Spectrum: When it's screwed up for whatever reason, reports for other ISPs also spike.
This happens even when those other systems are completely unaffected and working just fine.
Downdetector does provide useful data, but it's not necessarily clean data that establishes anything clear on its own.
"We would like to apologise for any inconvenience and continue to monitor the situation, which is also affecting other organisations."
This is affecting companies that put everything into a single GCP region in the UK for their entire business - aka most likely British companies or government with a data sovereignty requirement.
Sovereignty is when there is a UK company with multiple data centres in the country.
With that if one region goes down, another can carry on.
Most European countries have enough space within the country to do that, and, for most things, except banking typically, hosting within the EU is enough, it doesn't have to be within your own country.
A "full function" one in/around London. And a "tickbox" one for multi-region requirements near Cardiff.
The tickbox one has constant capacity constraints and a large number of Azure services aren't in it, but if you need to tick the "two regions and in country" requirement, you kinda/sorta can.
Wdym bigger fish, nuclear war has started and I can’t upload my video of a mushroom cloud to Tiktok!
But considering what these sovereignty rules aim to achive, and as someone mentioned, that the problem may be due to companies relying on a single region provided by a hypersclaler due to data sovereignity rules.
Then redundancy has to be considered by both the company and the rulers.
Sovereignty is just where stuff is, not how many place it is there.
But this is supposed to be about practical.
What's the purpose of ticking a box about housing data in UK when you can't make it available after a single incident.
I guess service reliability is just no longer something we consider a big deal in today's tech landscape.
I doubt it was of much interest outside the UK.
https://news.ycombinator.com/item?id=19280131
https://www.srf.ch/play/tv/kassensturz/video/internetbank-re...
Even the KYC process the first time I was making a larger transfer was surprisingly smooth.
Currency exchange fees/spread with "classical" banks are absolutely prohibitive.
They need to talk about some updates to the maintenance interact pricing...
Generally with acts like these: You don't have to be fully compliant in practice. You need to know how compliant you are, have a plan to improve things, and prove that you are working on it. But if you are "close enough" (depending on who audits you), non-compliance isn't a direct route to fines.