AWS says it can't restore some data from mideast facilities struck by Iran

Posted by berkeleyjunk 1 day ago

AWS says it can't restore some data from mideast facilities struck by Iran(www.wsj.com)

393 points | 331 commentspage 2

KingOfCoders 2 hours ago|

If all your backups are with AWS you don't have backups at all.

weinzierl 9 hours ago||

   me (Middle East)
   ├── me-south-1 (Bahrain)                                DOWN since 2026-04
   │   ├── mes1-az1                            me (Middle East)

├── me-south-1 (Bahrain) DOWN since 2026-04 │ ├── mes1-az1 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3

nerdile 4 hours ago|

Ok claude, now format that in a way humans can read

weinzierl 3 hours ago||

Sorry, I accidentally posted it twice while fixing it and on a slow connection.

dhanudhakne 3 hours ago|||

Yas

dhanudhakne 3 hours ago||||

Okk

dhanudhakne 2 hours ago|||

Iii

markive 1 day ago||

Does this mean that even with 3 availability zones for Amazon S3 storage, that some data is lost?

OrangeDelonge 1 day ago||

Did they say its S3 data? Could also be single-az EBS or RDS.

MiroslavPokorny 1 day ago|||

Obviously what you understand is different from the reality after you actually follow all the footnotes.

dhx 1 day ago|||

For me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a single government--something customers of AWS were hopefully aware of when they entrusted AWS with their data.

It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures.

I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider.

[1] https://news.ycombinator.com/item?id=49033240

[2] https://en.wikipedia.org/wiki/Failure_Mode,_Effects,_and_Cri...

0cf8612b2e1e 12 hours ago|||

As far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?

testplzignore 12 hours ago|||

https://aws.amazon.com/compliance/data-privacy-faq/

"You choose the AWS Region(s) in which your content is stored. You can replicate and back up your content in more than one AWS Region. We will not move or replicate your content outside of your chosen AWS Region(s) without your agreement."

ericpruitt 12 hours ago||||

That would be a legal nightmare. They don't necessarily know what customers' data residency requirements are.

bumblehean 11 hours ago||

This. We have (well, had) customers running in me-south-1 and once the first AZ went down we wanted to proactively move their data to other regions even just as cold backups. But our legal department slapped that down pretty quickly.

sparkling 11 hours ago|||

Most likely, their own data residency terms prohibit this. It would be interesting to know if, when 2 out of 3 AZs got destroyed, customers got a heads up to move their data to a different region?

leftbehind 11 hours ago||

We received repeated, constant heads up to move our data by the first AZ much less second. The problem is that nobody is storing data in Bahrain unless there are data residency requirements for it.

nobody wakes up one morning and chooses to launch instances, CDN or S3 and would choose Bahrain as that without a requirement to, we were contractually and legally forbidden (in the middle as a vendor) to copy even encrypted data where we don't have the key out for redundancy, so the best we could do was tell our subcustomers to download all of their buckets to their office or some employee laptops at their office

zmgsabst 12 hours ago|||

AZs weren’t meant to be disaster resistant, eg, an earthquake or hurricane could take out a whole region.

Regions were always the scale of disaster isolation on AWS.

flumpcakes 11 hours ago||

Regions are really the scale of disaster isolation only in extreme cases - such as global catastrophe (meteor strike taking out a city) or in this case, when actively targeted in war. I don't really see the same thing happening to a US or European region.

gregw2 12 hours ago||

I think so.

Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.

weinzierl 9 hours ago||

The data center layout should look something like this:

    me (Middle East)
    ├── me-south-1 (Bahrain) DOWN since 2026-04
    │   ├── mes1-az1 DOWN
    │   │   └── mes1-mct1-az1 (Oman, Muscat) ???
    │   ├── mes1-az2 DOWN since 2026-03-01
    │   └── mes1-az3 DOWN
    ├── me-central-1 (United Arab Emirates)
    │   ├── mec1-az1
    │   ├── mec1-az2 DOWN since 2026-03-01
    │   └── mec1-az3         DOWN since 2026-03-01
    └── il-central-1 (Israel, Tel Aviv)
        ├── ilc1-az1
        ├── ilc1-az2
        └── ilc1-az3

Not sure about the Muscat local zone, whole me-south-1 region has been reported down despite Muscat still being operational.

If someone had told me a year ago that a whole AWS region could go down I'd called them crazy, but now me is close to exactly that happening.

See also previous discussion: https://news.ycombinator.com/item?id=49033240

chasd00 11 hours ago||

They say "some" data, i wonder what percentage that really is. I haven't seen pictures but I find it hard to imagine all of me-south-1 was completely leveled to the point where's there's just nothing left. On the other hand, if you have 100 rows of racks and then randomly take out a contiguous 10% across both rows and columns it may be functionally equivalent to taking out everything.

tgsovlerkhgsel 11 hours ago|

I wouldn't be surprised if the engineers said "we can probably recover between 20-30% of the data but it will cost 200 hours of engineering and the data will be 7 months old by then" and the beancounters said "we'd rather have one news cycle rather than the news watching what we can and cannot recover + save those 200 hours, we'll just say it's all gone".

skybrian 12 hours ago||

I wonder if they'll start adding an underground bunker to new data centers so you can put an S3 replica there?

expedition32 1 hour ago||

I suppose this is why Microsoft and Google want to build data centers in the Netherlands. Its not cheap but it is safe.

genxy 2 hours ago||

So many AWS simps in this thread that don't actually understand how AWS scrimped out and fucked their customers. You can't all suck up to AWS at the same time, you need to serialize.

Eastmill 8 hours ago||

Seems like their multi-AZ redundancy didn't account for missile strikes. Good reminder to always have your own backups.

brightball 8 hours ago|

There are many regulations over there which prevent data from leaving the country.

Sometimes those rules can have serious consequences.

More comments...