The Word Doing All the Work in the AWS Data Loss Notice

AWS confirmed in September that data held exclusively in me-south-1 and mec1-az2 cannot be restored. Multi-AZ worked as designed.

Share
The Word Doing All the Work in the AWS Data Loss Notice. Feature image on debugly.dev

It ended at the whole lesson. It started at the region. Every hand-off in between did exactly what it was told, and this is the one that lied.

In mid-September 2026 AWS posted an update to its Health Dashboard saying that after a thorough assessment it is unable to restore access to the resources and data hosted exclusively in its Bahrain region, me-south-1, and exclusively in the mec1-az2 availability zone of its UAE region. The damage followed drone strikes in March, a second Bahrain availability zone disruption in April, and, in AWS's words, infrastructure damage that spanned multiple availability zones and exceeded what its regional and multi-AZ services are designed to withstand.

This is being discussed again this week, and it is worth reading carefully, because the engineering lesson has nothing to do with the conflict and everything to do with a distinction a lot of architectures quietly blur.

What multi-AZ actually protects against

Availability zones are physically separate data centres within a region, and AWS is explicit that they are separated by many kilometres but kept within roughly 100 kilometres of each other so they can exchange data over low-latency networks. That distance is a deliberate engineering trade. It is close enough for synchronous replication and far enough to survive the failure modes the design targets: power outages, fires, floods, cooling failures, lightning, tornadoes, earthquakes.

What it does not survive is an event that reaches several nearby facilities at once. A hundred kilometres is a large distance for a flood and a small distance for anything deliberately aimed. That is not a design flaw so much as a design boundary, and the boundary was documented long before anyone tested it.

Mike Chapple of the University of Notre Dame made the related point that losing several data centres within an availability zone can leave too little capacity to absorb the workloads even when other zones are still standing, which is a second failure mode sitting on top of the data loss.

Read the word exclusively

The AWS notice is precise, and the precision is the lesson. It cannot restore data hosted exclusively in me-south-1, or exclusively in mec1-az2. Customers who had replicated elsewhere recovered. Customers who had not, did not.

AWS said most affected customers re-established operations in other regions by restoring backups or copying data that remained accessible, and that recovery depended on each customer's own replication and disaster-recovery configuration. That is the shared responsibility model operating exactly as written, and it is the part that is easy to skim past: the platform's redundancy and your data's recoverability are related problems but they are not the same problem.

For many services, including standard S3 storage classes, data is stored redundantly across at least three availability zones in a region. That is real protection and it is genuinely useful. It is also, in a regional disaster, three copies of the same thing in the same hundred kilometres.

This is the same distinction as the write-ahead log and what committed actually promises, where a guarantee is only as strong as the boundary it was defined against. Durability against a process crash and durability against a region are different contracts, and reading the first as if it were the second is how the gap goes unnoticed until it cannot be closed.

The question nobody had to answer before

The interesting unresolved problem is geographic concentration, and it has a sharp edge that has nothing to do with warfare.

Cross-region replication is the standard answer to regional loss. Data residency law is frequently the reason you cannot use it. If your regulator requires that certain data never leave national borders, and your country has one region, then the architecturally correct answer and the legally permitted answer are mutually exclusive. That tension existed before March and it has no clean technical resolution.

There is also the quieter version of the same problem. Two copies in two availability zones feel like redundancy. Two copies in two regions feel like disaster recovery. Two copies in two regions operated by the same provider, on the same network, in the same legal jurisdiction, are something in between, and most architecture diagrams do not have a symbol for it.

What I would actually check

Inventory what lives in exactly one place. The AWS wording gives you the exact query to run against your own estate: which resources are hosted exclusively in a single availability zone, and which are exclusively in a single region. Single-AZ databases, single-AZ EBS volumes, unreplicated snapshots and anything pinned by a subnet are the usual answers.

Test the restore, not the backup. A backup that has never been restored is a hypothesis. The teams that recovered in March were the ones who had already done the restoration procedure, not the ones who had the most snapshots.

Decide the residency trade explicitly rather than by default. If residency rules prevent cross-region copies, write down what the accepted loss is. An undocumented default is a decision nobody made.

Do not read the uptime SLA as a durability guarantee. Availability and recoverability are measured differently and fail differently. A service can be restored to operation while data remains permanently inaccessible, which is precisely what happened here.

Treat distance as a variable. Ask how far apart two copies need to be to be independent against the failure you are actually worried about. For a fire, a kilometre. For a regional event, another region. For a jurisdictional event, another provider.

The rule

Multi-AZ redundancy protects against a single facility failing, not against an event that reaches several nearby facilities, and the difference is the entire scope of the guarantee. Inventory what lives in exactly one zone or one region, test restoration rather than backup existence, and make the residency-versus-replication trade an explicit documented decision.

The part of this story worth carrying forward is not that AWS lost data, it is that the notice had to use the word exclusively, and the reason it had to use that word is that redundancy within a region and recovery from a region are different contracts. Most of us read the first one and priced in the second, and until September that had never been tested at scale, so nobody had to notice.


Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud