I Accidentally Deleted Our Production S3 Bucket — Here's What Happened in the Next 4 Hours

The command ran in under a second. The recovery took 4 hours. Here's the full story, what we lost, what we saved, and what we now have in place so it can never happen again.

There's a moment every cloud engineer dreads.

The cursor blinks. The command runs. And then a silence that feels different from normal silence.

aws s3 rb s3://my-app-production-uploads --force

I was targeting the staging bucket. I was in the wrong terminal window.

In under a second, 3 years of user-uploaded files — profile pictures, documents, exported reports — were gone.

This is what happened next.

The First 60 Seconds

The first thing you do when something goes catastrophically wrong is nothing.

Not because you're calm. Because your brain genuinely cannot process what just happened. I stared at the terminal output for probably 10 seconds before the word production registered.

Then I checked the AWS console. The bucket was gone. Not empty — gone. No objects, no metadata, no bucket at all. S3 bucket deletion removes the bucket itself, not just the contents.

I looked at the clock: 11:47 PM.

I typed one message in Slack: "I need help. Production S3 bucket just got deleted. Staging terminal wrong window."

Three people came online within 2 minutes. That response time, at midnight, is what actually saved us.

What Was in the Bucket

Before I explain the recovery, you need to understand what we lost — or thought we lost:

• User profile pictures — every account photo uploaded since launch
• Document uploads — files users had submitted through the app
• Exported reports — generated PDFs that users could re-download
• Some application assets — a handful of static files referenced in the app

The user-facing impact was immediate. Profile pictures showed broken image icons. Document download links returned 403. The export feature threw errors.

It was 11:47 PM on a Tuesday. Low traffic. That was the only lucky part.

The Recovery Attempt — Hour 1

The first thing we checked: S3 Versioning.

S3 Versioning, when enabled, keeps every version of every object including delete markers. When you delete an object with versioning on, S3 adds a delete marker rather than actually removing the data. The object is "deleted" but recoverable.

We had versioning enabled on the production bucket.

The problem: deleting the bucket itself with --force does something different. It first deletes all objects and versions, then deletes the bucket. By the time the bucket is gone, the versioned objects are gone too.

Versioning alone didn't save us. But it was a clue that we were thinking about this correctly — we just needed another layer.

Then someone asked the question that changed the next 3 hours:

"Do we have Cross-Region Replication set up?"

The Recovery — Hour 2

We did have Cross-Region Replication (CRR) configured. Set up 8 months earlier as a compliance requirement, never actually tested, mostly forgotten about.

CRR continuously replicates objects from a source bucket to a destination bucket in another region. It runs automatically, in the background, silently.

We checked the destination bucket.

It was there. All of it. Every object, replicated.

Source bucket (deleted):      0 objects
Destination bucket (intact):  47,832 objects ✅

47,832 files. All of them. The replication had been running every time someone uploaded anything, for 8 months, without us ever consciously thinking about it.

The recovery process:

1. Create a new source bucket with the same name in the original region
2. Copy all objects from the replica bucket back to the new source bucket
3. Update bucket policies, CORS configuration, and lifecycle rules to match original
4. Re-enable CRR on the new bucket pointing to the replica
5. Verify application could read all files

# Copy everything from replica back to new source bucket
aws s3 sync \
  s3://my-app-production-uploads-replica \
  s3://my-app-production-uploads \
  --source-region us-west-2 \
  --region ap-south-1

The sync took 47 minutes for 47,832 files.

At 2:34 AM — 2 hours and 47 minutes after the deletion — every broken image was loading again. Every download link worked. Every export was accessible.

Total permanent data loss: zero.

What We Lost That We Couldn't Recover

Not everything was perfect.

The bucket policy was gone. We had to reconstruct it from memory and the IAM policy attached to our application role. Close enough, but not exact — there were edge case permissions we discovered were missing over the following week when specific features started throwing permission errors.

The lifecycle rules were gone. We had automated deletion of files older than 365 days for GDPR compliance. Those rules had to be manually recreated. We found them eventually in a Terraform file that was slightly out of date.

The CORS configuration took three attempts to get right. We had it in a config file but the file wasn't fully updated — there were two origin entries we'd added manually and never committed.

Lesson: the bucket itself is infrastructure. Treat it like infrastructure. Version-control every policy, every lifecycle rule, every CORS config.

The 4 Layers We Now Have in Place

Layer 1 — S3 Versioning (Was Already On)

Versioning doesn't protect against bucket deletion with --force but it protects against accidental object deletion and overwrites. Keep it on. Always.

Layer 2 — Cross-Region Replication (Was Already On, Saved Us)

CRR was our actual recovery mechanism. The lesson: don't set it up and forget it. Test it. We had never run a recovery drill. We got lucky that it was working correctly when we needed it.

Test your replication quarterly:

• Delete a test object from source
• Verify it still exists in replica
• Restore it
• Confirm restoration worked

Layer 3 — S3 Object Lock (Added After)

S3 Object Lock in Compliance mode prevents objects from being deleted by anyone — including root account users — for a defined retention period.

For our document uploads that have legal retention requirements, we now use Object Lock with a 90-day retention window. Even aws s3 rb --force cannot delete locked objects.

Layer 4 — IAM Boundary on Deployment Credentials (Added After)

The access key I was using had full S3 permissions. It didn't need them. Our deployment pipeline only needs PutObject and GetObject on specific buckets.

We added a permission boundary that explicitly denies s3:DeleteBucket and s3:DeleteObject on production buckets for all non-root credentials.

{
  "Effect": "Deny",
  "Action": [
    "s3:DeleteBucket",
    "s3:DeleteObject",
    "s3:DeleteObjectVersion"
  ],
  "Resource": [
    "arn:aws:s3:::my-app-production-*",
    "arn:aws:s3:::my-app-production-*/*"
  ]
}

This policy means even if someone runs aws s3 rb --force on a production bucket, the API call returns AccessDenied. The bucket stays.

The Conversation We Had to Have

At 3 AM, with everything recovered and the immediate crisis over, the four of us stayed on the call for another 30 minutes.

Not to assign blame. Not to write a post-mortem. Just to talk through what had actually happened and what it meant.

The thing nobody said but everyone was thinking: we got lucky.

Lucky that CRR was set up. Lucky that it had been running correctly for 8 months without us verifying it. Lucky that traffic was low. Lucky that three people responded at midnight.

A production incident that resolves with zero data loss still costs something. It costs 4 hours of sleep, a spike of cortisol that doesn't fully fade for days, and a team that now moves a little more carefully around production resources than they did before.

That carefulness is worth more than any technical safeguard.

What Actually Protects Your S3 Data

Not one thing. All of these, in layers:

Layer 1 — Versioning
  Protects against: accidental object deletion, overwrites
  Does NOT protect: bucket deletion with --force

Layer 2 — Cross-Region Replication  
  Protects against: bucket deletion, regional failure
  Requirement: test it before you need it

Layer 3 — S3 Object Lock
  Protects against: everything, including root account
  Best for: compliance data, legal holds, critical assets

Layer 4 — IAM Deny on DeleteBucket/DeleteObject
  Protects against: credential misuse, wrong terminal window
  Best practice: least privilege on all non-root credentials

Layer 5 — AWS Backup for S3
  Protects against: all of the above
  Adds: point-in-time recovery, automated backup schedules

You don't need all five on every bucket. But you need to know which layers you have, verify they work, and match protection level to data criticality.

The Checklist I Wish I Had Before That Night

Before every AWS CLI session:
☐ Check which terminal window is active
☐ Check which AWS profile is set (aws sts get-caller-identity)
☐ Double-check the resource name before any destructive command

For every production S3 bucket:
☐ Versioning: enabled
☐ Cross-Region Replication: enabled and tested
☐ MFA Delete: enabled for critical buckets
☐ IAM Deny policy: DeleteBucket + DeleteObject on prod buckets
☐ Bucket policy + lifecycle rules: version-controlled in git

Recovery drills (quarterly):
☐ Delete a test object, verify replica has it, restore it
☐ Confirm CORS, bucket policy, lifecycle rules are in source control
☐ Know the exact sync command to run if you need it at 2am

One Last Thing

Before you close this tab — run this:

aws sts get-caller-identity

Check which account you're in right now.

If the answer is your production account and you don't need to be there, switch profiles before you do anything else.

That command takes 2 seconds. The recovery takes 4 hours.

Have a recovery story of your own? Drop it in the comments. The more of these we share, the less likely someone else has to learn the same lesson at midnight.

Tags: #AWS #S3 #DevOps #Cloud #DataRecovery #Lessons #Backend #Infrastructure