I deleted a queue and my own CLI told me a resource had been added

Last week I finally did something I'd been putting off for months: I pointed my AWS drift detector at a fake AWS.

Not a mock in a unit test — a real emulator listening on a real port, speaking the real API. I created a handful of resources, scanned them, then deleted an SQS queue and scanned again to watch the tool notice. Here is what it printed:

Demo System / Production: 3 drifted (changed=1 added=2)
    ~ EC2 i-9538ffc1...   instance_state: running -> stopped
    + S3  demo-assets-2
    + SQS demo-jobs          <-- the queue I had just deleted

The web UI, looking at the same database, said the queue was removed. The CLI said it was added. Same data, opposite verdicts, and only one of them was going to be in front of someone at 2am.

Why a deletion looked like a creation

My scanner does not delete rows for resources that vanish from AWS. It stamps missing_since and leaves the row alone, so you can still answer "what was there, and when did it go?" months later.

Drift classification then looked at two fields. raw_data is the latest scan, raw_data_prev is the one before it. The CLI's rule read:

if not asset.raw_data_prev:      # never had a previous value
    added.append(asset)          # ...so this is the first time we've seen it

That reads fine right up until you notice raw_data_prev is empty in two opposite situations:

• a resource that just appeared and has only been scanned once
• a resource that appeared and disappeared before a second scan could confirm it

The field is a proxy for "we have no history here." It cannot tell you why there's no history. The only field that knows a resource is gone is missing_since — and the CLI never looked at it.

The correct order isn't clever, it's just an order. Existence first, attributes second:

if asset.missing_since:          # did it disappear?
elif not asset.raw_data_prev:    # did it appear?
else:                            # did its attributes change?

The core already did exactly this. So did the drift report. So did the history snapshot writer. The CLI did not.

The part that actually stings

I wrote the fix in about four minutes, and then went looking for how it happened — and found four copies of the same decision tree:

Four callers, four hand-written copies of "what category is this asset in."

This is not the first time it bit me. When I added the removed category, some copies learned it and some didn't. A later fix updated two of them. The CLI was the fourth, and it stayed wrong for two releases because nothing forces copies to agree.

The genuinely embarrassing part: my previous post ended with advice about this. I'd put the Auto Scaling ownership check in one shared helper so all four paths would inherit it, and wrote a smug little paragraph about how a tool that disagrees with itself is worse than no tool. All true — and the classification around that helper was still copy-pasted four times. I'd deduplicated the thing I had just been burned by, and left the frame it sat in alone.

The fix: callers stop deciding

One function now owns the decision, and returns nothing but the decision:

# asset_manager/drift.py
def classify(asset) -> tuple[str, list]:
    """Existence first, attributes second."""
    if asset.missing_since:
        return (AUTOSCALING if is_autoscaling_churn(asset.raw_data) else REMOVED), []
    if not asset.raw_data_prev:
        return (AUTOSCALING if is_autoscaling_churn(asset.raw_data) else ADDED), []
    changes = _compute_raw_diff(asset.raw_data_prev, asset.raw_data)
    return (CHANGED, changes) if changes else (UNCHANGED, [])

Each caller keeps only what genuinely differs — counts for the badge, JSON for the CLI, template rows for the page. None of them mentions missing_since any more, which means none of them can get this wrong again.

Worth noting what did not get shared: the output shapes. A badge needs integers, a report needs ORM objects, a CLI needs strings. Trying to unify those too is how you end up with a "flexible" function taking four keyword arguments that each caller sets differently — which is four copies again, wearing a trench coat. Share the decision, not the presentation.

The test I should have written two releases ago

Per-rule tests are the easy half (a deletion is REMOVED, a scale-in is churn, an ASG instance's attribute change is still real drift). The one that earns its keep is the one that doesn't test a rule at all:

def test_badge_snapshot_and_cli_report_the_same_counts(self):
    badge    = _get_env_drift_summary(self.env)
    snapshot = _record_drift_snapshot(self.env, DriftSnapshot.Source.SCAN)
    cli      = drift_for(self.env)

    self.assertEqual(badge['removed'], snapshot.removed_count)
    self.assertEqual(len(cli['removed']), snapshot.removed_count)

It asserts no specific behaviour. It asserts that my four answers are the same answer. If someone (me, in four months, in a hurry) writes a fifth copy that drifts, that's a red build instead of a support email.

Why the emulator found this and my test suite didn't

My unit tests mock AWS at the library layer, and they're good tests — 200+ of them, all green while this bug shipped. They were green because I wrote them from the same mental model that produced the bug. I never wrote "create a queue, delete it, scan twice, assert it reads as removed," because it didn't occur to me that it wouldn't.

The emulator is what made that sequence cheap enough to do idly. No AWS account, no credentials, no bill — start a container, make things, break things, watch the tool. The bug surfaced in about ten minutes of poking, not from rigor but from playing with the actual product.

Honest notes from doing it:

• The free tier of the emulator implements a subset of AWS. 10 of my 18 scanners returned data; the rest reported an error per service. That was useful in itself: it exercised the path where a scanner fails, and confirmed a failed scanner is excluded from deleted-resource detection — an unreadable service must never read as "the customer deleted their fleet."
• CloudTrail attribution ("who changed this?") doesn't work there at all. Some things still need a real account.
• :latest is now a licensed build that exits immediately without an auth token. Pin the major version tag or your first impression is a dead container.

Takeaways

• A field that's empty for two opposite reasons is a bug waiting for a quiet afternoon. raw_data_prev meant both "brand new" and "already gone." If you're branching on absence, ask what else could make it absent.
• Judge existence before attributes. Did it appear, did it disappear, and only then, did it change. Get the order wrong and deletions come out as creations.
• Deduplicating one helper doesn't deduplicate the decision around it. I shared the ASG check and left four copies of the classification that used it.
• Test that your surfaces agree, not just that each one works. Every copy passed its own tests while disagreeing with the others.
• Run your own tool for fun, not just in CI. An emulator makes that free, and playing found in ten minutes what 200 green tests missed for two releases.

This is a real, open-source (MIT) self-hosted tool that tracks how your live AWS drifts from Terraform — one docker compose up, and now with a --profile demo that runs it against a local emulator so you can try it with no AWS account at all: syncvey.com.

Have you ever found two parts of your own system confidently disagreeing about the same data — and which one did you trust first?

docker pull jiniie/syncvey