Eight of my eighteen AWS scanners failed against a fake AWS — and that was the point

You cannot try a self-hosted AWS tool without an AWS account. That is a real wall: the thing needs credentials, a role to assume, and something in the account worth scanning before it shows you anything at all.

So I pointed it at a fake AWS — LocalStack, in a Compose profile, with a seed script that fills it with resources. Then I ran the scanner.

Ten of the eighteen scanners worked. Eight failed. The free edition of the
emulator does not implement those services, so they returned errors.

That was the interesting part.

A scan that stops at the first error cannot tell you what was deleted

Drift detection is not only "this attribute changed". The category that actually bites is the resource that is no longer there — someone deleted it in the console, and terraform plan says nothing because it was never in the state.

To report a deletion, you have to finish the sweep. A resource is gone only if you looked everywhere it could have been and did not find it. Which means: a scanner that aborts on the first failing service is structurally incapable of detecting deletions. It cannot tell "not there" from "I stopped early".

I had written it to carry on and report an error per service, because that is what deletion detection needs. Running against an emulator that fails eight of eighteen services was the first time that design was tested for real — the sweep completed, the eight reported their own failures, and the deletion detection on the other ten still worked.

The emulator did not just make the tool easier to try. It exercised the failure mode the tool depends on being right about.

Try it

git clone https://github.com/MR-TABATA/SyncVey
cd SyncVey
docker compose --profile demo up -d
python scripts/seed_localstack.py

No account, no credentials, no bill. The README says which scanners work on the emulator's free edition and which do not, CloudTrail attribution included.

Starting it is also faster than it used to be: docker compose up -d now pulls a published image instead of building one on your laptop (0.4.0). Building from source is still one command away — cp docker-compose.override.yml.example docker-compose.override.yml — and gives you live reload.