I had a fleet of stateful worker instances behind an Auto Scaling Group On-Demand
floor for guaranteed baseline capacity, Spot instances above that for burst. Standard
setup. Then we added a second scaling signal to fix a real gap in the first one, and
the fleet started flipping size every 30–90 seconds. This is the story of why that
happened, and why it wasn't a tuning problem.
The setup
Each instance in the fleet holds a fixed pool of "slots" a bounded number of
concurrent stateful sessions it can serve. Autoscaling started with the obvious
signal: CPU. A TargetTrackingScaling policy on average CPU across the ASG, scaling
out when the fleet got busy.
It worked, until it didn't. CPU is a fleet-wide average. An individual instance
could be completely full zero free slots, rejecting new sessions while the
fleet's average CPU still looked comfortable, because three other instances were
idle. CPU told you the fleet was fine. The thing that actually mattered was any
single instance out of room was invisible to it.
So we added a second signal: a CloudWatch Alarm watching per-instance free-slot
count, tied to a Step Scaling policy, firing when any instance hit zero free slots
for a sustained window. This is a completely reasonable instinct CPU is a lagging,
indirect proxy for capacity; free-slot count is the real thing.
What went wrong
Now there were two independent scaling authorities, reacting to two different
metrics, with no way to reconcile a disagreement. CPU's policy would look at the
fleet-wide average, see it drop after a burst, and scale in. The pool-alarm's policy
would look at the resulting per-instance squeeze, see zero free slots, and scale
right back out. Neither policy knew the other existed. Neither was wrong on its own
terms. Together, they fought.
Real observed behavior, pulled from describe-scaling-activities: the fleet size
flipping between 5 and 6 instances every 30 to 90 seconds, repeatedly, for minutes
at a stretch.
First attempt: tune the timing. Lengthened the pool alarm's sustained-breach
window from 3 minutes to 10, on the theory the two policies just needed more
separation. Deployed. The real activity log afterward showed it was still
oscillating just on a 5–6 minute cadence instead of 30–90 seconds. The timing
change reduced frequency. It didn't touch the actual disagreement. Two authorities
still existed, still had different opinions, still had no way to agree.
The actual fix: one authority, not two
The right move wasn't tuning either policy. It was removing one of them.
The metric that actually reflects real capacity is occupancy slots in use divided
by slots available, aggregated properly across the fleet, not two separate numbers
each policy interprets differently. CloudWatch Metric Math makes this
computable: occupied_per_instance = pool_max_size - avg(free_slots), targeting a
real utilization band (in our case, 75%).
Two things worth knowing if you're building this yourself, because both cost real
debugging time:
SEARCH() is not supported inside CloudWatch Metric Alarms. Only inside
Dashboards and direct GetMetricData calls. First attempt at a true fleet-wide
aggregate (summing free slots across every instance, dynamically, without hardcoding
instance IDs) used SEARCH inside a plain Alarm's metric query. PutMetricAlarm
rejected it outright not a syntax error, a hard platform limitation. Confirmed
against AWS's own documentation, not assumed.
Target Tracking policies support Metric Math as a separate, genuinely different
feature from Alarms live since December 2022, and not subject to the same
SEARCH restriction in the same way, because Target Tracking evaluates the metric
expression directly rather than routing through the Alarms evaluation engine. This
is the detail that made the fix possible: replace both old policies the CPU
target-tracking policy and the pool-based step-scaling alarm with a single
TargetTrackingScaling policy computing occupancy via Metric Math.
One authority. One metric that actually reflects real capacity. Nothing left to
disagree with.
Verified before applying terraform plan showed exactly the intended diff, one
resource added, three removed, nothing else touched. Applied cleanly on the first
real attempt. Confirmed afterward via describe-policies: exactly one scaling
policy exists. No oscillation since.
The generalizable part
This wasn't really an AWS-specific bug. It's a distributed-systems pattern that
shows up anywhere two control loops react to different observations of the same
underlying system state, without a shared source of truth:
- Each loop is locally correct it's doing exactly what its own metric tells it to.
- Neither loop is aware the other exists, let alone that it just undid the other's decision.
- The oscillation isn't a bug in either policy. It's an emergent property of running two of them.
The fix generalizes too: don't tune the disagreement, find the metric that actually
represents the thing you care about, and have exactly one thing making the decision
based on it. If you're tempted to add a second scaling signal because the first one
has a real, valid blind spot that blind spot is real, but the fix is usually a
better single metric, not a second policy running in parallel.
AWS building blocks used: EC2 Auto Scaling Group (On-Demand + Spot mix), Target
Tracking Scaling Policies, CloudWatch Metric Math, CloudWatch Alarms, Step Scaling
Policies, SNS (for the alert path that first surfaced the oscillation as noisy
paging).
