Karpenter picked an instance family our Savings Plan never covered

Savings Plan coverage across the fleet went from 100% in June to about 41% two weeks into July, and not one thing on the platform team's dashboards moved. Node count flat. Pod count flat. p99 latency flat. Karpenter had quietly replaced most of a 4-cluster fleet with m7a nodes, a family none of the three EC2 Instance Savings Plans reaches, while the m5 EC2 Instance Savings Plan kept billing $18.40/hr for m5 hours nobody was using anymore. An EC2 Instance plan is scoped to a single family in a single region; the m5 plan buys m5. The answer was in the NodePool selector, not in Karpenter.

Problem signals:

• Savings Plan coverage percent drops sharply while total instance-hours stay inside normal variance
• A brand new instance-family line item appears in Cost Explorer that was at zero the prior month
• kubectl get nodes shows a family nobody added to any NodePool by name
• Cost Anomaly Detection has no notion of Savings Plan coverage; it models spend, so the stranded commitment is structurally invisible to it and the new on-demand family only alerts if it clears your subscription threshold
• Node count, pod count and latency are all flat across the window where the bill moved

What we checked first when Savings Plan coverage fell 59 points

Three explanations that all failed the arithmetic

The FinOps analyst at this healthtech SaaS (about 230 engineers, single us-east-1 footprint, four EKS clusters, EC2 running around $58k of a $92k monthly bill) pulls a coverage report on the first Monday of the month. June came back at 100%, consistent with the prior eight months. The partial-July pull came back at about 41%. Total instance-hours had moved from roughly 1,850 to 1,880 per day, which is inside the noise of ordinary customer onboarding. Only the coverage number had broken.

The platform lead's first answer was that nothing had changed, and it was an honest answer. Karpenter had gone from 1.0.6 to 1.1.4 in the June 30 maintenance window, two engineers had reviewed that PR, and the diff was a version string in a Terraform manifest. No NodePool CR had been edited. No large service had shipped. So we went looking for a billing-side explanation, and we burned most of a day on three of them.

The first was untagged usage falling out of the coverage denominator. That one dies on the API surface before you run a command: get-savings-plans-coverage exposes no tag dimension at all, so coverage data is never tag-scoped and tagging cannot move the number in either direction. Wrong tree. The second was a new linked account joining the Organization without Savings Plan sharing turned on, which genuinely does dilute aggregate coverage. aws organizations list-accounts returned the same eight accounts as June. Checking the sharing setting is not something you can do from a member account, which is worth saying because we tried it first: aws savingsplans describe-savings-plans returns only the plans the calling account owns, so a linked account never sees the payer's commitments through it, and the call says nothing about discount sharing either way. Sharing is a payer-level billing preference. We read it in the management account under Billing preferences, "Reserved Instance and Savings Plans discount sharing", and confirmed the discount was actually landing in the linked accounts from the payer, one call per account, filtering rather than grouping: aws ce get-savings-plans-coverage --time-period Start=2026-07-01,End=2026-07-22 --filter '{"Dimensions":{"Key":"LINKED_ACCOUNT","Values":["<acct-id>"]}}' --granularity MONTHLY. LINKED_ACCOUNT is a filter dimension on that API, not a GroupBy attribute; GroupBy accepts only INSTANCE_FAMILY, REGION or SERVICE, and --granularity is legal precisely because there is no GroupBy on the call. It was on. The third explanation was a service migrating from EC2-backed nodes onto Fargate, which an EC2 Instance-family plan does not reach. Both Fargate profiles were the same two that had existed for a year, and neither had picked up new work.

We also confirmed the boring one, that a plan had quietly retired. The account holds three EC2 Instance Savings Plans, one each for m5, m6i and r5, and all three had end dates in 2027 and 2028. Four explanations, four dead ends, and half a day gone against a billing cycle that closed on August 1.

The per-family coverage breakdown that ended the search in one query

22,250 hours on a family with zero coverage

The report the analyst pulls monthly is an aggregate. Aggregates hide exactly this class of problem, because a fleet swapping one family for another keeps every total roughly constant. Mid-afternoon we re-ran the coverage query grouped by instance family instead. That is not one extra argument on the same call, which cost us twenty minutes: get-savings-plans-coverage rejects --granularity when --group-by is set, so you issue one call per window and diff them yourself. It is also a spend API rather than a usage one, so the hours had to come from a second call to get-cost-and-usage.

# get-savings-plans-coverage rejects --granularity when --group-by is set,
# so run one call per window and diff them yourself.
aws ce get-savings-plans-coverage \
  --time-period Start=2026-06-01,End=2026-07-01 \
  --group-by Type=DIMENSION,Key=INSTANCE_FAMILY --output json

aws ce get-savings-plans-coverage \
  --time-period Start=2026-07-01,End=2026-07-22 \
  --group-by Type=DIMENSION,Key=INSTANCE_FAMILY --output json

# Coverage is spend-based. Each group returns only
# {CoveragePercentage, OnDemandCost, SpendCoveredBySavingsPlans, TotalCost}:
#   m5   "CoveragePercentage": "100" -> "100"
#   m7a  no usage in June            -> "0"   (~$10.3k OnDemandCost)
#   m6i  "CoveragePercentage": "100" -> "100"
#   r5   "CoveragePercentage": "100" -> "100"

# Hours are a different API, and get-cost-and-usage cannot group by
# INSTANCE_TYPE_FAMILY at all -- there it is a filter dimension only.
# Group by INSTANCE_TYPE and roll the sizes up into families yourself,
# or issue one call per family with
#   --filter '{"Dimensions":{"Key":"INSTANCE_TYPE_FAMILY","Values":["m7a"]}}'
aws ce get-cost-and-usage \
  --time-period Start=2026-06-01,End=2026-07-22 \
  --granularity MONTHLY --metrics UsageQuantity \
  --group-by Type=DIMENSION,Key=INSTANCE_TYPE --output json

# Sizes rolled up into families:
#          June, 30 days        Jul 1-22, 21 days
#   m5     47,000 (1,567/day)   11,280   (537/day)
#   m7a         0     (0/day)   22,250 (1,060/day)
#   m6i     2,300    (77/day)    1,610    (77/day)
#   r5      6,200   (207/day)    4,340   (207/day)
#   total  55,500 (1,850/day)   39,480 (1,880/day)

The aggregate said the fleet was fine. Grouped by family, 22,250 hours had walked onto a family with zero coverage while total hours per day did not move.

The July bucket is 21 days against June's 30, so read the per-day column and not the totals. Per day, m6i and r5 did not move at all, and the fleet total went from 1,850 to 1,880 hours a day, which is the variance the analyst had already dismissed. m5 had settled at roughly 29% of its June run rate; the 21-day average still reads higher than that because the swap itself sits inside the first day or two of the window. And a family that did not appear in June at all was carrying 22,250 hours at zero coverage. Confirming it in the cluster took one command, and the count told us how far it had gone.

kubectl get nodes \
  -o jsonpath='{range .items[*]}{.metadata.labels.node\.kubernetes\.io/instance-type}{"\n"}{end}' \
  | sort | uniq -c | sort -rn

     41 m7a.2xlarge
      7 m5.2xlarge

Nobody had typed m7a into a manifest anywhere in the repo. It was on 85% of the general-purpose pool.

The NodePool that governed that pool asked for instance-category In ["m", "r"], a set of allowed CPU counts, and an instance-family NotIn ["m5a", "m6a"] exclusion added in early 2024 after a numerical-precision problem with an AMD-based workload that had since been decommissioned. There was no generation constraint and no positive allow-list. The selector said, in effect, any m or r family AWS offers us, minus these two names. m7a sat inside that selector and no rule in the cluster objected, and it had been permitted long before the June 30 upgrade; nothing in that upgrade widened it. Karpenter's AWS provider enumerates the families it may pick from at runtime, from ec2:DescribeInstanceTypes and ec2:DescribeInstanceTypeOfferings against this account's own subnets and zones, then prices them from the AWS Pricing API. A family becomes selectable the day EC2 offers it in your zones: no controller release, no restart, no manifest change. The exclusion list was not out of date either, which is the uncomfortable part. m7a and m7i had both been generally available in us-east-1 since August 2023, five months before that NotIn was written. It was incomplete on the day it shipped, against families that already existed, because it named the two AMD generations the author had in mind and left everything else permitted.

Karpenter then did precisely its job, though not for the reason we assumed on day one. Our first theory was that it had found m7a cheaper. It had not, and the rate card says so plainly: in us-east-1, m7a.2xlarge lists at about $0.4637/hr against $0.384/hr for both m5.2xlarge and m6i.2xlarge, for the identical 8 vCPU / 32 GiB shape. Karpenter ranks candidate types on hourly price and, with consolidationPolicy: WhenEmptyOrUnderutilized, only replaces an underutilized node when the replacement is cheaper. A price-ranked replacement can never move a fleet from m5 onto on-demand m7a, and these hours billed at full on-demand list, so spot pricing was not in play either. What moved the fleet was capacity, and what asked for that much capacity at once was drift. The controller version bump was coincident rather than causal: bumping the chart restarts the controller pods and leaves existing NodeClaims alone, and Karpenter guards against mass drift across releases with the karpenter.sh/nodepool-hash-version annotation precisely so an upgrade does not roll the fleet. What did roll it went into the same maintenance window: the EC2NodeClass used an amiSelectorTerms alias rather than a pinned AMI ID, the alias resolved to a newly released AMI, and every node whose resolved AMI no longer matched was marked drifted and replaced. That needs no NodePool CR edit, which is why the version-string diff two engineers reviewed showed nothing. Karpenter hands EC2 Fleet a list of acceptable types in price order and Fleet launches what is actually available in the zone. m7a was on that list the whole time, ranked behind m5 and m6i on price; drift churned nodes fleet-wide, every replacement re-ran instance selection, and with that many launches compressed into one window the m5 and m6i capacity the fleet asked for was not there, so the requests fell through to the family that was. The hour table says how fast it finished. Steady-state m5 plus m7a is 1,596 hr/day once you subtract the flat 284 for m6i and r5, so with m5 settled at 29% of its June rate (454 hr/day), steady-state m7a is about 1,142 hr/day. The measured m7a average is 1,060 hr/day, 93% of steady state across the entire 21-day window, which only happens if the swap completed inside the first day or two of July. A migration trickling across three weeks would have landed near 12,000 hours, not 22,250. The m5 side agrees: 11,280 hours is only about a day and a half of full-rate m5 above the settled rate. What the provisioner cannot see either way is the Savings Plan portfolio, because commitments live in the billing system and are not part of the pricing signal it reads. Cheaper on the hourly rate card and cheaper after commitment amortization are two different questions, and Karpenter only answers the first.

Both halves of the loss run through a fleet that looks completely healthy from the cluster side.

That double charge is the part people miss. The commitment does not stop when you stop using it. $18.40/hr kept flowing whether or not m5 nodes existed, and with m5 usage down to 29%, roughly $310 a day of it bought nothing. Meanwhile the m7a hours billed at full on-demand list. Neither half appears as a line item labeled waste. The first shows up only as a coverage percent under 100, and the second shows up as a new usage type that nobody was watching for.

How we chose the fix with 9 days left on the billing cycle

Constrain the pool, buy a second commitment, or take the haircut

By late afternoon on day one we had three real options on the table and a cycle closing August 1. The analyst was on PTO from the 25th and the platform lead lost the 24th to a customer-facing latency incident, so the working window was about five days, not nine.

The change itself was small, and it was deliberately not another round of exclusions. A NotIn list plus a generation ceiling still permits every m and r family at generation 6 or below that nobody thought to name: m4, m5n, m5d, m5dn, m5zn, m6id, m6in, m6idn, r4, r5a, r5b, r5d, r5n, r6a, r6i, r6id, r6in. Not one of those is covered by any of the three plans, and the next thin-capacity hour lands the fleet on one of them exactly as it landed on m7a. m6a is itself a generation-6 family, which is why it still has to be named by hand, and that is the proof the generation bound cannot carry this on its own. So we replaced the deny-list with a positive allow-list naming exactly the three covered families, and kept the generation bound behind it as a second guard.

requirements:
  - key: karpenter.k8s.aws/instance-category
    operator: In
    values: ["m", "r"]
  - key: karpenter.k8s.aws/instance-family      # the load-bearing line
    operator: In
    values: ["m5", "m6i", "r5"]
  - key: karpenter.k8s.aws/instance-generation  # second guard, not the fix
    operator: Lt
    values: ["7"]

A NotIn list is a list of yesterday's problems, and a generation bound still admits every uncovered family under the ceiling. The In list is the one that holds against families nobody has named yet.

We planned it, had a second engineer read the plan, and applied to the largest cluster at 15:40 on day one. Karpenter picked up the new constraints on its next reconcile, well under a minute. Then the second-order problem showed up: unwinding this is itself a fleet-wide node replacement, and it moves no faster than the disruption budget and the drain times allow. The pool had disruption.budgets set to nodes: "10%", which is a setting we would defend on any production cluster, but read it correctly before you plan a window around it. That field is a concurrency cap, not a rate: at most 10% of the pool may be undergoing voluntary disruption at any one time, roughly four or five nodes here. There is no hourly-rate form of it; if you want a genuine time-based limit, that is a budgets entry with schedule and duration. Elapsed time is therefore set by how long each node takes to drain, PDB-gated pod eviction plus the replacement node registering, not by a per-hour quota. Draining 41 m7a nodes back onto m5 and m6i took a little over two hours. PDBs held, nothing paged, and every one of those hours billed m7a at on-demand while the m5 commitment was still under-consumed. We could have raised the budget to move faster. We did not, because trading a stable rollback for a couple of hundred dollars is a bad trade in a system where the failure mode we are recovering from is invisible to health checks.

The other thing that did not snap back was the number we were watching. Day two we finished the remaining three clusters by end of business. Day three, the trailing-window coverage query read 91.4%, not 100%, and that was correct rather than alarming. The coverage report looks back over a rolling window that still contained the uncovered m7a hours from earlier in July. Anyone treating a 24-hour coverage spot check as a pass/fail gate will either panic or declare victory early. Read the trend across several days, not the number on the day you shipped the fix.

FinOps closed the cycle at about $22,500 of overage across the 21 days before detection, plus roughly $2,400 during the mitigation week. Just under $25,000 on a month where the compute bill should have looked exactly like June's.

The 4 controls we shipped so the next new family cannot do this

Deny-lists inherit whatever the vendor ships next

The durable lesson is not about Karpenter and it is not about AMD. It is that a selector written as an exclusion list silently accepts every family you did not think to name, including the ones that already existed on the day you wrote it. Our NodePools were written as deny-lists, and a deny-list is a bet that you enumerated a set you do not control correctly, and that you will keep re-enumerating it every time AWS extends it. That bet loses on a long enough timeline, and the loss shows up in the billing system weeks later rather than in a failing test.

Four things landed in the platform IaC repo and the on-call docs over the two weeks after. The first is a rule in CI requiring instance-family to use operator: In with an enumerated list instead of NotIn. That flips the failure mode from silently adopting new families to silently ignoring them until a human adds one, which is the direction you want a cost-bearing default to fail in. We shipped it soft-failing for two weeks so teams could migrate, then made it blocking. It touched 11 NodePools across the four clusters. The second is a companion OPA policy, about 30 lines of rego, that fails the plan on any NodePool without an explicit karpenter.k8s.aws/instance-generation constraint. It is a backstop rather than the control that closes this hole, since a generation ceiling on its own still admits uncovered families below it, but the discovery set is now stated by us rather than inherited.

The third is the detection gap, and it is the one we would fix first if we could only do one. AWS Cost Anomaly Detection was watching the raw EC2 compute line at a $500/day threshold and never fired. The explanation we reached for first, that the m5 commitment kept billing and offset the new on-demand spend, is backwards and worth correcting out loud: a Savings Plan charges its fixed hourly amount whether or not you consume it, so a constant charge cannot offset anything. The raw line did move. Of the roughly $22,500 of overage, about $6,500 was commitment buying nothing at $310 a day, and the other $16,000 or so was genuinely new on-demand spend, about $760 a day, comfortably above the threshold, and it arrived as a step change in the first days of July rather than as a slow drift. That monitor should have caught it and did not, and we never got an account of its baselining out of it that we were willing to trust. What we could establish is what it does not evaluate at any threshold: coverage percent. So we stopped trying to tune a spend monitor into a coverage monitor.

So we wrote a nightly Lambda, about 80 lines of Python on an EventBridge schedule, that pulls the trailing seven days of coverage grouped by instance family and posts to the FinOps Slack channel on three conditions: a family carrying material uncovered spend that had none the week before, aggregate coverage moving more than 5 percentage points week over week, and Savings Plan utilization falling below 100%. Per-family coverage percentage is deliberately not one of them, and we learned that by writing the naive version first. Through this entire incident m5, m6i and r5 all read 100% and m7a read 0% from its first hour to its last, so a week-over-week delta on per-family coverage percent would never have crossed any threshold. What moved was which families carried the volume, and the aggregate, which went 100 to 41. Against this incident the alert as shipped would have fired within a day or two of the migration instead of 22, because m7a went from no usage at all to a full day of it. The fourth is a one-page runbook for the phrase "SP coverage dropped": run the coverage query grouped by family once per window and diff the two, pull the matching hours from get-cost-and-usage grouped by INSTANCE_TYPE and rolled up into families yourself (that API takes INSTANCE_TYPE_FAMILY only as a filter, never as a GroupBy), and if any family went from near zero to non-zero, go read the NodePool selector before you go read anything else.

We now treat any Kubernetes autoscaler as a component with a billing blast radius, not just a scheduling one, and we review its selectors during cost reviews rather than only during platform reviews. That framing is the same one we bring to Kubernetes and CI/CD stabilization work generally: the controller doing exactly what it was told is the most expensive kind of correct. If the shape of this feels familiar from the invoice side rather than the cluster side, the pattern is covered from that angle in cloud cost spikes.

Karpenter and Savings Plans: 4 questions worth answering before your next NodePool review

The questions the retro kept circling back to

These came up in the retro and have come up in every conversation we have had about this since.

• Does an EC2 Instance-family Savings Plan cover a different family in the same category? No. m5 and m7a are distinct families under the plan's scope rules, not size or generation variants of one another. An EC2 Instance Savings Plan commits to one instance family in one region, flexes only across size, OS and tenancy inside it, and buys a deeper discount in exchange for that narrowness. A Compute Savings Plan is the one that flexes across families, at a lower rate.
• Would Cost Anomaly Detection have caught this? Ours did not, at a $500/day threshold, and not for the reason we first told ourselves. The commitment did not mask the delta: a Savings Plan bills the same fixed amount whether you consume it or not, so it cannot offset anything. The raw EC2 line really did rise, by about $760/day of new on-demand spend, and it rose as a step change in the first days of July. It should have fired and it did not. Check what your monitor actually evaluates before you rely on it here; Cost Anomaly Detection models spend and has no notion of Savings Plan coverage, so a coverage collapse is structurally invisible to it, and the control that covers this failure is the per-family coverage alert.
• Is it safe to change a NodePool selector on a busy production cluster? It was for us, and the reason is disruption.budgets plus PDBs. Read the budget correctly first: nodes: "10%" caps how many nodes may be under voluntary disruption at once, not how many per hour. Karpenter replaced 41 nodes with four or five draining concurrently, a little over two hours end to end, with no service impact. Budget the elapsed time from your own drain and registration times, and resist raising the cap to finish faster; that is exactly where a cost incident turns into an availability incident.
• Should we just move to Compute Savings Plans? It removes this failure mode structurally, and it costs roughly 7 points of discount. Note that it is not a conversion: Savings Plans cannot be exchanged, modified or cancelled after the 7-day return window, so you either layer a Compute plan on top of current spend or replace the EC2 Instance plans as they expire. Nobody on the team was happy about the 7 points and nobody had a better answer. If your fleet is under an autoscaler that is free to pick families, price the immunity honestly rather than assuming the deeper discount is the cheaper option.

When the cluster looks healthy and the invoice does not agree

If your coverage number moved and your dashboards did not

The hard part of this class of incident is not the fix, which was a handful of lines in a NodePool. It is that every signal a platform team looks at daily stays green while the money leaves, and the one signal that would have caught it lives in a monthly FinOps artifact that the platform team does not own or read. Two functions each holding half the picture is how a 22-day detection lag happens to competent people.

We do this work on live clusters: reading autoscaler selectors against a commitment portfolio, finding the gap between what the scheduler optimizes and what the invoice charges, and leaving behind the CI policy and the alert so it does not come back the next time a new family lands in your zones.

If a coverage number just moved on you and the cluster looks fine, book an infrastructure review and we will get on a call the same day to find where your fleet went.

Originally published at https://infraforge.agency/insights/karpenter-savings-plan-coverage-drop/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.