Our ECS Fargate Task Was Silently Failing for Days — Here's Exactly How We Found It
No alarm fired. No Slack alert. No PagerDuty page. Just a service quietly burning compute while cycling through thousands of failed tasks. Here's the full story.
It started with a routine check.
I opened the ECS console to verify a deployment and noticed something wrong in the numbers. The running task count was 0. The desired count was 1. And the service had been in that state for longer than I wanted to admit.
No notification. No alert. No one on the team had noticed.
The task was starting, failing its health check, getting killed by ECS, and restarting — on a 60-second loop. Silently. Automatically. Expensively.
This is the story of how we found it, what caused it, and the three things we now have in place so it never goes undetected again.
What Was Happening
ECS Fargate has a deceptively simple failure loop:
Task starts
↓
Health check runs
↓
Health check fails
↓
ECS kills the task
↓
ECS starts a new task (desired count not met)
↓
Repeat indefinitely
From the outside, your service looks like it's "working" — ECS is doing exactly what it's designed to do, trying to maintain your desired count. From the inside, your application is never actually serving traffic. And unless you're watching the right metrics, you won't know.
The console shows this:
Service: my-backend
Desired: 1 | Running: 0 | Pending: 1
Recent events:
service my-backend has started 1 tasks
service my-backend: task failed health check, will be stopped
service my-backend has started 1 tasks
service my-backend: task failed health check, will be stopped
...
Every 60 seconds. Over and over.
The Root Cause
Our health check was configured to hit /api/health.
The container was serving the application — but on a different path after a recent refactor. The endpoint had moved. The health check hadn't been updated.
Health check expected: GET /api/health → 200 OK
Container was serving: GET /health → 200 OK
GET /api/health → 404
One forgotten update. That's it.
ECS saw a 404 on every health check. Interpreted it as unhealthy. Killed the task. Started a new one. Repeat.
The application code was perfect. The deployment was clean. The task definition was correct. The only problem was a single path string in the health check configuration.
How We Found It
Three signals that told us something was wrong — none of them were the alarm we should have had:
Signal 1 — ECS console service events
The events tab in the ECS service console shows every task start and stop. When a service is stuck in a failure loop, this list fills up fast. A healthy service might have 5-10 events. Ours had hundreds.
Signal 2 — CloudWatch metric: RunningTaskCount
This metric drops to 0 when all tasks fail. If you graph it, you see a flat line at 0 instead of the expected flat line at 1. We weren't graphing it. We should have been.
Signal 3 — CloudWatch logs — or the absence of them
Application logs stop when the task dies. If your log stream has a gap or just stops entirely, the task isn't running. Check the timestamp on the last log line.
None of these required any special setup. They were all there, visible, waiting. We just weren't watching.
The Fix — 3 Layers We Now Have in Place
Layer 1 — Health Check Path Validation
Before every deployment we now verify health check path matches the actual endpoint:
Application route: /health ✅
Health check path: /health ✅ match — good to deploy
Application route: /health ✅
Health check path: /api/health ❌ mismatch — fix before deploy
Simple manual check. Takes 30 seconds. Would have prevented the entire incident.
Layer 2 — CloudWatch Alarm on RunningTaskCount
The metric RunningTaskCount dropping below your desired count for more than 5 minutes means something is wrong. This alarm now fires to Slack the moment it happens:
Metric: ECS/RunningTaskCount
Condition: < 1 for 5 consecutive minutes
Action: SNS → Slack notification
We missed days of failure because this didn't exist. It takes about 10 minutes to set up.
Layer 3 — ECS Service Events in CloudWatch
ECS service events can be streamed to CloudWatch Logs via EventBridge. Once there, you can search them, alert on patterns, and see the full history without opening the console.
The event pattern to watch for:
{
"source": ["aws.ecs"],
"detail-type": ["ECS Service Action"],
"detail": {
"eventType": ["WARN", "ERROR"]
}
}
Any WARN or ERROR from ECS service events goes straight to a log group. From there a metric filter and alarm handles the rest.
What the Fixed Architecture Looks Like
ECS Task starts
↓
Health check: GET /health → 200 OK ✅
↓
Task marked healthy
↓
Traffic routed to task
+-- CloudWatch Alarm watching RunningTaskCount
| fires if task count < desired for 5 mins
|
+-- EventBridge capturing ECS WARN/ERROR events
| streams to CloudWatch Logs
|
+-- Slack notification on any alarm state
Three independent detection layers. Any one of them would have caught the original failure within minutes.
The Cost of Not Watching
Every failed task in ECS Fargate still consumes compute during its startup window. If your task takes 60 seconds to start before failing the health check, and it restarts every 60 seconds:
Fargate 0.25 vCPU + 0.5GB = ~$0.015/hour
Constant restart loop = full billing, zero useful work
Over 3 days undetected: 3 × 24 × $0.015 = $1.08 wasted
Not a huge number — but that's a small task. Scale this to larger Fargate tasks or multiple services and it adds up fast. More importantly, your application was completely down the entire time.
Lessons Learned
1. ECS will not tell you when it's failing — by design.
ECS is doing its job: trying to maintain desired count. It has no concept of "this has been broken for too long." That judgement is yours to implement.
2. Health check path is a contract.
Change your application routes, update your health check. Always. Make it part of your PR checklist.
3. RunningTaskCount alarm is non-optional.
If you run ECS Fargate and don't have this alarm, you are flying blind. It takes 10 minutes to set up and covers an entire class of silent failures.
4. Log gaps are signals.
If your application log stream suddenly stops, the task stopped. The absence of logs is as important as the presence of errors.
5. Silent failures are more dangerous than loud ones.
A crashing application that sends an alert is manageable. An application that silently fails for days while appearing to run is much harder to deal with — because by the time you find it, the damage is done.
Quick Checklist Before Your Next ECS Deployment
Before deploying:
☐ Health check path matches current application route
☐ Health check interval and threshold are reasonable
☐ Container port matches task definition port mapping
After deploying:
☐ RunningTaskCount reaches desired count within 5 minutes
☐ Application logs are appearing in CloudWatch
☐ Health check is returning 200 in the target group
Monitoring (set up once, never remove):
☐ CloudWatch alarm: RunningTaskCount < desired for 5 mins
☐ EventBridge rule: ECS service WARN/ERROR events to CloudWatch
☐ Slack/SNS notification on alarm state change
Final Thought
The most expensive outages are the ones nobody notices.
A loud failure — exception thrown, service returns 500, alarm fires — gets fixed fast. A silent failure — task cycling, health check failing, no traffic getting through — can run for days before anyone looks at the right dashboard.
ECS is excellent infrastructure. But it will not babysit your application for you. The monitoring layer is yours to own.
Set up the RunningTaskCount alarm today. It takes 10 minutes. Future you will be grateful.
Running ECS Fargate? What monitoring do you have in place for silent failures? Drop it in the comments — curious what patterns people are using.
Tags: #AWS #ECS #Fargate #DevOps #CloudWatch #Monitoring #Backend #SRE
