Your Kubernetes Deployments Are Killing Sidekiq Jobs

How a preStop hook, one Unix signal, and the right grace period stopped Kubernetes from killing our long-running Sidekiq jobs in the middle of their work.

Every deployment looked clean. Green pipeline, new pods up, old pods gone. And every deploy, a handful of background jobs quietly died halfway through.

We ran a big part of our product on Sidekiq workers in Kubernetes. Short jobs were fine. The long ones, the jobs that needed a few minutes to finish, were the problem. If a deploy landed while one of them was running, it got cut off. Sometimes Sidekiq retried it, and it finished later. Sometimes it left work half done, and someone had to clean up by hand.

Nothing was "broken." Kubernetes and Sidekiq were both doing exactly what they were told. They just weren't told the same thing.

This article walks through why it happens, the fix we put in place, and every piece of config you need to do the same: the preStop script, the Deployment, the Sidekiq config, the Dockerfile, and how to test it before you trust it.

The symptom

If you're running Sidekiq on Kubernetes with default settings, you've probably seen some of these without connecting them to deploys:

• Jobs show up in the Retry set right after a release, with errors that don't make sense for the job.
• Long jobs that "ran twice": emails sent twice, records processed twice, partial exports.
• Jobs that were running before the deploy and then simply vanished. No retry, no error, no trace.
• Sidekiq logs on the old pod that end abruptly, without the usual Bye! shutdown line.

The pattern is the giveaway: it only happens around deploys, scale-downs, and node drains. In other words, it happens whenever Kubernetes stops a pod.

What actually happens when Kubernetes stops a pod

When a rolling update replaces a pod, Kubernetes runs this sequence:

1. Marks the pod as Terminating and removes it from Service endpoints.
2. Runs the container's preStop hook, if one is defined, and waits for it to finish.
3. Sends SIGTERM to PID 1 in the container.
4. Waits until terminationGracePeriodSeconds runs out.
5. Sends SIGKILL. Whatever is still running dies on the spot.

Two details matter a lot here:

• The default terminationGracePeriodSeconds is 30 seconds.
• The grace period clock starts at step 1, not after the preStop hook. The hook's time is spent out of the same budget.

Here's the timeline with defaults:

t=0s     Pod marked Terminating
t=0s     preStop hook (none by default)
t=0s     SIGTERM -> Sidekiq starts shutting down
t=25s    Sidekiq shutdown timeout expires, unfinished jobs pushed back to Redis
t=30s    SIGKILL if the process is still alive

What Sidekiq does with signals

Sidekiq listens for a few Unix signals. Two of them matter for deploys:

• TSTP (quiet): stop fetching new jobs. Keep working on the jobs already running.
• TERM (shutdown): stop fetching, wait up to timeout seconds for running jobs, push unfinished jobs back to Redis, and exit.

Older Sidekiq versions used USR1 for quiet. Since Sidekiq 5, it's TSTP. The default shutdown timeout is 25 seconds.

So, out of the box, Kubernetes sends SIGTERM, and Sidekiq gives running jobs 25 seconds, pushes the rest back to the queue, and exits. Kubernetes allows 30 seconds total before SIGKILL.

We were running both defaults. For our long jobs, neither number was anywhere close.

Here's what happens to a job that's still running at the deadline:

• Best case: Sidekiq pushes it back to Redis. Another worker picks it up and starts again from zero. If the job isn't idempotent, it does half its work twice.
• Worst case: the process gets SIGKILL before it can push the job back. With open-source Sidekiq, which uses basic fetch, that job is simply gone. Sidekiq Pro's reliable fetch can recover it, but most teams run the open-source version.

The fix: quiet first, then wait, then stop

The idea we landed on is simple. Before Kubernetes sends SIGTERM, tell Sidekiq to go quiet so it stops taking new work. Then give the jobs it already has enough time to finish. Only then let the normal shutdown happen.

The new timeline looks like this:

t=0s      Pod marked Terminating
t=0s      preStop: kill -TSTP 1      -> Sidekiq stops fetching new jobs
t=0s..N   preStop waits while running jobs finish (new pod already takes new work)
t=N       preStop exits
t=N       SIGTERM -> Sidekiq has nothing left to do, exits in seconds
t<600s    Pod gone. No SIGKILL, no lost jobs.

It takes four pieces to make that work:

1. A preStop script that sends TSTP and waits for running jobs to finish.
2. A terminationGracePeriodSeconds longer than your longest job.
3. A Sidekiq config with a sensible shutdown timeout.
4. A container where Sidekiq is actually PID 1, so the signal reaches it.

Let's go through each one.

Piece 1: the preStop drain script

This script goes into your app image, for example at bin/sidekiq-prestop.sh:

#!/bin/sh
# bin/sidekiq-prestop.sh
#
# Called by the Kubernetes preStop hook.
# 1. Puts Sidekiq into quiet mode (stop fetching new jobs).
# 2. Waits until this pod has no busy jobs, or until MAX_WAIT runs out.
#
# MAX_WAIT must be lower than terminationGracePeriodSeconds minus the
# Sidekiq shutdown timeout, or Kubernetes will SIGKILL us mid-drain.

set -u

MAX_WAIT="${SIDEKIQ_DRAIN_SECONDS:-540}"
INTERVAL=5

log() {
  # Write to PID 1's stdout so the message shows up in `kubectl logs`.
  echo "[prestop] $(date -u +%H:%M:%S) $*" > /proc/1/fd/1
}

log "sending TSTP to Sidekiq (quiet mode)"
kill -TSTP 1

waited=0
while [ "$waited" -lt "$MAX_WAIT" ]; do
  # Prints "<quiet> <busy>", for example "true 1".
  # Prints nothing if the check fails (Redis down, process not found).
  state=$(bundle exec ruby -e '
    require "sidekiq/api"
    me = Sidekiq::ProcessSet.new.find { |p| p["hostname"] == ENV["HOSTNAME"] }
    puts "#{me["quiet"]} #{me["busy"].to_i}" if me
  ' 2>/dev/null)

  quiet=$(echo "$state" | cut -d" " -f1)
  busy=$(echo "$state" | cut -d" " -f2)

  if [ "$quiet" != "true" ]; then
    # No heartbeat since TSTP yet, or the check failed. Don't trust busy.
    log "waiting for a fresh heartbeat, waited ${waited}s of ${MAX_WAIT}s"
  elif [ "$busy" = "0" ]; then
    log "no busy jobs after ${waited}s, letting shutdown continue"
    exit 0
  else
    log "still ${busy} busy job(s), waited ${waited}s of ${MAX_WAIT}s"
  fi

  sleep "$INTERVAL"
  waited=$((waited + INTERVAL))
done

log "drain timeout reached with jobs still running, continuing shutdown"
exit 0

A few notes on this script:

• kill -TSTP 1 only works because Sidekiq is PID 1 in the container. More on that in piece 4.
• The busy check uses Sidekiq's own API. Each Sidekiq process registers in Redis with its hostname, and in Kubernetes the hostname is the pod name, so ENV["HOSTNAME"] finds this exact pod.
• It waits for a fresh heartbeat. The busy count comes from Sidekiq's heartbeat in Redis, which is a few seconds old. Right after TSTP, it can still say 0 while a job that was just fetched is running. So the script only trusts busy once the process reports quiet=true, which means a heartbeat ran after TSTP. If the check fails, it keeps waiting instead of assuming the pod is idle.
• It exits as soon as the pod is idle. A quiet pod with nothing running shuts down in about five seconds. A pod in the middle of a long job waits for it. You don't pay the full wait on every pod.
• It always exits 0. If the hook fails, Kubernetes records a FailedPreStopHook event and moves straight on to SIGTERM. Exiting cleanly keeps the shutdown predictable.
• Logs go to /proc/1/fd/1. The preStop output doesn't show up in kubectl logs by default. Writing to PID 1's stdout puts the drain messages right next to Sidekiq's own logs, which makes debugging much easier.

Make it executable in your image:

COPY bin/sidekiq-prestop.sh /app/bin/sidekiq-prestop.sh
RUN chmod +x /app/bin/sidekiq-prestop.sh

The one-line version

If you don't want the Ruby check, there's a simpler version:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "kill -TSTP 1 && sleep 540"]

It works, but it's blunt. Every pod waits the full 540 seconds, even an idle one, so rollouts and scale-downs get slow. It's fine to start with. The script version is better once you run more than a couple of replicas.

Piece 2: the grace period

spec:
  terminationGracePeriodSeconds: 600

The rule is:

terminationGracePeriodSeconds > preStop max wait + Sidekiq shutdown timeout + buffer

With the numbers in this article:

600 > 540 (SIDEKIQ_DRAIN_SECONDS) + 25 (Sidekiq :timeout:) + 35 (buffer)

Pick the drain time from your real job durations, not from a guess. If your longest job can run for 9 minutes, a 540-second drain covers it. If you have a job that runs for an hour, a preStop hook is the wrong tool. Split that job into smaller chunks instead.

Piece 3: the Sidekiq config

# config/sidekiq.yml
:concurrency: 10
:timeout: 25
:queues:
  - [critical, 3]
  - [default, 2]
  - [low, 1]

By the time SIGTERM arrives, the preStop hook has already drained the pod. The 25-second :timeout: only has to cover whatever is left, which is usually nothing. Keep it well under the grace period minus the drain time.

Piece 4: make sure Sidekiq is PID 1

This is the gotcha that quietly breaks the whole thing. kill -TSTP 1 sends the signal to whatever process is PID 1. If that's a shell, Sidekiq never hears it.

This is wrong:

# Shell form: /bin/sh is PID 1, Sidekiq is a child process
CMD bundle exec sidekiq -C config/sidekiq.yml

This is right:

# Exec form: Sidekiq is PID 1 and receives signals directly
CMD ["bundle", "exec", "sidekiq", "-C", "config/sidekiq.yml"]

If you need an entrypoint script for setup work, end it with exec so Sidekiq replaces the shell:

#!/bin/sh
# docker-entrypoint.sh
set -e

# ...any setup you need (wait for Redis, load secrets, etc.)

exec "$@"

You can check what's running as PID 1 inside a pod:

kubectl exec -it deploy/sidekiq-worker -- cat /proc/1/cmdline | tr '\0' ' '

If the output starts with sidekiq or bundle exec sidekiq, you're good. If it starts with /bin/sh, fix that first.

The full Deployment

Here's everything put together:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sidekiq-worker
  labels:
    app: sidekiq-worker
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: sidekiq-worker
  template:
    metadata:
      labels:
        app: sidekiq-worker
    spec:
      terminationGracePeriodSeconds: 600
      containers:
        - name: sidekiq
          image: registry.example.com/app:1.42.0
          command: ["bundle", "exec", "sidekiq", "-C", "config/sidekiq.yml"]
          env:
            - name: SIDEKIQ_DRAIN_SECONDS
              value: "540"
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: redis-url
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "/app/bin/sidekiq-prestop.sh"]
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              memory: "1Gi"

Two things are worth calling out:

• maxUnavailable: 0 with maxSurge: 1 means Kubernetes starts a new worker before an old one begins draining. The queue keeps moving during the deploy, and you never drop below your normal worker count.
• Watch your liveness probe. If you have one that checks Sidekiq, make sure a quiet, draining pod still passes it. Otherwise Kubernetes may restart the container in the middle of the drain.

Protect it during node drains too

Deploys aren't the only time pods get stopped. Cluster upgrades and kubectl drain evict pods as well. A PodDisruptionBudget stops Kubernetes from evicting too many workers at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: sidekiq-worker-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: sidekiq-worker

Evicted pods still run the preStop hook and respect the grace period, so the same drain logic applies. The PDB just makes sure they drain one at a time.

How to test it before you trust it

Don't wait for the next real deploy to find out if this works. Test it on purpose in a non-production environment.

1. Add a slow test job:

# app/jobs/slow_test_job.rb
class SlowTestJob
  include Sidekiq::Job

  def perform(seconds = 120)
    Sidekiq.logger.info "SlowTestJob starting, will run #{seconds}s"
    seconds.times { sleep 1 }
    Sidekiq.logger.info "SlowTestJob finished cleanly"
  end
end

2. Enqueue it:

kubectl exec -it deploy/sidekiq-worker -- \
  bundle exec rails runner 'SlowTestJob.perform_async(120)'

3. Find the pod that picked it up, then delete it:

kubectl get pods -l app=sidekiq-worker
kubectl delete pod sidekiq-worker-7d9f8c6b5-abcde

4. Watch the logs of the terminating pod:

kubectl logs -f sidekiq-worker-7d9f8c6b5-abcde

You should see something like this:

[prestop] 10:14:02 sending TSTP to Sidekiq (quiet mode)
INFO: Received TSTP, no longer accepting new work
[prestop] 10:14:07 still 1 busy job(s), waited 0s of 540s
[prestop] 10:14:12 still 1 busy job(s), waited 5s of 540s
...
INFO: SlowTestJob finished cleanly
[prestop] 10:15:57 no busy jobs after 115s, letting shutdown continue
INFO: Shutting down
INFO: Bye!

If you see SlowTestJob finished cleanly and Bye! before the pod disappears, the drain works. If the logs stop abruptly with no Bye!, the pod was killed. Check the grace period and the PID 1 setup.

5. Check events for hook failures:

kubectl get events --field-selector reason=FailedPreStopHook

An empty result is what you want.

Jobs still need to be idempotent

The preStop hook handles planned shutdowns: deploys, scale-downs, node drains. It doesn't handle everything. Nodes crash. Containers get OOM-killed. Someone runs kubectl delete pod --force --grace-period=0. In all of those cases, no hook runs.

So the real safety net is still a job that can safely run twice. A simple pattern is to record that the work is done and check before doing it again:

class SendInvoiceJob
  include Sidekiq::Job

  def perform(invoice_id)
    invoice = Invoice.find(invoice_id)
    return if invoice.sent_at.present?   # already done, safe to skip

    InvoiceMailer.with(invoice: invoice).deliver_now
    invoice.update!(sent_at: Time.current)
  end
end

This still leaves one gap. If the worker dies after the email goes out but before sent_at is saved, the retry sends the invoice again. No hook can close that window. For a plain notification, a rare duplicate may be fine. For anything tied to money, save a delivery record with a stable key before sending, pass that key to the provider if it supports idempotency, and check the record before retrying. Test it by killing the worker between the send and the database update.

For long jobs that loop over many records, save progress as you go, so a retry picks up where it stopped instead of starting over:

class BackfillJob
  include Sidekiq::Job

  def perform(batch_id)
    batch = Batch.find(batch_id)

    batch.items.where(processed: false).find_each do |item|
      process(item)
      item.update!(processed: true)   # checkpoint after each item
    end
  end
end

The drain makes deploys clean. Idempotency makes everything else survivable.

Checklist

Before you call this done:

• Sidekiq runs as PID 1 (exec form CMD, or exec in the entrypoint)
• preStop hook sends TSTP and waits for busy jobs
• terminationGracePeriodSeconds > drain time + Sidekiq timeout + buffer
• Drain time is based on your longest real job
• maxUnavailable: 0 so new workers start before old ones drain
• PodDisruptionBudget for node drains
• Liveness probe still passes while the pod is quiet
• Tested with a slow job and a manual pod delete
• Jobs are idempotent for the cases no hook can catch

What changed

After this went in, deploys stopped killing jobs. Long jobs finished on the old pod while new pods picked up new work. Nobody had to check the Retry set after a release, and nobody had to clean up half-processed work by hand.

The whole fix is a shell script, a couple of numbers, and one signal. The hard part was noticing that Kubernetes and Sidekiq each had a perfectly reasonable idea of "shutting down gracefully," and the two didn't match.

Updated: thanks to Raza Hussain for spotting the stale heartbeat in the busy check, and to Tim Urista for pointing out the crash window in the invoice example. Both are fixed above.

I write up real production fixes like this one. If your team runs Sidekiq on Kubernetes, go check your terminationGracePeriodSeconds right now. If it still says 30, you're probably losing jobs.