When Workflows Get Stuck: Troubleshooting and Preventing Task Deadlocks in Apache DolphinScheduler

One ClickHouse outage triggered a chain reaction: 200+ tasks went into aggressive retries, overwhelmed ClickHouse again, and the Serial Wait execution strategy eventually caused a deadlock that left the tasks completely stuck. This article documents the full troubleshooting process and the ultimate solution.

1. The Incident

1.1 What Happened

Here’s the timeline:

• February 3, 2026: ClickHouse went down, and DolphinScheduler tasks stopped running.
• February 4, 2026: The issue was discovered. After restarting DolphinScheduler, 200+ backfill tasks were triggered almost instantly.
• Result: The sudden task surge overwhelmed the freshly recovered ClickHouse and brought it down again.

After restarting ClickHouse, we found that all the tasks were blocked, with the entire UI basically turning red:

The symptoms were:

• The task at the very bottom showed Failed. After clicking Stop, its status changed to Ready to Stop and then remained stuck.
• All subsequent tasks showed Serial Wait.
• The entire workflow was effectively deadlocked and could no longer make progress.

Server configuration: 4 CPU cores and 32 GB RAM. Resources were not the problem—the issue was the scheduling strategy.

1.2 What Caused the Deadlock

The culprit was the Serial Wait task execution strategy:

Here’s where Serial Wait can get you into trouble:

1. When one task instance is running, subsequent instances are queued and wait.
2. If the running task enters an abnormal state, such as Failed or Ready to Stop, it may neither release the lock nor actually stop.
3. Subsequent tasks keep waiting for the “previous” instance to finish, but it never does → deadlock.

We also tried running the task again through Backfill, but it immediately returned an error along the lines of:

“There is already a task in progress.”

That was the general meaning of the error message; unfortunately, we didn’t preserve the exact message from the incident.

Restarting DolphinScheduler? No effect.

Manually stopping the task? Stuck at Ready to Stop.

Waiting for it to recover on its own? Don’t count on it.

2. The Solution: Go Straight to the Database

Since the UI could no longer resolve the issue, it was time to go straight to the source: the MySQL database.

2.1 Identify the Key Tables

Open the DolphinScheduler database and you’ll find 65 tables.

Based on the naming convention, tables beginning with t_ds_process_* are related to workflow and task execution:

The key tables are:

2.2 Inspect the Abnormal Data

Query the t_ds_process_instance table to find the blocked task instances:

SELECT id, name, state, start_time, end_time
FROM t_ds_process_instance
WHERE state NOT IN (7)  -- 7 = Success
ORDER BY id DESC
LIMIT 50;

Commonly used state codes:

Sure enough, there were plenty of records with state = 14 (Serial Wait) and state = 4 (Ready to Stop):

2.3 Clear the Blocked Data

There are two approaches.

Option 1: Delete the Records Directly

Simple and aggressive:

-- Delete blocked workflow instances (use with caution; back up the database first)
DELETE FROM t_ds_process_instance
WHERE state IN (4, 14)  -- Ready to Stop, Serial Wait
  AND process_definition_code = <YOUR_WORKFLOW_CODE>;

-- Also clean up the corresponding task instances
DELETE FROM t_ds_task_instance
WHERE process_instance_id IN (
    SELECT id FROM t_ds_process_instance
    WHERE state IN (4, 14)
);

Option 2: Change the State

A relatively gentler approach:

-- Change blocked tasks to Failed to release the lock
UPDATE t_ds_process_instance
SET state = 6  -- 6 = Failed
WHERE state IN (4, 14);

Important: Always back up the database before making changes. In a production environment, do not run DELETE blindly. Run a SELECT first to verify exactly which records will be affected.

2.4 Run the Backfill Again

Once the blocked data has been cleared, return to the DolphinScheduler UI:

Workflow Definition → Run → Backfill

This time, there was no error, and the tasks ran successfully:

3. How to Prevent It from Happening Again

Lessons learned. Here are a few practical ways to avoid falling into the same trap.

3.1 Use Serial Wait with Caution

Recommendation: Unless your business logic strictly requires execution in sequence, consider using Serial Discard instead of Serial Wait. This helps prevent tasks from piling up and eventually causing a deadlock.

3.2 Configure Task Timeouts

Set reasonable task timeouts so that a task cannot remain stuck indefinitely and occupy resources:

Timeout Alert + Timeout Failure

3.3 Set Up Monitoring and Alerts

• Monitor the number of pending DolphinScheduler tasks.
• Monitor the health of downstream services such as ClickHouse.
• Alert on task failures as soon as they occur. Don’t wait until the next morning to discover that something went wrong.

3.4 Build Resilience into Downstream Services

The root cause of this incident was the ClickHouse outage. Consider the following measures:

• Deploy ClickHouse as a cluster to avoid a single point of failure.
• Add retry mechanisms and circuit-breaking logic on the task side.
• Limit concurrency to prevent a sudden task surge after a service restart.

4. Summary

The root cause: Serial Wait + abnormal task states = deadlock

The solution: When the UI can no longer resolve the issue, go directly to the database and clear the abnormal records in the t_ds_process_instance table.

Key takeaways:

1. Serial Wait is a double-edged sword. If used incorrectly, it can lead to task accumulation and deadlocks.
2. The scheduling system’s database can be the last line of defense when the UI and normal operational controls can no longer recover a blocked workflow.
3. Monitoring and alerting need to be in place. The earlier you detect a problem, the smaller the impact and the easier the recovery.

Hopefully, you’ll never need the solution described in this article.

But if you ever do, back up the database before touching anything.