1. Why Apache DolphinScheduler?
1.1 What Is Apache DolphinScheduler?
Apache DolphinScheduler is a visual, distributed workflow orchestration and scheduling system designed for big data workloads.
Put simply, it serves as the “command center” for orchestrating and automatically executing complex ETL jobs. You can turn your existing SQL scripts, Shell scripts, and other tasks into a dependency-based workflow graph (DAG) through drag-and-drop, then configure a schedule, such as 2:00 a.m. every day. DolphinScheduler will execute the workflow automatically on schedule and monitor the status of each task.
1.2 Why choose DolphinScheduler?
In data development, workflow orchestration is one of those challenges you simply cannot avoid. In the early days, you might get by with Crontab, but once the number of tasks grows, dependencies quickly become difficult to manage. Failure recovery often means manual intervention, and being woken up in the middle of the night by an alert only to dig through logs becomes part of the routine.
DolphinScheduler is designed to solve exactly these problems. It uses visual DAGs to make task dependencies easy to understand, supports automatic retries, backfilling, and alerting, and offers a lightweight deployment experience without relying on an external database. DolphinScheduler is also an open-source project originally developed in China, with extensive Chinese-language documentation and community resources, making it easier for beginners in China to get started.
Compared with Airflow and Azkaban, it is lighter and easier to get started with. With Docker, you can deploy it with minimal setup and quickly build a reliable workflow scheduling system. That is why I chose DolphinScheduler as the scheduling foundation for this exploration.
1.3 Comparing Popular Workflow Scheduling Tools
- If you prioritize broad industry adoption, a rich ecosystem, and a team with strong Python expertise → choose Airflow. It is the “Swiss Army knife” of data engineering.
- If you prioritize ease of use and visual workflow design, have a team working primarily with Java/Scala, or want a lower barrier to entry → choose DolphinScheduler. It has become a widely adopted choice for enterprise data platform development in China.
- If you are maintaining a legacy Hadoop environment with a relatively small number of jobs → you can continue using Azkaban. However, it is not recommended for new projects.
2. Deployment Options and Why We Chose Docker
Why choose Docker to deploy DolphinScheduler?
After comparing the different options, I began my exploration of DolphinScheduler. Next, let’s look at how to deploy DolphinScheduler.
3. Deploying with Docker
With Docker, deployment is essentially a one-step process, without the need to spend time sorting out the environment.
3.1 Installing Docker
Open the Ubuntu terminal and run the following commands:
# Update the package index
sudo apt update
# Install dependencies
sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
# Add the official Docker GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
# Check your Ubuntu version
lsb_release -a
# Add the Docker stable repository based on your Ubuntu version, such as 20.04 or 22.04
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# Add the official Docker repository for Ubuntu 22.04 Jammy
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# Install Docker
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io
# Add the current user to the docker group to avoid using sudo every time
sudo usermod -aG docker $USER
# After running this command, log out and log back in,
# or run newgrp docker to apply the new group permissions.
# Enable Docker to start automatically at boot
sudo systemctl enable docker
sudo systemctl start docker
Verify that Docker is installed correctly:
docker --version # If the version number is displayed, the installation was successful
Then verify that the regular user can run Docker commands:
docker --version # If there is no permission error, the configuration is complete
Starting DolphinScheduler in Standalone Mode
When starting DolphinScheduler in standalone mode, I encountered a Docker image pull failure:
If you see a connection refused error, it means Docker cannot connect to the official Docker Hub image registry. This is usually a network issue, particularly when access to Docker Hub is restricted or slow from certain regions.
Solution: Try configuring a Docker registry mirror. You can configure mirrors provided by USTC, NetEase, Baidu, or other providers, or use a dedicated accelerator provided by Alibaba Cloud.
Another option is to deploy DolphinScheduler in a different way: download the DolphinScheduler binary package directly and run it without Docker.
- Note: If Docker image pulls continue to fail, consider switching to the binary package deployment method.
4. Binary Package Deployment
4.1 Download the Installation Package
If pulling the image from other registries is still unreliable, you can use the official Apache archive site (archive.apache.org), which provides a stable download source.
wget https://archive.apache.org/dist/dolphinscheduler/3.2.2/apache-dolphinscheduler-3.2.2-bin.tar.gz
# The Apache permanent archive link is more reliable than the dlcdn mirror
# and avoids potential 404 errors.
Then came the long wait. After three interrupted transfers and resumptions, the download took 4 hours and 31 minutes.
4.2 Extracting and Starting DolphinScheduler
Finally, the long wait was over. The package was downloaded successfully.
# Extract the installation package
tar -xvzf apache-dolphinscheduler-3.2.2-bin.tar.gz
# Enter the extracted directory
cd apache-dolphinscheduler-3.2.2-bin
# Start the Standalone Server
bash ./bin/dolphinscheduler-daemon.sh start standalone-server
To stop the service, use the corresponding stop command shown below:
Run the start command again whenever you need to restart the service. Once it is running, open your browser and you can start using DolphinScheduler.
cd apache-dolphinscheduler-3.2.2-bin
bash ./bin/dolphinscheduler-daemon.sh start standalone-server
5. Accessing the Web UI
The default username is admin, and the default password is dolphinscheduler123.
URL:
http://<your Ubuntu IP address>:12345/dolphinscheduler/ui
If you are running Ubuntu inside a virtual machine, make sure the network is configured in Bridged mode or with NAT port forwarding, so that the host machine can access the virtual machine's IP address.
6. Hands-On with DolphinScheduler
Next, let's run a scheduled workflow and get a Shell task working end to end.
Goal: Create a simple Shell workflow and run it on a schedule.
① Create a Tenant
A tenant is the Linux system user that DolphinScheduler uses to execute actual tasks. This ensures that tasks have the appropriate file system permissions.
After creating the tenant, assign it to the user in User Management. Find the tenant you just created in the dropdown list and click Submit. This gives the admin user permission to submit tasks.
Fill in the required information. The queue can remain default.
② Create a Project
All workflows must belong to a project.
③ Create a Workflow
This is where you build the DAG. Enter the visual drag-and-drop canvas, which is the core part of DolphinScheduler: designing DAGs (Directed Acyclic Graphs) visually.
From the toolbar on the left, drag the task types you need, such as a Shell node, onto the canvas. Enter the script directly or reference a resource file you have uploaded. Connections between nodes define the upstream and downstream dependencies in your ETL process. For example, the DWD cleansing task can run only after the ODS loading task has completed.
DAG (Directed Acyclic Graph): This is the core concept. All task dependencies are clearly represented in the graph. Put simply, a DAG shows which task runs first and which task runs next, with arrows representing the execution order.
Task types: DolphinScheduler natively supports dozens of task types, including Shell, Hive, Spark, and SQL, making it well suited to big data workloads. Hive SQL can be executed directly.
④ Release the Workflow
Enter the node name and add the script. Once everything is configured, click Save Workflow, followed by Release.
⑤ Schedule the Workflow
A dialog box will then ask whether you want to configure a schedule. Click to configure one if needed, or close the dialog if you do not need scheduling.
With a scheduled workflow, you can have the task run during the early hours of the morning, allowing your data warehouse ETL jobs to run automatically every day.
⑥ Check the Results
Click Workflow Instances on the left, then select the workflow instance you just created. A green status indicates that the workflow completed successfully.
Right-click any node and select View Log to see the output from the echo command you added to the script.
The Core Concepts of a DAG
-
Directed: The arrows indicate the execution direction.
A → Bmeans A runs before B. -
Acyclic: There are no cycles in the graph, so the workflow cannot enter an infinite loop such as
A → B → A.
What happens if you try to draw another connection from C back to A?
The system will tell you that creating a cycle is not allowed. This is exactly what the “acyclic” property means.
You can also change the task type to an SQL or Hive node to get a feel for how different task types work.
Workflow Summary
Create a tenant → Create a project → Drag tasks to build a DAG → Release → Schedule
This is the standard workflow for getting started with DolphinScheduler.
Issues Encountered
No installation candidate
After identifying your system version, troubleshoot the issue step by step using the following commands. Once the version number is displayed correctly, the installation issue should be resolved.
# 1. Remove any existing old configuration
sudo rm -f /etc/apt/sources.list.d/docker.list
sudo rm -f /etc/apt/sources.list.d/docker-ce.list
# Update the package list
sudo apt update
sudo apt install -y ca-certificates curl gnupg lsb-release # 2. Install required dependencies
# 3. Add the official Docker GPG key
# Create the keyring directory
sudo mkdir -p /etc/apt/keyrings
# Download and add the Docker GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
# Set permissions for the key file
sudo chmod a+r /etc/apt/keyrings/docker.gpg
# 4. Add the official Docker repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# Update the package list
sudo apt update
# 5. Install Docker
sudo apt install -y docker-ce docker-ce-cli containerd.io
7. FAQ
- How do I find the server IP address?
① Windows: Open the command prompt, run ipconfig, and look for the IPv4 address.
② Mac/Linux: Open the terminal, run ifconfig or ip addr, and look for the number following inet.
③ Cloud server: Check the public IP address provided in your cloud service provider's console.
④ If Docker is running locally: Simply access http://localhost:12345/dolphinscheduler/ui.
8. Summary
Get familiar with DolphinScheduler's drag-and-drop workflow orchestration → Connect to Hive/MySQL and schedule SQL tasks → Learn the fundamentals of DAG development with Airflow.
