Installing Apache Airflow on Ubuntu 24.04

Apache Airflow is an open-source workflow management platform that manages data pipelines and automates workflows such as Extract, Transform, and Load (ETL) processes. It uses Python-based Directed Acyclic Graphs (DAGs) to schedule and execute tasks while streamlining the management of all required dependencies for error-free execution. This guide installs Apache Airflow on Ubuntu 24.04, configures a secure environment behind Nginx with Let's Encrypt, and tests the deployment with a sample DAG. By the end, you'll have a working Apache Airflow instance served securely over HTTPS.

Prerequisites

• An Ubuntu 24.04 server with at least 4 GB RAM.
• A domain A record with your DNS provider pointing to the server's IP address.

Install Apache Airflow

Apache Airflow is available as a Python package installed with a package manager such as Pip.

1. Update the server's package index:

$ sudo apt update

2. View the available Python version:

$ python3 --version

Your output should be similar to:

Python 3.12.3

Install Python if it's not available on your server:

$ sudo apt install python3

3. Install the python3-venv virtual environment module and the PostgreSQL development library:

$ sudo apt install python3-venv libpq-dev -y

4. Create a new virtual environment, for example airflow_env:

$ python3 -m venv airflow_env

5. Activate the airflow_env virtual environment:

$ source ~/airflow_env/bin/activate

Verify that your shell prompt changes to the airflow_env virtual environment:

(airflow_env) linuxuser@example:~$

6. Use Pip to install Apache Airflow with PostgreSQL support:

$ pip install apache-airflow[postgres] psycopg2

7. Install PostgreSQL:

$ sudo apt install postgresql postgresql-contrib

8. Start the PostgreSQL service:

$ sudo systemctl start postgresql

9. Access the PostgreSQL console using the postgres user:

$ sudo -u postgres psql

Your output should be similar to:

psql (16.6 (Ubuntu 16.6-0ubuntu0.24.04.1))
Type "help" for help.

postgres=#

10. Create a new airflow PostgreSQL user with a strong password. Replace YourStrongPassword with your desired password:

postgres=# CREATE USER airflow PASSWORD 'YourStrongPassword';

11. Create a new database, for example airflowdb:

postgres=# CREATE DATABASE airflowdb;

12. Grant the airflow user full privileges to all tables in the public schema:

postgres=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO airflow;

13. Grant the airflow user ownership privileges to the airflowdb database:

postgres=# ALTER DATABASE airflowdb OWNER TO airflow;

14. Grant the airflow user all privileges on the public schema:

postgres=# GRANT ALL ON SCHEMA public TO airflow;

15. Exit the PostgreSQL console:

postgres=# exit;

16. Open the airflow.cfg file in your Airflow installation directory:

$ nano ~/airflow/airflow.cfg

Temporarily initialize the database and start the Airflow scheduler to create the necessary directories if the airflow directory is missing:

$ airflow db init; airflow scheduler

Press Ctrl+C to stop the scheduler.

17. Replace the default executor and sql_alchemy_conn values with the following configuration to enable parallel execution and set PostgreSQL as the metadata database:

executor = LocalExecutor
sql_alchemy_conn = postgresql+psycopg2://airflow:YourStrongPassword@localhost/airflowdb

Save and close the file.

18. Initialize the Airflow metadata database to apply the changes:

$ airflow db init

Your output should be similar to:

DB: postgresql+psycopg2://airflow:***@localhost/airflow
[2025-01-05T23:58:36.808+0000] {migration.py:207} INFO - Context impl PostgresqlImpl.
[2025-01-05T23:58:36.809+0000] {migration.py:210} INFO - Will assume transactional DDL.
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 5f2621c13b39
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
Initialization done

19. Create a new administrative user to use with Apache Airflow. Replace admin with your desired username:

$ airflow users create \
   --username admin \
   --password yourSuperSecretPassword \
   --firstname Admin \
   --lastname User \
   --role Admin \
   --email admin@example.com

20. Start the Airflow web server on port 8080 as a background process and redirect logs to webserver.log:

$ nohup airflow webserver -p 8080 > webserver.log 2>&1 &

21. Start the Airflow scheduler and redirect logs to scheduler.log:

$ nohup airflow scheduler > scheduler.log 2>&1 &

Configure Nginx as a Reverse Proxy to Expose Apache Airflow

Apache Airflow listens for connections on the default port 8080. Use Nginx to front that port and serve requests over HTTP and HTTPS.

1. Install Nginx:

$ sudo apt install -y nginx

2. Create a new airflow Nginx virtual host configuration file:

$ sudo nano /etc/nginx/sites-available/airflow

3. Add the following configuration to the file. Replace airflow.example.com with your actual domain:

server {
    listen 80;
    server_name airflow.example.com;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Save and close the file. This configuration listens for connections on your airflow.example.com domain and forwards them to the Apache Airflow port 8080.

4. Link the configuration to the Nginx sites-enabled directory to enable it:

$ sudo ln -s /etc/nginx/sites-available/airflow /etc/nginx/sites-enabled/

5. Test the Nginx configuration for errors:

$ sudo nginx -t

Your output should be similar to:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

6. Reload Nginx to apply the changes:

$ sudo systemctl reload nginx

7. Allow connections to the HTTP port 80 through the firewall:

$ sudo ufw allow 80/tcp

8. Reload UFW to apply the firewall configuration changes:

$ sudo ufw reload

9. Access your airflow.example.com domain in a web browser and verify that the Airflow login page displays.

http://airflow.example.com

Generate Trusted SSL Certificates to Secure Apache Airflow

SSL certificates encrypt the connection between a client and the Apache Airflow server. Use Certbot to generate Let's Encrypt SSL certificates.

1. Install the Certbot Let's Encrypt client:

$ sudo snap install --classic certbot

Install Snap first if it's not available on your workstation:

$ sudo apt install snapd -y

2. Move the Certbot binary to /usr/bin to enable it as a system-wide command:

$ sudo ln -s /snap/bin/certbot /usr/bin/certbot

3. Request a new Let's Encrypt SSL certificate using the Nginx plugin and your domain. Replace airflow.example.com with your actual domain and admin@example.com with your active email address:

$ sudo certbot --nginx --redirect -d airflow.example.com -m admin@example.com --agree-tos

Your output should be similar to the following when the certificate request succeeds:

...
Account registered.
Requesting a certificate for airflow.example.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/airflow.example.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/airflow.example.com/privkey.pem
This certificate expires on 2025-04-21.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

Deploying certificate
Successfully deployed certificate for airflow.example.com to /etc/nginx/sites-enabled/airflow
Congratulations! You have successfully enabled HTTPS on https://airflow.example.com
...

4. Verify that Certbot auto-renews the SSL certificate before it expires:

$ sudo certbot renew --dry-run

5. Restart Nginx to apply the SSL configuration changes:

$ sudo systemctl restart nginx

Access Apache Airflow

1. Access the Apache Airflow web interface using your domain:

https://airflow.example.com

Enter the credentials you set earlier to log in:

• Username: admin
• Password: yourSuperSecretPassword

Create and Run DAGs Using Apache Airflow

1. Create the dags directory in the Airflow installation directory:

$ mkdir ~/airflow/dags

2. Create a new my_first_dag.py Python application file in the dags directory:

$ nano ~/airflow/dags/my_first_dag.py

3. Add the following code to define a new DAG:

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta

with DAG(
    'my_first_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval=timedelta(days=1),
    catchup=False
) as dag:

    def print_hello():
        print('Hello from Airflow!')

    hello_task = PythonOperator(
        task_id='hello_task',
        python_callable=print_hello
    )

Save and close the file. This creates a my_first_dag sample DAG that runs daily and prints Hello from Airflow!.

4. In the Apache Airflow interface, navigate to the DAGs list, find the DAG, and enable it to trigger it manually.

5. Use the Graph View and Event Log to monitor the DAG.

Next Steps

Apache Airflow is running behind Nginx with HTTPS and executing your first DAG. From here you can:

• Add more DAGs to automate real ETL and data pipeline workflows
• Configure Airflow connections and variables for external systems (databases, APIs, object storage)
• Set up monitoring and alerting for task failures and SLA misses
• Review the Certbot renewal timer and Airflow's authentication/RBAC options before going to production

For the full guide with additional tips, visit the original article on Vultr Docs.