Run Spring Scheduler Tasks Only Once Across Multiple Instances With ShedLock

Introduction

A Spring Boot application with @Scheduled behaves as expected on a single instance. As soon as the application scales to multiple instances, however, every task runs on every instance. This can lead to conflicts.

This article shows how I solved this with ShedLock.

The starting point

My Spring Boot applications run on AWS Elastic Beanstalk and AWS Fargate. The number of instances varies from 1–2 to n, depending on the load. Some of these applications process tasks regularly with the Spring Scheduler:

@Scheduled(cron = "0 */5 * * * *")
public void processPendingJobs() {
    // ...
}

With two instances, processPendingJobs() runs twice every five minutes. In the best case, the task processes different jobs on each instance. In the worse case, the same job is executed twice.

To process jobs one after the other, ShedLock is a good fit. Because the instances scale horizontally, the lock has to be stored in a central place. ShedLock provides the JdbcTemplateLockProvider for this.

I deliberately did not want distributed processing, where multiple instances split the jobs among themselves and process them in parallel. This was not necessary for my cases and I wanted to keep the complexity low.

ShedLock

ShedLock is not a distributed scheduler. It extends the existing Spring Scheduler with a lock: before a task starts, the instance tries to set a lock for the task. If this succeeds, the task runs, otherwise the instance skips this run. The lock has an expiry date so that a task is not blocked forever if an instance dies in the middle of a run. A LockProvider determines where the lock is stored. Because my instances share a database, I use the JdbcTemplateLockProvider.

Dependencies

<dependency>
    <groupId>net.javacrumbs.shedlock</groupId>
    <artifactId>shedlock-spring</artifactId>
    <version>...</version>
</dependency>
<dependency>
    <groupId>net.javacrumbs.shedlock</groupId>
    <artifactId>shedlock-provider-jdbc-template</artifactId>
    <version>...</version>
</dependency>

The latest versions (shedlock-spring and shedlock-provider-jdbc-template) are available on Maven Central.

The table

The table belongs in the database migration (for example Flyway or Liquibase):

CREATE TABLE shedlock
(
    name       VARCHAR(64)  NOT NULL,
    lock_until TIMESTAMP(3) NOT NULL,
    locked_at  TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
    locked_by  VARCHAR(255) NOT NULL,
    PRIMARY KEY (name)
);

There is one row per task. name is the name of the lock, lock_until the expiry date and locked_at the time at which the lock was set. locked_by contains the instance that holds the lock.

The scheduler configuration

@Configuration
@EnableScheduling
@EnableSchedulerLock(defaultLockAtMostFor = "PT30S")
public class SchedulerConfig {

    @Bean
    public LockProvider lockProvider(DataSource dataSource) {
        return new JdbcTemplateLockProvider(
            JdbcTemplateLockProvider.Configuration.builder()
                .withJdbcTemplate(new JdbcTemplate(dataSource))
                .usingDbTime()
                .withLockedByValue(UUID.randomUUID().toString())
                .build()
        );
    }
}

withLockedByValue(String lockedBy) determines what is written to locked_by in the database. By default, ShedLock writes the hostname there. On Fargate, the hostname is derived from the private IP and can be reused by another task later. That is why I generate a UUID with UUID.randomUUID() at startup. Because the @Bean method only runs once, it stays the same for the lifetime of the application and identifies the instance uniquely. The other configuration methods can be found in the documentation.

A task gets its lock via @SchedulerLock:

@Scheduled(cron = "0 */5 * * * *")
@SchedulerLock(name = "processPendingJobs", lockAtLeastFor = "PT10S")
public void processPendingJobs() {
    // ...
}

The name is the key in the table and must be unique per task. lockAtLeastFor holds the lock for at least this long, even if the task finishes earlier.

After a run, the table looks like this, for example:

ShedLock locks per task and per run. So locked_by shows which instance currently holds the lock, not a permanently responsible instance. On the next run, a different instance can get the lock.

Optional: instance ID from AWS

A UUID is perfectly sufficient for the lock. In my projects, however, I already use the dependency aws-app-instance-id (Maven Central). I did not develop it for ShedLock, but I can reuse it for this. It provides a unique ID for the running instance:

• On Beanstalk this is the EC2 instance ID from the Instance Metadata Service (IMDSv2).
• On Fargate it is the task ID from the ECS Task Metadata Endpoint.

With this ID in locked_by, I can find the matching log entries in CloudWatch when an error occurs, because the logs can be filtered by the ID.

<dependency>
    <groupId>io.github.alaugks</groupId>
    <artifactId>aws-app-instance-id</artifactId>
    <version>0.1.0</version>
</dependency>

Here is the configuration for Fargate. On prod and stage the task ID is determined. Locally and in the CI/CD (dev and test) a mock ID is set, so that nothing depends on AWS. If the ID cannot be determined, the fallback uses a UUID:

@Configuration
public class AwsInstanceConfig {

    @Bean
    @Profile({"prod", "stage"})
    public InstanceService awsInstanceService() {
        return InstanceService
            .builder(
                FargateInstanceService.builder().build()
            )
            .fallback(e -> {
                log.warn("Could not determine instance ID", e);
                return "instance-" + UUID.randomUUID();
            })
            .build();
    }

    @Bean
    @Profile({"dev", "test"})
    public InstanceService mockInstanceService() {
        return InstanceService
            .builder(
                MockInstanceService.builder("mock-" + UUID.randomUUID()).build()
            )
            .build();
    }
}

In the LockProvider, the ID from the InstanceService replaces the UUID:

@Bean
public LockProvider lockProvider(DataSource dataSource, InstanceService instanceService) {
    return new JdbcTemplateLockProvider(
        JdbcTemplateLockProvider.Configuration.builder()
            .withJdbcTemplate(new JdbcTemplate(dataSource))
            .usingDbTime()
            .withLockedByValue(instanceService.instanceId())
            .build()
    );
}

For Beanstalk, I replace FargateInstanceService with BeanstalkInstanceService. Further details can be found in the README of the dependency.

Related article

• Webhook Hub With Spring Boot Using Strategy Pattern, Sub-Strategies and Collection Injection