Quartz integration in Spring

Why Quartz? motivation

Spring comes already with an own timer and this is fine if we have very simple requirements and we don’t really care about state or on which pod the timer is actually running. To be more precise the timer/ job is running in each of our spring containers independently.

Lets be honest, this is usually never really the case and so it is common to invite ShedLock to the party. Are we done then?

Usually no, honestly we want in the most cases just a bit more:

Run jobs in the cluster on different nodes simultaneous
Have a kind of state management
The ability to provide some kind of configuration to each run
Have some kind of way for easy retries etc.

Hey why don’t use Spring Batch then?

Because to have just a timer and not really a batch-job it is somehow the wrong tool. Furthermore Spring-Batch is build with a different goal in mind, where we start a container with one job inside to process mass of data. Quartz is more a timer, which we want to use to trigger either smaller task in our application or run kind of repeating small task and are more concerned to orchestrate and Monitor them.

Spring Batch = Batch Framework to handle large datasets which should be processed, usually in an own container / pod for the duration of the Job
Quartz = Timer which also supports a JDBC/ REDIS store for smaller timer jobs which should run in our app cluster

Steps for the integration

Setup the DB
Configure Quartz Scheduler
Configure CDI Job Factory
Register our timers
Set a thread priority for our job

Setup the DB

Overall quartz comes with SQL scripts and liquibase scripts you may just include. Read here. For unit test we can just simply use the build in feature:

spring:
  quartz:
    jdbc:
      initialize-schema: always

Configure Quartz Scheduler

We will use the CMT Store here, for more details because of the why please read the following tutorial.

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-quartz</artifactId>
		</dependency>

After adding the dependency we have to configure Quartz, in this example we set the thread count to 2, adjust this value based on your needs. The following configuration will usually create a LocalDataSourceJobStore which is an JobStoreCMT.

spring:
  quartz:
    wait-for-jobs-to-complete-on-shutdown: false
    job-store-type: jdbc
    overwrite-existing-jobs: true
    jdbc:
      initialize-schema: always
    properties:
      org.quartz.jobStore.isClustered: true
      org.quartz.scheduler.instanceId: AUTO
      org.quartz.scheduler.skipUpdateCheck: true
      org.quartz.threadPool.threadCount: 2
  
  datasource:
    url: jdbc:h2:file:./quartz-db
    username: sa
    password:

Note: We init the DB just with quartz itself. But quartz contains in the core a liquibase and a SQL file which you may include into your DB update process.

Yes thats basically it, we will discuss details in further deep dives, but for now lets gets a job running.

Create a timer Job

To setup a timer we have to:

Add a job class
Configure the timer

Add a Spring configuration

// runs only one of these jobs in out cluster
// note we have no spring annotation here, but it contains spring beans
@DisallowConcurrentExecution 
public class QuatzTimerJob implements Job {
	// we can use spring beans, yes we cam
    @Autowired YourService yourService;
    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
		yourService.doSomething();
    }

}

Configure the timer

@Configuration
public class QuartzTimerConfiguration {
	// this registers our job itself
    @Bean
    public JobDetailFactoryBean quatzTimerJob() {
        final var jobDetailFactory = new JobDetailFactoryBean();
        jobDetailFactory.setJobClass(QuatzTimerJob.class);
        jobDetailFactory.setDescription("Simple timer like @Scheduled - but already like with ShedLock");
        jobDetailFactory.setDurability(true);
        return jobDetailFactory;
    }
    // here we add the timer, in this case every 30s, with a lower priority as normal for the job
    @Bean
    public SimpleTriggerFactoryBean trigger(JobDetail quatzTimerJob) {
        final var trigger = new SimpleTriggerFactoryBean();
        trigger.setJobDetail(quatzTimerJob);
        trigger.setRepeatInterval(30_000);
        trigger.setPriority(Thread.NORM_PRIORITY - 1);
        trigger.setRepeatCount(SimpleTrigger.REPEAT_INDEFINITELY);
        return trigger;
    }
}

Trigger a job

In difference to a normal Spring timer we may now fire triggers for Quartz Jobs, even add data to the triggers itself, at let Quartz handle the heavy work for us. Handy if we want QUEUE some work, or just distribute the load in our cluster to all available nodes. For this we have to:

Create a job instance we want to execute
Configure the job – but now without any default trigger
Create somewhere in our code the triggers, usually with some payload data

Create a job

Very simple thing here, just getting one value from the job context and calling out service.

public class TriggeredJob implements Job {

    @Autowired YourService yourService;
    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        long count = context.getMergedJobDataMap().getLong("count"));
        yourService.doSomething(count);
    }

}

Configure the job

The only special thing here is that we request a recovery on a different node, should this node for some reason die during the execution of the job.

@Configuration
public class QuartzJobConfiguration {
    @Bean
    public JobDetailFactoryBean createStoreItemsJob() {
        final var jobDetailFactory = new JobDetailFactoryBean();
        jobDetailFactory.setJobClass(CreateStoreItemsJob.class);
        jobDetailFactory.setDescription("Job we directly trigger if we want to run it");
        jobDetailFactory.setDurability(true);
        jobDetailFactory.setRequestsRecovery(true);
        return jobDetailFactory;
    }
}

Trigger the job

Note: The usage of @Transactional, this is by purpose, to ensure the trigger gets removed if we happen to rollback the transaction. If it is missing, quartz will silently create an own one.

    @Autowired Scheduler scheduler;
    @Autowired JobDetail createStoreItemsJob;

    @Transactional
    public void triggerStoreItemCreation() throws SchedulerException {
        Trigger t = TriggerBuilder.newTrigger()
            .forJob(createStoreItemsJob)
            .startNow() // run on this node if possible, means we have threads
            .usingJobData("count", count)
            .build();
        
        scheduler.scheduleJob(t);
    }

Testing it

You may wonder why we used a file DB in this configuration. Only with a DB which contains the trigger data after a restart you may play around and also kill the running Spring application. After a restart the jobs are triggered again.

Alternatives

We could also go for a command queue using e.g. JMS. This aproach would require additional infrastructure in form of a persistent broker. Which could be easy in the cloud but rather difficult if you have to operate and maintain it on your own. In the end a data store which have to join our DB transactions in case ACID is an issue and it always is.