Quartz Clustering

The clustering model

The single most useful thing to internalise: Quartz cluster nodes never talk to each other. There is no gossip, no leader election, no consensus protocol, no shared cache. Every node is a peer running the identical configuration, and all coordination happens through rows in a shared database.

That gives clustering three properties worth knowing up front:

• The database is a hard dependency and a single point of failure. If the database is unreachable, the whole cluster stops scheduling. Quartz clustering buys you node redundancy, not overall redundancy.
• Load balancing is opportunistic, not fair. Whichever node happens to win the lock next claims the next batch of work. On a lightly loaded cluster one node often does most of the firing. This is normal, and there’s no knob to round-robin it.
• Scaling out increases lock contention, not throughput. Every node serialises through the same TRIGGER_ACCESS row lock. Nodes add failover capacity; past a handful they mostly add contention.

The flow for a single firing:

1. A node’s scheduler thread wakes and wants work.
2. It takes a FOR UPDATE lock on the TRIGGER_ACCESS row in QRTZ_LOCKS. Any other node attempting this blocks in the database.
3. It queries QRTZ_TRIGGERS for triggers due within the look-ahead window and marks them ACQUIRED.
4. It inserts rows into QRTZ_FIRED_TRIGGERS, stamped with its own INSTANCE_NAME.
5. It commits, releasing the lock. Another node can now acquire.
6. It executes the jobs on its own thread pool and, on completion, deletes the QRTZ_FIRED_TRIGGERS rows and computes each trigger’s next fire time.

Because step 3–4 happen under the lock, two nodes cannot claim the same trigger. That’s the whole guarantee — and it’s why the lock is the thing to understand.

The 11 QRTZ_ tables

Verified against tables_postgres.sql in the Quartz 2.5.2 distribution. Every bundled DDL script creates the same eleven tables; only column types differ per database.

Three of these are the ones you’ll actually query when debugging.

QRTZ_TRIGGERS is where you look first. TRIGGER_STATE tells you what Quartz thinks is happening:

SELECT trigger_name, trigger_group, trigger_state,
       to_timestamp(next_fire_time / 1000) AS next_fire
FROM qrtz_triggers
ORDER BY next_fire_time;

States you’ll see: WAITING (normal, scheduled), ACQUIRED (claimed by a node, about to run), EXECUTING, PAUSED, BLOCKED (waiting on a @DisallowConcurrentExecution job), and ERROR. A trigger stuck in ERROR will never fire again until you reset it — Quartz does not retry out of that state, and this is a common “my job just stopped” cause.

QRTZ_FIRED_TRIGGERS is your live view of the cluster:

SELECT instance_name, trigger_name, state, requests_recovery,
       to_timestamp(fired_time / 1000) AS fired
FROM qrtz_fired_triggers;

Rows here should be short-lived. Rows that persist for hours mean a node died mid-execution without cleaning up, or a job is genuinely hung. The INSTANCE_NAME column tells you which node owns each one — that’s how you attribute a stuck job to a pod.

QRTZ_SCHEDULER_STATE is the heartbeat table, and it’s tiny by design — four columns:

SCHED_NAME, INSTANCE_NAME, LAST_CHECKIN_TIME, CHECKIN_INTERVAL

One row per live node. SELECT * FROM qrtz_scheduler_state is the fastest way to answer “how many nodes does my cluster think it has?” If you see stale rows for pods that were deleted weeks ago, checkin-based failover isn’t completing — worth investigating.

Quartz manages all eleven tables itself. Don’t write to them from application code; the state machine in QRTZ_TRIGGERS is not something to hand-edit on a running system.

QRTZ_LOCKS and the two locks

QRTZ_LOCKS is the smallest and most important table. Its entire schema is:

CREATE TABLE QRTZ_LOCKS
(
  SCHED_NAME VARCHAR(120) NOT NULL,
  LOCK_NAME  VARCHAR(40)  NOT NULL,
  PRIMARY KEY (SCHED_NAME, LOCK_NAME)
);

No value column, no owner column, no timestamp. The rows exist purely as something to take a database row lock on. There are exactly two lock names in 2.5.2, both defined in JobStoreSupport:

• TRIGGER_ACCESS — held while claiming triggers to fire. This is the cluster’s serialisation point and where contention shows up.
• STATE_ACCESS — held while reading and writing cluster state during checkin and recovery.

The mechanism is the plainest possible use of pessimistic locking. StdRowLockSemaphore issues:

SELECT * FROM QRTZ_LOCKS
WHERE SCHED_NAME = ? AND LOCK_NAME = ?
FOR UPDATE

That’s it. The FOR UPDATE makes competing nodes block in the database until the holder commits or rolls back. Lock lifetime equals transaction lifetime, which is why a long-running transaction on one node stalls scheduling cluster-wide, and why a node that hangs while holding the lock can freeze the cluster until its database session is killed.

You do not need to seed the lock rows. This is worth stating plainly because a great deal of older Quartz material tells you to run INSERT INTO QRTZ_LOCKS VALUES('TRIGGER_ACCESS') by hand. In 2.5.2, none of the bundled DDL scripts insert lock rows — StdRowLockSemaphore has an INSERT_LOCK statement and creates each row lazily, the first time that lock is needed. An empty QRTZ_LOCKS on a fresh schema is correct.

For diagnosing contention, look at your database rather than Quartz. On PostgreSQL:

SELECT a.pid, a.state, a.query_start, a.wait_event_type, a.query
FROM pg_stat_activity a
WHERE a.query ILIKE '%QRTZ_LOCKS%'
ORDER BY a.query_start;

Sessions piling up on that statement mean nodes are queueing for TRIGGER_ACCESS — usually a sign of too many nodes, a slow database, or a job holding its transaction open too long.

Configuration and real defaults

A minimal clustered quartz.properties:

# Every node in the cluster MUST share this name...
org.quartz.scheduler.instanceName = MyClusteredScheduler
# ...and MUST have a distinct id. AUTO generates one per node.
org.quartz.scheduler.instanceId = AUTO

org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
org.quartz.jobStore.dataSource = quartzDS
org.quartz.jobStore.tablePrefix = QRTZ_

org.quartz.jobStore.isClustered = true
org.quartz.jobStore.clusterCheckinInterval = 7500
org.quartz.jobStore.misfireThreshold = 60000

org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 10

The instanceName / instanceId pair is the part people invert, so it’s worth being explicit: instanceName must be identical on every node — it’s the cluster’s identity, stored as SCHED_NAME in every table. instanceId must be unique per node — set it to AUTO and Quartz derives one from hostname and timestamp, which means one properties file works everywhere. Give two nodes different instanceNames and you don’t get one cluster; you get two independent clusters sharing tables, and every job fires twice.

The documented defaults are thin, so these were read directly from JobStoreSupport in 2.5.2:

Two of those are easy to get wrong from memory. clusterCheckinInterval is widely quoted as 15000 or 20000 because the documentation’s example config uses 20000 and never states the real default — the source says 7500L. And batchTriggerAcquisitionMaxCount lives under the org.quartz.scheduler. prefix even though it behaves like a job-store concern; putting it under org.quartz.jobStore. silently does nothing.

RAMJobStore cannot cluster. If clustering appears to do nothing at all, confirm you’re actually on JobStoreTX (or JobStoreCMT inside a JEE container) — not the in-memory default.

Checkin and failover

Each node runs a ClusterManager thread that writes its own heartbeat and inspects everyone else’s, every clusterCheckinInterval milliseconds.

The heartbeat is an update to that node’s row in QRTZ_SCHEDULER_STATE, setting LAST_CHECKIN_TIME to now. Detection is then arithmetic: a node is considered failed when

now − LAST_CHECKIN_TIME > CHECKIN_INTERVAL (+ a tolerance)

When a node finds failed instances — via findFailedInstances() — it runs clusterRecover() for them under the STATE_ACCESS lock. Recovery:

1. Reads the dead node’s rows from QRTZ_FIRED_TRIGGERS.
2. For any whose job has requestsRecovery = true, schedules an immediate recovery firing on the surviving node.
3. For the rest, releases the trigger back to WAITING so it fires on its normal schedule.
4. Deletes the dead node’s QRTZ_FIRED_TRIGGERS and QRTZ_SCHEDULER_STATE rows.

Two consequences worth designing around.

requestsRecovery is opt-in and defaults to false. Unless you ask for it, a job interrupted by a node dying is simply not re-run — the trigger goes back to waiting and you wait for the next scheduled time. For work that must complete, set it:

JobDetail job = JobBuilder.newJob(BillingJob.class)
    .withIdentity("nightly-billing")
    .requestRecovery(true)      // re-run me if my node dies mid-execution
    .storeDurably()
    .build();

Recovery re-runs a job that may have partially completed. Quartz knows the node vanished; it has no idea how far the job got. With requestsRecovery(true) you are opting into at-least-once semantics, so the job body must be idempotent. If it isn’t, recovery converts a node failure into duplicated side effects — double charges, double emails.

There’s also a first-startup subtlety: on its very first checkin a node treats every existing QRTZ_SCHEDULER_STATE row as a candidate for recovery, since it can’t distinguish a live peer from a stale row it should clean up. This is normally harmless, but it means a node joining a cluster whose clocks disagree can immediately “recover” jobs that are still running elsewhere.

Which is why the clock requirement is not advisory. The documentation is unusually blunt about it: never run clustering across separate machines unless their clocks are kept within one second of each other by a time-sync daemon. Liveness is a comparison between a stored timestamp and the local clock, so skew larger than the checkin interval makes a perfectly healthy node look dead — and its in-flight jobs get recovered and re-fired while they are still executing. On containers, that means the host clocks; a drifting node doesn’t just miss its own schedule, it corrupts the cluster’s view of everyone.

The acquireTriggersWithinLock trap

This is the setting nobody reads until something misfires twice, and the interaction it has with batch acquisition is documented only in a source comment.

By default, Quartz claims one trigger at a time (batchTriggerAcquisitionMaxCount = 1). Under load that’s a round-trip and a lock acquisition per firing, so the obvious optimisation is to claim several at once:

org.quartz.scheduler.batchTriggerAcquisitionMaxCount = 20
org.quartz.scheduler.batchTriggerAcquisitionFireAheadTimeWindow = 1000

Meanwhile acquireTriggersWithinLock defaults to false, and that default is deliberate. Quartz 1.6.3 changed the behaviour: claiming a trigger uses an UPDATE whose own atomicity is enough on most databases, so taking the explicit TRIGGER_ACCESS lock around it was, in the source’s words, “considered unnecessary for most databases … and therefore a superfluous performance hit.”

But those two settings interact, and the source is explicit about it:

However, if batch acquisition is used, it is important for this behavior to be used for all dbs.

The reason is that the single-trigger safety argument rests on one atomic UPDATE. Batch acquisition claims a set of triggers across multiple statements, and without the explicit lock held for the whole batch, two nodes can interleave and claim overlapping sets — the exact duplicate-firing that clustering exists to prevent.

So the rule is simple and non-obvious:

# If you raise this above 1...
org.quartz.scheduler.batchTriggerAcquisitionMaxCount = 20
# ...you MUST also set this. Not optional.
org.quartz.jobStore.acquireTriggersWithinLock = true

The failure mode is nasty precisely because it’s rare: it needs two nodes acquiring at the same instant, so it passes every test, survives staging, and shows up in production as an occasional duplicate that nobody can reproduce. If you’re chasing intermittent double-fires in a cluster and have tuned batch acquisition, check this pair first.

If you haven’t enabled batch acquisition, leave acquireTriggersWithinLock at its false default — turning it on gains you nothing and adds lock hold time on every firing.

Creating the schema safely

The DDL scripts ship inside the Quartz jar at org/quartz/impl/jdbcjobstore/tables_<database>.sql — tables_postgres.sql, tables_mysql_innodb.sql, tables_h2.sql, and around twenty others. Extract the one for your database and run it once, through whatever migration tool you already use.

Read the top of that file before you automate running it:

DROP TABLE IF EXISTS QRTZ_FIRED_TRIGGERS;
DROP TABLE IF EXISTS QRTZ_PAUSED_TRIGGER_GRPS;
DROP TABLE IF EXISTS QRTZ_SCHEDULER_STATE;
DROP TABLE IF EXISTS QRTZ_LOCKS;
...
DROP TABLE IF EXISTS QRTZ_JOB_DETAILS;
DROP TABLE IF EXISTS QRTZ_CALENDARS;

Every bundled script begins by dropping all eleven tables. It is a create-from-scratch script, not an idempotent migration — re-running it destroys every persisted job, trigger and calendar in the cluster.

Which makes this Spring Boot configuration a data-loss bug rather than a convenience:

spring:
  quartz:
    job-store-type: jdbc
    jdbc:
      initialize-schema: always   # ← DON'T. Re-runs the DROP script every boot.

With always, every application start wipes the schedule — silently, and worst of all successfully. The correct production setup:

spring:
  quartz:
    job-store-type: jdbc
    jdbc:
      initialize-schema: never
    properties:
      org.quartz.scheduler.instanceId: AUTO
      org.quartz.jobStore.isClustered: true
      org.quartz.jobStore.clusterCheckinInterval: 7500
      org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate

Set initialize-schema: never and own the schema in Flyway or Liquibase, having removed the DROP statements from the migration. embedded is acceptable only for throwaway in-memory databases in tests.

A note on delegates: use the one matching your database, not the generic StdJDBCDelegate. PostgreSQLDelegate exists because Postgres needs different blob handling; MSSQLDelegate because SQL Server’s locking differs. Using the generic delegate on Postgres usually works until it meets a JobDataMap and then fails on blob retrieval.

Production gotchas

A checklist of the things that actually break clusters, roughly in order of how often they do.

isClustered is still false. The default. Every node schedules independently and every job fires once per node. If you’re reading this page because something double-fired, check this first.

Mismatched instanceName. Nodes with different scheduler names are separate clusters that happen to share tables. Same symptom as above, different cause — and it’s easy to introduce when the name is templated from a hostname or pod name.

Clock skew. Covered above, and the cause of the most confusing failures because the symptom (jobs recovered while still running) looks nothing like the cause.

@DisallowConcurrentExecution is cluster-wide, and that surprises people in both directions. It genuinely prevents the same JobDetail running on two nodes at once — often exactly what you want. But a job that hangs leaves its trigger BLOCKED, and it stays blocked until the execution is cleaned up, so one stuck run stops all future runs of that job across the whole cluster.

Thread pool exhaustion looks like misfires. threadCount is per node. When every worker is busy, triggers can’t be executed, drift past misfireThreshold, and get handled by the misfire policy instead. The logs say “misfire”, the actual problem is capacity — check QRTZ_FIRED_TRIGGERS depth against threadCount before tuning misfire settings.

Long transactions inside jobs stall the cluster. The TRIGGER_ACCESS lock lives and dies with a database transaction. A job that holds a long transaction — or a connection pool starved by one — delays lock release and therefore delays every node’s acquisition.

Serialised JobDataMap is a versioning hazard. JobDataMap contents are Java-serialised into QRTZ_JOB_DETAILS. Store a custom class in there and later change its shape, and deserialisation of already-persisted jobs breaks. Keep JobDataMap to primitives and strings; put an id in the map and load the object inside the job.

Rolling deploys mix Quartz versions. During a rolling restart, old and new nodes run against the same tables. Quartz’s schema is stable across 2.x patch releases, so this is usually fine — but it’s a real consideration on a major upgrade, where the safe path is scaling to zero rather than rolling.

If most of this list reads as more operational surface than you want, that’s a legitimate conclusion to draw. Quartz clustering is the right tool when you need durable jobs, JobDataMap parameterisation, calendar exclusions, or misfire policies inside the JVM. When you only need “run this once, even though three replicas are deployed”, moving the schedule out to a Kubernetes CronJob that invokes one endpoint — or keeping Spring @Scheduled on a single-replica deployment with a database advisory lock — is a great deal less machinery for the same guarantee.