Graceful shutdown
When a pod stops, its workers should finish what they can and hand the rest back, so no job is lost or left waiting for a lease to run out. queue.close() does that, and the queue option signals calls it for you on SIGTERM.
This page sends itself SIGTERM while one short and one long job run, and shows what happens to each.
Example
import { setTimeout as sleep } from 'node:timers/promises'
import { MemoryStore, Queue } from '@sureq/core'
type Email = { to: string }
const store = new MemoryStore()
const queue = new Queue<Email>('emails', {
store,
signals: ['SIGTERM', 'SIGINT'],
shutdownTimeout: '500ms',
shutdownAbortGrace: '200ms',
})
const worker = queue.process(
async (job, ctx) => {
console.log('start', job.id)
try {
await sleep(job.id === 'quick' ? 100 : 60_000, undefined, { signal: ctx.signal })
} catch (err) {
console.log('stopped', job.id, ctx.signal.reason?.name)
throw err
}
console.log('done', job.id)
},
{ concurrency: 2 },
)
worker.on('error', ({ error }) => console.error('worker error', error))
queue.on('closing', () => console.log('closing'))
const closed = new Promise<void>((resolve) => queue.once('closed', () => resolve()))
await queue.add({ to: 'ada@example.com' }, { id: 'quick' })
await queue.add({ to: 'alan@example.com' }, { id: 'slow' })
await sleep(50)
// What your orchestrator sends when it stops the pod.
process.kill(process.pid, 'SIGTERM')
await closed
// The store is still open, so read the jobs back.
const reader = new Queue<Email>('emails', { store })
for (const id of ['quick', 'slow']) {
const job = await reader.getJob(id)
console.log(id, job?.state, job?.attempt)
}
await reader.close()
await store.close()Output:
start quick
start slow
closing
done quick
stopped slow ShutdownError
quick completed 1
slow waiting 0- On
SIGTERMthe queue callsclose(). Its workers stop claiming, and running handlers getshutdownTimeoutto finish.quickfinishes in time. - After that,
ctx.signalaborts withShutdownError, and handlers getshutdownAbortGracemore to stop. Then their jobs are released. - A released job goes back to
waitingat once, with its attempt refunded. Another worker picksslowup and it runs again from the start. - A second signal exits the process with code 1 without releasing jobs. Those jobs run again when their leases run out.
- SureQ never exits the process after a clean close. Close your other resources, such as an HTTP server, and Node.js exits by itself.
Common questions
What shutdownTimeout should I pick? Long enough for a typical job to finish, and short enough to fit your orchestrator's grace period. With the default settings close() can take about 28 s: 25 s of shutdownTimeout, 2 s of shutdownAbortGrace, and about 1 s for the releases. See Worker options: shutdown. That fits the Kubernetes default terminationGracePeriodSeconds of 30 s. A preStop hook counts against the same period, so with one, or with a longer shutdownTimeout, raise terminationGracePeriodSeconds above the total. A shutdownTimeout set on a worker overrides the queue's.
What if a handler ignores ctx.signal? Its job is released anyway, so it may run twice at once: once here, once on another worker. Pass ctx.signal to your I/O. See JobContext.
Can I close by hand instead of with signals? Yes. Call await queue.close() from your own shutdown code. close({ timeoutMs }) replaces shutdownTimeout for that call.