Dead letters and redrive
A job that fails for good is kept as a dead letter, with its data, its error and its attempt history. Once you fix the cause, you redrive it: the job goes back to waiting and runs again.
This page lets two jobs fail while the mail server is down, lists them, then redrives them.
Example
import { Queue } from '@sureq/core'
type Email = { to: string }
const queue = new Queue<Email>('emails', {
defaultJobOptions: { maxAttempts: 2, backoff: '100ms' },
})
let smtpUp = false
const worker = queue.process(
async (job) => {
if (!smtpUp) throw new Error('smtp down')
console.log('sent', job.data.to)
},
)
worker.on('deadLettered', ({ job, reason }) => console.log('dead letter', job.id, reason))
worker.on('error', ({ error }) => console.error('worker error', error))
const drained = () => new Promise<void>((resolve) => queue.once('drained', () => resolve()))
let idle = drained()
await queue.add({ to: 'ada@example.com' }, { id: 'ada' })
await queue.add({ to: 'alan@example.com' }, { id: 'alan' })
await idle
console.log('count', await queue.countDeadLetters())
const { jobs } = await queue.getDeadLetters({ limit: 10 })
for (const job of jobs) console.log(job.id, job.state, job.deadLetter?.reason, job.error?.message)
// The cause is fixed. Put the dead letters back.
smtpUp = true
idle = drained()
const summary = await queue.redriveAll()
console.log('redriven', summary.redriven, 'done', summary.done)
await idle
console.log('count', await queue.countDeadLetters())
const ada = await queue.getJob('ada')
console.log(ada?.state, ada?.redriveCount, ada?.lastError?.message)
await queue.close()Output:
dead letter ada exhausted
dead letter alan exhausted
count 2
ada failed exhausted smtp down
alan failed exhausted smtp down
redriven 2 done true
sent ada@example.com
sent alan@example.com
count 0
completed 1 smtp down- A dead letter is a
failedjob withjob.deadLetterset. Itsreasonsays why it failed for good: hereexhausted, because both attempts were used up. deadLetteredfires right afterfailed, once per job.getDeadLetterslists the oldest first, a page at a time.countDeadLetterscounts them. Both take areasonfilter.redriveAllmoves every dead letter back towaiting. It resets the attempt count by default, so each job gets its fullmaxAttemptsagain.- A redriven job keeps its id and its history.
redriveCountgoes up by one, andlastErrorkeeps the error from before the redrive.
Common questions
Can I redrive one job? Yes. queue.redrive(id) or job.retry(). Pass a list of ids to redrive for several.
Can I redrive later instead of now? Yes. Pass { delay: '5m' }, and the jobs wait as delayed first. See Redrive for priority and resetAttempts.
What can redriveAll filter, and what does it skip? Pass { reason, olderThan } as its first argument to pick dead letters, and limit and pageSize with the other options. It takes dead letters only, oldest first, and passes by superseded ones, whose id a newer job holds. Use redrive(id) for a failed job that is not a dead letter. It is not atomic: if it stops part way, the jobs already moved stay moved, so run it again for the rest. See Redrive.
What reasons can a dead letter have? exhausted, unrecoverable, retryRejected, crashLimit or legacy (a pod of the previous release failed it). A worker, as in the example, hears only the dead letters it makes. Those the queue's sweeper makes, such as every crashLimit one, reach only the queue, so listen with queue.on('deadLettered', ...) to hear both. A legacy dead letter fires no event in this release. See Admin: dead letters.
How long are dead letters kept? As long as deadLetter.keep says; see Queue options: deadLetter. clean('failed', ...) deletes them too.
How do I know when they pile up? health() warns with DEAD_LETTERS_HIGH past deadLetter.warnAbove. See Health and metrics.