Events

Queues, workers and producers emit events. Listen with on, once and off.

const stop = queue.on('failed', ({ job, reason, error }) => console.log(job.id, reason, error.message))
worker.once('closed', () => console.log('worker stopped'))
stop()
  • on and once return an Unsubscribe: () => void.
  • Listeners run in the order they were added. A listener may be async.
  • A listener that throws or rejects never breaks SureQ. Its failure is reported as an error event with source: 'listener'. A failing error listener is only logged.
  • An error event with no listener is logged. Always listen for error.
  • J below is the job snapshot, a Job<D, R>, and R is the result type.

Who emits what

EventQueueWorkerProducer
addedyesyes
duplicateyesyes
rejectedyesyes
activeyesyes
progressyesyes
completedyesyes
retryingyesyes
failedyesyes
deadLetteredyesyes
leaseLostyesyes
stalledyes
drainedyesyes
cancelledyes
redrivenyesyes
cleanedyesyes
obliteratedyesyes
pausedyesyes
resumedyesyes
closingyesyes
closedyesyes
erroryesyesyes
storeDisconnectedyesyes
storeReconnectedyesyes
degradedyes
versionSkewyes
alertyes
scheduleFiredyes
scheduleSkippedyes
scheduleQueuedyes
schedulePausedyesyes
scheduleResumedyesyes
scheduleDeletedyesyes

A queue passes on these events of its workers: active, progress, completed, retrying, failed, deadLettered, leaseLost, drained and error. So queue.on('completed', ...) hears every worker of that queue object. paused, resumed, closing and closed are emitted by the queue and the worker each for their own change.

The event name types are keyof QueueEvents for a queue, keyof WorkerEvents for a worker and ProducerEvent for a producer.

Adding

EventPayloadWhen
added{ job: J; origin? }A new job was stored.
duplicate{ id: string; action: 'ignored' | 'replaced' | 'merged' | 'rejected'; existingState: JobState; by: 'id' | 'singletonKey'; origin? }An add met a job holding its id or singleton key. id is that job's id.
rejected{ id: string | null; reason: string; reasonCode?: string; operation: 'add' | 'replace' | 'merge' | 'schedule'; origin? }The queue filter refused an add. Nothing was stored. id is null when SureQ would have made the id.
queue.on('added', ({ job }) => console.log('added', job.id))
queue.on('duplicate', ({ id, action, by }) => console.log(id, action, by))
queue.on('rejected', ({ reason, reasonCode }) => console.log(reason, reasonCode))

Running

EventPayloadWhen
active{ job: J; workerId: string; origin? }A handler started on the job. In a batch, once per job when the handler starts.
progress{ job: J; progress: JobProgress; origin? }A ctx.progress value was stored. At most once per progressInterval per job.
completed{ job: J; result: R; durationMs: number; removed?: boolean; origin? }The job completed. removed: true when retention deleted it at once (resultGrace: 0).
retrying{ job: J; error: SerializedError; delayMs: number; runAt: number; origin? }The attempt failed and the job will run again at runAt (store time).
failed{ job: J; error: SerializedError; reason: FailureReason; removed?: boolean; origin? }The job failed for good. reason is 'attempts-exhausted', 'unrecoverable' or 'not-retryable'.
deadLettered{ job: J; reason: DeadLetterReason; origin? }The failed job was kept as a dead letter. Right after its failed event.
leaseLost{ job: J; token: number; reason: 'expired' | 'superseded' | 'not-active'; origin? }The worker lost the job's lease. Its outcome is thrown away; it is not reported as failed.
stalled{ job: J; previousWorkerId: string; crashCount: number; action: 'requeued' | 'failed'; origin? }The sweeper found an expired lease and put the job back, or failed it when no attempts or crashes were left.
drained{ queue: string; origin? }No job is left waiting, delayed or running.
queue.on('active', ({ job, workerId }) => console.log(job.id, 'on', workerId))
queue.on('progress', ({ job, progress }) => console.log(job.id, progress))
queue.on('completed', ({ job, result, durationMs }) => console.log(job.id, result, durationMs))
queue.on('retrying', ({ job, delayMs }) => console.log(job.id, 'retry in', delayMs))
queue.on('failed', ({ job, reason }) => console.log(job.id, reason))
queue.on('deadLettered', ({ job, reason }) => console.log(job.id, reason))
queue.on('leaseLost', ({ job, reason }) => console.log(job.id, reason))
queue.on('stalled', ({ job, action, crashCount }) => console.log(job.id, action, crashCount))
queue.on('drained', ({ queue: name }) => console.log(name, 'is empty'))

DeadLetterReason is 'exhausted' | 'unrecoverable' | 'retryRejected' | 'crashLimit' | 'legacy'. See Admin.

A retrying, completed or failed event can go missing when the store wrote the outcome but its answer was lost and the job moved on before the worker could confirm it. The stored job is still correct.

Managing jobs

EventPayloadWhen
cancelled{ job: J; previousState: 'waiting' | 'delayed' | 'active'; origin? }This queue object's cancel cancelled the job.
redriven{ id: string; state: 'waiting' | 'delayed'; viaDeadLetter: boolean; origin? }This object moved a failed job back. viaDeadLetter is false for a plain failed job.
cleaned{ state: CleanableState; removed: number; origin? }This object's clean removed jobs. Once per call that removed any.
obliterated{ removed: number; origin? }This object's obliterate finished.
queue.on('cancelled', ({ job, previousState }) => console.log(job.id, previousState))
queue.on('redriven', ({ id, state }) => console.log(id, state))
queue.on('cleaned', ({ state, removed }) => console.log(state, removed))
queue.on('obliterated', ({ removed }) => console.log(removed))

CleanableState is 'completed' | 'failed' | 'cancelled' | 'waiting' | 'delayed'.

Pause and close

EventPayloadWhen
pausedPauseEventThe queue (scope 'global') or the worker (scope 'local') was paused. Once per change.
resumedPauseEventThe same, for a resume.
closing{ queue: string; workerId?: string }close() began. A worker's payload has its workerId.
closed{ queue: string; workerId?: string }close() finished.
queue.on('paused', ({ scope, source }) => console.log('paused', scope, source))
queue.on('resumed', ({ scope }) => console.log('resumed', scope))
worker.on('closing', ({ workerId }) => console.log(workerId, 'closing'))
worker.on('closed', ({ workerId }) => console.log(workerId, 'closed'))

PauseEvent:

FieldTypeMeaning
scopePauseScope'local': this worker only. 'global': the whole queue.
queuestringThe queue name.
workerIdstring, optionalSet on a worker's events.
source'self' | 'observed''self' for this pod's own call; 'observed' when it learned the change from the store.
atnumberWall-clock ms when emitted.

A global change also fires on each worker of the queue object. Other pods see it from their claims, up to one pollMax later.

Store and pods

EventPayloadWhen
errorErrorEventA problem that is not a job failure, such as a store error or a failing listener.
storeDisconnected{ error: Error }The store became unreachable. Once per outage, on every queue sharing the store object.
storeReconnected{ downForMs: number; attempts: number }A probe reached the store again.
degraded{ error: StoreFormatError }The store moved to a format this pod cannot read. The pod stops claiming and sweeping and lets its active jobs finish.
versionSkew{ workers: { workerId: string; sureqVersion: string }[] }Live workers run a release that is not next to this pod's. A warning only.
alertAlertEventOne of this pod's alert rules started firing or resolved. See Health and metrics.
queue.on('error', ({ error, source, context }) => console.error(source, context.operation, error.code))
queue.on('storeDisconnected', ({ error }) => console.warn('store down', error.message))
queue.on('storeReconnected', ({ downForMs, attempts }) => console.info('store back', downForMs, attempts))
queue.on('degraded', ({ error }) => console.warn(error.code))
queue.on('versionSkew', ({ workers }) => console.warn(workers))
queue.on('alert', ({ type, status, value, threshold }) => console.warn(type, status, value, threshold))

ErrorEvent:

FieldTypeMeaning
errorSureQErrorThe error. Its code names it. See Errors.
sourceErrorSource'store', 'listener', 'internal' or 'strategy' (a backoff strategy).
context{ operation?: string; jobId?: string; workerId?: string; queue: string; event?: string }Where it happened. event is set for a failing listener.
originEventOrigin, optionalSee below.

Schedules

EventPayloadWhen
scheduleFired{ key: string; scheduledFor: number; jobId: string }This pod's scheduler made the job for a tick.
scheduleSkipped{ key: string; scheduledFor: number; reason: 'overlap' | 'backlog-full' }This pod's scheduler dropped a tick.
scheduleQueued{ key: string; scheduledFor: number }This pod's scheduler put a tick in the backlog (overlap 'queue').
schedulePaused{ key: string }This object paused a schedule.
scheduleResumed{ key: string }This object resumed a schedule.
scheduleDeleted{ key: string }This object deleted a schedule.
queue.on('scheduleFired', ({ key, scheduledFor, jobId }) => console.log(key, scheduledFor, jobId))
queue.on('scheduleSkipped', ({ key, reason }) => console.log(key, reason))
queue.on('scheduleQueued', ({ key, scheduledFor }) => console.log(key, scheduledFor))
queue.on('schedulePaused', ({ key }) => console.log(key, 'paused'))
queue.on('scheduleResumed', ({ key }) => console.log(key, 'resumed'))
queue.on('scheduleDeleted', ({ key }) => console.log(key, 'deleted'))

See Schedules.

Worker events

WorkerEvents is every queue event except added, duplicate, rejected, cancelled, stalled, redriven, cleaned, obliterated, storeDisconnected, storeReconnected, degraded, versionSkew, alert and the schedule events. A worker listener has the type Listener<D, R, K>.

const onFailed: Listener<{ to: string }, unknown, 'failed'> = ({ job, reason }) => console.log(job.data.to, reason)
worker.on('failed', onFailed)

Producer events

ProducerEvent is 'added' | 'duplicate' | 'rejected' | 'error' | 'redriven' | 'cleaned' | 'obliterated' | 'storeDisconnected' | 'storeReconnected' | 'schedulePaused' | 'scheduleResumed' | 'scheduleDeleted'. See Producer.

producer.on('added', ({ job }) => console.log('added', job.id))

EventOrigin

EventOrigin is { podId: string; workerId?: string }: the pod, and worker, an event came from. Every payload that has origin? may carry it. This release emits only events from its own pod and leaves origin unset.

Was this page helpful?