Producer

A Producer is the add side of a queue, for a process that never runs jobs: an API server, a serverless function, a command-line tool. It adds, reads and manages jobs in a shared store. It runs no worker, sweeper, scheduler, heartbeat, alert loop or timer.

import { Producer } from '@sureq/core/producer'

const emails = new Producer<{ to: string }>('emails', { store, namespace: 'shop' })
await emails.add({ to: 'ada@example.com' })
  • new Producer<D, R>(name, opts). D is the job data type, R the result type. The name rules are the same as for a Queue.
  • Import it from @sureq/core/producer. That entry loads none of the worker, sweeper, scheduler, batching or alert code. Producer is also exported from @sureq/core; both entries share one copy of every class.
  • A producer needs a store that worker pods also use. Without one it throws InvalidOptionError.
  • After ready() and after every call, nothing is left running, so a frozen and resumed function costs nothing between calls.
  • close() only marks the producer closed. The store you passed stays open.
  • With no namespace, it logs one info line saying it uses 'default'.

Producer options

ProducerOptions<D> is the add side of Queue options, plus a required store. Any other queue option, such as alerts, metrics, deadLetter, signals or leaseDuration, throws InvalidOptionError.

NameTypeDefaultMeaning
storeStorerequiredThe store the worker pods use.
namespacestring'default'Must match the worker pods' namespace.
id(data: D) => stringnone: a UUIDv7 per jobMakes the job id from the data.
onDuplicateDuplicateRule<D>'ignore'What add does when the id is taken.
uniqueWhile'pending' | 'retained''pending'How long a job holds its id.
defaultJobOptionsJobDefaultsnonemaxAttempts, backoff and priority for every add.
order'fifo' | 'lifo''fifo'Claim order among jobs of equal priority.
maxPayloadBytesnumber1048576 (1 MiB)Largest job data, in UTF-8 bytes of its JSON.
maxBulkSizenumber1000Most inputs in one addBulk call.
filterFilterFn<D>noneChecks every add before it is stored.
filterTimeoutMsnumber5000Longest the filter may take, in ms.
loggerLogger | falseconsole at warnWhere log lines go. See Logging.
reconnectReconnectOptionssee reconnectHow the store is probed while it is unreachable.
onDisconnected'reject' | 'wait''reject'What add does while the store is unreachable.
maxWaitDuration'10s'Longest an add waits in 'wait' mode.
maxBufferedAddsnumber1000Most adds waiting in 'wait' mode.

Each option means the same as on a queue; the Queue options tables give the details.

new Producer('emails', { store })
new Producer('emails', { store, namespace: 'shop' })
new Producer<{ to: string }>('emails', { store, id: (data) => `welcome-${data.to}` })
new Producer('emails', { store, onDuplicate: 'replace' })
new Producer('emails', { store, uniqueWhile: 'retained' })
new Producer('emails', { store, defaultJobOptions: { maxAttempts: 5 } })
new Producer('emails', { store, order: 'lifo' })
new Producer('emails', { store, maxPayloadBytes: 64 * 1024 })
new Producer('emails', { store, maxBulkSize: 500 })
new Producer<{ to: string }>('emails', { store, filter: (data) => data.to.includes('@') })
new Producer('emails', { store, filter: async () => true, filterTimeoutMs: 1000 })
new Producer('emails', { store, logger: false })
new Producer('emails', { store, reconnect: { maxDelay: '1m' } })
new Producer('emails', { store, onDisconnected: 'wait' })
new Producer('emails', { store, onDisconnected: 'wait', maxWait: '30s' })
new Producer('emails', { store, onDisconnected: 'wait', maxBufferedAdds: 5000 })

A producer has no storeCallTimeout option and uses the default '30s'. The first queue or producer on a store object sets the call timeout and the reconnect policy for that store, so a producer created first fixes them at its values.

Methods

A producer has the same methods as a queue, minus the ones that run jobs. Each one behaves as on a Queue.

GroupMethods
Addingadd, addBulk
ReadinggetJob, getJobs, getCounts, getAttempts, getOldestWaitingAge, getStoreTime, getDeadLetters, countDeadLetters, getWorkers, metrics
Controllingcancel, remove, redrive, redriveAll, pause, resume, pauseState, isPaused
Adminclean, drain, obliterate
SchedulesupsertSchedule, getSchedule, listSchedules, pauseSchedule, resumeSchedule, deleteSchedule, nextRuns, previewSchedule
Life cycleready, health, close
Eventson, once, off

It has no process, processBatch, startScheduler, stopScheduler or getAlerts. Its properties are name, namespace, closed, connectionState and formatVersion.

const { job } = await emails.add({ to: 'ada@example.com' }, { maxAttempts: 5 })
console.log((await emails.getJob(job.id))?.state)
await emails.upsertSchedule('digest', { cron: '0 7 * * *', template: { data: { to: 'team@example.com' } } })

What a producer does not do

  • It never fires schedules. A schedule it creates fires where a worker of that queue runs the scheduler, or where startScheduler() was called. See Schedules.
  • It never sweeps expired leases, trims finished jobs or checks alerts. Those run on worker pods.
  • getWorkers() never lists it. health() shows no workers and no alerts.
  • metrics() holds sureq_jobs, sureq_oldest_waiting_age_seconds, and this process's sureq_jobs_added_total (and sureq_jobs_rejected_total once its filter rejected an add). See Health and metrics.
  • cancel of a running job uses the default cancelled retention policy and the job's own keep, because the producer cannot see the worker pods' options.

Events

A producer emits only ProducerEvents: added, duplicate, rejected, error, redriven, cleaned, obliterated, storeDisconnected, storeReconnected, schedulePaused, scheduleResumed and scheduleDeleted. Payloads are in Events.

emails.on('rejected', ({ reason }) => console.warn('add rejected:', reason))

Types

  • Producer<D, R> extends QueueBase<D, R, ProducerEvent>, the base it shares with Queue.
  • Producer and Queue both implement QueueClient (add, addBulk, ready, close, health). Type code that only adds jobs against it. See QueueClient.
  • @sureq/core/producer exports the producer, every error class, the store contract types, serializeError, deserializeError, describeMetrics and errorCatalogue. It does not export Queue, Worker, BatchWorker and their handler and option types, MemoryStore, consoleLogger, pinoLogger, isUnrecoverable, defineCapabilities or the contract version constants: import those from @sureq/core.

Was this page helpful?