Workers

Platform → Workers is “what the background jobs have been doing, and which of them stopped.”

Guarded on audit.read; the platform-health panels additionally need platform.read, and a panel your account cannot read renders nothing rather than firing a request that would be refused.

Six tabs:

Tab	What it holds
Job runs	The run history over a window — see The run history. Its tab shows how many failed.
Last run per job	“The cheapest answer to ‘is this job alive’: one row, by name, with no window” — see The job lookup.
Outbox	Events not yet published — see The outbox.
Partitions	Partition cover — see below.
Stuck claims	Idempotency claims left open — see Stuck claims.
Key migration	“Migration 000043. Not a job: there is no run and nothing to trigger.” How many secret keys are still on the legacy Argon2id hash, and the server’s own note on what to do.

The same conditions are emailed by worker-health-alerts, when it is configured. The email’s Affected line names the jobs, outbox topics or idempotency endpoints involved, such as create-partitions ×3 (oldest 2d 4h). This page is where to read the rest. See docs/operations.md, under Health alerting, for the recipient and thresholds.

The run table is the least alarming thing on this page

“Did the jobs run” is answered by the table. The panels around it answer “what is quietly broken”, which is a different and worse question — and they are ordered by how long it takes for the answer to cost something.

Partition cover — the shortest fuse in the platform

There is no default partition anywhere in this schema

A partitioned table with no partition covering a date does not degrade on that date. Every insert into it fails.

On the transactions table, that is the platform refusing payments at midnight on the first of a month — with nothing in the days before to suggest anything was wrong, and with the one person who would have noticed asleep.

The panel names both halves of the fact: which table and which date. A list of table names with no date is a configuration note; a date attached to the transactions table is a countdown.

The check compares real date ranges in UTC (since migration 074). Before that it compared text in the connection’s time zone, so behind a non-UTC connection — a pooler such as pgbouncer — it reported every table missing and could miss a real gap. A panel full of tables on an older build was that bug, not an outage.

It checks the first of next month, not today — today having a partition is nearly always true and tells nobody anything. The date it checked for is shown, so the answer can be interpreted.

A non-empty list also means a job is broken. The hourly sweeper creates three months ahead for every partitioned table, so anything listed here means that job has not run or could not do its work. Creating the partition by hand fixes this month; find the job, or you will be back here on the 28th.

The run history

Filter	Values
Job name	e.g. expire-stale-transactions.
Window	Last hour, 6 hours, 24 hours, last week.

The default is a day and the maximum is a week, because the job-runs table is unpartitioned and the queue jobs write a row a second.

The newest runs come first, and Load more appends older ones. There are no page numbers: new runs land at the top while you read, so numbered pages would show the same rows twice. From the API (GET /v1/workers/jobs/runs), pass the response’s next_cursor back as before.

The lists of failing outbox events and stuck claims are paged.

Column	Notes
Job	—
Status	Including Running and Open and abandoned.
Started / Took	—
Records	“Zero is normal: most of these jobs find nothing to do on most runs.”
Instance	Which worker ran it.
Error	“Truncated to 1000 characters by the worker. A panic is recorded here.”
Run key	—

Zero records processed is normal and common — that is what a sweeper is. The column is deliberately unremarkable at zero, because a page where two thirds of the rows look like a problem is a page nobody reads.

Open and abandoned

A run with no finish time, on a job that takes forty milliseconds, means the process died. Which runs those are is a question answered by the platform, using its own threshold, rather than by comparing timestamps against your browser’s clock.

An abandoned run is worth investigating: whatever it was holding may not have been released.

The health check closes some of these itself. A run left open by a killed worker, on a job that runs on every replica, is marked failed once the same job has finished a run since, with the error “abandoned: the worker stopped before finishing this run; the job has run since”. Those runs held nothing. A run of a job that runs on one worker at a time is never closed this way, because it holds that job’s window, so it stays open and keeps alerting.

The outbox

This is the silent one

Every settlement projection, every merchant webhook and every notification leaves the platform through the outbox. The payment path writes the row and commits, and a separate relay publishes it.

That is the whole point of the pattern — and the reason its failure is invisible: payments keep working perfectly while nobody is paid, told or alerted.

The figure to read is the age of the oldest pending event, not the count. A row exists for the fraction of a second between the commit that wrote it and the publish that clears it, so a busy platform always has pending events:

Pending	Oldest age	Means
Many	Seconds	A busy platform, working.
Few	Hours	The relay is stuck.

A count threshold is almost useless here — a dead relay on a quiet afternoon accumulates a handful of rows an hour and would never trip one. The age trips at once. The panel reads the two together and says in words which situation it is in.

If the relay is stuck: that is an incident. Merchant webhooks, settlement projections and notifications are all stopped, and nothing else on the console will say so.

Stuck claims

One merchant, one refused retry each. Small in volume and worth clearing, because each one is a merchant whose event is not moving.

Creating cover by hand

The panel offers it for the table it names, because the sweeper having failed is not a reason to wait for the sweeper.

Field	Notes
How far ahead	“Three is what the hourly job does, and matching it means the next gap is a month further out rather than next week.”
Why this is being done by hand	Required. The useful text here is what you found wrong with the job, since that is the part somebody has to fix afterwards.

Creating the partitions removes the countdown. It does not fix the sweeper — the same gap reappears next month unless somebody looks at why the hourly job stopped.

The job lookup

A job name, with no window — for the job the partition panel sends you to. It is how you confirm whether a specific sweeper has run at all, rather than whether anything ran in the last day.

A morning check, in thirty seconds

1. Partition cover. Anything listed? Fix it, then find out why the sweeper did not.
2. The outbox age. Hours old is an incident.
3. Open and abandoned runs. Anything stuck?
4. Errors in the run table for the overnight jobs.

Everything else on this page is history.