Job lifecycle

States, who moves a job between them, timings, retention and failure.

Reference

A job has five states, and no single part of the analysis service owns all the transitions. The API sets the first state, the scheduler sets every other one by looking at what exists in storage, and the worker sets none. This page gives the states, what moves a job between them and how often, how long a job may take, what a failure looks like, and how long jobs and their files are kept. The parts themselves are described in Broker, scheduler, worker and storage.

States

State	Set by	Meaning
`waiting`	The API, at submission	The job is recorded; its records have not been fetched
`queued`	The scheduler, packaging phase	The package is in the queue container
`running`	The scheduler, reconciliation phase (inferred)	The package has left the queue and no result has appeared
`complete`	The scheduler, reconciliation phase	The result package and its metadata sidecar are in the results container
`failed`	The scheduler, reconciliation phase	A failed package is in the results container, or the time limit has passed

There is no cancelled state. A cancelled or deleted job is removed outright. all appears in status queries as a filter meaning "any state"; it is not a state.

Transitions

Transition	What triggers it
new → `waiting`	The API accepts the request, writes the job record with a 7-day expiry and adds the job ID to the submitter's list
`waiting` → `queued`	A scheduler pass finds fewer than 10 packages in the queue, fetches the records with the submitter's key, builds and uploads `<title>.<id>.tar`
`queued` → `running`	A scheduler pass finds the package gone from the queue and nothing yet in results
any → `complete`	A scheduler pass finds `<title>.<id>.zip` and `<title>.<id>.metadata.json` in results and can read the sidecar
any except `complete` → `failed`	A scheduler pass finds `<title>.<id>.failed.zip` and `<title>.<id>.failed.metadata.json`, or the time since submission exceeds the time limit
`failed` → `complete`	A result package appears after the time limit expired
any → removed	Cancel: a user removes a `waiting`, `complete` or `failed` job; an administrator can force-remove a job in any state, including its queued package

The API also offers a manual override that sets any state with the system token; nothing in the workbench uses it.

Cadence and timing

Scheduler: every 30 seconds, with a lock so passes do not overlap. Every job that is not complete is re-examined on every pass, including failed ones.
Worker: one package per run, on a schedule set on the analysis host (documented as every minute). Runs are independent of one another, so several jobs can execute at the same time, and jobs do not run in submission or priority order.
Time limit: 6 hours (21,600 seconds) from submission. Time spent waiting and queued counts towards it. A job in any state other than complete is marked failed once the limit passes, even if its container is still running; if a result then arrives it becomes complete.
last_updated on a job record is rewritten on every pass for every unfinished job. It records when the job was last checked, not when its state last changed.

How running is inferred

The worker never contacts the API. It removes a package from the queue before it builds the image and starts the container, and it uploads a result only when the container has exited. Between those two moments the scheduler sees no package and no result, and calls that running.

The consequence is that any run that produces no result at all, for example because the image fails to build or the method name in the package does not match a method directory, leaves the job showing running until its time limit, after which it is marked failed with no reason recorded. The package is not returned to the queue, so such a job has to be submitted again.

What a failure looks like

Inside the container, the orchestrator runs the five stage scripts in order. A stage that exits with a non-zero code stops the job: the orchestrator writes error.json holding the stage script's name (script) and everything the stage printed (error), and exits with the stage number, 1 to 5. A package missing params.json or records.jsonl exits with 404 before any stage runs.

The worker then zips the job directory as it stands, names it <title>.<id>.failed.zip, and writes <title>.<id>.failed.metadata.json. The sidecar holds the resource-usage summary and an error object: the contents of error.json plus a message giving the container's exit code. On the next scheduler pass the job becomes failed and the error is stored on the job record, where the status query returns it.

In the workbench the Analysis panel shows the state but not the reason, and a failed job's package cannot be downloaded from the panel. The administrator's analysis dashboard lists every job and can download queue and result packages, so the error and the partial output are available there (see Tracking jobs and reading reports).

Good to know: The message in the sidecar does not name the stage; it gives the exit code, which is the stage number. The script field names the stage script, and error holds its output.

Retention

Job record: expires 7 days (604,800 seconds) after creation. Updates do not extend it. After expiry the job disappears from the Analysis panel, from status queries and from the result endpoints. The panel lists the last 7 days, so a job is visible until its record expires.
Packages and sidecars: kept in storage until the job is removed. Removing a complete or failed job deletes its result files; an administrator's forced removal also deletes a queued package. Nothing else deletes them, so files belonging to an expired job record remain in storage but are no longer listed anywhere in the platform.
Worker host: each run leaves a log and last_run_summary.json on the analysis host.

Good to know: Two clocks govern a job. The 6-hour limit decides when it fails; the 7-day expiry decides when it disappears.