Broker, scheduler, worker and storage
The parts of the analysis service and what each does.
ReferenceWorkflow animation
The analysis service is the part of the platform that runs methods over records. It is not one program but four cooperating parts: an API that records jobs, a scheduler that turns them into packages, two storage containers that hold packages and results, and a worker that runs each method in its own container. The workbench talks only to the API; the other parts communicate through what exists in storage. This page describes each part and how they fit. For the states a job passes through see Job lifecycle; for what comes back see Anatomy of a result package.
The parts
| Part | Where it runs | What it does |
|---|---|---|
Analysis API (bf-analysis-api) |
A web service | Records each job, answers status queries, serves result packages and reports, deletes jobs on cancel |
Scheduler (sync_analysis.py) |
A sidecar container beside the API, every 30 seconds | Packages waiting jobs, uploads them, and works out each job's current state |
| Queue and results storage | Two Azure Blob Storage containers, one for the queue and one for results | Hold job packages waiting to run and the result packages that come back |
Worker (execute_job.sh) |
The analysis host, on a schedule set on that server (documented as every minute) | Takes the oldest package, builds the method's container image, runs it, uploads the result |
The workbench is the only user-facing surface: the parameter form, the Analysis panel and the administrator's analysis dashboard all go through the API. See Platform architecture and data flow for how the service sits among the others.
The analysis API
The API is the system of record for jobs. When a job is submitted it creates a record in Redis under a short job ID, in state waiting, holding the job definition (the method's title and its parameters), the recordset code or set token that identifies the records, the record count the workbench computed, the submitter's API key and a snapshot of the submitter's username. The record expires 7 days after it is created.
There are two ways in. The workbench submits jobs on a stored recordset. The specimen and sequence ingest pipeline submits jobs with the records attached as a JSONL file, which the API stores compressed inside the job record (see Specimen and sequence ingest). Either way the caller authenticates with its API key; the platform's system token sees every job, a user's key sees only that user's jobs.
The API also streams a complete job's result package, extracts analysis_report.html from it for the in-browser preview, serves individual package files by name for the administrator's dashboard and the ingest pipeline, and deletes jobs on request. The endpoints are listed in the Analysis API reference.
The scheduler
The scheduler runs every 30 seconds with a lock so that passes never overlap. Each pass has two phases.
Packaging. For each waiting job, while fewer than 10 packages sit in the queue container, it fetches the job's records from the record store with the submitter's key (or decodes the attached records), applies the record selection made in the workbench, writes <title>.<id>/records.jsonl and params.json, bundles them as <title>.<id>.tar, uploads the package via BioStor, and marks the job queued. Records are fetched at this moment, not at submission.
Reconciliation. For every job that is not yet complete, it looks at what exists in storage and sets the state accordingly: a result package and its metadata sidecar mean complete, a failed package means failed, a package still in the queue means queued, and a package that has left the queue with nothing yet in results means running. The worker never reports anything; the scheduler infers it.
Storage
The two containers are the only link between scheduler and worker. Names follow the job:
| File | Container | Meaning |
|---|---|---|
<title>.<id>.tar |
queue | A packaged job waiting for a worker |
<title>.<id>.zip and <title>.<id>.metadata.json |
results | A completed job and its resource-usage sidecar |
<title>.<id>.failed.zip and <title>.<id>.failed.metadata.json |
results | A job whose container exited with an error |
The API reaches the containers through BioStor; the worker reads and writes them directly with the container command-line tool from azure-queue-observer. Both must be configured for the same containers.
The worker
Each run of the worker handles one package. It lists the queue, takes the oldest package by modification time, downloads it and removes it from the queue. The text before the first dot in the file name is the method name, which must match a directory in bf-analysis-jobs. The worker builds that directory's Docker image (on every run, so a deployed code change takes effect at the next job), then starts a container that runs the method's five-stage orchestrator on the package, with the job working directory mounted into the container.
While the container runs, the worker samples its CPU and memory every 10 seconds. When it exits, the samples are summarised into resource_stats.json, folded into the metadata sidecar together with an error field (null on success), and the whole job directory is zipped and uploaded with the sidecar. Each run writes a log and a last_run_summary.json on the analysis host.
Because runs are scheduled independently and each takes one package, several jobs can be executing at the same time when the queue is busy.
Good to know
Good to know: Jobs are not run in priority or submission order. The scheduler admits waiting jobs in the order it finds them, and the worker takes the oldest package in the queue.
Good to know: The scheduler is part of the service and runs wherever the API runs, including a developer's stack. The worker is a host-level scheduled task on the analysis host, so in a developer's stack it has to be run by hand (see Testing a method).
Good to know: The record count stored on the job is the one the workbench computed at submission; the records themselves are fetched when the package is built, so access or data changes in between are reflected in the job.