Platform architecture and data flow
The workbench, the record store, the analysis service and BioStor, and the stores they share.
ReferenceWorkflow animation
BOLD Forensics is not one program but four services that share a small number of data stores. The workbench is the only one people use directly; the others hold the records, store the files and run the analyses on its behalf. This page explains what each service is responsible for, how data moves between them, and where the shared stores sit, so that the rest of this layer can be read with the whole picture in mind.
The four services
| Service | Responsible for | Talks to |
|---|---|---|
| Workbench | Sign-in and sessions, every page and dialog, upload intake, job submission, statements, platform administration | All three other services |
| Record store (the data API) | Specimen and sequence records, record sets, search, taxonomy and geography reference lists, summaries, and the decision of which records each account may read | PostgreSQL, Redis |
| Analysis service | Recording analysis jobs, packaging the records each job needs, queueing packages for execution, holding results | Redis, Azure storage, the record store, BioStor |
| BioStor | Receiving, checking, processing and storing evidence files, and serving them back | PostgreSQL, Azure storage, RabbitMQ |
Each service runs in its own container and can be updated on its own. The workbench reaches the others over HTTP with a service key for each; it never reads their databases directly. See How services authenticate to each other.
How data moves
Records. A specimen or sequence upload becomes a row in the workbench's submission queue. A processor drains the queue once a minute, hands batch uploads to an ingest job on the analysis service, and loads the prepared records into the record store, which checks that the account may write to the project before accepting them. Searches, counts and summaries are then answered by the record store from its own tables and caches. See From upload to loaded record and Records, sets, search and summaries.
Evidence files. A file attached to a record goes to BioStor, either straight from the Enrich form or unpacked from a package by the same per-minute processor. BioStor records the file's metadata and checksum, copies it to Azure storage, runs its validation and post-processing workers, and keeps the record's process ID so the workbench can find the file again. When the workbench shows an image, a trace or a document, it asks BioStor for a short-lived link to that one object. See BioStor: what it stores and how files flow.
Analyses. Submitting an analysis records a job with the analysis service. A scheduler fetches the job's records from the record store, packages them with the parameters, and queues the package in Azure storage. A worker on the analysis host takes packages from that queue, builds the method's container and runs its five stages, then uploads the result package. The workbench reads the job's status from the analysis service and fetches the report and package from it. See Broker, scheduler, worker and storage.
Statements. Statements are produced and stored by the workbench itself, from the record, the deponent's profile and a template; the signed scan goes to BioStor like any other document.
The shared stores
| Store | Holds |
|---|---|
| PostgreSQL | Separate databases for the workbench (accounts, projects and datasets, submissions, statements), the record store (records and the search index), curation, and BioStor (object metadata) |
| Redis | Sessions, sign-in throttling, invitation and reset tokens, the record store's caches (per-account access lists, record and summary caches, search results), and the analysis job queue |
| Azure Blob Storage | Evidence file objects, queued analysis packages, and analysis result packages |
| RabbitMQ | BioStor's work queue for validation and post-processing |
Two consequences of this layout are worth knowing. First, caches mean that some numbers lag: summaries on a console and an account's access list are refreshed on a schedule or on specific events, not on every change, which is why Access, logging and retention describes delays. Second, files and records live in different stores and are linked only by the process ID, so an evidence file whose process ID is wrong is stored correctly but attached to the wrong record.
Reference data
Three reference lists are shared by every service. The Barcode Core Data Model and its forensic extension define the fields of a record, their types, the controlled vocabularies and the validation rules; the record store and the ingest jobs load the same definition files. The taxonomy and geography registries live in the record store and are what uploads are resolved against; administrators maintain them through the workbench. See the field dictionary and controlled vocabularies.
Scheduled work
Several things happen on a timer rather than when a user acts:
- The submission processor runs once a minute in the workbench.
- The analysis scheduler checks for new jobs every 30 seconds; the execution worker takes one queued package each time it runs.
- The record store rebuilds summary views and the activity report on its own schedule (the activity report hourly).
- BioStor's workers run continuously, taking work from the queue as it arrives.
This is why an upload, a job or an activity entry can take from a few seconds to several minutes to show, and why the documentation gives those timings where it can.