BioStor: what it stores and how files flow
Objects, metadata, Azure storage and the processing workers.
ReferenceWorkflow animation
BioStor is the file store behind BOLD Forensics. Every evidence file that is not a database record (specimen photographs, Sanger traces, sequence files, fragment-analysis files, PDFs, spreadsheets, audio recordings, tar archives and tabular datasets) is held by BioStor as an object: the file itself plus its metadata. The workbench, the analysis services and your own integrations all read and write files through it. This page explains what an object is, where the bytes live and how a file moves from upload to being ready to use.
What an object is
An object is one file and three layers of metadata:
| Layer | Holds | Where it comes from |
|---|---|---|
| Universal | File name, size, extension, MD5 checksum, owner, profile, is_public, processing status, timestamps |
Recorded at upload |
| File-type (by extension) | Technical statistics such as image dimensions, read counts, trace metadata, audio duration | Computed by post-processing |
| Profile (by purpose) | Domain fields such as process ID, document type, permit type, species, collection date | Supplied by the uploader, editable in the portal |
The file's extension chooses the technical pipeline that checks it and renders previews; its profile describes what the file is for and which domain fields it carries. The two are independent, so a PDF can be a permit, a chain-of-custody form or an analyst statement while passing through the same PDF pipeline. See File types, pipelines and size limits and Evidence profiles.
Object identifiers
Every object gets an identifier of the form biostor-{bucket}_{group}.{name}.{ext}, for example biostor-1_157.abc123def.jpg. The name part is a short random string, and ext is the canonical extension: .jpeg, .fas, .fa, .fq and .abi are accepted on upload but normalised to jpg, fasta, fastq and ab1 before the identifier is generated. The identifier is the only handle needed to fetch the original, a preview or the metadata.
Where files are kept
Metadata is stored in a PostgreSQL database. File content is stored in Azure Blob Storage, in a container chosen when the object is created and recorded with the object as its storage location. When you ask for a file, BioStor does not stream it; it answers with a redirect to a read-only signed link to the blob, so download bandwidth comes from storage rather than from the API. Links are valid for 24 hours by default. An MD5 checksum is computed while the upload is received and recorded with the object.
How a file flows
- Upload. The API checks the extension, the profile and the size limit, receives the file and computes its checksum. The universal, file-type and profile rows are written together with an entry in a processing outbox, in one database transaction, and the object identifier is returned.
- Outbox relay. A relay process publishes each committed outbox entry to a RabbitMQ
validationqueue. Because the metadata and the queue intent commit together, an object is never recorded without being queued, or queued without being recorded. The relay retries a publish up to 10 times. - Validation worker. A consumer runs the validation procedures bound to the extension (structural checks for images, PDF, XLSX, audio, tar and tabular files), copies the original to Azure, marks the object
validatedand hands it to thepostprocessingqueue. From this point the original can be downloaded. - Post-processing worker. A second consumer runs the post-processing procedures: statistics for the file type and renders such as thumbnails, trace graphs, spectrograms and tabular previews. These surrogates are uploaded beside the original, the object is marked
postprocessedand its status becomesready.
Each procedure runs as a separate program with a timeout and a critical flag. A critical failure stops the object at that stage; a non-critical failure is recorded against the object and processing continues. Accounts with a webhook secret receive signed validation.* and postprocessing.* events after each stage (BioStor webhooks). The full state sequence is in Object lifecycle and statuses.
Who uses it
- The workbench sends files on your behalf. Specimen images and traces uploaded from Enrich record go straight to BioStor under the
specimen_imageandchromatogramprofiles; document, FASTQ, microsatellite and spreadsheet uploads go as batch packages with a metadata manifest. When you open a file in the workbench, it looks the object up with its own service account and serves it through a link bound to your session. See Evidence files and Uploads queue. - The analysis service stores job packages and results through BioStor's container storage.
- Integrators use the REST API directly (BioStor API).
The portal
BioStor also has its own web portal for account holders. The Dashboard shows counts and upload trends; the Gallery is a thumbnail grid of recent objects filtered by extension or profile; Objects supports search, filters, CSV export and hand-off to another account; the Processing page lists pending and failed objects with the full error text and a requeue action; and Upload accepts single files or batch packages. Administrators get an additional console covering service health, users, API keys, failed objects, the audit log and reference pages for object types, procedures and profiles. Access rules are in Accounts and the access model.
Good to know
- An object identifier is returned as soon as the metadata is recorded and the file is queued. The original becomes downloadable when validation completes, and previews when post-processing completes.
- Replacing the file behind an existing identifier keeps the object's status and any existing previews.