BF Workbench is a restricted scientific system. Records are available only to registered, authorised accounts.

From upload to loaded record

The submission queue, the ingest jobs, validation and the database load.

ReferenceWorkflow animation

Every specimen and sequence enters the database by one route: a submission queue in the workbench, a processor that drains it every minute, an ingest job that validates the data, and a database load in the record store that checks your permission and writes the rows. Use this page when a batch reports an error at a named stage, or when a reviewer asks what was checked before a record was accepted.

The submission queue

Each upload becomes one row in the submission_queue table with a status of queued, processing, success or failed. The row's submission_packet holds the detail: the analysis job spawned, the load outcome and, on failure, a last_error naming the stage (intake, preprocessing, analysis, db_load, processing or biostor_upload).

Batch specimen spreadsheets are checked before they are queued. The preflight confirms a header row, data rows, a sampleid column, a consistent New or Update layout and, for New files, the required columns sampleid, country/ocean, phylum and inst. The record store then gives a per-row verdict: in New mode, that the sample ID is not already in use; in Update mode, that the row resolves to exactly one record you may edit (with Restrict to this project on, one already in the project). Any failure rejects the whole file with row-level errors; nothing is queued.

The per-minute processor

Once a minute the workbench runs the submission processors in order: images, traces, documents, sequences, specimens, spreadsheets, then genetic fingerprints. The evidence-file processors send files to BioStor; the other two lead to a record.

  • Batch specimens. Every queued spreadsheet from the same submitter is normalised into one BCDM JSON Lines file and submitted as one specimen-data-ingest job; the packet records analysis_job_id, analysis_status and db_load_status: pending. When a later tick finds the job complete, the processor extracts db_load_ready.jsonl from the result, loads it and sets db_load_status to success or failed with a db_load_summary of processed, inserted, updated and failed counts.
  • Sequences. The FASTA is parsed, each sequence is matched to its record by sample ID or process ID and given that record's taxonomy, and one sequence-data-ingest job is submitted per submission, then polled and loaded the same way.

What the ingest jobs validate

Both jobs run on the analysis service through the standard five stages. Neither writes to the database; their product is db_load_ready.jsonl.

specimen-data-ingest

  1. Acceptability: each field is checked for data type and format against the BCDM field definitions; sampleid, bold_recordset_code_arr, country/ocean, phylum and inst must be present; sample IDs and process IDs must be unique within the file.
  2. Controlled vocabularies: taxonomy and geography are checked against their registries, and vocabulary fields against the controlled vocabularies. An unrecognised value is dropped from the record, which is kept and loaded without it.
  3. Field-group filter, then conversion from BCDM to the database layout, including verbatim fields.
  4. Load preparation: taxonomy, geography, institution and primer names are resolved to database identifiers using registries the record store exports nightly.

sequence-data-ingest

  1. Records missing marker_code or nuc are rejected, and the rest are converted to the database layout.
  2. A stop-codon report is produced and merged into the load as stopcodon_present. It runs only on protein-coding markers; the translation matrix (nuclear or mitochondrial) is chosen from the record's taxonomy; forward frames 1, 2 and 3 are tried. A sequence is flagged only when all three forward frames contain a stop codon. The reverse strand is not checked, and a record whose taxonomy or matrix cannot be resolved is reported as unresolved.

The database load

The record store's db-load upsert takes db_load_ready.jsonl and applies it in one all-or-nothing transaction.

Before writing it checks the submitter's permission on every target project and dataset: a level that allows editing specimens (Project Manager, Edit All or Edit Specimens/Read Sequences on a project; Dataset Manager on a dataset); Super Users pass. For an existing record, at least one of its current recordsets must pass. The check uses the submitting user's own key.

For specimens, each row is matched by process ID or sample ID. If nothing matches, a new record is created and its process ID is minted from the project code, a counter and the year. The load writes the specimen, location and locality fields plus the project and dataset links, and creates the record's feature rollup row marked stale. For sequences, the record must already exist and the marker code must resolve; the sequence is inserted or replaced for that record and marker, and the first marker loaded takes over the record's initial rollup row.

After commit the store recomputes the coordinate and country flags, clears its caches and tokens (see Records, sets, search and summaries), marks the project and dataset statistics for refresh and writes New- and Modify- events to the activity log.

How a web-form specimen differs

A specimen entered on the Single Specimen, Research Record or Chain-of-Custody form is queued in immediate mode: the upload request runs the processor itself and waits up to 30 s for the result. The fields pass through the same normalisation as a spreadsheet row and the same db-load upsert with the same permission check, but no ingest job runs. Required fields and the BCDM validation rules are checked in the browser; the server requires only a sample ID or process ID, and the record store does not validate against the vocabularies or validation rules.

Good to know

  • Unrecognised taxonomy, geography and vocabulary values do not reject a record. They are removed and the record loads without them.
  • Spreadsheets queued by one submitter within the same minute become one job and one load; an error in one file fails the group.
  • New or Update mode and Restrict to this project are enforced when the file is accepted, not again at load time.
  • The processor waits for an ingest job for as long as it takes; there is no timeout on processing.