Specimen and sequence ingest

The validation each batch passes before loading, including stop-codon checks.

Reference

Two internal jobs, specimen-data-ingest and sequence-data-ingest, validate and transform every batch upload before anything reaches the database. You do not launch them from the Analysis menu; the upload queue launches them. The jobs write nothing to the database themselves. Each produces a load-ready file that the workbench then submits to the data service, as described in From upload to loaded record.

Where they fit

The queue normalises a batch specimen upload or a sequence upload into BCDM records and submits the matching job; the job validates, converts and resolves the records and packages db_load_ready.jsonl with a validation report; the queue posts that file for loading.

Both jobs judge records one at a time: a rejected record does not stop the batch, so a batch can load partially. Neither job has a form; the settings below are fixed by the job definition.

Specimen ingest

Acceptability

Required fields. sampleid, bold_recordset_code_arr, country/ocean, phylum and inst must be non-empty; a record missing any of them is rejected.
Unknown fields. A field not in the BCDM field definitions is removed with a warning; the record is kept.
Types and formats. Each value is checked against its definition: integers, numbers, dates in the defined format, coordinate pairs, lists and JSON. A single-character field is truncated to its first character rather than rejected, so Male becomes M. A record with any value that fails its check is rejected.
Uniqueness. sampleid and processid must each be unique within the batch; every record sharing a duplicated value is rejected.

Controlled vocabulary

Vocabulary fields are checked against the controlled vocabularies; taxonomy ranks from phylum to subspecies against the taxonomy registry, ignoring case; and country/ocean and province/state against the geography registry, ignoring case, spacing, slashes and dashes.

Good to know: A value that is not in the vocabulary does not reject the record. The field is dropped from that record, with a warning in the job log, and the record continues without it. A misspelt species name therefore loads as a record without a species, not as an error.

A field-group filter then keeps only the specimen, taxonomy, collection, record organization, forensic extension and forensic extension advanced groups; sequence, feature rollup and individualization extension fields are removed.

Conversion

Each BCDM field is mapped to its database table and column. A value that does not fit the column's type or length is dropped with a warning; an empty string becomes null. The submitted text of mapped fields is also kept in verbatim columns, so your wording survives resolution.

Resolution

Taxonomy. If identification matches a registry name, the candidates are narrowed by each rank you supplied. Otherwise the lowest rank you supplied is looked up, and the match is accepted only when your parent ranks agree with the registry's lineage (at least two supporting parents, or all the way to phylum). An unresolved taxonomy drops the link; the record still loads with its text ranks.
Geography. province/state is matched within country/ocean (by name, ISO-2 or ISO-3 code), falling back to the country alone.
Institutions and primers. inst, the biobank institution, the collector's affiliation and the sequencing site are matched by name, code, registered identifier (ROR, GRSciColl, GGBN, ISNI) or alias; primers by name or code. If a registry is unavailable, these are left for the loader.

The resolved record is grouped by database table into db_load_ready.jsonl.

Sequence ingest

Required fields. marker_code and nuc; a record missing either is rejected. There is no uniqueness or vocabulary check.
Conversion. The same field mapping and column-type checks as specimens; invalid fields are dropped. Linked fields such as primers and sequencing site are not resolved in the job but carried with their loading policy for the loader.
Stop-codon check. Runs on every record whose marker is flagged protein-coding in the marker registry: 1. The marker's genome (nuclear or mitochondrial) is taken from the registry. 2. The record's taxonomy is matched to the taxonomy registry, by identification or by its lowest rank, with every supplied rank required to agree. The matched row names the translation table for that taxon and genome. 3. The sequence is upper-cased, with U read as T, and translated in forward frames 1, 2 and 3. A codon containing any non-ACGT character is read as unknown (X), never as a stop. 4. The first frame without a stop codon gives the status no_stop_codons; if all three frames contain a stop, has_stop_codon. The reverse strand is not examined. 5. If the genome, the taxon or its translation table cannot be resolved, the status is unresolved and analysis_note says why.

The result is protein_coding_stop_codon_report.tsv (processid, sampleid, marker_code, marker_genome, translation_matrix_id, translation_matrix_label, selected_reading_frame, stop_codon_status, analysis_note). For each resolved record, record_feature_rollup.stopcodon_present is set to true or false in db_load_ready.jsonl, keyed by processid and marker_code. Unresolved records get no flag. If the batch has no protein-coding marker, no report is written.

Outputs

File	Content
`db_load_ready.jsonl`	One load-ready object per accepted record, grouped by table. This is what gets loaded.
`validation_result.json`, `validation_report.md`	Totals, `validation_status` (`valid`, `partial` or `invalid`), issue counts and file names
`validation_errors.jsonl`, `rejected_records.jsonl`	Per-record rejections and their reasons
`accepted_records.jsonl`, `submission_ready.jsonl`	The accepted records before and after conversion
`protein_coding_stop_codon_report.tsv`	Sequence ingest only, when a protein-coding marker is present
`analysis_report.html`, `analysis_report.pdf`	The standard report; its "filtered records" count is always 0 for ingest jobs

Failure modes

The whole job fails when the input is empty, the parameters are malformed, or (for sequence ingest) a registry file is missing; the upload is marked failed with the stage's error.
A record is rejected for a missing required field, a duplicate sampleid or processid, or a value that fails its type or format check; it appears in rejected_records.jsonl.
A field is dropped for an unrecognised vocabulary term, a value that does not fit its column, or an unresolved taxonomy, geography, institution or primer; the record loads without it. Drops appear only in the job log, so compare the loaded record with your upload when a field you supplied is missing.
Counts can disagree. The accepted and rejected counts come from two separate checks at different stages, so they need not sum to the total.