How data gets in
Records and evidence files take different routes; what happens after you upload.
ReferenceWorkflow animation
Everything you add to the workbench arrives by one of two routes. Records (specimens and their sequences) go into the record store, where they are searched, counted and analysed. Evidence files (photographs, chromatograms, PDF documents, genetic fingerprints and lab notebooks) go into BioStor, the platform's file store, and are linked to a record by its Process ID. The two routes are checked and processed differently, and knowing which one your upload takes tells you what to expect afterwards.
Why records and files are handled differently
A record is structured data. Every field has a defined type, many fields have a controlled vocabulary, and the taxonomy, geography and institution you enter must be resolved against the platform's reference lists before the record can be loaded. That work is done by an ingest job, and the result is a row in the database that other people can search and analyse. Writing a record is also permission-checked: the record store only accepts a load from someone with the right access to the project.
An evidence file is a binary object. The platform needs to know what kind of file it is (its profile), which record it belongs to, and some descriptive metadata, but it does not interpret the contents the way it interprets a specimen field. Files are stored as objects in BioStor and shown on the record through signed links. See Evidence files for what is and is not checked on the way in.
The record route
Three things travel this route:
| What | Where you start | How it is processed |
|---|---|---|
| A batch of specimens in a spreadsheet | Data Upload > Batch Specimen on a project | Checked up front, then an ingest job, then the database load |
| One specimen from a web form | Data Upload > Single Specimen, Research Record or COC Record | Loaded directly while you wait |
| Sequences in FASTA | Data Upload > Sequences | An ingest job, then the database load |
Every submission becomes a row in the submission queue. A processor runs once a minute and drains that queue. What happens next depends on the type.
Batch specimens. Your spreadsheet is checked before it is queued: the header row, the required columns, and whether each row's identifier exists and you may write to it. If any row fails, the whole file is rejected with row-level errors and nothing is queued. If it passes, the processor converts the rows to the platform's data model and sends them to the specimen-data-ingest job. On later passes it polls that job; when the job completes, the processor loads the prepared records into the record store. Expect this to take minutes rather than seconds, because the job runs on the analysis service's own schedule. See Batch specimen upload.
Single specimens. The web form skips the ingest job. The upload request runs the processor itself and waits up to 30 seconds for the record to load, then returns you to the project. See Adding or editing one specimen.
Sequences. The FASTA is parsed, each sequence is matched to its specimen, and the sequence-data-ingest job checks and prepares it (including a stop-codon report) before the load. Timing is the same as for batch specimens. See Uploading sequences.
The evidence-file route
Evidence files arrive either as a package (a .tar or .zip containing the files and a metadata.tsv that lists them and names their records) or one at a time through Enrich record on an open record.
Packages are queued and picked up by the per-minute processor, which unpacks the archive, reads the metadata file and sends each file (or the batch) to BioStor. When the files are stored, the record's evidence counts are refreshed and an activity-log entry is written. Large packages are sent in pieces so that the upload itself does not fail on size; see Uploading an evidence package.
Enrich record is synchronous: the file goes straight to BioStor while you wait, and the record view reloads with the new count. See Attaching a file to one record.
Following progress
Every queued submission appears in the uploads drawer in the top bar, with a status of queued, processing, success or failed and an error detail when something went wrong. The batch specimen page also has a table of that project's recent uploads. See Tracking your uploads.
Good to know
Good to know: A batch spreadsheet that fails its up-front checks never enters the queue, so it does not appear in the uploads drawer. The errors are shown on the batch page itself.
Good to know: Record writes are permission-checked against your project access. Evidence-file uploads are not checked against your access to the record you name, so take care that the Process ID in your metadata or on the Enrich form is the right one.