BF Workbench is a restricted scientific system. Records are available only to registered, authorised accounts.

Object lifecycle and statuses

From upload to processed, and what each status means.

Reference

Every BioStor object carries a status that records how far it has travelled from upload to ready for use. Knowing the states tells you when the original can be downloaded, when previews exist, and what a failure means and who can act on it. This page follows one object through the pipeline described in BioStor: what it stores and how files flow.

The states

Status Meaning What you can do with the object
pending_validation Accepted and queued; the validation worker has not finished with it Nothing yet; previews are served as stand-ins
pending_postprocessing Validated and copied to storage; statistics and renders are being produced Download the original
ready Fully processed Download the original and every surrogate
validation_failed A critical validation procedure failed, or validation timed out after three attempts Read the error; requeue or delete
postprocessing_failed A critical post-processing procedure failed Download the original; read the error; requeue or delete
dismissed An administrator has closed a failed object without fixing it Download the original if it had been validated

Two flags accompany the status: validated is set when the validation stage completes and postprocessed when post-processing completes.

What moves an object

  1. Upload sets pending_validation. The metadata rows and an outbox entry are committed together, and the identifier is returned to the caller.
  2. The relay queues it. The outbox relay publishes the entry to the validation queue, retrying up to 10 times.
  3. Validation completes. The validation procedures bound to the extension run, the original is uploaded to Azure, validated is set, the status becomes pending_postprocessing, a validation.complete webhook fires and the object is passed to the post-processing queue.
  4. Post-processing completes. The statistics and render procedures run and their outputs are uploaded as surrogates. postprocessed is set, the status becomes ready, a postprocessing.complete webhook fires and the working copy is removed.

Each stage runs every procedure in its group as a separate program with a timeout. A critical procedure (every validator and every statistics step except calculate_audio_stats and calculate_tar_stats) stops the stage when it fails. A non-critical procedure (renders, thumbnails, spectrograms, previews) records its failure as a derived_<procedure>_status field and the stage continues, so an object can be ready without one of its previews.

What happens on failure

When a critical procedure fails, the object's status becomes validation_failed or postprocessing_failed, its processing_attempts counter increases, the error text is stored in last_error, and a validation.failed or postprocessing.failed webhook fires. Validation itself is attempted three times, after waits of 2 s and 10 s, before the object is marked failed.

A failed object can be moved back to the pending state of the stage that failed:

  • Requeue from the portal's Processing page, by the owner or an administrator. This writes a new outbox entry using the uploaded file; if the file can no longer be found the requeue is refused.
  • Retry from the administrator's Failed objects page, which does the same from the failed stage.
  • Automatic requeue. A nightly job at 02:00 requeues every failed object that is more than 6 hours old and has fewer than 5 processing attempts.

An administrator can instead dismiss a failed object. Its status becomes dismissed; the metadata and any stored files remain, and the nightly job leaves it alone.

Where to see the status

  • In the portal, the Processing page lists your pending and failed objects, with the full error text for each failure. The Objects list can be filtered by status.
  • Through the API, the profile listing (GET /api/profile/{name}/objects) returns status and postprocessed for each object. The general metadata endpoint does not include status.
  • Webhooks deliver the four stage events to accounts that have a webhook secret (BioStor webhooks).
  • Polling GET /api/objects/{id}?url=true returns a link once the original is available, which is the simplest readiness check for a script.

Objects uploaded through the workbench are tracked there as well; see Uploads queue.

Deleting

The API deletes an object in any state, subject to the ownership rules in Accounts and the access model. In the portal, ready objects are deleted from the Objects list, objects in the four processing states from the Processing page, and administrators can bulk-delete ready objects. Deletion removes the database rows first, then the original and its surrogates from storage.

Good to know

  • The identifier is returned when the upload is accepted, not when the file is ready. If you need the original or a preview, wait for pending_postprocessing or ready, or subscribe to the webhooks.
  • A stand-in is served for a surrogate requested before post-processing finishes: the original for images, audio, archives and tabular files, a placeholder image for the other families.
  • Replacing the file behind an existing identifier does not restart processing. The status stays as it was and earlier surrogates remain until the object is requeued.
  • dismissed objects cannot be deleted from the portal; use the API.