Records, sets, search and summaries
What the record store holds, how searches match, how summaries are computed and cached.
Reference
The record store holds every specimen and sequence on the platform. When you open a recordset, run a search, read the statistics panel or draw a map, the workbench passes the request to the record store, which fetches the records, keeps only those you may read and computes the figures. Use this page to explain a number on the console to a reviewer or to understand why a search behaved as it did.
What the record store holds
A record is one specimen, identified by its processid. The laboratory's own sampleid can be used to find it instead. Each record carries the BCDM fields in the field dictionary: collection, specimen, taxonomy and forensic-extension fields, plus the sequence fields.
A record can hold one sequence per marker. The store reads records as one row per record and marker, so a record with a COI-5P and a rbcL sequence comes back as two rows sharing the specimen fields.
Alongside these sits a derived feature rollup per record and marker: counts of attached images, traces, FASTA and FASTQ files, documents, spreadsheets and genetic fingerprints; the flags coord_present, country_present and stopcodon_present; a compliance_level; and a features_updated timestamp. The coordinate and country flags are recomputed every minute on rows marked stale; a nightly job adds missing rows. compliance_level is stored with the record and passed through as held; the record store does not grade records.
How records are retrieved
Records are fetched by a list of process IDs, a list of sample IDs or a set token. Results stream as JSON Lines in batches of 5000 by default; a sorted variant returns one page ordered by any field, and a record can optionally carry its primer codes.
A set token is a saved list of process IDs, sample IDs, project codes, dataset codes or BIN URIs, stored under a token derived from its contents or a label the caller chooses. Reading a set returns the records you may read with the set's total and readable counts. Project pages, search results and analysis jobs all work from tokens. Treat them as short-lived: any specimen or sequence load clears every stored token, and the workbench rebuilds the set when you next open it.
Every record list, however addressed, is reduced to the records your account may read before it is returned; see How services authenticate to each other.
How term search matches
Term search runs against a separate search index, not the live record tables. The index holds a records table and a terms table of the names occurring in the taxonomy, geography (country, province, ecoregion, biome, realm and ISO code) and marker fields, each tagged with its field.
A search has two steps. First, each term you entered is resolved against the terms table; if any term is unknown, the search fails and names it. Second, the resolved terms become one query: values within a category are combined with OR (two taxonomy tags match either), and categories with AND (a taxonomy tag and a country must both hold). A marker can carry a sequence-length range, collection dates are a from-to range, and institution, recordset and tags narrow the result; public records can be excluded.
Matching is exact: Canis mesomelas matches that name and nothing else. There is no partial, fuzzy or free-text search, and free-text fields such as notes are not indexed. The result is saved as a search token, capped at 1,000,000 hits with a 120 s time limit; an identical search returns the existing token.
Because the index is built separately from the record tables, a record becomes searchable only when the index is next rebuilt, not when it is loaded. Retrieval by ID or token always reads the live tables.
Summaries
| Summary | What it reports |
|---|---|
| Set statistics | Record counts; breakdowns by phylum through species, country and province; coordinate and marker counts; coordinate and country completeness; compliance level as stored; latest dates |
| Canned completeness summary | Specimen view: species, specimens, coordinates, countries, barcodes and BINs. Sequence view: per-marker sequences, species and BINs. Optional bounding box |
| Categorical histogram | Value counts for any BCDM string, integer, number or date field, optionally grouped by a second field |
| Spatial histogram | Point counts on a degree grid or an approximately equal-area kilometre grid |
| Map image | A PNG world map with one dot per readable record that has coordinates |
Wherever a summary counts a barcode, it means a COI-5P, rbcL, matK, ITS or rbcLa sequence longer than 486 bp.
Every figure is computed over the readable records of the set when it is requested, then cached. The counts shown while browsing the taxonomy and geography consoles are platform-wide totals maintained separately; they are not filtered to your access.
Caching, from a user's point of view
Summary results are cached under a key that includes both the set and a hash of the recordsets you may read, so users with different access never see each other's figures. A cached aggregate lives for at least 30 s, longer for large results, so a figure can lag a change by that interval.
A specimen or sequence load refreshes the store deliberately: it drops the cached copies of the written records, clears all cached set statistics, histograms and tokens, and marks the project and dataset statistics view for refresh by a job that runs every minute.
Good to know
- Access is decided per record, not per marker. If you can read a record you see all of its markers.
- Search terms must be exact index names; use the typeahead on the forms to find the spelling the index holds.
- A set's total count can exceed its readable count; the console shows the readable records.