Barcode gap

Barcode Gap asks, for each species in a recordset, whether its sequences are more similar to each other than to the nearest sequence of any other species. The job computes the largest distance within each species, finds the smallest distance from that species to a neighbour, and reports the difference. A positive difference is a barcode gap; zero or a negative value means the species overlaps its neighbour.

Inputs

Records of the chosen marker that pass the shared filters. For this method the length window is fixed at 200 to 700 A, C, G and T bases regardless of the form (see the parameters table). The convert stage writes each record to FASTA with a header of record ID, phylum, class, order, family, subfamily, genus and species.

The distance is computed column by column, so the sequences must be aligned to a common length. Choose MAFFT under "Align Sequences" unless the stored sequences are already aligned.

Parameters

The report lists these parameters as submitted, so a report can read "Pairwise Distance" above K2P values and show a length filter that was not used.

Method

1. Sanitise. Each sequence is upper-cased, and every character other than A, C, G or T becomes a gap.
2. Distance. For each pair, positions with a gap in either sequence are skipped (pairwise deletion). Among the remaining positions the job counts transitions and transversions and computes the K2P distance as a percentage, rounded to two decimal places. A pair whose differences exceed the model's domain (saturation) gets an infinite distance and is left out of every statistic. A pair with no comparable positions gets 0.0.
3. Grouping. Records are grouped by species name. A record takes part only if it has a species, a genus and a family.
4. Within-species distances. All pairs within each species are computed, with no cap. They are split into pairs that share a process ID (replicate sequences of one specimen) and pairs from different process IDs; both subsets and the combined set get a count, minimum, median and maximum.
5. Nearest neighbour. The search for the closest other species is hierarchical. First, every pair of records from different species within the same genus is compared, and each species keeps its smallest distance to a congener. Second, only species that found no congener are compared against the other species in their family. Third, species still without a neighbour are compared against every other species in the dataset. The min_inter_scope column records which level supplied the neighbour.
6. Gap. The gap is the nearest-neighbour distance minus the maximum within-species distance. Genus and family summaries give the minimum, median and maximum of the species gaps beneath them, counting only species with a numeric gap.

Outputs

• species_gaps.tsv: one row per species with 26 columns: the taxonomy, record and process ID counts, the count, minimum, median and maximum of the within-species distances (all, replicate pairs and different-specimen pairs), the nearest-neighbour distance, its scope, the nearest species and process ID, and the gap. Missing values are NA.
• summary_stats.tsv: one row per genus and per family with the number of species that have a gap and the minimum, median and maximum gap.
• filtered_records.jsonl: the records included.

aligned_sequences.fas appears in the report's output list but is not produced.

The report contains five sections: Barcode Gap Analysis, a scatter of each species' maximum within-species distance against its nearest-neighbour distance with a 1:1 line, where points above the line have a gap; Barcode Gap Distribution, a histogram of gap values; Distance to Nearest Neighbour, a histogram of neighbour distances; Species Failing Barcode Gap, a table of species whose gap is zero, negative or unavailable, with the nearest species; and Family Barcode Gap Hotspots, a per-family count of failing species and genera.

Limitations and interpretation

• Singletons count as failures. A species with one record has no within-species distance, so its gap is NA. The report's failing tables treat NA the same as a missing gap. Check n_records in species_gaps.tsv before reading a "failure" as evidence of overlap.
• The neighbour search does not leave the genus. When a species has any congener in the dataset, its nearest neighbour is chosen among congeners only. A closer species in another genus is never considered. The family and dataset-wide searches run only for species with no congener present.
• Results depend on the dataset. The neighbour is the nearest species in this recordset, not in any reference library. A species can show a wide gap simply because its real sister species was not selected.
• No overlap control. Pairs with few or no comparable positions yield small or zero distances, which can erase a gap or create one.
• Missing taxonomy. A record whose species is empty reaches the analysis with the text null as its name. Such records are grouped together as a species called "null", and the check meant to exclude records without genus or species does not take effect. Remove unidentified records before running the job.
• Cost. Every distance is computed in Python; large genera and the dataset-wide fallback are the expensive steps.

Related

• The methods and their shared pipeline
• Distance summary
• Which method should I use?
• Known limitations