File types, pipelines and size limits

Which extension goes through which pipeline, what each procedure produces, and the limits.

Reference

BioStor accepts 21 file extensions. Each one is bound to a pipeline: an ordered set of procedures that run in two stages, validation and post-processing. This page lists every accepted extension, the procedures that run on it, what they produce and the size limit enforced at upload. Use it to know what to expect after you upload a file, and which previews and statistics a reviewer can rely on. For what a file is for, see Evidence profiles.

How a pipeline runs

A file is accepted by its extension; aliases are normalised first (jpeg to jpg, fas and fa to fasta, fq to fastq, abi to ab1). The extension selects a procedure group from BioStor's registry. Validation procedures run first in the validation worker; if they pass, the original is copied to Azure and the object moves on. Post-processing procedures then compute statistics (stored as derived_* fields on the object's file-type metadata) and render surrogates: additional files stored beside the original and fetched by name, such as thumbnail.jpg, 640.jpg, graph.png, spectrogram.png, preview.pdf or preview.json.

Every procedure has a timeout and a critical flag. All validators and all calculate_* statistics steps are critical, except calculate_audio_stats and calculate_tar_stats. Renderers, thumbnails, spectrograms and previews are not critical: if one fails, the failure is recorded as a derived_<procedure>_status field and the object still becomes ready, without that surrogate. Validation is attempted up to three times (after waits of 2 s and 10 s) before the object is marked failed.

Validation and size limits

Extension (aliases)	Pipeline	Validation	Size limit
`jpg` (`jpeg`)	`image_pipeline`	`validate_image`: image structure check	25 MB
`png`, `gif`	`image_pipeline`	`validate_image`	25 MB
`fasta` (`fas`, `fa`)	`fasta_pipeline`	No format check	20 MB
`fastq` (`fq`)	`fastq_pipeline`	No format check	20 MB
`ab1` (`abi`)	`ab1_pipeline`	No format check	5 MB
`scf`	`scf_pipeline`	No format check	5 MB
`fsa`	`fsa_pipeline`	No format check	1 MB
`pdf`	`pdf_pipeline`	`validate_pdf`: PDF structure check	50 MB
`xlsx`	`xlsx_pipeline`	`validate_xlsx`: workbook check	50 MB
`mp3`, `wav`	`audio_pipeline`	`validate_audio`: MP3 or WAV header check	100 MiB
`tar`	`tar_archive_pipeline`	`validate_tar_archive`: archive readability	under 2 GiB
`parquet`	`parquet_pipeline`	`validate_parquet`: Parquet footer check	500 MiB
`jsonl`	`jsonl_pipeline`	`validate_jsonl`: first 10 lines parsed	50 MiB
`tsv`	`tsv_pipeline`	`validate_tsv`: header row check	50 MiB

Limits marked MB are decimal (25 MB is 25,000,000 bytes); those marked MiB are binary (100 MiB is 104,857,600 bytes). Where no size limit is configured for an extension, as for fsa, the default of 1 MB applies.

Limit: sequence, trace and fragment-analysis files (fasta, fastq, ab1, scf, fsa) are accepted on their file name and are not format-checked at validation. A malformed file surfaces only if a statistics step fails during post-processing, and fsa has no statistics step.

Limit: a tar archive must be under 2 GiB.

Post-processing and what it produces

Extension	Procedures	Produces
`jpg`, `png`, `gif`	`calculate_image_stats`, `generate_thumbnails`	Aspect ratio and pixel count; thumbnails 320, 640 and 1024 px wide (first frame for GIF)
`fasta`	`calculate_fasta_stats`, `render_fasta`	Sequence count and mean length; `graph.png`
`fastq`	`calculate_fastq_stats`, `render_fastq`	Read count and mean length; `graph.png`
`ab1`	`calculate_ab1_stats`, `render_ab1`	Trace metadata; `graph.png`; base calls as `output.fas`
`scf`	`calculate_scf_stats`, `render_scf`	Trace metadata; `graph.png`; base calls with quality as `output.fastq`
`fsa`	`render_fsa`	`graph.png` only
`pdf`	`render_pdf`	First-page thumbnail
`xlsx`	`render_xlsx`	`preview.pdf` rendering plus a thumbnail
`mp3`, `wav`	`calculate_audio_stats`, `generate_spectrogram`	Duration, sample rate, channels, bitrate and codec; `spectrogram.png` titled with the species
`tar`	`calculate_tar_stats`	Entry count, total bytes and compression
`parquet`, `jsonl`, `tsv`	`calculate__stats`, `render__preview`	Row and column counts; `preview.json` with the column names and the first 3 rows

Two profiles add a second, profile-bound stage to tar archives after the generic pipeline:

Profile	Extra validation	Extra post-processing
`acoustic_survey`	`validate_acoustic_manifest`: checks the `index.tsv` and `sensor.json` manifests	`parse_acoustic_survey` reads sensor and index data; `generate_acoustic_spectrograms` renders one spectrogram per recording into `spectrograms.tar`
`paired_end_metabarcoding`	`validate_paired_reads`: checks that R1 and R2 are both present	`parse_paired_reads` computes paired read statistics

While processing is still running

If you request a surrogate before post-processing has finished, BioStor serves a stand-in so that pages do not break. For images, audio, tar archives and tabular files it serves the original. For FASTA, FASTQ, traces, FSA, PDF and XLSX it serves a placeholder image for that family. Once the object is ready the real surrogate is returned.

Good to know

Image statistics are limited to aspect ratio and pixel count; colour information and camera (EXIF) data are not extracted.
FASTA statistics cover sequence count and mean length; GC content is not computed.
The portal previews MP3 spectrograms; WAV objects still get a spectrogram.png but it is not shown in the portal.
Size limits are checked before anything is stored, so an oversized upload is refused immediately.