BF Workbench is a restricted scientific system. Records are available only to registered, authorised accounts.

Building a method from the template

The five stage scripts, the Docker image, inputs and outputs.

How-to

A method is a directory in the bf-analysis-jobs repository: an orchestrator, five stage scripts, a Dockerfile and, usually, a chart script. The worker builds the directory's image and runs its orchestrator on each job package; the analysis API needs no change, because it accepts any method name. This page gives the contract each piece must honour and walks through sequence-composition as a concrete example. Registering the method's form is on Registering a method in the workbench and running it on test data on Testing a method.

How a method runs

A job arrives as <name>.<id>.tar, containing a directory <name>.<id>/ with params.json and records.jsonl. The worker builds the image <name> from <name>/Dockerfile and runs bash /app/orchestrator.sh <name>.<id>.tar inside it, with the job working directory mounted at the same path and set as the current directory. The orchestrator unpacks the tar, changes into <name>.<id>/, checks both files exist, runs the five stages and leaves the directory in place. The worker then adds resource_stats.json, zips the directory and uploads it; the orchestrator must not zip anything itself.

Start from a working method

The template/ directory holds one file, orchestrator.template.sh, and it expects a .zip while real packages are .tar, so it is not usable as it stands. Copy a working method instead:

  1. cp -r sequence-composition <name> inside bf-analysis-jobs/. The name must contain no dots (the worker reads the method name as the text before the first dot of the package name) and must equal the title in the workbench definition.
  2. In orchestrator.sh, set JOB_NAME="<name>" and CONVERTED_INPUT to the file name your convert stage will write. Leave the rest alone; every method uses the same orchestrator logic.
  3. Replace the stage scripts and generate_charts.py.
  4. In the Dockerfile, change the COPY <name>/... lines and add the programs your method needs.
  5. Add test fixtures and run the harness.
  6. Register the form and menu entry in the workbench.
  7. Deploy: update the bf-analysis-jobs checkout on the analysis host (the worker rebuilds the image on every run) and redeploy the workbench.

What the scripts see

The orchestrator exports these before calling any stage:

Variable Value
JOB_NAME, JOB_ID, JOB_DIR The method name, the job ID, and <name>.<id>, which is the current directory while the stages run
PARAMS, RECORDS params.json and records.jsonl
ANALYTICS_SCRIPTS_DIR /app, where the method's scripts and the shared charting/ folder are copied
SHARED_TMP_ROOT /shared-tmp, a directory shared with the host

params.json holds the form values keyed by field name, plus title, recordset_title and recordset_descr. Slider values arrive as strings, checkboxes as booleans and multi-selects as arrays (see Registering a method). records.jsonl holds one record per line with the record store's field names (record_id, processid, marker_code, nuc, the taxonomic ranks, and so on). jq, Python 3 and a virtual environment with pandas and weasyprint are on the path.

The five stages

Stage Arguments Must produce
1.validate.sh params.json records.jsonl Nothing. Exit non-zero if a parameter is missing or invalid
2.filter.sh params.json records.jsonl The records to analyse, as JSONL on stdout; the orchestrator saves it as filtered_records.jsonl
3.convert.sh params.json filtered_records.jsonl <CONVERTED_INPUT> The converted input file
4.execute.sh params.json filtered_records.jsonl <CONVERTED_INPUT> The result files, written into the current directory
5.package.sh params.json records.jsonl <name>.<id>.zip analysis_charts.html, analysis_report.html and the PDF

The rules are the same for all five. Standard output is data: only the filter stage's is kept, the others' is discarded. Report progress on standard error. Signal failure by exit code; on a non-zero exit the orchestrator writes error.json with the script name and everything the stage printed, and exits with the stage number. Before the package stage it writes stage_timings.json. The third argument to 5.package.sh is only a name; the report generator uses it to recover the job ID.

Charts and the report

generate_charts.py builds the method's charts from its own result files and writes analysis_charts.html. Add ANALYTICS_SCRIPTS_DIR to sys.path and import from charting.chart_helpers: COLORS (the report palette), esc, svg_open, svg_close, nice_ceil for axis ceilings, and write_charts(html), which wraps your chart blocks in the Analysis section with the shared chart CSS, or writes an empty file when given nothing. Each chart is a <div class="chart-container"> with a chart-title, a chart-subtitle and an inline SVG.

5.package.sh then sources charting/generate_html_report.sh and calls:

generate_html_report "$PARAMS" "$RECORDS" "$JOB_PACKAGE" "<name>" \
    "result_table.tsv|what it contains" \
    "filtered_records.jsonl|records included in analysis"

Each extra argument is a file|description pair for the Outputs section. The function copies charting/ into the job directory, writes analysis_report.html and load_stats.json, and renders analysis_report.pdf with WeasyPrint on a best-effort basis. See Anatomy of a result package for what the report then contains.

The Dockerfile

The build context is the bf-analysis-jobs root, not the method directory, so COPY paths start with the method name. The existing Dockerfiles start from ubuntu:noble pinned by digest, install jq, zip, unzip, Python with venv, and the Pango and Cairo libraries WeasyPrint needs, create /opt/venv with pandas and weasyprint, set WORKDIR /app, copy the method's *.sh and *.py files to /app/ and charting/ to /app/charting/. Add your method's programs as further apt-get or pip installs. Only the two ingest methods need BCDM field definitions; they add COPY --from=bcdm_forensic_ext ..., and the worker and harness supply that named context. To build by hand:

cd bf-analysis-jobs
DOCKER_BUILDKIT=1 docker build -t <name> \
  --build-context bcdm_forensic_ext=../bcdm-forensic-ext \
  -f <name>/Dockerfile .

Walk-through: sequence-composition

  • 1.validate.sh checks that marker, filter_min_seqlength_bp and filter_max_seqlength_bp are present, counting errors and exiting 200 if any.
  • 2.filter.sh reads the marker and length bounds, pipes records.jsonl through filter_BCDM.py, which keeps records whose marker_code matches and whose sequence length is in range, and keeps only the fields the later stages need.
  • 3.convert.sh turns each filtered record into a FASTA entry in labelled_sequences.fas, with the record ID and the remaining fields in the header.
  • 4.execute.sh runs composition_by_tax.py on that FASTA to write record_composition.tsv and composition_summary.tsv.
  • 5.package.sh runs generate_charts.py (a GC-content histogram, composition by taxon and a length histogram) and calls generate_html_report listing the two tables and filtered_records.jsonl.

The same method shows the trap to avoid: its filter stage reads the stop-codon and misidentification checkboxes into variables but never applies them, so the report lists them as submitted while they have no effect. Read every parameter your form offers, and apply what you read.

Good to know

Good to know: The image is rebuilt on the host for every job, so pin program versions in the Dockerfile where reproducibility matters, and keep the build short.

Good to know: There are no per-method settings for time limits or resources. Every job has the same 6-hour limit from submission.

Good to know: If the package's method name matches no directory, the worker cannot run it and the job fails at its time limit with no result. Keep the directory name and the workbench title identical.