Testing a method

The test harness runs one method end to end on fixed test data, exactly as the worker would, without the analysis service, the queue or the workbench. It builds the method's image, runs the orchestrator in a container, records resource usage and leaves a result package you can open. This is the first check for any new or changed method; a run inside the platform comes after. Building a method is described in Building a method from the template.

The test harness

test_harness.sh lives in the bf-analysis-jobs root and must be run from there. It needs Docker on the machine and nothing else: no Azure access, no Redis, no API.

cd bf-analysis-jobs
bash test_harness.sh <name>

Each run:

1. Creates /tmp/test_harness/<name>.test/ with records.jsonl and params.json from the test data, and packs it as <name>.test.tar. The job ID is always test.
2. Builds the image <name> from <name>/Dockerfile, with the same build context and named bcdm_forensic_ext context the worker uses.
3. Starts a container running bash /app/orchestrator.sh <name>.test.tar with /tmp/test_harness mounted as the working directory, and samples the container's CPU and memory every 10 seconds (STATS_SAMPLE_INTERVAL overrides this).
4. Waits for the container, prints its log, writes resource_stats.json into the job directory and copies it out as the metadata sidecar.
5. Zips the job directory, removes it, and exits with the container's exit code.

Test data

Only diagnostic-characters and sequence-data-ingest ship their own records; every other method falls back to records.jsonl. To run on another file set TEST_RECORDS_FILE, for example TEST_RECORDS_FILE=test_data/records_10.jsonl bash test_harness.sh <name>.

Write the params file as the workbench would send it: every field name from the method's definition, with title, recordset_title and recordset_descr, and with the value types the form produces (slider values as strings, checkboxes as booleans, multi-selects as lists; see Registering a method). A params file that drifts from the form definition tests something the platform will never send.

What a run produces

The zip is always named <name>.test.zip, for a failed run as well as a successful one; the harness never produces a .failed.zip. A new run overwrites the previous outputs.

Checking a run

1. Check the exit code. A non-zero code names the stage; read the stage's output in the console, or extract the error record: unzip -p /tmp/test_harness/<name>.test.zip '*/error.json'.
2. On success, unzip the package and confirm the files described on Anatomy of a result package are there: filtered_records.jsonl with the count you expect for the filters in the params file, every result file your package stage lists, analysis_charts.html with content when the method draws charts, and analysis_report.html.
3. Open analysis_report.html. The Parameters section should show the params file, the Results counts should match your line counts, the Outputs list should name every file, and the Analysis section should show the charts.
4. Confirm analysis_report.pdf is present. If it is missing, the console log contains PDF generation failed; the usual cause is a missing WeasyPrint library in the Dockerfile.
5. Prove the parameters are applied: change one value in the params file, rerun, and compare the result files. A parameter whose change makes no difference is either unread or unused by the scripts, and should be fixed or removed from the form.
6. For runs longer than 10 seconds, check the sidecar has sample_count above zero and error equal to null.

Testing inside the platform

Once the harness passes, run the method through the service. Submit a job from the workbench on a small recordset (see Running an analysis). The scheduler is part of the service and runs every 30 seconds in a developer's stack too, so the job moves to QUEUED on its own; local-analysis-scheduled-tasks/2_queue_analysis.sh triggers one scheduler pass immediately if you do not want to wait. 5_list_queue.sh in the same folder lists the queue and results containers, which is the quickest way to confirm the package was uploaded.

The worker is not part of a developer's stack. On a host configured for the environment's containers, run bash execute_job.sh from the bf-analysis-jobs root to process the oldest queued package once; its log and last_run_summary.json are written to the run log directory (RUN_LOG_DIR). Within 30 seconds of the upload the job shows COMPLETE or FAILED in the Analysis panel, and the report preview and download work as they will for users. The administrator's analysis dashboard lists every job and can download the packages, including a failed one.

Good to know

Good to know: The harness builds the image on every run, like the worker. The first build of a new method is slow; later builds reuse cached layers.

Good to know: Every method directory receives the same orchestrator contract, so a method that passes the harness but fails in the platform almost always differs in its inputs: compare the real package's params.json and records.jsonl with your fixtures.

Good to know: Some method directories also contain older helpers (init_test_env.sh, test_orchestrator.sh) that stage a working directory under /tmp/working. The harness is the supported route.

Related

• Building a method from the template
• Registering a method in the workbench
• Anatomy of a result package
• Job lifecycle