stjudecloud · adthrasher · Dec 3, 2025 · Dec 3, 2025 · Dec 3, 2025 · Dec 3, 2025
diff --git a/.gitattributes b/.gitattributes
@@ -17,3 +17,8 @@ bin/check-job-alive !text !filter !merge !diff
 *.conf !text !filter !merge !diff
 .fa filter=lfs diff=lfs merge=lfs -text
 *.fa filter=lfs diff=lfs merge=lfs -text
+*.mmi filter=lfs diff=lfs merge=lfs -text
+*.gbz filter=lfs diff=lfs merge=lfs -text
+*.dist filter=lfs diff=lfs merge=lfs -text
+*.min filter=lfs diff=lfs merge=lfs -text
+*.zipcodes filter=lfs diff=lfs merge=lfs -text
diff --git a/.github/workflows/sprocket-test.yaml b/.github/workflows/sprocket-test.yaml
@@ -26,11 +26,11 @@ jobs:
     - uses: actions/checkout@v4
       with:
         lfs: true
-    - name: Update Rust
-      run: rustup update stable && rustup default stable
-    - name: Build Sprocket
+    - name: Install Sprocket
       run: |
-        cargo install sprocket --locked
+        SPROCKET_URL=$(curl -sSL https://api.github.com/repos/stjude-rust-labs/sprocket/releases/latest \
+          | jq -r '.assets[] | select(.name | test("x86_64-unknown-linux-gnu.tar.gz")) | .browser_download_url')
+        curl -sSL "$SPROCKET_URL" | tar -xz -C /usr/local/bin sprocket
     - name: Update containers
       run: |
         ./developer_scripts/update_container_tags.sh ${GITHUB_REF##*/}

diff --git a/AGENTS.md b/AGENTS.md
@@ -0,0 +1,141 @@
+# AGENTS.md
+
+Agent guidance for the St. Jude Cloud bioinformatics workflows repository.
+
+## What this repo is
+
+A collection of WDL 1.1 pipelines and tool wrappers for genomic analysis. Primary artifacts are `.wdl` files; there is no traditional build step. Python and R exist only as scripts embedded in Docker images under `scripts/`.
+
+## Layout
+
+```
+workflows/        # End-to-end pipelines (rnaseq/, dnaseq/, chipseq/, methylation/, etc.)
+tools/            # Individual WDL task wrappers (one tool per file)
+data_structures/  # WDL struct definitions
+docker/           # Dockerfiles + package.json version metadata per image
+scripts/          # Python and R scripts (not standalone; embedded in Docker images)
+tests/            # pytest-workflow YAML test definitions
+  tools/          # Per-tool test YAMLs
+  workflows/      # Per-workflow test YAMLs
+  input_json/     # Input JSON fixtures
+  input/          # Binary fixtures (BAMs, FASTQs, etc.) — tracked via git-lfs
+template/         # task-examples.wdl (canonical patterns) + common-parameter-meta.txt
+developer_scripts/
+  run_sprocket_or_miniwdl.sh   # Unified test runner
+  update_container_tags.sh     # Rewrites container tags for branch testing (do not commit)
+```
+
+## Developer commands
+
+### Setup
+
+```bash
+uv sync          # Python deps (canonical; use this, not pip)
+git lfs pull     # Required after clone to get test fixture files
+```
+
+### WDL lint / format / check (Sprocket)
+
+```bash
+sprocket lint .                    # Lint all WDL
+sprocket format tools/bwa.wdl      # Format a file
+sprocket check .                   # Thorough check (validates inputs, etc.)
+sprocket validate <wdl> <inputs>   # Validate inputs against a workflow
+```
+
+`sprocket.toml` disables the `ContainerUri` rule and sets `deny_notes = true` (notes in WDL cause check failure).
+
+### Python (scripts/ only)
+
+```bash
+uv run ruff format --check --diff scripts/
+uv run ruff format scripts/
+uv run ruff check scripts/
+```
+
+### R (scripts/ only)
+
+```r
+styler::style_dir()   # format
+lintr::lint_dir()     # lint
+```
+
+### Tests
+
+```bash
+# All tests
+uv run pytest --kwdof --wt $(nproc)
+
+# Single test by tag
+pytest --tag bwa
+
+# Single test by name
+pytest -k bwa_aln
+
+# Choose runner (defaults to sprocket)
+RUNNER=miniwdl pytest --tag bwa
+```
+
+`pytest.ini` always injects `--git-aware --symlink`. Use `--kwdof` to keep outputs on failure.
+
+Every test calls `./developer_scripts/run_sprocket_or_miniwdl.sh` internally — do not call runners directly in test commands.
+
+### Run a WDL task directly (outside pytest)
+
+```bash
+# sprocket
+sprocket run --output-dir output --target bwa_aln tools/bwa.wdl <input_file>
+
+# miniwdl
+miniwdl run --task bwa_aln --verbose --dir output/. -i tests/tools/input_json/bwa_aln.json tools/bwa.wdl
+```
+
+## CI pipeline
+
+All jobs trigger on push. Key facts:
+- `reference` and `slow` tags are **excluded** from the CI test matrix.
+- CI deletes `slow`-tagged tests from YAML files in-place before running — do not replicate this destructively locally.
+- CI builds Docker images and runs `update_container_tags.sh` to rewrite tags before pytest — do not commit the rewritten WDL files.
+- Every test runs against both `sprocket` and `miniwdl` runners as a matrix.
+- `sprocket` is built from source (`cargo install sprocket --locked`) in CI.
+- `requirements-ci.txt` pins older versions than `pyproject.toml`; use `uv` locally.
+
+## WDL conventions (non-obvious)
+
+- All WDL files must be `version 1.1`.
+- All tasks must include `set -euo pipefail` when using pipes or multiple commands.
+- Multi-core tasks: accept `use_all_cores: Boolean = false` and `ncpu: Int = 2`; use `$(nproc)` when `use_all_cores` is true. `use_all_cores` must be the last Boolean in the input block; `ncpu` precedes memory/disk inputs.
+- Resource inputs (`ncpu`, `modify_memory_gb`, `modify_disk_size_gb`) must be overridable.
+- Single output → `outfile_name`; multiple outputs or extension matters → `prefix`.
+- BAM/BAI companion pairs must be localized via `ln -s` to CWD and cleaned up with `rm` at task end.
+- Scripts go in `scripts/`; never embed Python/R directly in WDL `command` blocks.
+- Imports within the repo must use **relative paths**. External imports must pin a **tagged release** (not `main`/`master`) — enforced by `pull-check.yaml`.
+- Deprecated tasks: add `deprecated: true` and `warning: "**[DEPRECATED]**..."` to `meta`. Never add `deprecated: false`.
+- Update the relevant `CHANGELOG.md` when modifying any WDL under a subdirectory.
+
+## Parameter meta conventions
+
+Prefer names from `template/common-parameter-meta.txt`. Key ones:
+- `bam` (not `input_bam`, `in_bam`)
+- `bam_index` (not `bai`)
+- `read_one_fastq_gz` / `read_two_fastq_gz` (not `read1`/`read2`)
+- `paired_end` (not `paired`)
+
+## Docker image versioning
+
+Each image lives in `docker/<tool>/` with `Dockerfile` and `package.json`. `version` = underlying tool version; `revision` starts at `0`, increments for image-only changes, resets to `0` on tool version upgrades. Prefer BioContainers images over creating new custom images.
+
+## Testing quirks
+
+- Binary fixtures in `tests/input/` are in **git-lfs** — run `git lfs pull` before testing.
+- Test files prefixed with `_` (e.g., `_test_methylation-preprocess.yaml`) are disabled.
+- Sprocket outputs land in `output/runs/*/_latest/outputs.json`, then get copied to `output/outputs.json`. miniwdl outputs go directly to `output/`. Test YAMLs reference `output/outputs.json`.
+- Fixture inputs are downsampled to small chromosomes (chrY/chrM, chr9/chr22) for speed.
+
+## Existing instruction sources
+
+- `.github/instructions/wdl.instructions.md` — applies to `**/*.wdl`; references CONTRIBUTING.md, best-practices.md, template/task-examples.wdl, and template/common-parameter-meta.txt.
+- `template/task-examples.wdl` — canonical WDL task patterns; read before writing new tasks.
+- `template/common-parameter-meta.txt` — required/banned parameter_meta strings.
+- `CONTRIBUTING.md` — general coding style.
+- `best-practices.md` — WDL-specific best practices.
diff --git a/data_structures/read_group.wdl b/data_structures/read_group.wdl
@@ -425,3 +425,50 @@ task inner_read_group_to_string {
         maxRetries: 1
     }
 }
+
+task read_group_to_array {
+    meta {
+        description: "Converts a `ReadGroup` struct to a `Array[String]` **without any validation**."
+        outputs: {
+            converted_read_group: "Input `ReadGroup` as a `Array[String]`",
+        }
+    }
+
+    parameter_meta {
+        read_group: "`ReadGroup` struct to convert to array"
+    }
+
+    input {
+        ReadGroup read_group
+    }
+
+    String delimiter = "\n"
+
+    command <<<
+        {
+            echo -n "~{"ID:" + read_group.ID}"  # required field. All others optional
+            echo -n "~{delimiter + "BC:" + read_group.BC}"
+            echo -n "~{delimiter + "CN:" + read_group.CN}"
+            echo -n "~{delimiter + "DS:" + read_group.DS}"
+            echo -n "~{delimiter + "DT:" + read_group.DT}"
+            echo -n "~{delimiter + "FO:" + read_group.FO}"
+            echo -n "~{delimiter + "KS:" + read_group.KS}"
+            echo -n "~{delimiter + "LB:" + read_group.LB}"
+            echo -n "~{delimiter + "PG:" + read_group.PG}"
+            echo -n "~{delimiter + "PI:" + read_group.PI}"
+            echo -n "~{delimiter + "PL:" + read_group.PL}"
+            echo -n "~{delimiter + "PM:" + read_group.PM}"
+            echo -n "~{delimiter + "PU:" + read_group.PU}"
+            echo "~{delimiter + "SM:" + read_group.SM}"
+        } >> out.txt
+    >>>
+
+    output {
+        Array[String] converted_read_group = read_lines("out.txt")
+    }
+
+    runtime {
+        container: "ghcr.io/stjudecloud/util:3.0.3"
+        maxRetries: 1
+    }
+}
diff --git a/docker/bwamem2/Dockerfile b/docker/bwamem2/Dockerfile
@@ -0,0 +1,8 @@
+FROM quay.io/biocontainers/samtools:1.17--h00cdaf9_0 AS samtools
+FROM quay.io/biocontainers/bwa-mem2:2.3--he70b90d_0
+
+COPY --from=samtools /usr/local/bin/ /usr/local/bin/
+COPY --from=samtools /usr/local/lib/ /usr/local/lib/
+COPY --from=samtools /usr/local/libexec/ /usr/local/libexec/
+
+ENTRYPOINT [ "bwa-mem2" ]
diff --git a/docker/bwamem2/package.json b/docker/bwamem2/package.json
@@ -0,0 +1,5 @@
+{
+    "name": "bwamem2",
+    "version": "2.3",
+    "revision": "0"
+}
diff --git a/docker/hisat2/Dockerfile b/docker/hisat2/Dockerfile
@@ -0,0 +1,8 @@
+FROM quay.io/biocontainers/samtools:1.17--h00cdaf9_0 AS samtools
+FROM quay.io/biocontainers/hisat2:2.2.1--hdbdd923_7
+
+COPY --from=samtools /usr/local/bin/ /usr/local/bin/
+COPY --from=samtools /usr/local/lib/ /usr/local/lib/
+COPY --from=samtools /usr/local/libexec/ /usr/local/libexec/
+
+ENTRYPOINT [ "hisat2" ]
diff --git a/docker/hisat2/package.json b/docker/hisat2/package.json
@@ -0,0 +1,5 @@
+{
+    "name": "hisat2",
+    "version": "2.2.1",
+    "revision": "0"
+}
diff --git a/docker/minimap2/Dockerfile b/docker/minimap2/Dockerfile
@@ -0,0 +1,8 @@
+FROM quay.io/biocontainers/samtools:1.17--h00cdaf9_0 AS samtools
+FROM quay.io/biocontainers/minimap2:2.30--h577a1d6_0
+
+COPY --from=samtools /usr/local/bin/ /usr/local/bin/
+COPY --from=samtools /usr/local/lib/ /usr/local/lib/
+COPY --from=samtools /usr/local/libexec/ /usr/local/libexec/
+
+ENTRYPOINT [ "minimap2" ]
diff --git a/docker/minimap2/package.json b/docker/minimap2/package.json
@@ -0,0 +1,5 @@
+{
+    "name": "minimap2",
+    "version": "2.30",
+    "revision": "0"
+}
diff --git a/docker/ngsep/Dockerfile b/docker/ngsep/Dockerfile
@@ -0,0 +1,5 @@
+FROM eclipse-temurin:21
+
+RUN wget https://github.com/NGSEP/NGSEPcore/releases/download/v5.1.0/NGSEPcore_5.1.0.jar -O /usr/local/bin/NGSEPcore.jar
+
+ENTRYPOINT [ "java", "-jar", "/usr/local/bin/NGSEPcore.jar" ]
diff --git a/docker/ngsep/package.json b/docker/ngsep/package.json
@@ -0,0 +1,5 @@
+{
+    "name": "ngsep",
+    "version": "5.1.0",
+    "revision": "0"
+}
diff --git a/test/fixtures/bams/README.md b/test/fixtures/bams/README.md
@@ -40,6 +40,14 @@ Coordinate sorted BAM with 1 read group (`ID:test`). Aligned using BWA `aln` (us
 
 BAM index corresponding to `test.bwa_aln_pe.chrY_chrM.bam`.
 
+## test.bwa_aln_pe.with_variants.chrY_chrM.bam
+
+Coordinate sorted BAM with 1 read group (`ID:test`). Aligned using BWA `aln` (using the `bwa.bwa_aln_pe` WDL task). 100,000 Paired-End reads mapped to GRCh38 `chrY` and `chrM` from `test_variants_R1.fq.gz` and `test_variants_R2.fq.gz`.
+
+## test.bwa_aln_pe.with_variants.chrY_chrM.bam.bai
+
+BAM index corresponding to `test.bwa_aln_pe.with_variants.chrY_chrM.bam`.
+
 ## test.extra_RG.bam
 
 A duplicate of `test.bam` with an added `@RG` entry with the ID `no_match`. There are no reads corresponding to the `no_match` RG entry.

diff --git a/test/fixtures/bams/test.bwa_aln_pe.with_variants.chrY_chrM.bam b/test/fixtures/bams/test.bwa_aln_pe.with_variants.chrY_chrM.bam
diff --git a/test/fixtures/bams/test.bwa_aln_pe.with_variants.chrY_chrM.bam.bai b/test/fixtures/bams/test.bwa_aln_pe.with_variants.chrY_chrM.bam.bai
diff --git a/test/fixtures/bams/test.bwa_aln_pe.with_variants.chrY_chrM.bam.csi b/test/fixtures/bams/test.bwa_aln_pe.with_variants.chrY_chrM.bam.csi
diff --git a/test/fixtures/fastqs/README.md b/test/fixtures/fastqs/README.md
@@ -19,3 +19,11 @@ The following list is sorted alphabetically:
 ## test_R2.fq.gz
 
 10,000 reads in FASTQ format. Can be used by itself to represent a Single-End sample or used with `test_R1.fq.gz` to represent a Paired-End sample. Generated with `fq generate`.
+
+## test_variants_R1.fq.gz
+
+10,000 reads in FASTQ format. Can be used by itself to represent a Single-End sample or used with `test_variants.R2.fq.gz` to represent a Paired-End sample. Generated with `fq generate` on `chrY` and `chrM`. Simulated from a reference sequence containing variants.
+
+## test_variants_R2.fq.gz
+
+10,000 reads in FASTQ format. Can be used by itself to represent a Single-End sample or used with `test_variants.R1.fq.gz` to represent a Paired-End sample. Generated with `fq generate` on `chrY` and `chrM`. Simulated from a reference sequence containing variants.
diff --git a/test/fixtures/fastqs/test_variants_R1.fq.gz b/test/fixtures/fastqs/test_variants_R1.fq.gz
diff --git a/test/fixtures/fastqs/test_variants_R2.fq.gz b/test/fixtures/fastqs/test_variants_R2.fq.gz
diff --git a/test/fixtures/mutect2/README.md b/test/fixtures/mutect2/README.md
@@ -0,0 +1,29 @@
+# Mutect2 fixtures
+
+Inputs and intermediate artifacts used by the per-task tests in `tools/test/mutect2.yaml` for `tools/mutect2.wdl`. Files are downsampled to `chrY` and `chrM` to match the reference at `test/fixtures/reference/GRCh38.chrY_chrM.fa` and the BAMs at `test/fixtures/bams/test.bwa_aln_pe*.chrY_chrM.bam`.
+
+The following list is sorted alphabetically:
+
+## af-only-gnomad.hg38.chrY_chrM.vcf.gz
+
+GATK best-practices germline resource (`af-only-gnomad.hg38.vcf.gz` from the GATK resource bundle), subset to `chrY` and `chrM`. Used as the `germline_resource_vcf` input for the `mutect2` task and as both `intervals` and `variants` inputs for the `get_pileup_summaries` task.
+
+## af-only-gnomad.hg38.chrY_chrM.vcf.gz.tbi
+
+Tabix index for `af-only-gnomad.hg38.chrY_chrM.vcf.gz`.
+
+## test.bwa_aln_pe.chrY_chrM_pileup_summaries.table
+
+Output of `gatk GetPileupSummaries` run on `test/fixtures/bams/test.bwa_aln_pe.chrY_chrM.bam` (normal sample) over the sites in `af-only-gnomad.hg38.chrY_chrM.vcf.gz`. Used as the `normal_pileups` input for the `calculate_contamination` task.
+
+## test.bwa_aln_pe.with_variants.chrY_chrM_pileup_summaries.contamination.table
+
+Output of `gatk CalculateContamination` run on the tumor and normal pileup tables (matched-normal mode). Used as the `contamination_table` input for the `filter_mutect` task.
+
+## test.bwa_aln_pe.with_variants.chrY_chrM_pileup_summaries.segments.table
+
+Tumor segmentation output of the same `gatk CalculateContamination` invocation that produced the contamination table. Used as the `maf_segments` input for the `filter_mutect` task.
+
+## test.bwa_aln_pe.with_variants.chrY_chrM_pileup_summaries.table
+
+Output of `gatk GetPileupSummaries` run on `test/fixtures/bams/test.bwa_aln_pe.with_variants.chrY_chrM.bam` (tumor sample) over the sites in `af-only-gnomad.hg38.chrY_chrM.vcf.gz`. Used as the `tumor_pileups` input for the `calculate_contamination` task.
diff --git a/test/fixtures/mutect2/af-only-gnomad.hg38.chrY_chrM.vcf.gz b/test/fixtures/mutect2/af-only-gnomad.hg38.chrY_chrM.vcf.gz
diff --git a/test/fixtures/mutect2/af-only-gnomad.hg38.chrY_chrM.vcf.gz.tbi b/test/fixtures/mutect2/af-only-gnomad.hg38.chrY_chrM.vcf.gz.tbi
diff --git a/test/fixtures/mutect2/test.bwa_aln_pe.chrY_chrM_pileup_summaries.table b/test/fixtures/mutect2/test.bwa_aln_pe.chrY_chrM_pileup_summaries.table
@@ -0,0 +1,2 @@
+#<METADATA>SAMPLE=test
+contig	position	ref_count	alt_count	other_alt_count	allele_frequency
diff --git a/...ures/mutect2/test.bwa_aln_pe.with_variants.chrY_chrM_pileup_summaries.contamination.table b/...ures/mutect2/test.bwa_aln_pe.with_variants.chrY_chrM_pileup_summaries.contamination.table
@@ -0,0 +1,2 @@
+sample	contamination	error
+test	0.0	1.0
diff --git a/.../fixtures/mutect2/test.bwa_aln_pe.with_variants.chrY_chrM_pileup_summaries.segments.table b/.../fixtures/mutect2/test.bwa_aln_pe.with_variants.chrY_chrM_pileup_summaries.segments.table
@@ -0,0 +1,2 @@
+#<METADATA>SAMPLE=test
+contig	start	end	minor_allele_fraction
diff --git a/test/fixtures/mutect2/test.bwa_aln_pe.with_variants.chrY_chrM_pileup_summaries.table b/test/fixtures/mutect2/test.bwa_aln_pe.with_variants.chrY_chrM_pileup_summaries.table
@@ -0,0 +1,2 @@
+#<METADATA>SAMPLE=test
+contig	position	ref_count	alt_count	other_alt_count	allele_frequency
diff --git a/test/fixtures/reference/GRCh38.chrY_chrM.bwamem2_db.tar.gz b/test/fixtures/reference/GRCh38.chrY_chrM.bwamem2_db.tar.gz
diff --git a/test/fixtures/reference/GRCh38.chrY_chrM.dist b/test/fixtures/reference/GRCh38.chrY_chrM.dist
diff --git a/test/fixtures/reference/GRCh38.chrY_chrM.giraffe.gbz b/test/fixtures/reference/GRCh38.chrY_chrM.giraffe.gbz
diff --git a/test/fixtures/reference/GRCh38.chrY_chrM.mmi b/test/fixtures/reference/GRCh38.chrY_chrM.mmi
diff --git a/test/fixtures/reference/GRCh38.chrY_chrM.shortread.withzip.min b/test/fixtures/reference/GRCh38.chrY_chrM.shortread.withzip.min
diff --git a/test/fixtures/reference/GRCh38.chrY_chrM.shortread.zipcodes b/test/fixtures/reference/GRCh38.chrY_chrM.shortread.zipcodes
diff --git a/test/fixtures/reference/Homo_sapiens_assembly38.dbsnp138.top5000.vcf.gz b/test/fixtures/reference/Homo_sapiens_assembly38.dbsnp138.top5000.vcf.gz
diff --git a/test/fixtures/reference/Homo_sapiens_assembly38.dbsnp138.top5000.vcf.gz.tbi b/test/fixtures/reference/Homo_sapiens_assembly38.dbsnp138.top5000.vcf.gz.tbi
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		#<METADATA>SAMPLE=test
		contig position ref_count alt_count other_alt_count allele_frequency
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		#<METADATA>SAMPLE=test
		contig start end minor_allele_fraction