diff --git a/bbmri-miabis/README.md b/bbmri-miabis/README.md new file mode 100644 index 0000000..140f15a --- /dev/null +++ b/bbmri-miabis/README.md @@ -0,0 +1,64 @@ +## MIABIS-on-FHIR + BBMRI.de mixed-node Locator + +Tests a federated search across two nodes: one running MIABIS-on-FHIR data (focus with `CQL_FLAVOUR=miabis`) and one running BBMRI.de data (focus with default flavour). + +```bash +# Local mode (manual UI testing at http://localhost:3000/search/) +docker compose -f bbmri-miabis/compose.local.yaml up --pull always + +# Local mode with automated tests +docker compose -f bbmri-miabis/compose.local.yaml up -d +docker compose -f bbmri-miabis/compose.local.yaml wait tester && echo success +``` + +### What this tests + +Eight scenarios across a 2-node federation (MIABIS-on-FHIR node + BBMRI.de node). Counts are totals across both sites. + +**Test data summary:** +- MIABIS node (`proxy2`, `CQL_FLAVOUR=miabis`): p1 (female, C34) — Plasma/LN + TissueFixed/RT; p2 (male, C18) — WholeBlood/LN +- BBMRI.de node (`proxy3`, default CQL flavour): bbmri-p1 (female, C50) — blood-plasma/LN; bbmri-p2 (male, C61) — whole-blood/RT + +**Mixed-node scenarios (validate totals across both sites):** + +| Criterion | MIABIS | BBMRI.de | Total | +|---|---|---|---| +| *(empty AST)* | 2 | 2 | 4 | +| `storage_temperature = temperatureRoom` | 1 (p1, RT specimen) | 1 (bbmri-p2, Room) | 2 | +| `storage_temperature = temperatureLN` | 2 (p1 + p2, LN specimens) | 1 (bbmri-p1, LN) | 3 | +| `sample_kind = blood-plasma` | 1 (p1, Plasma) | 1 (bbmri-p1) | 2 | +| `sample_kind = whole-blood` | 1 (p2, WholeBlood) | 1 (bbmri-p2) | 2 | + +**Diagnosis scenarios (validate BBMRI.de node is unaffected by MIABIS workarounds):** + +| Criterion | MIABIS | BBMRI.de | Total | +|---|---|---|---| +| `diagnosis = C34` | 1 (p1) | 0 | 1 | +| `diagnosis = C50` | 0 | 1 (bbmri-p1) | 1 | +| `diagnosis = C61` | 0 | 1 (bbmri-p2) | 1 | + +### Key design points + +**`CQL_FLAVOUR=miabis` (per-site focus config):** Selects the MIABIS-on-FHIR CQL template in focus. Spot continues to send `PROJECT=bbmri` unchanged — only the per-site focus environment variable differs. The BBMRI.de focus instance runs without `CQL_FLAVOUR` and uses the default BBMRI template. + +**`CODE_WORKAROUNDS` translation:** Without it, Lens codes (`temperatureRoom`, `blood-plasma`, etc.) do not exist in MIABIS FHIR data and all filter queries silently return 0. The mixed-node temperature and sample_kind scenarios validate that translation works correctly on the MIABIS node while leaving the BBMRI.de node unaffected. + +### Test data + +`test-bundle.json` — MIABIS-on-FHIR FHIR transaction bundle: +- Patient p1 (female, dx C34): Specimen Plasma/LN, Specimen TissueFixed/RT +- Patient p2 (male, dx C18): Specimen WholeBlood/LN + +`test-bundle-bbmri.json` — BBMRI.de FHIR transaction bundle: +- Patient bbmri-p1 (female, dx C50): Specimen blood-plasma/LN +- Patient bbmri-p2 (male, dx C61): Specimen whole-blood/RT + +### Troubleshooting + +**`--exit-code-from tester` kills the stack immediately (Docker Compose v5):** `--exit-code-from` implies `--abort-on-container-exit`, so `pki-setup` exiting normally after PKI initialisation terminates the entire stack before Blaze becomes healthy. Use `up -d` and run `test.sh` directly instead (see commands above). + +**All scenarios return `got=` on second run:** Spot does not re-stream results for a task ID it has already processed. Run `docker compose -f bbmri-miabis/compose.local.yaml down -v` before re-running tests to get a clean stack with fresh task IDs. + +**Spot rejects non-UUID query IDs:** IDs must be in `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` hex format. Shorthand forms like `test-0001-0000-...` are silently rejected. + +**BBMRI.de filter queries silently return 0:** BBMRI.de focus CQL uses `fhir.bbmri.de` URLs for the StorageTemperature extension and SampleMaterialType CodeSystem. If `test-bundle-bbmri.json` is modified and the wrong namespace (`fhir.bbmri-eric.eu`) is used instead, all filter queries on the BBMRI.de node return 0 with no error. diff --git a/bbmri-miabis/compose.local.yaml b/bbmri-miabis/compose.local.yaml new file mode 100644 index 0000000..a1dfc92 --- /dev/null +++ b/bbmri-miabis/compose.local.yaml @@ -0,0 +1,123 @@ +services: + bbmri-sample-locator: + image: samply/bbmri-sample-locator:latest + ports: + - 3000:3000 + environment: + PUBLIC_SPOT_URL: http://localhost:8055 + + spot: + image: samply/rustyspot:latest + depends_on: + - proxy1 + ports: + - 8055:8055 + extra_hosts: + - host.docker.internal:host-gateway + environment: + BEAM_PROXY_URL: http://host.docker.internal:4001 + BEAM_APP_ID: spot.proxy1.broker + BEAM_SECRET: pass123 + CORS_ORIGIN: http://localhost:3000 + PRISM_URL: http://host.docker.internal:8056 + TRANSFORM: LENS + PROJECT: bbmri + SITES: &sites "proxy2,proxy3" + + prism: + image: samply/prism:main + depends_on: + test-data-loader-miabis: + condition: service_completed_successfully + test-data-loader-bbmri: + condition: service_completed_successfully + ports: + - 8056:8056 + extra_hosts: + - host.docker.internal:host-gateway + environment: + BIND_ADDR: 0.0.0.0:8056 + BEAM_PROXY_URL: http://host.docker.internal:4001 + BEAM_APP_ID_LONG: prism.proxy1.broker + API_KEY: pass123 + CORS_ORIGIN: any + PROJECT: bbmri + SITES: *sites + + focus-miabis: + image: samply/focus:develop + depends_on: + - proxy2 + extra_hosts: + - host.docker.internal:host-gateway + environment: + BEAM_PROXY_URL: http://host.docker.internal:4002 + BEAM_APP_ID_LONG: focus.proxy2.broker + API_KEY: pass123 + ENDPOINT_TYPE: blaze + BLAZE_URL: http://host.docker.internal:8080/fhir/ + OBFUSCATE: no + CQL_FLAVOUR: miabis + + blaze-miabis: + image: samply/blaze:0.32 + ports: + - 8080:8080 + healthcheck: + test: curl -f http://localhost:8080/fhir/metadata + start_period: 1m + + test-data-loader-miabis: + image: curlimages/curl:latest + depends_on: + blaze-miabis: + condition: service_healthy + volumes: + - ./test-bundle.json:/test-bundle.json:ro + entrypoint: ["curl", "-sf", "-X", "POST", "http://blaze-miabis:8080/fhir", "-H", "Content-Type: application/fhir+json", "-d", "@/test-bundle.json"] + + focus-bbmri: + image: samply/focus:develop + depends_on: + - proxy3 + extra_hosts: + - host.docker.internal:host-gateway + environment: + BEAM_PROXY_URL: http://host.docker.internal:4003 + BEAM_APP_ID_LONG: focus.proxy3.broker + API_KEY: pass123 + ENDPOINT_TYPE: blaze + BLAZE_URL: http://host.docker.internal:8081/fhir/ + OBFUSCATE: no + + blaze-bbmri: + image: samply/blaze:0.32 + ports: + - 8081:8080 + healthcheck: + test: curl -f http://localhost:8080/fhir/metadata + start_period: 1m + + test-data-loader-bbmri: + image: curlimages/curl:latest + depends_on: + blaze-bbmri: + condition: service_healthy + volumes: + - ./test-bundle-bbmri.json:/test-bundle-bbmri.json:ro + entrypoint: ["curl", "-sf", "-X", "POST", "http://blaze-bbmri:8080/fhir", "-H", "Content-Type: application/fhir+json", "-d", "@/test-bundle-bbmri.json"] + + tester: + image: alpine:latest + network_mode: host + depends_on: + test-data-loader-miabis: + condition: service_completed_successfully + test-data-loader-bbmri: + condition: service_completed_successfully + volumes: + - ./test.sh:/test.sh:ro + command: sh -c 'apk add -q curl jq bash && bash /test.sh' + +include: + - ../compose.localbeam.yaml diff --git a/bbmri-miabis/test-bundle-bbmri.json b/bbmri-miabis/test-bundle-bbmri.json new file mode 100644 index 0000000..e455d51 --- /dev/null +++ b/bbmri-miabis/test-bundle-bbmri.json @@ -0,0 +1,114 @@ +{ + "resourceType": "Bundle", + "type": "transaction", + "entry": [ + { + "fullUrl": "urn:uuid:bbmri-patient-1", + "resource": { + "resourceType": "Patient", + "id": "bbmri-patient-1", + "gender": "female", + "birthDate": "1972-05-10" + }, + "request": { "method": "PUT", "url": "Patient/bbmri-patient-1" } + }, + { + "fullUrl": "urn:uuid:bbmri-patient-2", + "resource": { + "resourceType": "Patient", + "id": "bbmri-patient-2", + "gender": "male", + "birthDate": "1968-11-03" + }, + "request": { "method": "PUT", "url": "Patient/bbmri-patient-2" } + }, + { + "fullUrl": "urn:uuid:bbmri-condition-1", + "resource": { + "resourceType": "Condition", + "id": "bbmri-condition-1", + "subject": { "reference": "Patient/bbmri-patient-1" }, + "code": { + "coding": [ + { "system": "http://hl7.org/fhir/sid/icd-10", "code": "C50" } + ] + } + }, + "request": { "method": "PUT", "url": "Condition/bbmri-condition-1" } + }, + { + "fullUrl": "urn:uuid:bbmri-condition-2", + "resource": { + "resourceType": "Condition", + "id": "bbmri-condition-2", + "subject": { "reference": "Patient/bbmri-patient-2" }, + "code": { + "coding": [ + { "system": "http://hl7.org/fhir/sid/icd-10", "code": "C61" } + ] + } + }, + "request": { "method": "PUT", "url": "Condition/bbmri-condition-2" } + }, + { + "fullUrl": "urn:uuid:bbmri-specimen-1", + "resource": { + "resourceType": "Specimen", + "id": "bbmri-specimen-1", + "subject": { "reference": "Patient/bbmri-patient-1" }, + "type": { + "coding": [ + { + "system": "https://fhir.bbmri.de/CodeSystem/SampleMaterialType", + "code": "blood-plasma" + } + ] + }, + "extension": [ + { + "url": "https://fhir.bbmri.de/StructureDefinition/StorageTemperature", + "valueCodeableConcept": { + "coding": [ + { + "system": "https://fhir.bbmri.de/CodeSystem/StorageTemperature", + "code": "temperatureLN" + } + ] + } + } + ] + }, + "request": { "method": "PUT", "url": "Specimen/bbmri-specimen-1" } + }, + { + "fullUrl": "urn:uuid:bbmri-specimen-2", + "resource": { + "resourceType": "Specimen", + "id": "bbmri-specimen-2", + "subject": { "reference": "Patient/bbmri-patient-2" }, + "type": { + "coding": [ + { + "system": "https://fhir.bbmri.de/CodeSystem/SampleMaterialType", + "code": "whole-blood" + } + ] + }, + "extension": [ + { + "url": "https://fhir.bbmri.de/StructureDefinition/StorageTemperature", + "valueCodeableConcept": { + "coding": [ + { + "system": "https://fhir.bbmri.de/CodeSystem/StorageTemperature", + "code": "temperatureRoom" + } + ] + } + } + ] + }, + "request": { "method": "PUT", "url": "Specimen/bbmri-specimen-2" } + } + ] +} diff --git a/bbmri-miabis/test-bundle.json b/bbmri-miabis/test-bundle.json new file mode 100644 index 0000000..48908ac --- /dev/null +++ b/bbmri-miabis/test-bundle.json @@ -0,0 +1,185 @@ +{ + "resourceType": "Bundle", + "type": "transaction", + "entry": [ + { + "fullUrl": "urn:uuid:patient-1", + "resource": { + "resourceType": "Patient", + "id": "patient-1", + "gender": "female", + "birthDate": "1980-03-15" + }, + "request": { "method": "PUT", "url": "Patient/patient-1" } + }, + { + "fullUrl": "urn:uuid:patient-2", + "resource": { + "resourceType": "Patient", + "id": "patient-2", + "gender": "male", + "birthDate": "1975-07-22" + }, + "request": { "method": "PUT", "url": "Patient/patient-2" } + }, + { + "fullUrl": "urn:uuid:condition-1", + "resource": { + "resourceType": "Condition", + "id": "condition-1", + "subject": { "reference": "Patient/patient-1" }, + "code": { + "coding": [ + { + "system": "http://hl7.org/fhir/sid/icd-10", + "code": "C34" + } + ] + }, + "onsetDateTime": "2015-06-01" + }, + "request": { "method": "PUT", "url": "Condition/condition-1" } + }, + { + "fullUrl": "urn:uuid:condition-2", + "resource": { + "resourceType": "Condition", + "id": "condition-2", + "subject": { "reference": "Patient/patient-2" }, + "code": { + "coding": [ + { + "system": "http://hl7.org/fhir/sid/icd-10", + "code": "C18" + } + ] + }, + "onsetDateTime": "2018-11-10" + }, + "request": { "method": "PUT", "url": "Condition/condition-2" } + }, + { + "fullUrl": "urn:uuid:specimen-1", + "resource": { + "resourceType": "Specimen", + "id": "specimen-1", + "subject": { "reference": "Patient/patient-1" }, + "extension": [ + { + "url": "https://fhir.bbmri-eric.eu/StructureDefinition/miabis-sample-collection-extension", + "valueIdentifier": { "value": "collection-a" } + } + ], + "type": { + "coding": [ + { + "system": "https://fhir.bbmri-eric.eu/CodeSystem/miabis-detailed-samply-type-cs", + "code": "Plasma" + } + ] + }, + "processing": [ + { + "extension": [ + { + "url": "https://fhir.bbmri-eric.eu/StructureDefinition/miabis-sample-storage-temperature-extension", + "valueCodeableConcept": { + "coding": [ + { + "system": "https://fhir.bbmri-eric.eu/CodeSystem/miabis-storage-temperature-cs", + "code": "LN" + } + ] + } + } + ] + } + ], + "collection": { "collectedDateTime": "2015-06-05" } + }, + "request": { "method": "PUT", "url": "Specimen/specimen-1" } + }, + { + "fullUrl": "urn:uuid:specimen-2", + "resource": { + "resourceType": "Specimen", + "id": "specimen-2", + "subject": { "reference": "Patient/patient-1" }, + "extension": [ + { + "url": "https://fhir.bbmri-eric.eu/StructureDefinition/miabis-sample-collection-extension", + "valueIdentifier": { "value": "collection-a" } + } + ], + "type": { + "coding": [ + { + "system": "https://fhir.bbmri-eric.eu/CodeSystem/miabis-detailed-samply-type-cs", + "code": "TissueFixed" + } + ] + }, + "processing": [ + { + "extension": [ + { + "url": "https://fhir.bbmri-eric.eu/StructureDefinition/miabis-sample-storage-temperature-extension", + "valueCodeableConcept": { + "coding": [ + { + "system": "https://fhir.bbmri-eric.eu/CodeSystem/miabis-storage-temperature-cs", + "code": "RT" + } + ] + } + } + ] + } + ], + "collection": { "collectedDateTime": "2015-06-05" } + }, + "request": { "method": "PUT", "url": "Specimen/specimen-2" } + }, + { + "fullUrl": "urn:uuid:specimen-3", + "resource": { + "resourceType": "Specimen", + "id": "specimen-3", + "subject": { "reference": "Patient/patient-2" }, + "extension": [ + { + "url": "https://fhir.bbmri-eric.eu/StructureDefinition/miabis-sample-collection-extension", + "valueIdentifier": { "value": "collection-b" } + } + ], + "type": { + "coding": [ + { + "system": "https://fhir.bbmri-eric.eu/CodeSystem/miabis-detailed-samply-type-cs", + "code": "WholeBlood" + } + ] + }, + "processing": [ + { + "extension": [ + { + "url": "https://fhir.bbmri-eric.eu/StructureDefinition/miabis-sample-storage-temperature-extension", + "valueCodeableConcept": { + "coding": [ + { + "system": "https://fhir.bbmri-eric.eu/CodeSystem/miabis-storage-temperature-cs", + "code": "LN" + } + ] + } + } + ] + } + ], + "collection": { "collectedDateTime": "2018-11-15" } + }, + "request": { "method": "PUT", "url": "Specimen/specimen-3" } + } + ] +} diff --git a/bbmri-miabis/test.sh b/bbmri-miabis/test.sh new file mode 100644 index 0000000..e19bfc6 --- /dev/null +++ b/bbmri-miabis/test.sh @@ -0,0 +1,112 @@ +#!/usr/bin/env bash +set -euo pipefail + +SPOT_URL="http://localhost:8055" +PASS=0 +FAIL=0 + +wait_for_spot() { + echo -n "Waiting for Spot..." + until curl -sf -o /dev/null "${SPOT_URL}/health" 2>/dev/null; do + printf '.'; sleep 2 + done + echo " ready." +} + +query_total() { + local ast_obj="$1" + local id="$2" + local payload query + payload=$(jq -cn --arg id "${id}" --argjson ast "${ast_obj}" '{"ast":$ast,"id":$id}' | base64 | tr -d '\n') + query=$(jq -cn --arg p "${payload}" '{"lang":"ast","payload":$p}' | base64 | tr -d '\n') + curl -sf -m 10 \ + -X POST "${SPOT_URL}/beam" \ + -H "Content-Type: application/json" \ + -d "$(jq -cn --arg q "${query}" --arg id "${id}" '{"query":$q,"id":$id}')" \ + || return 1 + curl -sf -m 30 -N \ + "${SPOT_URL}/beam/${id}" \ + | sed -n 's/^data: //p' \ + | jq -r 'if .status == "succeeded" then (.body | @base64d | fromjson | .totals.patient) else 0 end // 0' \ + | awk '{s+=$1} END {print s+0}' +} + +run_scenario() { + local desc="$1" + local ast_obj="$2" + local id="$3" + local expected="$4" + + local actual; actual=$(query_total "${ast_obj}" "${id}" || echo "") + + if [[ "${actual}" == "${expected}" ]]; then + printf " ok %s (total=%s)\n" "${desc}" "${actual}" + PASS=$((PASS + 1)) + else + printf " FAIL %s (expected=%s, got=%s)\n" "${desc}" "${expected}" "${actual}" + FAIL=$((FAIL + 1)) + fi +} + +wait_for_spot +echo "" + +echo "=== Mixed-node scenarios (MIABIS + BBMRI.de, total across both sites) ===" +echo "" + +run_scenario \ + "empty AST — all patients (2 MIABIS + 2 BBMRI.de)" \ + '{"operand":"AND","children":[]}' \ + "00000000-0000-0000-0000-000000000000" \ + 4 + +run_scenario \ + 'storage_temperature = temperatureRoom (MIABIS RT: p1=1, BBMRI.de Room: bbmri-p2=1)' \ + '{"operand":"OR","children":[{"operand":"AND","children":[{"operand":"OR","children":[{"key":"storage_temperature","type":"EQUALS","system":"","value":"temperatureRoom"}]}]}]}' \ + "00000000-0000-0000-0000-000000000001" \ + 2 + +run_scenario \ + 'storage_temperature = temperatureLN (MIABIS LN: p1+p2=2, BBMRI.de LN: bbmri-p1=1)' \ + '{"operand":"OR","children":[{"operand":"AND","children":[{"operand":"OR","children":[{"key":"storage_temperature","type":"EQUALS","system":"","value":"temperatureLN"}]}]}]}' \ + "00000000-0000-0000-0000-000000000002" \ + 3 + +run_scenario \ + 'sample_kind = blood-plasma (MIABIS Plasma: p1=1, BBMRI.de blood-plasma: bbmri-p1=1)' \ + '{"operand":"OR","children":[{"operand":"AND","children":[{"operand":"OR","children":[{"key":"sample_kind","type":"EQUALS","system":"","value":"blood-plasma"}]}]}]}' \ + "00000000-0000-0000-0000-000000000003" \ + 2 + +run_scenario \ + 'sample_kind = whole-blood (MIABIS WholeBlood: p2=1, BBMRI.de whole-blood: bbmri-p2=1)' \ + '{"operand":"OR","children":[{"operand":"AND","children":[{"operand":"OR","children":[{"key":"sample_kind","type":"EQUALS","system":"","value":"whole-blood"}]}]}]}' \ + "00000000-0000-0000-0000-000000000004" \ + 2 + +echo "" +echo "=== BBMRI.de node unaffected by MIABIS workarounds ===" +echo "" + +run_scenario \ + 'diagnosis C61 (MIABIS: 0, BBMRI.de: bbmri-p2=1)' \ + '{"operand":"OR","children":[{"operand":"AND","children":[{"operand":"OR","children":[{"key":"diagnosis","type":"EQUALS","system":"","value":"C61"}]}]}]}' \ + "00000000-0000-0000-0000-000000000005" \ + 1 + +run_scenario \ + 'diagnosis C50 (MIABIS: 0, BBMRI.de: bbmri-p1=1)' \ + '{"operand":"OR","children":[{"operand":"AND","children":[{"operand":"OR","children":[{"key":"diagnosis","type":"EQUALS","system":"","value":"C50"}]}]}]}' \ + "00000000-0000-0000-0000-000000000006" \ + 1 + +run_scenario \ + 'diagnosis C34 (MIABIS: p1=1, BBMRI.de: 0)' \ + '{"operand":"OR","children":[{"operand":"AND","children":[{"operand":"OR","children":[{"key":"diagnosis","type":"EQUALS","system":"","value":"C34"}]}]}]}' \ + "00000000-0000-0000-0000-000000000007" \ + 1 + +echo "" +echo "Results: ${PASS} passed, ${FAIL} failed" +echo "" +[ "${FAIL}" -eq 0 ] || exit 1