Skip to content

feat(nf-google): add configurable CLI staging transports and safer default unstaging for Google Batch#7031

Draft
tomiles wants to merge 6 commits intonextflow-io:masterfrom
tomiles:fix/google-batch-gloud-staging
Draft

feat(nf-google): add configurable CLI staging transports and safer default unstaging for Google Batch#7031
tomiles wants to merge 6 commits intonextflow-io:masterfrom
tomiles:fix/google-batch-gloud-staging

Conversation

@tomiles
Copy link
Copy Markdown

@tomiles tomiles commented Apr 14, 2026

This PR addresses #5653 by adding optional Google Batch stage-in/stage-out copy transports (posix, gcloud, gsutil) and aligning transfer behavior with other cloud executors while preserving backward-compatible defaults.

This design allows granular optimization of staging behavior at the process level (via stageInMode / stageOutMode), rather than forcing a single workflow-wide override. In practice, only processes that use copy for stage-in or stage-out (stage-out defaults to copy in Google Batch when unset) and have the CLI tools in their runtime environment will use the new transport path, while other modes keep existing behavior.

What changed

  • Added google.batch.stageInCopyTransport and google.batch.stageOutCopyTransport to select override for transfer transport in copy mode.
  • Added Google Batch transfer options:
    • google.batch.gcloudCli
    • google.batch.gsutilCli
    • google.batch.maxParallelTransfers
    • google.batch.maxTransferAttempts
    • google.batch.delayBetweenAttempts
  • Added/updated Google Batch bash staging helpers with CLI tool first behavior and POSIX fallback.
  • Folded in the Default stageOutMode to 'copy' for Google Batch executor #6917 behavior: default Google Batch unstaging to copy when stageOutMode is unset to avoid move failures in gcsfuse-mounted work-dir workflows (e.g. overlapping outputs/symlinked paths).

Docs

  • Updated docs/google.md with the new transport options and default unstaging rationale.
  • Updated docs/reference/config.md with all new google.batch.* config entries and defaults.

Acknowledgements

Thanks to @dmvanbeek and @rhassaine for input and hackathon discussions that helped shape this approach.

tomiles added 6 commits April 14, 2026 10:57
…, gcloud, gsutil)

Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
…lized CLI transport detection

Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
…out mode

Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
@netlify
Copy link
Copy Markdown

netlify bot commented Apr 14, 2026

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit b48e886
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69de0ff9d4b755000931b623
😎 Deploy Preview https://deploy-preview-7031--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants