feat(nf-google): add configurable CLI staging transports and safer default unstaging for Google Batch#7031
Draft
tomiles wants to merge 6 commits intonextflow-io:masterfrom
Draft
Conversation
…, gcloud, gsutil) Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
…lized CLI transport detection Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
…out mode Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
✅ Deploy Preview for nextflow-docs-staging ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR addresses #5653 by adding optional Google Batch stage-in/stage-out copy transports (
posix,gcloud,gsutil) and aligning transfer behavior with other cloud executors while preserving backward-compatible defaults.This design allows granular optimization of staging behavior at the process level (via
stageInMode/stageOutMode), rather than forcing a single workflow-wide override. In practice, only processes that usecopyfor stage-in or stage-out (stage-out defaults tocopyin Google Batch when unset) and have the CLI tools in their runtime environment will use the new transport path, while other modes keep existing behavior.What changed
google.batch.stageInCopyTransportandgoogle.batch.stageOutCopyTransportto select override for transfer transport incopymode.google.batch.gcloudCligoogle.batch.gsutilCligoogle.batch.maxParallelTransfersgoogle.batch.maxTransferAttemptsgoogle.batch.delayBetweenAttemptscopywhenstageOutModeis unset to avoidmovefailures in gcsfuse-mounted work-dir workflows (e.g. overlapping outputs/symlinked paths).Docs
docs/google.mdwith the new transport options and default unstaging rationale.docs/reference/config.mdwith all newgoogle.batch.*config entries and defaults.Acknowledgements
Thanks to @dmvanbeek and @rhassaine for input and hackathon discussions that helped shape this approach.