Poor File Staging/Unstaging Performance with Google Batch Executor

[KatieAtGordian]

Bug report

I am documenting this problem as a known issue with discussion here in slack.

There is no plan to support the fix to this issue (per slack discussion, the solution is "too clunky"), but I would like to document it so that others encountering this may chime in and/or save some time debugging why their google batch pipelines are taking so long to complete in comparison to other executors.

Expected behavior and actual behavior

Expected behavior:
File staging and unstaging using the default, built-in support for the Google Batch executor should complete in a reasonable amount of time. "Reasonable" means transfer speeds comparable to standard GCP file transfer achieved by using, e.g., gcloud storage or gsutil.

Actual behavior:
File unstaging/delocalization using the built-in support in the Google Batch executor is extremely slow, e.g., taking ~11 hours for 600GB of data. This is because it's using gcsfuse, which is significantly slower than gsutil which is used by the Google LifeSciences executor (via the nxf_gs_upload function in .command.run).

The officially recommended solution is to use Wave/Fusion; however, this solution is not appropriate for all users, especially those who cannot support the injection of 3rd party services into their workflows.

This is problematic especially because (1) poor performance is essentially the default behavior for the executor (as in, wave/fusion are the recommended solution but are not enabled by default), and (2) this reflects a disparity with the lifesciences executor, which is deprecated by GCP and will not be available after July 8, 2025.

Steps to reproduce the problem

• Config
  • process.executor = 'google-batch'
  • fusion.enabled = false or omit from config
  • wave.enabled = false or omit from config
• Run any process that generates at least a few GB of data.
  • See the example process below. Running this with 20GB of data takes ~27 mins with defaults and ~12 mins with a gsutil-enabled solution I wrote in a fork (~3-4 mins of this is generating the file).)

Here's an example process for testing purposes that just writes a file of the given size.

process DUMMY_WRITE {
    label 'process_single'
    disk { 2 * file_size_gb as nextflow.util.MemoryUnit * 1024**3 }
    

    input:
    val file_size_gb

    output:
    path 'dummy_dir/dummy.txt', emit: ch_dummy

    publishDir (
        path: "${params.publish_dir}/",
        mode: 'copy',
        )

    script:
    """
    # Write a file of size file_size_gb
    echo "Writing a file of size ${file_size_gb}GB."
    mkdir dummy_dir/
    dd if=/dev/zero of=dummy_dir/dummy.txt bs=1G count=${file_size_gb} 
    echo "Done writing file."
    """
}

Program output

Here are the performance characteristics that caused me to look into this issue. I have a workflow that performs the following steps:

1. Localizes data:

• Method: Manually staged using gsutil due to the known bucket underscore issue. See related issues: #3619, #1069, #1527.
• Size: ~300 GB.
• Duration: ~8 minutes.

2. Runs some code:

• Duration: ~2 hours.

3. Delocalizes data:

• Method: Using Nextflow's built-in gcsfuse support. Files are moved to the workdir only (no publishing).
• Size: ~600 GB.
• Duration: ~11 hours.
• Comparable upload using gsutil takes ~15 minutes.

Environment

• Nextflow version: 24.10.0
• Java version: openjdk 11.0.25 2024-10-15
• Operating system: Linux
• Bash version: zsh 5.8.1 (x86_64-ubuntu-linux-gnu)

Additional context

See this slack discussion for additional context.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tomiles]

We seem to be hitting similar issues in data-heavy tasks with the stageOut/In that uses cp/mv/rclone/rsync + gcsfuse. Staging seems to take a disproportionately longer time compared to the actual analysis time. Also, this staging system specific to the Google Batch Executor plugin seems inconsistent with the AWS/Azure implementations, where the respective CLI tools are used to do up and download to object storage buckets. We are still running some more detailed performance tests to check the transfer speed impact for the different approaches.

But it would be useful to know why this distinct design choice was made for Google Batch implementation versus how this was handled in AWS/Azure plugins. The fork @KatieAtGordian made, that (re)introduces nxf_gs_upload/download functionally analogous to nxf_az_upload/download and nxf_s3_upload/download does seem like a more consistent strategy.

[KatieAtGordian]

@tomiles Thanks for bumping this.

I know you weren't asking me, and of course I don't actually know the story, but from looking at it forensically it feels like an experiment to simplify and use more of the core nextflow functionality for moving files (seen in the basic mv backed by gcsfuse rather than nxf_gs_upload, no special GoogleBatchFileCopyStrategy, etc.).

This approach makes it much, much harder to tune performance because there's no good way to expose gcsfuse parameters or replace the upload utility entirely. I also believe that maintaining this side has been more or less abandoned in favor of fusion/wave, so I'm not sure we'll see any motion here. Truly haven't seen a downside to the nxf_gs_upload approach aside from code complexity though.

---

Sharing some additional details in case this is useful to you or others.

I ended up implementing a solution using a "new" executor/plugin nf-google-mycompany that duplicates the nf-google plugin (the one that's built in to nextflow itself). This blends the batch & lifesciences approach via nxf_gs_upload and also adds a helper function to handle underscores in paths (below), while making it easier to maintain (aka actually buildable on its own w/o having to build all of nextflow).

It's not public because I was messing around with internal plugin publishing & would need some real cleanup to strip that. Also not being kept up to date regularly.

This is totally in the wrong place, but here's a snippet for the underscore issue. Basically everywhere that the plugin calls Escape.uriPath() I use this method instead. There may be limitations to this but it sidesteps java's uri validator and that's all I needed.

/**
     * Helper method to safely handle GCS paths with underscores in bucket names
     * Avoids the Java URI validation that fails with underscores
     */
    static String safeGsPath(String path) {
        if (path == null) return ''
        String pathStr = path.toString()
        // If it's a GCS path, handle it directly as a string to avoid URI parsing
        if (pathStr.startsWith('gs://')) {
            // Escape any spaces or special characters, but preserve the underscore
            return "\"${pathStr.replace('\"', '\\\"')}\""
        }
        // For non-GCS paths, use the standard Escape.path method
        return Escape.path(path)
    }

[pditommaso]

I replicated the test comparing gcsfuse vs Fusion using the config below, and the results confirm the poor gcsfuse performance.

• gcsfuse: 23m 41s
• Fusion: 10m 21s

The rationale for using gcsfuse as default storage for the Google Batch executor is that it's the built-in solution provided by the Batch service. This allows accessing the data storage bucket without having to pre-install other tools in the container or on the running VM, making deployment much simpler.

Overall my take is that this is an issue on Google's side, not Nextflow.

/*
 * Pipeline parameters
 */
params {
    file_size_gb = 100
    publish_dir = 'gs://rnaseq-nf/scratch/results'
}

/*
 * Google Cloud configuration
 */
google {
    location = 'europe-west1'
}

/*
 * Executor configuration for Google Batch
 */
process {
    executor = 'google-batch'
    container = 'ubuntu:24.04'

    cpus = 4
    memory = '16 GB'
    machineType = 'n2d-standard-4'
    disk = [request: '375 GB', type: 'local-ssd']
}

[KatieAtGordian]

I can see why the built-in solution is preferred at first glance. Ultimately, though, I think the potential solutions are more reasonable than accepting poor performance. And surprisingly attainable.

• Idea 1: use startup scripts when launching VMs (--> in order to use gsutil instead of gcsfuse)

  • More detail: Add a startup script that automatically installs the google cloud SDK when nextflow launches a VM (as recommended by google, using their provided installation script). This is just an additional argument at launch.
  • Pros: Allows the nextflow-style approach of using the same container independent of executor
  • Cons: Installation can potentially fail (highest risk when the container uses out of date / unsupported python versions)

• Idea 2: Only use gsutil or gcloud storage if available, and fall back to simple fs operations (i.e., gcsfuse) if gsutil is unavailable

  • Pro & con: makes the user responsible for providing compatible containers or instance templates in exchange for optimized performance (I would be very happy to accept this tradeoff). Could be a BYO startup script or entire container with the SDK pre-installed.

• Idea 3: Use explicit profile variables to toggle gsutil & SDK installation

  • Add google.batch.enableGsutil = true to turn on gsutil for file transfer (similar to what exists for wave/fusion), and an optional google.batch.installSDK (same pattern as GPU drivers) if you want it installed at runtime for you.
  • Pro & con: Same as 2, but explicit control matches existing nextflow patterns and should make troubleshooting more straightforward. Will fail if google.batch.enableGsutil = true but gsutil is unavailable, which could be confusing, but this is the same behavior as fusion when not configured.

All of these would require some dev work to reintroduce the nxf_upload parallelism with other executors.

[dchaley-mayo]

@pditommaso , nice to e-meet you, I'm a software engineer working with Mayo Clinic & a Google Developer Expert as well. I'm working on scaling Nextflow pipelines.

The rationale for using gcsfuse as default storage

The rationale makes sense (use what's guaranteed to be available). And it's true that gcsfuse simply has poor performance at scale. That said, the google-batch executor doesn't use gcsfuse as the "default" but the "only" storage mechanism. So, users who follow standard procedure get really bad performance & have to pass paths as raw strings and handle transfer themselves to get acceptable transfer times.

I think @KatieAtGordian 's idea #3 provides a Nextflow-consistent, opt-in optimization for people who know their container is configured correctly. While preserving the default using gcsfuse mounts.

1. Create a GoogleStorageFileCopyStrategy that would be very similar to the Aws and Az versions.
2. Add new constructor to GoogleBatchScriptLauncher accepting a file copy strategy to pass to super (the BashWrapperBuilder)
3. Add new config flags google.batch.useGoogleStorageStrategy & google.batch.installGcloudSdk
4. If the strategy config flag is true, the GoogleBatchTaskHandler will create the GoogleBatchScriptLauncher which at last will pass the new copy strategy to the BashWrapperBuilder constructor.
5. When staging the task, the GoogleStorageFileCopyStrategy will use gcloud-storage (Google's recommended tool)
6. If the strategy install flag is true, the task handler will attempt to follow the official installation instructions

Error experience can be improved by detecting the cmdline util at the start of the process, and printing an explicit message that the flag may only be used when the tool is present in the runtime container.

I plan on packaging this up as a plugin but believe it could be in the core too & I'd be happy to contribute a PR. Do you think a PR following the above approach could be reviewed or is this approach not viable?
Thanks 😁

[tomiles]

We discussed staging optimizations for the Google Batch plugin at the last hackathon, and this draft PR is now open for comments, #7031. It should address most of the concerns raised. The only thing we haven’t implemented is installing CLI tooling as part of the plugin. We’ve decided to leave that to the user’s runtime side because Google’s ‘gcloud’ tooling has Python dependencies that aren’t as minimal to quickly inject into existing containers compared to other cloud providers. Perhaps there’s demand for this, and we could still add it in a separate later commit. However, we should be able to handle it with a proper process container build or inject it with Wave.