Make container for ANNDATAR_CONVERT.nf have packages preinstalled on it

[tzvi909]

Description of the bug

Hi all,

I understand the motivation for having the docker.io-nfcore-anndatar-20241129.img container that's used to run process: ANNDATAR_CONVERT (as a singularity/apptainer one at least) as a lightweight one.

However, I've noticed that it seems to rely on the user's R_lib path to grab the necessary libraries when running templates/anndatar_convert.R inside anndatar_convert.nf as an Apptainer/Singularity container on SLURM. This is great if everyone is using R v.4.2.2, however, if not (and this cannot be modified on your HPC cluster) then you obviously run into some errors importing dependencies for Seurat and SingleCellExperiment, particularly: Matrix and spam.

Right now, the container I've recreated using the dependencies of the current one is ~893MB so should be no harder to cache than the largest container in the pipeline right now (cellbender).

TIA.

Command used and terminal output

#!/bin/bash

## dependencies
module purge; module load bluebear
module load bear-apps/2022b
module load Nextflow/24.04.2


export NXF_SINGULARITY_CACHEDIR="/path/to/.apptainer"

#main

### cmd used before patching the aforementioned container
### used local pipeline dir after patching on my cluster
### cellranger_index comes straight from 10x website

nextflow run nf-core/scrnaseq -r 4.0.0 \
    --input ../samplesheet.csv \
    --outdir ../Outs_QC_and_counts_Mouse \
    -profile bluebear \
    --aligner cellranger \
    --cellranger_index ../../Refs/refdata-gex-GRCm39-2024-A

Relevant files

anndatar_convert_singularity_recipe.zip

This zipfile contains the apptainer/singularity .def recipe I have created.

Cheers.

System information

Nextflow version: 24.04.2
Hardware: HPC
Executor: Slurm
Container engine: Apptainer (Singularity)
OS: Red Hat Enterprise Linux 8.10 (Ootpa)
Version of nf-core/scrnaseq: 4.0.0

[grst]

Hi @tzvi909,

thanks for reporting. The pipeline should NEVER use anything outside the container, that defies the purpose of having the container in the first place. I'm also relatively sure that the original container contains everything that's required, otherwise it wouldn't run on the CI runner, which doesn't have an R installation on its own.

So I assume the root cause is something else here, e.g. your $HOME directory being mounted into the singularity container. Even though this should not happen in nextflow 24.04 anymore by default: nextflow-io/nextflow#4056

[tzvi909]

This may be true but because I managed to remake the container locally and reupload it to my hpc, point the module to it, I think there's an issue with its "self-containment". you might not be able to reproduce this on the ci because you don't already have a conflicting version of R installed. Therefore it does seem like an issue with the container recipe. I've not tested it on Docker but the current one in the repo definitely breaks with apptainer/singularity on my hpc cluster. The recipe I've made is relatively simple too.

[grst]

I think there's an issue with its "self-containment".

Yes, but how does your conflicting version of R end up in the container? It should not, and this is the root cause we should attempt to fix.

[tzvi909]

I'd need to see the container recipe you're using to be able to figure that one out. Can you link me to it?

…

On Mon, 2 Jun 2025, 14:14 Gregor Sturm, ***@***.***> wrote: *grst* left a comment (nf-core/scrnaseq#470) <#470 (comment)> I think there's an issue with its "self-containment". Yes, but how does your conflicting version of R end up in the container? It should not, and this is the root cause we should attempt to fix. — Reply to this email directly, view it on GitHub <#470 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWQ5R74RH2RPHAI7XYLP6433BRE3LAVCNFSM6AAAAAB6KLRCESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMZQGY2TQMRXGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[grst]

@fmalmeida, do you have the Dockerfile for the anndataR module? I couldn't find it in the pipeline, nor in this PR: https://github.com/nf-core/scrnaseq/pull/369/files

[fmalmeida]

Hi @grst ,
I will try to find. But it was a really simple one, starting with rocker R public image and then running the anndataR (pak::[pak](https://pak.r-lib.org/reference/pak.html)("scverse/anndataR")) and seurat package.

@apeltzer , I think it was you that did the build&push for the nf-core registry. Do you happen to have it?

[fmalmeida]

but to report to the issue, yes, it should not expect anything from outside, that is why it runs in the CI ... I think it may be something related to the global variables of the R home, $HOME, or something similar (e.g. the ones here).

[fmalmeida]

Hi @grst and @tzvi909 ,
This is the Dockerfile I had used in gitpod when testing the implementation, before the nf-core build&push:

FROM r-base

RUN apt-get update -y
RUN R -e 'install.packages(c("devtools", "BiocManager", "SingleCellExperiment", "SeuratObject"))'
RUN R -e 'devtools::install_github("scverse/anndataR")'

[tzvi909]

Ah strange guess you're all right then. It must be that singularity is incorrectly trying to mount $home

…

On Tue, 3 Jun 2025, 07:42 Felipe Marques de Almeida, < ***@***.***> wrote: *fmalmeida* left a comment (nf-core/scrnaseq#470) <#470 (comment)> Hi @grst <https://github.com/grst> and @tzvi909 <https://github.com/tzvi909> , This is the Dockerfile I had used in gitpod when testing the implementation, before the nf-core build&push: FROM r-base RUN apt-get update -y RUN R -e 'install.packages(c("devtools", "BiocManager", "SingleCellExperiment", "SeuratObject"))' RUN R -e 'devtools::install_github("scverse/anndataR")' — Reply to this email directly, view it on GitHub <#470 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWQ5R7Y6QCEFKSZN2NSA5G33BU7UHAVCNFSM6AAAAAB6KLRCESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMZTG4ZTKOJSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tzvi909]

so what can be done here then as this isn't ideal?

[grst]

Can you share the .command.run file of the corresponding process?

[fmalmeida]

Creating a branch to work on this ticket and replace #472 .

Not sure if it actually make sense to do it here, or this ticket should be closed and a "container update to use conda package" ticket should be raised.