[Bug] How to use SlurmClusterExecutor

[pmrv]

I'm trying to follow the docs over here

submission_template = """\
#!/bin/bash
#SBATCH --output=time.out
#SBATCH --job-name={{job_name}}
#SBATCH --chdir={{working_directory}}
#SBATCH --get-user-env=L
#SBATCH --partition={{partition}}
{%- if run_time_max %}
#SBATCH --time={{ [1, run_time_max // 60]|max }}
{%- endif %}
{%- if dependency %}
#SBATCH --dependency=afterok:{{ dependency | join(',') }}
{%- endif %}
{%- if memory_max %}
#SBATCH --mem={{memory_max}}G
{%- endif %}
#SBATCH --cpus-per-task={{cores}}

{{command}}
"""

with SlurmClusterExecutor(cache_directory="./cache") as exe:
    future = exe.submit(
        sum, [4, 4], 
        resource_dict={
            "submission_template": submission_template, 
            "run_time_max": 180,  # in seconds  
            "partition": "s.cmfe",
        })
    print(future.result())

but it prints this warning

[/lib/python3.12/site-packages/executorlib/task_scheduler/base.py:132](https://localhost:8000/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/task_scheduler/base.py#line=131): UserWarning: The following keys are not recognized and cannot be validated: ['submission_template', 'run_time_max', 'partition']
  self._validator(resource_dict=resource_dict)

The code itself runs in that the correct result is returned, but it doesn't actually submit a slurm job and seems to just run on the node the code is run on.

[pmrv]

If I switch over to block allocation with

submission_template = """\
#!/bin/bash
#SBATCH --output=time.out
#SBATCH --job-name={{job_name}}
#SBATCH --chdir={{working_directory}}
#SBATCH --get-user-env=L
#SBATCH --partition={{partition}}
{%- if run_time_max %}
#SBATCH --time={{ [1, run_time_max // 60]|max }}
{%- endif %}
{%- if dependency %}
#SBATCH --dependency=afterok:{{ dependency | join(',') }}
{%- endif %}
{%- if memory_max %}
#SBATCH --mem={{memory_max}}G
{%- endif %}
#SBATCH --cpus-per-task={{cores}}

{{command}}
"""

with SlurmClusterExecutor(
    max_cores=1,
    cache_directory="./cache", 
    resource_dict={
        "submission_template": submission_template, 
        "run_time_max": 180,  # in seconds  
        "partition": "s.cmfe",
    },
    block_allocation=True,
) as exe:
    future = exe.submit(
        sum, [4, 4])
    print(future.result())

I can at least get pysqa to try to submit something, but it then errors out with

Exception in thread Thread-5 (_execute_multiple_tasks):
Traceback (most recent call last):
  File "[/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/threading.py", line 1075](https://localhost:8000/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/threading.py#line=1074), in _bootstrap_inner
    self.run()
  File "[/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/threading.py", line 1012](https://localhost:8000/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/threading.py#line=1011), in run
    self._target(*self._args, **self._kwargs)
  File "[/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/task_scheduler/interactive/blockallocation.py", line 264](https://localhost:8000/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/task_scheduler/interactive/blockallocation.py#line=263), in _execute_multiple_tasks
    interface = interface_bootup(
                ^^^^^^^^^^^^^^^^^
  File "[/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/standalone/interactive/communication.py", line 229](https://localhost:8000/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/standalone/interactive/communication.py#line=228), in interface_bootup
    interface.bootup(
  File "[/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/standalone/interactive/communication.py", line 137](https://localhost:8000/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/standalone/interactive/communication.py#line=136), in bootup
    if not self._spawner.bootup(
           ^^^^^^^^^^^^^^^^^^^^^
  File "[/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/task_scheduler/interactive/spawner_pysqa.py", line 96](https://localhost:8000/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/task_scheduler/interactive/spawner_pysqa.py#line=95), in bootup
    self._process = self._start_process_helper(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/task_scheduler/interactive/spawner_pysqa.py", line 194](https://localhost:8000/u/zora/.conda/envs/2026_02_GRACE_MELTING_POINTS_FEP/lib/python3.12/site-packages/executorlib/task_scheduler/interactive/spawner_pysqa.py#line=193), in _start_process_helper
    return queue_adapter.submit_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: pysqa.queueadapter.QueueAdapter.submit_job() got multiple values for keyword argument 'run_time_max'

and hangs the notebook.

[jan-janssen]

The quick fix is to replace "run_time_max" with "run_time_limit" then it works.

The issue is that pysqa uses run_time_max and executorlib introduced run_time_limit in #930 as part of 1.9.0. For consistency I think it would be easier to switch the run_time_limit in executorlib to run_time_max`.

[pmrv]

Thanks, that seems to work better!

This snippets runs and submits now

submission_template = """\
#!/bin/bash
#SBATCH --output=time.out
#SBATCH --job-name={{job_name}}
#SBATCH --chdir={{working_directory}}
#SBATCH --get-user-env=L
#SBATCH --partition={{partition}}
{%- if run_time_max %}
#SBATCH --time={{ [1, run_time_max // 60]|max }}
{%- endif %}
{%- if dependency %}
#SBATCH --dependency=afterok:{{ dependency | join(',') }}
{%- endif %}
{%- if memory_max %}
#SBATCH --mem={{memory_max}}G
{%- endif %}
#SBATCH --cpus-per-task={{cores}}

{{command}}
"""

with SlurmClusterExecutor(
    cache_directory="./cache", 
    resource_dict={
        "submission_template": submission_template, 
        "run_time_limit": 180,  # in seconds  
        "partition": "s.cmfe",
        'cores': 1,
    },
) as exe:
    future = exe.submit(
        sum, [4, 4])
    print(future.result())

for a single core at least. Testing multi core submissions right now.
There's still a warning about submission_template, partition, though they appear to be used correctly:
The following keys are not recognized and cannot be validated: ['submission_template', 'partition']

Will test with the PR in a bit.

[pmrv]

Submission to multiple cores is not working, if I just change cores in the example above, I get
srun: error: Unable to create step for job 19867310: More processors requested than permitted
in time.out file in the cache dir and no error in the notebook.

I thought this may be due to the submission template, when I change cpus-per-task there to --ntasks, the submission runs, I get the expect return value from the future, but the slurm crashes with

  File "/cmmc/u/zora/pyiron/projects/2026/02_GRACE_MELTINGS_POINTS_FEP/software/executorlib/src/executorlib/backend/cache_parallel.py", line 72, in <module>
    main()
  File "/cmmc/u/zora/pyiron/projects/2026/02_GRACE_MELTINGS_POINTS_FEP/software/executorlib/src/executorlib/backend/cache_parallel.py", line 63, in main
    backend_write_file(
  File "/cmmc/u/zora/pyiron/projects/2026/02_GRACE_MELTINGS_POINTS_FEP/software/executorlib/src/executorlib/task_scheduler/file/backend.py", line 47, in backend_write_file
    os.rename(file_name, file_name_out + "_r.h5")
FileNotFoundError: [Errno 2] No such file or directory: '/cmmc/u/zora/pyiron/projects/2026/02_GRACE_MELTINGS_POINTS_FEP/multicache/sumf3976430477e8445fc62baddea187fde_i.h5' -> '/cmmc/u/zora/pyiron/projects/2026/02_GRACE_MELTINGS_POINTS_FEP/multicache/sumf3976430477e8445fc62baddea187fde_r.h5'
srun: error: cmti160: task 1: Exited with exit code 1
srun: Terminating StepId=19867315.0

in time.out.

[jan-janssen]

There's still a warning about submission_template, partition, though they appear to be used correctly:
The following keys are not recognized and cannot be validated: ['submission_template', 'partition']

Yes, partition is a variable the user can choose, you could als call it queue or something else so they cannot be validated. submission_template also cannot be validated, but it is obviously one of the standard keywords, so it should not complain.
#956

I thought this may be due to the submission template, when I change cpus-per-task there to --ntasks, the submission runs, I get the expect return value from the future, but the slurm crashes with

...

in time.out.

Basically, it tries to run the same python script twice rather than using mpi4py to run it in parallel. We can validate this using:

#!/bin/bash
#SBATCH --output=time.out
#SBATCH --get-user-env=L
#SBATCH --partition=s.cmfe
#SBATCH --time=3
#SBATCH --ntasks=2

srun -n 2 /cmmc/ptmp/janj/mambaforge/bin/python test.py

with test.py as:

from mpi4py import MPI
import numpy as np

comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()
print(size, rank)

The result should be 2 0 and 2 1 but instead it returns 1 0 and 1 0.

[pmrv]

Basically, it tries to run the same python script twice rather than using mpi4py to run it in parallel. We can validate this using:

So what you're saying is that this is expected behavior, because we should be using the original cpus-per-task?

My actual use case is closer to how pyiron used to run scripts. I submit a python function that should run on one core in an allocation with multiple cores which would then be picked up by a shell script run from within my python function. But I think it would make sense to discuss also the cases of a mpi4py parallel python script, that's also unclear to me still.

[jan-janssen]

It seems to be an issue with the pmi version, here is the fixed version - the important change pmi_mode="pmix":

submission_template = """\
#!/bin/bash
#SBATCH --output=time.out
#SBATCH --job-name={{job_name}}
#SBATCH --chdir={{working_directory}}
#SBATCH --get-user-env=L
#SBATCH --partition={{partition}}
{%- if run_time_max %}
#SBATCH --time={{ [1, run_time_max // 60]|max }}
{%- endif %}
{%- if dependency %}
#SBATCH --dependency=afterok:{{ dependency | join(',') }}
{%- endif %}
{%- if memory_max %}
#SBATCH --mem={{memory_max}}G
{%- endif %}
#SBATCH --ntasks={{cores}}

{{command}}
"""

with executorlib.SlurmClusterExecutor(
    cache_directory="./cache", 
    resource_dict={
        "submission_template": submission_template, 
        "run_time_limit": 180,  # in seconds  
        "partition": "s.cmfe",
        'cores': 2,
        
    },
    pmi_mode="pmix",
) as exe:
    future = exe.submit(
        sum, [4, 4])
    print(future.result())

[jan-janssen]

My actual use case is closer to how pyiron used to run scripts. I submit a python function that should run on one core in an allocation with multiple cores which would then be picked up by a shell script run from within my python function. But I think it would make sense to discuss also the cases of a mpi4py parallel python script, that's also unclear to me still.

The use case to submit a python function that should run on one core in an allocation with multiple cores, is demonstrated in the quantum espresso example:
https://executorlib.readthedocs.io/en/latest/5-2-quantum-espresso.html

The mpi4py parallel python script is demonstrated based on GPAW:
https://executorlib.readthedocs.io/en/latest/5-1-gpaw.html

With the fix above the mpi4py examples should work fine.

[pmrv]

It seems to be an issue with the pmi version, here is the fixed version - the important change pmi_mode="pmix":

submission_template = """\
#!/bin/bash
#SBATCH --output=time.out
#SBATCH --job-name={{job_name}}
#SBATCH --chdir={{working_directory}}
#SBATCH --get-user-env=L
#SBATCH --partition={{partition}}
{%- if run_time_max %}
#SBATCH --time={{ [1, run_time_max // 60]|max }}
{%- endif %}
{%- if dependency %}
#SBATCH --dependency=afterok:{{ dependency | join(',') }}
{%- endif %}
{%- if memory_max %}
#SBATCH --mem={{memory_max}}G
{%- endif %}
#SBATCH --ntasks={{cores}}

{{command}}
"""

with executorlib.SlurmClusterExecutor(
    cache_directory="./cache", 
    resource_dict={
        "submission_template": submission_template, 
        "run_time_limit": 180,  # in seconds  
        "partition": "s.cmfe",
        'cores': 2,
        
    },
    pmi_mode="pmix",
) as exe:
    future = exe.submit(
        sum, [4, 4])
    print(future.result())

Thanks, this works now, seems that pmix is indeed the issue.

[pmrv]

I also got the script submission working, thanks!