[Bug] [SLURM] sbatch errors hang executor

[pmrv]

If a user (ie. me :() e.g. makes a mistake in the resource_dict when using SlurmClusterExecutor, then sbatch will crash which will make the submitting thread of executorlib print a stacktrace saying so, but without the output from sbatch. That's awkward to debug, because afaict executorlib also doesn't write the output to a logfile. So to debug this situtation, I end up going manually to the executor cache directory, sbatch'ing the run script and then taking the log from there to debug. If executorlib would print this directly, it'd be much more helpful.

There's also a tangential issue, where if the situation occurs, but I'm wait()'ing on a future the executor returns, my notebook hangs, not getting interrupted by the failure inside the executor. This kinda annoying because if I manually interrupt my kernel at this point, the whole kernel crashes. I'm not sure how much control executorlib has over this part though.

[pmrv]

MWE as in the other issue:

submission_template = """\
#!/bin/bash
#SBATCH --output=time.out
#SBATCH --job-name={{job_name}}
#SBATCH --chdir={{working_directory}}
#SBATCH --get-user-env=L
#SBATCH --partition={{partition}}
{%- if run_time_max %}
#SBATCH --time={{ [1, run_time_max // 60]|max }}
{%- endif %}
{%- if dependency %}
#SBATCH --dependency=afterok:{{ dependency | join(',') }}
{%- endif %}
{%- if memory_max %}
#SBATCH --mem={{memory_max}}G
{%- endif %}
#SBATCH --ntasks={{cores}}

{{command}}
"""

with SlurmClusterExecutor(
    cache_directory="./executorcache",
    resource_dict={
        "submission_template": submission_template, 
        "run_time_max": 2*60*60,  # in seconds 
        "cores": 8,
        "partition": "MYNONEXISTINGPARTITION",         # <---------------- example for a wrong config
    },
    pmi_mode="pmix",
) as exe:
    future = exe.submit(
        mpi_sum, [1,2]
    )
    print(future.result())