Slurm Job Lifecycle And Resource Optimization

Goal: submit, monitor, diagnose, and right-size Euler jobs without wasting shared resources.

Prerequisites: Euler access, an approved storage plan, and the Euler share policy.

Execution context: all commands on this page run on an Euler login node - Bash, except commands explicitly described as running inside the job.

The Lifecycle#

estimate -> inspect script -> sbatch -> squeue/myjobs -> logs ->
sacct/seff -> adjust next request

The login node is for editing, transfer, compilation where appropriate, and job management. Run computations through Slurm.

Core Commands#

Conservative CPU Template#

Do not specify a partition. Current ETH guidance says to let Slurm select one unless the workload genuinely depends on a particular resource.

#!/usr/bin/env bash
#SBATCH --job-name=cpu-smoke
#SBATCH --account=es_fuge
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=1G
#SBATCH --output=logs/%x_%j.out
#SBATCH --error=logs/%x_%j.err

set -euo pipefail
echo "job_id=$SLURM_JOB_ID"
echo "host=$(hostname)"
echo "cpus=$SLURM_CPUS_PER_TASK"
sleep 10

For eligible small CPU work, --account=public may replace es_fuge. Do not add a GPU request to a public job.

Before submission:

mkdir -p logs
bash -n cpu-smoke.slurm
sed -n '1,80p' cpu-smoke.slurm

Then submit and record the returned ID:

sbatch cpu-smoke.slurm
squeue -u "$USER"

Expected result: Submitted batch job <job-id>. If the script or resources are wrong, cancel immediately with scancel <job-id>.

Freeze Inputs Before Submission#

sbatch transfers a copy of the batch script to the Slurm controller when you submit it. Editing that script later does not change the queued job. Slurm does not copy the other files used by the script: source code, configuration, datasets, and environments are opened from their paths when the job runs. A pending job can therefore see later edits to those external files.

Before a production submission:

1. Commit the reviewed source and either leave that checkout untouched or make an immutable, commit-specific source snapshot for the run.
2. Copy the exact configuration into a run-specific directory and make the job reference that copy.
3. Record the Git commit and a checksum or version for important inputs.
4. Do not overwrite files referenced by pending or running jobs.
5. If inputs must change, create a new run snapshot and submit a new job. Cancel the old pending job if it is no longer wanted.

Run these inspection commands from the project clone before submission:

git status --short
git rev-parse HEAD

git status --short should be empty for a commit-based run. A commit hash records provenance but does not freeze files in a checkout that someone later switches or edits. If intentional uncommitted files are part of an exploratory run, preserve them in the run-specific directory and label the run non-reproducible; do not imply that a commit alone contains those changes.

States, Pending Reasons, And Exit Codes#

Common states:

• PENDING: waiting for resources, limits, priority, or another condition;
• RUNNING: executing on a compute node;
• COMPLETED: finished with a successful job exit;
• FAILED: exited unsuccessfully;
• OUT_OF_MEMORY: exceeded allocated system memory;
• TIMEOUT: reached the requested wall-time limit;
• CANCELLED: cancelled by a user or administrator.

Inspect rather than repeatedly resubmitting:

Replace <job-id> before using these command templates:

myjobs -j <job-id>
squeue -j <job-id> -o '%.18i %.2t %.30R'

The REASON field explains a pending job. A limit reason does not mean the script should be submitted again.

Completed-Job Accounting#

Replace <job-id> before using these command templates:

sacct -j <job-id> \
  --format=JobID,JobName%24,Account,State,ExitCode,Elapsed,AllocCPUS,ReqMem,MaxRSS
seff <job-id>

Interpret the evidence:

• AllocCPUS is the CPU allocation, not proof the program used every CPU.
• ReqMem is requested system memory.
• MaxRSS is the maximum recorded resident memory for a reported step.
• Elapsed helps choose the next wall-time limit.
• State and ExitCode distinguish scheduler and application outcomes.
• seff summarizes CPU and memory efficiency; it does not replace application profiling or GPU monitoring.

For a running batch step, use sstat sparingly:

Replace <job-id> before using this command template:

sstat -j <job-id>.batch --format=JobID,AveCPU,MaxRSS,AveRSS

Some metrics appear only after a step has run long enough or completed.

Measurement-Driven Optimization#

For each representative workload:

1. Start with one task and the CPUs the program can demonstrably use.
2. Request modest memory and a realistic short time for the first sample.
3. Inspect logs, sacct, and seff after completion.
4. Reduce unused CPU or memory while leaving sensible variation headroom.
5. Increase a resource only after identifying the bottleneck.
6. Record the measurement and chosen production request in the project.

Examples:

• Low CPU efficiency in a single-threaded program -> request one CPU.
• MaxRSS far below requested memory -> reduce memory next time.
• OUT_OF_MEMORY with valid input -> investigate growth, then increase memory.
• Job finishes in 12 minutes with a four-hour limit -> use a shorter limit with appropriate headroom.
• High CPU use but poor throughput -> profile I/O and algorithm behavior before requesting more cores.

Job Arrays And Mandatory Concurrency Caps#

An array represents many similar tasks. The % value limits simultaneous tasks:

#SBATCH --array=0-19%2

This defines 20 tasks but permits at most two to run concurrently. Start with %1 during validation. Every beginner and assessment array must have an explicit cap.

The cap is per array, not per user. If three active arrays each use %2, up to six array tasks may become eligible simultaneously, in addition to ordinary jobs. Before another submission, inspect squeue -u "$USER" and calculate the combined CPUs, memory, and GPUs across all active jobs and arrays.

Inside the script:

config="configs/config_${SLURM_ARRAY_TASK_ID}.yaml"
python train.py --config "$config"

Use %A for the parent array ID and %a for the task index in log names:

#SBATCH --output=logs/%x_%A_%a.out
#SBATCH --error=logs/%x_%A_%a.err

Choose the cap from measured per-task resources, project urgency, personal limits, and collective limits. Do not use the maximum merely because it is allowed.

GPU Jobs#

GPU jobs require es_fuge; the public share contains no GPUs. Start with one.

RTX 4090 lab starter profile#

#SBATCH --account=es_fuge
#SBATCH --gpus=rtx_4090:1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=3G

RTX 3090 fallback#

#SBATCH --account=es_fuge
#SBATCH --gpus=rtx_3090:1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=3G

RTX PRO 6000 special-purpose profile#

The lab share currently includes two RTX PRO 6000 GPUs with 96 GiB GPU memory each. They use the Slurm identifier pro_6000:

#SBATCH --account=es_fuge
#SBATCH --gpus=pro_6000:1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=3G

Do not use this profile merely because an RTX 4090 is pending. ETH identifies RTX PRO 6000 as Blackwell hardware and currently warns that programs compiled with CUDA 12 libraries will not run on it. Request it only when the project has been tested with a compatible CUDA 13 toolchain and needs its capabilities, such as more than 24 GiB of GPU memory. A live sbatch --test-only check under es_fuge accepted this profile on 2026-08-04.

Choose one explicit model#

Use RTX 4090 by default. If RTX 4090 capacity is not available, use RTX 3090 as the general fallback rather than submitting duplicate jobs. RTX PRO 6000 is a separate compatibility/capability choice, not a queue-avoidance fallback.

Current ETH documentation shows the flexible form --gpus=rtx_3090,rtx_4090:1, but a live sbatch --test-only check under es_fuge rejected that expression on 2026-08-04. Do not use the combined form until the lab maintainer has reverified it successfully on Euler.

The 16-CPU/48-GiB profile is a balanced lab starting point based on the common eight-GPU node layout previously inspected for this share. ETH explicitly notes that CPU and RAM can vary even between nodes with the same GPU model. Measure the actual workload and reduce or adjust the request when evidence supports it.

Inside an allocated GPU job, inspect utilization during a representative phase:

nvidia-smi

Low GPU utilization with saturated CPUs may indicate data loading or preprocessing is the bottleneck. Low CPU and GPU use may indicate I/O, waiting, or incorrect device placement. A second GPU does not fix those problems.

Use checkpointing before the wall-time limit. Establish throughput and scaling on one GPU before requesting multiple GPUs.

Verification#

• The script contains an explicit account, time, CPU, and memory request.
• No partition is specified without a documented dependency.
• Public jobs contain no GPU request.
• Arrays have a low explicit concurrency cap.
• The job's source revision, configuration snapshot, and important input version are recorded.
• Completed jobs have reviewed sacct and seff evidence.
• Resource changes cite measurements rather than convenience.

Common Failures And Safe Recovery#

Use Euler troubleshooting for pending jobs, OOM, authentication, tunnel, quota, and GPU symptoms.

Ask For Help When#

The application needs MPI, multiple nodes, unusually large memory, more than one GPU, a policy exception, or metrics that cannot be interpreted from a small representative job.

Primary Sources#

• ETH HPC Slurm
• ETH HPC first job
• ETH HPC submission line advisor
• ETH HPC GPU nodes
• Slurm sbatch file-transfer semantics

Sources verified: 2026-08-06. GPU profiles verified on Euler: 2026-08-04. Review by: 2026-11-04. Owner: IDEAL Lab IT.