Slurm Job Lifecycle And Resource Optimization

Slurm is Euler's scheduler: it accepts a job, waits for suitable shared resources, and starts the program on a compute node. A job request states the CPU, memory, GPU, and maximum time the program needs.

Goal: submit, monitor, diagnose, and right-size Euler jobs without wasting shared resources.

Use this as a lookup for the step you are doing: submit once, inspect a pending job, or read completed-job accounting. You do not need to work through every template here. If you came from a Passport lesson, return there after finding the requested explanation; follow that lesson's own script and resource limits.

Prerequisites: Euler access, an approved storage plan, and the Euler share policy.

New to text commands? Read how to open a terminal and run one command before using a command block. It explains the labels used below and what to copy.

Execution context: all commands on this page run on an Euler login node - Bash, except commands explicitly described as running inside the job.

The Lifecycle#

estimate -> inspect script -> sbatch -> squeue/myjobs -> logs ->
sacct/seff -> adjust next request

The login node is for editing, transfer, compilation where appropriate, and job management. Run computations through Slurm.

Core Commands#

Conservative CPU Template#

Do not specify a partition. Current ETH guidance says to let Slurm select one unless the workload genuinely depends on a particular resource. Save the block below as cpu-smoke.slurm in your approved personal job folder on Euler. These are file contents, not commands to paste into the terminal.

#!/usr/bin/env bash
#SBATCH --job-name=cpu-smoke
#SBATCH --account=es_fuge
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=1G
#SBATCH --output=logs/%x_%j.out
#SBATCH --error=logs/%x_%j.err

set -euo pipefail
echo "job_id=$SLURM_JOB_ID"
echo "host=$(hostname)"
echo "cpus=$SLURM_CPUS_PER_TASK"
sleep 10

For eligible small CPU work, --account=public may replace es_fuge. Do not add a GPU request to a public job.

Before submission:

mkdir -p logs
bash -n cpu-smoke.slurm
sed -n '1,80p' cpu-smoke.slurm

Then submit once and save the returned ID. The guard below reuses an existing ID instead of creating a duplicate job:

(
set -eu
id_file="$HOME/cpu-smoke.id"
if [ -e "$id_file" ] || [ -L "$id_file" ]; then
  if [ ! -f "$id_file" ] || [ -L "$id_file" ] || [ ! -s "$id_file" ]; then
    printf 'STOP: stored job ID file is empty or not a regular file. Ask for help before submitting.\n' >&2
    exit 1
  fi
  job_id="$(cat "$id_file")"
  case "$job_id" in
    ''|*[^0-9]*) printf 'STOP: stored job ID is invalid.\n' >&2; exit 1 ;;
  esac
  printf 'Existing job, not resubmitted: %s\n' "$job_id"
else
  submission="$(sbatch --parsable cpu-smoke.slurm)"
  job_id="${submission%%;*}"
  case "$job_id" in
    ''|*[^0-9]*) printf 'STOP: sbatch did not return a numeric job ID.\n' >&2; exit 1 ;;
  esac
  printf '%s\n' "$job_id" > "$id_file"
  chmod 600 "$id_file"
  printf 'Submitted job: %s\n' "$job_id"
fi
squeue -j "$job_id" -o '%.18i %.2t %.30R'
)

Expected result: the command prints one numeric job ID. Running the block again says Existing job, not resubmitted. If the script or resources are wrong, cancel that recorded ID with scancel <job-id> (replace <job-id> with the printed numeric ID); do not submit a replacement until you understand the first result.

If the ID file is empty or invalid, stop and ask for help. Do not delete it or submit again: the previous job may already exist even if its ID was not saved. The job_id variable belongs to the block above; in a new command, substitute the numeric ID it printed.

Freeze Inputs Before Submission#

sbatch transfers a copy of the batch script to the Slurm controller when you submit it. Editing that script later does not change the queued job. Slurm does not copy the other files used by the script: source code, configuration, datasets, and environments are opened from their paths when the job runs. A pending job can therefore see later edits to those external files.

Before a production submission:

1. Commit the reviewed source and either leave that checkout untouched or make an immutable, commit-specific source snapshot for the run.
2. Copy the exact configuration into a run-specific directory and make the job reference that copy.
3. Record the Git commit and a checksum or version for important inputs.
4. Do not overwrite files referenced by pending or running jobs.
5. If inputs must change, create a new run snapshot and submit a new job. Cancel the old pending job if it is no longer wanted.

Run these inspection commands from the project clone before submission:

git status --short
git rev-parse HEAD

git status --short should be empty for a commit-based run. A commit hash records provenance but does not freeze files in a checkout that someone later switches or edits. If intentional uncommitted files are part of an exploratory run, preserve them in the run-specific directory and label the run non-reproducible; do not imply that a commit alone contains those changes.

States, Pending Reasons, And Exit Codes#

Common states:

• PENDING: waiting for resources, limits, priority, or another condition;
• RUNNING: executing on a compute node;
• COMPLETED: finished with a successful job exit;
• FAILED: exited unsuccessfully;
• OUT_OF_MEMORY: exceeded allocated system memory;
• TIMEOUT: reached the requested wall-time limit;
• CANCELLED: cancelled by a user or administrator.

Inspect rather than repeatedly resubmitting:

Replace <job-id> before using these command templates:

myjobs -j <job-id>
squeue -j <job-id> -o '%.18i %.2t %.30R'

The REASON field explains a pending job. A limit reason does not mean the script should be submitted again.

Delayed Starts And Job Dependencies#

Use this section when you deliberately want a later start, or when one job needs another job's successful output. Choose the condition before the first submission; these are alternatives, not extra jobs to submit as tests.

Delay Until A Time#

For an intentional one-hour delay, add this directive to your reviewed batch script before submission. This is a file line, not a terminal command:

#SBATCH --begin=now+1hour

Then use the submit-once procedure above and keep its job ID. sbatch submits the job immediately; Slurm waits before making it eligible for allocation. Resources and priority may delay it further. --begin does not reserve an exact start time, space array tasks apart, or reduce submission traffic. For an absolute date/time, confirm the Euler shell's time and timezone first. Do not add --begin to every job as a general cluster-protection measure.

Wait For Successful Results#

If a second job reads the first job's output, use afterok rather than a delay based on how long you think the first will take. Record and verify the first job's ID. While it is pending or running, submit the dependent job once using this command template:

sbatch --parsable --dependency=afterok:<previous-job-id> <reviewed-job-script>

Replace both placeholders. Use these arguments in place of the initial sbatch call in your job-ID recording procedure; do not run the template again if you already submitted the dependent job. A dependency does not replace the second script's own account, CPU, memory, and time requests.

Expected and verification: keep the returned ID and inspect that job with squeue -j <job-id> -o '%.18i %.2t %.30R', replacing the placeholder. You may see BeginTime or Dependency while the condition is unmet, then another pending reason while Slurm waits for resources. An array cap applies only to that array; still include your other active jobs in the resource calculation.

Recovery and help: if the predecessor fails, the afterok condition cannot be satisfied. Inspect its logs and accounting before deciding what to do with the dependent job. Do not remove the dependency, resubmit blindly, or assume a later successful rerun satisfies the original dependency. Ask the supervisor or lab IT if the predecessor ID, output readiness, or timing is unclear. Editing a script does not update a submitted job; see freeze inputs.

Scheduler-Friendly Automation#

Concurrency caps bound running work, not the number of scheduler requests. sbatch and squeue contact the shared Slurm controller, so many rapid calls can degrade service for everyone.

• For similar tasks, prefer one appropriately capped array over many separate submissions. Validate one representative task before scaling up.
• Keep status checks infrequent and limited to your jobs. Group checks rather than running a separate tight polling loop for every task.
• Retry only identified transient failures. Bound attempts and increase delays between retries; add a small random delay (jitter) in automated clients so many clients do not retry together. Follow any current cluster guidance.
• After an uncertain submission result, establish whether the job was created before trying again. A pending job is not a failed submission.
• Do not retry malformed scripts, access errors, or unchanged application failures. If many tasks fail identically, stop new submissions and debug one task; cancel only the affected jobs you are authorized to manage.

For automated workflows, verify both the total concurrent resource request and how often the workflow calls Slurm. If the controller is unavailable, pause the automation and use Euler troubleshooting rather than increasing the request rate.

Completed-Job Accounting#

Replace <job-id> before using these command templates:

sacct -j <job-id> \
  --format=JobID,JobName%24,Account%40,State,ExitCode,Elapsed,AllocCPUS,ReqMem,MaxRSS
seff <job-id>

Interpret the evidence:

• AllocCPUS is the CPU allocation, not proof the program used every CPU.
• ReqMem is requested system memory.
• MaxRSS is the maximum recorded resident memory for a reported step.
• Elapsed helps choose the next wall-time limit.
• State and ExitCode distinguish scheduler and application outcomes.
• Account%40 gives the account name more room. Euler may display normal/es_fuge; the Passport accepts that full name or es_fuge. A value ending in + is truncated; widen the column before interpreting it.
• seff summarizes CPU and memory efficiency; it does not replace application profiling or GPU monitoring.

For a running batch step, use sstat sparingly:

Replace <job-id> before using this command template:

sstat -j <job-id>.batch --format=JobID,AveCPU,MaxRSS,AveRSS

Some metrics appear only after a step has run long enough or completed.

Measurement-Driven Optimization#

For each representative workload:

1. Start with one task and the CPUs the program can demonstrably use.
2. Request modest memory and a realistic short time for the first sample.
3. Inspect logs, sacct, and seff after completion.
4. Reduce unused CPU or memory while leaving sensible variation headroom.
5. Increase a resource only after identifying the bottleneck.
6. Record the measurement and chosen production request in the project.

Examples:

• Low CPU efficiency in a single-threaded program -> request one CPU.
• MaxRSS far below requested memory -> reduce memory next time.
• OUT_OF_MEMORY with valid input -> investigate growth, then increase memory.
• Job finishes in 12 minutes with a four-hour limit -> use a shorter limit with appropriate headroom.
• High CPU use but poor throughput -> profile I/O and algorithm behavior before requesting more cores.

Job Arrays And Mandatory Concurrency Caps#

An array represents many similar tasks. The % value limits simultaneous tasks:

#SBATCH --array=0-19%2

This defines 20 tasks but permits at most two to run concurrently. Start with %1 during validation. Every beginner and assessment array must have an explicit cap.

The cap is per array, not per user. If three active arrays each use %2, up to six array tasks may become eligible simultaneously, in addition to ordinary jobs. Before another submission, inspect squeue -u "$USER" and calculate the combined CPUs, memory, and GPUs across all active jobs and arrays.

Inside the script:

config="configs/config_${SLURM_ARRAY_TASK_ID}.yaml"
python train.py --config "$config"

Use %A for the parent array ID and %a for the task index in log names:

#SBATCH --output=logs/%x_%A_%a.out
#SBATCH --error=logs/%x_%A_%a.err

Choose the cap from measured per-task resources, project urgency, personal limits, and collective limits. Do not use the maximum merely because it is allowed.

Bulk Jobs And Fair-Share Accounting#

Students must not use bulk jobs. Follow the student bulk-job restriction. The staff accounting description below does not apply to student priority or fair-share behavior.

While Euler's student-subgroup controls do not yet apply correctly to bulk jobs, a five-minute administrative guard cancels student bulk jobs charged to es_fuge. A student who believes a cancellation was incorrect should send the job ID to Soheyl. Do not resubmit the same bulk job while the restriction is in force.

Bulk jobs are an advanced option for restartable CPU workloads that can use spare Euler capacity and tolerate interruption. Staff must confirm suitability with the lab IT administrator before using them. The following directives are for approved staff workloads only; they are not a student starter script:

#SBATCH --account=es_fuge
#SBATCH --partition=bulk
#SBATCH --requeue

Selecting bulk is an intentional exception to the usual advice to omit the partition. Use it only when the workload can restart safely. --requeue makes the job eligible to return to the queue after interruption; it does not preserve application state. The program must checkpoint useful progress, write outputs safely, and tolerate running part of its work again.

The earlier Euler-support accounting clarification recorded on 2026-09-04 describes regular production jobs; do not extend it to the student partition:

• bulk jobs accrue ordinary raw usage in their bulk-specific accounting QOSs;
• UsageFactor=1.0 and the same TRES billing weights as a production QOS do not mean that bulk usage affects production-job priority;
• bulk usage is excluded from the fair-share usage and limits applied to regular production QOSs, so it currently does not reduce the lab's regular-job fair-share priority; and
• this is an Euler configuration choice rather than a permanent Slurm guarantee. Euler support will announce a future change before bulk usage begins affecting production jobs.

This distinction matters when reading RawUsage or EffectiveUsage: the bulk accounting records can increase even though those values are not fed into the production fair-share calculation. Do not infer production priority impact from the raw counters, UsageFactor, or TRES billing weights alone.

GPU Jobs#

GPU jobs require es_fuge; the public share contains no GPUs. Start with one.

RTX 4090 lab starter profile#

#SBATCH --account=es_fuge
#SBATCH --gpus=rtx_4090:1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=3G

RTX 3090 fallback#

#SBATCH --account=es_fuge
#SBATCH --gpus=rtx_3090:1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=3G

RTX PRO 6000 special-purpose profile#

The lab share currently includes two RTX PRO 6000 GPUs with 96 GiB GPU memory each. They use the Slurm identifier pro_6000:

#SBATCH --account=es_fuge
#SBATCH --gpus=pro_6000:1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=3G

Do not use this profile merely because an RTX 4090 is pending. ETH identifies RTX PRO 6000 as Blackwell hardware and currently warns that programs compiled with CUDA 12 libraries will not run on it. Request it only when the project has been tested with a compatible CUDA 13 toolchain and needs its capabilities, such as more than 24 GiB of GPU memory. A live sbatch --test-only check under es_fuge accepted this profile on 2026-08-04.

Choose one explicit model#

Use RTX 4090 by default. If RTX 4090 capacity is not available, use RTX 3090 as the general fallback rather than submitting duplicate jobs. RTX PRO 6000 is a separate compatibility/capability choice, not a queue-avoidance fallback.

Current ETH documentation shows the flexible form --gpus=rtx_3090,rtx_4090:1, but a live sbatch --test-only check under es_fuge rejected that expression on 2026-08-04. Do not use the combined form until the lab maintainer has reverified it successfully on Euler.

The 16-CPU/48-GiB profile is a balanced lab starting point based on the common eight-GPU node layout previously inspected for this share. ETH explicitly notes that CPU and RAM can vary even between nodes with the same GPU model. Measure the actual workload and reduce or adjust the request when evidence supports it.

Inside an allocated GPU job, inspect utilization during a representative phase:

nvidia-smi

Low GPU utilization with saturated CPUs may indicate data loading or preprocessing is the bottleneck. Low CPU and GPU use may indicate I/O, waiting, or incorrect device placement. A second GPU does not fix those problems.

Use checkpointing before the wall-time limit. Establish throughput and scaling on one GPU before requesting multiple GPUs.

Verification#

• The script contains an explicit account, time, CPU, and memory request.
• No partition is specified without a documented dependency.
• Public jobs contain no GPU request.
• Arrays have a low explicit concurrency cap.
• The job's source revision, configuration snapshot, and important input version are recorded.
• Completed jobs have reviewed sacct and seff evidence.
• Resource changes cite measurements rather than convenience.

Common Failures And Safe Recovery#

Use Euler troubleshooting for pending jobs, OOM, authentication, tunnel, quota, and GPU symptoms.

Ask For Help When#

The application needs MPI, multiple nodes, unusually large memory, more than one GPU, a policy exception, or metrics that cannot be interpreted from a small representative job.

Primary Sources#

• ETH HPC Slurm
• ETH HPC first job
• ETH HPC submission line advisor
• ETH HPC GPU nodes
• Slurm sbatch file-transfer semantics
• Slurm delayed starts and dependencies
• Slurm job arrays
• Slurm scheduler query load

Sources verified: 2026-08-06. GPU profiles verified on Euler: 2026-08-04. Delayed-start, dependency, array, and scheduler-load semantics verified: 2026-10-08. Review by: 2026-11-04. Owner: IDEAL Lab IT.