Lab: Safe Euler Job Arrays

Goal: submit repeated tasks without allowing all of them to run simultaneously.

Prerequisites: Euler CPU Track and one successful single-task representative job.

Execution context: review locally or on an Euler login node. Do not submit the full array during the first review.

Unsafe Starting Point#

#SBATCH --array=0-99

This creates 100 tasks with no explicit student-selected concurrency cap.

Safe Starting Point#

#SBATCH --array=0-99%1

After one task succeeds and its measured resource use is understood, a small justified cap may be used, for example:

#SBATCH --array=0-99%2

The array still contains 100 tasks, but at most two may run at once.

That statement applies to one array. Two separate arrays capped at %2 may make four tasks eligible at once. Existing arrays and ordinary jobs must be included in the total before another submission.

Build A Reviewable Script#

#!/usr/bin/env bash
#SBATCH --job-name=passport-array
#SBATCH --account=es_fuge
#SBATCH --time=00:10:00
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=1G
#SBATCH --array=0-9%1
#SBATCH --output=logs/%x_%A_%a.out
#SBATCH --error=logs/%x_%A_%a.err

set -euo pipefail
config="configs/config_${SLURM_ARRAY_TASK_ID}.yaml"
test -f "$config"
python process.py --config "$config"

Tasks#

1. Explain %A, %a, and $SLURM_ARRAY_TASK_ID.
2. State why the first cap is %1.
3. Calculate concurrent CPU, memory, and GPU use for a proposed cap.
4. Inspect squeue -u "$USER" and calculate total eligible resources across all active arrays and ordinary jobs.
5. State the measurements and lab limits used to approve a higher cap.
6. Explain how you would cancel the whole array or one task.

Do not submit this fixture unless the files and program exist in an approved training project.

Verification#

• Every array has an explicit %<cap>.
• The combined cap across overlapping arrays remains within justified limits.
• Log filenames distinguish parent job and task index.
• One representative task passes before the full range is enabled.
• Concurrent resources fit personal and collective policy.

Common Failures And Safe Recovery#

• Many tasks fail identically: cancel the array and debug one task.
• Cap consumes the personal GPU limit: reduce it; a limit is not a target.
• Duplicate arrays submitted: cancel unwanted parent job IDs rather than waiting for every task.

Understand Before Accepting AI Output#

• I calculated concurrent resources myself.
• I counted my other active arrays and ordinary jobs.
• I validated one representative task.
• I know the difference between array size and concurrency.
• I know how to cancel the submission.

Evidence#

Submit the corrected script and concurrency calculation. A live multi-task array is not required for the passport.

Ask For Help When#

Tasks depend on each other, inputs have very different resource needs, or the project requires concurrency near a lab limit.