Lab: Euler GPU Resource Review

Goal: review and correct a GPU plan before consuming a GPU.

Prerequisites: Euler CPU Track and the Euler GPU Track.

Execution context: review only. Do not submit a GPU job for this lab.

Unsafe Scenario#

#SBATCH --account=public
#SBATCH --partition=gpu
#SBATCH --gpus=rtx_4090:4
#SBATCH --cpus-per-task=64
#SBATCH --mem=500G
#SBATCH --time=48:00:00

The author says, "The dataset is large, so four GPUs should make it faster."

Review Tasks#

Identify and explain:

1. why public cannot be used;
2. why the partition should normally be omitted;
3. why dataset size does not prove multi-GPU scaling;
4. which one-GPU starter profile is appropriate;
5. the shortest useful smoke test;
6. CPU, system-memory, GPU-memory, and utilization metrics to inspect;
7. checkpoint and wall-time behavior;
8. evidence required before requesting two GPUs.

Corrected Batch Starting Point#

For RTX 4090:

#SBATCH --account=es_fuge
#SBATCH --gpus=rtx_4090:1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=3G

For RTX 3090 fallback, change only the GPU type to rtx_3090. Then add a project-specific job name, short measured time limit, logs, environment setup, and executable command.

The lab's two RTX PRO 6000 GPUs are not a routine fallback. Select pro_6000 only for a CUDA 13-compatible workload with a specific requirement such as more than 24 GiB GPU memory; use the canonical special-purpose profile.

The CPU/memory values are a lab starter profile, not a promise that every model needs those resources. Measure and right-size.

Bottleneck Scenarios#

Explain the next investigation, not an automatic resource increase:

• GPU utilization 15%, CPUs saturated.
• GPU utilization 5%, CPUs low, storage reads slow.
• GPU memory full, GPU compute high.
• One GPU works; two GPUs produce only 1.1x throughput.
• Tunnel remains allocated while no process is running.

Verification#

The final plan uses one GPU, es_fuge, no unnecessary partition, a bounded time, logs, monitoring, checkpoints, and a measurable criterion for scaling.

Common Failures And Safe Recovery#

• CUDA is available but code remains on CPU: fix device placement before scaling resources.
• GPU job is mostly idle: cancel it, inspect the bottleneck, and use CPU-only work for setup/log review.
• 4090 waits: consider an RTX 3090 only if software compatibility and performance needs permit it; do not submit duplicates of both jobs and do not switch to RTX PRO 6000 solely to bypass the queue.

Understand Before Accepting AI Output#

• I can justify GPU use from program behavior.
• I understand the monitoring plan.
• I have not equated larger data with multi-GPU scaling.
• I know when to use sbatch instead of a tunnel.

Evidence#

Submit the corrected script, review answers, and one-GPU scaling criterion. No live GPU allocation is required.

Ask For Help When#

The project requires multi-GPU/distributed training, unusual GPU memory, unsupported CUDA versions, or an exception to student limits.