Euler Storage And Data Lifecycle

Goal: put files in a location that matches their size, lifetime, performance, sharing, and recovery requirements.

Prerequisites: know the project's information owner and approved group storage path.

Execution context: inspection commands run on an Euler login node - Bash. Data processing belongs in a Slurm allocation.

Storage Decision Table#

Inspect Capacity And Paths#

Run on an Euler login node - Bash:

printf 'HOME=%s\nSCRATCH=%s\n' "$HOME" "$SCRATCH"
lquota
lquota /cluster/work/fuge

Expected result: paths plus quota tables. Do not assume the old 1 TB scratch value found in archived documentation; current ETH documentation states 2.5 TB and a two-week purge policy.

Stage High-I/O Data Through $TMPDIR#

For I/O-intensive work, request node-local temporary space in the Slurm script:

#SBATCH --tmp=20G

Choose a measured size rather than copying this value into every job. Slurm creates a unique directory, exposes its path as $TMPDIR inside the job, and deletes it when the job terminates. It is not backed up.

This is the safe stage-in/stage-out pattern. Adapt the input, program, and durable result paths in a reviewed job script before submission:

set -euo pipefail

source_dir="/replace/with/approved/input/path"
durable_results_root="/replace/with/approved/results/path"
durable_run_dir="$durable_results_root/$SLURM_JOB_ID"
local_input="$TMPDIR/input"
local_output="$TMPDIR/output"

test -d "$source_dir"
mkdir -p "$local_input" "$local_output" "$durable_run_dir"
rsync -a -- "$source_dir/" "$local_input/"

python "$SLURM_SUBMIT_DIR/process.py" \
  --input "$local_input" \
  --output "$local_output"

rsync -a -- "$local_output/" "$durable_run_dir/"

Replace both /replace/... values with supervisor-approved Euler paths before submitting; they are deliberately invalid defaults. The trailing / means "copy the contents" rather than nesting the source directory. Use a unique result directory so a failed or repeated run cannot overwrite another run. Do not copy the whole submission directory into and back out of $TMPDIR: that can overwrite source, configuration, or logs.

The final rsync runs only if the program succeeds. For a long job that must survive timeout or node failure, write periodic checkpoints to an approved durable run directory as well; the final copy from $TMPDIR is not a backup. Do not process high-I/O workloads directly from external NAS storage.

Recommended Project Layout#

Keep code in each person's Git clone. Use shared storage for data and artifacts:

/cluster/work/fuge/<project>/
  datasets/
  checkpoints/
  logs/
  results/
  nobackup/

A directory named nobackup and everything below it is excluded from Euler work-storage backups. Use it only for re-creatable data.

Shared Permissions#

Lab IT/ETH HPC must first provision the top-level shared project directory; ordinary users cannot create a new child directly under /cluster/work/fuge. Only the Unix owner designated to initialize that existing project directory should run the ACL procedure. Confirm the exact path and Unix group with the supervisor or lab IT; do not guess.

Run on an Euler login node - Bash. This block prompts for the exact path and group, validates ownership and write access, shows the intended target, and requires the full path a second time before creating child directories:

(
set -euo pipefail
read -r -p "Approved project directory under /cluster/work/fuge: " project_dir
project_dir="$(realpath -m -- "$project_dir")"
read -r -p "Approved Unix group: " project_group

case "$project_dir" in
  /cluster/work/fuge/?*) ;;
  *) printf 'STOP: path must be below /cluster/work/fuge\n' >&2; exit 1 ;;
esac
case "$project_group" in
  ''|*[!A-Za-z0-9_.-]*) printf 'STOP: invalid group name\n' >&2; exit 1 ;;
esac
if ! id -nG | tr ' ' '\n' | grep -Fqx "$project_group"; then
  printf 'STOP: your account is not a member of %s\n' "$project_group" >&2
  exit 1
fi
if [ ! -d "$project_dir" ]; then
  printf 'STOP: project root is not provisioned; ask lab IT to create it.\n' >&2
  exit 1
fi
if [ "$(stat -c '%U' "$project_dir")" != "$USER" ]; then
  printf 'STOP: only the directory owner may initialize its ACLs.\n' >&2
  exit 1
fi
if [ ! -w "$project_dir" ]; then
  printf 'STOP: project root is not writable by your account.\n' >&2
  exit 1
fi

printf 'Directory: %s\nGroup: %s\n' "$project_dir" "$project_group"
ls -ld "$project_dir"
read -r -p "Type the full directory again to approve initialization: " confirmation
if [ "$confirmation" != "$project_dir" ]; then
  printf 'STOP: confirmation did not match; nothing was changed.\n' >&2
  exit 1
fi

child_paths=(
  "$project_dir/datasets"
  "$project_dir/checkpoints"
  "$project_dir/logs"
  "$project_dir/results"
)
mkdir -p -- "${child_paths[@]}"
project_paths=("$project_dir" "${child_paths[@]}")
chgrp "$project_group" "${project_paths[@]}"
chmod 2770 "${project_paths[@]}"
setfacl -m "g:${project_group}:rwx" "${project_paths[@]}"
setfacl -d -m "g:${project_group}:rwx" "${project_paths[@]}"
getfacl "$project_dir" "$project_dir/logs"
)

This uses setgid on directories and a default ACL so new content inherits group access even when a collaborator uses a restrictive umask. The default ACL is applied to every initialized child directory, not only the project root. The procedure deliberately avoids broad recursive permission changes.

Each collaborator should test with their own account:

read -r -p "Approved project directory: " project_dir
project_dir="$(realpath -m -- "$project_dir")"
case "$project_dir" in
  /cluster/work/fuge/?*) ;;
  *) printf 'STOP: path must be below /cluster/work/fuge\n' >&2; project_dir='' ;;
esac
test_file="$project_dir/logs/$USER.write-test"
if [ -n "$project_dir" ]; then
  touch "$test_file" && ls -l "$test_file" && rm "$test_file"
fi

Data Lifecycle#

1. Record the authoritative source and owner.
2. Copy working data to the approved Euler location.
3. Verify size, count, and a checksum for important transfers.
4. Use $TMPDIR or $SCRATCH only for re-creatable intermediates.
5. Copy final results and required checkpoints to durable project storage.
6. Remove obsolete scratch and intermediate data.
7. Document what another researcher must retain at handover.

Verification#

• lquota shows expected capacity.
• Lab IT provisioned the top-level project root before ACL initialization.
• A collaborator can create and remove a harmless test file.
• The project README identifies authoritative, temporary, and backed-up copies.
• Jobs using $TMPDIR request it explicitly and copy required outputs to a unique durable run directory.
• No dataset or checkpoint is accidentally tracked by Git.

Common Failures And Safe Recovery#

• Disk quota exceeded: inspect both byte and file-count quotas. Remove or relocate known disposable files; do not delete unfamiliar project data.
• Permission denied: inspect ls -ld, getfacl, and group membership. Do not use chmod -R 777.
• Scratch file disappeared: scratch is intentionally unbacked and purged. Restore from the authoritative copy; report data loss if none exists.
• NAS processing is slow: stage data to approved Euler storage rather than loading it repeatedly over the external mount.

Ask For Help When#

The approved group is unknown, recursive permission repair appears necessary, the project root is missing/not owned by the initializer, the project has no authoritative copy, or a transfer is measured in terabytes.

Primary Source#

• ETH HPC storage

Verified: 2026-08-06. Review by: 2026-11-04. Owner: IDEAL Lab IT.