Python Environments On Euler | IDEAL IT Passport

A Python environment is a separate Python interpreter and package set for one project. Euler is a different computer from your laptop, so the environment is created again on Euler from the project's recorded dependency file.

Goal: create reproducible Python environments for Euler jobs without copying a laptop environment or filling unsuitable storage with small files.

Prerequisites: working ssh euler access and a project dependency definition such as environment.yml, requirements.txt, or pyproject.toml.

New to text commands? Read how to open a terminal and run one command before using a command block. It explains the labels used below and what to copy.

Execution context: inspection and the small training environment run on an Euler login node - Bash. Use a CPU allocation or batch job for dependency installation that compiles code, transfers substantial data, or performs meaningful computation.

The Rule#

Your laptop and Euler need separate environment directories. Commit the small text file that defines dependencies; do not commit or copy .venv, env, or a Conda installation directory.

For the Passport exercise:

use the dated Python module named by the mission;
keep the small training environment in $HOME/passport-euler/venvs/passport-python;
load the module and activate the environment inside every Slurm script;
never use sudo or install project packages into the shared module or Conda base environment.

The mission creates the environment at its final location and verifies activation, the selected interpreter, and pip there. Python virtual environments contain absolute paths; moving a completed environment can break it. An existing target is checked before use and is never overwritten. Follow that guided recipe rather than copying a second command sequence from this reference page.

Recover A Broken Passport Training Environment#

Use this only after the mission's inspection or verification fails, and only if $HOME/passport-euler/venvs/passport-python is the small environment you created for this exercise. If you used it for research, installed project packages, or are unsure what it contains, stop and ask lab IT first.

Machine and shell: an Euler login node, Bash, after ssh euler. Finish or stop your other environment-creation terminal first. This recovery submits no job and does not change your shell configuration or Passport progress.

Read the error from Inspect before creating. A moved environment can still run Python directly while activation or the pip command uses its old temporary path. Repeating python -m venv over it is not the recovery.
Run the block below only for the training directory described above. It recognizes the old Passport temporary-path recipe or a marker from the new recipe. It refuses symbolic-link targets, unsafe ownership/permissions, unknown directories, and a creation still marked as in progress.
A working environment is kept. An affected recognized environment is moved intact into a new private passport-python.saved-.../environment directory. Nothing inside it is deleted. Keep the printed backup location private.

(
set -eu
umask 077
module purge
module load stack/2024-06 python/3.11.6
[ "$(python -c 'import platform; print(platform.python_version())')" = 3.11.6 ] || {
  printf 'STOP: the loaded module must provide Python 3.11.6.\n' >&2; exit 1
}
root="$HOME/passport-euler"
base="$root/venvs"
venv="$base/passport-python"
# <!-- passport-snippet:euler-environment-check -->
check_directories() {
  python - "$@" <<'PY'
import os, stat, sys
for name in sys.argv[1:]:
    info = os.lstat(name)
    if not stat.S_ISDIR(info.st_mode) or info.st_uid != os.getuid() or info.st_mode & 0o022:
        raise SystemExit("STOP: training paths must be real directories owned by you, without group/other write access.")
PY
}
verify_environment() (
  check_directories "$root" "$base" "$venv" "$venv/bin" || exit 1
  [ -f "$venv/pyvenv.cfg" ] && [ ! -L "$venv/pyvenv.cfg" ] &&
    [ -f "$venv/bin/activate" ] && [ ! -L "$venv/bin/activate" ] &&
    [ -x "$venv/bin/python" ] && [ -x "$venv/bin/pip" ] || {
    printf 'STOP: the training environment is incomplete or unsafe.\n' >&2; exit 1
  }
  . "$venv/bin/activate" || exit 1
  [ "${VIRTUAL_ENV:-}" = "$venv" ] && [ "$(command -v python)" = "$venv/bin/python" ] || {
    printf 'STOP: activation points to another location; use the training-environment recovery instructions.\n' >&2; exit 1
  }
  python - "$venv" <<'PY' || exit 1
import pathlib, platform, sys
import pip
expected = pathlib.Path(sys.argv[1])
assert pathlib.Path(sys.prefix).resolve() == expected.resolve(), "wrong environment prefix"
assert pathlib.Path(sys.executable).parent.resolve() == (expected / "bin").resolve(), "wrong interpreter directory"
assert pathlib.Path(sys.executable).name == "python", "wrong interpreter"
assert platform.python_version() == "3.11.6", "wrong Python version"
assert expected.resolve() in pathlib.Path(pip.__file__).resolve().parents, "pip belongs to another environment"
PY
  module_pip="$(python -m pip --version)" || exit 1
  entry_pip="$("$venv/bin/pip" --version)" || exit 1
  [ "$module_pip" = "$entry_pip" ] || { printf 'STOP: pip entrypoint uses another environment.\n' >&2; exit 1; }
  printf '%s\n' "$module_pip"
  deactivate
)
# <!-- /passport-snippet:euler-environment-check -->
check_directories "$root" "$base" "$venv"
[ ! -e "$venv/.passport-create-in-progress" ] || {
  printf 'STOP: a creation may still be running; ask lab IT before recovery.\n' >&2; exit 1
}
if verify_environment >/dev/null 2>&1; then
  printf 'existing-environment-ok: no recovery is needed.\n'
  exit 0
fi
python - "$venv" <<'PY'
import fcntl, os, pathlib, re, stat, sys, tempfile
venv = pathlib.Path(sys.argv[1])
base = venv.parent
# Cooperating recovery attempts cannot move the same target twice.
fd = os.open(base / ".passport-recovery.lock", os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
with os.fdopen(fd, "w") as lock:
    info = os.fstat(lock.fileno())
    if not stat.S_ISREG(info.st_mode) or info.st_uid != os.getuid() or info.st_mode & 0o077:
        raise SystemExit("STOP: unsafe recovery lock; ask lab IT.")
    try:
        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        raise SystemExit("STOP: another recovery is running.")
    before = venv.lstat()
    if not stat.S_ISDIR(before.st_mode) or before.st_uid != os.getuid() or before.st_mode & 0o077:
        raise SystemExit("STOP: recovery requires a private training directory owned by you; nothing was moved.")
    if os.path.lexists(venv / ".passport-create-in-progress"):
        raise SystemExit("STOP: a creation may still be running; nothing was moved.")
    # The private target blocks access by other users, even when Python copied
    # a group-writable activation-script mode from its installation template.
    def read_owned(path):
        info = path.lstat()
        if not stat.S_ISREG(info.st_mode) or info.st_uid != os.getuid() or info.st_size > 65536:
            raise SystemExit("STOP: unexpected training metadata; ask lab IT.")
        return path.read_text(encoding="utf-8")
    marker = venv / ".passport-training-environment"
    known = False
    if os.path.lexists(marker):
        known = read_owned(marker) == "passport-python-training-v1\n"
    elif (venv / "pyvenv.cfg").exists() and (venv / "bin/activate").exists():
        config = dict(line.split(" = ", 1) for line in read_owned(venv / "pyvenv.cfg").splitlines() if " = " in line)
        check_bin = (venv / "bin").lstat()
        if not stat.S_ISDIR(check_bin.st_mode) or check_bin.st_uid != os.getuid() or check_bin.st_mode & 0o022:
            raise SystemExit("STOP: unsafe training bin directory; nothing was moved.")
        old = pathlib.Path(config.get("command", "").partition(" -m venv ")[2])
        known = (config.get("version") == "3.11.6" and old.parent == base
                 and re.fullmatch(r"\.passport-python\.[A-Za-z0-9]{6}", old.name) is not None
                 and str(old) in read_owned(venv / "bin/activate"))
    if not known:
        raise SystemExit("STOP: this is not a recognized broken Passport training environment; ask lab IT. Nothing was moved.")
    backup = pathlib.Path(tempfile.mkdtemp(prefix="passport-python.saved-", dir=base))
    current = venv.lstat()
    if (current.st_dev, current.st_ino) != (before.st_dev, before.st_ino):
        raise SystemExit("STOP: target changed during recovery; nothing was moved.")
    venv.rename(backup / "environment")
    print("saved-training-environment:", backup / "environment")
    print("training-target-ready-for-recreation")
PY
)

Expected: existing-environment-ok means no recovery was necessary. training-target-ready-for-recreation means the original directory was saved and the training path is now free. Return to Inspect before creating, then Create or reuse the environment and Verify the active interpreter in the Euler Python mission. The final success marker must be euler-python-env-ok.

If recovery stops: leave the files and permissions unchanged. A marker left by an abruptly killed creation is deliberately not cleared automatically; lab IT must first confirm no creation is running. Give the help route the failed step and sanitized error, without your full home path or private logs. Do not remove lock files or run a recursive delete to get past the checks.

Keep the saved environment until the replacement works and lab IT has reviewed anything you need to retain. Do not activate the saved directory: it is a preserved copy for inspection, not a portable environment. Reinstall approved project dependencies from their recorded definition rather than copying package directories back.

Storage Choice#

$HOME is backed up but has a limited quota and is appropriate only for small environments. Euler scratch is temporary. The shared Lustre work filesystem is not recommended for Conda environments because an environment contains many small files. For large dependency stacks, use the project-approved container workflow or ask for the project’s established environment location.

Check environment size periodically:

du -sh "$HOME/passport-euler/venvs/passport-python"

This command only measures the training environment. Do not delete an environment merely because it is larger than expected; identify its owner and dependencies first.

Inside A Slurm Script#

The batch script must declare the software context before the application:

module purge
module load stack/2024-06 python/3.11.6
. "$HOME/passport-euler/venvs/passport-python/bin/activate"
python your_script.py

Do not rely on whichever environment happened to be active when sbatch was typed. Record the module version, dependency definition, Git revision, command, and relevant application configuration with the run.

Project Dependencies#

The Passport environment deliberately installs no project package. For a real project:

inspect the committed dependency definition and project instructions;
load the documented module and activate the project environment;
install exactly the reviewed dependencies;
run the project’s smallest verification command;
record any resolved lock file or version snapshot required by the project.

If installation fails, fix the dependency or platform problem. Do not request more job memory or a GPU to hide an import error.

Source And Freshness#

The training module stack/2024-06 with python/3.11.6 was verified on Euler on 2026-09-05. Recheck it with module spider python when preparing a new Passport release. Storage guidance follows the ETH Euler Conda and storage documentation; those primary sources take precedence when the cluster changes.

The final-location and recovery behavior was checked against the Python venv documentation on 2026-09-15. This is separate from the dated Euler module observation above.