Automated Validation And Residual Risk

This page records what the generated-passport release candidate proves without an available student alpha or beta cohort. It distinguishes technical evidence from claims that automation cannot make.

Automated Evidence#

The test suite currently covers:

curriculum schemas, dependency expansion, cycles, platforms, and required mission sections;
exact drift protection for the critical Euler CPU and GPU starter profiles;
route generation for Windows, macOS, and Linux;
novice Git/Python/AI, Euler CPU, data, Blade, Euler GPU, supervisor, and universal-core-only personas;
30 isolated synthetic learners, each assessed three times concurrently at the same immutable state, with deterministic identical results;
one-current-mission state, OS-local resume registry, interruption/resume, and deterministic reruns;
secret-first stopping before learner JSON parsing, code execution, or AI;
protected route/workflow tampering, symlink escape, dataset/checkpoint, and unsafe GPU fixture attacks;
public exercise export, explicit pre-provisioning consent, separated transport and practice folders, exact fork/parent/author/branch validation, immutable passport.json, and completed closed-PR resume through mocks;
independent Git-practice verification for exact author, fork, branch, draft state, head SHA, changed file, and Conventional Commit;
signed Worker rejection of wrong signature, installation, repository, action, base, branch, draft state, malformed JSON, and oversized bodies;
minimal immutable completion records and append-only archive events;
disabled-by-default advisory AI review that cannot award a pass;
complete generated browser build, stable and exact-ref links, body/symptom search, canonical anonymous route expansion, JavaScript budget, and source-map/secret scans;
Chromium, Firefox, and WebKit journeys with mobile navigation, keyboard search, no-JavaScript mission access, serious/critical Axe checks, dashboard fail-closed behavior, 404 recovery, and attached desktop/mobile visual baselines;
an exhaustive generated-page scan at 320, 390, 768, and 1024 CSS pixels for one H1, unique IDs, and zero document-level horizontal overflow;
loopback sessions, Host/origin enforcement, exact-SHA/App-ID GitHub status, forged-cache downgrade, strict local-state contracts, symlink refusal, healthy single-instance reuse, and stale-instance recovery;
immutable GitHub Action references, App scope limited to public exercises, bounded private GITHUB_TOKEN persistence, credential-free isolated assessment, exact-SHA revalidation, and digest-pinned container base;
race-tested Go launcher packages and cross-compiled release-shaped binaries for Windows x64, macOS x64/arm64, and Linux x64/arm64, with checksums, SPDX, build-input provenance, installed build-dependency license inventory, and a host-local install/update/rollback plus offline-help smoke.

The CI matrix repeats curriculum, journey, portal, and platform-preflight tests on current GitHub-hosted Windows, macOS, and Ubuntu runners. Fake home paths contain spaces and malformed SSH configuration; preflight must report the problem without changing the file.

Reproduce Locally#

The release orchestrator stops at the first failing gate and writes a machine-readable report under the ignored dist/ directory:

PNPM=pnpm python3 scripts/qualify_release.py \
  --version 2.0.0 \
  --controller-app-id 123456

Replace 123456 with the reviewed immutable production Controller App ID. A dirty development run must add --allow-dirty; its binaries and report must not be published. CI additionally executes the Windows/Linux/macOS matrix and rejects high-severity production dependency advisories using the locked graph.

Build disposable outputs outside the repository:

python3 scripts/export_exercises_repo.py --help
python3 scripts/export_gh_passport.py --help
python3 scripts/export_onboarding_control.py --help
python3 scripts/build_portal.py --output /tmp/ideal-passport-portal
python3 scripts/qualify_release.py --help

Live Facts Checked#

On 2026-08-29, a read-only GitHub API query reported the IDEALLab organization on the Free plan. The public exercise repository can use protected main; protection is still a guard rather than endorsement authority. The external trusted manifest, controller state, exact-SHA reviewer approval, and private registry remain the authority.

No live GPU allocation was started for this release candidate. GPU onboarding uses static review fixtures and guarded command snippets. Existing dated Euler source checks remain in the canonical reference pages.

What Remains Unproven#

Automation cannot prove that an unknown novice understands a concept, that a heterogeneous population finds the journey appealing, that every current GUI label matches a future release, or that real organizational GitHub App credentials were configured with least privilege. These are explicit residual risks, not automated passes.

The overhaul is not ready for learner cutover until the static first-contact site, two public exports, private controller, signed Worker, protected exercise main, launcher release, and one synthetic non-admin transaction are live. Windows/Euler/NAS/Blade claims retain their explicit external gates. Measured learner usability must not be claimed before a beginner pilot occurs.

The launcher directly performs the bounded NAS write/read/delete probe. SSH, Euler accounting, and Blade checks validate sanitized observations entered by the learner; the central controller cannot independently act as that learner on those systems. The live pilot gates must therefore verify those procedures and must not be replaced by the synthetic receipts.