Automated Validation And Residual Risk

This page records standing technical gates and dated historical evidence. The active beta cohort has supplied feedback; that is not equivalent to a documented clean-machine or learning trial. Current release results are in the upgrade rollout.

Automated Evidence#

The test suite currently covers:

curriculum schemas, dependency expansion, cycles, platforms, and required mission sections;
exact drift protection for the critical Euler CPU and GPU starter profiles;
route generation for Windows, macOS, and Linux;
novice Git/Python/AI, Euler CPU, data, Blade, Euler GPU, supervisor, and universal-core-only personas;
30 isolated synthetic learners, each assessed three times concurrently at the same immutable state, with deterministic identical results;
one-current-mission state, OS-local resume registry, interruption/resume, and deterministic reruns;
secret-first stopping before learner JSON parsing, code execution, or AI;
protected route/workflow tampering, symlink escape, dataset/checkpoint, and unsafe GPU fixture attacks;
public exercise export, explicit pre-provisioning consent, separated transport and practice folders, exact fork/parent/author/branch validation, immutable passport.json, and completed closed-PR resume through mocks;
independent Git-practice verification for exact author, fork, branch, draft state, head SHA, changed file, and Conventional Commit;
signed Worker rejection of wrong signature, installation, repository, action, base, branch, draft state, malformed JSON, and oversized bodies;
minimal immutable completion records and append-only archive events;
disabled-by-default advisory AI review that cannot award a pass;
complete generated browser build, stable and exact-ref links, body/symptom search, canonical anonymous route expansion, JavaScript budget, and source-map/secret scans;
Chromium, Firefox, and WebKit journeys with mobile navigation, keyboard search, no-JavaScript mission access, serious/critical Axe checks, dashboard fail-closed behavior, 404 recovery, and attached desktop/mobile visual baselines;
an exhaustive generated-page scan at 320, 390, 768, and 1024 CSS pixels for one H1, unique IDs, and zero document-level horizontal overflow;
loopback sessions, Host/origin enforcement, exact-SHA/App-ID GitHub status, forged-cache downgrade, strict local-state contracts, symlink refusal, healthy single-instance reuse, and stale-instance recovery;
immutable GitHub Action references, App scope limited to public exercises, bounded private GITHUB_TOKEN persistence, credential-free isolated assessment, exact-SHA revalidation, and digest-pinned container base;
race-tested Go launcher packages and cross-compiled release-shaped binaries for Windows x64, macOS x64/arm64, and Linux x64/arm64, with checksums, SPDX, build-input provenance, installed build-dependency license inventory, and a host-local install/update/rollback plus offline-help smoke.

The private sandbox fast CI runs one Ubuntu job for each current source PR revision. It cancels obsolete revisions, builds the browser product only for relevant changes, and runs Chromium only for curriculum or interface changes. Known prose-only changes omit Go setup/testing; unknown paths, mixed changes and missing history retain it. Deleted product files still require checks. The manually requested release suite repeats curriculum, journey, launcher, and platform-preflight tests on current GitHub-hosted Windows, macOS, and Ubuntu runners. Full browser coverage runs sequentially per engine: Chromium and Firefox on Linux, WebKit on macOS for the closest Safari implementation. The Chromium job also builds all five launcher package shapes. No engine, viewport or assertion is omitted. Fake home paths contain spaces and malformed SSH configuration; preflight must report the problem without changing the file.

No workflow runs in IDEALLab/onboarding-IT. A signed repository webhook sends the exact same-repository PR SHA to the private sandbox control repository. The source repository's Actions switch remains disabled.

Control Qualification Cost#

Complete local verification before pushing a stable PR candidate. Group compatible edits for publication while keeping reviewable commits. Inspect the automatic fast run rather than manually dispatching it twice. Reserve the full matrix, including both macOS jobs, for the exact final product candidate. Rerun only failed jobs when the source SHA is unchanged; diagnose locally before paying for another run. A changed final candidate needs fresh full qualification. No final browser, OS, integrity or security gate is removed.

Agent-only reports need no launcher release. Read current paid usage before manual CI; at 75% of the owner-set monthly Actions budget, stop discretionary qualification and notify the owner. Student assessment shares this budget. See the agent verification playbook for scope and recovery commands. Local execution cannot prove an unavailable native OS or browser; record that limit and cover it in the final remote gate.

Reproduce Locally#

The 0.4.1 example below is historical. For a current qualification, supply the intended release version and reviewed Controller App ID from deployment config; do not publish an old-version or dirty candidate by copying this example.

The release orchestrator stops at the first failing gate and writes a machine-readable report under the ignored dist/ directory:

PNPM=pnpm python3 scripts/qualify_release.py \
  --version 0.4.1 \
  --controller-app-id 123456

Replace 123456 with the reviewed immutable production Controller App ID. A dirty development run must add --allow-dirty; its binaries and report must not be published. CI additionally executes the Windows/Linux/macOS matrix and rejects high-severity production dependency advisories using the locked graph.

Build disposable outputs outside the repository:

python3 scripts/export_exercises_repo.py --help
python3 scripts/export_gh_passport.py --help
python3 scripts/export_onboarding_control.py --help
python3 scripts/build_portal.py --output /tmp/ideal-passport-portal
python3 scripts/qualify_release.py --help

Live Facts Checked#

On 2026-08-29, a read-only GitHub API query reported the IDEALLab organization on the Free plan. The public exercise repository can use protected main; protection is still a guard rather than endorsement authority. The external trusted manifest, controller state, exact-SHA reviewer approval, and private registry remain the authority.

No live GPU allocation was started for this release candidate. GPU onboarding uses static review fixtures and guarded command snippets. Existing dated Euler source checks remain in the canonical reference pages.

What Remains Unproven#

Automation cannot prove that an unknown novice understands a concept, that a heterogeneous population finds the journey appealing, that every current GUI label matches a future release, or that real organizational GitHub App credentials were configured with least privilege. These are explicit residual risks, not automated passes.

The overhaul is not ready for learner cutover until the static first-contact site, two public exports, private controller, signed Worker, protected exercise main, launcher release, and one synthetic non-admin transaction are live. Windows/Euler/NAS/Blade claims retain their explicit external gates. Measured learner usability must not be claimed before a beginner pilot occurs.

The launcher directly performs the bounded NAS write/read/delete probe. SSH, Euler accounting, and Blade checks validate sanitized observations entered by the learner; the central controller cannot independently act as that learner on those systems. The live pilot gates must therefore verify those procedures and must not be replaced by the synthetic receipts.