Explicit project preparation and LSMS verification
Table of Contents
Project profile
dispatch submit --project-profile PROFILE.json freezes an optional project
bootstrap and nonsecret configuration before submitting the job. With no
profile, the existing dispatch lifecycle is unchanged. The bootstrap runs
after the exact worker checkout is verified and before any native agent starts.
It uses the receiving node's paths and a separate project_setup phase.
A profile is versioned JSON. Unknown fields refuse. bootstrap names a shell
script relative to the profile file (an absolute path is also accepted); its
bytes are copied privately, so a later edit to that original file is irrelevant.
Fields are:
| Field | Meaning |
|---|---|
schema_version |
1 |
timeout_seconds |
Total deadline for supervised asset copying and bootstrap; at most one day |
environment |
Explicit nonsecret environment strings |
local_directories |
Environment names mapped to private relative output directories |
assets |
Relative input names mapped to absolute shared file paths and SHA256 |
credentials |
Explicit private reader files, copied privately at submission |
credential_environment |
Environment names mapped to declared credential names |
output_environment |
Names the bootstrap may supply through its JSON result |
thread_limits |
Positive pool limits no greater than the receiving CPU budget |
require_absent_environment |
Provider keys whose presence would violate the project data policy |
Environment strings support {worktree}, {project_root}, {assets},
{credentials} and {local_root}. Dynamic paths are resolved only after the
receiving root is verified. Assets are copied to private local files and the
copied bytes are checked against their declared digest. A queued image change
therefore refuses setup. No mutable master venv is substituted.
The bootstrap receives SUCODER_PROJECT_WORKTREE, SUCODER_PROJECT_ROOT,
SUCODER_PROJECT_ASSETS and SUCODER_PROJECT_RESULT. It writes a private JSON
receipt at the last path:
{"schema_version":1,"state":"ready","environment":{"PROJECT_PYTHON":"/local/job123/venv_img/bin/python"},"path_prepend":["/local/job123/venv_img/bin"],"runtime":{"image_generation":"example"}}
Only declared output names are applied. PATH prefixes must resolve to local
directories; the native agent/Node toolchain remains available.
PATH prefixes containing node, codex, claude, git, bash or sh refuse
so project tools cannot shadow the native agent or supervisor toolchain.
HOME, native homes, scheduler/control/Git identities and baseline environment protections
cannot be replaced. Local and credential bindings cannot be returned as new
bootstrap outputs. A child shell's export is not an environment result.
Provider credential environment variables are excluded from bootstrap. Project reader files
use a separate private channel; their values are absent from manifests and
receipts. The single-use staged copies are removed after local staging.
project-runtime.json records nonsecret runtime facts durably and is also
published under the existing hashed evidence manifest. The worker prelude
names this receipt and the prepared runtime convention. Setup failures
and TERM clean up owned helpers/mount processes even before a native child
exists, and retain a failed setup result and private diagnostics.
LSMS profile
examples/lsms/make_profile.py generates a concrete profile for an explicitly
selected image. It hashes the image and snapshots the bootstrap through the
normal profile contract. It does not select an image or discover credentials.
python3 examples/lsms/make_profile.py --image /shared/LSMS_Library/.venv.sqfs --output /shared/private/lsms-synthetic.json
The bootstrap reuses bin/savio_venv.sh mount from the PINNED LSMS checkout.
LSMS_VENV_SHARE points to the verified local image, independently of the
worker's private Git origin. Existing mounted/extracted runtimes refuse rather
than using the mounter's fast path for a potentially different image. Runtime
identity includes the fresh image input, generation, interpreter/prefix, actual
package import, effective countries root, and local config/data/DVC paths.
It never invokes update or uses the writable Lustre master.
Synthetic work has empty private project configuration/data and no staged reader. Ambient AWS provider keys refuse, including in the credentialed profile, so an unrelated model credential cannot silently open the data gate. The example therefore targets Codex/Claude authentication that does not require those AWS keys. It disables instance metadata discovery and import-time auth; this is an explicit execution profile, not a network sandbox.
For approved data work, additionally pass --reader /private/s3_creds. Opening
the real project's credential gate and running the selected country checks are
required evidence; having a reader file alone is not a successful data test.
To reuse approved immutable L1 inputs, also pass --shared-l1 PATH --country C
(repeat --country as needed). Only read-only content-addressed blob files
referenced by that scope are projected into private directories. Missing blobs
remain misses; new data and all writable metadata stay private. No shared
directory is symlinked as a writable cache and no shared cache is cleared.
The receipt records the scope and linked/missing counts. L2 outputs are private
for each implementer, verifier and reviewer.
Country fix followed by independent review
Run dispatch from the Savio coordinator; keep the source repository, profile,
task briefs and result root on shared storage. Fill
examples/lsms/task-template.org with one country/table, allowed edits, exact
control, evidence requirements and a bounded data/read/resource budget.
Select synthetic or approved-reader preparation explicitly.
sucoder dispatch --root /shared/private/dispatch submit /shared/LSMS_Library \ --ref FULL_CONTROL_COMMIT --agent codex \ --task-file /shared/private/implementer.org \ --project-profile /shared/private/lsms-profile.json \ --account co_carleton --partition savio4_htc --qos carleton_htc4_normal \ --cpus 1 --mem 8G --time 00:30:00 --exclude COORDINATOR_NODE
Inspect the explicit dispatch result and private results.git. Select the
published final_head only after checking task outcome, publication and proof;
a job disappearing from the queue does not select a candidate. Use that exact
repository/SHA for the deterministic job below. Preserve a meaningful control
failure and candidate pass with the same regression selection.
Give the reviewer a separate filled brief, the exact candidate and proof
identities. Submit another worker from the implementer's private results.git
with --ref FULL_CANDIDATE_COMMIT and the same explicit project profile.
Exclude the implementer's node as well as the coordinator when requiring
cross-node acceptance. Each allocation gets its own source, runtime and
writable data; the reviewer publishes to its own result repository. Requested
amendments return to the implementer as a new explicit assignment. The human's
development PR follows review and coordinator acceptance.
Deterministic verification of a private commit
The existing LSMS burst runner must accept an explicit repository and immutable
commit. dispatch verify-stage freezes those inputs, the project profile,
helpers and either a JSON command array or the bytes of an existing RUN script.
It allocates no resources and starts no model process. Each job needs a fresh
stage; automatic retries and stage reuse refuse.
Run the receiving command through a new burst allocation. A stage prepared in
a coordinator allocation records that job ID and refuses execution inside it;
allocation-wide helper cleanup must never be pointed at the coordinator.
sucoder dispatch verify-stage /shared/private/worker/results.git --ref FULL_COMMIT --project-profile /shared/private/lsms-synthetic.json --command-file /shared/private/tests.json --directory /shared/private/check-candidate --cpus 2 --require-evidence junit.xml sbatch --account ACCOUNT --partition PARTITION --qos QOS --export=NONE examples/lsms/burst.sbatch /shared/private/check-candidate
The adapter in examples/lsms/burst.sbatch is a reviewable replacement for the
hardcoded branch/mirror/runtime prologue of the existing shared burst script.
It is not installed over that shared script automatically. Resource flags
remain explicit Slurm inputs. --script-file RUN.sh preserves the existing
PY, CLONE, DATA and JOBDIR contract with receiving-allocation values.
A command file is a JSON argv array, for example:
["{python}","-m","pytest","tests/test_gh798_datetime_mixed.py","-k","not warm_interview_date","--junitxml={evidence}/junit.xml","-q","-p","no:cacheprovider"]
The standalone driver fetches one exact commit and its complete current tree
without alternates, records its shallow verification-history policy, prepares
the SAME project profile and runs from the verified checkout. Full history for
an implementer's prior-art work remains the generic agent dispatch policy.
Source/profile failures, test exit, cleanup and evidence publication remain
separate facts. Receipts and command logs are private in the stage; selected
JUnit/operands under evidence/ get the existing hash manifest.
Use repeatable --require-evidence names for mandatory proof files; an exit-zero
test command with missing required evidence is a failed verification result.
After owned processes stop, the driver rechecks HEAD, the real index and tracked
bytes against the frozen commit. Source drift fails verification even when the
test exits zero; untracked and ignored outputs remain allowed. The receipt
records the observed final source identity.
Scientific acceptance remains with the coordinator.
First acceptance
Replay the already reviewed GH1225 control/candidate selection, then one
bounded credentialed country check. An implementer publishes only to its
private results Git, a deterministic job tests that exact candidate, and an
independent reviewer on another node uses the same pinned candidate and its own
writable data. Require control failure/candidate pass, runtime/import identity,
selected proof after turnover and unchanged human-mirror Git state. Country
briefs must require the full CONTENTS.org and relevant caveats; correct the
known stale import/cache passages before copying project skills into a brief.