Codex SQLite on node-local disk, with durable recovery

Table of Contents

Decision and scope

Extend SuCoder's local-disk tiering with an opt-in Codex state runner. Keep active SQLite databases outside the working clone, on node-local disk, and publish immutable recovery generations to private shared storage. Keep CODEX_HOME, authentication, global configuration, skills and hook trust at their existing locations. Use Codex's sqlite_home override at server startup, with a version-specific compatibility check.

This document specifies a design and the experiments needed before implementation and rollout. No proposed setting or command below is currently implemented. The existing architecture and test anchors are recorded in the prior-art ledger, sections 2–5.

The first production scope is SuCoder-managed Codex TUI and remote-control launches on Slurm targets with local_disk enabled. One service owns one state run, initially associated with one mirror. Connected clients may start several conversations in that service; they share its state run. Attaching a different client does not create a new database or change its placement.

Moving all agent homes, eliminating all shared-home access, migrating account-wide history automatically, merging independent conversation histories, and resuming an interrupted model request automatically are outside this change. Package caches and Git working trees already have their own local-tier mechanisms. Conversation recovery and Git WIP recovery are independent, with separate success reports.

Completion criteria for this design phase

  • Specify scope, identities, paths, startup, checkpoint publication, restore, shutdown, renewal, failures and retention.
  • Tie integration points to existing source and tests rather than introducing another allocation supervisor.
  • Identify claims that need an installed-Codex or Savio experiment.
  • Obtain independent storage and lifecycle reviews, record findings and dispositions, and revise the design.
  • Leave a runnable validation plan and handoff without starting a live cluster allocation.

Evidence and limits

Repository facts at main ffd0447

  • sucoder/local_tier.py:37 derives the allocation-local root; :57 exports cache and temporary paths, leaving agent state unchanged.
  • sucoder/mirror.py:2416 builds the confined batch body; :2480 prepares the local clone before starting its dedicated tmux server.
  • sucoder/mirror.py:2572 prepares the equivalent unconfined local clone.
  • sucoder/slurm_timer.py:204 is the existing warning/snapshot loop; sucoder/timer_lifecycle.py:9 deliberately puts ownership locks on node-local storage.
  • sucoder/session.py:84 saves launcher records atomically; those records are useful provenance, not authoritative evidence that a process is alive.
  • sucoder/renew.py:244 starts the replacement before :253 cancels the old allocation.
  • sucoder/cli.py:3601 writes a cooperative checkpoint sentinel; it does not acknowledge that Codex has stopped or that its databases are durable.
  • sucoder/cli.py:2932 releases allocations and may retire Git WIP refs; a conversation checkpoint must survive that retirement.

The measurements in the existing tiering note distinguish NFS home, Lustre mirror storage and node-local ext4 on Savio. Those measurements were made on 2026-09-09; this design session has not remeasured the cluster. In particular, SQLite's lock primitives must be tested directly; a successful flock probe is insufficient evidence for all SQLite modes.

Codex facts and compatibility gates

The official environment reference documents CODEX_SQLITE_HOME separately from CODEX_HOME, and says the sqlite_home configuration value takes precedence. The configuration reference documents sqlite_home for SQLite-backed runtime state. The launcher therefore supplies an absolute -c override; an environment export alone could be defeated by existing configuration.

Local inspection on this development host found Codex 0.162.1 and these database filenames under its current home:

Observed file Initial treatment
state5.sqlite Preserve; runtime metadata
threadhistory1.sqlite Preserve; conversation history
goals1.sqlite Preserve; verify continuation semantics
queue1.sqlite Preserve; verify replay semantics
memories1.sqlite Preserve; verify scope
logs2.sqlite Preserve until profile proves disposable

Filename presence is evidence of the storage inventory, not proof that all these files follow sqlite_home or that restoring them together is supported. The inventory is illustrative, not a hard-coded promise about every Codex version. The associated maintenance lock, possible additional databases, legacy rollout files, attachments and external paths must be included in the on-cluster inventory. An unclassified SQLite file disables recovery publication and makes the run visibly degraded until the compatibility profile is updated.

Installed codex --help and codex remote-control --help describe a shared background server and --no-daemon. The current adapter, sucoder/agent_mode.py:46, removes start without adding that flag. The earlier ledger's Codex 0.154.0 foreground assumption is not sufficient for this version. Managed state requires an independently owned foreground server: use a supported --no-daemon form and verify its actual PID, database paths and process lifetime. An existing daemon must never be stopped, retargeted or silently reused to satisfy this requirement.

Identities and layout

Stable scope, allocation and run

Scope
A SHA-256 digest of a canonical tuple: cluster identity, remote agent user, resolved shared mirror path, and service slot (initially default). Hash the unsanitized tuple; a display token is not a collision-resistant identity. Target aliases for the same storage must map to the same canonical scope; different clusters or users must not collide.
Allocation
Cluster, Slurm job ID, node, and scheduler start/submit identity when available. A numeric job ID can be reused and is not a durable primary key.
Run
A random UUID for one independently writable state history. The same run may restart in place within one allocation after its owner exits. A new allocation receives a new run UUID and records its exact parent generation when continuing earlier work.
Lineage
A UUID assigned to a fresh independent history and inherited by every sequential successor. At most one writer or unresolved successor attempt may own a lineage, regardless of which generation or transfer nonce it selected.
Generation
A monotonically increasing sequence within a run, plus a random suffix. A generation is immutable after publication and never overwrites a different run's generation.

Derive paths at runtime after SLURM_JOB_ID and the final cwd are known. Use the existing local-tier root helpers rather than concatenating job paths in several launchers.

LOCAL = /local/job<ID>/state/codex/<scope>/<run>
  db/                         active SQLite databases and sidecars
  snapshots/<generation>/     completed local backup bundles
  control.sock                private runner control endpoint
  owner.json                  allocation, run, PID/start identity
  status.json                 local state and last durable generation
  checkpoint.lock             local single-flight checkpoint lock

DURABLE = <shared-mirror-root>/.sucoder-state/codex/<scope>
  metadata.lock/              short atomic-mkdir critical section
  lineages/<lineage>.json      current owner/attempt authority, not a heartbeat
  transfers/<nonce>.json      offered/claimed/revoked transfer state
  pins/<parent-run>/<generation>/<child-run>.json
  runs/<run>/run.json          immutable origin/parent provenance
  runs/<run>/generations/<generation>/
    manifest.json
    db/<classified database files>
  runs/<run>/.incoming-<uuid>/  incomplete publication; never restorable

shared-mirror-root means the directory containing mirrors, not a repository's working tree. The default state root is therefore a sibling of all mirror repositories on the same durable filesystem. Reject a configured durable root inside any managed clone, inside its shared mirror repository, or under allocation-local storage. Do not infer a durable location from ~/.cache or place the state root on NFS merely because it is reachable there. Savio's proposed durable root is on the Lustre mirror filesystem; its exact path is confirmed in the spike.

Create agent-owned directories with mode 0700 and files with mode 0600. Refuse unsafe ownership, symlinked managed directories, path traversal and mismatched scope/run metadata. Resolve the configured root once and anchor filesystem operations to validated directory descriptors; never follow a changed root symlink during publication or restore. Validate ownership and permissions without chmod/chown of existing user directories; only newly generated private paths receive the required modes. Database bundles may contain private prompts and tool output even though credentials are excluded. No credential, auth file or wholesale copy of CODEX_HOME belongs in a generation.

Startup and placement

  1. Recognize a supported Codex launch before wrappers obscure the executable; management commands and already-connected clients are not new state owners.
  2. Reuse local-tier preparation and native instruction installation, resolving runtime paths inside the allocation and its existing cgroup.
  3. Acquire a local startup lock and inspect the existing run directory. If the same healthy owner exists, attach through the existing lifecycle; never reseed its database. A held lock with inconsistent metadata is an error, not permission to delete or kill anything.
  4. If local databases exist after a stopped owner, reuse them in place under the profile's supported restart/recovery policy; never replace them with an older shared snapshot. Validate identity and supported storage version first. Revoke outstanding transfer authority before reopening any writers, and refuse restart while a successor claim is unresolved.
  5. Otherwise select an exact predecessor under the restore rules below, or create an explicitly fresh run.
  6. Stage and validate the complete bundle locally, then atomically install the db/ directory before starting Codex.
  7. Pass sqlite_home=<absolute LOCAL/db> using the existing config_override TOML encoder, to the actual foreground server before it opens databases.
  8. Record the actual run and placement in local owner/status files and pane metadata; persist provenance in RemoteSession when the runtime identity is available. Confined job IDs are not known at script-render time, so the runtime record must be discoverable even if the launcher disconnects.
  9. Start the initial checkpoint promptly after Codex initializes its classified database set, without waiting a full checkpoint interval.

A managed launch becomes ready only when the runtime handshake names the requested scope/run, exact restored parent/hash, actual allocation/node, effective SQLite placement, and a healthy independently owned foreground Codex server. The handshake must distinguish initializing, pending allocation, ready and failed states and permit a live recheck before predecessor cancellation. Do not equate a returned launcher exit status or an existing tmux session with readiness: the existing surviving shell can keep tmux alive after the runner fails.

Keep user model, permission, prompt, global-home and hook settings intact. Opt-in managed placement conflicts with a caller's different explicit sqlite_home; report that conflict before launch rather than silently change either policy. The implementation must inspect user/CLI overrides and managed requirements and verify effective placement. If placement cannot be established, fail the opted-in launch with a specific error; never fall back to shared SQLite silently. Manual restarts from the surviving shell must go through the staged runner so they preserve placement and ownership. Exporting a hint can help, but it does not replace the runner or override effective Codex configuration.

Checkpoint protocol

Two consistency classes

Warm generation
Each classified database is copied using SQLite's online backup API while Codex continues running. Each backup is individually consistent, but the bundle spans a capture interval. SQLite does not make transactions across separate databases atomic as a set.
Sealed generation
A profile-verified graceful shutdown/quiescence has completed successfully, the owned Codex server and all its database writers have stopped, the runtime refuses new starts, and all classified databases are backed up before that run is reopened. The bundle represents a quiescent application state and carries an acknowledged handoff token when used for renewal.

An abnormal, forced or unknown exit creates a crash candidate even when every process is now stopped. Stopping writers alone cannot repair an application interrupted between commits to related databases. Such a candidate needs the same application recovery gate as a warm bundle and cannot acquire sealed status merely by being copied after exit.

The SQLite backup API supports live per-database snapshots. The WAL documentation explains the sidecars and the limits on cross-database atomicity. Do not copy live .sqlite, -wal and -shm files independently, force checkpoints on the running source as a backup substitute, or open shared copies as live databases. The backup API reads the source's committed state, including WAL content, into independent local destination databases. Set destination journal mode deliberately and close all destination connections before hashing or publishing; no required destination WAL may be omitted.

Initially only sealed generations from a supported shutdown/replay profile are eligible for automatic continuation. Warm generations are retained for explicit recovery experiments and are labelled per-database, not application-consistent. Promotion of warm generations to unattended recovery requires a version-specific test of state/history/goal/queue relationships and crash recovery. If those tests cannot establish safe recovery, add an upstream-supported export/quiescence mechanism or retain this restriction; do not paper over it with quick_check. Until that gate passes, a ten-minute capture interval is not a ten-minute guaranteed recovery window.

Capture and publication

  1. Coalesce timer, deadline, manual and shutdown requests under one local nonblocking checkpoint.lock. Permit one capture/publication worker per run; never queue an unbounded backlog.
  2. Discover the classified database set and reject unclassified SQLite files. An optional database absent before first use is explicitly recorded as absent; a previously present required store disappearing is a failure.
  3. Back up each required database into a new private local staging directory, with bounded busy retries and an overall capture budget. A single failed required database invalidates that whole candidate.
  4. Close destinations, verify PRAGMA quick_check, and calculate file sizes and SHA-256 hashes. For sealed generations, verify the owned server remains stopped throughout capture and publication.
  5. Write a local manifest containing format version, scope, run, lineage, allocation identity, parent generation, Codex version/build, storage profile, consistency class, per-database capture times, absent stores, sizes and hashes. Record the source cwd and external-file requirements without rewriting upstream database rows.
  6. Create .incoming-<uuid> under the destination run; copy only the completed local files. Use exclusive creation and file/directory synchronization appropriate to Lustre, then verify destination hashes.
  7. Write the manifest last, synchronize it and the staging directory, and atomically rename the directory to its unique final generation name on that same filesystem. Synchronize the containing directory where supported; the Savio spike must establish the filesystem's publication/durability contract.
  8. Only after success update local status.json with the durable generation and emit an acknowledgement carrying its identity and manifest hash. Lost publication acknowledgement is resolved by inspecting that exact generation, not by assuming it failed or selecting some other latest file. Transfer authority additionally requires the current durable transfer record below; a manifest alone cannot authorize continuation.

There is no shared mutable database and no cross-run latest pointer. The generation manifest is the commit record; directories under .incoming- are never candidates. A manifest's timestamp is explanatory, not a concurrency or ownership primitive.

Transfer fencing and parent pins

Use one short scope-wide metadata critical section for offer/claim/revoke, parent pinning and retention decisions. Acquire it by atomic creation of metadata.lock/ on the tested durable filesystem, with an immutable owner/attempt record; never use an NFS flock or a time-based lease. An acquisition that fails or returns an uncertain result does not grant ownership. Do not steal the lock because its timestamp is old; uncertain or abandoned ownership blocks metadata changes until explicit operator reconciliation proves that exact owner and operation can no longer run. Keep capture, hashing, upload, database opening and model work outside this section. Brief metadata writes still need the same synchronization/atomic-replace guarantees tested for publication.

After a graceful drain and complete sealed publication, create an offered transfer record naming source run/lineage, exact generation/hash, source ownership identity, nonce and intended successor attempt if already known. Ordinary clean-exit offers may leave the successor unbound; bind it exactly once under the claim critical section. Issue a transferable acknowledgement only after that record is durable. The old runner keeps its local start barrier and checks the durable record after any restart. Under the critical section, a successor claims the offer exactly once, binding its new run/allocation/attempt identity and pinning the exact parent generation before its database writers may open. Publish the child's immutable run.json and parent pin under that same section. If the child allocation ID is not yet assigned, claim to its attempt UUID first and bind the actual job identity before accepting runtime readiness. Duplicate requests for the same exact attempt are idempotent; a different attempt cannot consume a claimed offer.

Before the predecessor restarts, it must atomically revoke an unclaimed offer. It cannot revoke a claimed offer or reopen writers while successor startup is pending or unknown. Recovery after a failed claim requires positive evidence that the exact successor never opened writers or has stopped, then explicit transfer reconciliation; node unreachability is not sufficient. The same fencing applies to an ordinary clean-exit checkpoint selected for continuation while its old allocation still exists. Historical sealed metadata remains a valid snapshot description after restart, but no longer grants implicit continuation authority. Lost handoff acknowledgements are reconciled against both the exact generation and current offer/claim/revoke state.

An ended-allocation recovery choice also records a successor claim under this critical section, so two launchers cannot silently continue the same parent at once. Enforce exclusivity over the entire lineage, not only an offer or parent generation: different historical generations must not grant separate simultaneous owners of the same upstream thread identities. Reject ancestor recovery while any descendant owner or startup attempt is active or unknown. Update the lineage authority record together with the claim and parent pin; it records who is authorized, while scheduler/runtime evidence establishes liveness. Missing provenance, multiple valid children or an incomplete claimed attempt requires explicit reconciliation, not arbitrary last-writer-wins selection. The first version does not offer concurrent forks of a saved Codex history: copying a database into a new run UUID does not change upstream thread identities or external transcript paths. Fresh independent histories remain supported; a future fork feature needs its own upstream-identity and external-file isolation contract.

Retention validates pins and marks an unpinned generation retiring under the same critical section before deleting its files outside it. A new parent pin cannot be acquired after that transition. Failed child setup conservatively leaves its pin; remove it only after proving that no restore or writer can still depend on it. Deletion interruption leaves a non-restorable retiring record and does not grant permission to select a missing parent.

Scheduling and outages

Extend the existing timer to enqueue requests using a nonblocking local control path. Do not perform backup, checksumming, shared metadata access or upload inline in slurm_timer.py's warning loop. The runner services requests and owns its single-flight worker; tmux/Slurm remain the allocation supervisor. Use elapsed monotonic time for cadence rather than counting successful scheduler polls. The initial proposal is one capture every ten minutes plus existing deadline thresholds and clean shutdown.

A timeout is a budget for waiting and escalation, not proof that a stalled filesystem syscall can be killed immediately. Keep at most one in-flight worker, mark it stuck when overdue, and allow deadline reporting and the agent to continue. Never start replacement workers while the old one retains its local single-flight lock. Keep the last completed local candidate for a retry after a failed upload; bound local staging to that candidate and any active worker. Local-only candidates are explicitly not durable. Display the age of the last successful durable checkpoint; during a shared-storage outage the recovery window grows without a finite guarantee.

Restore and history selection

Normal continuation

Persist a validated managed-owner descriptor alongside the existing RemoteSession.service_launch: Codex mode, requested command/model before generated overrides, scope, last run, requested parent generation, and durable root identity. The existing service_launch validator accepts only remote-control; it cannot represent a CLI-selected Codex TUI whose configured default harness is different. Managed TUI renewal must preserve the selected Codex command/model explicitly, without changing nonmanaged TUI renewal behavior. Do not save credentials or a transient allocation path as the durable identity. Keep backward-compatible defaults for older session records and heal missing records from actual runtime/pane metadata as existing service discovery does.

Within one allocation, local state takes precedence over every archived generation. Across allocations, prefer the explicitly recorded predecessor or a caller-selected exact generation. The source must meet the applicable consistency/replay policy and obtain a successor claim: either through a currently valid offered transfer from a drained service or recovery from an allocation proved ended by the scheduler. An unanswered scheduler query, unreachable node, stale PID or old heartbeat is unknown state. Do not infer that a service stopped because its tmux session exists or because a launcher record vanished.

If session provenance is lost, list compatible runs and their ancestry. Offer automatic selection only when there is exactly one eligible terminal lineage; multiple independent histories require an explicit selection or an explicitly fresh run. Never merge SQLite rows or let the newest wall-clock timestamp decide between unrelated histories. Independent concurrent jobs may start fresh runs, with separate run UUIDs and upstream conversation identities. Concurrent restored forks are deferred until their external-file and thread-identity isolation is proved.

Validation and compatibility

Restore only complete published generations matching scope, format, storage profile and checksums. Copy into a new local staging directory and verify every required store before activation. If the selected generation is incomplete, incompatible or corrupt, preserve it, report the exact failure, and require explicit fallback; never silently create an empty history. Codex upgrades may migrate only the disposable local copy after a pre-upgrade generation exists; a downgrade cannot automatically consume a newer storage profile. Unsupported builds stay opted out until their storage and process behavior are tested.

Cross-node cwd changes are an explicit test case: use supported Codex resume/cwd APIs or flags. Do not SQL-rewrite old cwd or rollout-path columns as a guessed portability fix. Verify external transcripts and attachments are still available; if SQLite redirection leaves them under shared CODEX_HOME, state that dependency in the manifest and operational output. SQLite relocation does not promise a complete offline copy of every conversation asset.

Restoring persisted queue entries, goals or background jobs must not duplicate actions or paid requests. Before production recovery, establish the installed build's replay behavior and a supported way to hold pending work for operator resumption. If no safe hold exists, unattended restoration of that profile is unsupported. Checking database integrity alone is not an application-level recovery test.

First opt-in and older history

Do not automatically seed a repo-local run from account-wide shared databases that may still have writers on another node. First opt-in either starts a clearly announced empty managed history or uses an explicitly selected compatible import prepared while its source writers are stopped. The existing shared Codex history remains untouched and accessible through its original placement. A future import facility must identify all dependent stores and account for unrelated repositories and private data; filtering unknown schemas is not part of this implementation. Report this change of history scope before a user begins a managed conversation.

Shutdown, release and overlapping renewal

Owned process and local control

The state runner remains alive while its owned foreground Codex process runs, forwards terminal signals appropriately, and preserves Codex's exit status. It may stop only the process/group it created and verified; never use process-name killing or stale recorded PIDs. TTY job control, Ctrl-C, local app-server ownership and remote-control reconnect behavior require actual subprocess tests. Do not let a background daemon escape the run's ownership barrier.

The runner's private local control endpoint accepts status, checkpoint and drain requests carrying scope, run and nonce. Reach it through the existing executor/in-allocation srun --overlap route for confined targets, not compute-node SSH outside the cgroup. An acknowledgement includes the same nonce, a sealed generation and manifest hash, plus proof that the owned service has stopped. The runner holds its local startup barrier while drained, so a manual restart cannot invalidate that acknowledgement.

Clean exit and explicit release

After a profile-verified graceful Codex exit, take a sealed checkpoint and publish its fenced continuation offer before returning to the existing surviving shell. After an abnormal/unknown exit, retain local state and capture a crash candidate without upgrading its consistency class. If publication fails, retain local state and show the previous durable checkpoint; do not report a successful save. A deliberate restart in that still-live allocation reuses local state through the runner.

Before sucoder release cancels an allocation, request a bounded drain/checkpoint from each managed state owner affected by that cancellation. Keep existing sibling-allocation protection: detaching one mirror must not drain or cancel its siblings. If sibling-detach stops this mirror's managed owner, drain/checkpoint only that departing owner before stopping its tmux session; the shared allocation and other owners remain live. If an operation merely detaches a client and leaves its owner running, take at most a warm checkpoint and keep its run active rather than issuing a false sealed acknowledgement. Explicit release remains the user's cancellation instruction even if state publication fails; print the failure and last durable generation before proceeding. Never retire the last conversation checkpoint merely because a Git WIP ref is retired. SIGKILL, OOM, admin cancellation and node failure may bypass the clean-exit path; periodic recovery and its stated limits cover those cases.

Planned renewal sequence

  1. Preserve the existing cooperative handoff sentinel for repository work.
  2. Ask the old runner to drain, stopping new conversation work and its owned server, then publish a sealed generation. No unimplemented Codex application-drain API is assumed: the initial mechanism is an explicit graceful server shutdown, with active-chat disruption reported to connected clients.
  3. Require its nonce-matched acknowledgement before treating the state as transferable. If draining or publication is unconfirmed, defer automatic continuation while time remains; never start a replacement from unknown live state.
  4. Persist a replacement-attempt record retaining old allocation/node/run, offered token, new attempt/run UUID and pending/new job identity before allocation starts. Pass an explicit replacement authorization through launcher setup, excluding only the verified drained predecessor from the confined scheduler-adoption probe. The current _launch_confined at mirror.py:3058 rediscovers and adopts a still-live same-target job even after YAML is cleared; clearing RemoteSession.slurm_job_id alone cannot authorize overlap. Keep normal duplicate-job prevention and do not ignore any unrelated live job. Claim the exact sealed parent and launch the replacement according to the transfer protocol. The old allocation can remain alive, holding its drained startup barrier, even though its Codex service has exited.
  5. Require a different new job identity and the managed runtime readiness handshake, then recheck its run/parent/server liveness before cancelling the old allocation as the existing overlapping renewal loop does. tmux has-session alone is not readiness and must not trigger cancellation.
  6. If replacement allocation or launch fails, keep the old local databases and released-service barrier intact, and report the resumable old allocation. Keep pending/submitted replacement jobs in the attempt record and probe/adopt that exact attempt before retrying; a timeout must not leak a second replacement. A controlled restart of the old service requires the durable revoke/reconciliation protocol before its writers reopen; a later attempt must drain and seal again.

The generic renewal loop currently has a request_checkpoint() -> None hook and launches regardless of checkpoint success. Its extension needs typed drain and launch outcomes for managed Codex (ready, pending, retryable failure, or lost/unknown owner), with existing behavior retained for other harnesses. The current _relaunch_session at cli.py:3421 clears the old allocation fields before attempting a launch; retain old and pending-new records throughout the transaction rather than replacing old provenance early. If walltime overtakes a failed handoff, classify the old run as lost and apply the crash-recovery policy explicitly. Starting a replacement service does not automatically resume active chats or attach clients; recovered conversations become available for a supported explicit resume.

Proposed configuration and operator surface

Keep the feature off by default, separately from Git local-disk tiering. Superseded (issue 51, 2026-10-20): a Savio target with local_disk but no codex_state block left a remote-control Codex's SQLite on Lustre, and the session stopped making progress about ten minutes after launch, with its SQLite worker thread in uninterruptible I/O wait (not proven causal; no controlled comparison). enabled now defaults to whether local_disk is set (CodexStateConfig.implied marks that default); enabled: false opts out; an implied default with --no-local-disk launches unmanaged with a warning instead of refusing, while an explicit enabled: true still refuses; and every Codex launch on a SLURM target logs managed or unmanaged placement. Proposed target slurm keys, to be implemented together with parser tests, README.org and config.example.yaml:

slurm:
  local_disk: true
  codex_state:
    enabled: true
    checkpoint_minutes: 10
    checkpoint_timeout_seconds: 30
    durable_root: null

durable_root: null derives the private sibling directory under the resolved shared mirror root. checkpoint_minutes must be a positive integer and checkpoint_timeout_seconds a positive bounded duration; booleans are not accepted as integers. Reject enabled state without Slurm/local-disk support or without a supported owned-server launch. Require an explicit compatibility profile before enabling recovery for a Codex build; do not infer support solely from a version string. Use the same interval for periodic state capture and a separate configured Git WIP cadence where required; their outcomes remain independent.

Minimum operator surface: display scope/run, active path, last durable generation/class/age, selected predecessor, and degraded reason at launch and in session status. Expose exact-generation selection and an explicit fresh-history choice; propose --codex-state-from RUN/GENERATION and --codex-state-fresh as mutually exclusive launch options. Those options select history, never a free-form shell command or arbitrary database path. Provide a status/checkpoint operation through the staged helper so the cluster spike and operators can request a save without terminal-input injection. Log status transitions once and periodic degraded heartbeats; do not print SQL rows, transcript contents or credentials.

Retention and storage cost

Keep the newest three completed warm generations and the newest sealed generation per run; never delete a generation referenced as another run's parent. Prune only after a new generation is fully published, and only within that writer's run, using the serialized parent-pin/retiring protocol above. Partial upload cleanup requires proving no active writer owns that exact attempt; age alone is not that proof. The first version performs no automatic whole-run deletion across allocations. Expose per-scope bytes/run counts so operators can see accumulated histories; a later explicit retirement command must preserve pinned parents and require reliable ownership evidence. This differs deliberately from Git WIP retirement: cleanly ended conversations still have value.

Full database backups may grow with conversation history and are not assumed cheap. Measure capture time, bytes published, metadata operations and restore time with realistic state sizes before accepting a ten-minute default on Savio. If cost is excessive, change cadence, compression or a supported export strategy; never replace the correctness protocol with live-file rsync. Exclude diagnostic logs_2.sqlite only after the version-specific profile proves it is disposable and independent of recovery.

Failure contract

Event Required behavior
Same allocation, Codex restarts Reuse current local state; do not overwrite it from archive
Shared storage unavailable at restore Report blocked restore; require explicit fresh-history choice
Shared storage fails while running Continue locally; keep last durable generation and show growing age
Worker stalls in filesystem I/O One in-flight worker; watchdog and conversation remain independent
Local disk fills Report failure before new writes where possible; never fall back to shared SQLite
Capture, checksum or upload fails No new durable acknowledgement; preserve previous generations
Crash during upload Ignore incoming directory; use an eligible completed generation
Cross-database warm recovery unproven No unattended warm restore; retain recovery candidate for explicit tests
Unknown predecessor ownership No automatic continuation or destructive cleanup
Two jobs share a mirror Separate run directories and histories; no last-writer-wins pointer
Renewal handoff unacknowledged Defer continuity; do not restore potentially active state
Queue replay cannot be held safely Profile cannot enable unattended recovery
Codex storage layout changes Stop claiming compatibility; preserve evidence and update the profile
Explicit release after save failure Print failure/last durable checkpoint; honor cancellation

Implementation slices and validation

Slice 0: capability spike on Savio

Move development to an existing or explicitly allocated short Savio job after design review. Use a disposable mirror, synthetic conversation markers, an isolated test Codex home, and a private state root; do not relocate an active account's files. Read ~/.sucoder/bulletin.org before diagnosing an infrastructure failure and append observations before attempting infrastructure repair. Do not acquire or cancel another agent's allocation.

Run these read-only inventory commands inside the test allocation and capture host, UTC time, job identity and exit status:

hostname
date -u
codex --version
codex --help
codex remote-control --help
findmnt -T "$HOME"
findmnt -T "$SUCODER_LOCAL_ROOT"
findmnt -T "$SUCODER_SHARED_STATE_TEST_ROOT"

The test launcher must set SUCODER_LOCAL_ROOT and SUCODER_SHARED_STATE_TEST_ROOT to verified private directories before these commands. Use /proc/<owned-pid>/fd and, if available, strace in the synthetic test process to inventory every SQLite file, WAL, shared-memory file and maintenance lock actually opened. Exercise both sqlite_home and conflicting environment/config values and record the effective result. Verify the shared Codex home receives no SQLite writes from that managed process. Record non-SQLite accesses separately; retaining shared configuration does not eliminate them.

Use the fake-model app-server pattern in tests/test_codex_context.py:158 to create deterministic threads without paid model calls. Keep its fixtures' credentials/configuration isolated from the real account. Create a thread containing distinct known markers before and after capture; exercise compaction/history, a persisted goal and a queued action. Never test replay with a real external action or real credentials.

Required spike outcomes:

  1. Inventory and redirect every required SQLite store and maintenance lock to local disk in both TUI and remote-control modes.
  2. Prove each mode owns an independent foreground server and cannot reuse a peer daemon.
  3. Back up while history changes, stop the synthetic process abruptly at varied capture points, and restore into a different allocation path. Verify thread listing and actual supported resume expose all markers covered by the declared recovery generation, with no dangling history dependency.
  4. Verify queue/goal recovery does not duplicate or automatically execute test actions; establish a supported recovery hold or keep unattended recovery disabled.
  5. Drain and seal while the old allocation stays live, restore on the replacement, then test failed replacement and controlled restart of the old owner. Race two claims, delay acknowledgements, crash the old runner, and verify that token revocation or an unknown successor prevents reopening two writers.
  6. Inject upload failures, incomplete generations, scheduler errors, missing parent provenance and a lost acknowledgement; verify the failure contract.
  7. Compare shared-placement and local-placement synthetic workloads using identical build, seeded data, workload and concurrency. Record p50/p95 operation latency, failures, capture/upload size and duration, restore duration, and time since last durable generation. Do not generate cluster-wide contention deliberately or claim a speedup from an uncontended single run.

Use at least twenty randomized abrupt-stop/capture interleavings per supported storage profile with the seed recorded, plus deterministic failures at each publication boundary. This is targeted regression evidence, not a proof of cross-database atomicity. Review the upstream storage/recovery contract as well; if it remains unsupported, keep warm restoration manual. Interrupt between related database commits, then capture after all processes stop, to verify that post-crash capture never acquires sealed status. Archive scripts, exact commands, version/build, schema inventory, marker assertions and results with the design handoff.

Slice 1: placement and process ownership

Add a stdlib-only sucoder/codex_state.py runner/helper, private local paths and locking, supported-server launch adaptation, and typed configuration. Wire it after existing clone preparation and before Codex opens state; retain native context delivery. Use a fresh synthetic history for early tests; do not enable managed placement for valuable conversations before persistence exists. Tests must check actual database paths and owned-process lifetime, not merely generated argv strings.

Slice 2: generations and sequential continuation

Implement capture, complete publication, manifest verification, same-allocation reuse, sealed restore, status and bounded worker scheduling. Use real SQLite WAL writers, not mocked file copies, in targeted tests. Verify mid-capture failure, destination WAL handling, corrupt/missing files, unsafe paths, unknown stores, version mismatch, and no advance after failed publication. Race parent pinning against retention and test retiring records, abandoned metadata locks, partial claims and idempotent same-attempt retries. Race claims against different generations of the same lineage, reject ancestor restore with an unresolved descendant, and bind an ordinary-exit offer with no preselected successor. Test warm recovery eligibility separately and leave it disabled if its gate remains open.

Slice 3: lifecycle integration

Extend timer requests, RemoteSession serialization, live metadata discovery, release, and renewal with acknowledged state handoff. Exercise confined runtime job-ID expansion, unconfined placement, reattachment, two fresh runs sharing a mirror, duplicate starters, lost records, active predecessors, pending/recycled job IDs and failed replacement. Drive the real confined scheduler-adoption branch during renewal: a specifically authorized predecessor may be excluded, unrelated jobs may not, the new job must differ, and a surviving shell after runner failure must fail readiness. Verify renewal of a CLI-selected managed Codex TUI when the configured default harness is Claude. Verify unrelated harnesses retain existing renewal and release behavior and the Git snapshotter's index-preservation tests still pass. Run pytest from the SuCoder mirror root; document baseline environment failures rather than fixing unrelated paths here. Run GitNexus impact before editing runtime symbols and detect_changes before each implementation commit.

Slice 4: pilot and acceptance

Enable on one disposable Savio mirror first, then one deliberately selected real mirror after recovery gates pass. Demonstrate a clean turnover and recovery on a different allocation/node, last-checkpoint status during an upload failure, and independent histories for overlapping jobs. Promote only when all mandatory storage/process/replay gates pass and the measured snapshot cost supports the chosen cadence. Documentation must state any remaining external-file dependency, recovery restriction and interval limitation.

Rollback disables managed state on the next launch and leaves archives intact. It must not point ordinary Codex at a partly restored database or overwrite the original shared history. A user who needs the managed history after rollback selects a validated generation explicitly.

Review record and remaining decisions

Independent review findings and their dispositions will be recorded in the design review. Review readiness is distinct from production readiness: the Savio capability and recovery gates above remain open until measured.

Proposed defaults for review are a ten-minute capture interval, a thirty-second waiting budget, explicit first opt-in history scope, and sealed-only automatic continuation until warm recovery is established. The main unresolved capability decisions are complete SQLite redirection, independent server ownership, portable resume, queue replay control, and warm-bundle recovery. If a capability fails, revise the relevant contract before implementing around it.

Prepared by Sue.

Revision after Slice 0 (2026-10-15)

This section records what the Slice 0 spike changed and governs where it conflicts with the sections above; the measurements are in the spike record. The concern it answers is not Lustre's median latency but its episodes, when every operation takes minutes, so the measure that matters is the number of synchronous Lustre operations inside a turn, and the target is zero data operations.

What the spike established

  • All six databases follow sqlite_home; -c beats config.toml, which beats CODEX_SQLITE_HOME; a relative value resolves against the cwd; a missing directory is created with 0777 masked by umask; an unwritable one fails the launch with exit 1 and no fallback.
  • The rollout JSONL in CODEX_HOME/sessions/ is the authoritative conversation; the SQLite thread history is a projection of it with a byte watermark (thread_history_projection_state), every history read fails without the file, and the path is stored absolute (canonicalized, as the Slice 2 record below corrects; the spike read it as "as written").
  • A per-process bind mount (bwrap --unshare-user with the caller's own ids) of local directories over subdirectories of the shared home leaves the home untouched for every other process, keeps the stored path valid, and lets Codex's own sandbox run nested; with sessions, .tmp, tmp, thread-writer-locks, shell_snapshots, a copy of skills/ and the AGENTS.md, AGENTS.override.md and history.jsonl files bound, a turn contains no Lustre data operation and 1.7 metadata calls.
  • Appending the rollout's new bytes to a replica on Lustre after each turn costs 13 ms outside Codex's path and is byte-identical at the next turn/start; a database generation two intervals old plus the replica resumes fully.
  • A rollout ahead of the database is rebuilt into the projection; a rollout truncated mid-line is tolerated; a database ahead of the rollout leaves the SQLite history permanently wrong while the model's context stays correct.
  • A queued message executes once on unattended thread/resume; an active goal drives about twenty model requests per second until stopped; a goal set to paused is a usable hold and the queue has none.
  • Two servers with different homes on one sqlite_home are two writers with no refusal from Codex.
  • Stdin EOF leaves all data in WAL files; the backup API reads through them.

Layout

Where the earlier layout conflicts, this one applies:

LOCAL = <local root>/state/codex/<scope>/<run>
  db/                         sqlite_home: the six databases and sidecars
  home/sessions/              bound over CODEX_HOME/sessions for the managed server only
  home/.tmp/  home/tmp/  home/thread-writer-locks/  home/shell_snapshots/  home/app-server-control/
  home/skills/                a copy taken at launch, bound over CODEX_HOME/skills
  home/AGENTS.md  home/AGENTS.override.md  home/history.jsonl    file binds, only when the file exists in the home
  replica.json                per-thread replicated byte offsets
  snapshots/<generation>/     completed local bundles, as before
  startup.lock  owner.json  status.json  checkpoint.lock          as before

DURABLE adds, under runs/<run>/:
  rollouts/<thread-id>.jsonl   the append-only replica of each thread's rollout
  generations/<generation>/manifest.json records, per thread, next_rollout_byte_offset and the replica length at capture

CODEX_HOME, authentication, configuration and hook trust stay where they are; the managed server reads them through the bind namespace and nothing in the shared home is created, moved or rewritten. A bind target that does not exist in the home is skipped and recorded, never created: bwrap would otherwise create an empty file or directory in the shared home.

Launch

  • The launcher inserts -c sqlite_home with a placeholder at the argv stage, before any nvm or context-installer wrapping, and stages a stdlib runner the way it stages the native context installer; the runner runs outermost as python3 <runner> -- <command> and must be clean for the oldest python3 a node image offers (3.6 was found), because the staged installer is not.
  • At run time the runner resolves SUCODER_LOCAL_ROOT (exported before tmux in both launch flows), creates the layout with mode 0700, takes startup.lock, resolves CODEX_HOME, substitutes the placeholder with the absolute db path and refuses a path containing anything outside [A-Za-z0-9/_.-].
  • It refuses to launch when bwrap is unavailable, when the caller's argv, CODEX_SQLITE_HOME or config.toml already set sqlite_home, or when the home or local root is missing; it never falls back to shared placement.
  • It launches bwrap --dev-bind / / <binds> --unshare-user --uid <own> --gid <own> --die-with-parent -- <command> as a child in the same process group, ignores SIGINT (Codex handles it), forwards SIGTERM and SIGHUP, and exits with the child's code.
  • It verifies placement through /proc for the launch wrappers and native Codex processes, including a native app server started by the frontend. A non-Codex child of a native Codex process begins a tool subtree; that subtree's project databases and nested test Codex instances are excluded. Native processes are identified by /proc/<pid>/exe; unreadable metadata does not authorize excluding a process. Every .sqlite descriptor under db/ is placed; any inspected .sqlite descriptor elsewhere is misplaced, and the runner stops the child and fails with a specific error; a child that exits before opening a store reports "no stores opened".
  • owner.json and status.json record scope, run, allocation, the effective paths, the bound and skipped entries and every transition.

Checkpoint protocol, amended

Replication
a watcher in the runner follows each rollout under home/sessions by file growth, independent of launch mode, and appends new bytes to DURABLE/runs/<run>/rollouts/<thread-id>.jsonl with fsync, recording the replicated offset in replica.json. Replication lag is tolerated and shown as replica age; a Lustre episode slows replication, never the conversation.
Generation ordering
a database generation is captured only after every thread's replica is confirmed byte-identical to the local rollout at that instant; the manifest records each thread's next_rollout_byte_offset and the replica length.
Restore
a generation is restored together with the replicas, placed at the same relative paths under home/sessions and bound again; restore refuses any generation whose recorded offset exceeds the replica it would be paired with, because the projection would then skip real turns and show phantom ones.
Sealed
a backup-API copy taken after the owned server has stopped; a zero exit alone is not sealed.
Unattended continuation
before thread/resume without a client, the runner refuses when goals_1 holds an active goal or queue_1 holds rows for the thread, and offers paused as the hold for goals.

Residue and limits

The 1.7 remaining metadata calls per turn are the path-component lookups of CODEX_HOME (one per component, a depth-10 path today) and an occasional auth.json read; the skills calls were shown not to be load-bearing, the others were not failure-tested, and a shallower home path shortens the chain. Compaction with a real model summary, remote control beyond the relay step, and behavior during an actual Lustre episode remain unmeasured.

Slices, amended

Slice 1
placement and ownership through the runner above, with the typed slurm.codex_state configuration, no capture or replication yet.
Slice 2
replication, generations with the ordering and watermark rules, sealed capture, restore, status; implemented 2026-10-15, see the Slice 2 record below.
Slice 3 and Slice 4
as above.

Slice 2 record (2026-10-15)

What landed, what the implementation corrected in the record above, and what it leaves to Slice 3. The evidence is in the tests named here and in the commit messages; where this section conflicts with the sections above, this section applies.

What landed

Launcher (270e9cb)
every managed launch hands the staged runner --durable-root <BASE>, --checkpoint-minutes and --checkpoint-timeout, and --restore RUN/GENERATION or --fresh when the operator selects a history through --codex-state-from or --codex-state-fresh on agents-run and collaborate. BASE is slurm.codex_state.durable_root when set, else .sucoder-state next to the mirrors (the directory containing the resolved shared mirror path, a symlink onto Lustre on Savio); the launcher refuses a root that is, is inside, or contains the shared mirror repository, or that is under local_disk; the runner appends codex/<scope> and refuses a root whose realpath is under SUCODER_LOCAL_ROOT. The selector is validated on the launcher (32 hex, slash, six digits, dash, 8 hex) so it can only ever name a generation. Without a flag the runner reuses this allocation's run when one is intact and otherwise starts a fresh history; no generation is selected automatically across allocations.
Runner, replication and capture (f074a5d)
a replicator thread follows every rollout-*-<uuid>.jsonl under home/sessions by growth and appends the new bytes with fsync to DURABLE/runs/<run>/rollouts/<thread-id>.jsonl. The replica file's own size is the replicated offset, so a torn append can never double-write; a local rollout shorter than its replica is a "shrunk" anomaly that stops replication of that thread and fails later captures of it; durable errors stall only that thread and show as replica age and lag bytes in status.json. A checkpointer thread runs single-flight captures (nonblocking checkpoint.lock) on placement, every interval by monotonic time, on a touched checkpoint.request, and once after Codex exits; triggers during a capture are dropped and counted, never queued. Each store is backed up through the SQLite backup API, or the sqlite3 CLI where the interpreter lacks it (Python 3.6), never a file copy; the destination is switched to journal_mode=DELETE, quick_check=ed, closed, then hashed. The ordering rule is implemented in its monotone form, which amends the statement under "Checkpoint protocol, amended": the backup is taken first, each thread's local rollout size is measured right after the copies complete (and its =next_rollout_byte_offset read from the copy of thread_history_1), and the capture then waits, within its budget, until that thread's replica is at least that long; only then is the manifest written and the bundle published. Because the watermark equals the rollout size at every completed turn (next subsection), waiting for the local size subsumes the offset check and also covers a thread whose rollout the replicator has not discovered yet; f074a5d waited for the copy's offset only, and the review follow-up below strengthens it to the local size. Checking the replica before the backup would leave a window in which a turn completes between the check and the copy and the copy's watermark passes the replica. A candidate whose replicas do not catch up in budget stays local-only and is never acknowledged. Publication is .incoming-<uuid> with the copies hash-verified after fsync, the manifest written last, one rename into generations/<seq>-<hex8> and a directory fsync; a failure leaves the incoming directory and does not advance last_generation. The class is warm while Codex runs; after exit it is sealed on a zero exit with no pid still holding a descriptor under db/, and crash otherwise; the final capture is bounded by the timeout and never delays the exit code. The manifest records format, scope, run, generation, parent, class, allocation, codex_home, codex_home_real, codex_version (the highest threads.cli_version in the copy), backup_method, per-database sha256, bytes and quick_check, the absent optional stores, and per thread relative_path, next_rollout_byte_offset, next_rollout_ordinal, replica_bytes and rollout_sha256_prefix (the hash of the local rollout's first replica_bytes bytes, against which a restore verifies the replica). run.json holds the immutable provenance; status.json gains replica, checkpoint, last_generation and durable_age_seconds; the startup line announcing the fresh, separate managed history is printed before Codex owns the terminal.
Runner, restore and reuse (f92298b)
--restore RUN/GENERATION resolves DURABLE/runs/RUN/generations/GENERATION and validates before anything is created: the selector grammar (never a path; .incoming-* is never a candidate), the manifest (format 1, the same scope, the same run and generation, the same canonical CODEX_HOME because the stored rollout paths are absolute, a known class, no not_durable threads, only classified stores, usable relative rollout paths that carry the thread id) and every replica (at least as long as the recorded next_rollout_byte_offset, the database-ahead hazard, and hashing to the recorded prefix). The stores are copied to <run>/restore-staging/db/ and checked there (size, sha256, quick_check, no sidecar), the pending-work gate runs on those copies, each replica is copied whole in one read to home/sessions/<relative path> and to the new run's rollouts/<thread-id>.jsonl with both copies hashed back, and only then is the staging directory renamed into place as db/; run.json records the parent generation and later manifests carry it. An explicit restore accepts sealed, warm and crash generations and says so in the startup line (warm and crash rows are consistent per database only); --unattended accepts sealed only. Pending work is detected, never held by SQL: active rows in thread_goals and queued_items grouped by thread are reported in status.json and the startup line; under --unattended they refuse, as does an unreadable goals or queue table, because an active goal drives unprompted requests from the moment a thread resumes and a queued item executes once with no hold. Without --restore or --fresh the runner reuses this allocation's latest run in place when it has owner.json and db/ and its recorded runner and child pids are dead (a live pid refuses: a second writer on the same stores); the run id, local directory and durable run continue, bound files already present locally are kept, status transitions accumulate; --restore with an intact local run is ignored with a loud line and restore_ignored recorded, since within one allocation local state wins; --fresh always starts a new run. Verified against the real build: two turns, stdin EOF, a sealed generation, a restore in another local root, thread/resume, and a third turn whose model request carries both earlier texts. Verified with a real model as well: a nine-minute gpt-6-astra review session under the runner (ten warm generations at a one-minute cadence, a sealed final one, the 1.1 MB rollout replicated with offset = replica = local size), then its sealed generation restored under a different local root and resumed with codex exec resume, the model answering from the restored history alone; a retry after an upstream capacity error reused the run in place. Cost, as the retention section asked to measure before accepting the ten-minute default: with six stores totalling 1.1 MB (state_5 0.28 MB, thread_history_1 0.65 MB) after a nine-minute real session, a warm capture took 0.3-0.6 s wall including publication to Lustre, the rollout's replication added 13 ms per turn off Codex's path, and the restore of that generation plus its 1.1 MB replica took 0.2 s; at the default cadence this is six captures and about 6 MB per hour per managed run, which supports the default and makes retention (Slice 3) a housekeeping matter rather than a cost one.
Review fixes (f6e460f, 4298140, 3ff3608, 930eae4)
two read-only reviews by Codex (gpt-6-astra), run through the managed runner itself, found twenty-three defects across the two runner halves, each confirmed against the code before it was fixed, each fixed with a test. The ones that changed behaviour: a rollout that shrinks (or is replaced by another file at its path) while replication lags is an anomaly, never appended as growth; a copy whose threads or projection table cannot be read fails the capture instead of publishing watermark 0; a replica's bytes count as replicated only after its directory entry is synced; the acknowledgement of a generation is written under the in-flight lock and a generation whose generations/ sync failed is held as unsynced_generation until a retry succeeds; status.json is best effort and never stops the machinery; the exit path stats nothing and shares one shutdown deadline; existing durable components must be real directories owned by this user with no group or other write bits. On the restore side: a restore stages the stores and the replicas (the durable copies as .incoming-<thread-id>.jsonl), writes run.json, and activates last by renames, with a rollback that removes exactly what the attempt created; the status is restoring before the first durable read and blocked (exit 70) when the durable side does not answer within max(120 s, 4 x checkpoint_timeout); reuse in place requires the recorded canonical home and durable run to match and refuses unknown ownership; validation fails closed (fullmatch everywhere, the empty prefix hash verified, unlisted stores refused, the manifest hashed from the bytes parsed). The third pass extends the restore (and fresh-launch) transaction through bind preparation, owner.json and latest-run, so a failure there is rolled back too; fences the thread of a restore reported blocked at every durable write; creates and syncs missing ancestors above the durable base one by one; keeps the directory-sync retry inside the bounded final capture rather than on the exit path; clamps the restore timeout to one day; re-adopts a reused run's unsynced generation; and requires manifest counts to be integers. The fourth pass makes the fence and the local activation mutually exclusive under a lock (a deadline that passes after the activation has landed lets the launch take the completed result), refuses when the parent of an ancestor this launch created cannot be synced, and has a reused run judge its recorded unsynced generation by shape alone, with the existence check moved onto the capture worker. Accepted residuals: the prefix hash is computed from the local rollout and verified against the durable replica at restore; a thread whose rollout has 0 bytes passes without a replica; a blocked restore's abandoned thread rolls back when it wakes, so only a thread that never wakes leaves files under its run id, which no selector names; a restore fsyncs rollouts/ after renaming its replicas, so the replicator may seed their lengths on first sight; the sync of the durable base's own parent is best effort.
Launcher review fixes (c2efc73, d0d469e)
a history selection is refused when the launch would attach to or reuse a live session, because the selection travels only in the argv of a command those paths discard: a preflight probe refuses before anything is staged, and the selection also rides into each decision itself (_reuse_confined_job for the recorded or squeue-adopted job, the remote-service script when the session existed or won a race, and the TUI's tmux new-session built without -A so a duplicate fails instead of attaching), with has-session -t =name exact rather than by prefix; every path the durable-root refusals compare has its leading slashes collapsed (normpath keeps //); the resolved mirror path and the final root must be absolute, with an unexpanded ~ tolerated on a dry run only. Launches without a selection keep every path as it was, including the unselected service script's prefix matching.
Real-Codex harness (7f55e44, 6acc531)
tests/codex_app_server.py drives the installed Codex as an app-server against a recording fake model, and tests/test_codex_app_server.py establishes on the real build the contracts the runner rests on (next subsection).
Slice 3a, timer requests (18a3cab)
the Slurm deadline timer touches the run's checkpoint.request at the 30-, 15- and 5-minute warnings, after the WIP snapshot, through a best-effort shell function that creates nothing but that file and never touches the durable side; build_timer_script takes the scope digest and the local-disk root, and launch_agent passes the scope it already decided. Gap: an unconfined managed launch's timer is started from cli._start_slurm_timer, which does not know the scope yet.
Slice 3b, status surface (a9fa6ec)
sucoder codex-state-status <mirror> [--json] reads the recorded allocation's latest-run, status.json and owner.json with one cat each, inside the allocation through srun --jobid --overlap on a confined target (never compute-node SSH), and prints one line per fact: mode and run, state, replica age and lag and anomalies, last durable generation with class and age and any unsynced generation, checkpoint state, pending work per thread, the restore origin, and the refusal or blocked reason; a refused or never-placed launch, which writes no latest-run, is reported as no managed run recorded.
Slice 3c, drain and acknowledgement, runner half (a97674a)
a requester writes <run>/drain.request atomically ({nonce, mode release|renew, requestedat, requester}); the runner consumes it on its one-second poll, rejects a malformed one by renaming it, answers a repeated nonce from the existing acknowledgement and drops a second nonce while one is served. Who owns the terminal decides what a drain does, recorded once at launch as owner.json terminal: with none (an app-server, exec, remote control or --unattended) the runner stops Codex the way a forwarded SIGTERM does, with a SIGKILL bound at the checkpoint timeout, and the normal exit path seals (or records a crash); with tty a person may be mid-turn, so Codex is not signalled, a warm capture is taken now and acknowledged warm_only, and the final capture on the person's own exit updates the acknowledgement under the same nonce. The acknowledgement <run>/drain.ack (mirrored in status.json) carries the nonce, mode, outcome (sealed, crash, warm_only, or failed with a reason and the last durable generation), the generation's name, class, manifest hash and path, Codex's exit code, and the child's pid with its /proc start time and whether that exact process is gone, which is the stop proof (a dead pid alone is not, under pid reuse). After a release-mode acknowledgement the runner exits as before; after a renew-mode one it stays drained, holding startup.lock with nothing durable touched, until SIGTERM, SIGHUP, SIGINT or a release-mode request with a new nonce: that acknowledgement is the token Slice 3d's transfer record will carry.
Slice 3c, drain and acknowledgement, launcher half (e25d592)
sucoder release requests a release-mode drain from the managed owner before it cancels the allocation (and, on a sibling detach that stops the departing mirror's tmux session, before that stop), writing drain.request atomically through the same srun --jobid --overlap route the status reads use and polling drain.ack every two seconds until the nonce and run match or the bound (checkpoint_timeout_seconds + 30 s, --drain-timeout overrides, --no-drain skips) elapses; it prints one outcome line (sealed, warm only because a person holds the terminal, crash kept for recovery, failed with the reason, unacknowledged, or no managed run) and proceeds with the release in every case, because an explicit release is the user's instruction. A run that has already exited with a sealed or crash final generation, or that is refused, blocked or released, is answered without waiting; an unconfined job that is not running gets no request, because a release never allocates a node to ask. sucoder codex-state-drain <mirror> [--mode release|renew] [--timeout S] [--json] exposes the same request for operators and for Slice 3d, whose renewal will use mode renew and read the acknowledgement as its transfer token.
Slice 3d, renewal with acknowledged handoff (9f74f35)
sucoder renew replaces a managed Codex owner on the sealed generation its drain acknowledgement names, on a confined target. The session file records the owner at launch (managed_owner: command and model as given, scope, durable root) and the renewal in flight (replacement, a ReplacementAttempt: old job, node and run, the transfer token, the new job once sbatch answered, failures); older files load both as absent. After the git sentinel and its grace the loop drains the predecessor in mode renew; only a sealed acknowledgement with the child proven stopped becomes the token, a run that had exited cleanly with a sealed final generation counts too (the drain report now carries the run's state when there was no owner to ask), and warm only, crash, failed and no answer defer the replacement to the next turn. The replacement is launched detached with --restore on the token and --unattended (plumbed through CodexStateLaunch and runner_command), authorized to replace exactly the predecessor: launch_agent(replace_job)= has _launch_confined and the selection probe skip that one job in the id-keyed and squeue --name probes, any other live job under the mirror's name blocks the submission (raising rather than healing the record mid-transaction), and sbatch answering the predecessor's id is refused; the new job and node go into the attempt record while the session keeps naming the old allocation. Readiness (codex_state_readiness) is placed on exactly the token's run, generation and manifest hash, read from status.json through the status route; a job that left the queue, a refused, blocked, misplaced or exited run, a run reused in place or restored from anything else fails at once; the budget passing is pending, and the next turn probes the recorded job rather than submitting another. Ready promotes the job and retains the record until the predecessor's cancellation is confirmed; a failed replacement is cancelled by id and resubmitted next turn, three times at most, after which submissions stop while the loop keeps watching and says what to do. A predecessor gone before it could be drained, or recording no managed run, is relaunched with the owner's command and a fresh managed history, the log naming its generations for an explicit --codex-state-from; unconfined targets and dry runs take that path too. Residuals: the durable claim of the offered generation is Slice 3e's (two launchers could continue one sealed parent until it lands; the attempt record and the predecessor's held startup.lock cover the single-operator case); managed renewal on unconfined targets; the owner's run id is not healed from live metadata. Dogfooding this session: two Codex (gpt-6-astra) tasks were run under the Slice 2 runner on this node; both placed, published warm generations to the trial root, and were stopped by the runner's misplacement guard when Codex ran the runner's own test suite, whose nested fake-Codex stores the guard counted as state opened outside the managed placement (crash final generations). This false positive recurred on vantage on 2026-11-08 during test_resume_from_backup_plus_rollout_in_a_fresh_home; placement inspection now excludes tool subtrees as described above, so these tests can run inside a managed session. A drain request that arrives after the runner has exited is answered by the launcher side as no owner to ask, as 3c specified. Review fixes (355fad8): renewal resolves the active -T target for managed routing and the drain timeout; an unmanaged relaunch on a confined target excludes its known predecessor from scheduler adoption, including the lost-owner and no-run fallbacks; an exited run yields a token only with exit 0 and a sealed generation published at or after its newest exit transition, carried in the drain report; readiness checks the Codex pid and start time and the runner's presence through the allocation's status route, and rejects reused_run even without restore_ignored; a failed successor stays submitted and recorded until cancellation succeeds, with cancel failures logged and retried; promotion retains a ready record until predecessor cancellation is confirmed, the next renewal reconciles it first, and listings and release name whichever job the record holds beside the session's own; both managed renewal and its unmanaged fallback preserve the owner's remote-control mode, including explicit off. Confirmation pass (3ce05f3): transient liveness probe errors stay pending until a successful read can establish absence or a start-time mismatch; readiness rejects zombie/dead Codex and runner processes and checks the runner's newly recorded runner_starttime when present, retaining compatibility with older records; both release paths cancel the other job before forgetting its record, retain that record on cancellation failure, report the completed action and exit 1 so a later release can retry; every renewal probe reconciles a promoted ready record, including during HOLD, retiring it only on confirmed predecessor cancellation and logging failures for another poll; the 900-second no-status bound initially excluded only initial queue time, starting at the first observed RUNNING time (running_since) and deferring on scheduler probe errors. With the second confirmation below, each observed non-RUNNING state resets that bound, excluding queue time before the next RUNNING observation as well. An unconfined TUI attach keeps the existing managed owner after an exact-name tmux probe; two races remain tolerated and documented in _record_managed_owner: absent then created by another launcher before our attach records our command/model for their owner; present then ended before our create keeps the previous owner's command/model for our new owner. The service path's marker-based created logic identifies its own successful create exactly; the TUI path lacks it and its pane has no owner metadata marker. Second confirmation (159895d): requeue resets and saves running_since; only tmux probe exit 1 authorizes an absent-session owner record, other nonzero results log unknown presence and skip recording, and a present session logs that its owner record is kept; after predecessor cancellation, reconciliation reloads the session and retires only the same attempt and successor record, preserving allocation fields cleared by a concurrent release; the two TUI probe/create races above remain tolerated, with explicit documentation rather than exact creation detection. Recorded limitations: a refusal before latest-run remains bounded by the 900-second no-status timeout from the first RUNNING observation after each observed non-RUNNING state, with a separate startup-result file deferred to the runner in Slice 3e; a lost sbatch reply can leave a successor requiring operator reconciliation, with scheduler --comment attempt identity deferred and name-based duplicate protection retained; two confined allocations for one mirror on one node share a tmux socket, so a replacement can attach to its predecessor and hit the no-status bound, with a per-allocation socket deferred across attach, release, sessions and messaging.
Slice 3e, fencing, pins, lineages and retention, runner half (6fcf18e, review fixes in 178ee47)
the durable side of "Transfer fencing and parent pins" and "Retention and storage cost", without the offer, claim and revoke records, which follow Slice 3d. A scope-wide critical section DURABLE/codex/<scope>/metadata.lock/ is taken by atomic mkdir with an immutable owner record (run, allocation, runner pid and /proc start time, attempt, operation, acquiredat), waited for at most 3 s on the monotonic clock and never stolen; an abandoned lock is reported in status.json with its record, and the manual recovery step is in the module docstring and the --help epilog. A restore records pins/<parent-run>/<gen>/<child-run>.json and lineages/<child-run>.json under the section before the staged stores are activated, refuses a retiring or missing parent, and on rollback removes exactly its own records, keeping them when the lock cannot be taken. After each publication, within the publishing run and never on the final capture, retention keeps the newest three complete warm generations, the newest sealed and the newest crash generation, every pinned generation and lineage parent, and the one just published; the rest are renamed to retiring/<gen> under the section (tombstones that --restore and pins refuse) and their payload deleted outside it; no run is deleted whole. status.json gains retention, metadata_lock and scope_storage ({runs, bytes, complete, entries, measuredat}, bounded to 2 s and 100k entries), measured on the capture worker only. The review fixes: a lock whose mkdir returned is entered however long the call took, because the bound is on waiting for a held lock and treating a slow success as uncertain left the live runner's own lock behind for good; a failure to write or sync the owner record after a successful mkdir removes this attempt's own writes, and only a failed removal reports the lock unavailable; a failed release is logged and recorded as release_failed in status, never raised, because the pin, lineage and run.json are already durable and refusing the restore would send rollback to a lock it still held and orphan them; a fresh run writes its null-parent lineage and run.json outside the section, because that lineage protects nothing and a stalled retention cycle must not block every new managed launch in the scope; retention remembers the tombstones it has emptied and does not walk or sync them again in later cycles; and a cycle skipped because an earlier one is still running is noted in status, so the previous outcome is not shown as current. Recorded residuals: the first retention after an upgrade on a reused pre-3e run with N generations retires N-5 of them, with two syncs each, in one pass under the section, a one-time cost left uncapped; metadata_lock is rewritten on every acquire and release, and the scope_storage key names stand, since the launcher-side formatter belongs to another slice; a child restored before 3e has run.json parent but no pin or lineage, so if its parent run is reused in place under the new runner that run's retention can retire the child's parent generation, and a one-time backfill of lineages from runs/*/run.json is a follow-up; a fresh attempt's rollback still takes the section, so with the lock unavailable it keeps a null-parent lineage that retention ignores; after a failed release the live runner's later acquisitions report its own lock until an operator removes it.

Corrections to the record

  • threads.rollout_path is stored canonicalized (the realpath), not as written: with CODEX_HOME given through a symlink the row holds the real path, and Codex rewrites it on thread/resume to the new canonical location. The bind target is the realpath of the home, so the stored path resolves inside the namespace; a restore must place the rollout where the canonical path resolves.
  • next_rollout_byte_offset equals the rollout size at every turn/completed (42408, 49257 and 56106 bytes for three empty-output turns); the restore gate is offset less than or equal to replica length, never equality, and a longer replica is the tolerated "rollout ahead" case.
  • The database-ahead hazard needs a gap of several turns to reproduce: with a one-turn gap the next turn's append crosses the stale watermark and the projection resumes mid-record; with a three-turn gap the projection freezes and the resumed thread shows phantom turns, as the spike recorded.
  • thread/goal/set with status: active fires an unprompted turn within about ten milliseconds of the call; paused is a hold. No protocol request queues a message; codex queue --thread ID --message TEXT with the server stopped leaves one queued_items row that executes once on resume.
  • Under the bind layout thread/archive fails atomically with EXDEV: Codex renames the rollout from sessions/ to a flat archived_sessions/ and has no copy-and-delete fallback, so with archived_sessions unbound the move crosses devices, the row is untouched and the thread keeps working. Archiving a managed thread is therefore a known limitation until archived_sessions joins the bind set and replication, capture and restore cover it (an archived thread's path lies outside sessions/, and restoring one needs thread/unarchive before thread/resume).
  • The staged context installer was not valid for Python 3.6 and failed inside the runner on such node images; 96a77a4 makes it 3.6-clean with a test that runs the staged form under /usr/bin/python3.6.
  • Python 3.6's sqlite3 module has no backup API and its library (3.26) has no VACUUM INTO; the runner's CLI fallback depends on the sqlite3 binary being on the node image, and a capture without either fails rather than copying files.

Deferred to Slice 3

  • Transfer fencing, lineage pins and retention: generations accumulate unpruned under each run (a few hundred kilobytes per capture at the sizes seen here); only the local snapshots/ bundles are pruned to the newest three.
  • RemoteSession metadata and renewal with acknowledged handoff (timer checkpoint requests landed in Slice 3a; a drain request with a sealed acknowledgement is Slice 3c).
  • Warm generations remain per-database consistent only; the gate for unattended continuation stands.
  • Archiving a managed thread. Measured for Slice 3: binding archived_sessions as a second bind does not help, because each bwrap --bind is its own vfsmount and rename(2) across two mounts fails with EXDEV even for sibling directories on one filesystem; an archive succeeds only when sessions/ and archived_sessions/ share one mount (the whole home in one bind, or no bind), where it is a same-inode rename to a flat archived_sessions/<basename> with rollout_path rewritten and archived=1, the file immutable while archived, thread/resume refused until thread/unarchive moves it back. Under the managed layout thread/archive therefore remains an atomic, state-consistent failure whose only side effect is the dropped writer lock (re-created on the next resume); supporting it needs a single-mount layout or an upstream copy-and-delete fallback, and a replicator that follows a thread across the two directories by thread id with the manifest recording archived, archived_at and the archived path.

Slice 4 record (2026-10-16)

The first managed-state turnover pilot, what it found about a restored thread's working directory, how Codex 0.162.1 treats that directory, and the fix (issue 46). Where this section conflicts with the sections above, this section applies.

The pilot

SuCoder 9765aa1, Codex 0.162.1, target carleton-htc (confined, local disk, managed state on), run by the runbook codex-local-state-pilot.org. The predecessor was job 40038236 on n0059.savio4; the successor job 40038356 on n0037.savio4 restored the sealed parent generation c8f875045e1f4dcab946c52544f9bca8/000004-99f8c69f and reported mode: restore, state: placed and the parent's exact manifest hash. In the successor's TUI /resume listed the thread, and the model recalled two values from its history; the thread was present in the managed state database and absent from the shared one. Conversation continuity across allocations and nodes therefore passed. The resumed thread, however, kept the predecessor's working directory: threads.cwd and the TUI both named /local/job40038236/mirrors/codex-state-pilot-20261016, which does not exist on n0037, while the successor's clone was /local/job40038356/mirrors/codex-state-pilot-20261016 (issue 46). The pilot ran no tool in that directory. Separately, sucoder attach reported the running successor as an ended allocation because its scheduler probe treated a failed SSH probe's empty output as absence (issue 47, fixed on its own branch).

Measurements (Codex 0.162.1)

A scratch CODEX_HOME with an absolute sqlite_home and a loopback fake model that answers a marker message with an exec_command call running pwd; ls; one thread was started in the TUI in OLD (<tmp>/job111/mirrors/tok), OLD was then removed, and every cell below began from a pristine copy of the stores and the rollout, run from or toward NEW (<tmp>/job222/mirrors/tok). "Model cwd" is the <cwd> of the <environment_context> Codex sends; "tool" is where pwd ran. The measurement script, its JSON output and the TUI screens accompany issue 46.

Route Model cwd Tool threads.cwd after Rollout
Seed: TUI started in OLD OLD - OLD session_meta and turn_context record OLD
app-server started in NEW, thread/resume without cwd OLD fails: ENOENT OLD appends turn_context OLD
TUI in NEW, /resume, cwd prompt default ("Use session directory") OLD fails: ENOENT OLD appends
TUI in NEW, /resume, "Use current directory" NEW NEW NEW appends
TUI in NEW, /resume, "Always use current directory" NEW NEW NEW appends; writes [tui] resume_cwd = "current" to config.toml
TUI codex resume ID -C NEW NEW NEW NEW appends
codex exec resume ID run from NEW NEW NEW NEW appends
codex exec -C NEW resume ID NEW NEW NEW appends
codex exec resume ID -C NEW - - OLD refused: -C must precede resume
app-server thread/resume with cwd NEW, then a turn NEW NEW NEW (already before the turn) appends
app-server thread/resume with cwd NEW, no turn, stopped - - NEW appends one thread_settings_applied record (2914 bytes)
then a later thread/resume without cwd, from elsewhere NEW NEW NEW appends
app-server thread/resume cwd NEW first, then TUI in NEW /resume NEW NEW NEW listed under the default filter, no cwd prompt

Findings:

  • A resumed thread works where threads.cwd says, not where the client was started: the app-server without cwd and the TUI's default answer both take the stored value, and with the predecessor's clone gone every tool call fails with ENOENT. The TUI's prompt reads the stored value too ("Session = latest cwd recorded in the resumed session"); after the stored value changes, the TUI shows no prompt. codex exec resume is the exception and takes the process cwd.
  • Every supported override (thread/resume with cwd, -C before the subcommand, the prompt's "current" answer, tui.resume_cwd) moves the thread: Codex rewrites threads.cwd itself and only appends to the rollout; the restored bytes stay a byte-identical prefix, so the replica and prefix-hash invariants of the restore hold.
  • thread/resume with cwd persists without a turn and without calling the model.
  • The TUI's /resume picker filters by cwd by default and hides a thread recorded elsewhere; Right switches the filter to All and codex resume --all lists it. The pilot's "listed" therefore means the operator was on All.
  • thread/resume loads the configuration the thread was recorded with: a thread whose model provider was defined by a command-line -c fails with "Model provider … not found" when resumed without that override.
  • Scripted TUI homes must set check_for_update_on_startup = false: an Enter sent while the TUI showed its update offer accepted "Update now" and ran npm install -g @openai/codex, replacing the shared nvm install with 0.163.0 at 12:39; a reinstall at 12:44 then failed with EBUSY on an NFS silly-renamed binary still open elsewhere and left codex absent from PATH until 0.162.1 was reinstalled at 12:47, with the two half-removed package directories moved aside as lib/node_modules/.openai-codex*.broken-issue46-* (removable once the process holding the old binary has exited).

Lines 296-297 above stand: the move uses Codex's supported thread/resume with cwd, and no cwd or rollout-path column is written by SuCoder.

The fix

Launcher
right after it inserts the sqlite_home placeholder, the launcher derives a bare app-server command from the launch's own argv (codex_state.app_server_command: the executable, the placeholder and the launch's -c, --config, --enable and --disable overrides, then app-server; subcommands, prompts and other flags stay out), wraps it with nvm like the launch command, and hands it to the runner as --app-server JSON, before the selection flags.
Runner
on a restore or a reuse, after the stores are placed and before Codex starts, the runner reads threads read-only and classifies every row. A thread moves only when its cwd is the same mirror's clone, or a directory inside it, in another allocation on the same local disk: <disk>/job<N>/<rel>[/sub], where <disk>/job<M> is this launch's SUCODER_LOCAL_ROOT and <rel> (mirrors/<token>) is this launch's working directory relative to it; it moves to <cwd>/sub, or to the clone itself when that subdirectory does not exist here. The layout rule needs no job id from the manifest, so a thread recorded two allocations back moves as well. The runner then starts the app-server once under the same bwrap binds (the restored rollouts resolve only there) in its own working directory, sends initialize, then one thread/resume {threadId, cwd} per move, closes stdin and waits; the bound is max(60 s, 2 x checkpoint_timeout) (at most ten minutes), then SIGTERM and SIGKILL of its process group. The outcome is read back from threads and recorded in status.json as thread_cwd (clone, moved, failed with the error, kept with reasons, helper with exit code and duration), printed on the startup line, and shown by codex-state-status as one thread cwd: line. A missing, malformed or failing app-server never stops the launch: the history is intact, and the line names the remedy.
Policy for other cwds
left exactly as recorded and reported by reason: current (already this clone), archived (thread/resume refuses it until unarchived), pending_work (an active goal or queued item would run the moment the thread resumes, unattended and possibly paid), other_allocation_path (another allocation's local disk outside this mirror's clone, such as its tmp), external (anything else, including another mirror's clone). The app-server is not started when nothing moves.

Verification

  • tests/test_codex_app_server.py::test_a_restored_thread_works_in_the_successors_clone runs the real Codex under the runner as the pilot did: a two-turn thread in disk/job111/mirrors/tok, sealed; that allocation's disk removed; a restore under disk/job222 from its clone with --app-server; threads.cwd is the new clone before any client resumes the thread; a plain thread/resume and one turn recall both earlier messages and run pwd; ls in the new clone (its marker file is listed); the restored rollout is a prefix of the resumed one.
  • The pilot's own path, scripted once on this node with the same fake model: the runner with --restore and --app-server in front of the real TUI in disk/job222/mirrors/tok (tmux); /resume listed the restored thread under the default directory filter, picking it raised no directory prompt, and one turn carried the earlier message and ran pwd; ls in the successor's clone.
  • tests/test_codex_state_runner.py covers the layout rule, every policy category, the --app-server refusals, a hanging app-server stopped within its bound, a missing one, and a hermetic restore under the current interpreter and Python 3.6 whose child sees the moved cwds and whose appended record replicates as growth.

Residuals

  • A pre-existing thread that is archived or holds pending work keeps its old cwd; resume it with "Use current directory" (or codex resume ID -C <clone>) after dealing with the goal or queued item.
  • A thread whose configuration came from a command-line override the successor's launch no longer passes fails to move and is reported; its history is unaffected.
  • The app-server pass loads the user's configuration as an interactive resume does, with whatever hooks and MCP servers it configures; the measurements used a configuration with neither.
  • An app-server killed at its bound may leave its thread writer lock in thread-writer-locks, which Codex re-creates on the next resume.

Laptop cross-node acceptance retest (2026-10-16)

The repeat pilot tested PR 49 revision c1a9ede, tree 9cd6cbd4917896d770160b0ec549e4b3df48af37, including the reviewed attach correction (PR 48) and CI environment-test correction (PR 50), against Codex 0.162.1. Job 40042040 on n0043.savio4 sealed 55e7b62108264abeacc6265572fa657d/000004-635d426a; renewal restored that exact parent into job 40042340 on n0059.savio4 and confirmed predecessor cancellation. The successor reported mode: restore, state: placed, thread_cwd.moved_count: 1 and no failures; the cwd helper exited 0 after 10.5 seconds. Ownership was observed about 103 seconds after /exit. The pilot used two-minute private checkpoints, 50-minute allocations, a 49-minute drain threshold and ten-second checkpoint grace to accelerate the turnover; the normal target retained its ten-minute cadence. Both launches disabled the startup update prompt.

Normal sucoder attach and attach --via-srun reached the successor's existing confined tmux session. /resume listed the restored thread without asking which directory to use. It recalled the earlier phrase amber otter 942 and number 61, then a real shell tool ran pwd; head -n 1 README.org and printed /local/job40042340/mirrors/codex-state-retest-20261016 and the expected disposable-repository title. Read-only checks found the predecessor's clone absent on that node, the successor's clone present, the thread's stored cwd pointing at the successor, and no matching thread in the shared ~/.codex/state_5.sqlite. The original 104823-byte rollout prefix remained identical. This satisfies the live cross-node cwd acceptance item for issue 46 and repeats the live attach acceptance for issue 47; the broader storage, fencing, upload-failure and overlapping-history gates remain separate. The evidence and operator result are recorded on issue 46; raw JSON, renewal logs, the picker and tool output are retained in the private codex-state-retest-20261016 pilot bundle on the laptop and Savio.

GitHub Actions passed on Python 3.9 and 3.11 after the CI test fix. Locally, 407 combined CLI/remote/release tests and the three forced-color regressions passed. A full local run with umask 022 reported 1714 passed, six skipped and one pre-existing failure, test_restore_places_a_sealed_generation_in_another_allocation[current], also reproduced on the unchanged baseline: its final holder scan counts the runner itself while a snapshot reader is still open and conservatively records a crash before that reader drains. The cwd change does not alter that shutdown path; the actual two-node pilot sealed normally.

SuCoder — Home · GitHub