Codex Local State: Slice 4 Pilot Runbook
Table of Contents
How to run the first pilot of managed Codex state from a laptop against the carleton-htc target, what to expect at each step, and what to bring back.
The design is Codex Local State; its "Slice 4: pilot and acceptance" section is the gate this runbook serves.
The first pilot on 2026-10-16, using 9765aa1, completed a cross-node turnover and recovered the conversation on successor job 40038356 on n0037.savio4.
Normal attach and attach --via-srun falsely reported that running allocation as ended after an SSH probe failure (issue 47); the direct login-node route below completed the recovery check.
The SSH failure's exact cause was not established during the pilot. Conversation recovery alone does not establish full Slice 4 acceptance; the restored thread's working directory also needed correction (issue 46).
Prerequisites
- The laptop's SuCoder checkout at
mainb925a15or later, including the issue 46 cwd fix and the issue 47 attach probe correction (pull it from the shared mirror); reinstall ifsucoderis installed as a tool rather than run from the checkout. - A throwaway git repository on the laptop; nothing in it matters, and it must not be a repository whose history you care about.
- On the cluster, the state of this writing: Codex
0.162.1from the nvm Node 20 install,bwrapand thesqlite3CLI on thesavio4_htcnode images, Python 3.6 as/usr/bin/python3(the runner is written for it). A newer Codex is not verified against the storage profile; the runner stops Codex if any store opens outside the managed directory, so a layout change fails loudly rather than silently. - Your Codex login on the cluster (
codex login statusthere says "Logged in using ChatGPT"); the managed runner uses the same~/.codexfor auth and config, only the SQLite stores and rollouts move.
Config change (laptop, ~/.sucoder/config.yaml)
Managed state is switched on per target, under its slurm: block; there is no command-line switch.
The two lines above codex_state are the ones the target already needs for confined, tiered launches.
targets:
carleton-htc:
# ... gateway, transfer_host, mirror_root: ~/mirrors, cert_file ...
slurm:
partition: savio4_htc
account: co_carleton
qos: carleton_htc4_normal
cpus_per_task: 4
mem: 16G
confined: true
local_disk: true
codex_state:
enabled: true
checkpoint_minutes: 2 # pilot cadence; the default is 10
checkpoint_timeout_seconds: 30
# durable_root: null # null -> ~/mirrors/.sucoder-state on the cluster (Lustre)
checkpoint_minutes: 2 is for watching generations arrive; set it back to 10 afterwards.
The durable root defaults to the sibling of the mirrors directory, ~/mirrors/.sucoder-state, which on Savio is a symlink onto Lustre; it is created on first use.
A root inside or around a mirror, or on local disk, is refused.
The pilot, step by step
Three terminals on the laptop. A short courtesy time makes the turnover arrive about thirty minutes after the start.
1. Start the session (terminal A)
cd ~/Projects/throwaway sucoder -T carleton-htc --time 0:50:00 collaborate --harness codex
The mirror appears as ~/mirrors/throwaway on the cluster.
Before Codex owns the terminal the runner prints three lines: the stores' path under /local/job<ID>/state/codex/<scope>/<run>/db, that the managed history for this mirror starts empty and separate from ~/.codex, and that generations go to ~/mirrors/.sucoder-state/codex/<scope>.
Have two or three turns with Codex that you will recognise later (ask it to remember a word, say).
2. Watch the state (terminal B)
sucoder -T carleton-htc codex-state-status throwaway sucoder -T carleton-htc codex-state-status throwaway --json > pilot-before.json
Expect mode: fresh, state: placed, a replica line with one thread per conversation, and last durable generation: 00000N-xxxxxxxx (warm, age ...) advancing every two minutes.
pending work: none unless you set a goal or queued a message.
3. Renewal (terminal C)
sucoder -T carleton-htc --time 0:50:00 renew throwaway --drain-minutes 20 --poll-interval 60 2>&1 | tee pilot-renew.log
--time here sizes the successor job; without it the target's configured time applies.
The loop holds while more than twenty minutes remain.
4. The turnover
When renew logs "~20 min to –time; checkpointing then re-allocating" (the tmux status line flashes a warning at thirty minutes, and the deadline timer requests a capture at each warning), exit Codex in terminal A with /exit.
This is the attended-TUI rule: while a person sits in the TUI the drain does not signal Codex, takes a warm generation and answers "warm only", and renew defers to its next turn; your clean exit seals the final generation and that seal is the transfer token.
If the renewal drain request arrived first, the drained runner holds its startup lock until the predecessor is released; the pane need not drop to a shell. The pilot observed this hold.
Expected in renew's log, in this order:
drained: sealed generation RUN/GENERATIONorno managed owner to drain: run ... exited (exit 0) and its final generation ... is sealed(the second when the exit landed before the drain request);Job N drained: sealed generation RUN/GENERATION (nonce ...); launching its replacement.;- the confined submission and node lines for the new job;
Replacement job M owns the managed history now (placed on RUN/GENERATION; run ...); job N is released by the renew loop.;relaunched: job N -> M.
If instead you see deferring the replacement with "not sealed" or "warm only", you are still in the TUI; exit it and wait one poll.
5. Verify continuity (terminal B)
sucoder -T carleton-htc codex-state-status throwaway # mode: restore; restored from RUN/GENERATION sucoder -T carleton-htc codex-state-status throwaway --json > pilot-after.json sucoder -T carleton-htc sessions # only job M for throwaway sucoder -T carleton-htc attach throwaway # the successor's Codex TUI
Normal confined attach uses the login-node SSH control, verifies the scheduler response, then joins the existing allocation through srun --overlap.
If it reports an unknown/unreachable allocation, keep the session record: check sucoder -T carleton-htc tunnel status, restore the connection with sucoder -T carleton-htc tunnel up, then retry.
The direct route that worked during the pilot is below; replace <login alias>, J and <mirror> with the configured login-node alias (for example carleton-htc-ln), the independently verified job id and mirror name.
The pilot used ssh -S none -t carleton-htc-ln to bypass the failing mux, then the same srun command. This joins the existing confined tmux session without creating a new one.
ssh <login alias> squeue --job J --noheader -o '%i|%j|%T|%N' srun --jobid=J --overlap --pty tmux -L sucoder-<mirror> attach-session -t sucoder-<mirror>
In the successor's TUI use /resume: the earlier thread is listed; pick it and ask what you discussed.
The answer must come from the restored history, not from ~/.codex (which never saw this conversation).
Then ask Codex to run pwd: it must print the successor's clone, /local/job<M>/mirrors/<name>.
Before Codex started, the runner moved every restored thread recorded in the predecessor's clone into this one through Codex's own thread/resume; codex-state-status shows it as thread cwd: 1 moved to /local/job<M>/mirrors/<name>, and the TUI resumes the thread there without asking which directory to use.
If that line says NOT moved instead, the conversation is still intact: when the TUI asks for the working directory choose "Use current directory" (the default, "Use session directory", is the predecessor's clone, absent on this node, and every tool call there fails).
Note the successor's node in sessions; a different node from the predecessor's is the stronger result, but the same node is not a failure.
6. End
sucoder -T carleton-htc release throwaway
release drains first (a sealed final generation, since you exit the TUI before it or it answers warm only and seals when you do), then cancels; if a replacement were still pending it would cancel that too.
It asks for confirmation; the pilot used release --force only after independently verifying the successor's sealed final generation.
Set checkpoint_minutes back to 10.
The pilot's generations stay under ~/mirrors/.sucoder-state/codex/<scope>/runs/ (a few megabytes); remove that scope directory by hand if you want nothing left.
What to bring back
The Slice 4 record needs evidence, not impressions:
pilot-before.jsonandpilot-after.json, andpilot-renew.log;- the
/resumeanswer (a sentence is enough) and whether the successor ran on another node; ls ~/mirrors/.sucoder-state/codex/*/runs/*/generationsand.../retiringafter the pilot (retention should have kept the newest three warm, the newest sealed, and any pinned parent, and retired the rest);- the wall-clock from
/exitto "owns the managed history now"; - anything the runbook got wrong.
If something refuses
The design forbids silent fallback, so every failure is loud and local.
| Symptom | Where to look | Meaning |
|---|---|---|
attach says "the allocation has ended" |
the scheduler response via the login-node control; independently check squeue --job J |
With the issue 47 correction, this means a successful empty scheduler response or an explicit invalid-job-id reply. A failed/timeout SSH probe instead reports unknown/unreachable, preserves the session record, and directs you to tunnel status / tunnel up. On 9765aa1 the message could be false; use the direct route in step 5 before concluding the job is gone |
The launch exits with "managed placement is unavailable" or a sqlite_home conflict |
the pane; ~/.cache/sucoder/job-throwaway-<jobid>.out |
bwrap missing on that node image, or your Codex config sets sqlite_home; nothing was started |
codex-state-status says state: refused with a reason |
the same log | the runner refused before placing (durable root, lock, pending work); the reason names it |
state: blocked on the successor |
renew's log, then codex-state-status |
the durable side did not answer within two minutes; the replacement fails readiness, is cancelled, and is resubmitted next turn (three times at most) |
renew keeps logging "deferring the replacement" |
terminal A | you are still in the TUI (warm only), or the drain was not answered in time; exit Codex, or raise checkpoint_timeout_seconds |
renew logs "3 replacement attempts failed" |
the session file ~/.sucoder/sessions/throwaway--carleton-htc.yaml (replacement:) |
the old job stays allocated and drained; fix the cause, remove the replacement record, or release |
sessions shows a job with no record |
sucoder sessions |
a replacement whose sbatch reply was lost; scancel it by hand (recorded limitation) |
| The successor never leaves "no managed run recorded yet" and is cancelled after fifteen minutes | the successor's job log | its runner refused before writing latest-run (the log has the reason), or two jobs of this mirror landed on one node and shared the tmux socket (recorded limitation); the next attempt usually lands elsewhere |
Known limits during the pilot
- The managed history is separate from
~/.codex: nothing is migrated, and/resumelists only managed threads. - Archiving a thread inside the managed TUI fails with an
EXDEVerror and changes nothing; leave threads unarchived. - Continuity across turnover needs either a clean exit or an unattended launch; an attended TUI that is never exited is not drained, and when its allocation ends the successor starts a fresh history with the old generations named in the log for an explicit
--codex-state-from RUN/GENERATION. - Generations are per-run; a fresh history on the successor is a new run, and the old run's generations remain for explicit continuation.
- A restored thread works in the successor's clone only when it was recorded in the predecessor's clone of the same mirror (or a directory inside it); a thread started elsewhere keeps its recorded directory, and an archived thread or one with an active goal or queued item is not moved before launch (
codex-state-statusnames each case). Choose "Use current directory" when the TUI asks, or start it ascodex resume ID -C <clone>. - The TUI's
/resumelist filters by directory; a thread recorded elsewhere appears only afterRightswitches the filter to All.