Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Measure M04 by where the planted turn is, and catch a section of the model's own

Browse Source 
The M04 re-run on 0c7316f ran 101 turns with no failures, and the verdict
still came out `precondition_not_met`. That verdict was wrong. The planted
player turn (depth 1) was 65 actions outside the history window, whose floor
was 66. The sentinel's text was in recent history for two other reasons.

- On 5 turns the narrator wrote a section of its own, `## Established:` over
  indented facts, with the planted clue copied into it from the state section.
  The extractor passed it: it was a single heading, and the `## ` meant it did
  not match. The harness leak count passed it the same way, and read 1 where
  5 turns leaked.
- On 4 turns the narrator used the sentinel as a name inside ordinary
  sentences ("the SILVER-KEY-CRYPT-OLD-ABBEY, flickers with latent power"). That
  is story text and cannot be stripped.

So the sentinel's text in history can never be the precondition. M04's
written pass is "Fact/event remains recoverable without entire transcript in
prompt" (V1-ACCEPTANCE-TESTS.md). The owner agreed on 2026-09-13 that the
precondition is positional, and that recovery through authoritative state
counts; memory is not required.

Extractor:
- A state-section heading is recognised with any markdown the model wrapped
  it in (`## Established:`, `**Held:**`, `> Held:`).
- One heading with an indented entry under it now qualifies as protocol.
  Before, a block needed two headings, or one heading and the scene line. A
  heading followed by unindented prose is still story. The earlier guard test
  "Held:" with an indented line is now protocol, and the case was rewritten
  unindented.

Harness:
- It records `planted_depth` when the clue is planted, and carries it across
  --resume. No endpoint reports an action's depth, but a fresh campaign's path
  is the opening, the planted turn and its reply, so the depth is
  `total_actions - 2`. That was checked against the database in three runs.
- `_recall` reports `planted_depth`, `history_floor_depth` and
  `planted_turn_in_history_window`. The verdict is `precondition_not_met` only
  when the planted turn is still in the window, and `precondition_unknown`
  when its depth was never recorded. `clue_in_recent_history_window` stays as a
  fact about the prompt.
- The protocol-leak count matches a state heading with an indented entry under
  it, in any markdown.

Every AI turn in five real runs was replayed through the new extractor: draco
M01, the two 26-turn GPU trials, the f8d4010 M01 run and the M04 re-run. 443
turns in all. No turn the old extractor had left clean changed, and the M04
re-run lost 7 more leaks. The sentinel-as-a-name turns are story and remain.
Trial 1's model-invented headings remain, as before.

The M04 re-run's own evidence, reclassified under the new precondition from
its database and its recall-turn snapshot, reads
`recovered_through_state_only`. The original recall.json is kept unchanged
beside `recall-reclassified.json`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136VBTMUKWYeU6G9HgbDbND
This commit is contained in:
JesseMarkowitz
2026-09-13 22:10:33 -04:00
co-authored by Claude Opus 5
parent 0c7316f951
commit 96c1bf5ded
4 changed files with 156 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/tools/m11_long_run.py
+63 -15
View File
@@ -98,6 +98,7 @@ from __future__ import annotations
import argparse
import json
import os
import re
import socket
import subprocess
import sys
@@ -361,6 +362,9 @@ class Run:
#: How far `background_failures` has read `server.log`. Carried across a
#: resume, so a failure is reported once, not again every session.
self.log_offset = 0
#: The depth of the player turn that planted the clue. M04's
#: precondition is that this turn has left the history window.
self.planted_depth: int | None = None
# ------------------------------------------------------------ recording
@@ -388,6 +392,7 @@ class Run:
"elapsed_seconds": self.elapsed(),
"turns_target": self.turns_target,
"log_offset": self.log_offset,
"planted_depth": self.planted_depth,
"written": datetime.now().isoformat(timespec="seconds"),
}
tmp = self.out / (RESUME_FILE + ".tmp")
@@ -408,6 +413,7 @@ class Run:
self.completed_steps = set(prior.get("completed_steps") or [])
self.elapsed_before = prior.get("elapsed_seconds", 0)
self.log_offset = prior.get("log_offset", 0)
self.planted_depth = prior.get("planted_depth")
self.resumed = True
def reattach(self) -> None:
@@ -739,7 +745,17 @@ def main() -> int:
run.setup()
# ---- The planted clue, at the very beginning. ----
run.turn(f"I tell Mara quietly that {CLUE} — {CLUE_SENTINEL}.")
planting = run.turn(f"I tell Mara quietly that {CLUE} — {CLUE_SENTINEL}.")
if not planting.get("accepted"):
raise SystemExit(
"the turn that plants the clue was not accepted, so M04 has "
"no planted turn to measure. Stopping before the campaign starts.")
# Where the planted turn sits, which is what M04's precondition is
# measured against. No endpoint reports an action's depth, but a
# fresh campaign's path is the opening (0), this player turn (1) and
# its reply (2), and the history report counts that path. It is
# checked against the database in the M04 re-run evidence.
run.planted_depth = planting["total_actions"] - 2
run.correct([CLUE_FACT], note="the planted clue, as accepted state")
# Proved here, at turn one, where it costs a single request. M04
# asks whether the clue is still recoverable a hundred turns later,
@@ -749,7 +765,7 @@ def main() -> int:
planted = any(CLUE_SENTINEL in json.dumps(fact) for fact
in run.state()["document"].get("facts") or [])
run.note("clue_planted", sentinel=CLUE_SENTINEL,
verified_in_state=planted)
verified_in_state=planted, planted_depth=run.planted_depth)
if not planted:
raise SystemExit(
"the planted clue is not in accepted state, so M04 cannot "
@@ -1119,6 +1135,16 @@ def _recall(run: Run) -> dict:
history_text = " ".join(
s["text"] for s in report["sections"] if s["label"] in HISTORY_LABELS)
in_history = CLUE_SENTINEL in history_text
# The precondition proper: has the planted turn itself left the window?
# The sentinel's text is no guide. The narrator reuses it in its own prose,
# once copied from the state section and once as a name, so it can be in
# recent history long after the planted turn is gone. `floor_depth` is None
# when the whole path is included.
floor = report["history"].get("floor_depth")
if run.planted_depth is None:
planted_in_window = None
else:
planted_in_window = floor is None or floor <= run.planted_depth
# 2. Ask about the subject, and see what the application assembles.
run.turn("I think back to what I told Mara about the key, that first night.")
@@ -1131,7 +1157,12 @@ def _recall(run: Run) -> dict:
fact_present = any(
CLUE_SENTINEL in json.dumps(f) for f in document.get("facts") or [])
result = {
# The sentinel's text in recent history, from any source. A fact about
# the prompt, kept for the report, and no longer the precondition.
"clue_in_recent_history_window": in_history,
"planted_depth": run.planted_depth,
"history_floor_depth": floor,
"planted_turn_in_history_window": planted_in_window,
"clue_in_prompt": CLUE_SENTINEL in whole_prompt,
"in_state_section": CLUE_SENTINEL in sections.get(STATE_LABEL, ""),
"in_summary_section": CLUE_SENTINEL in sections.get(SUMMARY_LABEL, ""),
@@ -1161,12 +1192,19 @@ def _recall(run: Run) -> dict:
def _m04_verdict(recall: dict) -> str:
"""What the recall check proved, in one word the report can quote.
The first long run with the bank on found the clue "in the prompt" at turn
100, but only because the narrator had pasted the state section into its
prose and the paste was still in recent history. A clue in recent history is
no evidence about memory, so that case gets its own verdict and never counts
as recovery."""
if recall["clue_in_recent_history_window"]:
M04 passes when the planted fact is recoverable "without entire transcript
in prompt" (`V1-ACCEPTANCE-TESTS.md`). So the precondition is that the
planted turn has left the history window. Any recovery path then counts:
memory, summary or authoritative state. The owner accepted state-only
recovery on 2026-09-13.
An earlier version used the sentinel's text in recent history as the
precondition. The narrator reuses that text in its own prose, so a run whose
planted turn was 65 actions out of the window read `precondition_not_met`."""
in_window = recall.get("planted_turn_in_history_window")
if in_window is None:
return "precondition_unknown"
if in_window:
return "precondition_not_met"
if recall["in_memories_section"] or recall["in_summary_section"]:
return "recovered_through_memory_or_summary"
@@ -1175,18 +1213,28 @@ def _m04_verdict(recall: dict) -> str:
return "not_recovered"
#: Signs the application stored protocol as story: the state section's own
#: headings, and a proposal's event list. Copied rather than imported from
#: `app.narrative.render`, for the reason `HISTORY_LABELS` is copied.
PROTOCOL_LEAK_MARKERS = ("Who and what exists:", "\nHeld:\n", "\nEstablished:\n",
'"events"')
#: Signs the application stored protocol as story. The first is a state-section
#: heading with an indented entry under it, in any markdown, because
#: `## Established:` got past a plain substring match and the count read 1
#: where 5 turns leaked. The second is a proposal's event list. Both are copied
#: rather than imported from `app.narrative`, for the reason `HISTORY_LABELS` is.
PROTOCOL_LEAK_HEADING_RE = re.compile(
r"^[ \t#>*_]*(?:Who and what exists|Held|Established"
r"|No longer true — do not treat these as established|Between them|Still open)"
r":[ \t*_]*\n[ \t]+\S",
re.MULTILINE,
)
PROTOCOL_LEAK_EVENTS = '"events"'
def _leaks_protocol(text: str) -> bool:
return bool(PROTOCOL_LEAK_HEADING_RE.search(text)) or PROTOCOL_LEAK_EVENTS in text
def _protocol_leaks(bundle: dict) -> dict:
"""How many stored AI turns still carry protocol, on every branch."""
ai = [a for a in bundle.get("actions") or [] if a.get("type") == "ai"]
leaking = [a for a in ai
if any(marker in (a.get("text") or "") for marker in PROTOCOL_LEAK_MARKERS)]
leaking = [a for a in ai if _leaks_protocol(a.get("text") or "")]
return {"ai_actions": len(ai), "leaking": len(leaking),
"example_ids": [a.get("id") for a in leaking[:10]]}