Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Measure M04 by where the planted turn is, and catch a section of the model's own

Browse Source 
The M04 re-run on 0c7316f ran 101 turns with no failures, and the verdict
still came out `precondition_not_met`. That verdict was wrong. The planted
player turn (depth 1) was 65 actions outside the history window, whose floor
was 66. The sentinel's text was in recent history for two other reasons.

- On 5 turns the narrator wrote a section of its own, `## Established:` over
  indented facts, with the planted clue copied into it from the state section.
  The extractor passed it: it was a single heading, and the `## ` meant it did
  not match. The harness leak count passed it the same way, and read 1 where
  5 turns leaked.
- On 4 turns the narrator used the sentinel as a name inside ordinary
  sentences ("the SILVER-KEY-CRYPT-OLD-ABBEY, flickers with latent power"). That
  is story text and cannot be stripped.

So the sentinel's text in history can never be the precondition. M04's
written pass is "Fact/event remains recoverable without entire transcript in
prompt" (V1-ACCEPTANCE-TESTS.md). The owner agreed on 2026-09-13 that the
precondition is positional, and that recovery through authoritative state
counts; memory is not required.

Extractor:
- A state-section heading is recognised with any markdown the model wrapped
  it in (`## Established:`, `**Held:**`, `> Held:`).
- One heading with an indented entry under it now qualifies as protocol.
  Before, a block needed two headings, or one heading and the scene line. A
  heading followed by unindented prose is still story. The earlier guard test
  "Held:" with an indented line is now protocol, and the case was rewritten
  unindented.

Harness:
- It records `planted_depth` when the clue is planted, and carries it across
  --resume. No endpoint reports an action's depth, but a fresh campaign's path
  is the opening, the planted turn and its reply, so the depth is
  `total_actions - 2`. That was checked against the database in three runs.
- `_recall` reports `planted_depth`, `history_floor_depth` and
  `planted_turn_in_history_window`. The verdict is `precondition_not_met` only
  when the planted turn is still in the window, and `precondition_unknown`
  when its depth was never recorded. `clue_in_recent_history_window` stays as a
  fact about the prompt.
- The protocol-leak count matches a state heading with an indented entry under
  it, in any markdown.

Every AI turn in five real runs was replayed through the new extractor: draco
M01, the two 26-turn GPU trials, the f8d4010 M01 run and the M04 re-run. 443
turns in all. No turn the old extractor had left clean changed, and the M04
re-run lost 7 more leaks. The sentinel-as-a-name turns are story and remain.
Trial 1's model-invented headings remain, as before.

The M04 re-run's own evidence, reclassified under the new precondition from
its database and its recall-turn snapshot, reads
`recovered_through_state_only`. The original recall.json is kept unchanged
beside `recall-reclassified.json`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136VBTMUKWYeU6G9HgbDbND
This commit is contained in:
JesseMarkowitz
2026-09-13 22:10:33 -04:00
co-authored by Claude Opus 5
parent 0c7316f951
commit 96c1bf5ded
4 changed files with 156 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/tests/test_m11_long_run_memory.py
+31 -7
View File
@@ -273,13 +273,14 @@ def _with_adv(run):
# ------------------------------------------------------- what M04 actually proved
def test_the_m04_verdict_never_credits_a_clue_still_in_recent_history():
"""The first long run with the bank on found the clue at turn 100 only
because the narrator had pasted the state into recent history."""
base = {"clue_in_recent_history_window": False, "in_memories_section": False,
"in_summary_section": False, "in_state_section": False}
assert lr._m04_verdict({**base, "clue_in_recent_history_window": True,
def test_the_m04_verdict_never_credits_a_planted_turn_still_in_the_window():
base = {"planted_turn_in_history_window": False, "in_memories_section": False,
"in_summary_section": False, "in_state_section": False,
"clue_in_recent_history_window": False}
assert lr._m04_verdict({**base, "planted_turn_in_history_window": True,
"in_memories_section": True}) == "precondition_not_met"
assert lr._m04_verdict({**base, "planted_turn_in_history_window": None,
"in_state_section": True}) == "precondition_unknown"
assert lr._m04_verdict({**base, "in_memories_section": True}) == \
"recovered_through_memory_or_summary"
assert lr._m04_verdict({**base, "in_summary_section": True}) == \
@@ -289,12 +290,35 @@ def test_the_m04_verdict_never_credits_a_clue_still_in_recent_history():
assert lr._m04_verdict(base) == "not_recovered"
def test_the_sentinel_in_recent_history_does_not_decide_the_precondition():
"""The M04 re-run: the narrator reused the sentinel in its own prose while
the planted turn was 65 actions outside the window."""
recall = {"planted_turn_in_history_window": False,
"clue_in_recent_history_window": True,
"in_memories_section": False, "in_summary_section": False,
"in_state_section": True}
assert lr._m04_verdict(recall) == "recovered_through_state_only"
def test_the_planted_depth_survives_a_resume(run_for, tmp_path):
first = run_for(Reports(tmp_path / "server.log"))
first.adv, first.planted_depth = 1, 1
first.save_resume()
second = run_for(Reports(tmp_path / "server.log"))
second.adopt(json.loads((tmp_path / lr.RESUME_FILE).read_text()))
assert second.planted_depth == 1
def test_protocol_left_in_stored_narration_is_counted():
bundle = {"actions": [
{"id": 1, "type": "do", "text": '> You say {"events": []}'},
{"id": 2, "type": "ai", "text": "The rain eases."},
{"id": 3, "type": "ai", "text": "Beat.\n\nWho and what exists:\n mara: Mara"},
{"id": 4, "type": "ai", "text": 'Beat.\n\n> {"events": []}'},
{"id": 5, "type": "ai",
"text": "Rain.\n\n## Established:\n the crypt is sealed (SENTINEL)"},
{"id": 6, "type": "ai", "text": "The notice read:\n\nHeld:\nnothing at all."},
]}
assert lr._protocol_leaks(bundle) == {
"ai_actions": 3, "leaking": 2, "example_ids": [3, 4]}
"ai_actions": 5, "leaking": 3, "example_ids": [3, 4, 5]}
backend/tests/test_narrative_state.py
+34 -1
View File
@@ -1767,6 +1767,39 @@ def test_a_parroted_continue_hint_cut_off_mid_sentence_is_not_story():
assert prose == "Aldric's steps were firm."
def test_a_section_of_its_own_under_a_markdown_heading_is_not_story():
"""The M04 re-run: the narrator wrote its own `## Established:` with the
planted clue copied into it, which kept the clue in recent history on a
turn the extractor had passed as clean."""
reply = (
"The rain outside seems to echo their uncertainty.\n\n"
"## Established:\n"
" the silver key has an enchantment that unlocks the sealed crypt of the Old Abbey\n"
" The Old Abbey's crypt is sealed (SILVER-KEY-CRYPT-OLD-ABBEY)"
)
prose, _parsed, _raw = extract.split(reply)
assert prose == "The rain outside seems to echo their uncertainty."
assert "SILVER-KEY" not in prose
def test_a_bold_heading_mid_story_goes_and_the_story_either_side_stays():
reply = (
"Mara takes the key.\n\n**Held:**\n the silver key — Mara\n\n"
"They step out into the rain."
)
prose, _parsed, _raw = extract.split(reply)
assert prose == "Mara takes the key.\n\nThey step out into the rain."
def test_decorated_headings_count_toward_a_pasted_section():
reply = (
"Beat.\n\n### Who and what exists:\n mara: Mara (character)\n\n"
"### Held:\n the key — Mara"
)
prose, _parsed, _raw = extract.split(reply)
assert prose == "Beat."
def test_a_pasted_state_section_with_no_block_records_no_block():
"""A paste is not a proposal, so the turn is not marked unparseable."""
prose, parsed, raw = extract.split("The rain eases.\n\n" + PASTED_STATE)
@@ -1776,7 +1809,7 @@ def test_a_pasted_state_section_with_no_block_records_no_block():
@pytest.mark.parametrize("reply", [
"The notice on the door read:\n\nHeld:\n nothing, by order of the Watch.",
"The notice on the door read:\n\nHeld:\nnothing, by order of the Watch.",
"He chalked the first mark of a sum on the wall:\n{",
"The sign said only [continue at your own risk",
'She typed it out:\n```json\n{"name":',