Keep the state section and its proposal out of the story
The first complete M01 run with the memory bank on (f8d4010, 101 turns on
a GPU host) reported "complete". It still did not prove M04. The planted
clue was found at turn 100 only because the narrator had pasted the
narrative-state section into its own prose, and the paste was still in
recent history. No memory and no summary carried the clue.
The narrator is a small local model. It wrote protocol into its stored
narration on 42 of 104 turns, starting at depth 2, in four shapes:
- a copy of the state section: `Scene:`, `Who and what exists:`, `Held:`,
`Established:`, `Still open:`
- that copy above a correct ```state block, which was stripped while the
copy stayed
- the copy, then a bare `State` heading, then a `> {"events": ...}`
proposal quoted like a player turn, sometimes with story after it
- the same block cut off by the output-token limit, on 10 turns
Stored text is replayed verbatim as history, so each leak also put a second,
older account of the state into the next prompt. That is what M5 review
Finding 4 removed from history replay, and every leak gave the model another
example to copy.
The extractor now removes:
- a pasted state section, recognised by at least two of the renderer's own
headings as whole lines. The headings are constants in `render.py`, so the
renderer and the extractor cannot drift apart. One heading alone, or a
`Scene:` line of prose, is left.
- an unfenced proposal that starts a line, quoted or not, when it parses and
is a proposal. With no fence it becomes the turn's proposal. A `State`
heading directly above goes with it. Candidates are taken outermost first,
so a finished event line inside an unfinished block is never taken as a
proposal by itself.
- an unfinished unfenced proposal at the end that reads as protocol.
- whatever is left at the end: a `State` heading, a bare `>`, a parroted
reminder or continue hint (closed or not), and a ```json fence cut off
before it names its events. These are cut repeatedly until nothing more
comes off.
This also fixes an older bug. `_STATE_FENCE_RE` read "a ```state block"
inside a parroted reminder as a fence opening and cut out the middle of the
reminder. The label must now end its line or run straight into the payload.
A reply whose only removal is a pasted state section records no raw block,
so the turn is not marked unparseable for a block it never started.
Every AI turn in four real runs was replayed through the new extractor:
draco M01, the two 26-turn GPU trials, and this M01 run. 339 turns in all.
No turn the old extractor had left clean changed. Every leak of our own
protocol is gone: 42 of 42 in this M01 run, 5 in trial 2, 3 on draco.
Trial 1 still has model-invented headings ("Identifiers established:",
"Set of events made true:") on 10 turns. They paraphrase the instruction and
are not our renderer's text, so they are left, not guessed at.
The long-run harness now records an explicit M04 verdict, which is never a
recovery while the clue is still in recent history. It also counts the AI
turns in the export that still carry protocol.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136VBTMUKWYeU6G9HgbDbND
This commit is contained in:
co-authored by
Claude Opus 5
parent
f8d401029f
commit
0c7316f951
@@ -269,3 +269,32 @@ def test_a_run_with_no_summary_or_no_memory_is_not_complete(run_for, tmp_path):
|
||||
def _with_adv(run):
|
||||
run.adv = 1
|
||||
return run
|
||||
|
||||
|
||||
# ------------------------------------------------------- what M04 actually proved
|
||||
|
||||
def test_the_m04_verdict_never_credits_a_clue_still_in_recent_history():
|
||||
"""The first long run with the bank on found the clue at turn 100 only
|
||||
because the narrator had pasted the state into recent history."""
|
||||
base = {"clue_in_recent_history_window": False, "in_memories_section": False,
|
||||
"in_summary_section": False, "in_state_section": False}
|
||||
assert lr._m04_verdict({**base, "clue_in_recent_history_window": True,
|
||||
"in_memories_section": True}) == "precondition_not_met"
|
||||
assert lr._m04_verdict({**base, "in_memories_section": True}) == \
|
||||
"recovered_through_memory_or_summary"
|
||||
assert lr._m04_verdict({**base, "in_summary_section": True}) == \
|
||||
"recovered_through_memory_or_summary"
|
||||
assert lr._m04_verdict({**base, "in_state_section": True}) == \
|
||||
"recovered_through_state_only"
|
||||
assert lr._m04_verdict(base) == "not_recovered"
|
||||
|
||||
|
||||
def test_protocol_left_in_stored_narration_is_counted():
|
||||
bundle = {"actions": [
|
||||
{"id": 1, "type": "do", "text": '> You say {"events": []}'},
|
||||
{"id": 2, "type": "ai", "text": "The rain eases."},
|
||||
{"id": 3, "type": "ai", "text": "Beat.\n\nWho and what exists:\n mara: Mara"},
|
||||
{"id": 4, "type": "ai", "text": 'Beat.\n\n> {"events": []}'},
|
||||
]}
|
||||
assert lr._protocol_leaks(bundle) == {
|
||||
"ai_actions": 3, "leaking": 2, "example_ids": [3, 4]}
|
||||
|
||||
Reference in New Issue
Block a user