Keep the state section and its proposal out of the story
The first complete M01 run with the memory bank on (f8d4010, 101 turns on
a GPU host) reported "complete". It still did not prove M04. The planted
clue was found at turn 100 only because the narrator had pasted the
narrative-state section into its own prose, and the paste was still in
recent history. No memory and no summary carried the clue.
The narrator is a small local model. It wrote protocol into its stored
narration on 42 of 104 turns, starting at depth 2, in four shapes:
- a copy of the state section: `Scene:`, `Who and what exists:`, `Held:`,
`Established:`, `Still open:`
- that copy above a correct ```state block, which was stripped while the
copy stayed
- the copy, then a bare `State` heading, then a `> {"events": ...}`
proposal quoted like a player turn, sometimes with story after it
- the same block cut off by the output-token limit, on 10 turns
Stored text is replayed verbatim as history, so each leak also put a second,
older account of the state into the next prompt. That is what M5 review
Finding 4 removed from history replay, and every leak gave the model another
example to copy.
The extractor now removes:
- a pasted state section, recognised by at least two of the renderer's own
headings as whole lines. The headings are constants in `render.py`, so the
renderer and the extractor cannot drift apart. One heading alone, or a
`Scene:` line of prose, is left.
- an unfenced proposal that starts a line, quoted or not, when it parses and
is a proposal. With no fence it becomes the turn's proposal. A `State`
heading directly above goes with it. Candidates are taken outermost first,
so a finished event line inside an unfinished block is never taken as a
proposal by itself.
- an unfinished unfenced proposal at the end that reads as protocol.
- whatever is left at the end: a `State` heading, a bare `>`, a parroted
reminder or continue hint (closed or not), and a ```json fence cut off
before it names its events. These are cut repeatedly until nothing more
comes off.
This also fixes an older bug. `_STATE_FENCE_RE` read "a ```state block"
inside a parroted reminder as a fence opening and cut out the middle of the
reminder. The label must now end its line or run straight into the payload.
A reply whose only removal is a pasted state section records no raw block,
so the turn is not marked unparseable for a block it never started.
Every AI turn in four real runs was replayed through the new extractor:
draco M01, the two 26-turn GPU trials, and this M01 run. 339 turns in all.
No turn the old extractor had left clean changed. Every leak of our own
protocol is gone: 42 of 42 in this M01 run, 5 in trial 2, 3 on draco.
Trial 1 still has model-invented headings ("Identifiers established:",
"Set of events made true:") on 10 turns. They paraphrase the instruction and
are not our renderer's text, so they are left, not guessed at.
The long-run harness now records an explicit M04 verdict, which is never a
recovery while the clue is still in recent history. It also counts the AI
turns in the export that still carry protocol.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136VBTMUKWYeU6G9HgbDbND
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 5
Claude Opus 5
parent
f8d401029f
commit
0c7316f951
@@ -269,3 +269,32 @@ def test_a_run_with_no_summary_or_no_memory_is_not_complete(run_for, tmp_path):
|
||||
def _with_adv(run):
|
||||
run.adv = 1
|
||||
return run
|
||||
|
||||
|
||||
# ------------------------------------------------------- what M04 actually proved
|
||||
|
||||
def test_the_m04_verdict_never_credits_a_clue_still_in_recent_history():
|
||||
"""The first long run with the bank on found the clue at turn 100 only
|
||||
because the narrator had pasted the state into recent history."""
|
||||
base = {"clue_in_recent_history_window": False, "in_memories_section": False,
|
||||
"in_summary_section": False, "in_state_section": False}
|
||||
assert lr._m04_verdict({**base, "clue_in_recent_history_window": True,
|
||||
"in_memories_section": True}) == "precondition_not_met"
|
||||
assert lr._m04_verdict({**base, "in_memories_section": True}) == \
|
||||
"recovered_through_memory_or_summary"
|
||||
assert lr._m04_verdict({**base, "in_summary_section": True}) == \
|
||||
"recovered_through_memory_or_summary"
|
||||
assert lr._m04_verdict({**base, "in_state_section": True}) == \
|
||||
"recovered_through_state_only"
|
||||
assert lr._m04_verdict(base) == "not_recovered"
|
||||
|
||||
|
||||
def test_protocol_left_in_stored_narration_is_counted():
|
||||
bundle = {"actions": [
|
||||
{"id": 1, "type": "do", "text": '> You say {"events": []}'},
|
||||
{"id": 2, "type": "ai", "text": "The rain eases."},
|
||||
{"id": 3, "type": "ai", "text": "Beat.\n\nWho and what exists:\n mara: Mara"},
|
||||
{"id": 4, "type": "ai", "text": 'Beat.\n\n> {"events": []}'},
|
||||
]}
|
||||
assert lr._protocol_leaks(bundle) == {
|
||||
"ai_actions": 3, "leaking": 2, "example_ids": [3, 4]}
|
||||
|
||||
@@ -30,7 +30,7 @@ from app.database import Base, SessionLocal, engine, get_db
|
||||
from app.main import app
|
||||
from app.narrative import apply as napply
|
||||
from app.narrative import events as nevents
|
||||
from app.narrative import extract, model, store, validate
|
||||
from app.narrative import extract, model, render, store, validate
|
||||
from app.routers import adventures
|
||||
|
||||
from fakes import ScriptedProvider, state_block
|
||||
@@ -1558,3 +1558,236 @@ def test_ordinary_prose_is_never_trimmed(reply):
|
||||
story to satisfy a regex is a worse failure than leaving a stray bracket."""
|
||||
prose, _parsed, _raw = extract.split(reply)
|
||||
assert prose == reply
|
||||
|
||||
|
||||
# ============================ M11: the state section, pasted into the narration
|
||||
#
|
||||
# Found by the first M01 long run with the memory bank on. A small local model
|
||||
# pasted the narrative-state section into its prose on 42 of 104 turns, and
|
||||
# wrote its proposal unfenced under a bare `State` heading, sometimes quoted and
|
||||
# sometimes with more story after it. Stored text is replayed as history, so
|
||||
# each leak also put a second, older account of the state into the next prompt.
|
||||
# The replies below are cut down from that run's raw output, not imagined.
|
||||
|
||||
PASTED_STATE = (
|
||||
"Scene: Aldric, Mara, and Edrin in The Crooked Lantern, the silver key in "
|
||||
"Mara’s possession. (at The Crooked Lantern)\n"
|
||||
"\n"
|
||||
"Who and what exists:\n"
|
||||
" aldric: Aldric (character)\n"
|
||||
" mara: Mara (character)\n"
|
||||
" silver_key: the silver key (item)\n"
|
||||
"\n"
|
||||
"Held:\n"
|
||||
" the silver key — Mara\n"
|
||||
"\n"
|
||||
"Established:\n"
|
||||
" Aldric knows The Old Abbey the silver key opens the crypt beneath the Old "
|
||||
"Abbey (SILVER-KEY-CRYPT-OLD-ABBEY) [corrected by the player]"
|
||||
)
|
||||
|
||||
|
||||
def test_a_pasted_state_section_and_an_unfenced_trailing_block_leave_the_prose():
|
||||
reply = (
|
||||
"Edrin looks at them both. “Do you trust me, Mara?”\n\n"
|
||||
"> Aldric steps forward, placing the silver key in Mara’s hand. “I do.”\n\n"
|
||||
+ PASTED_STATE + "\n\nState\n"
|
||||
'{\n "events": [\n {"type": "set_possession", "item": "silver_key", '
|
||||
'"owner": "mara"}\n ]\n}'
|
||||
)
|
||||
prose, parsed, _raw = extract.split(reply)
|
||||
assert prose == (
|
||||
"Edrin looks at them both. “Do you trust me, Mara?”\n\n"
|
||||
"> Aldric steps forward, placing the silver key in Mara’s hand. “I do.”"
|
||||
)
|
||||
assert parsed["events"][0]["owner"] == "mara"
|
||||
|
||||
|
||||
def test_a_quoted_block_in_the_middle_of_the_story_is_taken_and_removed():
|
||||
reply = (
|
||||
"Aldric steps forward. “We need to be cautious,” he says.\n\n"
|
||||
+ PASTED_STATE + "\n\nState\n\n"
|
||||
'> {"events": [{"type": "set_current_location", "entity": "aldric", '
|
||||
'"location": "ridge_track"}]}\n\n'
|
||||
"The forest ahead was dense, and the road lower than expected."
|
||||
)
|
||||
prose, parsed, raw = extract.split(reply)
|
||||
assert prose == (
|
||||
"Aldric steps forward. “We need to be cautious,” he says.\n\n"
|
||||
"The forest ahead was dense, and the road lower than expected."
|
||||
)
|
||||
assert parsed["events"][0]["location"] == "ridge_track"
|
||||
assert raw.startswith('{"events"'), "the proposal is kept for the audit"
|
||||
|
||||
|
||||
def test_a_pasted_state_section_above_a_proper_block_is_removed_too():
|
||||
reply = "The rain eases.\n\n" + PASTED_STATE + '\n\n```state\n{"events": []}\n```'
|
||||
prose, parsed, _raw = extract.split(reply)
|
||||
assert prose == "The rain eases."
|
||||
assert parsed == {"events": []}
|
||||
|
||||
|
||||
def test_a_multi_line_quoted_block_is_read_without_its_markers():
|
||||
reply = (
|
||||
'Beat.\n\n> {\n> "events": [\n> {"type": "add_fact", '
|
||||
'"predicate": "the door opened"}\n> ]\n> }\n\nAfter.'
|
||||
)
|
||||
prose, parsed, _raw = extract.split(reply)
|
||||
assert prose == "Beat.\n\nAfter."
|
||||
assert parsed["events"][0]["predicate"] == "the door opened"
|
||||
|
||||
|
||||
def test_every_section_the_renderer_writes_is_recognised_when_pasted():
|
||||
"""The extractor finds a paste by the renderer's own headings. A section
|
||||
added to `render.for_prompt` without a constant would leak again unnoticed,
|
||||
so this renders every section and pastes the lot."""
|
||||
document = model.empty()
|
||||
document["entities"] = {
|
||||
"mara": {"type": "character", "name": "Mara"},
|
||||
"key": {"type": "item", "name": "the key"},
|
||||
}
|
||||
document["possessions"] = {"key": "mara"}
|
||||
document["facts"] = [
|
||||
{"id": "f1", "subject": "mara", "predicate": "knows",
|
||||
"value": "the door is locked", "status": "active"},
|
||||
{"id": "f2", "subject": "mara", "predicate": "believed",
|
||||
"value": "the door was open", "status": "invalidated"},
|
||||
]
|
||||
document["relationships"] = [
|
||||
{"source": "mara", "type": "guards", "target": "key", "status": "active"}]
|
||||
document["threads"] = {"door": {"title": "Who locked the door", "status": "open"}}
|
||||
document["scene"] = {"summary": "Mara at the door.", "location": None,
|
||||
"present": ["mara"]}
|
||||
|
||||
rendered = render.for_prompt(document)
|
||||
lines = rendered.split("\n")
|
||||
for heading in render.SECTION_HEADINGS:
|
||||
assert heading in lines, f"{heading!r} is not a line of the rendered section"
|
||||
|
||||
prose, _parsed, _raw = extract.split("Mara listens.\n\n" + rendered)
|
||||
assert prose == "Mara listens."
|
||||
|
||||
|
||||
def test_a_quoted_block_cut_off_by_the_token_limit_is_not_story():
|
||||
"""10 of 104 turns in the long run ended inside a quoted object."""
|
||||
reply = (
|
||||
"The sky above is a canvas of gray, the rain relentless.\n\n"
|
||||
+ PASTED_STATE + "\n\nState\n\n"
|
||||
'> {"events": [{"type": "set_entity_status", "entity":'
|
||||
)
|
||||
prose, parsed, raw = extract.split(reply)
|
||||
assert prose == "The sky above is a canvas of gray, the rain relentless."
|
||||
assert parsed is None
|
||||
assert raw, "the unfinished block is kept for the audit"
|
||||
|
||||
|
||||
def test_a_block_missing_only_its_last_brace_is_not_story():
|
||||
reply = (
|
||||
"Mara’s eyes widen.\n\nState\n\n"
|
||||
'> {"events": [{"type": "add_fact", "predicate": "Mara understands."}]'
|
||||
)
|
||||
prose, _parsed, _raw = extract.split(reply)
|
||||
assert prose == "Mara’s eyes widen."
|
||||
|
||||
|
||||
def test_an_unfinished_block_is_cut_at_its_outer_brace_not_an_inner_one():
|
||||
"""The finished event objects close; the block around them never does.
|
||||
Cutting at the last line that opens an object left the list in the story."""
|
||||
reply = (
|
||||
"They brace themselves for what they must face.\n\nState\n{\n"
|
||||
' "events": [\n'
|
||||
' { "type": "set_current_location", "entity": "aldric", "location": "docks" },\n'
|
||||
' { "type": "set_entity_attribute", "entity": "aldric", "attribute": "mood", '
|
||||
'"value": "tense"\n'
|
||||
" ]\n}"
|
||||
)
|
||||
prose, parsed, _raw = extract.split(reply)
|
||||
assert prose == "They brace themselves for what they must face."
|
||||
assert parsed is None
|
||||
|
||||
|
||||
def test_a_finished_event_inside_an_unfinished_block_is_not_taken_on_its_own():
|
||||
reply = (
|
||||
'Beat.\n\nState\n{\n "events": [\n'
|
||||
' {"type": "add_fact", "predicate": "the door opened"}\n'
|
||||
)
|
||||
prose, parsed, _raw = extract.split(reply)
|
||||
assert prose == "Beat."
|
||||
assert parsed is None, "one event line was taken as the whole proposal"
|
||||
|
||||
|
||||
def test_a_bare_quote_marker_left_at_the_end_is_not_story():
|
||||
reply = (
|
||||
"They are ready.\n\n"
|
||||
"[Reminder: end your reply with a ```state block listing the events your "
|
||||
"narration made true, with absolute values. Send an empty events list if "
|
||||
"nothing changed.]\n\n>"
|
||||
)
|
||||
prose, _parsed, _raw = extract.split(reply)
|
||||
assert prose == "They are ready."
|
||||
|
||||
|
||||
def test_a_parroted_reminder_above_an_unfinished_json_fence_is_all_removed():
|
||||
"""The reminder names "a ```state block". That phrase once read as a fence
|
||||
opening, and the middle of the reminder was cut out while the rest of it
|
||||
and the unfinished JSON stayed in the story."""
|
||||
reply = (
|
||||
"He turned back towards the town instead.\n\n"
|
||||
"[Reminder: end your reply with a ```state block listing the events your "
|
||||
"narration made true, with absolute values. Send an empty events list if "
|
||||
"nothing changed.]\n\n"
|
||||
'```json\n{\n "events": [\n {'
|
||||
)
|
||||
prose, parsed, _raw = extract.split(reply)
|
||||
assert prose == "He turned back towards the town instead."
|
||||
assert parsed is None
|
||||
|
||||
|
||||
def test_a_fence_cut_off_before_it_names_its_events_is_not_story():
|
||||
reply = (
|
||||
"The silver key was his only guide.\n\n"
|
||||
"[Reminder: end your reply with a ```state block listing the events your "
|
||||
"narration made true, with absolute values. Send an empty events list if "
|
||||
"nothing changed.]\n\n"
|
||||
'```json\n{\n "'
|
||||
)
|
||||
prose, _parsed, _raw = extract.split(reply)
|
||||
assert prose == "The silver key was his only guide."
|
||||
|
||||
|
||||
def test_a_parroted_continue_hint_cut_off_mid_sentence_is_not_story():
|
||||
reply = (
|
||||
"Aldric's steps were firm.\n\n"
|
||||
"[Reminder: end your reply with a ```state block listing the events your "
|
||||
"narration made true, with absolute values. Send an empty events list if "
|
||||
"nothing changed.]\n\n"
|
||||
"[Continue the story directly."
|
||||
)
|
||||
prose, _parsed, _raw = extract.split(reply)
|
||||
assert prose == "Aldric's steps were firm."
|
||||
|
||||
|
||||
def test_a_pasted_state_section_with_no_block_records_no_block():
|
||||
"""A paste is not a proposal, so the turn is not marked unparseable."""
|
||||
prose, parsed, raw = extract.split("The rain eases.\n\n" + PASTED_STATE)
|
||||
assert prose == "The rain eases."
|
||||
assert parsed is None
|
||||
assert raw == ""
|
||||
|
||||
|
||||
@pytest.mark.parametrize("reply", [
|
||||
"The notice on the door read:\n\nHeld:\n nothing, by order of the Watch.",
|
||||
"He chalked the first mark of a sum on the wall:\n{",
|
||||
"The sign said only [continue at your own risk",
|
||||
'She typed it out:\n```json\n{"name":',
|
||||
"Scene: the docks at dawn.\n\nThe gulls cried over the water.",
|
||||
'She typed:\n> {"name": "Mara"}\nand pressed enter.',
|
||||
"Aldric read the old charter.\n\nState\n\nof the realm, it began, is fragile.",
|
||||
"> You open the door.\n\nThe hinges complain.",
|
||||
])
|
||||
def test_story_that_only_resembles_the_protocol_is_kept(reply):
|
||||
"""One heading, a scene line on its own, JSON that is not a proposal, and a
|
||||
`State` line with story after it are all somebody's story."""
|
||||
prose, parsed, _raw = extract.split(reply)
|
||||
assert prose == reply
|
||||
assert parsed is None
|
||||
|
||||
Reference in New Issue
Block a user