Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v1.1: harden context window and narrator protocol boundary

Browse Source 
WP-A1 and WP-A2, implemented in sequence, plus the corrective work the owner
asked for at review. Reported in
planning/reports/v1.1/V1.1-WP-A1-A2-REPORT.md (corrective addendum §R).
Planning package v4.2.

WP-A1: context-window safety reserve
- The prompt leaves max(256, ceil(5% of the effective window)) tokens free
  beside the reply. That is 256 at 4,096 and 820 at 16,384. The value is fixed,
  not a setting, and not calibrated per model.
- M6's 64-token margin is gone. Separators and the chat hint are priced
  exactly; tokenizer drift is the reserve's job.
- Protected context that cannot fit raises ContextOverflow before the model
  is called.
- Streams set stream_options.include_usage. Measured on Ollama 0.33, a stream
  sent no usage without it.
- Each sent turn records fits, exceeded, truncation_suspected or unknown.
  The status is returned on the done event, logged when bad, and shown in the
  context inspector. The turn is always kept.
- Accounting is per-attempt data (attempts.ATTEMPT_KEYS).
- Corrective: a cold model is loaded before its turn is built. When the
  window is unverified but the server answered, contextwindow.ensure_window
  makes one bounded POST /api/generate naming only the model. It sends no
  prompt, generates nothing and writes nothing. It then probes again, and the
  turn is built to that answer. If the load fails, or the window is still
  unknown, the turn falls back to the old behaviour.
- Real host, 4,096 window:
  - v1 cold turn: sent 13,875, the server read 2,050.
  - Same turn after the correction: the window was verified, 3,082 sent,
    3,097 read, fits, 499 tokens left beside the reply.
  - Verified turns elsewhere left 275-2,297 tokens against v1's 23-42.

WP-A2: protocol echo and genre-neutral state prompting
- The vocabulary is shown as the JSON object the model sends, not as
  name(field, ...). This costs 121 tokens.
- The example uses character-1, item-1 and location-1.
- The extractor removes shapes anchored to application-owned text:
  - a vocabulary call line;
  - an echoed length hint;
  - the renderer's scene line left last;
  - an empty fence opener.
- Corrective R5: the echoed continue hint is recognised by its own sentence
  ("Output only story text"). A Hard-limit-opened bracket is removed only
  directly above an echo already cut from the same reply.
- Replay of all 518 real v1 replies: 9 changed, 0 flagged, and no story prose
  removed. That is unchanged by R5.
- Replay of 64 v1.1 replies: 3 changed, 0 flagged. The depth-16 instruction
  tail is removed.
- Identity diagnostic after the correction:
  - 0 identity signals;
  - 0 prompt example identifiers proposed;
  - 0/10 stored turns with protocol or instruction shapes.
- 50-turn run: 51 accepted, 0 of 54 stored turns carry protocol.
- SPECS, render.py and validate.py are identical to v1.0.0.

Compatibility: a real v1.0.0 database reads identically on v1.0.0 and v1.1,
field for field, with schema and user_version 94 unchanged. Undo, redo, Save
Point restore, export and import all work on it. There is no schema,
migration or bundle-format change.

Verification: the backend suite passes 1,534 with 17 skipped and 0 failed.
The frontend passes 165/165, and lint and the build are clean. The offline
container and the browser regression were re-run on this tree (see §R.3).

One test was re-calibrated, not weakened: test_history_block_trim's prefix
test had assumed which turn holds the floor at a 2,048 budget.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VvegagkhuCZoFPdv4M1egY
This commit is contained in:
JesseMarkowitz
2026-09-14 16:35:05 -04:00
co-authored by Claude Opus 5
parent ac465ed867
commit d63804f22e
26 changed files with 3903 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/narrative/extract.py
+188 -14
View File
@@ -37,17 +37,17 @@ EMIT_RULE = (
"appeared, record it.\n"
"\n"
"Every value is ABSOLUTE — the new state of things, never a change or a "
"difference. Use only these events:\n"
"difference. Use only these events, in exactly this shape:\n"
f"{events.vocabulary_for_prompt()}\n"
"\n"
"Identifiers are short lower-case slugs (mara, silver-key, old-abbey) and must "
"match the ones already in the state you were shown. Introduce a person, place "
"or thing with create_entity before referring to it. If the turn established "
"nothing, send an empty events list.\n"
"Identifiers are short lower-case slugs and must match the ones already in the "
"state you were shown; the example's identifiers are placeholders. Introduce a "
"person, place or thing with create_entity before referring to it. If the turn "
"established nothing, send an empty events list.\n"
"Example:\n"
'```state\n'
'{"events": [{"type": "set_possession", "item": "silver-key", "owner": "aldric"},'
' {"type": "set_current_location", "entity": "aldric", "location": "old-abbey"}]}\n'
'{"events": [{"type": "set_possession", "item": "item-1", "owner": "character-1"},'
' {"type": "set_current_location", "entity": "character-1", "location": "location-1"}]}\n'
'```'
)
@@ -58,6 +58,47 @@ EMIT_REMINDER = (
"nothing changed.]"
)
# v1.1 WP-A2: the length hint's own words, named once. `builder.length_hint`
# builds the hint from these, and the extractor recognises an echo of it by
# them, so the two cannot drift apart.
LENGTH_HINT_OPENING = "[Hard limit:"
LENGTH_HINT_TAIL = "Finish the narration and append the state block well inside the limit."
#: The application's wording inside a hint. A 3B narrator reworded the front
#: ("your next turn") and the end ("This story ends here."), and kept one or the
#: other of these every time.
_LENGTH_HINT_PHRASE_RE = re.compile(
r"append the state block|turn must not exceed \d+ words", re.IGNORECASE
)
#: v1.1 WP-A2: the rules that remove protocol a narrator copied, named so the
#: replay tool and the report can say which removed what.
RULE_EVENT_CALL = "event_call_line"
RULE_LENGTH_HINT = "echoed_length_hint"
RULE_SCENE_LINE = "rendered_scene_line"
RULE_EMPTY_FENCE = "empty_dangling_fence"
RULE_INSTRUCTION_TAIL = "echoed_instruction_tail"
#: v1.1 WP-A2 corrective (R5). The sentence `CHAT_CONTINUE_HINT` in
#: `providers/openai_compatible.py` carries, which a narrator echoed with the rest
#: of the hint reworded around it. Kept as a copy rather than an import, so the
#: narrative package does not depend on the provider; a test pins that the
#: hint still contains it.
CONTINUE_HINT_PHRASE = "Output only story text"
# R1. A whole line opening with a call to an event this protocol has. The names
# come from the vocabulary, so a call-shaped line naming anything else — a
# character's `open_door(north)` — is not matched.
_EVENT_CALL_LINE_RE = re.compile(
r"^[ \t]*(?:>[ \t]*)?(?:"
+ "|".join(re.escape(name) for name in events.SPECS)
+ r")[ \t]*\(",
re.IGNORECASE,
)
# R3. The renderer's scene line carries its location this way.
_RENDERED_SCENE_LOCATION_RE = re.compile(r"\(at [^()\n]+\)\s*$")
# R4. An opener with nothing after it.
_EMPTY_FENCE_LINE_RE = re.compile(r"```(?:json)?[ \t]*", re.IGNORECASE)
# Three patterns, and the difference between them is the whole of this module's
# safety. A story is allowed to contain code, and taking a code block out of
# someone's prose is a worse failure than leaving a stray proposal in it.
@@ -128,11 +169,42 @@ def _is_echoed_instruction(inner: str) -> bool:
# opening words only, because the echo is often cut off before it ends.
if low.lstrip().startswith("continue the story directly"):
return True
# v1.1 WP-A2 corrective (R5): the same hint, reworded at the front. The M11
# closeout-era identity re-run stored "[You don't need to continue; … Continue
# the story here, directly. Output only story text.]" as the last line of a
# reply, and because nothing recognised it, nothing above it was trailing.
if CONTINUE_HINT_PHRASE.lower() in low:
return True
# v1.1 WP-A2 (R2): the length hint, which names "state block" but not
# "events list", so it passed every check above.
if _is_length_hint(inner):
return True
# The reminder names both; prose about the protocol rarely names either the
# way the instruction does, and effectively never both.
return "state block" in low and "events list" in low
def _opens_like_length_hint(inner: str) -> bool:
"""R5. The bracket opens with the length hint's own `Hard limit:`, whatever follows.
Never enough on its own: an in-world "[Hard limit: forty days]" opens the same
way. `_clean` takes it only directly above an echoed instruction it has already
removed from the end of the same reply.
"""
return inner.lstrip().lower().startswith(LENGTH_HINT_OPENING[1:].lower())
def _is_length_hint(inner: str) -> bool:
"""Whether a bracket's contents are `builder.length_hint`, however reworded.
It must open the way the hint opens *and* carry the hint's own wording. An
in-world "Hard limit: forty days" has the opening and none of the wording.
"""
opening = LENGTH_HINT_OPENING[1:].lower()
return (inner.lstrip().lower().startswith(opening)
and bool(_LENGTH_HINT_PHRASE_RE.search(inner)))
# A heading the model writes above a block it did not fence: `State`, sometimes
# as `State:`, `**State**` or `### State`. It is removed only in two places:
# directly above a proposal that is removed, and as the last line of the reply.
@@ -145,7 +217,7 @@ _LINE_OBJECT_RE = re.compile(r"^[ \t]*(?:>[ \t]*)?\{", re.MULTILINE)
_QUOTE_PREFIX_RE = re.compile(r"^[ \t]*>[ \t]?")
def _clean(prose: str) -> str:
def _clean(prose: str, *, after_block: bool = False) -> str:
"""Removes protocol the block extraction could not, and nothing else.
Found by the M5 realistic-context run (§12), which is the failure class
@@ -160,9 +232,23 @@ def _clean(prose: str) -> str:
story after it. Stored text is replayed as history, so every leak also
showed the next prompt a second, older account of the state, which is what
M5 review Finding 4 removed from replayed history.
v1.1 WP-A2 added four shapes, from the M11 closeout's identity run and the
v1 corpus, each anchored to something the application owns rather than to
what prose looks like: a line opening with a vocabulary call (R1), the
length hint echoed at the end (R2), the renderer's scene line left last
(R3), and an empty fence opener left last (R4). `after_block` says a
proposal block was already taken out of this reply, which is what lets R3
remove a bare scene line that sat above it.
"""
cleaned, _found = _inline_proposals(prose)
cleaned, calls_removed = _strip_event_call_lines(prose)
cleaned, _found = _inline_proposals(cleaned)
cleaned = _strip_echoed_state(cleaned)
protocol_cut = after_block or calls_removed
# R5: set once an echoed instruction bracket has come off the end. Only then
# may a bracket that merely opens the way the length hint opens be taken as
# part of the same echoed tail.
instruction_cut = False
# The end of the reply is cut until nothing more comes off, because one kind
# of leftover can hide another. In a real reply, a `State` heading sat above
# a block the model never finished, and a parroted reminder sat above an
@@ -171,7 +257,12 @@ def _clean(prose: str) -> str:
before = cleaned
for pattern in (_TRAILING_BRACKET_RE, _UNCLOSED_BRACKET_RE):
bracket = pattern.search(cleaned)
if bracket is not None and _is_echoed_instruction(bracket.group(1)):
if bracket is None:
continue
if _is_echoed_instruction(bracket.group(1)):
cleaned = cleaned[: bracket.start()]
instruction_cut = True
elif instruction_cut and _opens_like_length_hint(bracket.group(1)):
cleaned = cleaned[: bracket.start()]
cleaned = _DANGLING_STATE_RE.sub("", cleaned)
dangling = _DANGLING_JSON_RE.search(cleaned)
@@ -182,10 +273,93 @@ def _clean(prose: str) -> str:
cleaned = _strip_trailing_state_heading(cleaned).rstrip()
# A bare quote marker, the start of a quoted block that never came.
cleaned = re.sub(r"\n[ \t]*>[ \t]*\Z", "", cleaned)
cleaned = _strip_empty_dangling_fence(cleaned)
if cleaned.rstrip() != before.rstrip():
protocol_cut = True
cleaned = _strip_trailing_scene_line(cleaned, protocol_cut)
if cleaned == before:
return cleaned.strip()
def _strip_event_call_lines(text: str) -> tuple[str, bool]:
"""R1. Removes whole lines that open with a call to a vocabulary event.
A line inside a fenced code block is the story's own code and is never
examined. Returns the text and whether anything was removed.
"""
kept: list[str] = []
in_fence = False
removed = False
for line in text.split("\n"):
if line.lstrip().startswith("```"):
in_fence = not in_fence
kept.append(line)
continue
if not in_fence and _EVENT_CALL_LINE_RE.match(line):
removed = True
continue
kept.append(line)
if not removed:
return text, False
return re.sub(r"\n{3,}", "\n\n", "\n".join(kept)), True
def _strip_empty_dangling_fence(text: str) -> str:
"""R4. A ```` ```json ```` or ```` ``` ```` opener as the last line, with nothing after it.
Only an *opener*: the fence lines are counted, and an even count means the
last one closes a story's own code block, which stays.
"""
lines = text.rstrip().split("\n")
if len(lines) < 2 or not _EMPTY_FENCE_LINE_RE.fullmatch(lines[-1].strip()):
return text
fences = sum(1 for line in lines if line.lstrip().startswith("```"))
if fences % 2 == 0:
return text
return "\n".join(lines[:-1]).rstrip()
def _strip_trailing_scene_line(text: str, protocol_cut: bool) -> str:
"""R3. The renderer's scene line, left as the last line of the reply.
Taken when it carries the renderer's own `(at <location>)`, or when protocol
was already cut from this reply, which makes a bare scene line part of the
same pasted tail. A final screenplay-style "Scene: …" line in a reply with
no protocol in it stays, and so does any scene line with story after it.
"""
lines = text.rstrip().split("\n")
if len(lines) < 2:
return text
last = lines[-1].strip()
if not last.startswith(render.HEADING_SCENE + " "):
return text
if not (_RENDERED_SCENE_LOCATION_RE.search(last) or protocol_cut):
return text
return "\n".join(lines[:-1]).rstrip()
def explain_removed_line(line: str) -> str | None:
"""Which v1.1 rule removes a line of this shape, for the replay report.
None means no v1.1 rule explains it, which the replay treats as a failure.
"""
stripped = line.strip()
if _EVENT_CALL_LINE_RE.match(line):
return RULE_EVENT_CALL
if stripped.startswith("["):
inner = stripped[1:]
inner = inner[:-1] if inner.endswith("]") else inner
if _is_length_hint(inner):
return RULE_LENGTH_HINT
if _is_echoed_instruction(inner) or _opens_like_length_hint(inner):
return RULE_INSTRUCTION_TAIL
if stripped.startswith(render.HEADING_SCENE + " "):
return RULE_SCENE_LINE
if _EMPTY_FENCE_LINE_RE.fullmatch(stripped):
return RULE_EMPTY_FENCE
return None
def _is_state_heading(line: str) -> bool:
return bool(_STATE_HEADING_RE.match(line))
@@ -451,7 +625,7 @@ def split(text: str) -> tuple[str, dict | None, str]:
if matches:
match = matches[-1]
raw = match.group(1).strip()
prose = _clean(text[: match.start()] + text[match.end():])
prose = _clean(text[: match.start()] + text[match.end():], after_block=True)
return prose, _tolerant_load(raw), raw
# A `json` or unlabelled fence is ours only when its contents are this
@@ -465,7 +639,7 @@ def split(text: str) -> tuple[str, dict | None, str]:
raw = match.group(1).strip()
parsed = _tolerant_load(raw)
if _looks_like_proposal(parsed) or _reads_as_protocol(raw):
prose = _clean(text[: match.start()] + text[match.end():])
prose = _clean(text[: match.start()] + text[match.end():], after_block=True)
return prose, parsed, raw
match = _TRAILING_RE.search(text)
@@ -473,7 +647,7 @@ def split(text: str) -> tuple[str, dict | None, str]:
raw = match.group(1)
parsed = _tolerant_load(raw)
if _looks_like_proposal(parsed):
return _clean(text[: match.start()]), parsed, raw
return _clean(text[: match.start()], after_block=True), parsed, raw
# An unfenced proposal on its own lines but not at the end: quoted, or
# followed by more story. The last one is the turn's proposal, as with
@@ -481,7 +655,7 @@ def split(text: str) -> tuple[str, dict | None, str]:
without, found = _inline_proposals(text)
if found:
parsed, raw = found[-1]
return _clean(without), parsed, raw
return _clean(without, after_block=True), parsed, raw
# No block at all — but the reply may still carry protocol the model wrote
# as prose, or a fence it never closed.