Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

M5: genre-neutral authoritative narrative state, with review corrections

Browse Source 
Replaces AI-DnD's RPG relative-delta world state with the genre-neutral typed
narrative state of ADR 010: explicit, absolute, allowlisted events proposed by
the model, validated by the application, applied to one authoritative document,
and snapshotted per position so restore stays a row read.

This commit includes the corrective pass that followed the independent review
in planning/reports/M5-IMPLEMENTATION-REPORT.md. The invariant it exists to
hold is:

    visible active transcript position == stored head == authoritative state

Narrator editing (D10, STORY-BRANCH-SEMANTICS §§14-15)

  A narrator edit no longer rewrites a row. It returns to the state before the
  turn, takes the reader's exact text as the accepted narration, re-derives the
  state that text implies, and becomes a new active continuation — while the
  original narration keeps its words, its live flag and its whole future as
  retained history. At the tip the correction is another take; with story below
  it, it forks. No new history machinery: this is the existing fork/take/head
  path with the reader's text in place of a generated reply. The §14A refusal
  is therefore gone for narrator turns, and remains only for player input.

Pre-M5 positions

  Migration 88 backfills the empty narrative document onto every action written
  before M5, and a missing snapshot now restores the empty document instead of
  leaving the previous position's state standing. Restoring to an old Save
  Point no longer leaves a later position's entities and facts on screen.

Narrator context

  Replayed history carries prose only; the machine-readable block is no longer
  reconstructed into past turns, where it contradicted the authoritative state
  in the same prompt. A fact withdrawn by a manual correction is now named as
  no longer true, with the reader's reason, rather than silently dropped.

Also

  - state_changes joins the action-list bulk read, removing one query per row.
  - Extraction takes only the application's own protocol payload: an ordinary
    ```json or ```python block in a story survives, and a mangled proposal
    still does not reach the reader.

Planning: ADR 013 records the authoritative document shape; §§14-15/14A, D10,
C04 and BUILD-MILESTONES are updated to describe what exists. Debt is recorded
against M8 (scenario editor UX) and M9 (export of the audit trail).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PWU4gTfLYY6Qq9U7aa9Qw2
This commit is contained in:
JesseMarkowitz
2026-09-05 07:01:50 -04:00
co-authored by Claude Opus 5
parent 62a997f364
commit b7005e6fdd
57 changed files with 7257 additions and 474 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/context/builder.py
+86 -61
View File
@@ -22,7 +22,7 @@ from dataclasses import dataclass
import tiktoken
from .. import models, worldstate
from .. import models, narrative, worldstate
from . import encoding, history
AUTHORS_NOTE_DEPTH = 3 # actions from the end of history
@@ -88,7 +88,7 @@ class Section:
return count_tokens(self.text)
def length_hint(max_output_tokens: int, *, has_ws: bool) -> str:
def length_hint(max_output_tokens: int) -> str:
"""Ask for a turn that fits inside the output cap, stated as a word budget.
Returns an empty string when the cap is too small to state usefully. The
@@ -98,12 +98,8 @@ def length_hint(max_output_tokens: int, *, has_ws: bool) -> str:
words = int((max_output_tokens - LENGTH_HEADROOM) * WORDS_PER_TOKEN * LENGTH_BUFFER)
if words < MIN_LENGTH_HINT_WORDS:
return ""
tail = (
" Finish the narration and append the state block well inside the limit."
if has_ws
else " Bring the turn to a close well inside the limit rather than "
"stopping mid-sentence."
)
tail = " Finish the narration and append the state block well inside the limit."
# State the number as a ceiling, never as a budget. In measurements, the
# wording "keep this turn under about N words" read to the model as a target
# to fill. It raised the average from 174 words to 246 across five runs, and
@@ -166,30 +162,58 @@ def _script_memory(adventure: models.Adventure) -> dict:
def _history_text(action: models.Action) -> str:
"""Returns an AI turn as the model should see it in replayed history.
The result is the narration with its state block appended again,
reconstructed from the stored delta. The app strips that block before
storing and displaying the turn. Without this function, every past AI turn
would appear to have emitted no state, and the model would copy that pattern
and stop emitting state itself. Player turns and turns with no block pass
through unchanged.
Replayed history is **prose only**. The protocol block is not reconstructed
into it, and the M5 corrective pass is why (review Finding 4).
The block replays the changes the engine ACCEPTED, not the ones the model
sent. Replaying what was sent showed the model a refused change standing as
though it had been applied, while the live values in the same prompt
disagreed with it. Nothing marked which of the two was true, so the model
read its own refused change as correct and sent it again.
Replaying the block was meant to teach the model the output format by
example. What it actually did was put a second, older account of the world
into the same prompt as the authoritative one, with nothing marking which
governed. A fact the reader had explicitly withdrawn through a manual
correction was dropped from the state section and then handed straight back
in the history section, as an accepted event, phrased exactly as the model
had first asserted it. C04 requires a correction to reach the narrator's
context; a correction the next prompt contradicts has not reached it.
This function reads `world_delta` rather than `context_snapshot`. It runs
for every action in the replayed history, and `context_snapshot` is deferred
so that a turn never loads the prompt archive from the database.
Two other things were wrong with it. The blocks are implementation
metadata, not story, and every other consumer of stored text — memory,
summaries, export, the transcript — treats an action's text as prose. And a
turn's accepted events are a record of what was true *then*, which is
precisely what a later correction, retcon or invalidation revises.
The format instruction survives without the examples: `EMIT_RULE` carries a
worked example in the system block and `EMIT_REMINDER` repeats the demand
last, where recency is strongest.
"""
text = action.text
wd = action.world_delta if isinstance(action.world_delta, dict) else None
if wd:
block = worldstate.render_delta_block(worldstate.applied_delta(wd))
if block:
text = f"{text}\n{block}"
return text
return action.text
def _canon_section(adventure: models.Adventure) -> str:
"""The campaign's own rules, rendered for the system block.
Canon is configuration (C01, J03): the campaign writes what is true and what
is forbidden, and both the prompt and the validator read the same field.
Putting it in the system block is what makes C01 a narration-time constraint
as well as a validation-time one — the model is told the rule rather than
only refused after breaking it.
"""
canon = adventure.campaign_canon
if not isinstance(canon, dict):
return ""
lines: list[str] = []
rules = canon.get("rules")
if isinstance(rules, list):
lines += [f"- {rule}" for rule in rules if isinstance(rule, str) and rule.strip()]
forbidden = canon.get("forbidden_status_changes")
if isinstance(forbidden, list):
for rule in forbidden:
if isinstance(rule, dict) and rule.get("from") and rule.get("to"):
lines.append(
f"- Nothing that is {rule['from']} can become {rule['to']}."
)
if not lines:
return ""
body = "\n".join(lines)
return f"Campaign canon (these are true and may not be contradicted):\n{body}"
def _visible_npcs(actions: list[models.Action], stat_schema: dict) -> dict[str, str]:
@@ -259,11 +283,14 @@ def build_context(
stat_schema = adventure.scenario.stat_schema if adventure.scenario else None
has_ws = worldstate.has_schema(stat_schema)
persona_name = adventure.persona_name.strip()
if has_ws:
guide = worldstate.render_reference(stat_schema, persona_name)
if guide:
system_sections.append(Section("world_state_guide", guide))
system_sections.append(Section("world_state_rule", worldstate.EMIT_RULE))
# M5: the typed-event protocol replaces the delta rule for every campaign,
# with or without an inherited stat schema. State is no longer an opt-in
# RPG layer — a story has entities, places and possessions whatever genre it
# is, so the rule is unconditional.
system_sections.append(Section("state_rule", narrative.extract.EMIT_RULE))
canon_text = _canon_section(adventure)
if canon_text:
system_sections.append(Section("campaign_canon", canon_text))
if isinstance(script_mem.get("context"), str) and script_mem["context"].strip():
system_sections.append(Section("script_context", script_mem["context"].strip()))
@@ -301,21 +328,20 @@ def build_context(
memories_section = Section("used_memories", f"Memories:\n{lines_text}")
world_state_section = None
refusal_note = ""
if has_ws:
# One read serves both the in-scene NPCs and the refusal note below.
recent = history.tail(adventure, NPC_WINDOW, exclude_action_id)
block = worldstate.render_state_section(
adventure.world_state, stat_schema, _visible_npcs(recent, stat_schema),
persona_name,
)
if block:
world_state_section = Section("world_state", block)
# Corrections for the previous AI turn only. A refusal the model has
# already had one chance to fix is stale, and repeating it every turn
# would price a correction into the whole rest of the adventure.
last_ai = next((a for a in reversed(recent) if a.type == "ai"), None)
if last_ai is not None:
refusal_note = worldstate.render_refusals(last_ai.world_delta)
# M5: the authoritative narrative state, as the model is shown it. Read from
# the campaign's live document, which head movement keeps pointed at the
# position being read — so an undone story is described by the state it had
# then, not by the state it reached later.
state_block = narrative.render.for_prompt(adventure.narrative_state)
if state_block:
world_state_section = Section("narrative_state", state_block)
# Corrections for the previous AI turn only. A refusal the model has
# already had one chance to fix is stale, and repeating it every turn
# would price a correction into the whole rest of the adventure.
recent = history.tail(adventure, NPC_WINDOW, exclude_action_id)
last_ai = next((a for a in reversed(recent) if a.type == "ai"), None)
if last_ai is not None:
refusal_note = narrative.extract.render_rejections(last_ai.state_rejections)
authors_note_text = adventure.authors_note.strip()
if isinstance(script_mem.get("authorsNote"), str) and script_mem["authorsNote"].strip():
@@ -326,7 +352,7 @@ def build_context(
if isinstance(script_mem.get("frontMemory"), str):
front_memory = script_mem["frontMemory"].strip()
length_note = length_hint(settings.max_output_tokens, has_ws=has_ws)
length_note = length_hint(settings.max_output_tokens)
# The live sections sit below the history, but they are still part of the
# prompt, so they still count against the budget. `world_lore` is the
@@ -341,7 +367,7 @@ def build_context(
+ count_tokens(authors_note)
+ count_tokens(front_memory)
+ count_tokens(length_note)
+ (count_tokens(worldstate.EMIT_REMINDER) if has_ws else 0)
+ count_tokens(narrative.extract.EMIT_REMINDER)
+ count_tokens(refusal_note)
)
available = max(256, settings.context_token_budget - reserved)
@@ -386,7 +412,7 @@ def build_context(
for action in reversed(actions):
# Budget against the text as it appears in the prompt, which includes
# the state block when this adventure tracks world state.
rendered = _history_text(action) if has_ws else action.text
rendered = _history_text(action)
tokens = count_tokens(rendered) + count_tokens(SEPARATOR)
if spent + tokens > history_budget:
if not included_actions:
@@ -407,7 +433,7 @@ def build_context(
# ----- Assemble the story text, with the author's note near the end -----
# Append each AI turn's state block again. The app strips it before storage,
# and the recent history has to show the model the pattern to follow.
texts = [_history_text(a) if has_ws else a.text for a in included_actions]
texts = [_history_text(a) for a in included_actions]
note_sections: list[Section] = []
if authors_note:
pos = max(0, len(texts) - AUTHORS_NOTE_DEPTH)
@@ -431,14 +457,13 @@ def build_context(
# applies to the block that follows it, so this is also the order in which
# the model acts.
note_sections.append(Section("length_hint", length_note))
if has_ws:
# A correction for the previous turn sits directly above the reminder
# to emit a block, which is the instruction it modifies.
if refusal_note:
note_sections.append(Section("world_state_refusals", refusal_note))
# The emit rule sits in the system block, far from where the model
# generates text, so repeat it last where it has the most effect.
note_sections.append(Section("world_state_reminder", worldstate.EMIT_REMINDER))
# A correction for the previous turn sits directly above the reminder to
# emit a block, which is the instruction it modifies.
if refusal_note:
note_sections.append(Section("state_refusals", refusal_note))
# The emit rule sits in the system block, far from where the model
# generates text, so repeat it last where it has the most effect.
note_sections.append(Section("state_reminder", narrative.extract.EMIT_REMINDER))
story_sections = [s for s in note_sections if s.text]
system_text = SEPARATOR.join(s.text for s in system_sections if s.text)