M5: genre-neutral authoritative narrative state, with review corrections

Replaces AI-DnD's RPG relative-delta world state with the genre-neutral typed
narrative state of ADR 010: explicit, absolute, allowlisted events proposed by
the model, validated by the application, applied to one authoritative document,
and snapshotted per position so restore stays a row read.

This commit includes the corrective pass that followed the independent review
in planning/reports/M5-IMPLEMENTATION-REPORT.md. The invariant it exists to
hold is:

    visible active transcript position == stored head == authoritative state

Narrator editing (D10, STORY-BRANCH-SEMANTICS §§14-15)

  A narrator edit no longer rewrites a row. It returns to the state before the
  turn, takes the reader's exact text as the accepted narration, re-derives the
  state that text implies, and becomes a new active continuation — while the
  original narration keeps its words, its live flag and its whole future as
  retained history. At the tip the correction is another take; with story below
  it, it forks. No new history machinery: this is the existing fork/take/head
  path with the reader's text in place of a generated reply. The §14A refusal
  is therefore gone for narrator turns, and remains only for player input.

Pre-M5 positions

  Migration 88 backfills the empty narrative document onto every action written
  before M5, and a missing snapshot now restores the empty document instead of
  leaving the previous position's state standing. Restoring to an old Save
  Point no longer leaves a later position's entities and facts on screen.

Narrator context

  Replayed history carries prose only; the machine-readable block is no longer
  reconstructed into past turns, where it contradicted the authoritative state
  in the same prompt. A fact withdrawn by a manual correction is now named as
  no longer true, with the reader's reason, rather than silently dropped.

Also

  - state_changes joins the action-list bulk read, removing one query per row.
  - Extraction takes only the application's own protocol payload: an ordinary
    ```json or ```python block in a story survives, and a mangled proposal
    still does not reach the reader.

Planning: ADR 013 records the authoritative document shape; §§14-15/14A, D10,
C04 and BUILD-MILESTONES are updated to describe what exists. Debt is recorded
against M8 (scenario editor UX) and M9 (export of the audit trail).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PWU4gTfLYY6Qq9U7aa9Qw2

backend/app/models.py

@@ -119,7 +119,28 @@ class Adventure(Base):
     script_state: Mapped[dict] = mapped_column(JSON, default=dict)
     # Phase 12: live RPG world state (world/player/npc stats + milestones),
     # instantiated from the scenario's stat_schema. Empty when there's no RPG layer.
+    #
+    # **Legacy as of M5**, and no longer authoritative. M5 replaced the
+    # relative-delta protocol this column served (ADR 010); the turn engine no
+    # longer writes it, and nothing reads it to decide anything. It stays so
+    # that a pre-M5 database opens unchanged and its numbers remain visible to
+    # whoever wants to look — `narrative_state` below is what the story means
+    # now. Reinterpreting these values as generic narrative facts would be
+    # inventing meaning the data does not carry, which the M5 brief forbids.
     world_state: Mapped[dict] = mapped_column(JSON, default=dict)
+    # M5: the authoritative narrative state, as it stands at the active head.
+    # Genre-neutral (ADR 006), written only by validated typed events (ADR 010),
+    # and restored from the destination node's snapshot whenever the head moves,
+    # so it always describes the story being read rather than a story the reader
+    # has stepped back from.
+    narrative_state: Mapped[dict] = mapped_column(
+        CompressedJSON, nullable=True, default=None
+    )
+    # Campaign canon: rules the story may not contradict, as configuration
+    # rather than code (C01, J03). A fantasy campaign forbidding resurrection
+    # and a science-fiction one forbidding faster-than-light travel use the same
+    # field and the same validator; neither word appears in the application.
+    campaign_canon: Mapped[dict | None] = mapped_column(JSON, nullable=True)
     # The ${Placeholder} answers collected when this adventure was started, kept
     # so "Update from scenario" can re-fill freshly copied scenario text with the
     # same values. NULL for adventures created before this column existed.
@@ -302,6 +323,97 @@ class Checkpoint(Base):
     )
 
 
+class StateProposal(Base):
+    """M5: what the model proposed, and what the application did about it.
+
+    `DATA-MODEL.md` §19 requires the model's proposal to be *distinct from*
+    accepted state, and this table is that separation made physical. The model
+    writes here; it never writes `state_events`, and it never writes a snapshot.
+
+    A row exists whether the proposal was accepted, partly accepted, rejected or
+    unparseable. A rejected proposal is not authoritative and changes nothing,
+    but it is the record that explains why the state does not say what the
+    narration seems to say — without it, a wrong-looking campaign has no trail
+    to follow. `raw_output` is kept for exactly the case that matters most: the
+    block that did not parse, which no structured column could hold.
+    """
+
+    __tablename__ = "state_proposals"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    adventure_id: Mapped[int] = mapped_column(
+        ForeignKey("adventures.id", ondelete="CASCADE")
+    )
+    # The node whose narration produced this. NULL only for a manual correction,
+    # which has a coordinate but no narration behind it.
+    action_id: Mapped[int | None] = mapped_column(
+        ForeignKey("actions.id", ondelete="CASCADE"), nullable=True
+    )
+    branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    # Which model produced it, so a later comparison of extraction quality has
+    # something to group by. Empty for a manual correction.
+    model_name: Mapped[str] = mapped_column(String(200), default="")
+    # `accepted_story` or `manual_correction` — who is asserting this.
+    source: Mapped[str] = mapped_column(String(40), default="accepted_story")
+    # accepted | partially_accepted | rejected | unparseable
+    status: Mapped[str] = mapped_column(String(30), default="accepted")
+    # The block as written, including when it did not parse.
+    raw_output: Mapped[str] = mapped_column(Text, default="")
+    # The parsed payload, the events accepted, and every rejection with its
+    # reason. Compressed for the same reason the prompt is: a busy turn's
+    # rejections are the largest thing here and nothing reads them in bulk.
+    detail: Mapped[dict | None] = mapped_column(
+        CompressedJSON, nullable=True, deferred=True
+    )
+    created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
+
+
+class StateEvent(Base):
+    """M5: one accepted change to the authoritative narrative state.
+
+    The audit half of `DATA-MODEL.md` §17's hybrid. Append-only, ordered, and
+    **never read to reconstruct state** — that is the snapshot's job, and mixing
+    the two would make restore proportional to campaign length, which ADR 012
+    and M4 both forbid.
+
+    What this table answers is §8's list: what changed, why, which turn caused
+    it, whether a model or the user asserted it, and what the value was before.
+    `before` is stored per event rather than derived, because deriving it would
+    mean replaying — the thing the hybrid exists to avoid.
+
+    Events carry the story coordinate as well as the action id. The coordinate
+    survives a retry replacing the live take at that position, exactly as a Save
+    Point's does; the action id says which attempt actually proposed it.
+    """
+
+    __tablename__ = "state_events"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    adventure_id: Mapped[int] = mapped_column(
+        ForeignKey("adventures.id", ondelete="CASCADE")
+    )
+    proposal_id: Mapped[int | None] = mapped_column(
+        ForeignKey("state_proposals.id", ondelete="SET NULL"), nullable=True
+    )
+    action_id: Mapped[int | None] = mapped_column(
+        ForeignKey("actions.id", ondelete="CASCADE"), nullable=True
+    )
+    branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    # Order within one proposal, so a turn's events replay for a reader in the
+    # order they were applied.
+    sequence: Mapped[int] = mapped_column(Integer, default=0)
+    event_type: Mapped[str] = mapped_column(String(60), default="")
+    payload: Mapped[dict | None] = mapped_column(JSON, nullable=True)
+    # What the affected value was immediately before this event, so the audit
+    # can answer "what did it used to be" without reconstruction. NULL when the
+    # event established something that did not exist.
+    before: Mapped[dict | None] = mapped_column(JSON, nullable=True)
+    source: Mapped[str] = mapped_column(String(40), default="accepted_story")
+    created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
+
+
 class Memory(Base):
     """Phase 6: an auto-summarized (or hand-written) fact about the adventure.
 
@@ -504,10 +616,75 @@ class Action(Base):
     world_state_after: Mapped[dict | None] = mapped_column(
         JSON, nullable=True, deferred=True
     )
+    # M5: the authoritative narrative state as it stood after this node played.
+    # The genre-neutral successor to `world_state_after`, and the reason Undo,
+    # Redo and Save Point restore stay bounded: a position's state is one row
+    # read, not a replay of every event since the campaign began
+    # (`TECHNICAL-DESIGN.md` §10.4, and the M4 note that made it load-bearing
+    # for Save Points too).
+    #
+    # `DATA-MODEL.md` §17 selects the hybrid — validated events for audit, a
+    # snapshot for reads and restore. `state_events` is the audit half; this
+    # column is the restore half, and nothing reconstructs a document from
+    # events.
+    #
+    # Deferred and compressed for the reasons `context_snapshot` is: only the
+    # single node being moved to reads it, and a document carrying a campaign's
+    # entities and facts is larger than the RPG dict it replaces. `world_delta`
+    # has an M5 counterpart in `state_changes` for the bulk read.
+    narrative_state_after: Mapped[dict | None] = mapped_column(
+        CompressedJSON, nullable=True, deferred=True
+    )
+    # The small slice needed in bulk: the events accepted here, the ones
+    # refused, and short lines for the chip under an AI message. Same role
+    # `world_delta` played, and a separate column for the same reason — the
+    # context builder reads it for every action in the replayed history, and
+    # the snapshot beside it is deferred so a turn never loads the prompt
+    # archive. Shape: {"accepted": [...], "rejected": [...], "summary": [...]}.
+    state_changes: Mapped[dict | None] = mapped_column(JSON, nullable=True)
     created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
 
     adventure: Mapped[Adventure] = relationship(back_populates="actions")
 
+    @property
+    def state_events_replay(self) -> list[dict]:
+        """M5: the accepted events this turn produced, for replay into the prompt.
+
+        Read from `state_changes`' companion slice in the bulk-read column
+        rather than from the deferred snapshot, because the context builder
+        calls this for every action in the replayed history and loading the
+        prompt archive per action is the egress mistake this project keeps a
+        regression test about.
+        """
+        changes = self.state_changes
+        if not isinstance(changes, dict):
+            return []
+        events = changes.get("accepted")
+        return events if isinstance(events, list) else []
+
+    @property
+    def state_rejections(self) -> list[dict]:
+        """M5: what this turn proposed that the application refused.
+
+        Fed back to the model as a correction for one turn only. A refusal it
+        has already had a chance to fix is stale, and repeating it forever would
+        price one bad turn into the rest of the campaign.
+        """
+        changes = self.state_changes
+        if not isinstance(changes, dict):
+            return []
+        rejected = changes.get("rejected")
+        return rejected if isinstance(rejected, list) else []
+
+    @property
+    def state_summary(self) -> list[str]:
+        """M5: the short lines shown under an AI message: what changed here."""
+        changes = self.state_changes
+        if not isinstance(changes, dict):
+            return []
+        lines = changes.get("summary")
+        return [str(line) for line in lines] if isinstance(lines, list) else []
+
     @property
     def world_changes(self) -> list[dict]:
         """Compact per-turn RPG state changes (Phase 12), for the inline summary