M5: genre-neutral authoritative narrative state, with review corrections

Replaces AI-DnD's RPG relative-delta world state with the genre-neutral typed
narrative state of ADR 010: explicit, absolute, allowlisted events proposed by
the model, validated by the application, applied to one authoritative document,
and snapshotted per position so restore stays a row read.

This commit includes the corrective pass that followed the independent review
in planning/reports/M5-IMPLEMENTATION-REPORT.md. The invariant it exists to
hold is:

    visible active transcript position == stored head == authoritative state

Narrator editing (D10, STORY-BRANCH-SEMANTICS §§14-15)

  A narrator edit no longer rewrites a row. It returns to the state before the
  turn, takes the reader's exact text as the accepted narration, re-derives the
  state that text implies, and becomes a new active continuation — while the
  original narration keeps its words, its live flag and its whole future as
  retained history. At the tip the correction is another take; with story below
  it, it forks. No new history machinery: this is the existing fork/take/head
  path with the reader's text in place of a generated reply. The §14A refusal
  is therefore gone for narrator turns, and remains only for player input.

Pre-M5 positions

  Migration 88 backfills the empty narrative document onto every action written
  before M5, and a missing snapshot now restores the empty document instead of
  leaving the previous position's state standing. Restoring to an old Save
  Point no longer leaves a later position's entities and facts on screen.

Narrator context

  Replayed history carries prose only; the machine-readable block is no longer
  reconstructed into past turns, where it contradicted the authoritative state
  in the same prompt. A fact withdrawn by a manual correction is now named as
  no longer true, with the reader's reason, rather than silently dropped.

Also

  - state_changes joins the action-list bulk read, removing one query per row.
  - Extraction takes only the application's own protocol payload: an ordinary
    ```json or ```python block in a story survives, and a mangled proposal
    still does not reach the reader.

Planning: ADR 013 records the authoritative document shape; §§14-15/14A, D10,
C04 and BUILD-MILESTONES are updated to describe what exists. Debt is recorded
against M8 (scenario editor UX) and M9 (export of the audit trail).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PWU4gTfLYY6Qq9U7aa9Qw2

backend/app/migrations.py

@@ -361,6 +361,47 @@ MIGRATIONS: list[tuple[int, str | dict[str, str]]] = [
     # inventing one would be inventing the decision.
     (80, "CREATE INDEX IF NOT EXISTS ix_checkpoints_adventure "
          "ON checkpoints (adventure_id)"),
+    # M5: genre-neutral authoritative narrative state. `create_all` builds the
+    # two new tables — `state_proposals` and `state_events` — as it did
+    # `memories`, `branches` and `checkpoints`; these are the columns it cannot
+    # add to tables that already exist, plus the indexes the audit reads need.
+    #
+    # **No backfill, deliberately.** The inherited RPG world state is numbers
+    # against a stat schema: `player.gold = 70`, `npc.gwen.trust = 3`. Nothing
+    # in that says who Gwen is, where anyone stands, or what anyone holds, and a
+    # narrative fact invented from a number would be fiction the campaign never
+    # established — exactly what the M5 brief forbids. So the old columns are
+    # left intact and non-authoritative, and every campaign starts M5 with an
+    # empty narrative state that its next turns fill in.
+    #
+    # The campaign's own `narrative_state` is left NULL: an adventure with no
+    # M5 turns yet has no document, and the first one writes it.
+    #
+    # Per-action snapshots are a different question, and the M5 corrective pass
+    # settled it the other way (review Finding 3). This block originally left
+    # those NULL too, reasoning that an empty document would be "a claim, not an
+    # absence". The consequence was worse than the claim: restoring to an old
+    # position left the state of a *later* position standing, so the transcript
+    # and the state described different moments. Backfilling the empty document
+    # at version 88 says the only true thing about a pre-M5 position — the
+    # narrative-state system established nothing there, because it did not yet
+    # exist — and keeps head, transcript and state in agreement. The legacy RPG
+    # columns are untouched and still restored beside it.
+    (81, "ALTER TABLE adventures ADD COLUMN narrative_state BLOB"),
+    (82, "ALTER TABLE adventures ADD COLUMN campaign_canon JSON"),
+    (83, "ALTER TABLE actions ADD COLUMN narrative_state_after BLOB"),
+    (84, "ALTER TABLE actions ADD COLUMN state_changes JSON"),
+    (85, "CREATE INDEX IF NOT EXISTS ix_state_events_adventure "
+         "ON state_events (adventure_id, id)"),
+    (86, "CREATE INDEX IF NOT EXISTS ix_state_events_action "
+         "ON state_events (action_id)"),
+    (87, "CREATE INDEX IF NOT EXISTS ix_state_proposals_adventure "
+         "ON state_proposals (adventure_id, id)"),
+    # M5 corrective pass. No DDL — 83 already added the column. This version
+    # exists to carry the data pass that fills it in for rows that predate it,
+    # so that every position an existing campaign can be restored to has a
+    # snapshot. See `_backfill_narrative_snapshots`.
+    (88, "-- narrative snapshot backfill (data pass only)"),
 ]
 
 LATEST_VERSION = max((v for v, _ in MIGRATIONS), default=1)
@@ -374,6 +415,7 @@ TREE_BACKFILL_VERSION = 52
 CURSOR_ANCHOR_VERSION = 56
 SIBLING_SPLIT_VERSION = 60
 PARENT_BACKFILL_VERSION = 64
+NARRATIVE_SNAPSHOT_VERSION = 88
 
 # An adventure with no actions has no tip. A value of -1 keeps the rule that the
 # next node goes at `head_depth + 1` true without a special case. This matches
@@ -391,6 +433,30 @@ SNAPSHOT_BATCH = 50
 BACKFILL_BATCH = 200
 
 
+def _backfill_narrative_snapshots(conn) -> None:
+    """Gives every pre-M5 action the empty narrative document as its outcome.
+
+    One statement, no row loop: the document is identical for every row, so it
+    is encoded once in Python and bound as a single parameter. `narrative.model`
+    owns the shape and `compression.pack` owns the encoding, so this cannot
+    drift from what `snapshot_outcome` writes.
+
+    Why the empty document rather than NULL is argued at migration 81. In short:
+    a position with no snapshot used to mean "leave the live state alone", which
+    let a later position's state stand while the reader was somewhere else.
+    """
+    from .compression import pack
+    from .narrative import model as narrative_model
+
+    conn.execute(
+        text(
+            "UPDATE actions SET narrative_state_after = :document "
+            "WHERE narrative_state_after IS NULL"
+        ),
+        {"document": pack(narrative_model.empty())},
+    )
+
+
 def _backfill_world_delta(conn) -> None:
     """Populates `actions.world_delta` from the existing `context_snapshot`.
 
@@ -1095,9 +1161,12 @@ def bootstrap(engine: Engine, through: int = LATEST_VERSION) -> None:
             if current < version <= through:
                 statement = _for_dialect(sql, conn.dialect.name)
                 # Skip the DDL when it has already run. The data pass below it
-                # still runs.
-                if not (_column_already_there(conn, statement)
-                        or _column_already_gone(conn, statement)):
+                # still runs. A version whose whole content is a data pass
+                # carries a comment in place of DDL and executes nothing.
+                if not statement.lstrip().startswith("--") and not (
+                    _column_already_there(conn, statement)
+                    or _column_already_gone(conn, statement)
+                ):
                     conn.execute(text(statement))
                 if version == WORLD_DELTA_VERSION:
                     _backfill_world_delta(conn)
@@ -1127,5 +1196,7 @@ def bootstrap(engine: Engine, through: int = LATEST_VERSION) -> None:
                 # exist.
                 if version == PARENT_BACKFILL_VERSION:
                     _backfill_parents(conn)
+                if version == NARRATIVE_SNAPSHOT_VERSION:
+                    _backfill_narrative_snapshots(conn)
                 current = version
         _set_version(conn, current)