M6: branch-safe context, summaries and long-term story memory

Aligns the inherited AI-DnD memory and context foundation with the history,
authority and state model M3-M5 established. Long stories now reach the narrator
through a bounded, lineage-safe, inspectable context rather than a growing
transcript.

This commit includes the corrective work that followed the independent review in
planning/reports/M6-IMPLEMENTATION-REPORT.md. The first implementation reported
E03 as passing and it was not; the report records that history rather than
hiding it.

What was already correct, and was kept rather than rebuilt

  Memory lineage. Memories already carried (branch_id, depth) and retrieval
  already filtered through the capped-path clause; the ten-step negative control
  was measured passing against b7005e6 before any change here. M6 adds the
  regression tests that pin it, plus provenance and authority on the result.

Summary lineage — both halves

  A summary is a row carrying the coordinate of the last node it covers, and
  eligibility is the same head-capped lineage clause memories use. That alone
  was not enough: generation was seeded from adventures.story_summary, a
  campaign-global column with no lineage, so after a divergence the summariser
  was handed the abandoned line's prose and asked to update it. The row it
  produced was correctly anchored and therefore looked safe while its sentences
  described a story the reader had left.

  Generation is now seeded from summaries.current — the same question the
  context builder asks — so the input and the output are scoped by one rule.
  adventures.story_summary remains a reader-facing mirror for the Plot panel and
  the export bundle, kept in step when a summary is written and when the head
  moves, and nothing authoritative reads it.

Retrieval redundancy

  With a real embedding model, four near-identical memories crowded out the one
  distinctive clue, which survived only because the default memory_top_k is 5.
  Retrieval now drops a candidate that repeats one already chosen, never across
  authority classes, at a threshold measured against the configured embedding
  model. The clue is retrieved at top_k 5, 4 and 3. Ranking itself is unchanged;
  the further factors CONTEXT-AND-MEMORY §20 contemplates remain unimplemented
  and are recorded as such.

Memory authority, budgeting, observability

  Memory.authority is accepted_story or heuristic, classified by the application
  and marked in the prompt; retrieval never writes state. The reply is reserved
  out of the context budget, and an impossible configuration fails clearly
  instead of overflowing. Each derived pass records ok/idle/failed per campaign,
  served by GET /adventures/{id}/derived and shown in Insights, so the M2
  failure — a dead memory bank with a green suite — is visible if it recurs.
  Provider-wiring tests mock no factory.

Also: two pre-existing test-suite leaks fixed; two fixtures that stored one
vector in every memory now use distinct ones, so lineage assertions stay
readable alongside redundancy suppression.

Planning: CONTEXT-AND-MEMORY, TECHNICAL-DESIGN, DATA-MODEL, V1-ACCEPTANCE-TESTS,
BUILD-MILESTONES, VERSION and planning/README updated to describe what exists,
including that a valid E03 test must regenerate a summary after diverging. The
M5 report was rotated to planning/archive/milestone-reports/. No new ADR — every
choice implements a decision the package had already settled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PWU4gTfLYY6Qq9U7aa9Qw2

backend/app/routers/adventures/crud.py

@@ -10,7 +10,8 @@ from sqlalchemy.orm import Session
 from sqlalchemy.orm.attributes import set_committed_value
 
 from ... import (
-    attempts, head, images, limits, memorybank, models, schemas, tree, worldstate,
+    attempts, head, images, limits, memorybank, models, schemas, summaries, tree,
+    worldstate,
 )
 from ...database import get_db
 
@@ -294,8 +295,18 @@ def update_adventure(
     db: Session = Depends(get_db),
     adventure: models.Adventure = Depends(current_adventure),
 ):
-    for field, value in payload.model_dump(exclude_unset=True).items():
+    fields = payload.model_dump(exclude_unset=True)
+    for field, value in fields.items():
         setattr(adventure, field, value)
+    # M6: a summary the reader typed is still a summary, so it is anchored to
+    # the position they typed it at rather than left in a column with no
+    # lineage. Otherwise a hand-written summary would survive an Undo and a
+    # divergence that its generated equivalent correctly does not (E03).
+    if "story_summary" in fields:
+        typed = (fields["story_summary"] or "").strip()
+        held = summaries.current(db, adventure)
+        if typed and (held is None or held.text.strip() != typed):
+            summaries.record(db, adventure, typed, trigger="manual")
     db.commit()
     return adventure
 

backend/app/routers/adventures/insights.py

@@ -7,8 +7,8 @@ returns the prompt a turn was actually generated from. Neither writes anything.
 from fastapi import Depends, HTTPException
 from sqlalchemy.orm import Session
 
-from ... import memorybank, models
-from ...context import build_context
+from ... import derived, memorybank, models, summaries
+from ...context import ContextOverflow, build_context
 from ...database import get_db
 from ..settings import get_settings
 
@@ -24,10 +24,55 @@ async def dry_run_context(
     """Returns what the app would send to the AI if the player continued now."""
     settings = get_settings(db, user)
     memories = await memorybank.retrieve_memories(adventure, settings, update_stats=False)
-    _, _, report = build_context(adventure, settings, memories)
+    try:
+        _, _, report = build_context(adventure, settings, memories)
+    except ContextOverflow as exc:
+        # M6: a dry run of a prompt that cannot be built is still an answer, and
+        # a more useful one than a 500. The reader opened this panel to find out
+        # what would be sent; "nothing, because the protected context does not
+        # fit, and here is by how much" is exactly that.
+        raise HTTPException(422, str(exc)) from exc
     return report
 
 
+@router.get("/{adventure_id}/derived")
+def derived_status(
+    db: Session = Depends(get_db),
+    adventure: models.Adventure = Depends(current_adventure),
+):
+    """M6: whether background memory, summary and embedding work is healthy.
+
+    The surface that makes a dead memory bank findable. M2 shipped with the
+    whole bank failing inside a fire-and-forget task and nothing anywhere said
+    so — not the UI, not a log a player would read, not a failing test
+    (`BUILD-MILESTONES.md`, note from M2). This endpoint is where that now
+    shows.
+    """
+    # Resolved once, not once per row: which summary the current head is
+    # entitled to. Asking inside the comprehension would be one query per
+    # summary, which is the shape M5 spent a finding removing.
+    eligible = summaries.current(db, adventure)
+    eligible_id = eligible.id if eligible is not None else None
+    status = derived.report(db, adventure.id)
+    return {
+        "status": status,
+        "failing": [row["kind"] for row in status if row["status"] == "failed"],
+        "summaries": [
+            {
+                "id": row.id,
+                "branch_id": row.branch_id,
+                "depth": row.depth,
+                "trigger": row.trigger,
+                "model": row.model_name,
+                "eligible": row.id == eligible_id,
+                "created_at": row.created_at.isoformat() if row.created_at else None,
+                "preview": row.text[:200],
+            }
+            for row in summaries.all_for(db, adventure)
+        ],
+    }
+
+
 @router.get("/{adventure_id}/actions/{action_id}/context")
 def action_context(
     adventure_id: int,

backend/app/routers/adventures/turns.py

@@ -16,7 +16,7 @@ from ... import (
     attempts, head, limits, memorybank, models, narrative, schemas, tree,
     worldstate,
 )
-from ...context import build_context, cursors
+from ...context import ContextOverflow, build_context, cursors
 from ...database import get_db
 from ...providers import OpenAICompatibleProvider, PromptParts, ProviderError
 from ...sse import SSE_HEADERS, sse, turn_error
@@ -164,9 +164,18 @@ async def _generate_turn(
     memories = await memorybank.retrieve_memories(
         adventure, settings, update_stats=True, exclude_action_id=replacing_id
     )
-    system_text, story_text, snapshot = build_context(
-        adventure, settings, memories, exclude_action_id=replacing_id
-    )
+    try:
+        system_text, story_text, snapshot = build_context(
+            adventure, settings, memories, exclude_action_id=replacing_id
+        )
+    except ContextOverflow as exc:
+        # M6: the protected context does not fit in the configured budget, so
+        # there is no prompt to send. This is a settings problem the reader can
+        # fix, and the message says how — reporting it as a failed turn keeps
+        # the story intact and tells them what to change, where building the
+        # prompt anyway would return a silently truncated reply.
+        yield turn_error(str(exc))
+        return
 
     parts = PromptParts(system=system_text, story=story_text)