M9: a campaign you can actually get back

A campaign could already be exported and imported. What could not survive the
trip was everything that explains it: the state events behind the authoritative
document, the prompt each turn was actually given, the passages it was shown,
the summaries that carry long-story continuity, and which take belonged to which
turn. An imported campaign could be read and could no longer say why it was what
it was — and a manual correction, the one state change no narration explains,
was indistinguishable from something the story had established.

The bundle is now `ai-dnd-adventure-v3`, and the version is the design rather
than a side effect. Everything added here could have been another optional key,
the way persona, Save Points, narrative state and imported knowledge each were.
That mechanism stops working at exactly this addition: a v2 file with no prompt
provenance is ambiguous between "written before M9" and "written by M9 from a
campaign that has none", and those are different facts about a campaign. A
version number is how a recovery file states what it was capable of recording.
v1 and v2 still import, and every seam from pre-active-head onward is tested for
the rule that an older file is never reinterpreted under a newer assumption.

Two categories became three. "Chosen travels, derived is recomputed" was enough
until stored prompts had to be decided: they are derived, and they must travel
anyway. The test that separates evidence from cache is not "could this be
recomputed" but "would a recomputation answer the same question" — a rebuilt
search index answers the same question, a rebuilt prompt says what the turn
would be told *now*, which is the opposite of what the inspector is for.

Also here: a real SQLite backup, through the online backup API rather than a
file copy, taken while the application is running and verified before it is
kept; story cards settled as compatibility-only legacy data and taken out of the
narrator's prompt, because they were the untracked path around knowledge
authority that IMPORTED-KNOWLEDGE-DESIGN §73 already forbade; and no schema
change at all, proved against a database M8's own code wrote.

Three defects, found by running the milestone's own tests rather than by reading
them. Deleting a campaign leaked its FTS index rows, and SQLite then handed the
freed ids to the next source imported into any campaign, which failed with an
integrity error that Reindex could not repair — both ends are closed, and a
database already carrying the damage now repairs itself. An imported node with
no state snapshot was being stamped with the campaign's head state, so an Undo
to turn 2 showed what the story knew at turn 20. And the snapshot relink did not
persist at all, because it mutated a dict in place on a column SQLAlchemy tracks
by assignment: it looked correct in memory and wrote the wrong ids to disk.

Carrying per-turn prompts looked like it would halve the length of campaign that
can be restored. Measured — and after compressing them inside the file —
everything M9 added costs 12% of it: the import ceiling moves from about 318
turns to about 279, against a 100-turn certification target. The dominant cost
is not M9's at all. The per-position narrative state document is 74% of a
bundle, and v2 already carried it.

Backend 1,102 passed / 14 skipped / 0 failed. Frontend 145 passed. Lint,
production build and Docker build clean. Verified across two server processes
with two data directories, and in a real browser against a real narrator.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qyn3oRd4D6pi72nKBG725B

backend/tests/test_m9_legacy_bundles.py

@@ -0,0 +1,365 @@
+"""M9: every older bundle still imports, and none is reinterpreted.
+
+A backup that stops importing is not a backup, so the importer keeps every
+version it has ever written. That is the easy half. The hard half is the rule
+`V1-ACCEPTANCE-TESTS.md` I07 states about the head and this file generalises:
+
+> Do not reinterpret missing legacy data using modern assumptions that did not
+> exist when the file was written.
+
+An older file is missing things because its **format** could not carry them, not
+because the campaign lacked them, and the two demand opposite treatment. A file
+written before the head was carried opens at its tip, because tip was the only
+position that format could represent — reproducing what it recorded. A file
+written before state events existed opens with no state events, because
+manufacturing an audit trail from the snapshots it does carry would be this
+build's reading of a history it never saw, handed to a reader as the record of
+what happened.
+
+Each seam below is built by taking a real v3 export and removing exactly what
+the older format could not hold. That is deliberate: a checked-in fixture file
+drifts, and a hand-written one tests a shape nothing ever wrote.
+
+    python -m pytest tests/test_m9_legacy_bundles.py -v
+"""
+
+import copy
+
+import pytest
+from fastapi import Depends
+from fastapi.testclient import TestClient
+
+from app import auth, bundle, limits, memorybank, models
+from app.database import Base, SessionLocal, engine, get_db
+from app.knowledge import embeddings
+from app.main import app
+from app.routers import adventures
+
+import m9_fixture
+from fakes import ScriptedProvider
+from test_m9_portability import StubDerived
+
+
+@pytest.fixture()
+def client(monkeypatch):
+    Base.metadata.create_all(bind=engine)
+    memorybank._vector_cache.clear()
+    embeddings._cache.clear()
+    setup = SessionLocal()
+    user = models.User(is_guest=False, email="legacy@example.com")
+    setup.add(user)
+    setup.flush()
+    setup.add(models.Settings(
+        user_id=user.id, model="test-model", embedding_model="stub-embed",
+        context_token_budget=4000, max_output_tokens=400,
+    ))
+    adventure = models.Adventure(
+        user_id=user.id, title="Source",
+        campaign_canon=m9_fixture.CAMPAIGN_CANON,
+    )
+    setup.add(adventure)
+    setup.flush()
+    setup.add(models.Action(
+        adventure_id=adventure.id, type="start", text=m9_fixture.OPENING,
+    ))
+    setup.commit()
+    adv_id, user_id = adventure.id, user.id
+    setup.close()
+
+    monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
+    monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
+    monkeypatch.setattr(memorybank, "embedding_provider", lambda s: StubDerived())
+    monkeypatch.setattr(memorybank, "summary_provider", lambda s: StubDerived())
+    app.dependency_overrides[auth.get_current_user] = (
+        lambda db=Depends(get_db): db.get(models.User, user_id)
+    )
+    test_client = TestClient(app)
+    test_client.adv_id = adv_id
+    try:
+        yield test_client
+    finally:
+        app.dependency_overrides.clear()
+        adventures.turns._active_turns.clear()
+        memorybank._vector_cache.clear()
+        embeddings._cache.clear()
+        Base.metadata.drop_all(bind=engine)
+
+
+@pytest.fixture()
+def current(client):
+    """A real v3 export of the M9 fixture, to age backwards from."""
+    m9_fixture.build(client, client.adv_id)
+    response = client.get(f"/api/adventures/{client.adv_id}/export")
+    assert response.status_code == 200
+    return response.json()
+
+
+# ---------------------------------------------------- ageing a bundle backwards
+
+def as_of(payload: dict, era: str) -> dict:
+    """The same campaign as an export from an earlier era.
+
+    Each step removes only what that era's format genuinely could not carry, so
+    the result is the file a build of that vintage would have produced from this
+    campaign — not a mutilated modern one.
+    """
+    older = copy.deepcopy(payload)
+    eras = ("pre-m9", "pre-m7", "pre-m5", "pre-save-points", "pre-active-head")
+    assert era in eras, era
+    reached = eras.index(era)
+
+    # M9 (v3): the evidence sections and the node identities.
+    older["format"] = bundle.TREE_FORMAT
+    for key in ("stateEvents", "stateProposals", "summaries"):
+        older.pop(key, None)
+    for action in older["actions"]:
+        for key in ("contextSnapshot", "contextSnapshotZ", "id", "parentId"):
+            action.pop(key, None)
+    for memory in older.get("memories") or []:
+        memory.pop("authority", None)
+    for source in older.get("knowledge") or []:
+        for key in ("sourceId", "parserVersion", "chunkingVersion"):
+            source.pop(key, None)
+    if reached == 0:
+        return older
+
+    # M7: the imported knowledge library.
+    older.pop("knowledge", None)
+    if reached == 1:
+        return older
+
+    # M5: the authoritative narrative state, its per-position snapshots, and
+    # the campaign's own canon.
+    for key in ("narrativeState", "campaignCanon"):
+        older.pop(key, None)
+    for action in older["actions"]:
+        for key in ("narrativeStateAfter", "stateChanges"):
+            action.pop(key, None)
+    if reached == 2:
+        return older
+
+    # M4: named Save Points.
+    older.pop("checkpoints", None)
+    if reached == 3:
+        return older
+
+    # M3: the chosen head. Such a file could only ever be read at its tip.
+    older.pop("headDepth", None)
+    return older
+
+
+def bring_back(client, payload) -> int:
+    response = client.post("/api/adventures/import", json=payload)
+    assert response.status_code == 201, response.text[:500]
+    return response.json()["id"]
+
+
+def _rows(adv_id, model) -> int:
+    with SessionLocal() as db:
+        return db.query(model).filter(model.adventure_id == adv_id).count()
+
+
+def _tree_size(client, adv_id) -> int:
+    """Every retained row, which is what "no accepted story was lost" means."""
+    return len(client.get(f"/api/adventures/{adv_id}/export").json()["actions"])
+
+
+# --------------------------------------------------------------- every era
+
+@pytest.mark.parametrize("era", [
+    "pre-m9", "pre-m7", "pre-m5", "pre-save-points", "pre-active-head",
+])
+def test_no_accepted_story_is_lost_at_any_seam(client, current, era):
+    """The floor under every case below: the turns all arrive.
+
+    Counted over the whole retained tree rather than the active path, because
+    the head moves between eras and a count of what is on screen would move
+    with it.
+    """
+    copy_id = bring_back(client, as_of(current, era))
+    assert _tree_size(client, copy_id) == len(current["actions"])
+
+
+@pytest.mark.parametrize("era", [
+    "pre-m9", "pre-m7", "pre-m5", "pre-save-points", "pre-active-head",
+])
+def test_nothing_is_invented_to_fill_a_gap_the_format_left(client, current, era):
+    """Absent means the format could not say. It never means "make one up".
+
+    Each era is checked against what that era's files could hold: a pre-M9 file
+    gets no audit trail and no summaries, a pre-M7 file no knowledge, a pre-M5
+    file no state, a pre-Save-Point file no Save Points.
+    """
+    copy_id = bring_back(client, as_of(current, era))
+    reached = ("pre-m9", "pre-m7", "pre-m5", "pre-save-points",
+               "pre-active-head").index(era)
+
+    assert _rows(copy_id, models.StateEvent) == 0
+    assert _rows(copy_id, models.StateProposal) == 0
+    assert _rows(copy_id, models.Summary) == 0
+    if reached >= 1:
+        assert _rows(copy_id, models.KnowledgeSource) == 0
+        assert _rows(copy_id, models.KnowledgeChunk) == 0
+    if reached >= 2:
+        state = client.get(f"/api/adventures/{copy_id}/state").json()
+        assert state["document"]["facts"] == []
+        assert state["document"]["entities"] == {}
+        assert client.get(f"/api/adventures/{copy_id}").json()["canon_rules"] == []
+    if reached >= 3:
+        assert _rows(copy_id, models.Checkpoint) == 0
+
+
+# ------------------------------------------------------ the head, era by era
+
+def test_a_pre_m9_file_still_opens_at_the_head_it_recorded(client, current):
+    """v2 carried the head, so it is honoured exactly as before."""
+    copy_id = bring_back(client, as_of(current, "pre-m9"))
+    with SessionLocal() as db:
+        assert db.get(models.Adventure, copy_id).head_depth == current["headDepth"]
+
+
+def test_a_pre_active_head_file_opens_at_its_tip(client, current):
+    """I07's compatibility clause. Not a degraded path.
+
+    Such a file was written when the head could not be anywhere but the tip, so
+    opening it there reproduces the position it recorded. An import that refused
+    it, or that guessed some other position, would be the failure.
+    """
+    copy_id = bring_back(client, as_of(current, "pre-active-head"))
+    with SessionLocal() as db:
+        adventure = db.get(models.Adventure, copy_id)
+        tip = max(
+            row.depth for row in
+            db.query(models.Action).filter(
+                models.Action.adventure_id == copy_id,
+                models.Action.branch_id == adventure.head_branch_id,
+            )
+        )
+        assert adventure.head_depth == tip
+        assert adventure.head_depth > current["headDepth"], (
+            "the fixture's head must really be behind its tip, or this proves nothing"
+        )
+
+
+def test_a_pre_active_head_file_offers_no_redo_because_it_is_at_the_tip(
+    client, current
+):
+    copy_id = bring_back(client, as_of(current, "pre-active-head"))
+    page = client.get(f"/api/adventures/{copy_id}").json()
+    assert page["can_redo"] is False
+    assert page["can_undo"] is True
+
+
+# ----------------------------------------------------- what each era can do
+
+def test_a_pre_m5_campaign_can_be_played_on_and_gains_state_from_there(
+    client, current
+):
+    """The M5 rule, applied to an import: no backfill, and no obstacle either.
+
+    An old campaign starts with an empty state because its narration was never
+    read by a state extractor. The next turn fills it in, which is what makes
+    "no backfill" a decision rather than a loss.
+    """
+    copy_id = bring_back(client, as_of(current, "pre-m5"))
+    assert client.get(f"/api/adventures/{copy_id}/state").json()["empty"] is True
+
+    ScriptedProvider.replies = [
+        "The door gives at last.\n" + __import__("fakes").state_block([
+            {"type": "add_fact", "predicate": "tally", "value": 500,
+             "fact_id": "tally-500"}
+        ])
+    ]
+    played = client.post(f"/api/adventures/{copy_id}/actions",
+                         json={"type": "do", "text": "push harder"})
+    assert played.status_code == 200, played.text[:300]
+    after = client.get(f"/api/adventures/{copy_id}/state").json()
+    assert after["empty"] is False
+    assert any(f["predicate"] == "tally" for f in after["document"]["facts"])
+
+
+def test_a_pre_m7_campaign_needs_no_source_and_can_import_one(client, current):
+    copy_id = bring_back(client, as_of(current, "pre-m7"))
+    assert client.get(f"/api/adventures/{copy_id}/knowledge").json() == []
+    # It plays without one.
+    assert client.get(f"/api/adventures/{copy_id}/context").status_code == 200
+    # And gains one.
+    landed = m9_fixture.upload(
+        client, copy_id, "canon.md", m9_fixture.CANON_MD, "canon",
+    )
+    library = client.get(f"/api/adventures/{copy_id}/knowledge").json()
+    assert [s["id"] for s in library] == [landed]
+    assert library[0]["index_state"] == "ready"
+
+
+def test_a_pre_save_point_campaign_can_be_given_one(client, current):
+    copy_id = bring_back(client, as_of(current, "pre-save-points"))
+    assert client.get(f"/api/adventures/{copy_id}/checkpoints").json() == []
+    made = client.post(f"/api/adventures/{copy_id}/checkpoints",
+                       json={"name": "From here", "note": ""})
+    assert made.status_code == 201, made.text[:300]
+    assert made.json()["resolved"] is True
+
+
+def test_a_pre_m9_campaign_re_exports_as_v3_without_gaining_evidence(
+    client, current
+):
+    """Re-exporting an old campaign does not turn absence into presence.
+
+    The file it writes is a v3 file, because that is what this build writes. Its
+    evidence sections are empty, because the campaign genuinely has none — and a
+    later reader can therefore trust a v3 file's empty `stateEvents` to mean
+    "this campaign has no audit trail" rather than "the file could not say".
+    """
+    copy_id = bring_back(client, as_of(current, "pre-m9"))
+    again = client.get(f"/api/adventures/{copy_id}/export").json()
+    assert again["format"] == bundle.FORMAT
+    assert again["stateEvents"] == []
+    assert again["stateProposals"] == []
+    assert again["summaries"] == []
+    assert not any(a.get("contextSnapshotZ") for a in again["actions"])
+    # And the story it does have survives a second round trip unchanged.
+    twice = bring_back(client, again)
+    assert _tree_size(client, twice) == _tree_size(client, copy_id)
+
+
+def test_a_v1_file_still_imports_and_reads_in_order(client):
+    """The flat format, with its retries as a repeating group."""
+    copy_id = bring_back(client, {
+        "format": bundle.LEGACY_FORMAT,
+        "title": "An old flat file",
+        "memory": "Kept from before the tree.",
+        "actions": [
+            {"index": 0, "type": "start", "text": "It begins."},
+            {"index": 1, "type": "do", "text": "look around"},
+            {"index": 2, "type": "ai", "text": "Take two.",
+             "variants": [{"text": "Take one."}, {"text": "Take two."}],
+             "variantIndex": 1},
+        ],
+    })
+    page = client.get(f"/api/adventures/{copy_id}").json()
+    assert [a["text"] for a in page["actions"]] == [
+        "It begins.", "look around", "Take two.",
+    ]
+    assert page["memory"] == "Kept from before the tree."
+    # Both attempts arrived; only one is the story.
+    with SessionLocal() as db:
+        rows = db.query(models.Action).filter(
+            models.Action.adventure_id == copy_id, models.Action.type == "ai",
+        ).all()
+    assert sorted(r.text for r in rows) == ["Take one.", "Take two."]
+    assert sum(1 for r in rows if r.live) == 1
+
+
+def test_a_pre_m2_file_with_scripting_still_imports(client, current):
+    """M2 removed campaign scripting. Its keys are ignored, not rejected.
+
+    The story, the tree and everything else in such a file are still worth
+    importing, and refusing the campaign over a subsystem that no longer exists
+    would lose all of it to reject one key.
+    """
+    payload = as_of(current, "pre-m5")
+    payload["scripts"] = [{"name": "onTurn", "code": "state.gold += 10"}]
+    payload["scriptState"] = {"gold": 70}
+    copy_id = bring_back(client, payload)
+    assert _tree_size(client, copy_id) == len(current["actions"])