From a7bf47a35e77c936f8f357cbdb61fa42bf4a7165 Mon Sep 17 00:00:00 2001 From: parththakkar106 Date: Tue, 18 Aug 2026 12:28:49 +0530 Subject: [PATCH] Let a backup carry a story that went two ways MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A bundle had one list and a forked adventure has two stories, so export was emitting every branch's turns interleaved by index — a mangled story rather than lost data, and unreachable only because forking has no UI yet. `ai-dnd-adventure-v2` carries the branches, the depth each one left its parent at, which attempt at every turn is the story, and what each node left behind. That last one is not decoration: the after-snapshots are what a branch switch puts back, and a bundle without them imports a tree nobody can switch inside. `app/bundle.py` owns both formats and nothing else knows either. The v1 reader stays — those files are already on people's disks — and it is now the only place a `variants` array exists anywhere. The rule the module is built on is that a bundle carries what was chosen and never what is derived. The head branch, the fork points, the live flags and the anchors are decisions somebody made. The lineage, the head depth, the legacy `index` and the variant ordinals are computed from those and are rebuilt on the way in, because a bundle is a text file anybody can edit and a derived field shipped beside its source is a chance for the file to disagree with itself where no read would report it. `index` is the one that stops being academic here. It agreed with `depth` until SP5, and this is the first writer that has to fill it for a forked story, where two branches both hold a node at depth 4. It is allocated one per turn instead: siblings share it, no two coordinates do. Everything a hand-edited file can get wrong about the shape of a tree is a 400 raised before the adventure row exists, because a half-applied import is exactly the failure this phase exists to end — a story that goes quiet. A file wrong about which attempt is live is corrected rather than refused; that is an invariant of the database, not of the format. Measured on the 600-action fixture: 587 kB to 911 kB, and all of the increase is the outcomes at 489 B a node — the coordinates themselves save 57.5 B a node against the old turn-and-variants shape. Twenty forks add 660 B. 4.3% of the import body cap. 381 tests green, 16 of them new in test_bundle_v2.py. No migration, no vacuum owed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015H5qiyiR7gtFQaoDphHZ3g --- backend/app/bundle.py | 609 ++++++++++++++++++++++ backend/app/context/cursors.py | 17 +- backend/app/limits.py | 8 +- backend/app/routers/adventures.py | 206 ++------ backend/tests/test_attempt_siblings.py | 17 +- backend/tests/test_bundle_v2.py | 539 +++++++++++++++++++ backend/tests/test_retry_variants.py | 9 +- backend/tests/test_story_tree_baseline.py | 15 +- backend/tools/measure_bundle.py | 149 ++++++ plan/14-phase-story-tree.md | 88 +++- plan/STATUS.md | 90 +++- 11 files changed, 1532 insertions(+), 215 deletions(-) create mode 100644 backend/app/bundle.py create mode 100644 backend/tests/test_bundle_v2.py create mode 100644 backend/tools/measure_bundle.py diff --git a/backend/app/bundle.py b/backend/app/bundle.py new file mode 100644 index 0000000..78853e0 --- /dev/null +++ b/backend/app/bundle.py @@ -0,0 +1,609 @@ +"""Phase 14, SP6 — the export bundle, as a tree. + +A bundle is the one place a story leaves the database, and the only part of the +tree no migration can ever reach: a file downloaded today has to still import +into a build shipped next year. So the format is versioned, both versions live +here, and nothing else in the app knows either of them. + +**v1** is a flat list of turns, each with an optional `variants` array — the +repeating group SP4 unpacked into rows. Nothing writes that shape any more. +The *reader* stays, because bundles already on people's disks still have it and +a backup that stops importing is not a backup. + +**v2** carries the tree. Three things it holds that v1 could not, each +load-bearing: + +* **the branches**, because a forked adventure is two stories and a flat list + can hold one — v1 export interleaved them by `index`, which read as a mangled + story rather than as lost data; +* **`live`**, because a coordinate can hold several attempts at one turn and + exactly one of them is the story; +* **both after-snapshots**, because they are what a branch switch and an undo + put back. A bundle carrying the actions but not the outcomes would import a + tree nobody could switch inside. + +## The rule about what a bundle carries + +**What was chosen, never what is derived.** The head *branch*, the fork points, +the live flags and the anchors are decisions somebody made; they are in the +file. `lineage`, the head *depth*, `index` and the variant ordinals are all +computed from those, and they are recomputed on import instead: + +* `lineage` is a cache of `parent` + `fork_depth`. Shipping it too would put a + second source of truth for one fact in a file anybody can hand-edit, and the + two could then disagree in a way no read would ever report. +* the head depth is the tip of the head branch, which is a fact about the nodes + that arrived with it. +* `index` is the legacy column SP8 drops. Its one remaining job is to hand the + next row a number nothing else holds — a fact about the *adventure*, not + about a path — so `depth` cannot be it: two branches have a node at depth 4. + The import allocates one per turn instead, which keeps `max_action_index` + honest and keeps siblings sharing an index the way SP4 leaves them. +* the variant ordinals are `attempts.renumber`'s to maintain, and it is the + only place allowed to. + +Every hand-editable coordinate is therefore checked before a row is written +(`plan`), not fixed up afterwards: an import that fails halfway leaves an +adventure holding half a tree, and a tree missing a branch is a story that +silently stops rather than one that reports. +""" + +from datetime import datetime + +from fastapi import HTTPException +from sqlalchemy import insert, update +from sqlalchemy.orm import Session, undefer + +from . import attempts, models +from .context import cursors, lineage + +FORMAT = "ai-dnd-adventure-v2" +LEGACY_FORMAT = "ai-dnd-adventure-v1" + +# VARCHAR(20) on `actions.type`; a raw-dict import bypasses the schemas. +TYPE_MAX = 20 + + +# ---------------------------------------------------------------- exporting + +def export(db: Session, adventure: models.Adventure) -> dict: + """The whole adventure as a v2 bundle. + + Deliberately un-pathed: a backup wants the entire tree, not the branch its + owner happens to be standing on. Both after-snapshots are undeferred in the + one query — they are per-node columns nothing else reads in bulk, and asking + for them a row at a time would be a query per turn. + + No context snapshots. A bundle has never carried the assembled prompts and + still does not: they are ~163 kB a turn, they are an explanation of a + generation rather than part of the story, and the Insights viewer they feed + is reading the adventure it came from. + """ + branches = ( + db.query(models.Branch) + .filter(models.Branch.adventure_id == adventure.id) + .order_by(models.Branch.id) + .all() + ) + # Branch ids are local to the file — positions in this list — because the + # database ids they had here are already taken over there. + local = {branch.id: i for i, branch in enumerate(branches)} + nodes = ( + db.query(models.Action) + .filter(models.Action.adventure_id == adventure.id) + .options( + undefer(models.Action.state_after), + undefer(models.Action.world_state_after), + ) + .order_by( + models.Action.branch_id, models.Action.depth, + models.Action.variant_index, models.Action.id, + ) + .all() + ) + return { + "format": FORMAT, + "title": adventure.title, + "memory": adventure.memory, + "authorsNote": adventure.authors_note, + "aiInstructions": adventure.ai_instructions, + "storySummary": adventure.story_summary, + "scriptState": adventure.script_state, + "worldState": adventure.world_state, + "autoSummarize": adventure.auto_summarize, + "memoryBankEnabled": adventure.memory_bank_enabled, + # A root entry even for an adventure whose branch row was never created + # — a story with no branch is a pre-tree one, and the tree it belongs to + # is the root. `_local` puts its nodes there. + "branches": [_exported_branch(b, local) for b in branches] or [_ROOT], + "headBranch": local.get(adventure.head_branch_id, 0), + "memoryCursor": _exported_anchor(adventure, cursors.MEMORY, local), + "summaryCursor": _exported_anchor(adventure, cursors.SUMMARY, local), + "memories": [_exported_memory(m, local) for m in adventure.memories], + "storyCards": [ + {"type": c.type, "name": c.name, "keys": c.keys, + "entry": c.entry, "notes": c.notes} + for c in adventure.story_cards + ], + "scripts": [ + { + "position": s.position, "enabled": s.enabled, + "name": s.name, "description": s.description, + "library": s.library_js, "input": s.input_js, + "context": s.context_js, "output": s.output_js, + } + for s in adventure.scripts + ], + "actions": [_exported_node(a, local) for a in nodes], + } + + +_ROOT = {"parent": None, "forkDepth": None} + + +def _local(branch_id: int | None, local: dict[int, int]) -> int: + return local.get(branch_id, 0) if branch_id is not None else 0 + + +def _exported_branch(branch: models.Branch, local: dict[int, int]) -> dict: + parent = ( + local.get(branch.parent_branch_id) + if branch.parent_branch_id is not None else None + ) + if parent is None: + return dict(_ROOT) + return {"parent": parent, "forkDepth": branch.fork_depth} + + +def _exported_node(action: models.Action, local: dict[int, int]) -> dict: + node = { + "branch": _local(action.branch_id, local), + # A pre-tree row's depth is the number `index` already held. + "depth": action.depth if action.depth is not None else action.index, + "live": bool(action.live), + "type": action.type, + "text": action.text, + "createdAt": action.created_at.isoformat() if action.created_at else None, + } + if action.reasoning: + node["reasoning"] = action.reasoning + # `{}` and absent mean different things — "this node left an empty + # scoreboard behind" against "nobody knows, leave the live state alone" — + # so an empty snapshot is written out rather than trimmed. It costs about + # eighteen bytes a row and it is the difference between an undo that clears + # a score and one that leaves it standing. + if action.state_after is not None: + node["stateAfter"] = action.state_after + if action.world_state_after is not None: + node["worldStateAfter"] = action.world_state_after + if action.world_delta: + node["worldDelta"] = action.world_delta + return node + + +def _exported_memory(memory: models.Memory, local: dict[int, int]) -> dict: + return { + "text": memory.text, "pinned": memory.pinned, "forgotten": memory.forgotten, + "sourceStart": memory.source_start, "sourceEnd": memory.source_end, + "useCount": memory.use_count, + # The node it hangs off. A hand-written memory summarises no node, so it + # has a branch and no depth, and keeps that shape here. + "branch": _local(memory.branch_id, local), + "depth": memory.depth, + } + + +def _exported_anchor( + adventure: models.Adventure, cursor: cursors.Cursor, local: dict[int, int] +) -> dict: + branch_id, depth = cursor.stored(adventure) + return { + "branch": local.get(branch_id) if branch_id is not None else None, + "depth": depth, + } + + +# ---------------------------------------------------------------- importing + +def check_format(bundle: dict) -> str: + """The bundle's version, or 400.""" + fmt = bundle.get("format") + if fmt in (FORMAT, LEGACY_FORMAT): + return fmt + raise HTTPException( + 400, + f"Not an adventure export file (expected format {FORMAT} or {LEGACY_FORMAT}).", + ) + + +def plan(bundle: dict, version: str) -> dict: + """The bundle's tree, checked and normalised, before a row is written. + + Pure: no session, no adventure, nothing created. Everything a hand-edited + file can get wrong about the *shape* of a tree is caught here, because the + alternative is an import that fails partway and leaves an adventure holding + a story with a hole in it. + + Both versions land in the same shape, so `write` never learns there are two + formats: a v1 bundle is a linear story, which is a tree with one branch, and + its `variants` array is a sibling group written the old way. + """ + branches = ( + _planned_branches(bundle) if version == FORMAT else [dict(_ROOT)] + ) + nodes = ( + _planned_nodes(bundle, len(branches)) if version == FORMAT + else _planned_v1_nodes(bundle) + ) + return { + "branches": branches, + "nodes": nodes, + "memories": _planned_memories(bundle, len(branches)), + "head": _as_index(bundle.get("headBranch"), len(branches), default=0), + # v2 knows where the derived work got to; v1 counted it, and a count + # cannot be turned into a node until the nodes exist (see `settle`). + "anchors": _planned_anchors(bundle, len(branches)) if version == FORMAT else None, + "positions": None if version == FORMAT else { + "memory": _as_int(bundle.get("memoryCursor"), 0), + "summary": _as_int(bundle.get("summaryCursor"), 0), + }, + } + + +def _planned_branches(bundle: dict) -> list[dict]: + raw = bundle.get("branches") + entries = [b for b in raw if isinstance(b, dict)] if isinstance(raw, list) else [] + if not entries: + return [dict(_ROOT)] + specs: list[dict] = [] + for i, entry in enumerate(entries): + parent = entry.get("parent") + if parent is None: + specs.append(dict(_ROOT)) + continue + # A branch may only fork from one listed before it. That is how the + # export writes them — branches are numbered in creation order and a + # parent always exists first — and requiring it here buys acyclicity for + # the price of a comparison: a lineage is computed by walking to the + # parent, and a cycle would be an import that never returns. + if not _is_int(parent) or not 0 <= parent < i: + raise HTTPException( + 400, + f"Branch {i} forks from branch {parent!r}, which is not one of " + f"the {i} branches listed before it.", + ) + fork_depth = entry.get("forkDepth") + if not _is_int(fork_depth): + raise HTTPException( + 400, + f"Branch {i} forks from branch {parent} but does not say at " + f"what depth.", + ) + specs.append({"parent": parent, "forkDepth": fork_depth}) + return specs + + +def _planned_nodes(bundle: dict, branches: int) -> list[dict]: + raw = bundle.get("actions") + nodes: list[dict] = [] + for entry in raw if isinstance(raw, list) else []: + if not isinstance(entry, dict) or not str(entry.get("text") or ""): + continue + branch = entry.get("branch", 0) + if not _is_int(branch) or not 0 <= branch < branches: + raise HTTPException( + 400, + f"An action names branch {branch!r}, but the file lists " + f"{branches}.", + ) + depth = entry.get("depth") + if not _is_int(depth) or depth < 0: + raise HTTPException( + 400, f"An action on branch {branch} has no depth to sit at." + ) + nodes.append({ + "branch": branch, + "depth": depth, + "live": bool(entry.get("live", True)), + "type": str(entry.get("type") or "story")[:TYPE_MAX], + "text": str(entry.get("text") or ""), + "reasoning": _as_text(entry.get("reasoning")), + "stateAfter": _as_dict(entry.get("stateAfter")), + "worldStateAfter": _as_dict(entry.get("worldStateAfter")), + "worldDelta": _as_dict(entry.get("worldDelta")), + "createdAt": _as_time(entry.get("createdAt")), + }) + return nodes + + +def _planned_v1_nodes(bundle: dict) -> list[dict]: + """A v1 bundle's turns as the nodes they describe: one per attempt. + + The `variants` array is the repeating group SP4 unpacked, so reading one is + the same split migration 60 does — every attempt becomes a row at the turn's + coordinate and `variantIndex` picks which is live. Clamped, because a + hand-edited bundle can name an attempt its own list does not have, and a + turn with no live node is a turn no read can see. + + The depth is the bundle's `index`: v1 is one branch, where the two agree. + """ + raw = bundle.get("actions") + nodes: list[dict] = [] + for i, entry in enumerate(raw if isinstance(raw, list) else []): + if not isinstance(entry, dict) or not str(entry.get("text") or ""): + continue + depth = _as_int(entry.get("index"), i) + kind = str(entry.get("type") or "story")[:TYPE_MAX] + variants = [v for v in (entry.get("variants") or []) if isinstance(v, dict)] + if not variants: + variants = [{"text": entry["text"], "reasoning": entry.get("reasoning")}] + live = min(max(_as_int(entry.get("variantIndex"), 0), 0), len(variants) - 1) + for n, variant in enumerate(variants): + text = str(variant.get("text") or "") + if not text: + continue + nodes.append({ + "branch": 0, + "depth": max(depth, 0), + "live": n == live, + "type": kind, + "text": text, + "reasoning": _as_text(variant.get("reasoning")), + "stateAfter": None, + "worldStateAfter": None, + "worldDelta": None, + "createdAt": _as_time(variant.get("createdAt")), + }) + return nodes + + +def _planned_memories(bundle: dict, branches: int) -> list[dict]: + raw = bundle.get("memories") + out: list[dict] = [] + for entry in raw if isinstance(raw, list) else []: + if not isinstance(entry, dict) or not str(entry.get("text") or "").strip(): + continue + out.append({ + "text": str(entry["text"]), + "pinned": bool(entry.get("pinned", False)), + "forgotten": bool(entry.get("forgotten", False)), + "sourceStart": entry.get("sourceStart"), + "sourceEnd": entry.get("sourceEnd"), + "useCount": _as_int(entry.get("useCount"), 0), + # Out of range rather than absent means a file that disagrees with + # itself; the root is the safe reading, because a memory on a branch + # nothing can see is a memory that never reaches a prompt again. + "branch": _as_index(entry.get("branch"), branches, default=0), + "depth": entry.get("depth") if _is_int(entry.get("depth")) else None, + }) + return out + + +def _planned_anchors(bundle: dict, branches: int) -> dict: + anchors = {} + for name in ("memory", "summary"): + raw = bundle.get(f"{name}Cursor") + raw = raw if isinstance(raw, dict) else {} + branch = raw.get("branch") + anchors[name] = ( + _as_index(branch, branches) if branch is not None else None, + _as_int(raw.get("depth"), lineage.NO_DEPTH), + ) + return anchors + + +# ------------------------------------------------------------------ writing + +def write(db: Session, adventure: models.Adventure, story: dict) -> None: + """Write a planned tree onto a freshly created adventure. + + Order matters and is not negotiable: branches first, because a node needs an + id to hang off; then the nodes, because the head and the anchors name one. + """ + ids = _write_branches(db, adventure, story["branches"]) + _write_nodes(db, adventure, story["nodes"], ids) + _write_memories(db, adventure, story["memories"], ids) + _point_the_head(adventure, story, ids) + _write_anchors(adventure, story, ids) + + +def _write_branches( + db: Session, adventure: models.Adventure, specs: list[dict] +) -> list[int]: + """One row per branch, lineage computed rather than read. + + Inserted through Core and the lineage written second, for the reason + `tree.root_branch` spells out: the lineage names the row's own id. The + parent's cached ancestry is capped at this fork, which is the same + arithmetic `tree.fork` does — a fork made now and a fork made a year ago and + exported must produce the same rows. + """ + ids: list[int] = [] + lineages: list[list[list]] = [] + for spec in specs: + parent = spec.get("parent") + fork_depth = spec.get("forkDepth") + new_id = db.execute( + insert(models.Branch).values( + adventure_id=adventure.id, + parent_branch_id=ids[parent] if parent is not None else None, + fork_depth=fork_depth if parent is not None else None, + lineage=[], + created_at=models.utcnow(), + ) + ).inserted_primary_key[0] + entries = [[new_id, None]] + if parent is not None: + entries += [ + [branch_id, fork_depth if cap is None else min(cap, fork_depth)] + for branch_id, cap in lineages[parent] + ] + db.execute( + update(models.Branch) + .where(models.Branch.id == new_id) + .values(lineage=entries) + ) + ids.append(new_id) + lineages.append(entries) + return ids + + +def _write_nodes( + db: Session, adventure: models.Adventure, specs: list[dict], ids: list[int] +) -> None: + """The nodes, grouped into the turns they are attempts at. + + Two things are allocated here rather than trusted from the file. `index` is + handed out one per *turn*, in the order the bundle lists them, so siblings + share one and no two coordinates do — which is what `max_action_index` needs + to keep issuing numbers nothing holds. And exactly one attempt in each group + is made live: a file can name none or several, and a turn with no live node + is a turn that vanishes from the story. + """ + groups: dict[tuple[int, int], list[models.Action]] = {} + indices: dict[tuple[int, int], int] = {} + for spec in specs: + key = (spec["branch"], spec["depth"]) + if key not in indices: + indices[key] = len(indices) + action = models.Action( + adventure_id=adventure.id, + branch_id=ids[spec["branch"]], + depth=spec["depth"], + index=indices[key], + type=spec["type"], + text=spec["text"], + reasoning=spec["reasoning"], + live=spec["live"], + state_after=spec["stateAfter"], + world_state_after=spec["worldStateAfter"], + world_delta=spec["worldDelta"], + ) + if spec["createdAt"] is not None: + action.created_at = spec["createdAt"] + db.add(action) + groups.setdefault(key, []).append(action) + + for rows in groups.values(): + live = next((row for row in rows if row.live), rows[0]) + for row in rows: + row.live = row is live + attempts.renumber(rows) + + +def _write_memories( + db: Session, adventure: models.Adventure, specs: list[dict], ids: list[int] +) -> None: + for spec in specs: + memory = models.Memory( + adventure_id=adventure.id, + text=spec["text"], + pinned=spec["pinned"], + forgotten=spec["forgotten"], + source_start=spec["sourceStart"], + source_end=spec["sourceEnd"], + use_count=spec["useCount"], + branch_id=ids[spec["branch"]], + depth=spec["depth"], + ) + # A v1 memory has no depth of its own; `source_end` is the index of the + # last action it summarises, which on one branch is that node's depth. + if memory.depth is None and memory.source_end is not None: + memory.depth = memory.source_end + db.add(memory) + + +def _point_the_head( + adventure: models.Adventure, story: dict, ids: list[int] +) -> None: + """Where the story is being played, and how deep it goes. + + The branch comes from the file and the depth does not: the tip of a branch + is whatever arrived on it, and a branch with nothing of its own sits at its + fork point — the last node its story contains, borrowed but the tip all the + same. The same rule as `tree.refresh_head`, applied before a flush. + """ + head = story["head"] + adventure.head_branch_id = ids[head] + depths = [n["depth"] for n in story["nodes"] if n["branch"] == head] + if depths: + adventure.head_depth = max(depths) + return + fork_depth = story["branches"][head].get("forkDepth") + adventure.head_depth = fork_depth if fork_depth is not None else lineage.NO_DEPTH + + +def _write_anchors( + adventure: models.Adventure, story: dict, ids: list[int] +) -> None: + """How far the memories and the summary have read — v2 only. + + A v1 bundle counts instead, and a count cannot be resolved to a node until + the nodes are in the database; `settle` does that half afterwards. + """ + anchors = story["anchors"] + if anchors is None: + return + for cursor in cursors.ALL: + branch, depth = anchors[cursor.name] + cursor.anchor(adventure, ids[branch] if branch is not None else None, depth) + + +def settle(db: Session, adventure: models.Adventure, story: dict) -> None: + """Line up the two coordinate systems, once the nodes exist. + + The anchors and the legacy counts describe the same boundary in different + words, and each version of the bundle brings one of them. A v1 file brings + the count, so the anchor is found by counting that far along the story; a v2 + file brings the anchor, so the count is read back off it. The legacy columns + are still written either way — they are what a rolled-back build reads. + + Called after the flush, because both directions need the actions queryable. + """ + positions = story["positions"] + for cursor in cursors.ALL: + if positions is not None: + setattr(adventure, f"{cursor.name}_cursor", positions[cursor.name]) + cursors.anchor_at_position(adventure, cursor, positions[cursor.name]) + else: + setattr( + adventure, f"{cursor.name}_cursor", + cursors.position_of(adventure, cursor.depth(db, adventure)), + ) + + +# ------------------------------------------------------------------ reading +# Small coercions. A raw-dict import bypasses the schemas entirely, so +# everything out of a bundle is whatever JSON happened to hold. + +def _is_int(value) -> bool: + """`True` is an `int` in Python, and is not one in a coordinate.""" + return isinstance(value, int) and not isinstance(value, bool) + + +def _as_int(value, default: int) -> int: + return value if _is_int(value) else default + + +def _as_index(value, count: int, default: int | None = None) -> int | None: + """A local branch number, or `default` when the file names one that is not + there. Out of range is a file disagreeing with itself, not a shape a read + can be handed.""" + return value if _is_int(value) and 0 <= value < count else default + + +def _as_text(value) -> str | None: + return str(value) if value else None + + +def _as_dict(value) -> dict | None: + return value if isinstance(value, dict) else None + + +def _as_time(value) -> datetime | None: + if not isinstance(value, str): + return None + try: + return datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError: + return None diff --git a/backend/app/context/cursors.py b/backend/app/context/cursors.py index 2e4eb30..35032f9 100644 --- a/backend/app/context/cursors.py +++ b/backend/app/context/cursors.py @@ -66,6 +66,18 @@ class Cursor: # ------------------------------------------------------------- writing + def anchor( + self, adventure: models.Adventure, branch_id: int | None, depth: int + ) -> None: + """Put the anchor at a coordinate given outright. + + The plain setter under `anchor_at`. Only an import has a coordinate + without a node to read it off — a v2 bundle carries the anchor itself + (`app/bundle.py`), and the node it named lives in another database. + """ + setattr(adventure, self.branch_field, branch_id) + setattr(adventure, self.depth_field, max(depth, NO_DEPTH)) + def anchor_at(self, adventure: models.Adventure, node: models.Action) -> None: """Mark the work done up to and including `node`. @@ -73,9 +85,8 @@ class Cursor: actions can end before the fork this branch was made at, and the coverage belongs where the ground is. """ - setattr(adventure, self.branch_field, node.branch_id) - setattr(adventure, self.depth_field, lineage.NO_DEPTH - if node.depth is None else node.depth) + self.anchor(adventure, node.branch_id, lineage.NO_DEPTH + if node.depth is None else node.depth) def rewind_to( self, adventure: models.Adventure, branch_id: int | None, depth: int diff --git a/backend/app/limits.py b/backend/app/limits.py index 52d3a27..4f48b49 100644 --- a/backend/app/limits.py +++ b/backend/app/limits.py @@ -158,6 +158,11 @@ MAX_SCRIPTS_PER_USER = 200 MAX_STORY_CARDS_PER_OWNER = 200 # per scenario or adventure MAX_MEMORIES_PER_ADVENTURE = 1000 MAX_ACTIONS_PER_ADVENTURE = 5000 +# Phase 14, SP6. A branch per divergence somebody built a story on, so a tree +# with more of them than a story has turns is a file, not a game. Import-only +# for now: forking is a POST that adds one row and has no cap of its own, and +# the cap that matters there is `MAX_ACTIONS_PER_ADVENTURE` above it. +MAX_BRANCHES_PER_ADVENTURE = 1000 def check_row_cap( @@ -232,12 +237,13 @@ _BUNDLE_LIST_CAPS = { "story_cards": MAX_STORY_CARDS_PER_OWNER, "memories": MAX_MEMORIES_PER_ADVENTURE, "actions": MAX_ACTIONS_PER_ADVENTURE, + "branches": MAX_BRANCHES_PER_ADVENTURE, } def check_bundle_lists(**lists) -> None: """409 when an import bundle's lists exceed the same caps live creation - enforces (kwargs: story_cards=, memories=, actions=).""" + enforces (kwargs: story_cards=, memories=, actions=, branches=).""" if not auth.MULTI_USER: return for name, value in lists.items(): diff --git a/backend/app/routers/adventures.py b/backend/app/routers/adventures.py index ba24292..6086f5b 100644 --- a/backend/app/routers/adventures.py +++ b/backend/app/routers/adventures.py @@ -9,7 +9,8 @@ from sqlalchemy.orm import Session, load_only, undefer from sqlalchemy.orm.attributes import set_committed_value from .. import ( - attempts, auth, images, limits, memorybank, models, schemas, tree, worldstate, + attempts, auth, bundle, images, limits, memorybank, models, schemas, tree, + worldstate, ) from ..context import build_context, cursors from ..context import history as context_history @@ -1303,185 +1304,55 @@ def undo_turn( def export_adventure( adventure_id: int, db: Session = Depends(get_db), user: models.User = CurrentUser ): - """Full backup: plot components, story cards, scripts (+state), every action.""" + """Full backup: plot components, story cards, scripts (+state), the tree. + + The format is `app/bundle.py`'s alone — both versions of it. A backup is the + one thing here that outlives the schema, so nothing about its shape is + decided at a call site. + """ adv = get_adventure_or_404(adventure_id, db, user) - # The one read deliberately left un-pathed: a backup wants the whole - # adventure, not the branch its owner happens to be standing on. `index` - # orders it because the v1 bundle is a flat list keyed on index and its - # reader has no idea branches exist — which is exactly why SP6 replaces the - # format rather than quietly widening this query. - # - # Attempts at one turn share that index (SP4), so the flat list is built by - # folding each group back into the `variants` array the format expects. - # That array is the *only* remaining producer of the v1 shape: nothing in - # the database holds one any more. - # - # A *forked* adventure has no honest v1 rendering — the format has one - # story and there are two — so this emits every branch's turns interleaved - # by index, which reads as a mangled story rather than as lost data. SP6's - # v2 bundle is what fixes it, and SP7 is where a player first gets a way to - # fork at all, so the order those two ship in is the order that matters. - exported_actions = ( - db.query(models.Action) - .filter(models.Action.adventure_id == adv.id) - .order_by(models.Action.index, models.Action.variant_index, models.Action.id) - .all() - ) - turns: list[list[models.Action]] = [] - for action in exported_actions: - if turns and turns[-1][0].index == action.index: - turns[-1].append(action) - else: - turns.append([action]) - return { - "format": "ai-dnd-adventure-v1", - "title": adv.title, - "memory": adv.memory, - "authorsNote": adv.authors_note, - "aiInstructions": adv.ai_instructions, - "storySummary": adv.story_summary, - "scriptState": adv.script_state, - "worldState": adv.world_state, - "autoSummarize": adv.auto_summarize, - "memoryBankEnabled": adv.memory_bank_enabled, - # The bundle's coordinate system is a position in the story, and the - # cursors are nodes now, so they are counted back into one. A v1 bundle - # has to stay readable by builds that never heard of a depth — SP6's v2 - # format carries the anchors themselves. - "memoryCursor": cursors.position_of(adv, cursors.MEMORY.depth(db, adv)), - "summaryCursor": cursors.position_of(adv, cursors.SUMMARY.depth(db, adv)), - "memories": [ - { - "text": m.text, "pinned": m.pinned, "forgotten": m.forgotten, - "sourceStart": m.source_start, "sourceEnd": m.source_end, - "useCount": m.use_count, - } - for m in adv.memories - ], - "storyCards": [ - {"type": c.type, "name": c.name, "keys": c.keys, "entry": c.entry, "notes": c.notes} - for c in adv.story_cards - ], - "scripts": [ - { - "position": s.position, "enabled": s.enabled, - "name": s.name, "description": s.description, - "library": s.library_js, "input": s.input_js, - "context": s.context_js, "output": s.output_js, - } - for s in adv.scripts - ], - "actions": [_exported_turn(group) for group in turns], - } - - -def _exported_turn(group: list[models.Action]) -> dict: - """One turn as a v1 bundle entry: the attempt in the story, plus the rest. - - Narration only — a bundle carries no context snapshots, so the per-attempt - script/world state a switch would restore isn't there to export either. - """ - live = next((a for a in group if a.live), group[0]) - return { - "index": live.index, "type": live.type, "text": live.text, - "reasoning": live.reasoning, - "variants": [ - {"text": a.text, "reasoning": a.reasoning, - "createdAt": a.created_at.isoformat() if a.created_at else None} - for a in group - ] if len(group) > 1 else None, - "variantIndex": group.index(live), - "createdAt": live.created_at.isoformat(), - } - - -def _imported_turn( - adventure: models.Adventure, entry: dict, index: int -) -> list[models.Action]: - """A v1 bundle entry as the nodes it describes: one per attempt. - - A bundle's `variants` array is the repeating group SP4 unpacked, so - importing one is the same split the migration does — every attempt gets a - row at the turn's coordinate, and `variantIndex` picks which is live. - Clamped, because a hand-edited bundle can name an attempt its own list - doesn't have, and a turn with no live node is a turn no read can see. - """ - kind = str(entry.get("type") or "story")[:20] # VARCHAR(20) - variants = [v for v in (entry.get("variants") or []) if isinstance(v, dict)] - if not variants: - variants = [{"text": entry["text"], "reasoning": entry.get("reasoning")}] - live = min(max(int(entry.get("variantIndex", 0)), 0), len(variants) - 1) - rows = [] - for i, variant in enumerate(variants): - text = str(variant.get("text") or "") - reasoning = variant.get("reasoning") - rows.append(models.Action( - adventure_id=adventure.id, - index=index, - type=kind, - text=text, - reasoning=str(reasoning) if reasoning else None, - live=(i == live), - variant_index=i, - variant_count=len(variants) if len(variants) > 1 else 0, - )) - return rows + return bundle.export(db, adv) @router.post("/import", response_model=schemas.AdventureOut, status_code=201) def import_adventure( request: Request, - bundle: dict = Body(...), + payload: dict = Body(...), db: Session = Depends(get_db), user: models.User = CurrentUser, ): - if bundle.get("format") != "ai-dnd-adventure-v1": - raise HTTPException(400, "Not an adventure export file (expected format ai-dnd-adventure-v1).") + version = bundle.check_format(payload) limits.rate_limit("import", request, user) limits.check_row_cap("adventures", db, user) limits.check_bundle_lists( - story_cards=bundle.get("storyCards"), - memories=bundle.get("memories"), - actions=bundle.get("actions"), + story_cards=payload.get("storyCards"), + memories=payload.get("memories"), + actions=payload.get("actions"), + branches=payload.get("branches"), ) + # The tree is checked before the adventure row exists, so a file that + # disagrees with itself is a 400 and not a half-imported adventure holding a + # story with a hole in it. + story = bundle.plan(payload, version) # Raw-dict import bypasses the schemas — clamp strings headed for VARCHAR # columns (Postgres enforces the widths; see schemas.py). adventure = models.Adventure( user_id=user.id, - title=str(bundle.get("title") or "Imported Adventure")[:schemas.NAME_MAX], - memory=str(bundle.get("memory") or ""), - authors_note=str(bundle.get("authorsNote") or ""), - ai_instructions=str(bundle.get("aiInstructions") or ""), - story_summary=str(bundle.get("storySummary") or ""), - script_state=bundle.get("scriptState") or {}, - world_state=bundle.get("worldState") or {}, - auto_summarize=bool(bundle.get("autoSummarize", False)), - memory_bank_enabled=bool(bundle.get("memoryBankEnabled", False)), - memory_cursor=int(bundle.get("memoryCursor", 0)), - summary_cursor=int(bundle.get("summaryCursor", 0)), + title=str(payload.get("title") or "Imported Adventure")[:schemas.NAME_MAX], + memory=str(payload.get("memory") or ""), + authors_note=str(payload.get("authorsNote") or ""), + ai_instructions=str(payload.get("aiInstructions") or ""), + story_summary=str(payload.get("storySummary") or ""), + script_state=payload.get("scriptState") or {}, + world_state=payload.get("worldState") or {}, + auto_summarize=bool(payload.get("autoSummarize", False)), + memory_bank_enabled=bool(payload.get("memoryBankEnabled", False)), ) db.add(adventure) db.flush() - # A v1 bundle is a linear story, which is a tree with one branch. SP6's v2 - # format carries the branches themselves. - tree.head_branch(db, adventure) - for m in bundle.get("memories") or []: - if isinstance(m, dict) and str(m.get("text") or "").strip(): - memory = models.Memory( - adventure_id=adventure.id, - text=str(m["text"]), - pinned=bool(m.get("pinned", False)), - forgotten=bool(m.get("forgotten", False)), - source_start=m.get("sourceStart"), - source_end=m.get("sourceEnd"), - use_count=int(m.get("useCount", 0)), - ) - tree.place_memory(db, adventure, memory) - db.add(memory) - - for card in bundle.get("storyCards") or []: + for card in payload.get("storyCards") or []: if isinstance(card, dict): db.add(models.StoryCard( adventure_id=adventure.id, @@ -1492,7 +1363,7 @@ def import_adventure( notes=str(card.get("notes") or ""), )) - for i, s in enumerate(bundle.get("scripts") or []): + for i, s in enumerate(payload.get("scripts") or []): if isinstance(s, dict): db.add(models.AdventureScript( adventure_id=adventure.id, @@ -1506,21 +1377,14 @@ def import_adventure( output_js=str(s.get("output") or ""), )) - for i, a in enumerate(bundle.get("actions") or []): - if isinstance(a, dict) and str(a.get("text") or ""): - for action in _imported_turn(adventure, a, int(a.get("index", i))): - tree.place_action(db, adventure, action) - db.add(action) + bundle.write(db, adventure, story) - # The bundle's cursors are positions in a flat story and the marks are - # nodes, so the translation waits until the actions exist — this is the - # only moment the two coordinate systems can be lined up against each - # other. The legacy columns keep the numbers the bundle gave: they are what - # a rolled-back build would read. + # The anchors and the legacy counts are the same boundary in two coordinate + # systems, and lining them up needs the actions queryable — this is the only + # moment both exist. db.flush() db.expire(adventure, ["actions"]) - cursors.anchor_at_position(adventure, cursors.MEMORY, adventure.memory_cursor) - cursors.anchor_at_position(adventure, cursors.SUMMARY, adventure.summary_cursor) + bundle.settle(db, adventure, story) db.commit() db.refresh(adventure) diff --git a/backend/tests/test_attempt_siblings.py b/backend/tests/test_attempt_siblings.py index 4df9586..41fa3a7 100644 --- a/backend/tests/test_attempt_siblings.py +++ b/backend/tests/test_attempt_siblings.py @@ -380,18 +380,25 @@ def test_attempts_module_agrees_with_the_endpoint(client): db.close() -def test_export_folds_the_group_back_into_one_v1_entry(client): +def test_export_carries_every_attempt_as_its_own_node(client): + """SP6 changed the answer here, and the reason is the whole of that subphase. + + A v1 bundle had one entry per turn and folded the group back into a + `variants` array, because the format had nowhere else to put a second take. + A v2 bundle has coordinates, so an attempt is a node in the file exactly as + it is a node in the database, and `live` says which one is the story. + """ ScriptedProvider.replies = ["One.", "Two."] _play(client) _retry(client) bundle = client.get(f"/api/adventures/{client.adv_id}/export").json() ai = [a for a in bundle["actions"] if a["type"] == "ai"] - assert len(ai) == 1, "a v1 bundle carries one entry per turn, not per attempt" - assert [v["text"] for v in ai[0]["variants"]] == ["One.", "Two."] - assert ai[0]["variantIndex"] == 1 + assert [(a["text"], a["live"]) for a in ai] == [("One.", False), ("Two.", True)] + assert len({(a["branch"], a["depth"]) for a in ai}) == 1, "one turn, two takes" + assert "variants" not in ai[0], "nothing writes the repeating group any more" - # ...and importing it splits it back out into the rows it describes. + # ...and importing it puts the group back exactly as it stood. imported = client.post("/api/adventures/import", json=bundle).json()["id"] rows = _rows(imported) ai_rows = [a for a in rows if a.type == "ai"] diff --git a/backend/tests/test_bundle_v2.py b/backend/tests/test_bundle_v2.py new file mode 100644 index 0000000..7b1c276 --- /dev/null +++ b/backend/tests/test_bundle_v2.py @@ -0,0 +1,539 @@ +"""Phase 14 SP6 — the export bundle carries the tree. + +A bundle is the only part of this phase a migration can never reach: the file +is already on somebody's disk. So there are two formats, and the two halves of +this file watch different things. + +**v2 has to be lossless for a story that forked**, which v1 could not be — it +had one list and there were two stories, so it interleaved them by `index` and +read as a mangled story. Losslessness here means the *tree*: every branch, the +fork point it left its parent at, which attempt at each turn is the story, and +what each node left behind — because that last one is what a branch switch puts +back, and a tree nobody can switch inside is not the tree that was exported. + +**v1 has to still import**, because a backup that stops importing is not a +backup. + +Underneath both is the rule the module is built on: a bundle carries what was +*chosen* and never what is *derived*. The lineage, the head depth, the legacy +`index` and the variant ordinals are all rebuilt on the way in, so a +hand-edited file cannot disagree with itself — and the tests that matter most +here are the ones that hand it a file which does. + + python -m pytest tests/test_bundle_v2.py -v +""" +import os +import tempfile + +_tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False) +_tmp.close() +os.environ["AIDND_DB_PATH"] = _tmp.name +os.environ.pop("AIDND_DATABASE_URL", None) +os.environ.pop("DATABASE_URL", None) + +import pytest +from fastapi import Depends +from fastapi.testclient import TestClient + +from app import auth, bundle, limits, models +from app.context import lineage +from app.database import Base, SessionLocal, engine, get_db +from app.main import app +from app.providers import PromptParts +from app.routers import adventures + +SCHEMA = {"player": {"hp": {"min": 0, "max": 100, "initial": 100}}} + +# Ten gold a turn, so the script scoreboard is a number that says how many turns +# the story behind it has — which makes an after-snapshot visible from outside. +GOLD_SCRIPT = """ +const modifier = (text) => { + state.gold = (state.gold || 0) + 10; + return { text }; +}; +modifier(text); +""" + +OPENING = "You enter a cave." + + +class ScriptedProvider: + replies: list = [] + calls = 0 + + def __init__(self, *a, **k): + pass + + async def generate(self, parts: PromptParts, *, temperature, max_tokens): + index = min(ScriptedProvider.calls, len(ScriptedProvider.replies) - 1) + ScriptedProvider.calls += 1 + yield ("text", ScriptedProvider.replies[index]) + + +@pytest.fixture() +def client(monkeypatch): + Base.metadata.create_all(bind=engine) + setup = SessionLocal() + user = models.User(is_guest=False, email="bundle@example.com") + setup.add(user) + setup.flush() + setup.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="test-model")) + scenario = models.Scenario(user_id=user.id, title="S", stat_schema=SCHEMA) + setup.add(scenario) + setup.flush() + adv = models.Adventure( + user_id=user.id, title="Cave", scenario_id=scenario.id, + script_state={}, world_state={"player": {"hp": 100}}, + ) + setup.add(adv) + setup.flush() + setup.add(models.Action(adventure_id=adv.id, index=0, type="start", text=OPENING)) + setup.add(models.AdventureScript( + adventure_id=adv.id, position=0, enabled=True, name="Gold", output_js=GOLD_SCRIPT, + )) + setup.commit() + adv_id, user_id = adv.id, user.id + setup.close() + + ScriptedProvider.replies = ["A reply."] + ScriptedProvider.calls = 0 + monkeypatch.setattr(adventures, "OpenAICompatibleProvider", ScriptedProvider) + monkeypatch.setattr(auth, "resolve_provider_config", lambda s: auth.ProviderConfig( + "http://fake", "k", "test-model", False)) + monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None) + monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None) + + def _current_user(db=Depends(get_db)): + return db.get(models.User, user_id) + + app.dependency_overrides[auth.get_current_user] = _current_user + c = TestClient(app) + c.adv_id = adv_id + try: + yield c + finally: + app.dependency_overrides.clear() + adventures._active_turns.clear() + Base.metadata.drop_all(bind=engine) + + +# ------------------------------------------------------------------ helpers + +def _play(client, adv_id, text="look around", type="do"): + r = client.post(f"/api/adventures/{adv_id}/actions", json={"type": type, "text": text}) + assert r.status_code == 200, r.text + + +def _retry(client, adv_id): + assert client.post(f"/api/adventures/{adv_id}/retry").status_code == 200 + + +def _export(client, adv_id) -> dict: + r = client.get(f"/api/adventures/{adv_id}/export") + assert r.status_code == 200, r.text + return r.json() + + +def _import(client, payload): + return client.post("/api/adventures/import", json=payload) + + +def _imported(client, payload) -> int: + r = _import(client, payload) + assert r.status_code == 201, r.text + return r.json()["id"] + + +def _branches(client, adv_id) -> list[dict]: + r = client.get(f"/api/adventures/{adv_id}/branches") + assert r.status_code == 200, r.text + return r.json() + + +def _switch(client, adv_id, branch_id): + r = client.post(f"/api/adventures/{adv_id}/branches/{branch_id}/switch") + assert r.status_code == 200, r.text + + +def _texts(client, adv_id) -> list[str]: + return [a["text"] for a in client.get(f"/api/adventures/{adv_id}").json()["actions"]] + + +def _every_branch_story(client, adv_id) -> list[list[str]]: + """What each branch tells, in branch order — the whole tree as text.""" + stories = [] + for branch in _branches(client, adv_id): + _switch(client, adv_id, branch["id"]) + stories.append(_texts(client, adv_id)) + return stories + + +def _adventure_count() -> int: + db = SessionLocal() + try: + return db.query(models.Adventure).count() + finally: + db.close() + + +def _rows(adv_id) -> list[models.Action]: + db = SessionLocal() + try: + return ( + db.query(models.Action) + .filter(models.Action.adventure_id == adv_id) + .order_by(models.Action.branch_id, models.Action.depth, + models.Action.variant_index) + .all() + ) + finally: + db.close() + + +def _branch_rows(adv_id) -> list[models.Branch]: + db = SessionLocal() + try: + return ( + db.query(models.Branch) + .filter(models.Branch.adventure_id == adv_id) + .order_by(models.Branch.id) + .all() + ) + finally: + db.close() + + +def _script_state(adv_id) -> dict: + db = SessionLocal() + try: + return db.get(models.Adventure, adv_id).script_state + finally: + db.close() + + +def _forked_story(client) -> int: + """A story that went two ways, and stayed both. + + root: start · do · [attempt two] · do · next turn + fork: [ATTEMPT ONE] · do · elsewhere + + The *discarded* attempt is the one that gets promoted, because a fork moves + the take you are leaving for and leaves the line you came from untouched. + + Returns the adventure id, with the head on the fork. + """ + adv_id = client.adv_id + ScriptedProvider.replies = ["Attempt one.", "Attempt two.", "Next turn.", "Elsewhere."] + _play(client, adv_id) + _retry(client, adv_id) + _play(client, adv_id, "go deeper") + discarded = [a.id for a in _rows(adv_id) if a.type == "ai" and not a.live][0] + assert client.post(f"/api/adventures/{adv_id}/actions/{discarded}/fork").status_code == 200 + _play(client, adv_id, "go sideways") + return adv_id + + +# ------------------------------------------------------- the round trip (v2) + +def test_a_forked_story_survives_the_round_trip(client): + """The headline: both futures come back, and both are still readable. + + This is the thing v1 could not do. The check is not "the same rows" — the + ids are new — but "the same stories", read the way a player reads them: by + switching to a branch and looking at what it says. + """ + original = _forked_story(client) + before = _every_branch_story(client, original) + assert len(before) == 2, "the fixture forked" + assert before[0] != before[1], "and the two branches tell different stories" + + copy = _imported(client, _export(client, original)) + assert copy != original + assert _every_branch_story(client, copy) == before + + +def test_the_fork_point_comes_back_where_it_was_put(client): + """`fork_depth` is stored, never inferred — including through a file. + + Inferring it from where two branches' nodes first differ would be a guess + about how the story was played, and a wrong one the moment an attempt + happens to repeat its parent's text. + """ + original = _forked_story(client) + before = [(b["parent_branch_id"] is None, b["fork_depth"]) for b in _branches(client, original)] + + copy = _imported(client, _export(client, original)) + assert [(b["parent_branch_id"] is None, b["fork_depth"]) + for b in _branches(client, copy)] == before + + +def test_the_head_comes_back_on_the_branch_it_was_left_on(client): + original = _forked_story(client) + head_before = [b["is_head"] for b in _branches(client, original)] + assert head_before == [False, True], "the fixture left the head on the fork" + + copy = _imported(client, _export(client, original)) + assert [b["is_head"] for b in _branches(client, copy)] == head_before + # And the tip it sits at is derived from the nodes that arrived, not read + # out of the file — the bundle never says how deep a branch goes. + assert _texts(client, copy) == _texts(client, original) + + +def test_a_switch_in_the_copy_restores_what_that_branch_left_behind(client): + """The after-snapshots are why the bundle carries them. + + The gold script adds ten a turn, so the scoreboard is a count of the story + behind it. A bundle that carried the actions but not the outcomes would + import a tree that reads correctly and switches wrong. + """ + original = _forked_story(client) + # One more turn on the fork, so the two tips are genuinely different + # numbers: played turn for turn, the branches earn the same gold and a + # switch that restored nothing at all would still look right. + ScriptedProvider.replies = ["Further still."] + _play(client, original, "press on") + + per_branch = [] + for branch in _branches(client, original): + _switch(client, original, branch["id"]) + per_branch.append(_script_state(original).get("gold")) + assert len(set(per_branch)) == len(per_branch), "the tips are at different totals" + + copy = _imported(client, _export(client, original)) + restored = [] + for branch in _branches(client, copy): + _switch(client, copy, branch["id"]) + restored.append(_script_state(copy).get("gold")) + assert restored == per_branch + + +def test_a_memory_comes_back_on_the_node_it_hangs_off(client): + """Derived work is addressed by coordinate, so the coordinate is carried.""" + original = _forked_story(client) + tip = [a for a in _rows(original) if a.live][-1] + db = SessionLocal() + try: + db.add(models.Memory( + adventure_id=original, text="They met a goblin.", + source_start=0, source_end=tip.depth, + branch_id=tip.branch_id, depth=tip.depth, + )) + db.commit() + finally: + db.close() + + exported = _export(client, original) + assert [(m["branch"], m["depth"]) for m in exported["memories"]] == [(1, tip.depth)] + + copy = _imported(client, exported) + db = SessionLocal() + try: + memories = db.query(models.Memory).filter( + models.Memory.adventure_id == copy).all() + branches = [b.id for b in _branch_rows(copy)] + assert [(branches.index(m.branch_id), m.depth) for m in memories] == [(1, tip.depth)] + finally: + db.close() + + +# --------------------------------------------------- what is not in the file + +def test_the_lineage_is_rebuilt_rather_than_carried(client): + """A cache of `parent` + `fork_depth` is not a second thing to ship. + + The file says where each branch forked; the ancestry that makes the fork + readable is computed from that on the way in, capped at the fork exactly as + `tree.fork` caps it. Shipping the cache too would put two sources of truth + for one fact in a file anybody can hand-edit. + """ + original = _forked_story(client) + exported = _export(client, original) + assert all("lineage" not in b for b in exported["branches"]) + + root, forked = _branch_rows(_imported(client, exported)) + assert root.lineage == [[root.id, None]] + assert forked.lineage == [[forked.id, None], [root.id, forked.fork_depth]] + # Which is the arithmetic the reader depends on: the parent is capped one + # depth short of the attempt that was promoted, so the fork cannot see it. + assert lineage.entries_of(forked) == [(forked.id, None), (root.id, forked.fork_depth)] + + +def test_the_legacy_index_is_reissued_so_two_branches_never_share_one(client): + """`index` is a fact about the adventure, and `depth` is one about a path. + + Two branches have a node at depth 3, so depth cannot be the number + `max_action_index` hands out next. The import allocates one per turn + instead: siblings share it, the way SP4 leaves them, and no two coordinates + do. + """ + copy = _imported(client, _export(client, _forked_story(client))) + rows = _rows(copy) + by_index: dict[int, set] = {} + for row in rows: + by_index.setdefault(row.index, set()).add((row.branch_id, row.depth)) + assert all(len(coords) == 1 for coords in by_index.values()), \ + "one index per turn, whatever branch it is on" + assert len(by_index) == len({(r.branch_id, r.depth) for r in rows}) + + # The case that makes the rule necessary: the fork and the line it left + # both hold a turn at depth 2, and they are not the same turn. + at_depth_2 = [r for r in rows if r.depth == 2] + assert len({r.branch_id for r in at_depth_2}) == 2 + assert len({r.index for r in at_depth_2}) == 2, "same depth, different turns" + + +# ------------------------------------------------------- a file that is wrong + +def test_a_node_naming_a_branch_the_file_does_not_list_is_refused(client): + """Refused, not half-applied. A tree missing a branch is a story that + silently stops, which is the failure this whole phase exists to end.""" + payload = _export(client, _forked_story(client)) + payload["branches"] = payload["branches"][:1] + before = _adventure_count() + + r = _import(client, payload) + assert r.status_code == 400, r.text + assert "branch" in r.json()["detail"].lower() + assert _adventure_count() == before, "nothing was created" + + +def test_a_branch_forking_from_one_listed_after_it_is_refused(client): + """The ordering rule buys acyclicity for the price of a comparison — and a + cycle would be an import that never returns rather than one that fails.""" + payload = _export(client, _forked_story(client)) + payload["branches"] = [{"parent": 1, "forkDepth": 0}, {"parent": None, "forkDepth": None}] + before = _adventure_count() + + r = _import(client, payload) + assert r.status_code == 400, r.text + assert _adventure_count() == before + + +def test_a_fork_with_no_depth_is_refused(client): + payload = _export(client, _forked_story(client)) + payload["branches"][1].pop("forkDepth") + before = _adventure_count() + + r = _import(client, payload) + assert r.status_code == 400, r.text + assert "depth" in r.json()["detail"].lower() + assert _adventure_count() == before + + +def test_more_branches_than_the_cap_is_refused(client, monkeypatch): + monkeypatch.setattr(auth, "MULTI_USER", True) + payload = { + "format": bundle.FORMAT, "title": "Too many", + "branches": [{"parent": None, "forkDepth": None}] + * (limits.MAX_BRANCHES_PER_ADVENTURE + 1), + "actions": [], + } + before = _adventure_count() + + r = _import(client, payload) + assert r.status_code == 409, r.text + assert _adventure_count() == before + + +def test_a_turn_the_file_gives_no_live_attempt_still_tells_one(client): + """A coordinate with nothing live is a turn no read can see. + + The file is allowed to be wrong about this — it is a text file — so the + import picks the first attempt rather than importing a story with a hole. + """ + payload = _export(client, _forked_story(client)) + for node in payload["actions"]: + node["live"] = False + copy = _imported(client, payload) + + assert _texts(client, copy), "the story is readable" + live = [(r.branch_id, r.depth) for r in _rows(copy) if r.live] + assert len(live) == len(set(live)), "exactly one live attempt per coordinate" + assert len(live) == len({(r.branch_id, r.depth) for r in _rows(copy)}) + + +# ------------------------------------------------------------- the v1 reader + +def test_a_v1_bundle_still_imports(client): + """The reader stays after the writer goes: those files are already saved.""" + payload = { + "format": bundle.LEGACY_FORMAT, + "title": "Old backup", + "memoryCursor": 0, "summaryCursor": 0, + "actions": [ + {"index": 0, "type": "start", "text": OPENING}, + {"index": 1, "type": "do", "text": "> You go north."}, + { + "index": 2, "type": "ai", "text": "Two.", + "variants": [{"text": "One."}, {"text": "Two."}], + "variantIndex": 1, + }, + ], + } + copy = _imported(client, payload) + + assert _texts(client, copy) == [OPENING, "> You go north.", "Two."] + # One branch, and the `variants` array split back into the sibling group it + # always described. + assert len(_branches(client, copy)) == 1 + ai = [r for r in _rows(copy) if r.type == "ai"] + assert [(r.text, r.live) for r in ai] == [("One.", False), ("Two.", True)] + assert len({(r.branch_id, r.depth) for r in ai}) == 1 + + +def test_a_v1_bundle_with_a_cursor_lands_it_on_a_node(client): + """v1 counts covered actions; the tree anchors them. The translation needs + the nodes to exist, so it happens after they are written.""" + payload = { + "format": bundle.LEGACY_FORMAT, "title": "Old backup", + "memoryCursor": 2, "summaryCursor": 2, + "actions": [ + {"index": 0, "type": "start", "text": OPENING}, + {"index": 1, "type": "story", "text": "A corridor."}, + {"index": 2, "type": "story", "text": "A door."}, + ], + } + copy = _imported(client, payload) + + db = SessionLocal() + try: + adventure = db.get(models.Adventure, copy) + assert adventure.memory_cursor == 2, "the legacy count is kept as given" + assert adventure.memory_cursor_branch_id is not None + assert adventure.memory_cursor_depth == 1, "the second story action" + finally: + db.close() + + +def test_a_v2_bundle_brings_its_anchors_back(client): + """The other direction: v2 carries the anchor and the count is read off it.""" + original = _forked_story(client) + tip = [a for a in _rows(original) if a.live][-1] + db = SessionLocal() + try: + adventure = db.get(models.Adventure, original) + adventure.memory_cursor_branch_id = tip.branch_id + adventure.memory_cursor_depth = tip.depth + db.commit() + finally: + db.close() + + exported = _export(client, original) + assert exported["memoryCursor"] == {"branch": 1, "depth": tip.depth} + + copy = _imported(client, exported) + db = SessionLocal() + try: + adventure = db.get(models.Adventure, copy) + branches = [b.id for b in _branch_rows(copy)] + assert branches.index(adventure.memory_cursor_branch_id) == 1 + assert adventure.memory_cursor_depth == tip.depth + assert adventure.memory_cursor > 0, "the legacy count was read back off it" + finally: + db.close() + + +def test_an_unknown_format_is_refused(client): + r = _import(client, {"format": "ai-dnd-adventure-v3", "title": "From the future"}) + assert r.status_code == 400, r.text + assert bundle.FORMAT in r.json()["detail"] diff --git a/backend/tests/test_retry_variants.py b/backend/tests/test_retry_variants.py index 7eaa9fe..8e65278 100644 --- a/backend/tests/test_retry_variants.py +++ b/backend/tests/test_retry_variants.py @@ -330,9 +330,12 @@ def test_export_and_import_round_trips_variants(client): _play(client) _retry(client) bundle = client.get(f"/api/adventures/{client.adv_id}/export").json() - ai = [a for a in bundle["actions"] if a["type"] == "ai"][0] - assert [v["text"] for v in ai["variants"]] == ["One.", "Two."] - assert ai["variantIndex"] == 1 + # SP6: the attempts are nodes in the bundle too, sharing one coordinate, + # and `live` says which of them the story tells. The `variants` array + # survives only in the v1 *reader* — see the hand-edited bundle below. + ai = [a for a in bundle["actions"] if a["type"] == "ai"] + assert [(a["text"], a["live"]) for a in ai] == [("One.", False), ("Two.", True)] + assert len({(a["branch"], a["depth"]) for a in ai}) == 1 r = client.post("/api/adventures/import", json=bundle) assert r.status_code == 201, r.text diff --git a/backend/tests/test_story_tree_baseline.py b/backend/tests/test_story_tree_baseline.py index 3745266..795270c 100644 --- a/backend/tests/test_story_tree_baseline.py +++ b/backend/tests/test_story_tree_baseline.py @@ -451,9 +451,11 @@ def test_memories_are_created_listed_and_deleted(client): def test_export_carries_the_whole_story(client): """A bundle is a backup: every action, not the window. - NOTE for SP6 — the `format` assertion below is the one line in this file - expected to change, when the bundle becomes `ai-dnd-adventure-v2`. It is - correct through SP1-SP5. + SP6 changed the format assertion below, as this note said it would. It also + changed one more line in this file than the note allowed for — the + `variants` array in `test_export_keeps_retry_attempts`, which is the same + fact seen from the other side: a bundle that has coordinates has no use for + a repeating group. Everything else here still passes unmodified. """ ScriptedProvider.replies = ["One.", "Two."] _play(client, "go north") @@ -462,7 +464,7 @@ def test_export_carries_the_whole_story(client): r = client.get(f"/api/adventures/{client.adv_id}/export") assert r.status_code == 200, r.text bundle = r.json() - assert bundle["format"] == "ai-dnd-adventure-v1" + assert bundle["format"] == "ai-dnd-adventure-v2" assert bundle["title"] == "Cave" assert [a["text"] for a in bundle["actions"]] == [ OPENING, "> You go north.", "One.", "> You go south.", "Two.", @@ -490,8 +492,9 @@ def test_export_keeps_retry_attempts(client): client.post(f"/api/adventures/{client.adv_id}/retry") bundle = client.get(f"/api/adventures/{client.adv_id}/export").json() - ai = [a for a in bundle["actions"] if a["type"] == "ai"][-1] - assert [v["text"] for v in ai["variants"]] == ["Attempt one.", "Attempt two."] + ai = [a for a in bundle["actions"] if a["type"] == "ai"] + assert [a["text"] for a in ai] == ["Attempt one.", "Attempt two."] + assert [a["live"] for a in ai] == [False, True] # -------------------------------------------------------------- world state diff --git a/backend/tools/measure_bundle.py b/backend/tools/measure_bundle.py new file mode 100644 index 0000000..cc05faf --- /dev/null +++ b/backend/tools/measure_bundle.py @@ -0,0 +1,149 @@ +"""What a v2 bundle costs, measured on a production-sized adventure. + +SP6 gave the bundle coordinates, and coordinates cost bytes: a branch number +and a depth on every node, plus the two after-snapshots a switch needs to put +back. This puts a number on that, against the v1 shape it replaces, on the same +fixture the egress work is measured on. + + python -m tools.measure_bundle --actions 600 --rich + python -m tools.measure_bundle --actions 600 --rich --forks 20 + +Reuses `tools.stress_session`'s fixture, so read that module's warnings first: +a `--rich` run is a correctness fixture, and its byte figures are not +comparable to a plain one. +""" +import json +import random +import sys + +# `stress_session` FIRST, and it is not a style preference. Importing it is what +# points `AIDND_DB_PATH` at a throwaway file, and `app.database` reads that at +# module scope — so an `app` import above this line silently runs the whole +# fixture against `backend/data.db` instead. It fails by *working*: the first +# run seeds a synthetic user and adventure into the local database and reports +# perfectly good numbers, and only the second run trips over the unique email. +from tools import stress_session as stress # noqa: I001 (see above) + +from app import bundle, models, tree +from app.database import SessionLocal + + +def _v1_shape(v2: dict) -> dict: + """The same story as v1 would have written it, for a like-for-like count. + + One entry per turn, the siblings folded back into a `variants` array, no + coordinates and no outcomes — which is exactly what v1 could carry. + """ + turns: dict[tuple[int, int], list[dict]] = {} + order: list[tuple[int, int]] = [] + for node in v2["actions"]: + key = (node["branch"], node["depth"]) + if key not in turns: + turns[key] = [] + order.append(key) + turns[key].append(node) + actions = [] + for i, key in enumerate(order): + group = turns[key] + live = next((n for n in group if n["live"]), group[0]) + actions.append({ + "index": i, "type": live["type"], "text": live["text"], + "reasoning": live.get("reasoning"), + "variants": [ + {"text": n["text"], "reasoning": n.get("reasoning"), + "createdAt": n["createdAt"]} + for n in group + ] if len(group) > 1 else None, + "variantIndex": group.index(live), + "createdAt": live["createdAt"], + }) + old = {k: v for k, v in v2.items() if k not in ("branches", "headBranch")} + old["format"] = bundle.LEGACY_FORMAT + old["actions"] = actions + old["memories"] = [ + {k: v for k, v in m.items() if k not in ("branch", "depth")} + for m in v2["memories"] + ] + old["memoryCursor"] = 0 + old["summaryCursor"] = 0 + return old + + +def _bytes(obj) -> int: + return len(json.dumps(obj).encode("utf-8")) + + +def main(argv=None) -> int: + argv = list(sys.argv[1:] if argv is None else argv) + forks = 0 + if "--forks" in argv: + i = argv.index("--forks") + forks = int(argv[i + 1]) + del argv[i:i + 2] + args = stress.parse_args(argv) + rng = random.Random(args.seed) + stress._build_text(args, random.Random(args.seed ^ 0x5F5F)) + adv_id, _ = stress.build_fixture(args, rng) + + db = SessionLocal() + try: + adventure = db.get(models.Adventure, adv_id) + if forks: + # Twenty divergences off one line, each a little deeper — the shape + # SP5 measured the fork cost on. The nodes are chosen up front and + # the session is flushed after every fork: `fork` moves a row onto + # its new branch, and with `autoflush=False` a query issued before + # that move is written still finds the node where it used to be. + root = adventure.head_branch_id + candidates = [ + node.id for node in + db.query(models.Action) + .filter( + models.Action.adventure_id == adventure.id, + models.Action.branch_id == root, + models.Action.depth > 1, + ) + .order_by(models.Action.depth) + .all() + ] + step = max(len(candidates) // (forks + 1), 1) + made = 0 + for node_id in candidates[::step][:forks]: + tree.fork(db, adventure, db.get(models.Action, node_id)) + db.flush() + made += 1 + db.commit() + print(f"forked {made} times") + + v2 = bundle.export(db, adventure) + v1 = _v1_shape(v2) + nodes = len(v2["actions"]) + turns = len({(n["branch"], n["depth"]) for n in v2["actions"]}) + branches = len(v2["branches"]) + + stripped = json.loads(json.dumps(v2)) + for node in stripped["actions"]: + node.pop("stateAfter", None) + node.pop("worldStateAfter", None) + node.pop("worldDelta", None) + + v2_bytes, v1_bytes, bare = _bytes(v2), _bytes(v1), _bytes(stripped) + print(f"{turns} turns · {nodes} nodes · {branches} branches") + print(f"v1 shape {v1_bytes:>12,} B") + print(f"v2 {v2_bytes:>12,} B " + f"{v2_bytes / v1_bytes:.3f}× v1") + print(f"v2 w/o outcomes {bare:>12,} B " + f"{bare / v1_bytes:.3f}× v1") + print(f"the outcomes {v2_bytes - bare:>12,} B " + f"{(v2_bytes - bare) / nodes:.1f} B/node") + print(f"coordinates {bare - v1_bytes:>12,} B " + f"{(bare - v1_bytes) / nodes:+.1f} B/node") + print(f"import cap {bundle.__name__}: " + f"{v2_bytes / (20 * 1024 * 1024):.1%} of MAX_IMPORT_BODY_BYTES") + finally: + db.close() + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/plan/14-phase-story-tree.md b/plan/14-phase-story-tree.md index 415ed8c..5b249cc 100644 --- a/plan/14-phase-story-tree.md +++ b/plan/14-phase-story-tree.md @@ -37,8 +37,10 @@ existed for undo and delete; see the trap note below. Two rows want a footnote: stale exactly as before. What *has* changed is that the machinery to fix it now exists — an edit could write a sibling and switch to it, which is a retry the player typed — so it is a small change whenever it is wanted. -- **The 1NF violation** is resolved in the database. The `variants` array survives in - exactly one place: the v1 export bundle, which SP6 replaces. +- **The 1NF violation** is resolved. Nothing writes a `variants` array any more, in the + database or out of it — SP6 replaced the bundle that was its last producer, and the + array survives only in the v1 *reader*, which exists so files already saved still + import. ## Design decisions (settled 2026-08-16) @@ -610,6 +612,88 @@ bundle's linear actions plus `variants` onto one branch with siblings. **Verify:** v2 round-trip of a branched adventure is lossless; a v1 bundle still imports; a bundle claiming more branches than rows is rejected rather than half-applied. +**Done, 2026-08-18** (branch `sp6-bundle-v2`). **381 tests green**, the 365 SP5 finished +with plus 16 in the new `test_bundle_v2.py`. `app/bundle.py` owns both formats; the two +endpoints in `routers/adventures.py` are a delegation and the shared plumbing, and the +`variants` array now exists nowhere but the v1 *reader*. + +**The rule the module is built on: a bundle carries what was *chosen*, never what is +*derived*.** The head branch, the fork points, the live flags and the anchors are +decisions somebody made, and they are in the file. `lineage`, the head *depth*, `index` +and the variant ordinals are computed from those and are rebuilt on the way in. That is +not tidiness — a bundle is a text file anybody can edit, and every derived field shipped +beside its source is a chance for the file to disagree with itself in a way no read would +report. It is also the answer to "is the round trip lossless?": everything omitted is +reconstructed, and the tests assert the reconstruction rather than the bytes. + +Six things worth not rediscovering: + +- **`depth` cannot be the legacy `index`, and this is where that stops being academic.** + They agreed until SP5, and a bundle is the first writer that has to fill `index` for a + *forked* story — where two branches both have a node at depth 4. `index`'s one + remaining job is handing the next row a number nothing else holds, which is a fact + about the adventure rather than about a path, so the import allocates one per turn in + bundle order: siblings share it, the way SP4 leaves them, and no two coordinates do. +- **Validation happens before the adventure row exists.** Everything a hand-edited file + can get wrong about the shape of a tree — a node naming a branch that is not listed, a + fork with no depth, a branch forking from one listed after it — is a 400 raised by + `plan`, which touches no session. The alternative is an adventure holding a story with + a hole in it, and this whole phase exists because a story with a hole in it fails by + going quiet. +- **A branch may only fork from one listed before it.** That is how the export writes + them, and requiring it buys acyclicity for the price of a comparison — a cycle in the + parent chain would be an import that never returns rather than one that fails. +- **`{}` and absent are different snapshots.** An empty `state_after` means "this node + left an empty scoreboard behind"; a missing one means "nobody knows, leave the live + state alone" (`attempts.restore_state`). Trimming empty dicts on the way out would have + saved eighteen bytes a row and turned an undo that clears a score into one that leaves + it standing. Only `worldDelta`, which is display, is dropped when empty. +- **A file is allowed to be wrong about which attempt is live, and the import corrects it + rather than refusing.** "Exactly one sibling in a group is live" is an invariant of the + *database*, not of the format; a coordinate with none is a turn no read can see, so the + first attempt is made live. That is a different class from a missing branch, which is + structure, and is refused. +- **`limits.MAX_BRANCHES_PER_ADVENTURE` (1000) has no live counterpart.** Forking is a + POST that adds one row and has no cap of its own, so this is the one bundle cap that + does not mirror something creation enforces. Worth closing if branch management ever + makes forking cheap to repeat. + +**The verify line above was slightly wrong, and the code does the honest version.** "More +branches than rows" fails on an adventure with no actions, which legitimately has one +branch and no rows. It became two rules instead: a cap on the branch list, and *every +branch a node names must exist*. + +Measured with `tools/measure_bundle.py` on the 600-action `--rich` fixture — 600 turns, +750 nodes, because 150 of them were retried: + +| | bytes | vs v1 | +|---|---|---| +| v1 shape | 587,475 | — | +| **v2** | **911,229** | **1.551×** | +| v2 without the outcomes | 544,318 | 0.927× | + +**The tree is free; the outcomes are what cost.** Coordinates *save* 57.5 B a node +against v1's turn-and-variants shape, and the entire 1.55× is `state_after` / +`world_state_after` at 489 B a node — which are there because a bundle without them +imports a tree nobody can switch inside. Twenty forks add 660 B to the same file, **33 B +a branch**, so the format is as indifferent to fork count as the reads are. The longest +adventure production holds exports at 4.3 % of `MAX_IMPORT_BODY_BYTES`. + +**Two lines of the SP0 baseline changed, not the one it predicted.** The note in +`test_story_tree_baseline.py` allowed for the `format` assertion; the `variants` array in +`test_export_keeps_retry_attempts` is the same fact from the other side — a bundle with +coordinates has no use for a repeating group. Everything else in that file still passes +unmodified. + +**No migration, no vacuum.** SP6 adds no column and rewrites no row. + +**And one trap, paid for once.** `tools/measure_bundle.py` imported `app` before +`tools.stress_session`, which is what points `AIDND_DB_PATH` at a throwaway file — +`app.database` reads it at module scope. It fails by *working*: the first run seeded a +synthetic user and adventure into the local `backend/data.db` and printed perfectly good +numbers, and only the second run tripped over the unique email. Anything importing that +harness must import it first, and the file now says so where the imports are. + ### SP7 — Frontend: full tree visualisation `VariantPager` is removed. A spatial tree view replaces it, plus switch, rename and diff --git a/plan/STATUS.md b/plan/STATUS.md index 28da522..7dad0db 100644 --- a/plan/STATUS.md +++ b/plan/STATUS.md @@ -78,37 +78,41 @@ needed; nothing requires reading a row of anyone's story. ## Pick up here -**`plan/14-phase-story-tree.md`, SP6 — export/import v2.** SP0–SP5 are done and green -(**365 tests**); **nothing is deployed yet**. The tree is complete as a storage model: a -retry writes a sibling node, continuing from a discarded attempt forks a branch, and -`GET /branches`, `POST /branches/{id}/switch` and `POST /actions/{id}/fork` are the -endpoints SP7's tree view will be drawn on. What is left is the bundle format (SP6), the -frontend (SP7) and dropping the legacy columns (SP8). +**`plan/14-phase-story-tree.md`, SP7 — the frontend.** SP0–SP6 are done and green +(**381 tests**); **nothing is deployed yet**. The tree is complete everywhere except the +screen: a retry writes a sibling node, continuing from a discarded attempt forks a branch, +and a backup carries the whole thing (`ai-dnd-adventure-v2`, with the v1 reader kept so +existing bundles still import). What is left is the frontend (SP7) and dropping the legacy +columns (SP8). -**Do SP6 before SP7, and the reason is a live gap.** A forked adventure has no honest v1 -export — the format has one story and there are two — so export currently emits every -branch's turns interleaved by `index`, which reads as a mangled story. Nobody can reach -that state through the product yet, because forking has no UI until SP7. That ordering is -the whole mitigation, so keep it. +**SP7 is the release gate, and it is unscoped.** `VariantPager` comes out, a spatial tree +view replaces it, and branch management — switch, rename, delete-with-confirm — is a hard +dependency rather than a nice-to-have, because nothing auto-prunes and storage otherwise +grows without limit. `api.js` gains `GET /branches`, `POST /branches/{id}/switch` and +`POST /actions/{id}/fork`, which SP5 built for exactly this. + +**Drive the 600-action `--keep` fixture in a browser by hand.** That is SP7's own verify +line and the standing open gap in this project: the scroll path has never been driven by +hand and has already hidden one bug. A vitest + jsdom harness covers the prepend +arithmetic, but jsdom has no layout, so scroll position still needs eyes. **The schema is live in code but not on production.** When this ships, the deploy needs one `VACUUM FULL actions;` on the direct (non-`-pooler`) endpoint afterwards — SP1's migration rewrites every row and SP4's rewrites it three times more, so **two vacuums are -owed and one run settles both**. SP3's and SP5's changes touch `adventures` only (SP5 adds -no migration at all) and need none. See the 144 MB lesson at the top of this file. +owed and one run settles both**. SP3's, SP5's and SP6's changes need none (SP5 and SP6 add +no migration at all). See the 144 MB lesson at the top of this file. -Three things to carry into SP6: +Three things to carry into SP7: -- **The `variants` array now exists in exactly one place: the export bundle.** Nothing in - the database holds one. `export_adventure` folds each sibling group back into the shape - the v1 reader expects, and `_imported_turn` splits one back out into rows. Those two - functions are the whole v1 surface, and v2 replaces them. -- **A v2 bundle needs branches, `live`, and both after-snapshots.** `state_after` / - `world_state_after` are what a branch switch restores; a bundle that carried the - actions but not the outcomes would import a tree nobody could switch inside. - `limits.check_bundle_lists` has to learn about branches too. -- **Weigh new columns in bytes.** `actions` is already the table that fills the disk. - `tests/test_egress.py` has byte ceilings — they will tell you. +- **The bundle is the one thing here a migration can never reach.** `app/bundle.py` owns + both formats and nothing else knows either. Its rule — *carry what was chosen, never + what is derived* — is worth borrowing anywhere else state has to leave the database. +- **`variant_count` and `variant_index` die with the pager.** SP4 left them as a + maintained cache of the sibling group's shape because the pager reads both for every + row of a page. They are dead the moment the tree view replaces it, and SP8 drops them. +- **Nothing on the screen has ever seen a second branch.** Forking has no UI, which is + why the v1-export gap could be left open through SP5 — SP7 is the subphase that makes + a fork reachable, so it is also the one that makes every branch-shaped bug reachable. And one known cost, not a bug: the two memory marks are a single pair on the adventure, so switching branches makes the mark on the branch being left unreadable from the new one @@ -125,6 +129,44 @@ drive it before rewriting it. --- +## What happened on 2026-08-18, part three — the tree, SP6 + +The backup learned the tree. `ai-dnd-adventure-v2` carries branches, the fork point each +one left its parent at, which attempt at every turn is the story, and what each node left +behind — that last one because it is what a branch switch puts back, and a bundle that +imported a tree nobody could switch inside would be a backup of the wrong thing. The v1 +*reader* stays: those files are already on disk. **381 tests green**, 16 of them new in +`test_bundle_v2.py`. Branch `sp6-bundle-v2`, no migration, no vacuum. + +The gap SP5 left is closed — a forked adventure now has an honest export — so the +ordering constraint that has governed the last two subphases is discharged, and SP7 is +free. + +Three things to carry forward: + +- **Carry what was chosen, never what is derived.** The head branch, the fork points, the + live flags and the anchors are decisions, and they are in the file. `lineage`, the head + depth, the legacy `index` and the variant ordinals are computed from those, so they are + rebuilt on import instead. A bundle is a text file anybody can edit, and a derived field + shipped beside its source is a chance for the file to contradict itself in a way no read + reports. It also turns "is the round trip lossless?" into a testable question: every + omitted field is reconstructed, and the tests assert the reconstruction. +- **Check the shape before creating the row.** A node naming a branch the file does not + list is a 400 raised by a pure function, not a half-written adventure. The failure this + phase exists to end is a story that goes quiet, and a half-applied import is exactly + that. +- **The tree is free; the outcomes are what cost.** On the 600-action fixture the bundle + goes from 587 kB to 911 kB, and *all* of it is `state_after`/`world_state_after` at + 489 B a node — the coordinates themselves save 57.5 B a node against v1's + turn-and-variants shape. Twenty forks add 660 B. 4.3 % of the import body cap at + production's longest adventure. + +Paid for once, and worth not repeating: a new measuring script imported `app` before +`tools.stress_session`, which is what redirects `AIDND_DB_PATH` at a throwaway file. It +failed by *working* — the first run seeded a synthetic user into the local `data.db` and +printed good numbers; the second tripped over the unique email. **A harness that decides +where the database lives has to be imported before anything that reads it.** + ## What happened on 2026-08-18, part two — the tree, SP4 and SP5 A retry stopped rewriting a row, and a story learned to go two ways at once.