# Phase 17: refactor for readability The code works and it is tested. It is also hard to read, because four files hold most of it, one schema migration is half finished, and the published guide is a hand-maintained copy of another file. This phase fixes those three things without changing what the app does. ## How to use this file This file is both the plan and the running status. Update the progress table at the end of every stage. Read the table first when you pick the work back up. Each stage lands as its own commit on `refactor-17`. No stage starts until the stage before it is green. ## Progress | Stage | Work | Status | Landed | |---|---|---|---| | 0 | Hygiene: worktrees, branches, undocumented settings | done, except the branch deletion | 2026-08-29 | | 1 | Split the four largest files | tests share one setup; the router is a package; three files left | | | 2 | Remove duplication | not started | | | 3 | SP8: drop the legacy columns | not started | | | 4 | Documentation | not started | | | 5 | A frontend test runner | not started | | ## Baseline Record these numbers before you start. They are how you tell a refactor from a rewrite. - 549 backend tests pass in 103 seconds. - `backend/app` holds 13,486 lines across 45 files. - `backend/tests` holds 12,221 lines across 38 files. - `frontend/src` holds 9,171 lines across 21 files. - The app exposes 70 endpoints and applies 64 migrations on boot. Run the suite with `cd backend && .venv/Scripts/python.exe -m pytest tests/ -q`. ## What the review found ### The code is clean at the statement level A scan for unreferenced functions across `backend/app` returned only FastAPI route handlers, which the decorator references rather than the name. A scan of `index.css` found 5 unused class names out of 373, and every one of the 5 is either CodeMirror's or built from a template string. There is no dead code to delete and no `TODO` or `FIXME` anywhere in the tree. Read that as a constraint. The gains in this phase come from moving code, not from finding rot. ### Four files hold most of the complexity | File | Lines | What it holds | |---|---|---| | `backend/app/routers/adventures.py` | 2353 | Nine concerns: the turn engine, branches, takes, import and export, adventure scripts, refresh from scenario, insights, memory CRUD, and action CRUD | | `frontend/src/pages/Play.jsx` | 2280 | 27 components, plus a 722-line `Play()` holding 18 `useState` calls and 7 `useEffect` calls | | `frontend/src/index.css` | 2865 | One stylesheet. The `max-width: 720px` block at line 2655 overrides rules written 2000 lines above it | | `backend/app/worldstate/engine.py` | 918 | Four jobs: parse a delta, apply a delta, render the context sections, and instantiate a schema | ### The schema is half migrated `plan/14-phase-story-tree.md` defines SP8, which drops the columns the tree replaced. SP8 was gated on the tree running in production. It now does. Eight columns are still written on every turn and read by nothing: - `actions.index`, `variants`, `variant_index`, `variant_count` - `actions.state_before`, `world_state_before` - `adventures.memory_cursor`, `summary_cursor` Until they go, `models.py` carries two vocabularies for one idea, and every reader has to be told which one is live. ### The published guide is a hand-written copy `docs/guide.html` is 84 KB of hand-written HTML covering the same material as the 63 KB `docs/GUIDE.md`. Nothing generates one from the other. They have already drifted: the HTML last changed on 2026-08-20 and the Markdown on 2026-08-28, and section 3.6, "Counting visits", exists only in the Markdown. GitHub Pages publishes the HTML, so readers get the stale copy. `docs/architecture.html` is 87 KB of hand-written HTML with no Markdown source at all. ### Duplication the tests already protect - `worldstate.apply_delta` and `apply_override` each re-implement the same path routing for `flags.`, `milestones.`, and `npc..`. That is six parallel branches at `engine.py:573`, `587`, `617`, `675`, `697`, and `740`. - 35 of the 38 test files repeat the same 8-line temporary-database prologue. There is no `conftest.py`. - `ScriptedProvider` or `FakeProvider` is defined 12 times, once per test file that needs a fake model. - The debounced autosave helper is copied into `Play.jsx:204`, `ScenarioEditor.jsx:23`, and `ScriptEditor.jsx:22`. - `get_adventure_or_404` is called by hand in about 20 handlers. It is a function, not a dependency, so every handler repeats the call. - `routers/chat.py:22` imports `SSE_HEADERS` and `sse` from `routers/adventures.py`. One router reaches into another for shared plumbing. ### One setting is documented nowhere `AIDND_TRUSTED_PROXY_HOPS` is read at `limits.py:55`. It does not appear in `backend/.env.example`, `README.md`, or `render.yaml`. It sets how many proxy hops `client_ip()` trusts, which is what stops a rotated `X-Forwarded-For` header from buying a fresh rate-limit bucket. If a deployment adds a proxy hop and nobody sets this variable, the bypass comes back silently. The unprefixed `DATABASE_URL` that `database.py:24` also accepts turned out to be documented already, inside the `AIDND_DATABASE_URL` entry in `backend/.env.example`. No change needed there. ### `Settings.stream` is dead `models.py:622` defines the column, and `schemas.py:452` and `499` expose it. Nothing reads it in the backend or the frontend. This is item S1 in `docs/self-review.md`. ### Local clutter Everything here is gitignored, so it costs repository weight nothing. It costs disk and attention. - Two abandoned worktrees, each a full copy of the tree: `.claude/worktrees/simplify-comments` and `.claude/worktrees/sp7-tree-ui`. Both branches landed on `main` as squashes. - 21 local branches. 13 read as unmerged to `git branch --no-merged`, and `plan/STATUS.md` already records that they landed as squashes. - Three stale SQLite files in `backend/`: `data.backup-2026-07-06.db`, `data.backup-pre-phase8.db`, and `scroll_fixture.db`, 2.9 MB together. The live local database, `backend/data.db`, is not one of them and is not touched. ## Stage 0: hygiene No source changes. Do this first, so later stages run against a quiet tree. 1. Remove both worktrees with `git worktree remove`, then delete the branches that landed as squashes. Verify each one first: a branch is safe to delete when `git log main..` names only commits whose message already appears in `git log main`. 2. Delete the three stale `.db` files. Leave `backend/data.db` alone. 3. Add `AIDND_TRUSTED_PROXY_HOPS` to `backend/.env.example`, with the reason it exists. Add it to the deploy section of `README.md` too, because that is where somebody putting the app behind another proxy looks. **Check:** the suite still passes, and `git status` is clean. ### What Stage 0 actually did, 2026-08-29 Both worktrees are gone, which freed about 104 MB. Removing `sp7-tree-ui` needed one extra step: a Vite dev server had been running out of that worktree since 2026-08-18, holding `frontend/.vite` open and owning port 5173. It was serving a tree 54 commits behind `main`. Stopping it freed the directory. Restart the real one with `start.ps1`. The three stale `.db` files are deleted. `backend/data.db` is untouched. `AIDND_TRUSTED_PROXY_HOPS` is now in `backend/.env.example` and in the README deploy section. **Still owed.** The 19 squash-landed branches are still there. `git branch -D` is blocked by the permission classifier, which is a reasonable guard on a destructive command. All 19 are verified safe by the rule above. Run this to clear them: ``` git branch -D bugfix-code-review docs-story-tree fix-prepend-autoscroll \ fix-silent-clamps-and-milestone-ids fix-store-refusals-and-delta-wording \ handover-2026-08-17 measure-post-vacuum-sizes phase-14-story-tree \ phase-7-public-repo phase-8-accounts sp1-tree-schema sp10-memory-bank-eviction \ sp2-branch-clause sp3-node-cursors sp4-sibling-nodes sp5-fork-on-continue \ sp6-bundle-v2 sp7-tree-ui sp7b-take-pager worktree-keep-fixture-and-status \ worktree-simplify-comments ``` `worktree-keep-fixture-and-status` is the one that needed checking by hand. Its commit subject appears nowhere in `main`, but the `--keep` flag it adds is on `main` at `tools/stress_session.py:764`, along with both gotchas the message describes, at `:686` and `:692`. Only the message differs. The remote branches are left alone. Deleting those is a separate decision. ## Stage 1: split the four largest files Every change in this stage moves code. None of it changes behavior. The 549 tests are the check, and they must pass without being edited, except where this section says otherwise. ### `routers/adventures.py` becomes a package Split it into `backend/app/routers/adventures/`: | Module | Holds | Source lines | |---|---|---| | `__init__.py` | The `APIRouter`, the shared dependencies, and re-exports | | | `paging.py` | `ACTION_LIST_COLUMNS`, `ACTION_PAGE`, `action_window`, `annotate_takes`, `current_window` | 27-146, 1203-1261 | | `crud.py` | List, create, get, patch, delete, plus the script-state and world-state readers | 225-580 | | `turns.py` | The turn engine: `generate_turn`, `_generate_turn`, `run_player_turn`, the turn lock, and the SSE helpers | 581-1004 | | `takes.py` | Retry, variants, takes, forking, and undo | 1005-1192, 1503-1778 | | `branches.py` | The branch endpoints | 1193-1502 | | `bundle_io.py` | Export and import | 1779-1837 | | `scripts.py` | The per-adventure script endpoints | 1838-1941 | | `refresh.py` | Refresh from scenario | 1942-2152 | | `insights.py` | The context dry run and the per-action context | 2153-2189 | | `memories.py` | Memory bank CRUD | 2190-2279 | Target no file above 450 lines. **The tests couple to the module, so read this before you start.** Twelve test files call `monkeypatch.setattr(adventures, "OpenAICompatibleProvider", ...)`. `monkeypatch` replaces a name in the module where the calling code looks it up, so re-exporting from `__init__.py` does not keep those patches working. Once `_generate_turn` lives in `turns.py`, the target becomes `adventures.turns.OpenAICompatibleProvider`. Do Stage 2's `conftest.py` work first if you want that to be a one-line change instead of twelve. Otherwise retarget all twelve here. The other patched names are `adventures.limits`, `adventures.check_demo_cap`, `adventures.generate_turn`, and `adventures._active_turns`, each used once. Keep these importable from the package root, because tests import them by name: `ACTION_PAGE`, `SNIPPET_MAX`, `_snippet`, `world_delta_of`, `acquire_turn_lock`, `retry_action`, `undo_turn`, and `_active_turns`. `_active_turns` is module-level mutable state guarded by a lock. It must live in exactly one module, `turns.py`, and every other module must import the module and reach through it. If two modules import the set by value, the lock guards two different sets and the turn lock stops working. ### What the router split actually did, 2026-08-29 `backend/app/routers/adventures.py` is now a package of 14 modules. The largest is `turns.py` at 443 lines. Four modules exist that the table above does not list, because the plan's eleven still mixed unrelated work: | Extra module | Why it exists | |---|---| | `deps.py` | Holds the `APIRouter` and the ownership check. It imports nothing else in the package, so every endpoint module can import the router without importing its siblings. | | `scenario_text.py` | Copying a scenario's text and cards has two callers, `crud.create_adventure` and `refresh`. Leaving it in either one made the other import an endpoint module. | | `nodes.py` | Story-tree navigation that four modules use: `last_action`, `next_index`, `next_depth`, `stand_on`, `db_tip`, `delete_turn`. | | `actions.py` | The three action endpoints. They page and delete rather than play a turn, so they do not belong in `crud.py`. | Two decisions differ from the plan above. **The package root does not re-export `acquire_turn_lock` or `_active_turns`.** The plan said to keep them importable, but that makes a broken patch look like a working one. Rebinding `adventures.generate_turn` changes the alias and leaves every caller reading the original, and the test still passes. Leaving those names off the package root raises `AttributeError` instead. Eighteen test call sites and four in `backend/tools/` now say `adventures.turns.`. The package root still re-exports the pure helpers, so `adventures.ACTION_PAGE`, `adventures.undo_turn`, and `chat.py`'s `from .adventures import SSE_HEADERS, sse` are unchanged. **`world_delta_of` lives in `turns.py`.** It reads as a world-state helper and sat beside the world-state endpoints, but `_generate_turn` is its only caller. `_active_turns` behaved as the plan warned. Every module reaches it as `turns._active_turns`, and a check confirms the four modules see one set object and one lock. One test coupled to the router for an unrelated module: it called `adventures.worldstate.instantiate`. It imports `app.worldstate` directly now. The split moved text rather than retyping it. An AST comparison against the pre-split file confirms all 86 definitions are identical, once the `turns.` prefix is normalized away. The 549 tests pass, and the OpenAPI schema still lists the same 35 operations. ### `worldstate/engine.py` becomes a package Split `backend/app/worldstate/engine.py` into four modules under `backend/app/worldstate/`: - `schema.py`: `has_schema`, `instantiate`, `reconcile`, `band_label`, `npc_name`, `npc_triggers`, `_initials`. - `parse.py`: `extract_delta`, `_tolerant_load`, `render_delta_block`, `applied_delta`, `refusals`, `render_refusals`. - `apply.py`: `apply_delta`, `apply_override`, and the shared path resolver Stage 2 introduces. - `render.py`: `render_state_section`, `render_reference`, `_stat_line`, `_describe_stat`. `worldstate/__init__.py` re-exports every public name it exports today, so no call site changes. The engine is imported as a module, not monkeypatched, so this split carries none of the router's test coupling. ### `pages/Play.jsx` becomes a directory Split it into `frontend/src/pages/Play/`: - `index.jsx`: the page component. - `usePlaySession.js`: the 18 `useState` calls and 7 `useEffect` calls that drive one adventure, behind one hook. - `panels/`: `PlotPanel`, `MemoryPanel`, `ScriptsPanel`, `InsightsPanel`, `BranchPanel`. - `drawers/`: `StatusDrawer`, `WorldStateDrawer`, and the `StatRow`, `StatGroup`, `StateTree`, and `StateValue` parts they use. - `reports/`: `StateChangeChips`, `WorldStateReport`, `ScriptReport`, `CacheReport`, `TokenBreakdown`. - `TakePager.jsx`, `RefreshModal.jsx`. There is no frontend test runner until Stage 5, so this split is verified by `npm run lint`, `npm run build`, and by driving the Play screen in a browser. Drive it. `plan/STATUS.md` records that the last two Play bugs were both found by hand and were unreachable from any test that existed. ### `index.css` becomes a directory Split it into `frontend/src/styles/`, imported in order from `index.css`: `tokens.css`, `base.css`, `nav.css`, `cards.css`, `play.css`, `drawers.css`, `schema-editor.css`, `insights.css`, `modals.css`, and `theme.css`. Move each `@media` block next to the rules it overrides, rather than leaving one `max-width: 720px` block at the end. Keep the source order identical when you move rules, because CSS resolves ties by order and the file relies on that in at least two known places, recorded in `plan/STATUS.md` and in the comments at `index.css:1011` and `1082`. **Check:** 549 tests pass, `npm run lint` and `npm run build` are clean, and the Play screen works in a browser at desktop and at 500 px wide. ## Stage 2: remove duplication Each item here is small and is covered by tests that already exist. 1. **One path resolver in `worldstate`.** `apply_delta` and `apply_override` route `flags.`, `milestones.`, `npc..`, `world.`, and `player.` with parallel code. Extract a resolver that returns the container, the stat definition, and the kind, then let the two functions differ only in the write rule. Their rules genuinely differ, so do not merge the functions themselves. `apply_override` ignores `cooldown`, `max_delta_per_turn`, and the rule that a counter cannot decrease, and it lets a milestone be un-set. `tests/test_worldstate.py` covers both. 2. **Add `backend/tests/conftest.py`.** Move the temporary-database prologue there, so the other 35 files drop 8 lines each. The prologue has to run before `from app.main import app`, because `main.py` calls `bootstrap(engine)` at import time. A `conftest.py` runs before any test module, which satisfies that. 3. **One fake provider.** Move `ScriptedProvider` and `FakeProvider` to `conftest.py` as one fixture, replacing 12 copies. 4. **Move the SSE helpers out of the router.** Put `sse`, `SSE_HEADERS`, and `turn_error` in `backend/app/sse.py`, so `chat.py` stops importing from `adventures`. 5. **Make `get_adventure_or_404` a dependency.** About 20 handlers repeat the call. A `Depends` removes the line from each one and puts the ownership check in the signature, where a reader looking for it expects it. 6. **One `useDebouncedSave` hook** in `frontend/src/hooks/`, replacing the three copies. 7. **Delete `Settings.stream`.** Remove the column, the two schema fields, and add the migration that drops it. This is item S1 in `docs/self-review.md`. Mark it applied there. **Check:** 549 tests pass. Test count may drop if consolidating fixtures removes a duplicate case. If it does, say which case and why in the commit message. ## Stage 3: SP8, drop the legacy columns This is the only stage that touches the production database. Follow `plan/14-phase-story-tree.md`, which specifies it. 1. Confirm nothing reads the eight columns. `ACTION_LIST_COLUMNS` in the adventures package lists `index`, `variant_count`, and `variant_index` today, so that tuple changes here. `bundle.py:37` and `context/history.py:424` both describe `index` as unread; verify that rather than trusting the comment. 2. Add the migration that drops them. Remove the columns from `models.py`, and remove the fields from `schemas.py` and from `ActionOut`. 3. Deploy, then run `VACUUM FULL actions;` on the direct Neon endpoint, not the `-pooler` one. Dropping a column rewrites toasted values, and nothing reclaims that space on its own. `plan/STATUS.md` records what happened the last time nobody ran it: the database reached 144.2 MB against a 512 MB tier. 4. `VACUUM FULL` takes an `ACCESS EXCLUSIVE` lock, so the app blocks on `actions` for the duration. It took 5.5 seconds at 144 MB. **Check:** 549 tests pass, `/api/health` answers `{"ok":true}` after the deploy, and one existing adventure opens, takes a turn, retries it, and pages back through the transcript. ## Stage 4: documentation 1. **Generate `docs/guide.html` from `docs/GUIDE.md`.** Write a small build script that renders the Markdown into the existing hand-written HTML shell, keeping the current styles and metadata. After that, one edit updates both. Note in `README.md` that the HTML is generated and that you edit the Markdown. 2. **Decide what `docs/architecture.html` is.** It has no Markdown source. Either give it one and generate it the same way, or state at the top of the file that it is hand-written, so the next person does not look for a source that does not exist. 3. **Split `plan/STATUS.md`.** It is 63 KB, and most of it is dated session logs. Keep "Pick up here", "Things worth remembering", and "Running things" in `STATUS.md`. Move the dated entries to `plan/history/`, newest file first. 4. **Add `docs/DEVELOPING.md`.** Record how to run the app, how to run the tests, where each subsystem lives, and the invariants a newcomer breaks first: - Anything that reads `adventure.actions` during generation takes `exclude_action_id`. It leaks into four places, not one. - `memory_cursor` and `summary_cursor` are positions into `story_actions()`, and `Memory.source_start` and `source_end` are `Action.index` values. The two spaces diverge as soon as anything is deleted. - `/auth/me` must not raise. It is the SPA's bootstrap call, so anything it touches that can raise takes the whole frontend down. - The global field rule in the stylesheet keys off `input[type=...]`, so a bare `` and any new input type get browser default styling until you add them. - A modal opened from `.side-panel` needs `createPortal`, because the panel's filling transform animation makes it the containing block for `position: fixed`. **Check:** the generated HTML matches the Markdown section for section, including section 3.6. Every link in `README.md` and `docs/index.html` resolves. ## Stage 5: a frontend test runner `plan/STATUS.md` names the missing runner as the reason this project keeps finding UI bugs by hand. Two shipped bugs were unreachable from any test that existed. 1. Add Vitest, React Testing Library, and jsdom. Add a `test` script to `frontend/package.json`. 2. Add the step to the existing `frontend` job in `.github/workflows/ci.yml`, between lint and build. 3. Write the first tests against the two bug classes that already recurred: - The take pager renders when a retry's reply arrives over SSE. The stream builds its own `ActionOut`, so it was the one payload that never carried the annotation. - Retaking a player turn does not produce `> You > You`. The editor is seeded with stored text that is already formatted. 4. Add a test for `usePlaySession` from Stage 1, since that hook now holds the state the page used to hold inline. **Check:** `npm test` passes locally and in CI. ## Risks | Risk | Where | How you catch it | |---|---|---| | A monkeypatch silently stops patching, and a test passes while calling a real provider | Stage 1, `routers/adventures` split | After retargeting, break the fake on purpose and confirm the tests that use it fail | | `_active_turns` ends up imported by value into two modules, so the turn lock guards two sets | Stage 1 | `tests/test_state_revert.py` reaches for `adventures._active_turns`. Keep that import path working, and confirm a second concurrent turn still returns 409 | | A CSS rule changes meaning because the split reorders it | Stage 1, stylesheet split | Compare the built CSS before and after. Order within each section must not change | | The column drop rewrites the table and nobody reclaims the space | Stage 3 | Run `VACUUM FULL actions;` on the direct endpoint, and read sizes from `sum(octet_length(col))` rather than `n_live_tup` | | Splitting `Play.jsx` breaks something no test covers | Stage 1 | Drive the Play screen by hand at both widths. Stage 5 exists to shrink this risk for next time | ## What this phase does not do - It does not replace the hand-rolled migration runner with Alembic. 64 migrations run correctly, and `docs/GUIDE.md` section 2.6 already explains the choice. - It does not move `bootstrap(engine)` out of import time in `main.py`. The current behavior means a failed migration means no service, which is deliberate. Stage 2's `conftest.py` removes the boilerplate that import-time bootstrapping forces on tests, which is the part that actually hurts. - It does not change any API shape, any database content, or any prompt.