interactive-story/plan/17-refactor.md

Phase 17: refactor for readability

The code works and it is tested. It is also hard to read, because four files hold most of it, one schema migration is half finished, and the published guide is a hand-maintained copy of another file. This phase fixes those three things without changing what the app does.

How to use this file

This file is both the plan and the running status. Update the progress table at the end of every stage. Read the table first when you pick the work back up.

Each stage lands as its own commit on refactor-17. No stage starts until the stage before it is green.

Progress

Stage	Work	Status	Landed
0	Hygiene: worktrees, branches, undocumented settings	done, except the branch deletion	2026-08-29
1	Split the four largest files	tests share one setup; the router is a package; three files left
2	Remove duplication	not started
3	SP8: drop the legacy columns	not started
4	Documentation	not started
5	A frontend test runner	not started

Baseline

Record these numbers before you start. They are how you tell a refactor from a rewrite.

• 549 backend tests pass in 103 seconds.
• backend/app holds 13,486 lines across 45 files.
• backend/tests holds 12,221 lines across 38 files.
• frontend/src holds 9,171 lines across 21 files.
• The app exposes 70 endpoints and applies 64 migrations on boot.

Run the suite with cd backend && .venv/Scripts/python.exe -m pytest tests/ -q.

What the review found

The code is clean at the statement level

A scan for unreferenced functions across backend/app returned only FastAPI route handlers, which the decorator references rather than the name. A scan of index.css found 5 unused class names out of 373, and every one of the 5 is either CodeMirror's or built from a template string. There is no dead code to delete and no TODO or FIXME anywhere in the tree.

Read that as a constraint. The gains in this phase come from moving code, not from finding rot.

Four files hold most of the complexity

File	Lines	What it holds
backend/app/routers/adventures.py	2353	Nine concerns: the turn engine, branches, takes, import and export, adventure scripts, refresh from scenario, insights, memory CRUD, and action CRUD
frontend/src/pages/Play.jsx	2280	27 components, plus a 722-line Play() holding 18 useState calls and 7 useEffect calls
frontend/src/index.css	2865	One stylesheet. The max-width: 720px block at line 2655 overrides rules written 2000 lines above it
backend/app/worldstate/engine.py	918	Four jobs: parse a delta, apply a delta, render the context sections, and instantiate a schema

The schema is half migrated

plan/14-phase-story-tree.md defines SP8, which drops the columns the tree replaced. SP8 was gated on the tree running in production. It now does. Eight columns are still written on every turn and read by nothing:

• actions.index, variants, variant_index, variant_count
• actions.state_before, world_state_before
• adventures.memory_cursor, summary_cursor

Until they go, models.py carries two vocabularies for one idea, and every reader has to be told which one is live.

The published guide is a hand-written copy

docs/guide.html is 84 KB of hand-written HTML covering the same material as the 63 KB docs/GUIDE.md. Nothing generates one from the other. They have already drifted: the HTML last changed on 2026-08-20 and the Markdown on 2026-08-28, and section 3.6, "Counting visits", exists only in the Markdown. GitHub Pages publishes the HTML, so readers get the stale copy. docs/architecture.html is 87 KB of hand-written HTML with no Markdown source at all.

Duplication the tests already protect

• worldstate.apply_delta and apply_override each re-implement the same path routing for flags., milestones., and npc.<id>.. That is six parallel branches at engine.py:573, 587, 617, 675, 697, and 740.
• 35 of the 38 test files repeat the same 8-line temporary-database prologue. There is no conftest.py.
• ScriptedProvider or FakeProvider is defined 12 times, once per test file that needs a fake model.
• The debounced autosave helper is copied into Play.jsx:204, ScenarioEditor.jsx:23, and ScriptEditor.jsx:22.
• get_adventure_or_404 is called by hand in about 20 handlers. It is a function, not a dependency, so every handler repeats the call.
• routers/chat.py:22 imports SSE_HEADERS and sse from routers/adventures.py. One router reaches into another for shared plumbing.

One setting is documented nowhere

AIDND_TRUSTED_PROXY_HOPS is read at limits.py:55. It does not appear in backend/.env.example, README.md, or render.yaml. It sets how many proxy hops client_ip() trusts, which is what stops a rotated X-Forwarded-For header from buying a fresh rate-limit bucket. If a deployment adds a proxy hop and nobody sets this variable, the bypass comes back silently.

The unprefixed DATABASE_URL that database.py:24 also accepts turned out to be documented already, inside the AIDND_DATABASE_URL entry in backend/.env.example. No change needed there.

Settings.stream is dead

models.py:622 defines the column, and schemas.py:452 and 499 expose it. Nothing reads it in the backend or the frontend. This is item S1 in docs/self-review.md.

Local clutter

Everything here is gitignored, so it costs repository weight nothing. It costs disk and attention.

• Two abandoned worktrees, each a full copy of the tree: .claude/worktrees/simplify-comments and .claude/worktrees/sp7-tree-ui. Both branches landed on main as squashes.
• 21 local branches. 13 read as unmerged to git branch --no-merged, and plan/STATUS.md already records that they landed as squashes.
• Three stale SQLite files in backend/: data.backup-2026-07-06.db, data.backup-pre-phase8.db, and scroll_fixture.db, 2.9 MB together. The live local database, backend/data.db, is not one of them and is not touched.

Stage 0: hygiene

No source changes. Do this first, so later stages run against a quiet tree.

1. Remove both worktrees with git worktree remove, then delete the branches that landed as squashes. Verify each one first: a branch is safe to delete when git log main..<branch> names only commits whose message already appears in git log main.
2. Delete the three stale .db files. Leave backend/data.db alone.
3. Add AIDND_TRUSTED_PROXY_HOPS to backend/.env.example, with the reason it exists. Add it to the deploy section of README.md too, because that is where somebody putting the app behind another proxy looks.

Check: the suite still passes, and git status is clean.

What Stage 0 actually did, 2026-08-29

Both worktrees are gone, which freed about 104 MB. Removing sp7-tree-ui needed one extra step: a Vite dev server had been running out of that worktree since 2026-08-18, holding frontend/.vite open and owning port 5173. It was serving a tree 54 commits behind main. Stopping it freed the directory. Restart the real one with start.ps1.

The three stale .db files are deleted. backend/data.db is untouched.

AIDND_TRUSTED_PROXY_HOPS is now in backend/.env.example and in the README deploy section.

Still owed. The 19 squash-landed branches are still there. git branch -D is blocked by the permission classifier, which is a reasonable guard on a destructive command. All 19 are verified safe by the rule above. Run this to clear them:

git branch -D bugfix-code-review docs-story-tree fix-prepend-autoscroll \
  fix-silent-clamps-and-milestone-ids fix-store-refusals-and-delta-wording \
  handover-2026-08-17 measure-post-vacuum-sizes phase-14-story-tree \
  phase-7-public-repo phase-8-accounts sp1-tree-schema sp10-memory-bank-eviction \
  sp2-branch-clause sp3-node-cursors sp4-sibling-nodes sp5-fork-on-continue \
  sp6-bundle-v2 sp7-tree-ui sp7b-take-pager worktree-keep-fixture-and-status \
  worktree-simplify-comments

worktree-keep-fixture-and-status is the one that needed checking by hand. Its commit subject appears nowhere in main, but the --keep flag it adds is on main at tools/stress_session.py:764, along with both gotchas the message describes, at :686 and :692. Only the message differs.

The remote branches are left alone. Deleting those is a separate decision.

Stage 1: split the four largest files

Every change in this stage moves code. None of it changes behavior. The 549 tests are the check, and they must pass without being edited, except where this section says otherwise.

routers/adventures.py becomes a package

Split it into backend/app/routers/adventures/:

Module	Holds	Source lines
__init__.py	The APIRouter, the shared dependencies, and re-exports
paging.py	ACTION_LIST_COLUMNS, ACTION_PAGE, action_window, annotate_takes, current_window	27-146, 1203-1261
crud.py	List, create, get, patch, delete, plus the script-state and world-state readers	225-580
turns.py	The turn engine: generate_turn, _generate_turn, run_player_turn, the turn lock, and the SSE helpers	581-1004
takes.py	Retry, variants, takes, forking, and undo	1005-1192, 1503-1778
branches.py	The branch endpoints	1193-1502
bundle_io.py	Export and import	1779-1837
scripts.py	The per-adventure script endpoints	1838-1941
refresh.py	Refresh from scenario	1942-2152
insights.py	The context dry run and the per-action context	2153-2189
memories.py	Memory bank CRUD	2190-2279

Target no file above 450 lines.

The tests couple to the module, so read this before you start. Twelve test files call monkeypatch.setattr(adventures, "OpenAICompatibleProvider", ...). monkeypatch replaces a name in the module where the calling code looks it up, so re-exporting from __init__.py does not keep those patches working. Once _generate_turn lives in turns.py, the target becomes adventures.turns.OpenAICompatibleProvider.

Do Stage 2's conftest.py work first if you want that to be a one-line change instead of twelve. Otherwise retarget all twelve here. The other patched names are adventures.limits, adventures.check_demo_cap, adventures.generate_turn, and adventures._active_turns, each used once.

Keep these importable from the package root, because tests import them by name: ACTION_PAGE, SNIPPET_MAX, _snippet, world_delta_of, acquire_turn_lock, retry_action, undo_turn, and _active_turns.

_active_turns is module-level mutable state guarded by a lock. It must live in exactly one module, turns.py, and every other module must import the module and reach through it. If two modules import the set by value, the lock guards two different sets and the turn lock stops working.

What the router split actually did, 2026-08-29

backend/app/routers/adventures.py is now a package of 14 modules. The largest is turns.py at 443 lines. Four modules exist that the table above does not list, because the plan's eleven still mixed unrelated work:

Extra module	Why it exists
deps.py	Holds the APIRouter and the ownership check. It imports nothing else in the package, so every endpoint module can import the router without importing its siblings.
scenario_text.py	Copying a scenario's text and cards has two callers, crud.create_adventure and refresh. Leaving it in either one made the other import an endpoint module.
nodes.py	Story-tree navigation that four modules use: last_action, next_index, next_depth, stand_on, db_tip, delete_turn.
actions.py	The three action endpoints. They page and delete rather than play a turn, so they do not belong in crud.py.

Two decisions differ from the plan above.

The package root does not re-export acquire_turn_lock or _active_turns. The plan said to keep them importable, but that makes a broken patch look like a working one. Rebinding adventures.generate_turn changes the alias and leaves every caller reading the original, and the test still passes. Leaving those names off the package root raises AttributeError instead. Eighteen test call sites and four in backend/tools/ now say adventures.turns.<name>. The package root still re-exports the pure helpers, so adventures.ACTION_PAGE, adventures.undo_turn, and chat.py's from .adventures import SSE_HEADERS, sse are unchanged.

world_delta_of lives in turns.py. It reads as a world-state helper and sat beside the world-state endpoints, but _generate_turn is its only caller.

_active_turns behaved as the plan warned. Every module reaches it as turns._active_turns, and a check confirms the four modules see one set object and one lock.

One test coupled to the router for an unrelated module: it called adventures.worldstate.instantiate. It imports app.worldstate directly now.

The split moved text rather than retyping it. An AST comparison against the pre-split file confirms all 86 definitions are identical, once the turns. prefix is normalized away. The 549 tests pass, and the OpenAPI schema still lists the same 35 operations.

worldstate/engine.py becomes a package

Split backend/app/worldstate/engine.py into four modules under backend/app/worldstate/:

• schema.py: has_schema, instantiate, reconcile, band_label, npc_name, npc_triggers, _initials.
• parse.py: extract_delta, _tolerant_load, render_delta_block, applied_delta, refusals, render_refusals.
• apply.py: apply_delta, apply_override, and the shared path resolver Stage 2 introduces.
• render.py: render_state_section, render_reference, _stat_line, _describe_stat.

worldstate/__init__.py re-exports every public name it exports today, so no call site changes. The engine is imported as a module, not monkeypatched, so this split carries none of the router's test coupling.

pages/Play.jsx becomes a directory

Split it into frontend/src/pages/Play/:

• index.jsx: the page component.
• usePlaySession.js: the 18 useState calls and 7 useEffect calls that drive one adventure, behind one hook.
• panels/: PlotPanel, MemoryPanel, ScriptsPanel, InsightsPanel, BranchPanel.
• drawers/: StatusDrawer, WorldStateDrawer, and the StatRow, StatGroup, StateTree, and StateValue parts they use.
• reports/: StateChangeChips, WorldStateReport, ScriptReport, CacheReport, TokenBreakdown.
• TakePager.jsx, RefreshModal.jsx.

There is no frontend test runner until Stage 5, so this split is verified by npm run lint, npm run build, and by driving the Play screen in a browser. Drive it. plan/STATUS.md records that the last two Play bugs were both found by hand and were unreachable from any test that existed.

index.css becomes a directory

Split it into frontend/src/styles/, imported in order from index.css: tokens.css, base.css, nav.css, cards.css, play.css, drawers.css, schema-editor.css, insights.css, modals.css, and theme.css.

Move each @media block next to the rules it overrides, rather than leaving one max-width: 720px block at the end. Keep the source order identical when you move rules, because CSS resolves ties by order and the file relies on that in at least two known places, recorded in plan/STATUS.md and in the comments at index.css:1011 and 1082.

Check: 549 tests pass, npm run lint and npm run build are clean, and the Play screen works in a browser at desktop and at 500 px wide.

Stage 2: remove duplication

Each item here is small and is covered by tests that already exist.

1. One path resolver in worldstate. apply_delta and apply_override route flags.<name>, milestones.<id>, npc.<id>.<stat>, world.<stat>, and player.<stat> with parallel code. Extract a resolver that returns the container, the stat definition, and the kind, then let the two functions differ only in the write rule. Their rules genuinely differ, so do not merge the functions themselves. apply_override ignores cooldown, max_delta_per_turn, and the rule that a counter cannot decrease, and it lets a milestone be un-set. tests/test_worldstate.py covers both.
2. Add backend/tests/conftest.py. Move the temporary-database prologue there, so the other 35 files drop 8 lines each. The prologue has to run before from app.main import app, because main.py calls bootstrap(engine) at import time. A conftest.py runs before any test module, which satisfies that.
3. One fake provider. Move ScriptedProvider and FakeProvider to conftest.py as one fixture, replacing 12 copies.
4. Move the SSE helpers out of the router. Put sse, SSE_HEADERS, and turn_error in backend/app/sse.py, so chat.py stops importing from adventures.
5. Make get_adventure_or_404 a dependency. About 20 handlers repeat the call. A Depends removes the line from each one and puts the ownership check in the signature, where a reader looking for it expects it.
6. One useDebouncedSave hook in frontend/src/hooks/, replacing the three copies.
7. Delete Settings.stream. Remove the column, the two schema fields, and add the migration that drops it. This is item S1 in docs/self-review.md. Mark it applied there.

Check: 549 tests pass. Test count may drop if consolidating fixtures removes a duplicate case. If it does, say which case and why in the commit message.

Stage 3: SP8, drop the legacy columns

This is the only stage that touches the production database. Follow plan/14-phase-story-tree.md, which specifies it.

1. Confirm nothing reads the eight columns. ACTION_LIST_COLUMNS in the adventures package lists index, variant_count, and variant_index today, so that tuple changes here. bundle.py:37 and context/history.py:424 both describe index as unread; verify that rather than trusting the comment.
2. Add the migration that drops them. Remove the columns from models.py, and remove the fields from schemas.py and from ActionOut.
3. Deploy, then run VACUUM FULL actions; on the direct Neon endpoint, not the -pooler one. Dropping a column rewrites toasted values, and nothing reclaims that space on its own. plan/STATUS.md records what happened the last time nobody ran it: the database reached 144.2 MB against a 512 MB tier.
4. VACUUM FULL takes an ACCESS EXCLUSIVE lock, so the app blocks on actions for the duration. It took 5.5 seconds at 144 MB.

Check: 549 tests pass, /api/health answers {"ok":true} after the deploy, and one existing adventure opens, takes a turn, retries it, and pages back through the transcript.

Stage 4: documentation

1. Generate docs/guide.html from docs/GUIDE.md. Write a small build script that renders the Markdown into the existing hand-written HTML shell, keeping the current styles and metadata. After that, one edit updates both. Note in README.md that the HTML is generated and that you edit the Markdown.
2. Decide what docs/architecture.html is. It has no Markdown source. Either give it one and generate it the same way, or state at the top of the file that it is hand-written, so the next person does not look for a source that does not exist.
3. Split plan/STATUS.md. It is 63 KB, and most of it is dated session logs. Keep "Pick up here", "Things worth remembering", and "Running things" in STATUS.md. Move the dated entries to plan/history/, newest file first.
4. Add docs/DEVELOPING.md. Record how to run the app, how to run the tests, where each subsystem lives, and the invariants a newcomer breaks first:
   • Anything that reads adventure.actions during generation takes exclude_action_id. It leaks into four places, not one.
   • memory_cursor and summary_cursor are positions into story_actions(), and Memory.source_start and source_end are Action.index values. The two spaces diverge as soon as anything is deleted.
   • /auth/me must not raise. It is the SPA's bootstrap call, so anything it touches that can raise takes the whole frontend down.
   • The global field rule in the stylesheet keys off input[type=...], so a bare <input> and any new input type get browser default styling until you add them.
   • A modal opened from .side-panel needs createPortal, because the panel's filling transform animation makes it the containing block for position: fixed.

Check: the generated HTML matches the Markdown section for section, including section 3.6. Every link in README.md and docs/index.html resolves.

Stage 5: a frontend test runner

plan/STATUS.md names the missing runner as the reason this project keeps finding UI bugs by hand. Two shipped bugs were unreachable from any test that existed.

1. Add Vitest, React Testing Library, and jsdom. Add a test script to frontend/package.json.
2. Add the step to the existing frontend job in .github/workflows/ci.yml, between lint and build.
3. Write the first tests against the two bug classes that already recurred:
   • The take pager renders when a retry's reply arrives over SSE. The stream builds its own ActionOut, so it was the one payload that never carried the annotation.
   • Retaking a player turn does not produce > You > You. The editor is seeded with stored text that is already formatted.
4. Add a test for usePlaySession from Stage 1, since that hook now holds the state the page used to hold inline.

Check: npm test passes locally and in CI.

Risks

Risk	Where	How you catch it
A monkeypatch silently stops patching, and a test passes while calling a real provider	Stage 1, routers/adventures split	After retargeting, break the fake on purpose and confirm the tests that use it fail
_active_turns ends up imported by value into two modules, so the turn lock guards two sets	Stage 1	tests/test_state_revert.py reaches for adventures._active_turns. Keep that import path working, and confirm a second concurrent turn still returns 409
A CSS rule changes meaning because the split reorders it	Stage 1, stylesheet split	Compare the built CSS before and after. Order within each section must not change
The column drop rewrites the table and nobody reclaims the space	Stage 3	Run VACUUM FULL actions; on the direct endpoint, and read sizes from sum(octet_length(col)) rather than n_live_tup
Splitting Play.jsx breaks something no test covers	Stage 1	Drive the Play screen by hand at both widths. Stage 5 exists to shrink this risk for next time

What this phase does not do

• It does not replace the hand-rolled migration runner with Alembic. 64 migrations run correctly, and docs/GUIDE.md section 2.6 already explains the choice.
• It does not move bootstrap(engine) out of import time in main.py. The current behavior means a failed migration means no service, which is deliberate. Stage 2's conftest.py removes the boilerplate that import-time bootstrapping forces on tests, which is the part that actually hurts.
• It does not change any API shape, any database content, or any prompt.