Update planning package after Phase 0B

planning/BUILD-MILESTONES.md

@@ -1,163 +1,551 @@
-# Adventure Storyteller — Build Milestones
+# Adventure Storyteller — Production Build Milestones
 
-**Status:** Placeholder / intentionally incomplete  
-**Do not use for production implementation yet.**
+**Status:** Planning-ready; implementation not yet authorized  
+**Base:** AI-DnD `d72f7c1bda0f34fccd84afb7a25c34eb01c901de`
 
 ## 1. Purpose
 
-The detailed production implementation plan will be created at the end of **Phase 0 — Research, Validation & Architecture**.
+This document defines the production implementation sequence after Phase 0.
 
-A precise plan cannot responsibly be written before the project has selected:
+It is intentionally a milestone plan, **not a Codex execution prompt**. Each milestone should be converted into a separate coding brief only after the previous milestone has been reviewed and accepted.
 
-- the base repository or build strategy,
-- the final persistence/story-tree design,
-- the final browser architecture,
-- the Ollama integration model,
-- the memory/retrieval strategy,
-- the local-only hardening approach,
-- the migration/reuse plan for inherited code.
+The milestone order prioritizes load-bearing correctness and offline safety before broad feature work.
 
-## 2. Why This Document Is Deliberately Limited
+## 2. Global Rules for Every Milestone
 
-Different fork choices create fundamentally different engineering work.
+Each implementation milestone must:
 
-Example:
+- preserve upstream provenance and licensing notices,
+- keep the application runnable whenever practical,
+- add or update automated tests for changed behavior,
+- avoid unrelated refactors,
+- update affected documentation,
+- record schema/migration changes,
+- preserve local-only defaults,
+- stop at the milestone boundary for review,
+- avoid implementing later milestones opportunistically.
 
-### If Open Dungeon is selected
-Early milestones may require:
+The product specification and acceptance tests outrank convenience inherited from AI-DnD.
 
-- adding immutable story-tree persistence,
-- adding branch-aware state restoration,
-- introducing structured narrative state,
-- adding long-term semantic memory.
+## 3. End-User Capability Progression
 
-### If AI-DnD is selected
-Early milestones may instead require:
+The application is expected to remain runnable throughout the sequence. **M8 does not create the browser UI from scratch**; AI-DnD already provides a browser interface from M1. M8 is where that inherited/adapted interface is completed and simplified into the intended v1 storyteller experience.
 
-- removing RPG mechanics,
-- removing cloud providers,
-- removing account/hosted assumptions,
-- simplifying world state while preserving story-tree behavior.
+- **M1 — First playable baseline:** The user can open the inherited browser UI, start or resume a story, send prompts to Ollama, receive streamed narration, and persist the story across restart. The interface may still look and behave substantially like AI-DnD, and advanced history/state features are not yet converted to final semantics.
+- **M2 — Clean local storyteller baseline:** Normal story play still works, but cloud/hosted/account/scripting surfaces are removed and Ollama can be either same-host or on an explicitly configured trusted-LAN machine. From the user's perspective this is still basic play, but the environment now matches the intended privacy/deployment model.
+- **M3 — Safe history controls:** The user can play normally with production-grade Undo, Redo, Retry, alternate takes, and divergence without deleting old story history. Export/import also preserves the exact current position even when the user has undone several turns.
+- **M4 — Save Points:** The user can name a point in the story, continue playing, restart, return to that Save Point, and take a different path without losing the later story. This is the first milestone where deliberate long-form experimentation/recovery should feel comfortable.
+- **M5 — Reliable story state:** Characters, locations, possessions, relationships, facts, threads, and current scene become application-owned genre-neutral state rather than RPG-stat machinery. The user can begin inspecting/correcting what the storyteller believes, although the final polished state UI comes later.
+- **M6 — Long-story memory:** The storyteller becomes much better suited to long campaigns because summaries and retrieved older memories remain branch-safe and inspectable. The user should be able to return to old people, promises, clues, and events without abandoned paths contaminating the active story.
+- **M7 — Local knowledge library:** The user can import `.txt`/`.md` material as Canon, Reference, or Inspiration and have it retrieved locally with provenance. The functionality is usable, but some management/inspection surfaces may still be utilitarian until M8.
+- **M8 — Finished v1 browser experience:** The existing browser UI is reorganized and polished around storytelling: streamlined campaign setup, transcript/input, Undo/Redo/Retry, Save Points, state, knowledge, context inspection, and model status. This milestone makes the product feel like the intended storyteller rather than an adapted AI-DnD application.
+- **M9 — Portable/recoverable campaigns:** The user can reliably export, back up, import, and recover complete campaigns including history, Save Points, state, knowledge, and active position. This is where moving or restoring a campaign becomes a supported user workflow rather than merely an underlying capability.
+- **M10 — Media-ready, still text-first:** Little or nothing visibly changes for normal play. The story/scene data and provider boundaries are prepared so image/video/audio/TTS/STT can be added later without redesigning the core application.
+- **M11 — Release-quality v1:** The user experience should be functionally complete; this milestone proves it stays correct through long campaigns, repeated history operations, offline use, failures, export/import, and multiple genres. It is the release-validation milestone rather than a major new feature milestone.
 
-### If ai-adventure is selected
-Early milestones may instead require:
+---
 
-- adding an Ollama adapter,
-- adding a browser API,
-- building the browser UI,
-- extending lore retrieval beyond current behavior.
+# M1 — Establish Production Fork and Offline Baseline
 
-A single detailed build plan written now would therefore contain false precision.
+## Objective
 
-## 3. Expected High-Level Production Phases
+Create the production fork from the pinned AI-DnD commit and make ordinary local story operation genuinely offline-capable before larger changes.
 
-These are directional only and must be rewritten after Phase 0.
+## Scope
 
-### Phase 1 — Production Foundation
-- establish production fork/repository,
-- preserve upstream provenance,
-- remove or isolate unwanted functionality,
-- establish development/test environment,
-- confirm local Ollama integration.
+- establish production repository/fork lineage,
+- record upstream commit and license provenance,
+- establish reproducible dev/test environment,
+- vendor/cache/replace the `tiktoken` first-use encoding dependency so story generation works without Internet,
+- eliminate runtime Google Fonts/remote font dependency,
+- tighten CSP for local runtime assets,
+- verify same-host Ollama story generation with outbound Internet blocked,
+- verify explicitly configured trusted-LAN Ollama story generation while the storyteller UI/API remains loopback-bound,
+- establish baseline regression/test report.
 
-### Phase 2 — Authoritative Story Persistence
-- immutable/recoverable turn history,
-- branch parentage,
-- checkpoints,
-- restore,
-- retry/edit semantics,
-- transactional commits.
+## Explicit Non-Scope
 
-### Phase 3 — Narrative State
-- generic entities,
+- no history rewrite yet,
+- no RPG-state redesign,
+- no imported knowledge,
+- no UI redesign,
+- no media generation.
+
+## Tests / Acceptance
+
+Must demonstrate:
+
+- A01-A06 as applicable to the inherited base, including the separate-host trusted-LAN inference path,
+- H01-H03 and H11,
+- no unexpected DNS/HTTP requests during ordinary story generation after setup,
+- no remote font request,
+- no tokenizer/BPE download on first production turn,
+- existing relevant AI-DnD tests remain green except documented upstream/environment exceptions.
+
+## Definition of Done
+
+A clean production build can start, open the browser UI, generate and persist story turns through either same-host Ollama or an explicitly configured trusted-LAN Ollama host, restart, and resume with outbound Internet blocked. The storyteller UI/API remains loopback-bound by default.
+
+---
+
+# M2 — Remove Hosted, Cloud, Scripting, and Unneeded Deployment Surface
+
+## Objective
+
+Reduce AI-DnD to the intended single-user local product trust boundary without destabilizing the story/tree/memory foundation.
+
+## Scope
+
+Remove or isolate as appropriate:
+
+- multi-user/account/guest/auth flows,
+- demo API-key behavior,
+- hosted analytics,
+- Render/Neon deployment paths,
+- Postgres/`psycopg` support,
+- cloud providers not required by v1,
+- arbitrary remote model-provider UI/configuration,
+- QuickJS/campaign scripting,
+- AI-Dungeon compatibility code that exists primarily to support scripting/hosted behavior,
+- hosted-only rate-limit/account infrastructure.
+
+Add:
+
+- explicit Ollama endpoint policy: same-host loopback default plus user-configured trusted-LAN inference,
+- clear local-model connection diagnostics,
+- migration/test instrumentation replacements for any tests that depended on JS hooks or removed hosted paths.
+
+## Explicit Non-Scope
+
+- do not redesign story history,
+- do not replace RPG state yet,
+- do not build knowledge/media systems.
+
+## Tests / Acceptance
+
+- local story play still works,
+- existing tree/retry/memory/context behavior remains intact,
+- application requires no cloud API keys,
+- approved trusted-LAN Ollama endpoints work while arbitrary public/Internet provider endpoints are rejected or absent from normal production configuration,
+- removed provider/auth/analytics/scripting paths are no longer reachable from normal production configuration.
+
+## Definition of Done
+
+The codebase has a narrow single-user/local-only surface and the inherited story foundation still passes its relevant regression suite.
+
+---
+
+# M3 — Production Non-Destructive History, Redo, and Active-Head Export
+
+## Objective
+
+Replace destructive Undo with the demonstrated head-cursor model and make Undo/Redo/divergence/export/import production-safe.
+
+## Scope
+
+Promote the Phase 0B spike concept into maintainable production code:
+
+- active head can move behind retained tip,
+- Undo moves head and deletes zero accepted turns,
+- Redo follows the retained continuation,
+- new write behind tip forks on first write,
+- retry/add-take/edit paths use the same safe fork/head rules,
+- prevent Undo from walking before the campaign opening/root semantics,
+- mark displaced/inactive futures/takes as retained/disposable using implementation-appropriate metadata,
+- keep ordinary Redo invalidated after divergence,
+- add browser Redo control and correct disabled/enabled states,
+- keep branch complexity out of normal UI,
+- export active head coordinate/depth as well as active branch,
+- import honors exported active head,
+- preserve backward compatibility for older bundles that lack the new field.
+
+## Explicit Non-Scope
+
+- named checkpoints belong in M4,
+- no general branch-tree UI,
+- no abandoned-history cleanup feature.
+
+## Tests / Acceptance
+
+Must cover:
+
+- D01-D10,
+- D02 minimum-five Undo,
+- D03 practical unlimited Undo behavior across retained history,
+- D04-D05 Redo semantics,
+- E01-E04 lineage isolation,
+- I01-I03 and I07 (undone-head round trip),
+- L01-L02,
+- branch-scoped memory remains isolated under Undo/Redo/divergence,
+- row counts demonstrate Undo deletes zero accepted turns.
+
+## Definition of Done
+
+History operations are non-destructive, Redo works, divergence preserves old futures, and export/import reopens at the exact active head.
+
+---
+
+# M4 — Named Save Points / Checkpoints
+
+## Objective
+
+Add durable user-facing Save Points on top of the active-head model.
+
+## Scope
+
+- named checkpoint storage,
+- checkpoint create/list/rename/delete,
+- restore by moving the active head,
+- later history retained rather than deleted,
+- fork only on first new continuation after restore,
+- checkpoint persistence across restart,
+- checkpoint export/import,
+- browser Save Point UX per `BROWSER-UX-SPEC.md`.
+
+## Explicit Non-Scope
+
+- no branch merge,
+- no automatic discarded-history cleanup,
+- no complex branch explorer required.
+
+## Tests / Acceptance
+
+- D11-D14,
+- E lineage tests after checkpoint restore/divergence,
+- I04,
+- L03.
+
+## Definition of Done
+
+The user can create a named Save Point, continue, restart, restore it, and continue differently without losing later history.
+
+---
+
+# M5 — Genre-Neutral Authoritative Narrative State
+
+## Objective
+
+Replace/generalize AI-DnD's RPG-specific relative-delta state system with the approved genre-neutral typed-event model.
+
+## Scope
+
+Establish generic authoritative state for:
+
+- entities,
+- characters/locations/organizations/items/vehicles as descriptive entity categories,
 - facts,
 - relationships,
+- possession/location/status/conditions,
 - story threads,
-- state extraction/validation,
-- state inspector.
+- scene state,
+- manual state/canon corrections,
+- state proposal provenance.
 
-### Phase 4 — Long-Term Memory
-- summaries,
-- older-turn retrieval,
-- token budgeting,
-- provenance,
-- continuity handling.
+Implement:
 
-### Phase 5 — Local Knowledge Library
-- local file imports,
-- Canon / Reference / Inspiration classes,
-- chunking,
-- local indexing,
-- optional local embeddings,
-- retrieval inspection.
+- explicit typed state proposal schema,
+- absolute/unambiguous event semantics,
+- event allowlist,
+- validation,
+- atomic accepted-event commit,
+- current/historical snapshot/cache,
+- rollback/reconstruction tied to active lineage,
+- browser current-state inspector foundation.
 
-### Phase 6 — Browser UX Completion
-- campaign management,
-- transcript,
-- branching visualization,
-- checkpoints,
-- state/editor,
-- library management,
-- prompt inspection,
-- responsive local UI.
+Remove or demote:
 
-### Phase 7 — Local-Only Hardening
-- remove remote providers,
-- remove telemetry/analytics,
-- remove runtime CDN dependencies,
-- enforce/validate local endpoints,
-- network tests,
-- offline operation tests.
+- D&D-specific stats/bands/cooldowns/mechanics from the core product model,
+- relative-delta semantics as the generic state protocol.
 
-### Phase 8 — Export, Backup, and Recovery
-- campaign export,
-- import,
-- backups,
-- migration,
-- corruption/error recovery.
+## Explicit Non-Scope
 
-### Phase 9 — Future-Media Hooks
-- scene snapshots,
-- visual character/location descriptors,
-- asset schema,
-- media-provider interfaces,
-- no required image/video implementation.
+- no optional RPG module,
+- no imported knowledge yet,
+- no final rich state-editing UX if a simpler inspector is enough for this milestone.
 
-### Phase 10 — v1 Validation and Release
-- regression testing,
-- long-story testing,
-- rollback/branch tests,
-- offline test,
-- migration test,
-- documentation,
-- release packaging.
+## Tests / Acceptance
 
-## 4. Gate Before This Plan Becomes Active
+- C01-C04 and C06,
+- D/E tests verifying state follows head movement,
+- H05 invalid state event rejection,
+- L01-L02,
+- malformed JSON/state proposal handling,
+- realistic-context tests against representative local models,
+- fantasy and science-fiction state fixtures use the same schema.
 
-Do not convert the high-level phases above into Codex implementation prompts until all of the following exist:
+## Definition of Done
 
-- `SPECIFICATION.md` v1.0,
-- `TECHNICAL-DESIGN.md` v1.0,
-- completed Phase 0 research reports,
-- approved fork/build ADR,
-- approved licensing/reuse review.
+Accepted story state is genre-neutral, auditable, reconstructable, and no longer depends on ambiguous relative deltas.
 
-## 5. Required Format for the Final Build Plan
+---
 
-When rewritten after Phase 0, every production milestone should contain:
+# M6 — Branch-Safe Context, Summaries, and Long-Term Story Memory
 
-- objective,
-- scope,
-- explicit non-scope,
-- prerequisite milestones,
-- files/components expected to change,
-- implementation tasks,
-- data/schema changes,
-- tests required,
-- security/privacy checks,
-- acceptance criteria,
-- rollback/migration notes,
-- documentation updates,
-- definition of done.
+## Objective
 
-The final document should be suitable for handing directly to Codex one milestone at a time.
+Align inherited AI-DnD memory/context behavior with the final authority and history model.
+
+## Scope
+
+- preserve recent active-lineage history selection,
+- ensure summaries are anchored to source turn ranges/lineage rather than positional list assumptions,
+- preserve branch-scoped memory isolation,
+- classify memory authority (accepted vs heuristic/inferred),
+- maintain explicit token budgets,
+- preserve/extend prompt-context inspection,
+- ensure memory/summary failures do not corrupt accepted story state,
+- test memory behavior after Undo/Redo/checkpoint/divergence,
+- validate realistic-context prompt construction.
+
+## Explicit Non-Scope
+
+- imported document knowledge belongs in M7,
+- no remote embeddings/vector DB.
+
+## Tests / Acceptance
+
+- F01-F08,
+- E lineage safety,
+- no abandoned-future term appears in active prompt after divergence,
+- negative control proves eligible memory returns when the relevant lineage is active,
+- summary lineage survives head movement correctly,
+- prompt inspector identifies included memories/summaries and token costs.
+
+## Definition of Done
+
+Long-running story context is lineage-safe, authority-aware, local, inspectable, and bounded.
+
+---
+
+# M7 — First-Class Imported Knowledge Library
+
+## Objective
+
+Implement the separate local knowledge subsystem required by the specification rather than overloading AI-DnD Story Cards.
+
+## Scope
+
+- `.txt` and `.md` import,
+- source validation and local copy/storage,
+- Canon / Reference / Inspiration classification,
+- enable/disable/delete,
+- content hash and provenance,
+- heading/paragraph-aware chunking,
+- SQLite FTS5 lexical index,
+- local Ollama embeddings where semantic retrieval is enabled,
+- hybrid retrieval/reranking,
+- authority-aware context insertion,
+- campaign scoping,
+- source/chunk inspector,
+- prompt retrieval provenance,
+- export/import preservation,
+- no automatic URL/image fetch,
+- prompt-injection framing as untrusted data.
+
+## Explicit Non-Scope
+
+- PDF/DOCX/EPUB import,
+- remote URL ingestion,
+- remote vector stores,
+- general plugin/tool framework.
+
+## Tests / Acceptance
+
+- G01-G10,
+- C05 canon beats reference/inspiration,
+- hidden/abandoned story facts cannot leak through lineage-derived knowledge,
+- H06-H09 as applicable to rendering/import/archive handling,
+- I05 knowledge provenance survives export/import.
+
+## Definition of Done
+
+A campaign can import local Canon/Reference/Inspiration files, retrieve them locally with provenance, and maintain authority boundaries.
+
+---
+
+# M8 — Browser UX Completion for v1 Story Operations
+
+## Objective
+
+Turn the adapted AI-DnD interface into the focused interactive-story workspace defined by `BROWSER-UX-SPEC.md`.
+
+## Scope
+
+- campaign library/setup streamlined for non-RPG stories,
+- story transcript and streaming polish,
+- one primary natural-language input flow plus story-direction affordance,
+- Undo/Redo/Retry/alternate-take controls,
+- Save Points UI,
+- edit user input/narrator output with safe lineage semantics,
+- current scene/state side panel,
+- knowledge panel,
+- context inspector,
+- local Ollama status/model selection,
+- no cloud-provider UI,
+- reserve microphone/STT affordance without implementing STT.
+
+## Explicit Non-Scope
+
+- no full branch-management UI,
+- no image/video/TTS/STT implementation,
+- no mobile-native app.
+
+## Tests / Acceptance
+
+- B01-B04,
+- D user-facing behavior,
+- relevant UX acceptance checks,
+- browser state remains consistent after restart/Undo/Redo/checkpoint/failed generation.
+
+## Definition of Done
+
+Normal story creation and play feels like a focused local storyteller rather than an RPG or developer console.
+
+---
+
+# M9 — Export, Backup, Recovery, and Migration Hardening
+
+## Objective
+
+Make campaigns portable and recoverable without losing lineage, state, knowledge, or the active head.
+
+## Scope
+
+- finalize documented campaign bundle format/versioning,
+- preserve active branch/head, retained history, retries, checkpoints,
+- state events/snapshots,
+- prompt/retrieval provenance as selected,
+- knowledge source metadata/content/index rebuild information,
+- scene/media metadata placeholders,
+- safe SQLite backup behavior,
+- import validation,
+- backward-compatible migration rules,
+- corruption/failure handling where practical.
+
+## Tests / Acceptance
+
+- I01-I06,
+- L01-L04,
+- undone-head round trip,
+- branched campaign round trip,
+- checkpoint round trip,
+- knowledge provenance round trip,
+- derived indexes can be rebuilt.
+
+## Definition of Done
+
+A campaign can be safely exported, imported into a clean data directory, and reopened at the exact intended active position with authoritative history/state intact.
+
+---
+
+# M10 — Future Media Extension Hooks Only
+
+## Objective
+
+Preserve the approved future media interfaces without adding a media-generation dependency to v1.
+
+## Scope
+
+- scene snapshots/packets suitable for future providers,
+- optional visual character/location/item descriptors,
+- provider-neutral media request/job/asset types or reserved schema as justified,
+- branch/lineage association for scenes/assets,
+- local-only endpoint contract for future providers,
+- STT contract: local transcription -> editable draft -> normal submission,
+- no core story-engine dependency on media availability.
+
+## Explicit Non-Scope
+
+Do not implement:
+
+- image generation,
+- video generation,
+- TTS,
+- STT,
+- ambience/audio generation,
+- ComfyUI/FLUX integration.
+
+## Tests / Acceptance
+
+- K01-K04 architecture/media-readiness tests,
+- no v1 story flow requires a media service,
+- scene data follows active lineage after Undo/restore/divergence.
+
+## Definition of Done
+
+Future media providers can be added through defined local interfaces without redesigning core story authority/history.
+
+---
+
+# M11 — v1 Security, Long-Run, and Release Validation
+
+## Objective
+
+Validate the full product against the release contract after all functional milestones are integrated.
+
+## Scope
+
+- full outbound-network blocked run,
+- dependency/runtime audit,
+- restrictive browser/CORS/CSP checks,
+- failed-model-call recovery,
+- long-running 100-turn campaign,
+- repeated Undo/Redo/Retry/checkpoint cycles,
+- branch/memory/summary leakage checks,
+- realistic-context state extraction across selected recommended models,
+- fantasy and science-fiction fixtures,
+- export/import/recovery tests,
+- migration tests,
+- documentation and packaging.
+
+## Tests / Acceptance
+
+Release gate in `V1-ACCEPTANCE-TESTS.md`:
+
+- all REQUIRED FOR V1 tests pass,
+- approved exceptions documented in ADRs,
+- 100-turn test passes,
+- offline operation passes,
+- branch/memory isolation passes,
+- export/import recovery passes,
+- both genre fixtures pass.
+
+## Definition of Done
+
+The build meets the v1 black-box acceptance contract and can be packaged as the first production release.
+
+---
+
+## 4. Milestone Dependency Summary
+
+```text
+M1 Production/offline foundation
+  -> M2 Local-only surface reduction
+      -> M3 Non-destructive history + export head
+          -> M4 Save Points
+              -> M5 Narrative state
+                  -> M6 Context/memory
+                      -> M7 Imported knowledge
+                          -> M8 Browser UX completion
+                              -> M9 Export/recovery hardening
+                                  -> M10 Future-media hooks
+                                      -> M11 v1 validation/release
+```
+
+Some implementation work may overlap internally, but milestone acceptance should remain sequential so architectural regressions are discovered early.
+
+## 4. Prompting Rule
+
+When implementation begins, prepare **one Codex prompt per milestone**.
+
+Do not hand the entire plan to Codex as one production task.
+
+Each prompt should contain only:
+
+- milestone objective,
+- relevant source documents,
+- scope/non-scope,
+- specific acceptance tests,
+- stop condition and required report/diff.
+
+No implementation prompt is included in the current planning revision.