Files

T

JesseMarkowitzandClaude Opus 5 480414efe0 M7: a first-class imported knowledge library

A campaign can import local .txt and .md files as Canon, Reference or
Inspiration, and the class is load-bearing rather than a label: it decides the
words a passage is framed with in the prompt, the weight it carries when
passages are ranked, and which budget it competes in when the context is tight.

This is a separate subsystem, which is the Phase 0B decision
(IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification,
provenance, content identity, chunking, an index or a lifecycle, and they were
not promoted into something that does. Nothing here reads or writes one.

The subsystem, in backend/app/knowledge/:

  classes      the three classes, their weights, and the prompt framing
  chunking     deterministic, heading-aware, 60-800 tokens, no overlap
  fts          SQLite FTS5 with porter stemming; scoped and bounded in SQL
  importer     validate, hash, store, chunk, index — in one transaction
  embeddings   local Ollama vectors through the shared provider
  retrieval    query construction, hybrid merge, rerank
  inject       the budgeted cut and the rendered prompt sections

Relevance admission is a separate stage from ranking, and that separation is
the milestone's most expensive lesson. An independent review found the first
implementation deciding relevance with a floor expressed as a share of the best
candidate — which the best clears by construction — so a passage was admitted on
every turn regardless of the scene. A query about tide tables and container
tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden
Canon among them.

So the pipeline is now:

  candidate generation -> admission -> ranking -> class weighting -> budget

Admission reads raw, candidate-set-independent signals: the cosine the model
returned, and how many distinct meaningful query terms a passage contains.
Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero
is not zero. Normalization decides order among things that matched; it can never
decide whether anything matched. Authority is applied after admission, so a
class orders what matched and never rescues what did not.

Retrieval may therefore return nothing, and on a scene unrelated to the library
it does.

The other decisions that each replaced an obvious wrong one:

- The class multiplies relevance rather than adding to it. An additive bonus
  satisfies "Canon outranks Reference" and makes "do not include irrelevant
  Canon" impossible, because a large enough constant wins on its own.
- The semantic floor is measured, not guessed: 113 production-path pairs against
  nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at
  0.36-0.56, and 0.58 sits between them. Because it is a property of that model
  and not of cosine similarity, it is keyed to the model rather than applied to
  whatever is configured: an embedding model with no measured calibration in
  this build does not borrow the number. Semantic admission is skipped, the
  campaign retrieves lexically, and the reason is stated in the knowledge status
  and in the turn's provenance. Degrading to lexical keeps the library usable;
  lending the threshold to an unmeasured model is how the admitted-everything
  defect would return.
- One lexical term is not evidence. Two distinct meaningful terms, or one that
  is neither a standing campaign entity nor a negligible share of the query.
  The stop list grew from 42 words to 261, all function words — no subject
  matter, because a stop list that removes subject matter stops finding "The
  Silver Key".
- Lexical retrieval is a production path, not a fallback. It finds the proper
  nouns and invented terms a setting bible is made of, and the library is fully
  usable with no embedding model configured.

Safety is structural rather than filtered. Imported text reaches the prompt
whole, inside a section that says what it is, under a rule stating the authority
order in words and refusing every instruction inside it. No endpoint accepts a
filesystem path, so H08 has no mechanism to escape from. Nothing renders
imported content as HTML, so a script tag is five visible characters and a
remote image is never fetched. Import, chunking, indexing, retrieval and a turn
open no socket at all; only embeddings do, through the endpoint allowlist the
memory bank already uses.

Provenance is the rendered text, not a foreign key: deleting a source cannot
turn a historical turn's evidence into dangling ids.

Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5
virtual table attached to knowledge_chunks as a DDL hook so it is created and
dropped with the table it indexes. Migration 92. A pre-M7 database opens
unchanged and needs no sources to play.

Bundle: the source content and the reader's judgements about it travel; the
passages, index rows and vectors are rebuilt on import, so a restored campaign
is searchable immediately without a reindex step.

One runtime dependency: python-multipart, Starlette's multipart parser. It is
what makes the upload surface possible, and the upload surface is why no
pathname is ever accepted.

The test doubles were the reason the defect shipped, so they were corrected too.
The retrieval stub scored unrelated text at 0.06-0.20 where the real model
scores it at 0.43-0.44, and its docstring said it had deliberately removed the
constant component that "would put a similarity floor under every pair" — which
is exactly the property real models have. The stub now has that floor, one test
fails if it is ever removed, and another reproduces the superseded rule and
asserts it is still fooled by the same fixture. Run against the pre-corrective
implementation, the new suite fails 13 of 18.

Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of
which mocks nothing between itself and Ollama and re-measures the similarity
separation on every run. 43/43 checks in a real Firefox, reproduced.
Docker build clean.

Four other defects found by review or by the browser run were fixed here rather
than carried: an unreachable relevance constant that appeared to enforce
something and did not; acceptance tests using the wrong fixture files, so G07's
trap was never exercised; a bidirectional override surviving into displayed
filenames; and, from the implementation pass, the Insights panel showing M5's
two state sections as raw keys and the source inspector refetching on every
keystroke.

M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK
REQUIRED. Both blocking findings are closed, and closeout resolved the
embedding-model calibration boundary the corrective pass had left as debt.
planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective
closeout and the closeout verification in sequence, none overwriting another.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b

2026-09-06 15:40:13 -04:00

README.md

Adventure Storyteller — Planning Package

This file is the index. Start here.

Current state: Phase 0 complete; AI-DnD forked as the production base; milestones M1 through M6 implemented and accepted (M3 and M4: 2026-09-03; M5: 2026-09-04; M6: 2026-09-06). M5 and M6 were each accepted only after an independent review found a real defect and a corrective pass fixed it.

M7 — First-Class Imported Knowledge Library — is complete (2026-09-06), after an independent review, a corrective pass and a closeout verification. Its report keeps all three in sequence: reports/M7-IMPLEMENTATION-REPORT.md. Next: M8 — Browser UX Completion for v1 Story Operations. Its brief has not been written yet, and writing it is the current action.

Package version: see VERSION.md, which records what each revision changed and why.

Where documentation lives

planning/
    active specifications and implementation planning — authoritative

planning/DECISIONS/
    active architectural decisions (ADRs) — authoritative

planning/reports/
    the current milestone's implementation report — evidence, not instruction

planning/archive/
    historical evidence and completed planning material;
    not authoritative for current implementation

You do not need to open planning/archive/ to do ordinary milestone work. Go there only when an active document sends you for a specific piece of historical evidence. If an archived document contradicts an active one, the active one is right.

Outside planning/: README.md describes the application, DEVELOPMENT.md is setup and local operation, and PROVENANCE.md records what came from upstream AI-DnD and what each milestone changed.

Which document controls

When two documents appear to conflict, the one higher in this list wins:

SPECIFICATION.md — product requirements and required behavior.
The detailed behavior/design documents: STORY-BRANCH-SEMANTICS.md, CONTEXT-AND-MEMORY.md, IMPORTED-KNOWLEDGE-DESIGN.md, SECURITY-THREAT-MODEL.md, MEDIA-EXTENSION-CONTRACT.md, BROWSER-UX-SPEC.md, DATA-MODEL.md.
V1-ACCEPTANCE-TESTS.md — the observable pass/fail contract.
TECHNICAL-DESIGN.md — the selected implementation architecture.
The ADRs in DECISIONS/.
BUILD-MILESTONES.md — implementation sequence. It sequences work; it does not override product behavior.
The current milestone report in reports/, then planning/archive/ — evidence and historical findings, authoritative over nothing.

Two standing qualifications:

Candidate repositories and research reports are not specifications. In particular, ai-adventure is an implementation reference for selected patterns; its behavior does not override this package.
Where an ADR records the architecture chosen to implement a requirement stated elsewhere, both stand: ADR 005 states the history requirement and ADR 012 states the architecture that implements it.

The active documents

Document	What it is for
`SPECIFICATION.md`	What the product must do. The top of the authority order.
`TECHNICAL-DESIGN.md`	The selected architecture, including what M1-M7 built, recorded as fact.
`DATA-MODEL.md`	Entities, the stored head, branch disposition, and the export shape.
`STORY-BRANCH-SEMANTICS.md`	Undo/Redo/Retry/branch/take behavior, including the M3 ratifications.
`CONTEXT-AND-MEMORY.md`	Prompt assembly, summarization, branch-safe memory.
`IMPORTED-KNOWLEDGE-DESIGN.md`	Canon / Reference / Inspiration knowledge as a first-class subsystem.
`SECURITY-THREAT-MODEL.md`	The trust boundary, and the inference endpoint policy as implemented.
`MEDIA-EXTENSION-CONTRACT.md`	The contract future image/video/audio/TTS/STT work must fit.
`BROWSER-UX-SPEC.md`	The browser surface, and what is deliberately not in it.
`BUILD-MILESTONES.md`	M1-M11, what each delivers, what is done, and the notes each milestone leaves its successors.
`V1-ACCEPTANCE-TESTS.md`	The pass/fail contract v1 is measured against.
`TEST-CAMPAIGN-FIXTURE.md`	The standard campaign the acceptance tests are run on.
`VERSION.md`	Package revision history: what each closeout changed.
`PROJECT-SOURCES.md`	Which files to upload as ChatGPT Project Sources.

Reading order for a new coding agent

This file.
SPECIFICATION.md
TECHNICAL-DESIGN.md
BUILD-MILESTONES.md — find the milestone you are being asked to do.
STORY-BRANCH-SEMANTICS.md
DATA-MODEL.md
CONTEXT-AND-MEMORY.md
IMPORTED-KNOWLEDGE-DESIGN.md
SECURITY-THREAT-MODEL.md
BROWSER-UX-SPEC.md
V1-ACCEPTANCE-TESTS.md
DECISIONS/ — all of them; they are short.
reports/M7-IMPLEMENTATION-REPORT.md, for what the last accepted milestone actually left behind. Nothing in planning/archive/ unless sent there.

Architectural decisions

Active ADRs, all of which still constrain current or future work:

ADR	Decision
`001-browser-first.md`	The UI is a browser application.
`002-ollama-only-v1.md`	Ollama-compatible local inference only; trusted-LAN HTTPS with a private CA is verified, never bypassed.
`003-authoritative-local-state.md`	The application owns authoritative state; the model does not.
`004-local-only-production.md`	Local-only runtime; no cloud, telemetry or runtime remote assets.
`005-branch-preserving-history.md`	History is preserved, not overwritten. The requirement.
`006-genre-agnostic-core.md`	The core state model is genre-agnostic.
`007-future-media-extension.md`	Media generation stays optional and decoupled.
`009-ai-dnd-production-base.md`	AI-DnD at `d72f7c1b…` is the production base.
`010-explicit-typed-narrative-state-events.md`	Explicit typed events / absolute assignments, not relative deltas.
`011-local-inference-endpoint-policy.md`	Address allowlist, deny by default, checked twice, TLS mandatory.
`012-active-head-non-destructive-history.md`	The stored active head. The architecture implementing ADR 005.
`013-authoritative-narrative-state-document.md`	The authoritative state document, its pipeline, and where each part lives. The architecture implementing ADR 010.

008-phase0-before-build-plan.md was a process gate — do not begin production work until Phase 0 closes — which Phase 0 satisfied on 2026-09-01. It is in archive/decisions/ and constrains nothing now. ADR numbering continues from 012; 008 is not reused.

Milestone reports

reports/ holds the report for the milestone most recently completed, because that is the one the next milestone's planning has to consult:

reports/M7-IMPLEMENTATION-REPORT.md — M7's independent review, its corrective closeout and its closeout verification, in that order and none overwriting another. It is the longest report in the package because M7 is the milestone whose first implementation was most wrong, and the sequence is the point: what was claimed, what was measured, what that forced.

Completed earlier milestones are in archive/milestone-reports/, which M6's report joined at M7's closeout: a milestone report is useful during the immediate next milestone and historical afterwards. M1-M6 are all there, unedited.

The decision this package rests on

Phase 0A static research and Phase 0B local validation closed on 2026-09-01 with one decision:

Fork AI-DnD at upstream commit d72f7c1bda0f34fccd84afb7a25c34eb01c901de.

It was chosen on measured behavior, not feature count: AI-DnD already had the browser UI, the FastAPI service boundary, SQLite persistence, parent-linked story history, alternate takes, branch-aware state snapshots, local Ollama operation, branch-scoped memory, prompt inspection, streaming, export/import and a substantial test suite. The composition of ideas selected around it:

AI-DnD production base
  + ai-adventure state/event/commit/checkpoint/privacy patterns
  + Open Dungeon story-reading and future-media UX patterns
  + Chronicler memory-authority concepts
  + Interactive Fiction Framework canon/validation concepts
  + Gamentic provider-neutral media concepts

This is not a repository merge. AI-DnD is the ownership center; the others are implementation references only.

Phase 0B also corrected several Phase 0A assumptions, and those corrections are now built into the active documents rather than needing to be read from the research: shipped Undo was destructive with no Redo (ADR 012 replaced it); relative-delta world state can be semantically wrong while syntactically valid (ADR 010); tiktoken and Google Fonts broke offline operation (fixed in M1); export had to carry the head position (M3); Story Cards were not a sufficient imported-knowledge store (IMPORTED-KNOWLEDGE-DESIGN.md).

The evidence is in archive/phase0/, and archive/phase0/PHASE-0B-RECOMMENDATION.md is the strongest single document there. It is research evidence, not a specification, and it was not rewritten to match later conclusions.

Foundational Decisions

The following are settled for v1:

browser-first UI,
local-only runtime across user-controlled local infrastructure,
Ollama inference with same-host loopback as default and explicitly configured trusted-LAN inference supported,
single-user deployment,
AI-DnD production base at the pinned Phase 0B commit,
application-owned authoritative state,
complete retained transcript/history,
simple user-facing Undo/Redo/Retry/Save Point semantics,
non-destructive head-cursor history internally,
new write after moving backward creates a new continuation while retaining the old future,
abandoned history is retained and marked disposable; no automatic cleanup is required in v1,
named checkpoints remain until explicitly deleted,
genre-agnostic core state,
explicit typed narrative-state events/absolute assignments rather than ambiguous relative deltas,
hybrid state model: validated events plus state snapshots/cache,
branch/lineage-safe summaries and memories,
imported knowledge is a separate first-class subsystem rather than an extension of AI-DnD Story Cards,
knowledge classes: Canon / Reference / Inspiration,
local lexical retrieval plus local semantic retrieval where practical,
prompt/context provenance and inspection,
export/import must preserve active branch and active head position, including an undone position,
no cloud/Internet inference, telemetry, automatic web retrieval, remote runtime assets, shell/MCP/general plugin execution,
LAN inference is distinct from LAN exposure of the storyteller UI/API; the latter is not required for v1,
future local image/video/audio/TTS/STT support remains optional and decoupled from the story engine.

Still open, and deliberately so

These do not block the architecture and are not v1 requirements:

comparative recommendation of narrator/state models for real users,
multi-hour/100-turn long-run behavior,
detailed concurrency behavior beyond the single-user turn lock,
actual future image/video/TTS/STT provider integration,
abandoned-history cleanup UI/policy (not required in v1).

Where the work stands

Phase 0 research and spikes         COMPLETE
Architecture/fork decision          COMPLETE
Planning package approved           COMPLETE
        |
        v
Milestone M1                        COMPLETE (2026-09-02)
  fork + offline baseline           archive/milestone-reports/M1-*.md
        |
        v
Milestone M2                        COMPLETE (2026-09-02)
  local-only surface + endpoint     archive/milestone-reports/M2-*.md
  policy
        |
        v
Milestone M3                        COMPLETE (2026-09-03)
  non-destructive undo/redo,        archive/milestone-reports/M3-*.md
  active-head export                and ADR 012
        |
        v
Milestone M4                        COMPLETE (2026-09-03)
  named Save Points                 archive/milestone-reports/M4-*.md
        |                           browser verification: PASS (M3 + M4)
        v
Milestone M5                        COMPLETE (2026-09-04)
  authoritative narrative state     archive/milestone-reports/M5-*.md
        |                           and ADR 013
        v
Milestone M6                        COMPLETE (2026-09-06)
  branch-safe context, summaries    archive/milestone-reports/M6-*.md
  and long-term story memory
        |
        v
Milestone M7                        COMPLETE (2026-09-06)
  first-class imported knowledge    reports/M7-IMPLEMENTATION-REPORT.md
  library                           review + corrective + closeout, in sequence
        |
        v
M8-M11, one at a time               see BUILD-MILESTONES.md

Stop Rule

One milestone at a time. Do not begin a milestone before its brief exists.

No M8 brief has been prepared. Writing one is the current action, informed by the M7 report and by the debt BUILD-MILESTONES.md records against M7 — in particular that the knowledge panel and the Insights knowledge rows are functional rather than designed, that a "nothing was relevant enough" result and an uncalibrated-embedding-model warning are both surfaced plainly and want a considered treatment, and that BROWSER-UX-SPEC.md §47's import preview and §52's retrieval-usage count are not built.

The M6 retrieval debt this milestone was warned about is partly addressed and partly still open. Imported material does not compete with story memory for one budget — M7 gave it a separate bounded budget of its own — and ranking for imported knowledge is now relevance × class rather than similarity plus a pin. Story-memory ranking is unchanged, and cross-layer duplication (CONTEXT-AND-MEMORY.md §22) is still open: the same fact can still appear in state, memory, history and now an imported passage at once.

No conditions remain open on M1-M6. The browser smoke condition that M3 and M4 both carried was satisfied at M4 closeout: a real Firefox exercised the rendered DOM for both milestones' controls, 44/44 checks passing. The M3 report's §M.2 and the M4 report's §M record the condition as it stood; the M4 report's §W records it closed.

What each milestone closeout corrected

These tables are the audit trail: what implementation evidence forced back into the planning package, milestone by milestone. VERSION.md narrates the same changes.

Post-M3 corrections applied (2026-09-03)

M3's review recommended planning changes and, following the M2 pattern, reported rather than applied them. All are now applied, together with the closeout work the milestone itself required:

Document	Correction
`DECISIONS/012-active-head-non-destructive-history.md`	New ADR. The architecture selected to implement ADR 005: head stored not derived, one capped read path, one movement mechanism, state from the node, divergence on first write below the head, Redo decided by the lineage, advisory disposition metadata, and the head as an exported decision.
`STORY-BRANCH-SEMANTICS.md` §5	Undo crosses fork points and continues to the campaign opening. The old refusal was a consequence of destructive deletion, not a product decision.
`STORY-BRANCH-SEMANTICS.md` §10	Switching which take is live is refused while a later story is off screen, with the two resolutions the user has.
`STORY-BRANCH-SEMANTICS.md` §14A	New. In-place editing before §14-15 exist: refuse when story descends from the turn and is not on screen. States explicitly that the full narrator-edit requirement stands and is completed in M5.
`TECHNICAL-DESIGN.md` §8.7, §9.1	New. The implemented active-head model and bundle behaviour, recorded as fact.
`TECHNICAL-DESIGN.md` §10.4	Constraint from M3: the snapshot half of the hybrid state model is a requirement, or head movement becomes proportional to campaign length.
`DATA-MODEL.md` §4, §5, §29	The head as campaign-stored rather than derived; the branch disposition as implemented and deliberately advisory; the export as carrying a chosen position.
`BUILD-MILESTONES.md` M3	Marked COMPLETE, with inherited capabilities, the open browser condition, and carried debt.
`BUILD-MILESTONES.md` M4	Note: a Save Point is a durable pointer; restore by reusing M3's head movement rather than building a second restore path.
`BUILD-MILESTONES.md` M5	Note: keep state efficiently recoverable at a position; move the test instrumentation rather than the assertions; complete the narrator edit.
`V1-ACCEPTANCE-TESTS.md` D03, D10, I07, L01	D03's result recorded as a full pass; D10's milestone ownership stated without weakening any pass condition; I07's pre-M3 bundle clause added; the apparent L01/A05 conflict resolved.
`README.md`	Corrected to describe the current local-only single-user application. The scripting, accounts, analytics, hosted-demo, cloud-provider, Postgres and Render material described subsystems M2 removed.

SPECIFICATION.md and SECURITY-THREAT-MODEL.md were deliberately not changed. M3 altered no product requirement and touched no path in the threat model.

Post-M2 corrections applied (2026-09-03)

M2's review recommended six planning changes and reported rather than applied them. All six are now applied, plus three additions drawn from the same evidence:

Document	Correction
`SECURITY-THREAT-MODEL.md`	New §10A records the inference endpoint policy as implemented — address allowlist, enforced on save and before every request, TLS never traded against it — with both residual limits stated. §71A item 5 marked resolved; §77 notes the required defaults are now met.
`TECHNICAL-DESIGN.md` §5.1	Items 3 and 4 marked done; all five hardening items are now resolved.
`TECHNICAL-DESIGN.md` §5.2	New: the M1/M2 production architecture recorded as fact — SQLite, Ollama-only, loopback storyteller, trusted-LAN inference accepted, public endpoints refused.
`TECHNICAL-DESIGN.md` §18.1	New wiring rule from the M2 regressions: test a real consumer path when removing a setting, and prove a new setting reaches its component.
`DECISIONS/011-local-inference-endpoint-policy.md`	New ADR. Address-based allowlist over hostname matching, deny by default, checked twice, mandatory TLS — with the `ipaddress`-classification finding as the reason the CIDRs are spelled out.
`BUILD-MILESTONES.md` M2	Marked COMPLETE with the capabilities it delivered and the debt it carried forward.
`BUILD-MILESTONES.md` M5	Note: eight rollback tests now use the world-state engine as instrumentation, not as endorsement; move the instrumentation when M5 replaces the protocol, and rework rather than delete those tests.
`BUILD-MILESTONES.md` M6	Note: background memory failure must be observable, at least one real provider-construction path must be tested, and derived-memory failure must not corrupt accepted story state.
`V1-ACCEPTANCE-TESTS.md` H10	Strengthened: a wildcard origin must be rejected at startup, and an unknown `/api/...` path must 404 rather than returning the SPA with HTTP 200.
`V1-ACCEPTANCE-TESTS.md` H12	New. Inference endpoint enforcement, including the defence-in-depth case: a public endpoint written into the database behind the settings API must still be refused at request time.

SPECIFICATION.md was deliberately not changed. M2 altered no product requirement; it removed capability the specification never asked for.

Post-M1 corrections applied (2026-09-02)

Implementation evidence contradicted or under-specified six places in this package, and a seventh was added on review. All seven are now corrected:

Document	Correction
`DECISIONS/002-ollama-only-v1.md`	New section: a trusted-LAN Ollama may be HTTPS with a private CA; verify against the OS trust store; full certificate and hostname checking; no bypass option.
`DECISIONS/004-local-only-production.md`	New Testing Consequence: offline tests need a fresh cache and no route out. Vendored runtime artifacts should be integrity-verifiable.
`V1-ACCEPTANCE-TESTS.md` A05	Pass conditions reworded around accepted history; explicit note that the user's submitted text is deliberately retained.
`V1-ACCEPTANCE-TESTS.md` A06	Now requires a real second machine and an HTTPS endpoint with a locally issued certificate; a plain-HTTP LAN test is no longer sufficient evidence.
`V1-ACCEPTANCE-TESTS.md` §3	Record CPU/GPU/RAM: cold model load on a CPU-only host exceeded the inherited 120 s timeout.
`BUILD-MILESTONES.md`	M1 marked COMPLETE with the capabilities it delivered; M2's endpoint-policy line reframed from inventing trusted-LAN support to narrowing it.
`TECHNICAL-DESIGN.md` §5	Runtime boundary restated: the storyteller is loopback-only, inference may be same-host or trusted-LAN, and the two are independent. Adds the TLS/private-CA rule.

README.md Unescape Escape