M7: a first-class imported knowledge library

A campaign can import local .txt and .md files as Canon, Reference or
Inspiration, and the class is load-bearing rather than a label: it decides the
words a passage is framed with in the prompt, the weight it carries when
passages are ranked, and which budget it competes in when the context is tight.

This is a separate subsystem, which is the Phase 0B decision
(IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification,
provenance, content identity, chunking, an index or a lifecycle, and they were
not promoted into something that does. Nothing here reads or writes one.

The subsystem, in backend/app/knowledge/:

  classes      the three classes, their weights, and the prompt framing
  chunking     deterministic, heading-aware, 60-800 tokens, no overlap
  fts          SQLite FTS5 with porter stemming; scoped and bounded in SQL
  importer     validate, hash, store, chunk, index — in one transaction
  embeddings   local Ollama vectors through the shared provider
  retrieval    query construction, hybrid merge, rerank
  inject       the budgeted cut and the rendered prompt sections

Relevance admission is a separate stage from ranking, and that separation is
the milestone's most expensive lesson. An independent review found the first
implementation deciding relevance with a floor expressed as a share of the best
candidate — which the best clears by construction — so a passage was admitted on
every turn regardless of the scene. A query about tide tables and container
tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden
Canon among them.

So the pipeline is now:

  candidate generation -> admission -> ranking -> class weighting -> budget

Admission reads raw, candidate-set-independent signals: the cosine the model
returned, and how many distinct meaningful query terms a passage contains.
Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero
is not zero. Normalization decides order among things that matched; it can never
decide whether anything matched. Authority is applied after admission, so a
class orders what matched and never rescues what did not.

Retrieval may therefore return nothing, and on a scene unrelated to the library
it does.

The other decisions that each replaced an obvious wrong one:

- The class multiplies relevance rather than adding to it. An additive bonus
  satisfies "Canon outranks Reference" and makes "do not include irrelevant
  Canon" impossible, because a large enough constant wins on its own.
- The semantic floor is measured, not guessed: 113 production-path pairs against
  nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at
  0.36-0.56, and 0.58 sits between them. Because it is a property of that model
  and not of cosine similarity, it is keyed to the model rather than applied to
  whatever is configured: an embedding model with no measured calibration in
  this build does not borrow the number. Semantic admission is skipped, the
  campaign retrieves lexically, and the reason is stated in the knowledge status
  and in the turn's provenance. Degrading to lexical keeps the library usable;
  lending the threshold to an unmeasured model is how the admitted-everything
  defect would return.
- One lexical term is not evidence. Two distinct meaningful terms, or one that
  is neither a standing campaign entity nor a negligible share of the query.
  The stop list grew from 42 words to 261, all function words — no subject
  matter, because a stop list that removes subject matter stops finding "The
  Silver Key".
- Lexical retrieval is a production path, not a fallback. It finds the proper
  nouns and invented terms a setting bible is made of, and the library is fully
  usable with no embedding model configured.

Safety is structural rather than filtered. Imported text reaches the prompt
whole, inside a section that says what it is, under a rule stating the authority
order in words and refusing every instruction inside it. No endpoint accepts a
filesystem path, so H08 has no mechanism to escape from. Nothing renders
imported content as HTML, so a script tag is five visible characters and a
remote image is never fetched. Import, chunking, indexing, retrieval and a turn
open no socket at all; only embeddings do, through the endpoint allowlist the
memory bank already uses.

Provenance is the rendered text, not a foreign key: deleting a source cannot
turn a historical turn's evidence into dangling ids.

Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5
virtual table attached to knowledge_chunks as a DDL hook so it is created and
dropped with the table it indexes. Migration 92. A pre-M7 database opens
unchanged and needs no sources to play.

Bundle: the source content and the reader's judgements about it travel; the
passages, index rows and vectors are rebuilt on import, so a restored campaign
is searchable immediately without a reindex step.

One runtime dependency: python-multipart, Starlette's multipart parser. It is
what makes the upload surface possible, and the upload surface is why no
pathname is ever accepted.

The test doubles were the reason the defect shipped, so they were corrected too.
The retrieval stub scored unrelated text at 0.06-0.20 where the real model
scores it at 0.43-0.44, and its docstring said it had deliberately removed the
constant component that "would put a similarity floor under every pair" — which
is exactly the property real models have. The stub now has that floor, one test
fails if it is ever removed, and another reproduces the superseded rule and
asserts it is still fooled by the same fixture. Run against the pre-corrective
implementation, the new suite fails 13 of 18.

Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of
which mocks nothing between itself and Ollama and re-measures the similarity
separation on every run. 43/43 checks in a real Firefox, reproduced.
Docker build clean.

Four other defects found by review or by the browser run were fixed here rather
than carried: an unreachable relevance constant that appeared to enforce
something and did not; acceptance tests using the wrong fixture files, so G07's
trap was never exercised; a bidirectional override surviving into displayed
filenames; and, from the implementation pass, the Insights panel showing M5's
two state sections as raw keys and the source inspector refetching on every
keystroke.

M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK
REQUIRED. Both blocking findings are closed, and closeout resolved the
embedding-model calibration boundary the corrective pass had left as debt.
planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective
closeout and the closeout verification in sequence, none overwriting another.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b

planning/VERSION.md

@@ -1,8 +1,136 @@
 # Planning Package Version
 
-- **Package:** Adventure Storyteller Planning Package v2.7
+- **Package:** Adventure Storyteller Planning Package v3.0
 - **Revision date:** 2026-09-06
-- **Status:** Phase 0 complete; architecture selected; **Milestones M1-M6 implemented and accepted**; M7 is next to brief.
+- **Status:** Phase 0 complete; architecture selected; **Milestones M1-M7 implemented and accepted**; M8 is next to brief.
+
+## v3.0 — M7 Closeout (2026-09-06)
+
+M7 is complete. Two items the corrective pass had left open are resolved.
+
+**The embedding-model calibration boundary.** `SEMANTIC_FLOOR = 0.58` was
+measured against `nomic-embed-text`, and the corrective pass documented only the
+safe half of that: a model scoring everything lower degrades to lexical-only. A
+model scoring unrelated material *higher* would have recreated M7-F1 on a build
+whose tests all pass. Semantic admission is now **per model**: an uncalibrated
+model does not inherit the threshold, semantic retrieval is skipped for it with
+the reason reported, and the library degrades to lexical-only. Recorded in
+`TECHNICAL-DESIGN.md` §13.3 and `IMPORTED-KNOWLEDGE-DESIGN.md` §76.
+
+**The ambiguous `export/import 53/54`.** The 54th case was a false positive in
+the independent review's own harness — its "no filesystem path" assertion was a
+substring test that fired on `text/markdown`, a MIME type. Replaced with three
+precise checks; the suite is **56/56** and no product behaviour was involved.
+
+**What closeout changed in the active documents:**
+
+- `TECHNICAL-DESIGN.md` §13.3 — **new.** A similarity threshold is a property of
+  the model, the two ways a different model breaks it are not symmetric, and the
+  product refuses to apply a threshold to a model it has not measured.
+- `IMPORTED-KNOWLEDGE-DESIGN.md` §76 — the same, in the design's own terms:
+  §25's local-Ollama embedding stands; what is added is that a *threshold* must
+  be measured before it is trusted.
+- `BUILD-MILESTONES.md` M7 — marked COMPLETE, with the capabilities later
+  milestones inherit and the debt carried forward, including that calibrating
+  further embedding models is a measurement rather than a guess.
+- `V1-ACCEPTANCE-TESTS.md` — the M7 results promoted from implementation-pass
+  evidence to reviewed results.
+
+## v2.9 — M7 Independent Review and Corrective Pass (2026-09-06)
+
+The review returned *PASS WITH CORRECTIVE WORK REQUIRED*. It closed the five
+acceptance conditions the implementation had flagged as unmeasured — C05, G06,
+G07, G10 and hidden Canon, all exercised against a real narrator and all
+passing — and found two blocking defects, both now corrected.
+
+**M7-F1 — imported knowledge was injected regardless of relevance.** Relevance
+was decided by a floor expressed as a share of the best candidate, which the
+best clears by construction. A query about tide tables and container tonnage
+retrieved all five sources of a fantasy campaign, narrator-only hidden Canon
+among them. Corrected by separating relevance **admission** from **ranking**.
+
+**M7-F2 — the retrieval suite could not detect it.** Its stub scored unrelated
+text an order of magnitude lower than the real model, so the broken gate passed.
+Corrected with a stub that has the real model's similarity floor, plus a test
+that fails if the floor is removed and one that shows the superseded rule still
+being fooled. The new suite fails 13/18 against the pre-corrective code.
+
+**What the corrective pass forced into the active documents:**
+
+- `TECHNICAL-DESIGN.md` §13.2 — **new.** Relevance admission is a separate stage
+  from ranking, and the general rule behind it: a relevance decision must rest on
+  a signal meaningful on its own, because normalization answers "which of these
+  is best" and can never answer "is any of these any good". A pipeline that ranks
+  first and cuts second has no way to return nothing.
+- `TECHNICAL-DESIGN.md` §13.1 — the pipeline diagram gains the admission stage.
+- `IMPORTED-KNOWLEDGE-DESIGN.md` §76 — retrieval corrected: admission before
+  authority, and the plain statement that **retrieval may return nothing**,
+  which is what §30 means when every source is irrelevant.
+- `BUILD-MILESTONES.md` M7 § Status — both findings, their corrections, and the
+  model-specific calibration recorded as carried debt.
+
+Three non-blocking findings were folded in: a relevance constant that could
+never fire was removed rather than re-tuned; the acceptance tests moved to the
+standard `TEST-CAMPAIGN-FIXTURE.md` §12 files so G07's trap is finally
+exercised; and Unicode format characters are stripped from displayed filenames.
+A pre-existing M5 narrator-protocol issue was recorded and deliberately left
+with M5.
+
+## v2.8 — M7 Implementation Pass (2026-09-06)
+
+**Not a closeout.** M7 is implemented, not accepted, and this revision records
+what the implementation pass built and measured so that an independent review
+has something to verify against. No milestone report was written: the
+convention this package follows puts the report in `reports/` and has the
+*reviewer* write it, treating the build summary as claims to check.
+
+**M7 — First-Class Imported Knowledge Library.** A campaign can import local
+`.txt` and `.md` files as Canon, Reference or Inspiration; retrieval is hybrid
+(SQLite FTS5 plus local Ollama embeddings), reranked by relevance × class,
+bounded by its own token budget, framed in the prompt as untrusted data with the
+authority order stated in words, and fully traceable in the Insights panel. It is
+a separate subsystem: AI-DnD's Story Cards were not promoted into it and are
+untouched.
+
+**What implementation forced into the active documents:**
+
+- `TECHNICAL-DESIGN.md` §13.1 — **new.** The implemented pipeline, and four
+  decisions that each replaced an obvious wrong one: the class multiplies
+  relevance rather than adding to it; both retrieval scores are normalized per
+  query against the best of their own path; the relevance floor is therefore
+  relative rather than absolute; and lexical retrieval is a production path
+  rather than a fallback.
+- `DATA-MODEL.md` §24A — **new.** The three tables and the FTS5 virtual table,
+  and the line between what the reader gave the campaign and what the machine
+  derived from it. Only a source's content and its classification are not
+  derivable.
+- `DATA-MODEL.md` §25 — the retrieval record is **not** a table. It lives in the
+  turn's own context snapshot and carries the *rendered text*, because a table of
+  foreign keys would turn every historical turn's evidence into dangling
+  references the moment a source were deleted.
+- `DATA-MODEL.md` §29 — what the bundle carries for imported knowledge, and why
+  passages, index rows and vectors are rebuilt rather than exported.
+- `CONTEXT-AND-MEMORY.md` §29, §41-42, §46 — the knowledge budget as implemented
+  (a protected cap for always-included Canon, a share of the rest filled in
+  authority order), always-include as a Canon-only mechanism, and hidden Canon as
+  prompt discipline rather than as filtering.
+- `IMPORTED-KNOWLEDGE-DESIGN.md` §76 — **new.** Where this document offered
+  options, which was chosen and why; and, named rather than left to be
+  discovered, the six things it contemplates that M7 does **not** implement —
+  entity linking, tags, manual priority, scene pinning, Canon-versus-Canon
+  conflict detection, and source versioning.
+- `V1-ACCEPTANCE-TESTS.md` — results for G01-G10, C05, F05, F06, I05 and
+  H06-H09, marked as implementation-pass evidence rather than review findings.
+  F05 and F06 move from *PARTIAL / PASS for story memory* to complete. H09 is
+  recorded NOT APPLICABLE with a test that fails if an archive extractor is ever
+  added to this surface. **C05 is recorded as a pass on the assembled prompt
+  with the gap stated**: no real narrator generation was run against it.
+- `BUILD-MILESTONES.md` M7 § Status — **new.** What was built beyond the scope
+  list, and the debt carried forward, deliberately.
+
+**One runtime dependency was added**: `python-multipart`, Starlette's multipart
+parser. It is what makes the upload surface possible, and the upload surface is
+why no endpoint in the knowledge API accepts a filesystem path.
 
 ## v2.7 — M5 and M6 Closeout (2026-09-06)