M7: a first-class imported knowledge library

A campaign can import local .txt and .md files as Canon, Reference or
Inspiration, and the class is load-bearing rather than a label: it decides the
words a passage is framed with in the prompt, the weight it carries when
passages are ranked, and which budget it competes in when the context is tight.

This is a separate subsystem, which is the Phase 0B decision
(IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification,
provenance, content identity, chunking, an index or a lifecycle, and they were
not promoted into something that does. Nothing here reads or writes one.

The subsystem, in backend/app/knowledge/:

  classes      the three classes, their weights, and the prompt framing
  chunking     deterministic, heading-aware, 60-800 tokens, no overlap
  fts          SQLite FTS5 with porter stemming; scoped and bounded in SQL
  importer     validate, hash, store, chunk, index — in one transaction
  embeddings   local Ollama vectors through the shared provider
  retrieval    query construction, hybrid merge, rerank
  inject       the budgeted cut and the rendered prompt sections

Relevance admission is a separate stage from ranking, and that separation is
the milestone's most expensive lesson. An independent review found the first
implementation deciding relevance with a floor expressed as a share of the best
candidate — which the best clears by construction — so a passage was admitted on
every turn regardless of the scene. A query about tide tables and container
tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden
Canon among them.

So the pipeline is now:

  candidate generation -> admission -> ranking -> class weighting -> budget

Admission reads raw, candidate-set-independent signals: the cosine the model
returned, and how many distinct meaningful query terms a passage contains.
Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero
is not zero. Normalization decides order among things that matched; it can never
decide whether anything matched. Authority is applied after admission, so a
class orders what matched and never rescues what did not.

Retrieval may therefore return nothing, and on a scene unrelated to the library
it does.

The other decisions that each replaced an obvious wrong one:

- The class multiplies relevance rather than adding to it. An additive bonus
  satisfies "Canon outranks Reference" and makes "do not include irrelevant
  Canon" impossible, because a large enough constant wins on its own.
- The semantic floor is measured, not guessed: 113 production-path pairs against
  nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at
  0.36-0.56, and 0.58 sits between them. Because it is a property of that model
  and not of cosine similarity, it is keyed to the model rather than applied to
  whatever is configured: an embedding model with no measured calibration in
  this build does not borrow the number. Semantic admission is skipped, the
  campaign retrieves lexically, and the reason is stated in the knowledge status
  and in the turn's provenance. Degrading to lexical keeps the library usable;
  lending the threshold to an unmeasured model is how the admitted-everything
  defect would return.
- One lexical term is not evidence. Two distinct meaningful terms, or one that
  is neither a standing campaign entity nor a negligible share of the query.
  The stop list grew from 42 words to 261, all function words — no subject
  matter, because a stop list that removes subject matter stops finding "The
  Silver Key".
- Lexical retrieval is a production path, not a fallback. It finds the proper
  nouns and invented terms a setting bible is made of, and the library is fully
  usable with no embedding model configured.

Safety is structural rather than filtered. Imported text reaches the prompt
whole, inside a section that says what it is, under a rule stating the authority
order in words and refusing every instruction inside it. No endpoint accepts a
filesystem path, so H08 has no mechanism to escape from. Nothing renders
imported content as HTML, so a script tag is five visible characters and a
remote image is never fetched. Import, chunking, indexing, retrieval and a turn
open no socket at all; only embeddings do, through the endpoint allowlist the
memory bank already uses.

Provenance is the rendered text, not a foreign key: deleting a source cannot
turn a historical turn's evidence into dangling ids.

Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5
virtual table attached to knowledge_chunks as a DDL hook so it is created and
dropped with the table it indexes. Migration 92. A pre-M7 database opens
unchanged and needs no sources to play.

Bundle: the source content and the reader's judgements about it travel; the
passages, index rows and vectors are rebuilt on import, so a restored campaign
is searchable immediately without a reindex step.

One runtime dependency: python-multipart, Starlette's multipart parser. It is
what makes the upload surface possible, and the upload surface is why no
pathname is ever accepted.

The test doubles were the reason the defect shipped, so they were corrected too.
The retrieval stub scored unrelated text at 0.06-0.20 where the real model
scores it at 0.43-0.44, and its docstring said it had deliberately removed the
constant component that "would put a similarity floor under every pair" — which
is exactly the property real models have. The stub now has that floor, one test
fails if it is ever removed, and another reproduces the superseded rule and
asserts it is still fooled by the same fixture. Run against the pre-corrective
implementation, the new suite fails 13 of 18.

Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of
which mocks nothing between itself and Ollama and re-measures the similarity
separation on every run. 43/43 checks in a real Firefox, reproduced.
Docker build clean.

Four other defects found by review or by the browser run were fixed here rather
than carried: an unreachable relevance constant that appeared to enforce
something and did not; acceptance tests using the wrong fixture files, so G07's
trap was never exercised; a bidirectional override surviving into displayed
filenames; and, from the implementation pass, the Insights panel showing M5's
two state sections as raw keys and the source inspector refetching on every
keystroke.

M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK
REQUIRED. Both blocking findings are closed, and closeout resolved the
embedding-model calibration boundary the corrective pass had left as debt.
planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective
closeout and the closeout verification in sequence, none overwriting another.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b

planning/TECHNICAL-DESIGN.md

@@ -760,6 +760,134 @@ Story Cards may remain a useful reference or authored-rule mechanism, but the im
 
 If a future knowledge item is derived from story history rather than imported as global campaign material, it must carry lineage/source-turn information sufficient to avoid abandoned-path leakage.
 
+### 13.1 As implemented in M7
+
+Every item above is built, in `backend/app/knowledge/`. Story Cards were not
+promoted into it and are untouched. The pipeline, and where each decision lives:
+
+```text
+upload (multipart; no pathname is ever accepted)
+  -> validate      size, strict UTF-8, real text, allowed extension, class
+  -> hash          SHA-256 of the normalized text; the duplicate test
+  -> store         the text in SQLite, under application control
+  -> chunk         deterministic, heading-aware, 60-800 tokens
+  -> index         SQLite FTS5, porter-stemmed
+  ---- one transaction ends here; the source is now `ready` ----
+  -> embed         local Ollama, best-effort, through the shared provider
+```
+
+```text
+query   built from the head-capped story tail and the authoritative state
+  -> FTS5 lexical candidates      (LIMIT in SQL)
+  +  semantic candidates          (when an embedding model is configured)
+  -> ADMISSION, absolute and per path:
+       semantic  raw cosine >= SEMANTIC_FLOOR
+       lexical   >= 2 distinct meaningful terms, or 1 that is neither a
+                 standing entity nor a negligible share of the query
+     a passage needs evidence from at least one path, or it is discarded
+  -> RANKING, among survivors only:
+       normalize each score against the best surviving value of its own path
+       relevance = max(lex, sem) + 0.15 x min(lex, sem)
+       score     = relevance x class weight  (canon 1.00, ref 0.85, insp 0.70)
+  -> suppress redundancy, never across classes, before the budget cut
+  -> fill Canon, then Reference, then Inspiration, each against a cap
+  -> render with class framing and per-passage provenance
+```
+
+### 13.2 Relevance admission is a separate stage from ranking
+
+**This is M7's most expensive lesson and it generalises beyond knowledge
+retrieval.** M7 shipped with only a ranking stage: both scores were normalized
+against the best candidate of their own path, and the relevance floor was
+expressed as a share of that best. A floor defined as a share of the best is
+structurally incapable of rejecting anything, because the best candidate clears
+a share of itself by construction. With the semantic path scoring every embedded
+chunk there was always a best, so **something was admitted on every turn**
+whatever the reader was doing — a query about tide tables and container tonnage
+retrieved all five sources of a fantasy campaign, narrator-only hidden Canon
+among them.
+
+The rule that follows:
+
+> A relevance decision must be made on a signal that means something on its own.
+> Normalization answers "which of these is best"; it can never answer "is any of
+> these any good". A pipeline that ranks first and cuts second has no way to
+> return nothing.
+
+So the two stages are separated, and they consume different quantities:
+
+- **Admission** reads the *raw* signals — the cosine the model returned, and how
+  many distinct meaningful query terms a passage contains. Neither is computed
+  by comparison with the other candidates.
+- **Ranking** reads the *normalized* signals, because `bm25` has no fixed range
+  and cosine's zero is not zero, so the two paths are not otherwise comparable.
+  It decides order among things that matched.
+
+Authority is applied in the second stage only. That is what makes
+`IMPORTED-KNOWLEDGE-DESIGN.md` §30's two consecutive sentences —
+`Canon > Reference > Inspiration`, and "do not include irrelevant Canon merely
+because it is authoritative" — compatible rather than contradictory: the class
+orders what matched and can never rescue what did not.
+
+An absolute threshold on an embedding similarity is a property of the model, not
+of the product, so it is measured, written down beside the constant, and
+re-measured by a real-model test on every run that has one — the same discipline
+`memorybank.REDUNDANT_SIMILARITY` already follows.
+
+### 13.3 Semantic admission is calibrated per embedding model
+
+**Semantic admission is calibrated for `nomic-embed-text`; uncalibrated
+embedding models fall back safely rather than borrowing its threshold.**
+
+The threshold is therefore **not portable**, and the two ways a different model
+can break it are not symmetric. A model whose similarity scale sits *below* the
+calibrated one admits nothing and degrades to lexical-only, which is a supported
+path. A model whose scale sits *above* it would put unrelated material past the
+threshold and reproduce the M7-F1 defect on a build whose tests all pass.
+
+So the product does not apply a threshold to a model it has not measured:
+
+```text
+SEMANTIC_CALIBRATION = {"nomic-embed-text": 0.58}
+
+calibrated model      -> semantic admission at its measured floor
+uncalibrated model    -> semantic retrieval skipped entirely, reason reported,
+                         retrieval degrades to lexical-only
+```
+
+The model's identity is the one already stored on each vector row, so no second
+mechanism was introduced, and an uncalibrated configuration reports
+`semantic_enabled: false` rather than claiming a semantic index that is never
+consulted. Adding a model is a measurement — run the real-model retrieval test
+against it and confirm the targeted and off-topic populations separate — not a
+guess. Generic cross-model calibration is out of scope for v1.
+
+The cost is stated rather than hidden: under an uncalibrated model a
+conceptual-only paraphrase is not retrieved. That is a missing passage rather
+than an irrelevant one, which is the direction this product prefers to fail in.
+
+Four decisions are worth recording, because each replaced an obvious wrong one:
+
+- **The class multiplies relevance; it does not add to it.** An additive class
+  bonus satisfies "Canon outranks Reference" and makes "do not include
+  irrelevant Canon" impossible, because a large enough constant wins alone.
+- **Both retrieval scores are normalized per query, against the best of their
+  own path — for ranking only.** `bm25` has no fixed range; cosine's zero is not zero, and a real
+  embedding model scores any two pieces of English around 0.3-0.6. Blended raw,
+  a lexical hit beats every semantic hit on every query.
+- **Admission does not use those normalized values at all.** A normalized score
+  cannot express "no match", which was M7's blocking defect; the correction is
+  the two-stage separation §13.2 records.
+- **Lexical retrieval is a production path**, not a fallback. It is what finds
+  proper nouns and invented terms — most of what a setting bible is made of —
+  and the library is fully usable with no embedding model at all.
+
+Abandoned-path safety is met at the query rather than by a lineage coordinate on
+the source, because an imported file has no lineage: the query is built from
+`context.history.tail`, which reads through the head-capped clause, and from
+`adventures.narrative_state`, which head movement repoints. Nothing reads the
+uncapped action table.
+
 ## 14. Prompt and Provenance Inspection
 
 Preserve and extend AI-DnD's Insights/context-snapshot capability.