M7: a first-class imported knowledge library

A campaign can import local .txt and .md files as Canon, Reference or
Inspiration, and the class is load-bearing rather than a label: it decides the
words a passage is framed with in the prompt, the weight it carries when
passages are ranked, and which budget it competes in when the context is tight.

This is a separate subsystem, which is the Phase 0B decision
(IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification,
provenance, content identity, chunking, an index or a lifecycle, and they were
not promoted into something that does. Nothing here reads or writes one.

The subsystem, in backend/app/knowledge/:

  classes      the three classes, their weights, and the prompt framing
  chunking     deterministic, heading-aware, 60-800 tokens, no overlap
  fts          SQLite FTS5 with porter stemming; scoped and bounded in SQL
  importer     validate, hash, store, chunk, index — in one transaction
  embeddings   local Ollama vectors through the shared provider
  retrieval    query construction, hybrid merge, rerank
  inject       the budgeted cut and the rendered prompt sections

Relevance admission is a separate stage from ranking, and that separation is
the milestone's most expensive lesson. An independent review found the first
implementation deciding relevance with a floor expressed as a share of the best
candidate — which the best clears by construction — so a passage was admitted on
every turn regardless of the scene. A query about tide tables and container
tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden
Canon among them.

So the pipeline is now:

  candidate generation -> admission -> ranking -> class weighting -> budget

Admission reads raw, candidate-set-independent signals: the cosine the model
returned, and how many distinct meaningful query terms a passage contains.
Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero
is not zero. Normalization decides order among things that matched; it can never
decide whether anything matched. Authority is applied after admission, so a
class orders what matched and never rescues what did not.

Retrieval may therefore return nothing, and on a scene unrelated to the library
it does.

The other decisions that each replaced an obvious wrong one:

- The class multiplies relevance rather than adding to it. An additive bonus
  satisfies "Canon outranks Reference" and makes "do not include irrelevant
  Canon" impossible, because a large enough constant wins on its own.
- The semantic floor is measured, not guessed: 113 production-path pairs against
  nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at
  0.36-0.56, and 0.58 sits between them. Because it is a property of that model
  and not of cosine similarity, it is keyed to the model rather than applied to
  whatever is configured: an embedding model with no measured calibration in
  this build does not borrow the number. Semantic admission is skipped, the
  campaign retrieves lexically, and the reason is stated in the knowledge status
  and in the turn's provenance. Degrading to lexical keeps the library usable;
  lending the threshold to an unmeasured model is how the admitted-everything
  defect would return.
- One lexical term is not evidence. Two distinct meaningful terms, or one that
  is neither a standing campaign entity nor a negligible share of the query.
  The stop list grew from 42 words to 261, all function words — no subject
  matter, because a stop list that removes subject matter stops finding "The
  Silver Key".
- Lexical retrieval is a production path, not a fallback. It finds the proper
  nouns and invented terms a setting bible is made of, and the library is fully
  usable with no embedding model configured.

Safety is structural rather than filtered. Imported text reaches the prompt
whole, inside a section that says what it is, under a rule stating the authority
order in words and refusing every instruction inside it. No endpoint accepts a
filesystem path, so H08 has no mechanism to escape from. Nothing renders
imported content as HTML, so a script tag is five visible characters and a
remote image is never fetched. Import, chunking, indexing, retrieval and a turn
open no socket at all; only embeddings do, through the endpoint allowlist the
memory bank already uses.

Provenance is the rendered text, not a foreign key: deleting a source cannot
turn a historical turn's evidence into dangling ids.

Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5
virtual table attached to knowledge_chunks as a DDL hook so it is created and
dropped with the table it indexes. Migration 92. A pre-M7 database opens
unchanged and needs no sources to play.

Bundle: the source content and the reader's judgements about it travel; the
passages, index rows and vectors are rebuilt on import, so a restored campaign
is searchable immediately without a reindex step.

One runtime dependency: python-multipart, Starlette's multipart parser. It is
what makes the upload surface possible, and the upload surface is why no
pathname is ever accepted.

The test doubles were the reason the defect shipped, so they were corrected too.
The retrieval stub scored unrelated text at 0.06-0.20 where the real model
scores it at 0.43-0.44, and its docstring said it had deliberately removed the
constant component that "would put a similarity floor under every pair" — which
is exactly the property real models have. The stub now has that floor, one test
fails if it is ever removed, and another reproduces the superseded rule and
asserts it is still fooled by the same fixture. Run against the pre-corrective
implementation, the new suite fails 13 of 18.

Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of
which mocks nothing between itself and Ollama and re-measures the similarity
separation on every run. 43/43 checks in a real Firefox, reproduced.
Docker build clean.

Four other defects found by review or by the browser run were fixed here rather
than carried: an unreachable relevance constant that appeared to enforce
something and did not; acceptance tests using the wrong fixture files, so G07's
trap was never exercised; a bidirectional override surviving into displayed
filenames; and, from the implementation pass, the Insights panel showing M5's
two state sections as raw keys and the source inspector refetching on every
keystroke.

M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK
REQUIRED. Both blocking findings are closed, and closeout resolved the
embedding-model calibration boundary the corrective pass had left as debt.
planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective
closeout and the closeout verification in sequence, none overwriting another.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b

backend/tests/test_knowledge_chunking.py

@@ -0,0 +1,187 @@
+"""M7: the chunker, on its own.
+
+Chunking is derived data that three other things assume is reproducible: an
+export carries only the source text, an import rebuilds the passages from it,
+and a reindex throws them away and rebuilds them again. All three are wrong if
+the same bytes can produce different passages, so determinism is asserted here
+directly rather than inferred from those features working once.
+
+The cases cover what `IMPORTED-KNOWLEDGE-DESIGN.md` §15-18, §59 and §61 ask of
+chunking — a small file, multi-heading Markdown, a long paragraph, Unicode text,
+and a file near the import limit — plus the two failure shapes the sizing rules
+exist to prevent.
+
+    python -m pytest tests/test_knowledge_chunking.py -v
+"""
+
+import pytest
+
+from app.knowledge import chunking, fts, importer
+
+
+def hashes(passages):
+    return [p.content_hash for p in passages]
+
+
+def test_the_same_source_always_produces_the_same_passages():
+    """Determinism, over a document with every structure in it at once."""
+    source = (
+        "# Setting\n\nA world of rain and stone.\n\n"
+        "## Westhaven\n\nA town on the north road, five miles south of the abbey.\n\n"
+        "### The Abbey\n\nThe crypt bears a broken circle.\n\n"
+        "```\ncode = 'not a # heading'\n```\n\n"
+        "## Rules\n\nResurrection is impossible.\n"
+    )
+    first = chunking.chunk(source)
+    for _ in range(5):
+        again = chunking.chunk(source)
+        assert hashes(again) == hashes(first)
+        assert [p.text for p in again] == [p.text for p in first]
+        assert [p.heading_path for p in again] == [p.heading_path for p in first]
+        assert [p.index for p in again] == list(range(len(first)))
+
+
+def test_a_small_file_is_one_passage():
+    passages = chunking.chunk("The Old Abbey lies five miles north of Westhaven.\n")
+    assert len(passages) == 1
+    assert passages[0].index == 0
+    assert passages[0].token_count > 0
+    assert passages[0].heading_path == ""
+
+
+def test_markdown_headings_become_the_passage_trail():
+    source = "\n\n".join(
+        ["# Setting"]
+        + ["A paragraph about the setting. " * 20]
+        + ["## Westhaven"]
+        + ["A paragraph about the town. " * 20]
+        + ["### The Old Abbey"]
+        + ["A paragraph about the abbey and its crypt. " * 20]
+    )
+    passages = chunking.chunk(source)
+    trails = [p.heading_path for p in passages]
+    assert "Setting" in trails
+    assert "Setting > Westhaven" in trails
+    assert "Setting > Westhaven > The Old Abbey" in trails
+    # A trail is context, so it goes into the index as well as onto the row.
+    line = fts.index_line(passages[-1].heading_path, passages[-1].text)
+    assert "The Old Abbey" in line
+
+
+def test_a_run_of_tiny_sections_does_not_become_a_run_of_fragments():
+    """The failure the packing rule exists to prevent."""
+    source = "\n\n".join(
+        f"## Section {n}\n\nOne short line about section {n}." for n in range(40)
+    )
+    passages = chunking.chunk(source)
+    assert len(passages) < 40, "every heading became its own fragment"
+    assert all(p.token_count >= chunking.MIN_TOKENS for p in passages[:-1])
+    # Nothing was lost: every section's body is still findable, and so is its
+    # heading — as the passage's own trail for whichever section opened it, and
+    # written into the text for every section packed in after that.
+    joined = "\n".join(p.text for p in passages)
+    trails = {p.heading_path for p in passages}
+    for n in range(40):
+        assert f"section {n}." in joined
+        assert f"Section {n}" in joined or f"Section {n}" in trails
+
+
+def test_a_long_paragraph_is_split_and_a_long_section_does_not_become_one_giant():
+    long_paragraph = "The abbey stands above the salt flats. " * 400
+    passages = chunking.chunk(f"# Abbey\n\n{long_paragraph}")
+    assert len(passages) > 1
+    assert all(p.token_count <= chunking.TARGET_MAX for p in passages)
+    assert all(p.heading_path == "Abbey" for p in passages)
+    # And the text survives the split.
+    assert "The abbey stands above the salt flats." in passages[0].text
+    assert "The abbey stands above the salt flats." in passages[-1].text
+
+
+def test_a_single_unbroken_run_of_text_still_terminates():
+    """A wall of characters with no sentence, no word break and no heading.
+
+    The point is that it terminates and stays inside the ceiling. This is the
+    last-resort cut, which joins its slices with whitespace — so the characters
+    are all still there, and the boundaries between slices are not exactly where
+    they were. That is a documented consequence for a pathological input (a
+    base64 blob, or an unsegmented script) rather than something that happens to
+    prose, and it is asserted here so a change to it is deliberate.
+    """
+    passages = chunking.chunk("x" * 60_000)
+    assert len(passages) > 1
+    assert all(p.token_count <= chunking.TARGET_MAX for p in passages)
+    recovered = "".join(p.text for p in passages)
+    assert "".join(recovered.split()) == "x" * 60_000
+
+
+def test_unicode_text_is_chunked_and_hashed_stably():
+    source = (
+        "# Café de la Résistance\n\n"
+        "Le vieux marin regardait la pluie tomber sur les volets sombres. " * 20
+        + "\n\n## Ελληνικά\n\n"
+        + "Ο ταξιδιώτης μπήκε σε μια σιωπηλή αίθουσα. " * 20
+        + "\n\n## 日本語\n\n"
+        + "旅人は静かな広間に入った。雨が暗い雨戸を叩いていた。" * 20
+    )
+    passages = chunking.chunk(source)
+    assert passages
+    assert hashes(chunking.chunk(source)) == hashes(passages)
+    joined = "\n".join(p.text for p in passages)
+    assert "Résistance" in "\n".join(p.heading_path for p in passages) or "Résistance" in joined
+    assert "ταξιδιώτης" in joined
+    assert "旅人" in joined
+
+
+def test_normalization_is_stable_across_line_endings_and_unicode_forms():
+    """§61: one normalization for hashing, duplicate detection and search."""
+    # The same accented character, composed and decomposed.
+    composed = "Café de la Résistance\n"
+    decomposed = "Café de la Résistance\n"
+    assert chunking.digest(composed) == chunking.digest(decomposed)
+    # ...and the same file through Windows.
+    assert chunking.digest("a\nb\n") == chunking.digest("a\r\nb\r\n")
+    # Trailing whitespace is invisible and must not make two files differ.
+    assert chunking.digest("a\nb\n") == chunking.digest("a   \nb\t\n")
+    # But real differences still differ.
+    assert chunking.digest("a\nb\n") != chunking.digest("a\nc\n")
+
+
+def test_a_file_at_the_import_limit_chunks_within_bounds():
+    """The largest source the importer accepts, chunked end to end."""
+    paragraph = "The crypt beneath the abbey is cold and the walls are damp. "
+    body = "\n\n".join(paragraph * 12 for _ in range(1400))
+    body = body[: importer.MAX_SOURCE_BYTES - 100]
+    assert len(body.encode("utf-8")) <= importer.MAX_SOURCE_BYTES
+
+    passages = chunking.chunk(body)
+    assert len(passages) <= importer.MAX_CHUNKS_PER_SOURCE
+    assert all(p.token_count <= chunking.TARGET_MAX for p in passages)
+    assert len({p.index for p in passages}) == len(passages)
+
+
+def test_a_fenced_code_block_is_not_read_as_headings():
+    source = (
+        "# Real Heading\n\nProse about the setting.\n\n"
+        "```python\n# not a heading\n## also not a heading\n```\n\n"
+        "More prose about the setting.\n"
+    )
+    passages = chunking.chunk(source)
+    assert all(p.heading_path in ("", "Real Heading") for p in passages)
+    joined = "\n".join(p.text for p in passages)
+    assert "# not a heading" in joined
+
+
+def test_plain_text_takes_the_same_packing_with_no_headings():
+    source = "\n\n".join(f"Paragraph {n} of the notes. " * 12 for n in range(20))
+    passages = chunking.chunk(source, markdown=False)
+    assert len(passages) > 1
+    assert all(p.heading_path == "" for p in passages)
+    assert all(p.token_count <= chunking.TARGET_MAX for p in passages)
+    # A `#` in plain text is a character, not a heading.
+    hashy = chunking.chunk("# not a heading\n\nsome text\n", markdown=False)
+    assert "# not a heading" in hashy[0].text
+
+
+@pytest.mark.parametrize("source", ["", "   \n\n  \n", "\n"])
+def test_an_empty_source_produces_no_passages(source):
+    assert chunking.chunk(source) == []