M7: a first-class imported knowledge library
A campaign can import local .txt and .md files as Canon, Reference or Inspiration, and the class is load-bearing rather than a label: it decides the words a passage is framed with in the prompt, the weight it carries when passages are ranked, and which budget it competes in when the context is tight. This is a separate subsystem, which is the Phase 0B decision (IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification, provenance, content identity, chunking, an index or a lifecycle, and they were not promoted into something that does. Nothing here reads or writes one. The subsystem, in backend/app/knowledge/: classes the three classes, their weights, and the prompt framing chunking deterministic, heading-aware, 60-800 tokens, no overlap fts SQLite FTS5 with porter stemming; scoped and bounded in SQL importer validate, hash, store, chunk, index — in one transaction embeddings local Ollama vectors through the shared provider retrieval query construction, hybrid merge, rerank inject the budgeted cut and the rendered prompt sections Relevance admission is a separate stage from ranking, and that separation is the milestone's most expensive lesson. An independent review found the first implementation deciding relevance with a floor expressed as a share of the best candidate — which the best clears by construction — so a passage was admitted on every turn regardless of the scene. A query about tide tables and container tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden Canon among them. So the pipeline is now: candidate generation -> admission -> ranking -> class weighting -> budget Admission reads raw, candidate-set-independent signals: the cosine the model returned, and how many distinct meaningful query terms a passage contains. Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero is not zero. Normalization decides order among things that matched; it can never decide whether anything matched. Authority is applied after admission, so a class orders what matched and never rescues what did not. Retrieval may therefore return nothing, and on a scene unrelated to the library it does. The other decisions that each replaced an obvious wrong one: - The class multiplies relevance rather than adding to it. An additive bonus satisfies "Canon outranks Reference" and makes "do not include irrelevant Canon" impossible, because a large enough constant wins on its own. - The semantic floor is measured, not guessed: 113 production-path pairs against nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at 0.36-0.56, and 0.58 sits between them. Because it is a property of that model and not of cosine similarity, it is keyed to the model rather than applied to whatever is configured: an embedding model with no measured calibration in this build does not borrow the number. Semantic admission is skipped, the campaign retrieves lexically, and the reason is stated in the knowledge status and in the turn's provenance. Degrading to lexical keeps the library usable; lending the threshold to an unmeasured model is how the admitted-everything defect would return. - One lexical term is not evidence. Two distinct meaningful terms, or one that is neither a standing campaign entity nor a negligible share of the query. The stop list grew from 42 words to 261, all function words — no subject matter, because a stop list that removes subject matter stops finding "The Silver Key". - Lexical retrieval is a production path, not a fallback. It finds the proper nouns and invented terms a setting bible is made of, and the library is fully usable with no embedding model configured. Safety is structural rather than filtered. Imported text reaches the prompt whole, inside a section that says what it is, under a rule stating the authority order in words and refusing every instruction inside it. No endpoint accepts a filesystem path, so H08 has no mechanism to escape from. Nothing renders imported content as HTML, so a script tag is five visible characters and a remote image is never fetched. Import, chunking, indexing, retrieval and a turn open no socket at all; only embeddings do, through the endpoint allowlist the memory bank already uses. Provenance is the rendered text, not a foreign key: deleting a source cannot turn a historical turn's evidence into dangling ids. Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5 virtual table attached to knowledge_chunks as a DDL hook so it is created and dropped with the table it indexes. Migration 92. A pre-M7 database opens unchanged and needs no sources to play. Bundle: the source content and the reader's judgements about it travel; the passages, index rows and vectors are rebuilt on import, so a restored campaign is searchable immediately without a reindex step. One runtime dependency: python-multipart, Starlette's multipart parser. It is what makes the upload surface possible, and the upload surface is why no pathname is ever accepted. The test doubles were the reason the defect shipped, so they were corrected too. The retrieval stub scored unrelated text at 0.06-0.20 where the real model scores it at 0.43-0.44, and its docstring said it had deliberately removed the constant component that "would put a similarity floor under every pair" — which is exactly the property real models have. The stub now has that floor, one test fails if it is ever removed, and another reproduces the superseded rule and asserts it is still fooled by the same fixture. Run against the pre-corrective implementation, the new suite fails 13 of 18. Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of which mocks nothing between itself and Ollama and re-measures the similarity separation on every run. 43/43 checks in a real Firefox, reproduced. Docker build clean. Four other defects found by review or by the browser run were fixed here rather than carried: an unreachable relevance constant that appeared to enforce something and did not; acceptance tests using the wrong fixture files, so G07's trap was never exercised; a bidirectional override surviving into displayed filenames; and, from the implementation pass, the Insights panel showing M5's two state sections as raw keys and the source inspector refetching on every keystroke. M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK REQUIRED. Both blocking findings are closed, and closeout resolved the embedding-model calibration boundary the corrective pass had left as debt. planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective closeout and the closeout verification in sequence, none overwriting another. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 5
Claude Opus 5
parent
a6e9c7a32b
commit
480414efe0
@@ -60,6 +60,9 @@ from sqlalchemy.orm import Session, undefer
|
||||
|
||||
from . import attempts, models, schemas
|
||||
from .context import cursors, lineage
|
||||
from .knowledge import chunking as knowledge_chunking
|
||||
from .knowledge import classes as knowledge_classes
|
||||
from .knowledge import importer as knowledge_importer
|
||||
from .narrative import model as narrative_model
|
||||
|
||||
FORMAT = "ai-dnd-adventure-v2"
|
||||
@@ -158,10 +161,48 @@ def export(db: Session, adventure: models.Adventure) -> dict:
|
||||
"entry": c.entry, "notes": c.notes}
|
||||
for c in adventure.story_cards
|
||||
],
|
||||
# M7. The imported knowledge library, carried by the same rule as
|
||||
# everything else here: what somebody chose goes in the file, what a
|
||||
# machine derives does not.
|
||||
#
|
||||
# So the source text and the reader's judgements about it travel —
|
||||
# content, classification, enabled, visibility, always-include, the
|
||||
# title and filename, the hash. Passages, FTS rows and vectors do not:
|
||||
# they are a deterministic function of the content, and the import
|
||||
# rebuilds them. That keeps a bundle a readable record of a campaign
|
||||
# rather than a database dump, and it keeps a campaign exported on one
|
||||
# machine importable on another whose embedding model is different.
|
||||
#
|
||||
# `contentHash` is exported although it is derivable, because it is the
|
||||
# identity the reader can check a restored file against — the one place
|
||||
# a derived value earns a place in the file is when its purpose is to
|
||||
# detect that the thing it describes has changed underneath it. The
|
||||
# import verifies it rather than trusting it.
|
||||
#
|
||||
# A bundle written before M7 has no key here and imports with an empty
|
||||
# library, which is what such a campaign had.
|
||||
"knowledge": [_exported_source(k) for k in adventure.knowledge_sources],
|
||||
"actions": [_exported_node(a, local) for a in nodes],
|
||||
}
|
||||
|
||||
|
||||
def _exported_source(source: models.KnowledgeSource) -> dict:
|
||||
"""One knowledge source, as it goes into the file."""
|
||||
return {
|
||||
"title": source.title,
|
||||
"originalFilename": source.original_filename,
|
||||
"classification": source.classification,
|
||||
"enabled": source.enabled,
|
||||
"visibility": source.visibility,
|
||||
"alwaysInclude": source.always_include,
|
||||
"contentHash": source.content_hash,
|
||||
"mediaType": source.media_type,
|
||||
"notes": source.notes,
|
||||
"importedAt": source.imported_at.isoformat() if source.imported_at else None,
|
||||
"content": source.content,
|
||||
}
|
||||
|
||||
|
||||
_ROOT = {"parent": None, "forkDepth": None}
|
||||
|
||||
|
||||
@@ -356,9 +397,85 @@ def plan(bundle: dict, version: str) -> dict:
|
||||
"memory": _as_int(bundle.get("memoryCursor"), 0),
|
||||
"summary": _as_int(bundle.get("summaryCursor"), 0),
|
||||
},
|
||||
# M7. Checked here with everything else, before a row is written, so a
|
||||
# hand-edited library fails the import rather than half-landing in it.
|
||||
"knowledge": _planned_knowledge(bundle),
|
||||
}
|
||||
|
||||
|
||||
def _planned_knowledge(bundle: dict) -> list[dict]:
|
||||
"""The knowledge sources in a bundle, checked and normalized.
|
||||
|
||||
Every field is validated here rather than at write time, for the same reason
|
||||
the tree is: a file anyone can edit must be found wrong before it has
|
||||
written anything. A source that fails validation refuses the import — it is
|
||||
not silently dropped. A campaign whose imported Canon quietly did not arrive
|
||||
is a campaign whose narrator has stopped being told the rules, and the
|
||||
reader would have no way to notice.
|
||||
|
||||
The one thing not trusted from the file is the hash. It is recomputed from
|
||||
the content that actually arrived, and a mismatch is reported: that is the
|
||||
whole reason a derived value is in the file at all.
|
||||
"""
|
||||
entries = bundle.get("knowledge")
|
||||
if entries is None:
|
||||
return []
|
||||
if not isinstance(entries, list):
|
||||
raise HTTPException(400, "The knowledge section of this file is not a list.")
|
||||
if len(entries) > knowledge_importer.MAX_SOURCES_PER_ADVENTURE:
|
||||
raise HTTPException(
|
||||
400,
|
||||
f"This file contains {len(entries)} knowledge sources — the limit "
|
||||
f"is {knowledge_importer.MAX_SOURCES_PER_ADVENTURE}.",
|
||||
)
|
||||
planned: list[dict] = []
|
||||
for i, entry in enumerate(entries):
|
||||
if not isinstance(entry, dict):
|
||||
raise HTTPException(400, f"Knowledge source {i + 1} is not an object.")
|
||||
content = entry.get("content")
|
||||
if not isinstance(content, str) or not content.strip():
|
||||
raise HTTPException(400, f"Knowledge source {i + 1} carries no content.")
|
||||
if len(content.encode("utf-8")) > knowledge_importer.MAX_SOURCE_BYTES:
|
||||
raise HTTPException(
|
||||
400, f"Knowledge source {i + 1} is larger than the import limit."
|
||||
)
|
||||
classification = entry.get("classification")
|
||||
if not knowledge_classes.is_class(classification):
|
||||
raise HTTPException(
|
||||
400,
|
||||
f"Knowledge source {i + 1} has no valid classification "
|
||||
"(expected canon, reference or inspiration).",
|
||||
)
|
||||
visibility = entry.get("visibility")
|
||||
if not knowledge_classes.is_visibility(visibility):
|
||||
visibility = knowledge_classes.NORMAL
|
||||
filename = knowledge_importer.safe_filename(
|
||||
str(entry.get("originalFilename") or "")
|
||||
)
|
||||
stated = entry.get("contentHash")
|
||||
actual = knowledge_chunking.digest(content)
|
||||
planned.append({
|
||||
"title": str(entry.get("title") or filename or "Imported source")[:200],
|
||||
"original_filename": filename,
|
||||
"classification": classification,
|
||||
"enabled": bool(entry.get("enabled", True)),
|
||||
"visibility": visibility,
|
||||
"always_include": bool(entry.get("alwaysInclude", False))
|
||||
and classification == knowledge_classes.CANON,
|
||||
"media_type": (
|
||||
str(entry.get("mediaType"))
|
||||
if entry.get("mediaType") in ("text/plain", "text/markdown")
|
||||
else "text/markdown"
|
||||
),
|
||||
"notes": str(entry.get("notes") or ""),
|
||||
"imported_at": _as_time(entry.get("importedAt")),
|
||||
"content": content,
|
||||
"content_hash": actual,
|
||||
"hash_mismatch": isinstance(stated, str) and bool(stated) and stated != actual,
|
||||
})
|
||||
return planned
|
||||
|
||||
|
||||
def _derived_tip(branches: list[dict], nodes: list[dict], head: int) -> int:
|
||||
"""Returns where the head branch's story ends, which is where a file that
|
||||
does not state a head depth is opened.
|
||||
@@ -668,6 +785,71 @@ def write(db: Session, adventure: models.Adventure, story: dict) -> None:
|
||||
_point_the_head(adventure, story, ids)
|
||||
_write_checkpoints(db, adventure, story["checkpoints"], ids)
|
||||
_write_anchors(adventure, story, ids)
|
||||
_write_knowledge(db, adventure, story.get("knowledge") or [])
|
||||
|
||||
|
||||
def _write_knowledge(
|
||||
db: Session, adventure: models.Adventure, specs: list[dict]
|
||||
) -> None:
|
||||
"""Restores the imported library, and rebuilds the index it needs.
|
||||
|
||||
The bundle carries the source and not its passages, so this is where they
|
||||
come back: `build_index` runs the same deterministic chunker the original
|
||||
import ran, against the same text, and produces the same passages. Lexical
|
||||
retrieval therefore works the moment the import finishes, with no reindex
|
||||
step and no explanation owed to the reader.
|
||||
|
||||
Vectors do not come back, because they were never in the file. The source
|
||||
lands `embed_state = "idle"` with no vectors, and the next turn's post-turn
|
||||
pass builds them against whatever embedding model *this* machine has — which
|
||||
is the right answer, and the reason exporting the vectors would have been
|
||||
the wrong one.
|
||||
|
||||
A source whose passages cannot be built is recorded as `failed` with the
|
||||
reason rather than raising. By this point the story, its tree, its head and
|
||||
its Save Points are already written, and refusing the whole campaign over a
|
||||
rebuildable index would trade the valuable thing for the cheap one. The
|
||||
failure is visible on the source and in the knowledge status endpoint, and
|
||||
Reindex is the repair.
|
||||
"""
|
||||
for spec in specs:
|
||||
source = models.KnowledgeSource(
|
||||
adventure_id=adventure.id,
|
||||
title=spec["title"],
|
||||
original_filename=spec["original_filename"],
|
||||
classification=spec["classification"],
|
||||
enabled=spec["enabled"],
|
||||
visibility=spec["visibility"],
|
||||
always_include=spec["always_include"],
|
||||
content=spec["content"],
|
||||
content_hash=spec["content_hash"],
|
||||
byte_size=len(spec["content"].encode("utf-8")),
|
||||
media_type=spec["media_type"],
|
||||
notes=spec["notes"],
|
||||
parser_version=knowledge_chunking.PARSER_VERSION,
|
||||
chunking_version=knowledge_chunking.CHUNKING_VERSION,
|
||||
index_state="pending",
|
||||
embed_state="idle",
|
||||
)
|
||||
if spec["imported_at"] is not None:
|
||||
source.imported_at = spec["imported_at"]
|
||||
if spec["hash_mismatch"]:
|
||||
# Not a refusal. The content is what it is, and the recomputed hash
|
||||
# above is the one stored — but the file said something different,
|
||||
# which means it was edited after it was written, and the reader
|
||||
# should be able to find that out.
|
||||
source.notes = (
|
||||
f"{source.notes}\n[import] The content hash in the export file "
|
||||
"did not match the content it carried; the stored hash was "
|
||||
"recomputed from what arrived."
|
||||
).strip()
|
||||
db.add(source)
|
||||
db.flush()
|
||||
try:
|
||||
knowledge_importer.build_index(db, source)
|
||||
except Exception as exc: # noqa: BLE001 - recorded, not raised
|
||||
source.index_state = "failed"
|
||||
source.index_detail = f"{type(exc).__name__}: {exc}"[:2000]
|
||||
|
||||
|
||||
def _write_branches(
|
||||
|
||||
Reference in New Issue
Block a user