M7: a first-class imported knowledge library
A campaign can import local .txt and .md files as Canon, Reference or Inspiration, and the class is load-bearing rather than a label: it decides the words a passage is framed with in the prompt, the weight it carries when passages are ranked, and which budget it competes in when the context is tight. This is a separate subsystem, which is the Phase 0B decision (IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification, provenance, content identity, chunking, an index or a lifecycle, and they were not promoted into something that does. Nothing here reads or writes one. The subsystem, in backend/app/knowledge/: classes the three classes, their weights, and the prompt framing chunking deterministic, heading-aware, 60-800 tokens, no overlap fts SQLite FTS5 with porter stemming; scoped and bounded in SQL importer validate, hash, store, chunk, index — in one transaction embeddings local Ollama vectors through the shared provider retrieval query construction, hybrid merge, rerank inject the budgeted cut and the rendered prompt sections Relevance admission is a separate stage from ranking, and that separation is the milestone's most expensive lesson. An independent review found the first implementation deciding relevance with a floor expressed as a share of the best candidate — which the best clears by construction — so a passage was admitted on every turn regardless of the scene. A query about tide tables and container tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden Canon among them. So the pipeline is now: candidate generation -> admission -> ranking -> class weighting -> budget Admission reads raw, candidate-set-independent signals: the cosine the model returned, and how many distinct meaningful query terms a passage contains. Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero is not zero. Normalization decides order among things that matched; it can never decide whether anything matched. Authority is applied after admission, so a class orders what matched and never rescues what did not. Retrieval may therefore return nothing, and on a scene unrelated to the library it does. The other decisions that each replaced an obvious wrong one: - The class multiplies relevance rather than adding to it. An additive bonus satisfies "Canon outranks Reference" and makes "do not include irrelevant Canon" impossible, because a large enough constant wins on its own. - The semantic floor is measured, not guessed: 113 production-path pairs against nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at 0.36-0.56, and 0.58 sits between them. Because it is a property of that model and not of cosine similarity, it is keyed to the model rather than applied to whatever is configured: an embedding model with no measured calibration in this build does not borrow the number. Semantic admission is skipped, the campaign retrieves lexically, and the reason is stated in the knowledge status and in the turn's provenance. Degrading to lexical keeps the library usable; lending the threshold to an unmeasured model is how the admitted-everything defect would return. - One lexical term is not evidence. Two distinct meaningful terms, or one that is neither a standing campaign entity nor a negligible share of the query. The stop list grew from 42 words to 261, all function words — no subject matter, because a stop list that removes subject matter stops finding "The Silver Key". - Lexical retrieval is a production path, not a fallback. It finds the proper nouns and invented terms a setting bible is made of, and the library is fully usable with no embedding model configured. Safety is structural rather than filtered. Imported text reaches the prompt whole, inside a section that says what it is, under a rule stating the authority order in words and refusing every instruction inside it. No endpoint accepts a filesystem path, so H08 has no mechanism to escape from. Nothing renders imported content as HTML, so a script tag is five visible characters and a remote image is never fetched. Import, chunking, indexing, retrieval and a turn open no socket at all; only embeddings do, through the endpoint allowlist the memory bank already uses. Provenance is the rendered text, not a foreign key: deleting a source cannot turn a historical turn's evidence into dangling ids. Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5 virtual table attached to knowledge_chunks as a DDL hook so it is created and dropped with the table it indexes. Migration 92. A pre-M7 database opens unchanged and needs no sources to play. Bundle: the source content and the reader's judgements about it travel; the passages, index rows and vectors are rebuilt on import, so a restored campaign is searchable immediately without a reindex step. One runtime dependency: python-multipart, Starlette's multipart parser. It is what makes the upload surface possible, and the upload surface is why no pathname is ever accepted. The test doubles were the reason the defect shipped, so they were corrected too. The retrieval stub scored unrelated text at 0.06-0.20 where the real model scores it at 0.43-0.44, and its docstring said it had deliberately removed the constant component that "would put a similarity floor under every pair" — which is exactly the property real models have. The stub now has that floor, one test fails if it is ever removed, and another reproduces the superseded rule and asserts it is still fooled by the same fixture. Run against the pre-corrective implementation, the new suite fails 13 of 18. Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of which mocks nothing between itself and Ollama and re-measures the similarity separation on every run. 43/43 checks in a real Firefox, reproduced. Docker build clean. Four other defects found by review or by the browser run were fixed here rather than carried: an unreachable relevance constant that appeared to enforce something and did not; acceptance tests using the wrong fixture files, so G07's trap was never exercised; a bidirectional override surviving into displayed filenames; and, from the implementation pass, the Insights panel showing M5's two state sections as raw keys and the source inspector refetching on every keystroke. M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK REQUIRED. Both blocking findings are closed, and closeout resolved the embedding-model calibration boundary the corrective pass had left as debt. planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective closeout and the closeout verification in sequence, none overwriting another. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 5
Claude Opus 5
parent
a6e9c7a32b
commit
480414efe0
@@ -24,6 +24,8 @@ import tiktoken
|
||||
from sqlalchemy.orm import object_session
|
||||
|
||||
from .. import derived, models, narrative, summaries, worldstate
|
||||
from ..knowledge import inject as knowledge_inject
|
||||
from ..knowledge import records as knowledge_records
|
||||
from . import encoding, history
|
||||
|
||||
AUTHORS_NOTE_DEPTH = 3 # actions from the end of history
|
||||
@@ -286,11 +288,28 @@ def build_context(
|
||||
settings: models.Settings,
|
||||
memory_bank: dict | None = None,
|
||||
exclude_action_id: int | None = None,
|
||||
knowledge: knowledge_records.Result | None = None,
|
||||
) -> tuple[str, str, dict]:
|
||||
"""Returns (system_text, story_text, context_report). `memory_bank` is the
|
||||
result of memorybank.retrieve_memories (None when the bank is off);
|
||||
`exclude_action_id` omits one action from the story (see history.py)."""
|
||||
`exclude_action_id` omits one action from the story (see history.py).
|
||||
|
||||
M7: `knowledge` is the result of `knowledge.retrieval.retrieve` — the ranked
|
||||
imported passages, before any budget has been applied. It arrives already
|
||||
retrieved for the same reason `memory_bank` does: retrieval may need an
|
||||
embedding call, this function is synchronous, and a prompt builder that can
|
||||
make network requests is a prompt builder that can fail halfway through a
|
||||
prompt. None means the campaign has no library, or the caller did not ask.
|
||||
"""
|
||||
script_mem = _script_memory(adventure)
|
||||
# M7: priced before anything else, because the answer changes what is left.
|
||||
# `plan` prices only the protected half — the untrusted-data rule and any
|
||||
# always-in-force Canon — and both are counted with the system block below.
|
||||
knowledge_plan = knowledge_inject.plan(
|
||||
knowledge if knowledge is not None else knowledge_records.Result(),
|
||||
count_tokens,
|
||||
settings.context_token_budget,
|
||||
)
|
||||
|
||||
# ----- The static block, which is identical on every turn -----
|
||||
# This ordering exists to reduce cost. Prompt caching matches a prefix. The
|
||||
@@ -317,6 +336,21 @@ def build_context(
|
||||
if canon_text:
|
||||
system_sections.append(Section("campaign_canon", canon_text))
|
||||
|
||||
# M7: the imported-knowledge framing rule, and any Canon the campaign has
|
||||
# marked as always in force. Both go here, directly *below* the campaign's
|
||||
# own canon, which is the authority order stated in words in
|
||||
# `knowledge.classes.KNOWLEDGE_RULE` and reinforced by the position.
|
||||
#
|
||||
# In the system block rather than among the live sections, for two reasons.
|
||||
# They change only when the reader edits their library, so they belong in
|
||||
# the cached prefix; and being counted with the protected sections is what
|
||||
# makes an over-large always-include a `ContextOverflow` with an explanation
|
||||
# rather than a prompt that silently loses its history.
|
||||
for protected_section in knowledge_plan.protected:
|
||||
system_sections.append(
|
||||
Section(protected_section.label, protected_section.text)
|
||||
)
|
||||
|
||||
if isinstance(script_mem.get("context"), str) and script_mem["context"].strip():
|
||||
system_sections.append(Section("script_context", script_mem["context"].strip()))
|
||||
if adventure.ai_instructions.strip():
|
||||
@@ -443,20 +477,44 @@ def build_context(
|
||||
)
|
||||
available = settings.context_token_budget - protected
|
||||
|
||||
# ----- M7: retrieved imported knowledge, out of a share of `available` -----
|
||||
#
|
||||
# Chosen here, before the history window is sized, because what knowledge
|
||||
# spends is what the history does not get: a window fetched against the
|
||||
# whole of `available` would read turns there was never room for.
|
||||
#
|
||||
# Bounded rather than trimmed afterwards. The passages that fit are selected
|
||||
# against a share of the budget and the rest is recorded as dropped, so the
|
||||
# section stops growing when the budget is exhausted however large the
|
||||
# library becomes. Always-included Canon is not spent from this — it was
|
||||
# priced into `reserved` above — so Reference and Inspiration cannot crowd
|
||||
# out a standing campaign rule, and none of them can reach the current
|
||||
# state, the reader's input or the reply reserve, which are all above.
|
||||
knowledge_sections = [
|
||||
Section(section.label, section.text)
|
||||
for section in knowledge_inject.select(knowledge_plan, available)
|
||||
]
|
||||
knowledge_spent = sum(
|
||||
section.tokens + count_tokens(SEPARATOR) for section in knowledge_sections
|
||||
)
|
||||
available_after_knowledge = max(0, available - knowledge_spent)
|
||||
|
||||
# Only the newest actions can reach the prompt, because the code below
|
||||
# either truncates the text to `available` tokens or stops at the budget.
|
||||
# Fetch a window that is provably larger than that and no larger. Otherwise
|
||||
# a long adventure reads its whole history on every turn and uses only the
|
||||
# end of it.
|
||||
actions = history.window_covering(
|
||||
adventure, available, count_tokens, exclude_action_id
|
||||
adventure, available_after_knowledge, count_tokens, exclude_action_id
|
||||
)
|
||||
|
||||
# ----- Story cards: triggered by recent story text (the window history could fill) -----
|
||||
trigger_window = truncate_to_last_tokens(SEPARATOR.join(a.text for a in actions), available)
|
||||
trigger_window = truncate_to_last_tokens(
|
||||
SEPARATOR.join(a.text for a in actions), available_after_knowledge
|
||||
)
|
||||
triggered = match_cards(adventure.story_cards, trigger_window)
|
||||
|
||||
card_budget = int(available * CARD_BUDGET_SHARE)
|
||||
card_budget = int(available_after_knowledge * CARD_BUDGET_SHARE)
|
||||
card_records = []
|
||||
lore_lines: list[str] = []
|
||||
used = 0
|
||||
@@ -476,7 +534,7 @@ def build_context(
|
||||
)
|
||||
|
||||
# ----- Story history: newest first until the remaining budget is spent -----
|
||||
history_budget = available - used
|
||||
history_budget = available_after_knowledge - used
|
||||
included_actions: list[models.Action] = []
|
||||
spent = 0
|
||||
oldest_truncated = False
|
||||
@@ -518,7 +576,22 @@ def build_context(
|
||||
# The live sections, ordered from least to most volatile. See the comment
|
||||
# where they are built. They go below the history so that the history stays
|
||||
# cached, and above the final sections so that those stay last.
|
||||
for live in (summary_section, lore_section, memories_section, world_state_section):
|
||||
#
|
||||
# M7 inserts the retrieved knowledge between the lore and the memories, in
|
||||
# ascending authority: Inspiration, then Reference, then imported Canon,
|
||||
# then the story's own memories, and the current authoritative state last of
|
||||
# all. A model weights what it read most recently, so the section it reads
|
||||
# last is the one that settles a conflict — which is the ordering
|
||||
# `knowledge.classes.KNOWLEDGE_RULE` states in words. Both are needed. C05
|
||||
# is not satisfied by section order alone, and a stated order the layout
|
||||
# contradicts is worse than either.
|
||||
for live in (
|
||||
summary_section,
|
||||
lore_section,
|
||||
*reversed(knowledge_sections),
|
||||
memories_section,
|
||||
world_state_section,
|
||||
):
|
||||
if live is not None:
|
||||
note_sections.append(live)
|
||||
if front_memory:
|
||||
@@ -568,6 +641,17 @@ def build_context(
|
||||
# campaign. A dead memory bank is visible here rather than only in a log
|
||||
# nobody reads (F08).
|
||||
"derived": derived.report(db, adventure.id) if db is not None else [],
|
||||
# M7: every imported passage this turn was given — which source, which
|
||||
# file, which class, which visibility, which passage, how it was found,
|
||||
# what each path scored it, and what it cost — plus what was considered,
|
||||
# what was set aside as redundant, and what there was no budget for.
|
||||
#
|
||||
# The rendered text travels in this record, not a reference to the chunk
|
||||
# row it came from. That is what makes a historical turn's evidence
|
||||
# survive the source being deleted
|
||||
# (`IMPORTED-KNOWLEDGE-DESIGN.md` §49-50): the snapshot says what the
|
||||
# narrator was actually shown, and it goes on saying it.
|
||||
"knowledge": knowledge_inject.report(knowledge_plan),
|
||||
"history": {
|
||||
"included": len(included_actions),
|
||||
# The count covers the whole story rather than the window fetched
|
||||
|
||||
Reference in New Issue
Block a user