M7: a first-class imported knowledge library
A campaign can import local .txt and .md files as Canon, Reference or Inspiration, and the class is load-bearing rather than a label: it decides the words a passage is framed with in the prompt, the weight it carries when passages are ranked, and which budget it competes in when the context is tight. This is a separate subsystem, which is the Phase 0B decision (IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification, provenance, content identity, chunking, an index or a lifecycle, and they were not promoted into something that does. Nothing here reads or writes one. The subsystem, in backend/app/knowledge/: classes the three classes, their weights, and the prompt framing chunking deterministic, heading-aware, 60-800 tokens, no overlap fts SQLite FTS5 with porter stemming; scoped and bounded in SQL importer validate, hash, store, chunk, index — in one transaction embeddings local Ollama vectors through the shared provider retrieval query construction, hybrid merge, rerank inject the budgeted cut and the rendered prompt sections Relevance admission is a separate stage from ranking, and that separation is the milestone's most expensive lesson. An independent review found the first implementation deciding relevance with a floor expressed as a share of the best candidate — which the best clears by construction — so a passage was admitted on every turn regardless of the scene. A query about tide tables and container tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden Canon among them. So the pipeline is now: candidate generation -> admission -> ranking -> class weighting -> budget Admission reads raw, candidate-set-independent signals: the cosine the model returned, and how many distinct meaningful query terms a passage contains. Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero is not zero. Normalization decides order among things that matched; it can never decide whether anything matched. Authority is applied after admission, so a class orders what matched and never rescues what did not. Retrieval may therefore return nothing, and on a scene unrelated to the library it does. The other decisions that each replaced an obvious wrong one: - The class multiplies relevance rather than adding to it. An additive bonus satisfies "Canon outranks Reference" and makes "do not include irrelevant Canon" impossible, because a large enough constant wins on its own. - The semantic floor is measured, not guessed: 113 production-path pairs against nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at 0.36-0.56, and 0.58 sits between them. Because it is a property of that model and not of cosine similarity, it is keyed to the model rather than applied to whatever is configured: an embedding model with no measured calibration in this build does not borrow the number. Semantic admission is skipped, the campaign retrieves lexically, and the reason is stated in the knowledge status and in the turn's provenance. Degrading to lexical keeps the library usable; lending the threshold to an unmeasured model is how the admitted-everything defect would return. - One lexical term is not evidence. Two distinct meaningful terms, or one that is neither a standing campaign entity nor a negligible share of the query. The stop list grew from 42 words to 261, all function words — no subject matter, because a stop list that removes subject matter stops finding "The Silver Key". - Lexical retrieval is a production path, not a fallback. It finds the proper nouns and invented terms a setting bible is made of, and the library is fully usable with no embedding model configured. Safety is structural rather than filtered. Imported text reaches the prompt whole, inside a section that says what it is, under a rule stating the authority order in words and refusing every instruction inside it. No endpoint accepts a filesystem path, so H08 has no mechanism to escape from. Nothing renders imported content as HTML, so a script tag is five visible characters and a remote image is never fetched. Import, chunking, indexing, retrieval and a turn open no socket at all; only embeddings do, through the endpoint allowlist the memory bank already uses. Provenance is the rendered text, not a foreign key: deleting a source cannot turn a historical turn's evidence into dangling ids. Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5 virtual table attached to knowledge_chunks as a DDL hook so it is created and dropped with the table it indexes. Migration 92. A pre-M7 database opens unchanged and needs no sources to play. Bundle: the source content and the reader's judgements about it travel; the passages, index rows and vectors are rebuilt on import, so a restored campaign is searchable immediately without a reindex step. One runtime dependency: python-multipart, Starlette's multipart parser. It is what makes the upload surface possible, and the upload surface is why no pathname is ever accepted. The test doubles were the reason the defect shipped, so they were corrected too. The retrieval stub scored unrelated text at 0.06-0.20 where the real model scores it at 0.43-0.44, and its docstring said it had deliberately removed the constant component that "would put a similarity floor under every pair" — which is exactly the property real models have. The stub now has that floor, one test fails if it is ever removed, and another reproduces the superseded rule and asserts it is still fooled by the same fixture. Run against the pre-corrective implementation, the new suite fails 13 of 18. Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of which mocks nothing between itself and Ollama and re-measures the similarity separation on every run. 43/43 checks in a real Firefox, reproduced. Docker build clean. Four other defects found by review or by the browser run were fixed here rather than carried: an unreachable relevance constant that appeared to enforce something and did not; acceptance tests using the wrong fixture files, so G07's trap was never exercised; a bidirectional override surviving into displayed filenames; and, from the implementation pass, the Insights panel showing M5's two state sections as raw keys and the source inspector refetching on every keystroke. M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK REQUIRED. Both blocking findings are closed, and closeout resolved the embedding-model calibration boundary the corrective pass had left as debt. planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective closeout and the closeout verification in sequence, none overwriting another. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 5
Claude Opus 5
parent
a6e9c7a32b
commit
480414efe0
@@ -0,0 +1,288 @@
|
||||
"""M7: opening a genuine pre-M7 database, and playing on afterwards.
|
||||
|
||||
Two databases are exercised, because they fail differently:
|
||||
|
||||
* **Fresh.** Everything is built by `create_all`, which is the path a new
|
||||
install takes — and the path the FTS5 index nearly missed, because a virtual
|
||||
table is not something SQLAlchemy's metadata describes.
|
||||
* **A real M6 database.** Built by dropping every M7 table and index and
|
||||
rewinding the stamp to 91, so the M7 migration runs its real statements
|
||||
against a schema that genuinely lacks them. A current schema with an old stamp
|
||||
would skip the DDL and test half the change (the lesson
|
||||
`tests/schema_rewind.py` was written for).
|
||||
|
||||
What the second one has to prove is not "the migration completed". It is that a
|
||||
campaign written before M7 existed still behaves: its history, head, branches,
|
||||
Save Points, narrative state, summaries, memories, derived status and prompt
|
||||
provenance are all intact, it needs no knowledge sources to play, and it can
|
||||
then import one and use it.
|
||||
|
||||
python -m pytest tests/test_knowledge_migration.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from fastapi import Depends
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import inspect, select, text
|
||||
|
||||
from app import auth, limits, memorybank, migrations, models
|
||||
from app.database import Base, SessionLocal, engine, get_db
|
||||
from app.knowledge import fts
|
||||
from app.main import app
|
||||
from app.routers import adventures
|
||||
|
||||
from fakes import ScriptedProvider, state_block
|
||||
|
||||
M6_VERSION = 91
|
||||
M7_VERSION = 92
|
||||
|
||||
#: Everything M7 adds to the schema. Dropping all of it and rewinding the stamp
|
||||
#: is what makes the fixture a real M6 database rather than a current one
|
||||
#: wearing an old number.
|
||||
M7_TABLES = ("knowledge_embeddings", "knowledge_chunks", "knowledge_sources")
|
||||
|
||||
|
||||
class StubEmbedder:
|
||||
async def embed(self, texts):
|
||||
return [[1.0, float(len(t) % 7), 0.5] for t in texts]
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(monkeypatch):
|
||||
Base.metadata.create_all(bind=engine)
|
||||
memorybank._vector_cache.clear()
|
||||
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
|
||||
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
|
||||
monkeypatch.setattr(memorybank, "embedding_provider", lambda s: StubEmbedder())
|
||||
monkeypatch.setattr(memorybank, "summary_provider", lambda s: StubEmbedder())
|
||||
try:
|
||||
yield _make_client()
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
memorybank._vector_cache.clear()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
|
||||
|
||||
def _make_client():
|
||||
setup = SessionLocal()
|
||||
user = models.User(is_guest=False, email="m7mig@example.com")
|
||||
setup.add(user)
|
||||
setup.flush()
|
||||
setup.add(models.Settings(
|
||||
user_id=user.id, model="test-model", embedding_model="",
|
||||
context_token_budget=4000, max_output_tokens=300,
|
||||
))
|
||||
adventure = models.Adventure(user_id=user.id, title="Pre-M7 Campaign")
|
||||
setup.add(adventure)
|
||||
setup.flush()
|
||||
setup.add(models.Action(adventure_id=adventure.id, type="start",
|
||||
text="The road forks at the Crooked Lantern."))
|
||||
setup.commit()
|
||||
adv_id, user_id = adventure.id, user.id
|
||||
setup.close()
|
||||
app.dependency_overrides[auth.get_current_user] = (
|
||||
lambda db=Depends(get_db): db.get(models.User, user_id)
|
||||
)
|
||||
test_client = TestClient(app)
|
||||
test_client.adv_id = adv_id
|
||||
test_client.user_id = user_id
|
||||
return test_client
|
||||
|
||||
|
||||
def play(client, text_, prose="The road bends on past the treeline.", events=None):
|
||||
ScriptedProvider.replies = [f"{prose}\n{state_block(events or [])}"]
|
||||
response = client.post(f"/api/adventures/{client.adv_id}/actions",
|
||||
json={"type": "do", "text": text_})
|
||||
assert response.status_code == 200, response.text[:300]
|
||||
return response
|
||||
|
||||
|
||||
def rewind_to_m6():
|
||||
"""Makes the database genuinely M6: no M7 tables, no M7 index, stamp 91."""
|
||||
with engine.begin() as conn:
|
||||
for table in M7_TABLES:
|
||||
conn.execute(text(f"DROP TABLE IF EXISTS {table}"))
|
||||
conn.execute(text(f"DROP TABLE IF EXISTS {fts.TABLE}"))
|
||||
conn.execute(text(f"PRAGMA user_version = {M6_VERSION}"))
|
||||
|
||||
|
||||
def stamp():
|
||||
with engine.begin() as conn:
|
||||
return conn.execute(text("PRAGMA user_version")).scalar()
|
||||
|
||||
|
||||
def upload(client, name, body, classification):
|
||||
return client.post(
|
||||
f"/api/adventures/{client.adv_id}/knowledge",
|
||||
files={"file": (name, body.encode(), "text/markdown")},
|
||||
data={"classification": classification},
|
||||
)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- fresh
|
||||
|
||||
def test_a_fresh_database_gets_every_m7_table_and_the_fts_index(client):
|
||||
"""The `create_all` path, including the virtual table it cannot describe."""
|
||||
tables = set(inspect(engine).get_table_names())
|
||||
for table in M7_TABLES:
|
||||
assert table in tables
|
||||
assert fts.TABLE in tables
|
||||
assert stamp() == migrations.LATEST_VERSION == M7_VERSION
|
||||
|
||||
# And it works end to end on that fresh database.
|
||||
assert upload(client, "canon.md",
|
||||
"# Abbey\n\nThe Old Abbey lies north of Westhaven.\n",
|
||||
"canon").status_code == 201
|
||||
play(client, "Aldric asks about the Old Abbey north of Westhaven.")
|
||||
report = client.get(f"/api/adventures/{client.adv_id}/context").json()
|
||||
assert "canon.md" in [u["filename"] for u in report["knowledge"]["used"]]
|
||||
|
||||
|
||||
# ------------------------------------------------------------ a real M6 db
|
||||
|
||||
def test_a_real_m6_database_migrates_and_keeps_everything_it_had(client):
|
||||
"""The migration, against a database that genuinely predates M7."""
|
||||
# --- build a campaign with one of everything M6 owns ---
|
||||
play(client, "Aldric leaves the tavern.")
|
||||
play(client, "Aldric walks the north road.",
|
||||
events=[{"type": "create_entity", "entity": "aldric", "name": "Aldric",
|
||||
"entity_type": "character"}])
|
||||
play(client, "Aldric reaches the abbey gate.",
|
||||
events=[{"type": "add_fact", "fact_id": "at-gate", "subject": "aldric",
|
||||
"predicate": "stands at", "value": "the abbey gate"}])
|
||||
save_point = client.post(f"/api/adventures/{client.adv_id}/checkpoints",
|
||||
json={"name": "At the gate"}).json()
|
||||
assert client.post(f"/api/adventures/{client.adv_id}/undo").status_code == 200
|
||||
play(client, "Aldric turns back instead.", prose="He turns back toward the town.")
|
||||
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
db.add(models.Summary(
|
||||
adventure_id=adventure.id, text="Aldric has been walking north.",
|
||||
branch_id=adventure.head_branch_id, depth=adventure.head_depth,
|
||||
source_start=0, source_end=adventure.head_depth, trigger="interval",
|
||||
))
|
||||
memory = models.Memory(
|
||||
adventure_id=adventure.id, text="Aldric left the Crooked Lantern.",
|
||||
branch_id=adventure.head_branch_id, depth=adventure.head_depth,
|
||||
)
|
||||
memorybank.set_vector(memory, [1.0, 2.0, 3.0])
|
||||
db.add(memory)
|
||||
db.add(models.DerivedStatus(
|
||||
adventure_id=adventure.id, kind="summary", status="ok"))
|
||||
db.commit()
|
||||
|
||||
before = {
|
||||
"actions": client.get(f"/api/adventures/{client.adv_id}/actions").json(),
|
||||
"branches": client.get(f"/api/adventures/{client.adv_id}/branches").json(),
|
||||
"checkpoints": client.get(f"/api/adventures/{client.adv_id}/checkpoints").json(),
|
||||
"state": client.get(f"/api/adventures/{client.adv_id}/state").json(),
|
||||
"derived": client.get(f"/api/adventures/{client.adv_id}/derived").json(),
|
||||
"memories": client.get(f"/api/adventures/{client.adv_id}/memories").json(),
|
||||
}
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
head_before = (adventure.head_branch_id, adventure.head_depth)
|
||||
state_before = adventure.narrative_state
|
||||
ai_action = next(a for a in reversed(before["actions"]["actions"])
|
||||
if a["type"] == "ai")
|
||||
snapshot_before = client.get(
|
||||
f"/api/adventures/{client.adv_id}/actions/{ai_action['id']}/context"
|
||||
).json()
|
||||
|
||||
# --- make it an M6 database, then migrate it ---
|
||||
rewind_to_m6()
|
||||
tables = set(inspect(engine).get_table_names())
|
||||
assert not (set(M7_TABLES) & tables)
|
||||
assert fts.TABLE not in tables
|
||||
assert stamp() == M6_VERSION
|
||||
|
||||
migrations.bootstrap(engine)
|
||||
|
||||
assert stamp() == M7_VERSION
|
||||
tables = set(inspect(engine).get_table_names())
|
||||
for table in M7_TABLES + (fts.TABLE,):
|
||||
assert table in tables, table
|
||||
|
||||
# --- everything M6 had still behaves ---
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/actions").json() \
|
||||
== before["actions"]
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/branches").json() \
|
||||
== before["branches"]
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/checkpoints").json() \
|
||||
== before["checkpoints"]
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/state").json() \
|
||||
== before["state"]
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/memories").json() \
|
||||
== before["memories"]
|
||||
derived_after = client.get(f"/api/adventures/{client.adv_id}/derived").json()
|
||||
assert derived_after["summaries"] == before["derived"]["summaries"]
|
||||
assert derived_after["status"] == before["derived"]["status"]
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
assert (adventure.head_branch_id, adventure.head_depth) == head_before
|
||||
assert adventure.narrative_state == state_before
|
||||
|
||||
# Prompt provenance from before the migration is still readable, and its
|
||||
# M6 components are unchanged.
|
||||
snapshot_after = client.get(
|
||||
f"/api/adventures/{client.adv_id}/actions/{ai_action['id']}/context"
|
||||
).json()
|
||||
assert snapshot_after["sections"] == snapshot_before["sections"]
|
||||
assert snapshot_after["summary"] == snapshot_before["summary"]
|
||||
assert snapshot_after["memories"] == snapshot_before["memories"]
|
||||
|
||||
# The campaign needs no knowledge sources to keep playing.
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/knowledge").json() == []
|
||||
report = client.get(f"/api/adventures/{client.adv_id}/context").json()
|
||||
assert report["knowledge"]["used"] == []
|
||||
assert not any(s["label"].startswith("imported_") for s in report["sections"])
|
||||
play(client, "Aldric keeps walking.")
|
||||
|
||||
# Undo, Redo and Save Point restore all still work after the migration.
|
||||
assert client.post(f"/api/adventures/{client.adv_id}/undo").status_code == 200
|
||||
assert client.post(f"/api/adventures/{client.adv_id}/redo").status_code == 200
|
||||
assert client.post(
|
||||
f"/api/adventures/{client.adv_id}/checkpoints/{save_point['id']}/restore"
|
||||
).status_code == 200
|
||||
|
||||
# --- and it can now use the new subsystem ---
|
||||
assert upload(client, "canon.md",
|
||||
"# The Abbey\n\nThe Old Abbey lies five miles north of "
|
||||
"Westhaven and its crypt bears a broken circle.\n",
|
||||
"canon").status_code == 201
|
||||
play(client, "Aldric asks about the Old Abbey and its broken-circle symbol.")
|
||||
report = client.get(f"/api/adventures/{client.adv_id}/context").json()
|
||||
assert "canon.md" in [u["filename"] for u in report["knowledge"]["used"]]
|
||||
|
||||
|
||||
def test_the_migration_is_idempotent(client):
|
||||
"""Running it twice is not a second migration."""
|
||||
rewind_to_m6()
|
||||
migrations.bootstrap(engine)
|
||||
upload(client, "canon.md", "# Abbey\n\nThe abbey stands.\n", "canon")
|
||||
with SessionLocal() as db:
|
||||
rows = len(db.execute(select(models.KnowledgeChunk)).scalars().all())
|
||||
|
||||
migrations.bootstrap(engine)
|
||||
assert stamp() == M7_VERSION
|
||||
with SessionLocal() as db:
|
||||
assert len(db.execute(select(models.KnowledgeChunk)).scalars().all()) == rows
|
||||
assert len(client.get(f"/api/adventures/{client.adv_id}/knowledge").json()) == 1
|
||||
|
||||
|
||||
def test_the_fts_index_is_dropped_with_the_table_it_indexes():
|
||||
"""`create_all`/`drop_all` carry the virtual table both ways.
|
||||
|
||||
Without this, a teardown would leave the index holding rowids for chunks
|
||||
that no longer exist, and the next campaign's first passage would inherit a
|
||||
stranger's search results.
|
||||
"""
|
||||
Base.metadata.create_all(bind=engine)
|
||||
assert fts.TABLE in inspect(engine).get_table_names()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
assert fts.TABLE not in inspect(engine).get_table_names()
|
||||
Base.metadata.create_all(bind=engine)
|
||||
with engine.begin() as conn:
|
||||
assert conn.execute(text(f"SELECT count(*) FROM {fts.TABLE}")).scalar() == 0
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
Reference in New Issue
Block a user