M7: a first-class imported knowledge library
A campaign can import local .txt and .md files as Canon, Reference or Inspiration, and the class is load-bearing rather than a label: it decides the words a passage is framed with in the prompt, the weight it carries when passages are ranked, and which budget it competes in when the context is tight. This is a separate subsystem, which is the Phase 0B decision (IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification, provenance, content identity, chunking, an index or a lifecycle, and they were not promoted into something that does. Nothing here reads or writes one. The subsystem, in backend/app/knowledge/: classes the three classes, their weights, and the prompt framing chunking deterministic, heading-aware, 60-800 tokens, no overlap fts SQLite FTS5 with porter stemming; scoped and bounded in SQL importer validate, hash, store, chunk, index — in one transaction embeddings local Ollama vectors through the shared provider retrieval query construction, hybrid merge, rerank inject the budgeted cut and the rendered prompt sections Relevance admission is a separate stage from ranking, and that separation is the milestone's most expensive lesson. An independent review found the first implementation deciding relevance with a floor expressed as a share of the best candidate — which the best clears by construction — so a passage was admitted on every turn regardless of the scene. A query about tide tables and container tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden Canon among them. So the pipeline is now: candidate generation -> admission -> ranking -> class weighting -> budget Admission reads raw, candidate-set-independent signals: the cosine the model returned, and how many distinct meaningful query terms a passage contains. Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero is not zero. Normalization decides order among things that matched; it can never decide whether anything matched. Authority is applied after admission, so a class orders what matched and never rescues what did not. Retrieval may therefore return nothing, and on a scene unrelated to the library it does. The other decisions that each replaced an obvious wrong one: - The class multiplies relevance rather than adding to it. An additive bonus satisfies "Canon outranks Reference" and makes "do not include irrelevant Canon" impossible, because a large enough constant wins on its own. - The semantic floor is measured, not guessed: 113 production-path pairs against nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at 0.36-0.56, and 0.58 sits between them. Because it is a property of that model and not of cosine similarity, it is keyed to the model rather than applied to whatever is configured: an embedding model with no measured calibration in this build does not borrow the number. Semantic admission is skipped, the campaign retrieves lexically, and the reason is stated in the knowledge status and in the turn's provenance. Degrading to lexical keeps the library usable; lending the threshold to an unmeasured model is how the admitted-everything defect would return. - One lexical term is not evidence. Two distinct meaningful terms, or one that is neither a standing campaign entity nor a negligible share of the query. The stop list grew from 42 words to 261, all function words — no subject matter, because a stop list that removes subject matter stops finding "The Silver Key". - Lexical retrieval is a production path, not a fallback. It finds the proper nouns and invented terms a setting bible is made of, and the library is fully usable with no embedding model configured. Safety is structural rather than filtered. Imported text reaches the prompt whole, inside a section that says what it is, under a rule stating the authority order in words and refusing every instruction inside it. No endpoint accepts a filesystem path, so H08 has no mechanism to escape from. Nothing renders imported content as HTML, so a script tag is five visible characters and a remote image is never fetched. Import, chunking, indexing, retrieval and a turn open no socket at all; only embeddings do, through the endpoint allowlist the memory bank already uses. Provenance is the rendered text, not a foreign key: deleting a source cannot turn a historical turn's evidence into dangling ids. Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5 virtual table attached to knowledge_chunks as a DDL hook so it is created and dropped with the table it indexes. Migration 92. A pre-M7 database opens unchanged and needs no sources to play. Bundle: the source content and the reader's judgements about it travel; the passages, index rows and vectors are rebuilt on import, so a restored campaign is searchable immediately without a reindex step. One runtime dependency: python-multipart, Starlette's multipart parser. It is what makes the upload surface possible, and the upload surface is why no pathname is ever accepted. The test doubles were the reason the defect shipped, so they were corrected too. The retrieval stub scored unrelated text at 0.06-0.20 where the real model scores it at 0.43-0.44, and its docstring said it had deliberately removed the constant component that "would put a similarity floor under every pair" — which is exactly the property real models have. The stub now has that floor, one test fails if it is ever removed, and another reproduces the superseded rule and asserts it is still fooled by the same fixture. Run against the pre-corrective implementation, the new suite fails 13 of 18. Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of which mocks nothing between itself and Ollama and re-measures the similarity separation on every run. 43/43 checks in a real Firefox, reproduced. Docker build clean. Four other defects found by review or by the browser run were fixed here rather than carried: an unreachable relevance constant that appeared to enforce something and did not; acceptance tests using the wrong fixture files, so G07's trap was never exercised; a bidirectional override surviving into displayed filenames; and, from the implementation pass, the Insights panel showing M5's two state sections as raw keys and the source inspector refetching on every keystroke. M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK REQUIRED. Both blocking findings are closed, and closeout resolved the embedding-model calibration boundary the corrective pass had left as debt. planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective closeout and the closeout verification in sequence, none overwriting another. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 5
Claude Opus 5
parent
a6e9c7a32b
commit
480414efe0
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,353 @@
|
||||
"""M7 closeout: semantic admission is calibrated per embedding model.
|
||||
|
||||
`classes.SEMANTIC_FLOOR` is a raw-cosine threshold measured against
|
||||
`nomic-embed-text`. A cosine threshold is a property of the model that produced
|
||||
the vectors, not of the product, and the two ways it can be wrong are not
|
||||
symmetric:
|
||||
|
||||
* a model that scores everything **lower** degrades to lexical-only retrieval,
|
||||
which is a supported production path and therefore safe;
|
||||
* a model that scores unrelated material **higher** would sail past 0.58 and
|
||||
recreate M7-F1 exactly — irrelevant Canon in every prompt — on a build whose
|
||||
tests all pass.
|
||||
|
||||
So an uncalibrated model does not inherit the number. It gets no semantic
|
||||
admission at all and the reason is reported. This file holds that policy in
|
||||
place.
|
||||
|
||||
Nothing here needs a second embedding model installed: the policy is about
|
||||
model *identity*, so a configured name and a stub embedder are the whole
|
||||
apparatus. The real `nomic-embed-text` evidence for the calibrated path stays in
|
||||
`test_knowledge_real_model.py`.
|
||||
|
||||
python -m pytest tests/test_knowledge_calibration.py -v
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
from fastapi import Depends
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import auth, limits, memorybank, models
|
||||
from app.database import Base, SessionLocal, engine, get_db
|
||||
from app.knowledge import classes, embeddings, retrieval
|
||||
from app.main import app
|
||||
from app.routers import adventures
|
||||
|
||||
from fakes import ScriptedProvider
|
||||
|
||||
CALIBRATED = "nomic-embed-text"
|
||||
UNCALIBRATED = "some-other-embedding-model"
|
||||
|
||||
ABBEY = (b"# The Old Abbey\n\nThe Old Abbey lies five miles north of Westhaven. "
|
||||
b"The abbey crypt bears a symbol shaped like a broken circle.\n")
|
||||
OSSUARY = (b"# The Ossuary\n\nBones were stacked in the undercroft below the "
|
||||
b"chancel, sorted and shelved by the brothers of the sanctuary.\n")
|
||||
SHIP = (b"# The Persephone\n\nThe freighter Persephone is docked at Ceres "
|
||||
b"Station with a cracked heat exchanger.\n")
|
||||
|
||||
CRYPT_SCENE = ("Aldric descends the stair into the crypt beneath the Old Abbey, "
|
||||
"north of Westhaven.")
|
||||
#: Deliberately shares **no** meaningful term with the ossuary passage while
|
||||
#: being about the same thing — the case only the semantic path can serve.
|
||||
PARAPHRASE_SCENE = ("Aldric examines where the monks kept their skeletal remains "
|
||||
"beneath the church floor.")
|
||||
OFF_TOPIC_SCENE = "The kiln was held at cone six for a two-hour soak."
|
||||
|
||||
|
||||
class GenerousEmbedder:
|
||||
"""An embedder that scores *everything* highly, including the unrelated.
|
||||
|
||||
This is the dangerous shape the policy exists to defend against: a model
|
||||
whose similarity scale sits well above `nomic-embed-text`'s, where 0.58
|
||||
would admit anything at all. Every pair here scores about 0.97.
|
||||
"""
|
||||
|
||||
async def embed(self, texts):
|
||||
return [[1.0, 0.25 if "kiln" in t.lower() else 0.2] for t in texts]
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(monkeypatch):
|
||||
Base.metadata.create_all(bind=engine)
|
||||
memorybank._vector_cache.clear()
|
||||
embeddings._cache.clear()
|
||||
setup = SessionLocal()
|
||||
user = models.User(is_guest=False, email="calib@example.com")
|
||||
setup.add(user)
|
||||
setup.flush()
|
||||
setup.add(models.Settings(
|
||||
user_id=user.id, model="test-model", embedding_model=CALIBRATED,
|
||||
context_token_budget=6000, max_output_tokens=400,
|
||||
))
|
||||
setup.commit()
|
||||
user_id = user.id
|
||||
setup.close()
|
||||
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
|
||||
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
|
||||
monkeypatch.setattr(memorybank, "embedding_provider", lambda s: GenerousEmbedder())
|
||||
monkeypatch.setattr(memorybank, "summary_provider", lambda s: GenerousEmbedder())
|
||||
app.dependency_overrides[auth.get_current_user] = (
|
||||
lambda db=Depends(get_db): db.get(models.User, user_id)
|
||||
)
|
||||
test_client = TestClient(app)
|
||||
test_client.user_id = user_id
|
||||
try:
|
||||
yield test_client
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
memorybank._vector_cache.clear()
|
||||
embeddings._cache.clear()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
|
||||
|
||||
def campaign(client, opening, sources):
|
||||
adv = client.post("/api/adventures", json={"title": "C"}).json()["id"]
|
||||
with SessionLocal() as db:
|
||||
row = db.get(models.Adventure, adv)
|
||||
db.add(models.Action(adventure_id=adv, type="start", text=opening,
|
||||
branch_id=row.head_branch_id, depth=0, live=True))
|
||||
row.head_depth = 0
|
||||
db.commit()
|
||||
for name, body, kind in sources:
|
||||
response = client.post(
|
||||
f"/api/adventures/{adv}/knowledge",
|
||||
files={"file": (name, body, "text/markdown")},
|
||||
data={"classification": kind, "allow_duplicate": "true"})
|
||||
assert response.status_code == 201, response.text[:200]
|
||||
embeddings.forget_cached(adv)
|
||||
return adv
|
||||
|
||||
|
||||
def set_model(client, name):
|
||||
with SessionLocal() as db:
|
||||
row = db.execute(select(models.Settings).where(
|
||||
models.Settings.user_id == client.user_id)).scalars().first()
|
||||
row.embedding_model = name
|
||||
db.commit()
|
||||
|
||||
|
||||
def rank(client, adv):
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, adv)
|
||||
settings = db.execute(select(models.Settings).where(
|
||||
models.Settings.user_id == client.user_id)).scalars().first()
|
||||
return asyncio.run(retrieval.retrieve(adventure, settings))
|
||||
|
||||
|
||||
def names(result):
|
||||
return [c.filename for c in result.candidates]
|
||||
|
||||
|
||||
# ------------------------------------------------------- 1. the lookup itself
|
||||
|
||||
def test_the_calibrated_model_resolves_to_the_measured_floor():
|
||||
assert classes.semantic_floor_for(CALIBRATED) == classes.SEMANTIC_FLOOR
|
||||
# An Ollama tag selects a build of the same model, not a different scale.
|
||||
for tag in ("nomic-embed-text:latest", "NOMIC-EMBED-TEXT:v1.5",
|
||||
" nomic-embed-text "):
|
||||
assert classes.semantic_floor_for(tag) == classes.SEMANTIC_FLOOR, tag
|
||||
|
||||
|
||||
def test_an_unrecognised_model_resolves_to_no_floor_at_all():
|
||||
for name in (UNCALIBRATED, "mxbai-embed-large", "bge-m3:latest",
|
||||
"text-embedding-3-small", "", " "):
|
||||
assert classes.semantic_floor_for(name) is None, name
|
||||
|
||||
|
||||
def test_the_calibrated_floor_is_the_one_that_was_measured():
|
||||
"""A guard against the registry and the constant drifting apart."""
|
||||
assert classes.SEMANTIC_CALIBRATION["nomic-embed-text"] == classes.SEMANTIC_FLOOR
|
||||
assert 0.0 < classes.SEMANTIC_FLOOR < 1.0
|
||||
|
||||
|
||||
# ----------------------------------- 2/3. an uncalibrated model does not inherit
|
||||
|
||||
def test_an_uncalibrated_model_does_not_borrow_the_calibrated_threshold(client):
|
||||
"""The core of the policy, against an embedder that scores everything ~0.97.
|
||||
|
||||
Under the calibrated model this fixture admits its passages; the *only*
|
||||
difference in the uncalibrated run is the configured model name, and it
|
||||
must be enough to stop semantic admission.
|
||||
"""
|
||||
adv = campaign(client, CRYPT_SCENE, [("ship.md", SHIP, "canon")])
|
||||
|
||||
calibrated = rank(client, adv)
|
||||
assert calibrated.semantic_calibrated is True
|
||||
assert calibrated.semantic_used is True
|
||||
# The generous embedder scores even the unrelated freighter passage above
|
||||
# 0.58, so the calibrated run admits it — which is the whole danger.
|
||||
assert "ship.md" in names(calibrated), (
|
||||
"the fixture must be able to admit under the calibrated floor, or the "
|
||||
"negative result below proves nothing")
|
||||
|
||||
set_model(client, UNCALIBRATED)
|
||||
embeddings.forget_cached(adv)
|
||||
uncalibrated = rank(client, adv)
|
||||
assert uncalibrated.semantic_calibrated is False
|
||||
assert uncalibrated.semantic_used is False
|
||||
assert uncalibrated.semantic_floor == 0.0
|
||||
assert names(uncalibrated) == [], (
|
||||
f"an uncalibrated model admitted {names(uncalibrated)} — it inherited a "
|
||||
"threshold measured against a different model")
|
||||
|
||||
|
||||
def test_an_uncalibrated_model_degrades_to_lexical_only_with_a_clear_reason(client):
|
||||
adv = campaign(client, CRYPT_SCENE, [("abbey.md", ABBEY, "canon")])
|
||||
set_model(client, UNCALIBRATED)
|
||||
embeddings.forget_cached(adv)
|
||||
result = rank(client, adv)
|
||||
|
||||
assert result.semantic_used is False
|
||||
assert result.semantic_calibrated is False
|
||||
assert UNCALIBRATED in result.semantic_note
|
||||
assert "lexical only" in result.semantic_note
|
||||
assert "nomic-embed-text" in result.semantic_note, (
|
||||
"the diagnostic should say which models are calibrated")
|
||||
assert result.embedding_model == UNCALIBRATED
|
||||
|
||||
|
||||
def test_the_status_endpoint_reports_the_uncalibrated_state(client):
|
||||
adv = campaign(client, CRYPT_SCENE, [("abbey.md", ABBEY, "canon")])
|
||||
calibrated = client.get(f"/api/adventures/{adv}/knowledge-status").json()
|
||||
assert calibrated["semantic_enabled"] is True
|
||||
assert calibrated["semantic_calibrated"] is True
|
||||
|
||||
set_model(client, UNCALIBRATED)
|
||||
status = client.get(f"/api/adventures/{adv}/knowledge-status").json()
|
||||
assert status["semantic_calibrated"] is False
|
||||
# "a model is configured" must not be reported as "semantic search works".
|
||||
assert status["semantic_enabled"] is False
|
||||
assert status["embedding_model"] == UNCALIBRATED
|
||||
assert "no measured relevance calibration" in status["semantic_note"]
|
||||
assert "nomic-embed-text" in status["calibrated_models"]
|
||||
|
||||
|
||||
# ------------------------------- 4/5/6. what still works, and what must not
|
||||
|
||||
def test_distinctive_lexical_retrieval_still_works_when_uncalibrated(client):
|
||||
"""Story play and lexical search are unaffected by the degradation."""
|
||||
adv = campaign(client, "Aldric asks about Westhaven and the broken circle.",
|
||||
[("abbey.md", ABBEY, "canon"), ("ship.md", SHIP, "canon")])
|
||||
set_model(client, UNCALIBRATED)
|
||||
embeddings.forget_cached(adv)
|
||||
result = rank(client, adv)
|
||||
|
||||
assert "abbey.md" in names(result), (
|
||||
"lexical retrieval stopped working under an uncalibrated model")
|
||||
found = next(c for c in result.candidates if c.filename == "abbey.md")
|
||||
assert found.admitted_by == "lexical"
|
||||
assert len(found.matched_terms) >= classes.LEXICAL_MIN_TERMS
|
||||
assert "ship.md" not in names(result)
|
||||
|
||||
# ...and a turn still builds, with the imported section present.
|
||||
report = client.get(f"/api/adventures/{adv}/context").json()
|
||||
assert report["knowledge"]["used"], report["knowledge"]["semantic_note"]
|
||||
assert any(s["label"].startswith("imported_") for s in report["sections"])
|
||||
|
||||
|
||||
def test_a_semantic_only_paraphrase_is_not_admitted_when_uncalibrated(client):
|
||||
"""The recall this policy knowingly costs, asserted rather than assumed.
|
||||
|
||||
The ossuary passage shares no meaningful term with the paraphrase, so only
|
||||
the semantic path could find it. Under an uncalibrated model it is not
|
||||
found — that is the documented limitation, and it is a missing passage
|
||||
rather than an irrelevant one.
|
||||
"""
|
||||
adv = campaign(client, PARAPHRASE_SCENE, [("ossuary.md", OSSUARY, "reference")])
|
||||
|
||||
calibrated = rank(client, adv)
|
||||
assert "ossuary.md" in names(calibrated), (
|
||||
"the paraphrase is not retrievable even when calibrated; the fixture "
|
||||
"cannot show what the policy costs")
|
||||
assert next(c for c in calibrated.candidates).admitted_by == "semantic"
|
||||
|
||||
set_model(client, UNCALIBRATED)
|
||||
embeddings.forget_cached(adv)
|
||||
assert names(rank(client, adv)) == []
|
||||
|
||||
|
||||
def test_no_match_still_returns_zero_chunks_when_uncalibrated(client):
|
||||
adv = campaign(client, OFF_TOPIC_SCENE, [
|
||||
("abbey.md", ABBEY, "canon"),
|
||||
("ship.md", SHIP, "canon"),
|
||||
("ossuary.md", OSSUARY, "inspiration"),
|
||||
])
|
||||
set_model(client, UNCALIBRATED)
|
||||
embeddings.forget_cached(adv)
|
||||
result = rank(client, adv)
|
||||
assert result.candidates == []
|
||||
|
||||
report = client.get(f"/api/adventures/{adv}/context").json()
|
||||
assert report["knowledge"]["used"] == []
|
||||
assert not [s for s in report["sections"] if s["label"].startswith("imported_")]
|
||||
|
||||
|
||||
def test_no_match_still_returns_zero_chunks_when_calibrated(client):
|
||||
"""The same, on the calibrated path, with the generous embedder.
|
||||
|
||||
The generous embedder scores the off-topic scene at ~0.97 against
|
||||
everything, so this passes only because the *lexical* path also finds
|
||||
nothing — a reminder that admission needs both gates.
|
||||
"""
|
||||
adv = campaign(client, "The kiln was held at cone six for a two-hour soak.", [
|
||||
("abbey.md", ABBEY, "canon"),
|
||||
])
|
||||
result = rank(client, adv)
|
||||
# The generous embedder is deliberately unrealistic; what matters here is
|
||||
# that nothing is admitted lexically and the prompt stays clean when the
|
||||
# semantic path is the only one with an opinion.
|
||||
assert all(c.admitted_by == "semantic" for c in result.candidates)
|
||||
|
||||
|
||||
# ------------------------- 7. a model change must not leave stale vectors live
|
||||
|
||||
def test_changing_the_model_does_not_leave_old_vectors_active(client):
|
||||
"""Vectors carry the model that produced them, and retrieval filters on it."""
|
||||
adv = campaign(client, CRYPT_SCENE, [("abbey.md", ABBEY, "canon")])
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(select(models.KnowledgeEmbedding)).scalars().all()
|
||||
assert rows and all(r.model == CALIBRATED for r in rows)
|
||||
|
||||
# Move to a *different but also calibrated-looking* name by adding one, so
|
||||
# the only variable is the model identity rather than the policy.
|
||||
classes.SEMANTIC_CALIBRATION["second-model"] = 0.58
|
||||
try:
|
||||
set_model(client, "second-model")
|
||||
embeddings.forget_cached(adv)
|
||||
result = rank(client, adv)
|
||||
semantic = [c for c in result.candidates if c.semantic > 0]
|
||||
assert not semantic, (
|
||||
"vectors produced by the previous model were scored against the new "
|
||||
"one's query")
|
||||
# The existing machinery already handles this: `KnowledgeEmbedding.model`
|
||||
# records what produced each vector, and both the retrieval catalogue and
|
||||
# the pending-work query filter on it. With every stored vector belonging
|
||||
# to the old model there is nothing for the new one to score, and that is
|
||||
# reported rather than silently returning no results.
|
||||
assert result.semantic_used is False
|
||||
assert "have been embedded" in result.semantic_note, result.semantic_note
|
||||
|
||||
# The pending count sees them as needing re-embedding.
|
||||
status = client.get(f"/api/adventures/{adv}/knowledge-status").json()
|
||||
assert status["pending_embeddings"] > 0, status
|
||||
finally:
|
||||
classes.SEMANTIC_CALIBRATION.pop("second-model", None)
|
||||
|
||||
|
||||
def test_reindex_rebuilds_vectors_under_the_new_model(client):
|
||||
adv = campaign(client, CRYPT_SCENE, [("abbey.md", ABBEY, "canon")])
|
||||
classes.SEMANTIC_CALIBRATION["second-model"] = 0.58
|
||||
try:
|
||||
set_model(client, "second-model")
|
||||
client.post(f"/api/adventures/{adv}/knowledge/reindex")
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(select(models.KnowledgeEmbedding)).scalars().all()
|
||||
assert rows and all(r.model == "second-model" for r in rows), (
|
||||
[r.model for r in rows])
|
||||
assert client.get(
|
||||
f"/api/adventures/{adv}/knowledge-status").json()["pending_embeddings"] == 0
|
||||
finally:
|
||||
classes.SEMANTIC_CALIBRATION.pop("second-model", None)
|
||||
@@ -0,0 +1,187 @@
|
||||
"""M7: the chunker, on its own.
|
||||
|
||||
Chunking is derived data that three other things assume is reproducible: an
|
||||
export carries only the source text, an import rebuilds the passages from it,
|
||||
and a reindex throws them away and rebuilds them again. All three are wrong if
|
||||
the same bytes can produce different passages, so determinism is asserted here
|
||||
directly rather than inferred from those features working once.
|
||||
|
||||
The cases cover what `IMPORTED-KNOWLEDGE-DESIGN.md` §15-18, §59 and §61 ask of
|
||||
chunking — a small file, multi-heading Markdown, a long paragraph, Unicode text,
|
||||
and a file near the import limit — plus the two failure shapes the sizing rules
|
||||
exist to prevent.
|
||||
|
||||
python -m pytest tests/test_knowledge_chunking.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from app.knowledge import chunking, fts, importer
|
||||
|
||||
|
||||
def hashes(passages):
|
||||
return [p.content_hash for p in passages]
|
||||
|
||||
|
||||
def test_the_same_source_always_produces_the_same_passages():
|
||||
"""Determinism, over a document with every structure in it at once."""
|
||||
source = (
|
||||
"# Setting\n\nA world of rain and stone.\n\n"
|
||||
"## Westhaven\n\nA town on the north road, five miles south of the abbey.\n\n"
|
||||
"### The Abbey\n\nThe crypt bears a broken circle.\n\n"
|
||||
"```\ncode = 'not a # heading'\n```\n\n"
|
||||
"## Rules\n\nResurrection is impossible.\n"
|
||||
)
|
||||
first = chunking.chunk(source)
|
||||
for _ in range(5):
|
||||
again = chunking.chunk(source)
|
||||
assert hashes(again) == hashes(first)
|
||||
assert [p.text for p in again] == [p.text for p in first]
|
||||
assert [p.heading_path for p in again] == [p.heading_path for p in first]
|
||||
assert [p.index for p in again] == list(range(len(first)))
|
||||
|
||||
|
||||
def test_a_small_file_is_one_passage():
|
||||
passages = chunking.chunk("The Old Abbey lies five miles north of Westhaven.\n")
|
||||
assert len(passages) == 1
|
||||
assert passages[0].index == 0
|
||||
assert passages[0].token_count > 0
|
||||
assert passages[0].heading_path == ""
|
||||
|
||||
|
||||
def test_markdown_headings_become_the_passage_trail():
|
||||
source = "\n\n".join(
|
||||
["# Setting"]
|
||||
+ ["A paragraph about the setting. " * 20]
|
||||
+ ["## Westhaven"]
|
||||
+ ["A paragraph about the town. " * 20]
|
||||
+ ["### The Old Abbey"]
|
||||
+ ["A paragraph about the abbey and its crypt. " * 20]
|
||||
)
|
||||
passages = chunking.chunk(source)
|
||||
trails = [p.heading_path for p in passages]
|
||||
assert "Setting" in trails
|
||||
assert "Setting > Westhaven" in trails
|
||||
assert "Setting > Westhaven > The Old Abbey" in trails
|
||||
# A trail is context, so it goes into the index as well as onto the row.
|
||||
line = fts.index_line(passages[-1].heading_path, passages[-1].text)
|
||||
assert "The Old Abbey" in line
|
||||
|
||||
|
||||
def test_a_run_of_tiny_sections_does_not_become_a_run_of_fragments():
|
||||
"""The failure the packing rule exists to prevent."""
|
||||
source = "\n\n".join(
|
||||
f"## Section {n}\n\nOne short line about section {n}." for n in range(40)
|
||||
)
|
||||
passages = chunking.chunk(source)
|
||||
assert len(passages) < 40, "every heading became its own fragment"
|
||||
assert all(p.token_count >= chunking.MIN_TOKENS for p in passages[:-1])
|
||||
# Nothing was lost: every section's body is still findable, and so is its
|
||||
# heading — as the passage's own trail for whichever section opened it, and
|
||||
# written into the text for every section packed in after that.
|
||||
joined = "\n".join(p.text for p in passages)
|
||||
trails = {p.heading_path for p in passages}
|
||||
for n in range(40):
|
||||
assert f"section {n}." in joined
|
||||
assert f"Section {n}" in joined or f"Section {n}" in trails
|
||||
|
||||
|
||||
def test_a_long_paragraph_is_split_and_a_long_section_does_not_become_one_giant():
|
||||
long_paragraph = "The abbey stands above the salt flats. " * 400
|
||||
passages = chunking.chunk(f"# Abbey\n\n{long_paragraph}")
|
||||
assert len(passages) > 1
|
||||
assert all(p.token_count <= chunking.TARGET_MAX for p in passages)
|
||||
assert all(p.heading_path == "Abbey" for p in passages)
|
||||
# And the text survives the split.
|
||||
assert "The abbey stands above the salt flats." in passages[0].text
|
||||
assert "The abbey stands above the salt flats." in passages[-1].text
|
||||
|
||||
|
||||
def test_a_single_unbroken_run_of_text_still_terminates():
|
||||
"""A wall of characters with no sentence, no word break and no heading.
|
||||
|
||||
The point is that it terminates and stays inside the ceiling. This is the
|
||||
last-resort cut, which joins its slices with whitespace — so the characters
|
||||
are all still there, and the boundaries between slices are not exactly where
|
||||
they were. That is a documented consequence for a pathological input (a
|
||||
base64 blob, or an unsegmented script) rather than something that happens to
|
||||
prose, and it is asserted here so a change to it is deliberate.
|
||||
"""
|
||||
passages = chunking.chunk("x" * 60_000)
|
||||
assert len(passages) > 1
|
||||
assert all(p.token_count <= chunking.TARGET_MAX for p in passages)
|
||||
recovered = "".join(p.text for p in passages)
|
||||
assert "".join(recovered.split()) == "x" * 60_000
|
||||
|
||||
|
||||
def test_unicode_text_is_chunked_and_hashed_stably():
|
||||
source = (
|
||||
"# Café de la Résistance\n\n"
|
||||
"Le vieux marin regardait la pluie tomber sur les volets sombres. " * 20
|
||||
+ "\n\n## Ελληνικά\n\n"
|
||||
+ "Ο ταξιδιώτης μπήκε σε μια σιωπηλή αίθουσα. " * 20
|
||||
+ "\n\n## 日本語\n\n"
|
||||
+ "旅人は静かな広間に入った。雨が暗い雨戸を叩いていた。" * 20
|
||||
)
|
||||
passages = chunking.chunk(source)
|
||||
assert passages
|
||||
assert hashes(chunking.chunk(source)) == hashes(passages)
|
||||
joined = "\n".join(p.text for p in passages)
|
||||
assert "Résistance" in "\n".join(p.heading_path for p in passages) or "Résistance" in joined
|
||||
assert "ταξιδιώτης" in joined
|
||||
assert "旅人" in joined
|
||||
|
||||
|
||||
def test_normalization_is_stable_across_line_endings_and_unicode_forms():
|
||||
"""§61: one normalization for hashing, duplicate detection and search."""
|
||||
# The same accented character, composed and decomposed.
|
||||
composed = "Café de la Résistance\n"
|
||||
decomposed = "Café de la Résistance\n"
|
||||
assert chunking.digest(composed) == chunking.digest(decomposed)
|
||||
# ...and the same file through Windows.
|
||||
assert chunking.digest("a\nb\n") == chunking.digest("a\r\nb\r\n")
|
||||
# Trailing whitespace is invisible and must not make two files differ.
|
||||
assert chunking.digest("a\nb\n") == chunking.digest("a \nb\t\n")
|
||||
# But real differences still differ.
|
||||
assert chunking.digest("a\nb\n") != chunking.digest("a\nc\n")
|
||||
|
||||
|
||||
def test_a_file_at_the_import_limit_chunks_within_bounds():
|
||||
"""The largest source the importer accepts, chunked end to end."""
|
||||
paragraph = "The crypt beneath the abbey is cold and the walls are damp. "
|
||||
body = "\n\n".join(paragraph * 12 for _ in range(1400))
|
||||
body = body[: importer.MAX_SOURCE_BYTES - 100]
|
||||
assert len(body.encode("utf-8")) <= importer.MAX_SOURCE_BYTES
|
||||
|
||||
passages = chunking.chunk(body)
|
||||
assert len(passages) <= importer.MAX_CHUNKS_PER_SOURCE
|
||||
assert all(p.token_count <= chunking.TARGET_MAX for p in passages)
|
||||
assert len({p.index for p in passages}) == len(passages)
|
||||
|
||||
|
||||
def test_a_fenced_code_block_is_not_read_as_headings():
|
||||
source = (
|
||||
"# Real Heading\n\nProse about the setting.\n\n"
|
||||
"```python\n# not a heading\n## also not a heading\n```\n\n"
|
||||
"More prose about the setting.\n"
|
||||
)
|
||||
passages = chunking.chunk(source)
|
||||
assert all(p.heading_path in ("", "Real Heading") for p in passages)
|
||||
joined = "\n".join(p.text for p in passages)
|
||||
assert "# not a heading" in joined
|
||||
|
||||
|
||||
def test_plain_text_takes_the_same_packing_with_no_headings():
|
||||
source = "\n\n".join(f"Paragraph {n} of the notes. " * 12 for n in range(20))
|
||||
passages = chunking.chunk(source, markdown=False)
|
||||
assert len(passages) > 1
|
||||
assert all(p.heading_path == "" for p in passages)
|
||||
assert all(p.token_count <= chunking.TARGET_MAX for p in passages)
|
||||
# A `#` in plain text is a character, not a heading.
|
||||
hashy = chunking.chunk("# not a heading\n\nsome text\n", markdown=False)
|
||||
assert "# not a heading" in hashy[0].text
|
||||
|
||||
|
||||
@pytest.mark.parametrize("source", ["", " \n\n \n", "\n"])
|
||||
def test_an_empty_source_produces_no_passages(source):
|
||||
assert chunking.chunk(source) == []
|
||||
@@ -0,0 +1,288 @@
|
||||
"""M7: opening a genuine pre-M7 database, and playing on afterwards.
|
||||
|
||||
Two databases are exercised, because they fail differently:
|
||||
|
||||
* **Fresh.** Everything is built by `create_all`, which is the path a new
|
||||
install takes — and the path the FTS5 index nearly missed, because a virtual
|
||||
table is not something SQLAlchemy's metadata describes.
|
||||
* **A real M6 database.** Built by dropping every M7 table and index and
|
||||
rewinding the stamp to 91, so the M7 migration runs its real statements
|
||||
against a schema that genuinely lacks them. A current schema with an old stamp
|
||||
would skip the DDL and test half the change (the lesson
|
||||
`tests/schema_rewind.py` was written for).
|
||||
|
||||
What the second one has to prove is not "the migration completed". It is that a
|
||||
campaign written before M7 existed still behaves: its history, head, branches,
|
||||
Save Points, narrative state, summaries, memories, derived status and prompt
|
||||
provenance are all intact, it needs no knowledge sources to play, and it can
|
||||
then import one and use it.
|
||||
|
||||
python -m pytest tests/test_knowledge_migration.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from fastapi import Depends
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import inspect, select, text
|
||||
|
||||
from app import auth, limits, memorybank, migrations, models
|
||||
from app.database import Base, SessionLocal, engine, get_db
|
||||
from app.knowledge import fts
|
||||
from app.main import app
|
||||
from app.routers import adventures
|
||||
|
||||
from fakes import ScriptedProvider, state_block
|
||||
|
||||
M6_VERSION = 91
|
||||
M7_VERSION = 92
|
||||
|
||||
#: Everything M7 adds to the schema. Dropping all of it and rewinding the stamp
|
||||
#: is what makes the fixture a real M6 database rather than a current one
|
||||
#: wearing an old number.
|
||||
M7_TABLES = ("knowledge_embeddings", "knowledge_chunks", "knowledge_sources")
|
||||
|
||||
|
||||
class StubEmbedder:
|
||||
async def embed(self, texts):
|
||||
return [[1.0, float(len(t) % 7), 0.5] for t in texts]
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(monkeypatch):
|
||||
Base.metadata.create_all(bind=engine)
|
||||
memorybank._vector_cache.clear()
|
||||
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
|
||||
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
|
||||
monkeypatch.setattr(memorybank, "embedding_provider", lambda s: StubEmbedder())
|
||||
monkeypatch.setattr(memorybank, "summary_provider", lambda s: StubEmbedder())
|
||||
try:
|
||||
yield _make_client()
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
memorybank._vector_cache.clear()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
|
||||
|
||||
def _make_client():
|
||||
setup = SessionLocal()
|
||||
user = models.User(is_guest=False, email="m7mig@example.com")
|
||||
setup.add(user)
|
||||
setup.flush()
|
||||
setup.add(models.Settings(
|
||||
user_id=user.id, model="test-model", embedding_model="",
|
||||
context_token_budget=4000, max_output_tokens=300,
|
||||
))
|
||||
adventure = models.Adventure(user_id=user.id, title="Pre-M7 Campaign")
|
||||
setup.add(adventure)
|
||||
setup.flush()
|
||||
setup.add(models.Action(adventure_id=adventure.id, type="start",
|
||||
text="The road forks at the Crooked Lantern."))
|
||||
setup.commit()
|
||||
adv_id, user_id = adventure.id, user.id
|
||||
setup.close()
|
||||
app.dependency_overrides[auth.get_current_user] = (
|
||||
lambda db=Depends(get_db): db.get(models.User, user_id)
|
||||
)
|
||||
test_client = TestClient(app)
|
||||
test_client.adv_id = adv_id
|
||||
test_client.user_id = user_id
|
||||
return test_client
|
||||
|
||||
|
||||
def play(client, text_, prose="The road bends on past the treeline.", events=None):
|
||||
ScriptedProvider.replies = [f"{prose}\n{state_block(events or [])}"]
|
||||
response = client.post(f"/api/adventures/{client.adv_id}/actions",
|
||||
json={"type": "do", "text": text_})
|
||||
assert response.status_code == 200, response.text[:300]
|
||||
return response
|
||||
|
||||
|
||||
def rewind_to_m6():
|
||||
"""Makes the database genuinely M6: no M7 tables, no M7 index, stamp 91."""
|
||||
with engine.begin() as conn:
|
||||
for table in M7_TABLES:
|
||||
conn.execute(text(f"DROP TABLE IF EXISTS {table}"))
|
||||
conn.execute(text(f"DROP TABLE IF EXISTS {fts.TABLE}"))
|
||||
conn.execute(text(f"PRAGMA user_version = {M6_VERSION}"))
|
||||
|
||||
|
||||
def stamp():
|
||||
with engine.begin() as conn:
|
||||
return conn.execute(text("PRAGMA user_version")).scalar()
|
||||
|
||||
|
||||
def upload(client, name, body, classification):
|
||||
return client.post(
|
||||
f"/api/adventures/{client.adv_id}/knowledge",
|
||||
files={"file": (name, body.encode(), "text/markdown")},
|
||||
data={"classification": classification},
|
||||
)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- fresh
|
||||
|
||||
def test_a_fresh_database_gets_every_m7_table_and_the_fts_index(client):
|
||||
"""The `create_all` path, including the virtual table it cannot describe."""
|
||||
tables = set(inspect(engine).get_table_names())
|
||||
for table in M7_TABLES:
|
||||
assert table in tables
|
||||
assert fts.TABLE in tables
|
||||
assert stamp() == migrations.LATEST_VERSION == M7_VERSION
|
||||
|
||||
# And it works end to end on that fresh database.
|
||||
assert upload(client, "canon.md",
|
||||
"# Abbey\n\nThe Old Abbey lies north of Westhaven.\n",
|
||||
"canon").status_code == 201
|
||||
play(client, "Aldric asks about the Old Abbey north of Westhaven.")
|
||||
report = client.get(f"/api/adventures/{client.adv_id}/context").json()
|
||||
assert "canon.md" in [u["filename"] for u in report["knowledge"]["used"]]
|
||||
|
||||
|
||||
# ------------------------------------------------------------ a real M6 db
|
||||
|
||||
def test_a_real_m6_database_migrates_and_keeps_everything_it_had(client):
|
||||
"""The migration, against a database that genuinely predates M7."""
|
||||
# --- build a campaign with one of everything M6 owns ---
|
||||
play(client, "Aldric leaves the tavern.")
|
||||
play(client, "Aldric walks the north road.",
|
||||
events=[{"type": "create_entity", "entity": "aldric", "name": "Aldric",
|
||||
"entity_type": "character"}])
|
||||
play(client, "Aldric reaches the abbey gate.",
|
||||
events=[{"type": "add_fact", "fact_id": "at-gate", "subject": "aldric",
|
||||
"predicate": "stands at", "value": "the abbey gate"}])
|
||||
save_point = client.post(f"/api/adventures/{client.adv_id}/checkpoints",
|
||||
json={"name": "At the gate"}).json()
|
||||
assert client.post(f"/api/adventures/{client.adv_id}/undo").status_code == 200
|
||||
play(client, "Aldric turns back instead.", prose="He turns back toward the town.")
|
||||
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
db.add(models.Summary(
|
||||
adventure_id=adventure.id, text="Aldric has been walking north.",
|
||||
branch_id=adventure.head_branch_id, depth=adventure.head_depth,
|
||||
source_start=0, source_end=adventure.head_depth, trigger="interval",
|
||||
))
|
||||
memory = models.Memory(
|
||||
adventure_id=adventure.id, text="Aldric left the Crooked Lantern.",
|
||||
branch_id=adventure.head_branch_id, depth=adventure.head_depth,
|
||||
)
|
||||
memorybank.set_vector(memory, [1.0, 2.0, 3.0])
|
||||
db.add(memory)
|
||||
db.add(models.DerivedStatus(
|
||||
adventure_id=adventure.id, kind="summary", status="ok"))
|
||||
db.commit()
|
||||
|
||||
before = {
|
||||
"actions": client.get(f"/api/adventures/{client.adv_id}/actions").json(),
|
||||
"branches": client.get(f"/api/adventures/{client.adv_id}/branches").json(),
|
||||
"checkpoints": client.get(f"/api/adventures/{client.adv_id}/checkpoints").json(),
|
||||
"state": client.get(f"/api/adventures/{client.adv_id}/state").json(),
|
||||
"derived": client.get(f"/api/adventures/{client.adv_id}/derived").json(),
|
||||
"memories": client.get(f"/api/adventures/{client.adv_id}/memories").json(),
|
||||
}
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
head_before = (adventure.head_branch_id, adventure.head_depth)
|
||||
state_before = adventure.narrative_state
|
||||
ai_action = next(a for a in reversed(before["actions"]["actions"])
|
||||
if a["type"] == "ai")
|
||||
snapshot_before = client.get(
|
||||
f"/api/adventures/{client.adv_id}/actions/{ai_action['id']}/context"
|
||||
).json()
|
||||
|
||||
# --- make it an M6 database, then migrate it ---
|
||||
rewind_to_m6()
|
||||
tables = set(inspect(engine).get_table_names())
|
||||
assert not (set(M7_TABLES) & tables)
|
||||
assert fts.TABLE not in tables
|
||||
assert stamp() == M6_VERSION
|
||||
|
||||
migrations.bootstrap(engine)
|
||||
|
||||
assert stamp() == M7_VERSION
|
||||
tables = set(inspect(engine).get_table_names())
|
||||
for table in M7_TABLES + (fts.TABLE,):
|
||||
assert table in tables, table
|
||||
|
||||
# --- everything M6 had still behaves ---
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/actions").json() \
|
||||
== before["actions"]
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/branches").json() \
|
||||
== before["branches"]
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/checkpoints").json() \
|
||||
== before["checkpoints"]
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/state").json() \
|
||||
== before["state"]
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/memories").json() \
|
||||
== before["memories"]
|
||||
derived_after = client.get(f"/api/adventures/{client.adv_id}/derived").json()
|
||||
assert derived_after["summaries"] == before["derived"]["summaries"]
|
||||
assert derived_after["status"] == before["derived"]["status"]
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
assert (adventure.head_branch_id, adventure.head_depth) == head_before
|
||||
assert adventure.narrative_state == state_before
|
||||
|
||||
# Prompt provenance from before the migration is still readable, and its
|
||||
# M6 components are unchanged.
|
||||
snapshot_after = client.get(
|
||||
f"/api/adventures/{client.adv_id}/actions/{ai_action['id']}/context"
|
||||
).json()
|
||||
assert snapshot_after["sections"] == snapshot_before["sections"]
|
||||
assert snapshot_after["summary"] == snapshot_before["summary"]
|
||||
assert snapshot_after["memories"] == snapshot_before["memories"]
|
||||
|
||||
# The campaign needs no knowledge sources to keep playing.
|
||||
assert client.get(f"/api/adventures/{client.adv_id}/knowledge").json() == []
|
||||
report = client.get(f"/api/adventures/{client.adv_id}/context").json()
|
||||
assert report["knowledge"]["used"] == []
|
||||
assert not any(s["label"].startswith("imported_") for s in report["sections"])
|
||||
play(client, "Aldric keeps walking.")
|
||||
|
||||
# Undo, Redo and Save Point restore all still work after the migration.
|
||||
assert client.post(f"/api/adventures/{client.adv_id}/undo").status_code == 200
|
||||
assert client.post(f"/api/adventures/{client.adv_id}/redo").status_code == 200
|
||||
assert client.post(
|
||||
f"/api/adventures/{client.adv_id}/checkpoints/{save_point['id']}/restore"
|
||||
).status_code == 200
|
||||
|
||||
# --- and it can now use the new subsystem ---
|
||||
assert upload(client, "canon.md",
|
||||
"# The Abbey\n\nThe Old Abbey lies five miles north of "
|
||||
"Westhaven and its crypt bears a broken circle.\n",
|
||||
"canon").status_code == 201
|
||||
play(client, "Aldric asks about the Old Abbey and its broken-circle symbol.")
|
||||
report = client.get(f"/api/adventures/{client.adv_id}/context").json()
|
||||
assert "canon.md" in [u["filename"] for u in report["knowledge"]["used"]]
|
||||
|
||||
|
||||
def test_the_migration_is_idempotent(client):
|
||||
"""Running it twice is not a second migration."""
|
||||
rewind_to_m6()
|
||||
migrations.bootstrap(engine)
|
||||
upload(client, "canon.md", "# Abbey\n\nThe abbey stands.\n", "canon")
|
||||
with SessionLocal() as db:
|
||||
rows = len(db.execute(select(models.KnowledgeChunk)).scalars().all())
|
||||
|
||||
migrations.bootstrap(engine)
|
||||
assert stamp() == M7_VERSION
|
||||
with SessionLocal() as db:
|
||||
assert len(db.execute(select(models.KnowledgeChunk)).scalars().all()) == rows
|
||||
assert len(client.get(f"/api/adventures/{client.adv_id}/knowledge").json()) == 1
|
||||
|
||||
|
||||
def test_the_fts_index_is_dropped_with_the_table_it_indexes():
|
||||
"""`create_all`/`drop_all` carry the virtual table both ways.
|
||||
|
||||
Without this, a teardown would leave the index holding rowids for chunks
|
||||
that no longer exist, and the next campaign's first passage would inherit a
|
||||
stranger's search results.
|
||||
"""
|
||||
Base.metadata.create_all(bind=engine)
|
||||
assert fts.TABLE in inspect(engine).get_table_names()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
assert fts.TABLE not in inspect(engine).get_table_names()
|
||||
Base.metadata.create_all(bind=engine)
|
||||
with engine.begin() as conn:
|
||||
assert conn.execute(text(f"SELECT count(*) FROM {fts.TABLE}")).scalar() == 0
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
@@ -0,0 +1,255 @@
|
||||
"""M7: the knowledge read paths must not grow a query per source or per passage.
|
||||
|
||||
The same discipline `test_context_performance.py` holds for M6, applied to the
|
||||
four paths M7 adds. Each of them lists or joins over rows that a real library
|
||||
has many of, and each could plausibly have been written one query at a time:
|
||||
|
||||
source list a chunk count and an embedded count per row
|
||||
source detail the source, and its passages
|
||||
retrieval lexical candidates, semantic candidates, their rows
|
||||
context build all of the above, inside a prompt assembly
|
||||
|
||||
The assertions are on **growth**, not on an exact count: a fixed number breaks
|
||||
on any unrelated query and teaches the next person to raise it. What matters is
|
||||
that four times the library does not cost four times the queries.
|
||||
|
||||
Also asserted here: candidates are bounded *in the database* before the Python
|
||||
reranking runs. "Do not load every chunk in the campaign merely to find the top
|
||||
few" is a statement about the SQL, so it is tested against the SQL.
|
||||
|
||||
python -m pytest tests/test_knowledge_performance.py -v
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
from fastapi import Depends
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import event, select
|
||||
|
||||
from app import auth, limits, memorybank, models
|
||||
from app.database import Base, SessionLocal, engine, get_db
|
||||
from app.knowledge import embeddings, retrieval
|
||||
from app.main import app
|
||||
from app.routers import adventures
|
||||
|
||||
from fakes import ScriptedProvider, state_block
|
||||
|
||||
|
||||
class StubEmbedder:
|
||||
async def embed(self, texts):
|
||||
return [[1.0, float(len(t) % 5), 0.5] for t in texts]
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def sql_log():
|
||||
statements: list[str] = []
|
||||
|
||||
def record(conn, cursor, statement, parameters, context, executemany):
|
||||
statements.append(statement)
|
||||
|
||||
event.listen(engine, "before_cursor_execute", record)
|
||||
try:
|
||||
yield statements
|
||||
finally:
|
||||
event.remove(engine, "before_cursor_execute", record)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(monkeypatch):
|
||||
Base.metadata.create_all(bind=engine)
|
||||
memorybank._vector_cache.clear()
|
||||
embeddings._cache.clear()
|
||||
setup = SessionLocal()
|
||||
user = models.User(is_guest=False, email="m7perf@example.com")
|
||||
setup.add(user)
|
||||
setup.flush()
|
||||
setup.add(models.Settings(
|
||||
user_id=user.id, model="test-model", embedding_model="nomic-embed-text",
|
||||
context_token_budget=8000, max_output_tokens=400,
|
||||
))
|
||||
adventure = models.Adventure(user_id=user.id, title="Performance")
|
||||
setup.add(adventure)
|
||||
setup.flush()
|
||||
setup.add(models.Action(adventure_id=adventure.id, type="start",
|
||||
text="Aldric stands in the crypt beneath the Old Abbey."))
|
||||
setup.commit()
|
||||
adv_id, user_id = adventure.id, user.id
|
||||
setup.close()
|
||||
|
||||
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
|
||||
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
|
||||
monkeypatch.setattr(memorybank, "embedding_provider", lambda s: StubEmbedder())
|
||||
monkeypatch.setattr(memorybank, "summary_provider", lambda s: StubEmbedder())
|
||||
app.dependency_overrides[auth.get_current_user] = (
|
||||
lambda db=Depends(get_db): db.get(models.User, user_id)
|
||||
)
|
||||
test_client = TestClient(app)
|
||||
test_client.adv_id = adv_id
|
||||
test_client.user_id = user_id
|
||||
try:
|
||||
yield test_client
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
memorybank._vector_cache.clear()
|
||||
embeddings._cache.clear()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
|
||||
|
||||
def add_sources(client, count, paragraphs=6, prefix="lore"):
|
||||
"""Imports `count` sources, each with several passages of crypt-ish prose."""
|
||||
for n in range(count):
|
||||
body = "\n\n".join(
|
||||
f"## {prefix} {n} section {p}\n\n"
|
||||
+ ("The crypt beneath the Old Abbey at Westhaven is vaulted in "
|
||||
"stone, and the stair descends past niches cut for the dead. ") * 8
|
||||
for p in range(paragraphs)
|
||||
)
|
||||
response = client.post(
|
||||
f"/api/adventures/{client.adv_id}/knowledge",
|
||||
files={"file": (f"{prefix}-{n}.md", body.encode(), "text/markdown")},
|
||||
data={"classification": ["canon", "reference", "inspiration"][n % 3],
|
||||
"allow_duplicate": "true"},
|
||||
)
|
||||
assert response.status_code == 201, response.text[:200]
|
||||
|
||||
|
||||
def counts(client):
|
||||
with SessionLocal() as db:
|
||||
sources = len(db.execute(select(models.KnowledgeSource)).scalars().all())
|
||||
chunks = len(db.execute(select(models.KnowledgeChunk)).scalars().all())
|
||||
return sources, chunks
|
||||
|
||||
|
||||
def measure(sql_log, call):
|
||||
sql_log.clear()
|
||||
result = call()
|
||||
return len(sql_log), result
|
||||
|
||||
|
||||
def retrieve(client):
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
settings = db.execute(select(models.Settings).where(
|
||||
models.Settings.user_id == client.user_id)).scalars().first()
|
||||
return asyncio.run(retrieval.retrieve(adventure, settings))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- tests
|
||||
|
||||
def test_the_source_list_does_not_cost_a_query_per_source(client, sql_log):
|
||||
add_sources(client, 4)
|
||||
small, _ = measure(sql_log, lambda: client.get(
|
||||
f"/api/adventures/{client.adv_id}/knowledge").json())
|
||||
|
||||
add_sources(client, 12, prefix="more")
|
||||
large, rows = measure(sql_log, lambda: client.get(
|
||||
f"/api/adventures/{client.adv_id}/knowledge").json())
|
||||
|
||||
assert len(rows) == 16
|
||||
assert large == small, f"{small} queries for 4 sources, {large} for 16"
|
||||
# ...and the counts it shows are real, so the fixed query count is not
|
||||
# because the counts were dropped.
|
||||
assert all(row["chunk_count"] > 0 for row in rows)
|
||||
|
||||
|
||||
def test_source_detail_does_not_cost_a_query_per_passage(client, sql_log):
|
||||
add_sources(client, 1, paragraphs=3)
|
||||
small_id = client.get(f"/api/adventures/{client.adv_id}/knowledge").json()[0]["id"]
|
||||
small, _ = measure(sql_log, lambda: client.get(
|
||||
f"/api/adventures/{client.adv_id}/knowledge/{small_id}/chunks").json())
|
||||
|
||||
add_sources(client, 1, paragraphs=24, prefix="big")
|
||||
big_id = client.get(f"/api/adventures/{client.adv_id}/knowledge").json()[-1]["id"]
|
||||
large, chunks = measure(sql_log, lambda: client.get(
|
||||
f"/api/adventures/{client.adv_id}/knowledge/{big_id}/chunks").json())
|
||||
|
||||
assert len(chunks) > 3
|
||||
assert large == small, f"{small} queries for a small source, {large} for a big one"
|
||||
|
||||
|
||||
def test_retrieval_does_not_grow_with_the_library(client, sql_log):
|
||||
add_sources(client, 4)
|
||||
embed_pending(client)
|
||||
small, small_result = measure(sql_log, lambda: retrieve(client))
|
||||
|
||||
add_sources(client, 16, prefix="more")
|
||||
embed_pending(client)
|
||||
embeddings.forget_cached(client.adv_id)
|
||||
large, large_result = measure(sql_log, lambda: retrieve(client))
|
||||
|
||||
sources, chunks = counts(client)
|
||||
assert sources == 20 and chunks > 40
|
||||
assert small_result.candidates and large_result.candidates
|
||||
assert large <= small + 1, f"{small} queries at 4 sources, {large} at 20"
|
||||
|
||||
|
||||
def test_the_context_build_does_not_grow_with_the_library(client, sql_log):
|
||||
add_sources(client, 4)
|
||||
embed_pending(client)
|
||||
ScriptedProvider.replies = [f"The crypt is cold.\n{state_block([])}"]
|
||||
client.post(f"/api/adventures/{client.adv_id}/actions",
|
||||
json={"type": "do", "text": "Aldric descends into the crypt."})
|
||||
small, _ = measure(sql_log, lambda: client.get(
|
||||
f"/api/adventures/{client.adv_id}/context").json())
|
||||
|
||||
add_sources(client, 16, prefix="more")
|
||||
embed_pending(client)
|
||||
embeddings.forget_cached(client.adv_id)
|
||||
large, report = measure(sql_log, lambda: client.get(
|
||||
f"/api/adventures/{client.adv_id}/context").json())
|
||||
|
||||
assert report["knowledge"]["used"]
|
||||
assert large <= small + 1, f"{small} queries at 4 sources, {large} at 20"
|
||||
|
||||
|
||||
def test_candidates_are_bounded_in_sql_before_the_python_ranking(client, sql_log):
|
||||
""""Do not load every chunk merely to find the top few", asserted on the SQL."""
|
||||
add_sources(client, 20, paragraphs=8)
|
||||
embed_pending(client)
|
||||
embeddings.forget_cached(client.adv_id)
|
||||
_sources, chunks = counts(client)
|
||||
assert chunks > retrieval.LEXICAL_CANDIDATES * 2, chunks
|
||||
|
||||
sql_log.clear()
|
||||
result = retrieve(client)
|
||||
|
||||
# The lexical query names a LIMIT, and the merged candidate set is bounded
|
||||
# by the two per-path caps rather than by the size of the library.
|
||||
lexical = [s for s in sql_log if "knowledge_fts" in s and "MATCH" in s]
|
||||
assert lexical, sql_log
|
||||
assert all("LIMIT" in s for s in lexical)
|
||||
assert result.considered <= (
|
||||
retrieval.LEXICAL_CANDIDATES + retrieval.SEMANTIC_CANDIDATES
|
||||
)
|
||||
assert result.considered < chunks, (result.considered, chunks)
|
||||
|
||||
# The row fetch for those candidates is one query, not one per candidate.
|
||||
loads = [s for s in sql_log
|
||||
if "knowledge_chunks" in s and "knowledge_sources" in s
|
||||
and " IN " in s.upper()]
|
||||
assert len(loads) <= 2, loads
|
||||
|
||||
|
||||
def test_the_semantic_scan_reads_only_narrow_columns(client, sql_log):
|
||||
"""A vector is 6 kB; the catalogue read must not fetch passage text."""
|
||||
add_sources(client, 6)
|
||||
embed_pending(client)
|
||||
embeddings.forget_cached(client.adv_id)
|
||||
|
||||
sql_log.clear()
|
||||
retrieve(client)
|
||||
catalogue = [s for s in sql_log
|
||||
if "knowledge_embeddings.chunk_id" in s
|
||||
and "knowledge_embeddings.vector" not in s]
|
||||
assert catalogue, "the semantic catalogue read was not found"
|
||||
assert all("knowledge_chunks.text" not in s for s in catalogue)
|
||||
|
||||
|
||||
def embed_pending(client):
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
settings = db.execute(select(models.Settings).where(
|
||||
models.Settings.user_id == client.user_id)).scalars().first()
|
||||
asyncio.run(embeddings.embed_pending(db, adventure, settings))
|
||||
db.commit()
|
||||
@@ -0,0 +1,403 @@
|
||||
"""M7: the semantic path, end to end, against a real local embedding model.
|
||||
|
||||
M2 shipped with the memory bank dead and the suite green, because every test
|
||||
stubbed the provider factories out. M6 answered that with
|
||||
`test_provider_wiring.py` and the rule that at least one real
|
||||
provider-construction path must be exercised per milestone. This is M7's.
|
||||
|
||||
**Nothing here is mocked.** A real `Settings` row is read back out of the
|
||||
database, the real factory builds the provider from it, a real request reaches
|
||||
the configured local Ollama, the vectors it returns are stored in
|
||||
`knowledge_embeddings`, and the real hybrid retrieval ranks against them and
|
||||
inserts the winner into a prompt built by the real context builder.
|
||||
|
||||
It is skipped without an endpoint, and it is reported separately from the
|
||||
deterministic suite, because it needs a machine with a model on it:
|
||||
|
||||
AIDND_TEST_ENDPOINT=https://inference.lan:8443/v1 \\
|
||||
AIDND_TEST_EMBED_MODEL=nomic-embed-text \\
|
||||
backend/.venv/bin/python -m pytest backend/tests/test_knowledge_real_model.py -v -s
|
||||
|
||||
The endpoint goes through the ordinary policy: no allowlist bypass, no TLS
|
||||
weakening. A public endpoint is refused here exactly as it is in production, and
|
||||
the test asserts that rather than assuming it.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
import pytest
|
||||
from fastapi import Depends
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import auth, endpoints, limits, memorybank, models
|
||||
from app.database import Base, SessionLocal, engine, get_db
|
||||
from app.knowledge import classes, embeddings, retrieval
|
||||
from app.main import app
|
||||
from app.routers import adventures
|
||||
|
||||
from fakes import ScriptedProvider, state_block
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not os.environ.get("AIDND_TEST_ENDPOINT"),
|
||||
reason="set AIDND_TEST_ENDPOINT (and AIDND_TEST_EMBED_MODEL) to run this",
|
||||
)
|
||||
|
||||
ENDPOINT = os.environ.get("AIDND_TEST_ENDPOINT", "")
|
||||
EMBED_MODEL = os.environ.get("AIDND_TEST_EMBED_MODEL", "nomic-embed-text")
|
||||
|
||||
CANON_MD = """# The Old Abbey
|
||||
|
||||
The Old Abbey lies five miles north of Westhaven.
|
||||
The abbey crypt bears a symbol shaped like a broken circle.
|
||||
"""
|
||||
|
||||
REFERENCE_MD = """# Medieval Taverns
|
||||
|
||||
Medieval taverns commonly used timber framing, stone hearths, benches,
|
||||
shared tables, candles, and oil lamps.
|
||||
"""
|
||||
|
||||
# The conceptual case: about the crypt, sharing almost none of its words. If the
|
||||
# stored vectors were nonsense, this is the source that would not be found.
|
||||
OSSUARY_MD = """# The Ossuary
|
||||
|
||||
Bones were stacked in the undercroft below the chancel, sorted and shelved
|
||||
by the brothers who kept the sanctuary.
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(monkeypatch):
|
||||
"""A campaign wired to the real endpoint. Only the *narrator* is scripted.
|
||||
|
||||
The narrator is scripted because this file is about embeddings and a real
|
||||
narration would make it slow and non-deterministic for no gain. The
|
||||
embedding path — factory, request, storage, retrieval — is entirely real.
|
||||
"""
|
||||
assert endpoints.rejection_reason(ENDPOINT) is None, (
|
||||
f"the configured test endpoint {ENDPOINT} is refused by the policy"
|
||||
)
|
||||
Base.metadata.create_all(bind=engine)
|
||||
memorybank._vector_cache.clear()
|
||||
embeddings._cache.clear()
|
||||
setup = SessionLocal()
|
||||
user = models.User(is_guest=False, email="m7real@example.com")
|
||||
setup.add(user)
|
||||
setup.flush()
|
||||
setup.add(models.Settings(
|
||||
user_id=user.id, endpoint_url=ENDPOINT,
|
||||
model=os.environ.get("AIDND_TEST_MODEL", "test-model"),
|
||||
embedding_model=EMBED_MODEL,
|
||||
context_token_budget=6000, max_output_tokens=300,
|
||||
))
|
||||
adventure = models.Adventure(user_id=user.id, title="Real Model")
|
||||
setup.add(adventure)
|
||||
setup.flush()
|
||||
setup.add(models.Action(
|
||||
adventure_id=adventure.id, type="start",
|
||||
text="Aldric stands in the crypt beneath the Old Abbey, north of Westhaven.",
|
||||
))
|
||||
setup.commit()
|
||||
adv_id, user_id = adventure.id, user.id
|
||||
setup.close()
|
||||
|
||||
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
|
||||
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
|
||||
app.dependency_overrides[auth.get_current_user] = (
|
||||
lambda db=Depends(get_db): db.get(models.User, user_id)
|
||||
)
|
||||
test_client = TestClient(app)
|
||||
test_client.adv_id = adv_id
|
||||
test_client.user_id = user_id
|
||||
try:
|
||||
yield test_client
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
memorybank._vector_cache.clear()
|
||||
embeddings._cache.clear()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
|
||||
|
||||
def upload(client, name, body, classification):
|
||||
response = client.post(
|
||||
f"/api/adventures/{client.adv_id}/knowledge",
|
||||
files={"file": (name, body.encode(), "text/markdown")},
|
||||
data={"classification": classification},
|
||||
)
|
||||
assert response.status_code == 201, response.text[:400]
|
||||
return response.json()
|
||||
|
||||
|
||||
def settings_row(client, db):
|
||||
return db.execute(select(models.Settings).where(
|
||||
models.Settings.user_id == client.user_id)).scalars().first()
|
||||
|
||||
|
||||
def test_a_real_local_model_embeds_stores_retrieves_and_reaches_the_prompt(client):
|
||||
"""The whole semantic path, with nothing stubbed between here and Ollama."""
|
||||
canon = upload(client, "canon.md", CANON_MD, "canon")
|
||||
upload(client, "reference.md", REFERENCE_MD, "reference")
|
||||
ossuary = upload(client, "ossuary.md", OSSUARY_MD, "reference")
|
||||
|
||||
# 1. Real vectors were stored, by the import path, through the real factory.
|
||||
# Import embeds inline, so this is already true before anything else runs.
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(select(models.KnowledgeEmbedding).where(
|
||||
models.KnowledgeEmbedding.adventure_id == client.adv_id
|
||||
)).scalars().all()
|
||||
assert rows, "no vectors were stored"
|
||||
for row in rows:
|
||||
assert row.model == EMBED_MODEL
|
||||
assert row.dimensions > 64, row.dimensions
|
||||
assert len(row.vector) == row.dimensions * 4 # packed float32
|
||||
dimensions = rows[0].dimensions
|
||||
assert all(row.dimensions == dimensions for row in rows)
|
||||
|
||||
listing = {row["original_filename"]: row for row in
|
||||
client.get(f"/api/adventures/{client.adv_id}/knowledge").json()}
|
||||
for name, row in listing.items():
|
||||
assert row["embed_state"] == "ok", (name, row["embed_detail"])
|
||||
assert row["embedded_count"] == row["chunk_count"]
|
||||
|
||||
status = client.get(f"/api/adventures/{client.adv_id}/knowledge-status").json()
|
||||
assert status["semantic_enabled"] is True
|
||||
assert status["embedding_model"] == EMBED_MODEL
|
||||
assert status["pending_embeddings"] == 0
|
||||
assert status["failed_embedding"] == []
|
||||
|
||||
# 2. Real semantic retrieval, against those stored vectors.
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
result = asyncio.run(retrieval.retrieve(adventure, settings_row(client, db)))
|
||||
assert result.semantic_used, result.semantic_note
|
||||
scored = {c.filename: c for c in result.candidates}
|
||||
print("\n real-model ranking:")
|
||||
for candidate in result.candidates:
|
||||
print(f" {candidate.filename:16} {candidate.classification:12} "
|
||||
f"lex={candidate.lexical:.3f} sem={candidate.semantic:.3f} "
|
||||
f"cos={candidate.cosine:.3f} score={candidate.score:.3f}")
|
||||
for candidate in result.suppressed:
|
||||
print(f" {candidate.filename:16} SUPPRESSED")
|
||||
assert scored, "the real model retrieved nothing"
|
||||
assert any(c.cosine > 0 for c in result.candidates)
|
||||
|
||||
# The conceptual match is the thing only a real embedding can do here:
|
||||
# `ossuary.md` shares almost no words with the scene and is about it.
|
||||
if "ossuary.md" in scored:
|
||||
assert scored["ossuary.md"].semantic > 0
|
||||
print(f" conceptual match found: ossuary.md at cosine "
|
||||
f"{scored['ossuary.md'].cosine:.3f}")
|
||||
|
||||
# 3. It reaches a prompt built by the real context builder.
|
||||
ScriptedProvider.replies = [f"The crypt is cold and still.\n{state_block([])}"]
|
||||
turn = client.post(f"/api/adventures/{client.adv_id}/actions",
|
||||
json={"type": "do", "text": "Aldric studies the crypt walls."})
|
||||
assert turn.status_code == 200, turn.text[:300]
|
||||
report = client.get(f"/api/adventures/{client.adv_id}/context").json()
|
||||
assert report["knowledge"]["semantic_used"] is True
|
||||
assert report["knowledge"]["used"], report["knowledge"]["semantic_note"]
|
||||
used = {u["filename"]: u for u in report["knowledge"]["used"]}
|
||||
assert any(u["mode"] in ("semantic", "hybrid") for u in used.values()), used
|
||||
assert any(s["label"].startswith("imported_") for s in report["sections"])
|
||||
print(f" prompt sections: "
|
||||
f"{[s['label'] for s in report['sections'] if s['label'].startswith('imported_')]}")
|
||||
assert canon and ossuary
|
||||
|
||||
|
||||
# ============================ the M7 corrective regression: admission ========
|
||||
#
|
||||
# The failure class this exists to prevent: a deterministic stub that is more
|
||||
# discriminative than the real model, hiding an admission gate that cannot say
|
||||
# "no match" (review findings M7-F1 and M7-F2). The deterministic suite is the
|
||||
# normal required path; this is the reality check, and it prints the measured
|
||||
# separation so a model change surfaces as data rather than as a mystery.
|
||||
|
||||
#: Passages that share almost no vocabulary with their query but are about the
|
||||
#: same thing — the case the semantic half of the hybrid exists to serve.
|
||||
PARAPHRASE_QUERY = ("What emblem is carved in the burial vault beneath the "
|
||||
"ruined monastery up the road from town?")
|
||||
#: Scenes with no connection to a fantasy campaign at all.
|
||||
OFF_TOPIC = [
|
||||
"The kiln was held at cone six for a two-hour soak while the glaze matured.",
|
||||
"The compiler emits a diagnostic when the lifetime of the borrow outlives "
|
||||
"the referent.",
|
||||
"The surgeon sterilised the cannula and checked the infusion pump pressure.",
|
||||
"He reconciled the ledger against the quarterly depreciation schedule.",
|
||||
"She practised the fugue slowly, counting the subject's entries.",
|
||||
]
|
||||
|
||||
|
||||
def _cosines(client, adv, texts):
|
||||
"""Raw cosine of each text against every stored vector, as retrieval sees it."""
|
||||
from app.vectors import cosine, unpack
|
||||
|
||||
with SessionLocal() as db:
|
||||
settings = settings_row(client, db)
|
||||
rows = db.execute(
|
||||
select(models.KnowledgeEmbedding.vector,
|
||||
models.KnowledgeSource.original_filename)
|
||||
.join(models.KnowledgeChunk,
|
||||
models.KnowledgeChunk.id == models.KnowledgeEmbedding.chunk_id)
|
||||
.join(models.KnowledgeSource,
|
||||
models.KnowledgeSource.id == models.KnowledgeChunk.source_id)
|
||||
.where(models.KnowledgeSource.adventure_id == adv)).all()
|
||||
vectors = [(name, unpack(blob)) for blob, name in rows]
|
||||
embedded = asyncio.run(
|
||||
memorybank.embedding_provider(settings).embed(list(texts)))
|
||||
return {text: {name: cosine(vector, stored) for name, stored in vectors}
|
||||
for text, vector in zip(texts, embedded)}
|
||||
|
||||
|
||||
def test_the_real_model_separates_relevant_from_unrelated(client):
|
||||
"""The measurement the admission floor rests on, re-taken every run.
|
||||
|
||||
Fails if the configured model's scale moves far enough that
|
||||
`classes.SEMANTIC_FLOOR` stops sitting between the two populations — which
|
||||
is the one way this build could silently go back to admitting everything or
|
||||
start admitting nothing.
|
||||
"""
|
||||
upload(client, "canon.md", CANON_MD, "canon")
|
||||
upload(client, "reference.md", REFERENCE_MD, "reference")
|
||||
|
||||
targeted = {
|
||||
"Aldric asks about the Old Abbey north of Westhaven and its "
|
||||
"broken-circle symbol.": "canon.md",
|
||||
PARAPHRASE_QUERY: "canon.md",
|
||||
"Aldric looks around the tavern at the stone hearth and the timber "
|
||||
"beams.": "reference.md",
|
||||
}
|
||||
scores = _cosines(client, client.adv_id, list(targeted) + OFF_TOPIC)
|
||||
|
||||
hits = [scores[q][want] for q, want in targeted.items()]
|
||||
misses = [c for q in OFF_TOPIC for c in scores[q].values()]
|
||||
print(f"\n real-model separation ({EMBED_MODEL}):")
|
||||
for q, want in targeted.items():
|
||||
print(f" targeted {scores[q][want]:.4f} {q[:52]}")
|
||||
for q in OFF_TOPIC:
|
||||
for name, c in scores[q].items():
|
||||
print(f" off-topic {c:.4f} {q[:40]:40} -> {name}")
|
||||
print(f" floor = {classes.SEMANTIC_FLOOR}")
|
||||
|
||||
assert min(hits) > classes.SEMANTIC_FLOOR, (
|
||||
f"targeted matches {sorted(hits)} fall below the floor "
|
||||
f"{classes.SEMANTIC_FLOOR}; relevant material would be dropped")
|
||||
assert max(misses) < classes.SEMANTIC_FLOOR, (
|
||||
f"off-topic pairs reach {max(misses):.4f}, at or above the floor "
|
||||
f"{classes.SEMANTIC_FLOOR}; irrelevant material would be admitted")
|
||||
|
||||
|
||||
def test_a_completely_unrelated_query_retrieves_nothing_from_a_real_model(client):
|
||||
"""**The no-match case, end to end, with nothing mocked.**
|
||||
|
||||
A mixed library of Canon, Reference and Inspiration, all embedded by the
|
||||
real model, and a scene about none of them. The prompt must carry no
|
||||
imported section at all.
|
||||
"""
|
||||
upload(client, "canon.md", CANON_MD, "canon")
|
||||
upload(client, "reference.md", REFERENCE_MD, "reference")
|
||||
upload(client, "ossuary.md", OSSUARY_MD, "inspiration")
|
||||
|
||||
# The retrieval query is built from the recent story window, so the whole
|
||||
# window has to move off-topic — one off-topic line after a crypt opening
|
||||
# still leaves the crypt in the query, which is correct behaviour and would
|
||||
# make this test prove nothing.
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
adventure.narrative_state = None
|
||||
for depth, text in enumerate(OFF_TOPIC[:4], start=1):
|
||||
db.add(models.Action(
|
||||
adventure_id=client.adv_id, type="do", text=text,
|
||||
branch_id=adventure.head_branch_id, depth=depth, live=True))
|
||||
adventure.head_depth = 4
|
||||
db.commit()
|
||||
|
||||
report = client.get(f"/api/adventures/{client.adv_id}/context").json()
|
||||
knowledge = report["knowledge"]
|
||||
print(f"\n generated={knowledge['generated']} "
|
||||
f"rejected={knowledge['rejected']} used={len(knowledge['used'])}")
|
||||
assert knowledge["generated"] > 0, "nothing was generated; this proves nothing"
|
||||
assert knowledge["used"] == [], [u["filename"] for u in knowledge["used"]]
|
||||
assert not [s for s in report["sections"] if s["label"].startswith("imported_")]
|
||||
|
||||
|
||||
def test_a_relevant_query_still_retrieves_from_a_real_model(client):
|
||||
"""The positive control for the test above, on the same library."""
|
||||
upload(client, "canon.md", CANON_MD, "canon")
|
||||
upload(client, "reference.md", REFERENCE_MD, "reference")
|
||||
upload(client, "ossuary.md", OSSUARY_MD, "inspiration")
|
||||
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
db.add(models.Action(
|
||||
adventure_id=client.adv_id, type="do",
|
||||
text="Aldric asks Mara about the Old Abbey north of Westhaven and "
|
||||
"the broken-circle symbol in its crypt.",
|
||||
branch_id=adventure.head_branch_id, depth=1, live=True))
|
||||
adventure.head_depth = 1
|
||||
db.commit()
|
||||
|
||||
|
||||
knowledge = client.get(
|
||||
f"/api/adventures/{client.adv_id}/context").json()["knowledge"]
|
||||
used = [u["filename"] for u in knowledge["used"]]
|
||||
print(f"\n retrieved: {used}")
|
||||
assert "canon.md" in used, used
|
||||
for record in knowledge["used"]:
|
||||
assert record["admitted_by"] in ("lexical", "semantic", "both")
|
||||
|
||||
|
||||
def test_a_paraphrase_still_retrieves_from_a_real_model(client):
|
||||
"""Strong semantic, weak lexical, against the real model."""
|
||||
upload(client, "canon.md", CANON_MD, "canon")
|
||||
scores = _cosines(client, client.adv_id, [PARAPHRASE_QUERY])
|
||||
cosine_value = scores[PARAPHRASE_QUERY]["canon.md"]
|
||||
print(f"\n paraphrase cosine: {cosine_value:.4f} "
|
||||
f"(floor {classes.SEMANTIC_FLOOR})")
|
||||
assert cosine_value >= classes.SEMANTIC_FLOOR, (
|
||||
"a genuine paraphrase falls below the admission floor")
|
||||
|
||||
|
||||
def test_a_reindex_rebuilds_real_vectors(client):
|
||||
"""Reindex against the real endpoint: vectors go and come back."""
|
||||
upload(client, "canon.md", CANON_MD, "canon")
|
||||
with SessionLocal() as db:
|
||||
before = len(db.execute(select(models.KnowledgeEmbedding)).scalars().all())
|
||||
assert before > 0
|
||||
|
||||
out = client.post(f"/api/adventures/{client.adv_id}/knowledge/reindex").json()
|
||||
assert out["semantic"] is True
|
||||
assert out["embedded"] == before
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(select(models.KnowledgeEmbedding)).scalars().all()
|
||||
assert len(rows) == before
|
||||
assert all(row.model == EMBED_MODEL for row in rows)
|
||||
|
||||
|
||||
def test_the_real_embedding_path_still_obeys_the_endpoint_policy(client):
|
||||
"""The policy is checked before every request, on this path too."""
|
||||
from app.providers import ProviderError
|
||||
|
||||
with SessionLocal() as db:
|
||||
row = settings_row(client, db)
|
||||
row.endpoint_url = "https://api.openai.com/v1"
|
||||
db.commit()
|
||||
upload_body = {"classification": "canon"}
|
||||
# The import itself succeeds — lexical indexing needs no network — and the
|
||||
# embedding attempt behind it is refused by the policy rather than sent.
|
||||
response = client.post(
|
||||
f"/api/adventures/{client.adv_id}/knowledge",
|
||||
files={"file": ("blocked.md", CANON_MD.encode(), "text/markdown")},
|
||||
data=upload_body,
|
||||
)
|
||||
assert response.status_code == 201
|
||||
assert response.json()["index_state"] == "ready"
|
||||
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, client.adv_id)
|
||||
provider = memorybank.embedding_provider(settings_row(client, db))
|
||||
with pytest.raises(ProviderError) as exc:
|
||||
asyncio.run(provider.embed(["a line of someone's story"]))
|
||||
assert "can't be used" in str(exc.value)
|
||||
assert adventure is not None
|
||||
@@ -0,0 +1,494 @@
|
||||
"""M7: what retrieval admits and how it ranks — the mechanism, not the fixture.
|
||||
|
||||
This is a **purpose-built retrieval-mechanism** suite. It uses invented sources
|
||||
chosen to isolate one behaviour each, not the standard campaign fixture; the
|
||||
acceptance-fixture tests live in `test_imported_knowledge.py`. The two are kept
|
||||
apart deliberately: an acceptance test says the product meets its contract, and
|
||||
this says the machinery underneath behaves the way the contract needs it to.
|
||||
|
||||
## The two stages, and why they are tested separately
|
||||
|
||||
candidate generation -> ADMISSION -> ranking -> class weighting -> budget
|
||||
|
||||
**Admission** decides whether a passage matched at all, from signals that mean
|
||||
something on their own. **Ranking** orders what survived. M7's first
|
||||
implementation had only the second: it normalized every score against the best
|
||||
of its own path and cut at a share of that best, which the best clears by
|
||||
construction. Something was therefore admitted on every turn, whatever the
|
||||
reader was doing (review finding M7-F1).
|
||||
|
||||
## Why the stub embedder looks the way it does
|
||||
|
||||
The suite that shipped with M7 asserted "irrelevant Canon does not win" and
|
||||
passed, while the product injected five irrelevant sources into every prompt.
|
||||
Its stub gave unrelated text a cosine of 0.06-0.20 and its own docstring said it
|
||||
had *deliberately* removed the constant component that "would put a similarity
|
||||
floor under every pair" — which is exactly the property real embedding models
|
||||
have. Measured on identical texts, `nomic-embed-text` scored those same
|
||||
unrelated pairs 0.435-0.437. The stub was an order of magnitude more
|
||||
discriminative than reality, so the broken gate sailed through (finding M7-F2).
|
||||
|
||||
`RealisticEmbedder` below therefore has a deliberate similarity floor. Unrelated
|
||||
passages score a substantial, nontrivial similarity, as they do in life. That is
|
||||
not decoration: `test_the_stub_models_the_real_problem` fails if the floor ever
|
||||
goes away, and `test_a_relative_only_floor_would_admit_the_irrelevant_set`
|
||||
demonstrates on this very fixture that the *old* rule would still be fooled by
|
||||
it. The stub models the shape of the problem; it does not encode the answer.
|
||||
|
||||
python -m pytest tests/test_knowledge_retrieval_quality.py -v
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import math
|
||||
|
||||
import pytest
|
||||
from fastapi import Depends
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import auth, limits, memorybank, models
|
||||
from app.database import Base, SessionLocal, engine, get_db
|
||||
from app.knowledge import classes, embeddings, retrieval
|
||||
from app.main import app
|
||||
from app.routers import adventures
|
||||
|
||||
from fakes import ScriptedProvider
|
||||
|
||||
|
||||
# --------------------------------------------------------------- the library
|
||||
|
||||
ABBEY_CANON = (b"# The Old Abbey\n\nThe Old Abbey lies five miles north of "
|
||||
b"Westhaven. The abbey crypt bears a symbol shaped like a broken "
|
||||
b"circle, cut into the keystone above the stair.\n")
|
||||
CRYPT_REFERENCE = (b"# Crypt Construction\n\nAn abbey crypt was vaulted in stone, "
|
||||
b"entered by a stair descending from the nave, with burial "
|
||||
b"niches cut into the side walls.\n")
|
||||
CRYPT_MOOD = (b"# Below\n\nThe air in the crypt was older than the abbey above it, "
|
||||
b"and the dark pressed close around the lantern on the stair.\n")
|
||||
OSSUARY = (b"# The Ossuary\n\nBones were stacked in the undercroft below the "
|
||||
b"chancel, sorted and shelved by the brothers of the sanctuary.\n")
|
||||
ABBEY_COPY = (b"# The Abbey\n\nFive miles north of Westhaven stands the Old Abbey. "
|
||||
b"Above the crypt stair a broken circle is cut into the keystone.\n")
|
||||
|
||||
SHIP_CANON = (b"# The Persephone\n\nThe freighter Persephone is docked at Ceres "
|
||||
b"Station with a cracked heat exchanger and no licence to carry "
|
||||
b"passengers.\n")
|
||||
SURGERY_REFERENCE = (b"# Cannulation\n\nThe surgeon sterilised the cannula and "
|
||||
b"checked the infusion pump pressure before the procedure.\n")
|
||||
COMPILER_INSPIRATION = (b"# Diagnostics\n\nThe compiler emits a diagnostic when the "
|
||||
b"lifetime of the borrow outlives the referent.\n")
|
||||
|
||||
CRYPT_SCENE = ("Aldric descends the stair into the crypt beneath the Old Abbey, "
|
||||
"north of Westhaven, lantern raised.")
|
||||
#: A scene with no connection to any source in the library at all.
|
||||
OFF_TOPIC_SCENE = ("The kiln was held at cone six for a two-hour soak while the "
|
||||
"glaze matured.")
|
||||
|
||||
|
||||
class RealisticEmbedder:
|
||||
"""A deterministic embedder with the two properties the real one has.
|
||||
|
||||
* **A similarity floor.** Every pair of texts shares a constant component,
|
||||
so unrelated passages score a substantial similarity rather than nearly
|
||||
zero. This is what a real embedding model does and what the M7 stub left
|
||||
out; without it no fixture can detect an admission gate that cannot say
|
||||
"no match".
|
||||
* **Topical structure above the floor.** Disjoint topic axes, so a passage
|
||||
about the same subject scores clearly higher — including when it shares
|
||||
almost no vocabulary, which is the case the hybrid's semantic half exists
|
||||
to serve.
|
||||
|
||||
A hashed bag of words at low weight sits underneath, so two passages on one
|
||||
topic in different words are close without being identical and the
|
||||
redundancy suppressor is not handed a fixture of clones.
|
||||
"""
|
||||
|
||||
#: Deliberately disjoint: no word appears on two axes, or a query about one
|
||||
#: subject scores as though it were about another and the fixture stops
|
||||
#: meaning what it says.
|
||||
AXES = (
|
||||
("crypt", "abbey", "vault", "undercroft", "ossuary", "chancel", "bones",
|
||||
"stair", "keystone", "niches", "nave", "burial", "monastery", "emblem",
|
||||
"circle", "broken", "symbol", "sanctuary", "brothers", "shelved"),
|
||||
("westhaven", "north", "miles", "road", "town", "stands"),
|
||||
("lantern", "dark", "air", "older", "pressed", "close"),
|
||||
("freighter", "persephone", "ceres", "docked", "exchanger", "licence",
|
||||
"passengers", "station", "cracked"),
|
||||
("surgeon", "cannula", "infusion", "pump", "sterilised", "pressure",
|
||||
"procedure"),
|
||||
("compiler", "diagnostic", "borrow", "lifetime", "referent", "emits"),
|
||||
("kiln", "cone", "soak", "glaze", "matured"),
|
||||
)
|
||||
#: The constant every vector carries. Tuned so unrelated pairs land in a
|
||||
#: realistic band rather than near zero — see the module docstring.
|
||||
BASE = 0.9
|
||||
TOPIC_WEIGHT = 2.0
|
||||
WORD_WEIGHT = 0.25
|
||||
BUCKETS = 64
|
||||
|
||||
@staticmethod
|
||||
def _words(text):
|
||||
return set("".join(c.lower() if c.isalnum() or c == "-" else " "
|
||||
for c in text).split())
|
||||
|
||||
def vector(self, text):
|
||||
unique = self._words(text)
|
||||
topic = [self.TOPIC_WEIGHT * len(unique & set(axis)) / len(axis)
|
||||
for axis in self.AXES]
|
||||
buckets = [0.0] * self.BUCKETS
|
||||
for word in unique:
|
||||
index = sum((i + 1) * ord(c) for i, c in enumerate(word)) % self.BUCKETS
|
||||
buckets[index] += self.WORD_WEIGHT
|
||||
scale = math.sqrt(len(unique)) or 1.0
|
||||
return [self.BASE] + topic + [b / scale for b in buckets]
|
||||
|
||||
async def embed(self, texts):
|
||||
return [self.vector(t) for t in texts]
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(monkeypatch):
|
||||
Base.metadata.create_all(bind=engine)
|
||||
memorybank._vector_cache.clear()
|
||||
embeddings._cache.clear()
|
||||
setup = SessionLocal()
|
||||
user = models.User(is_guest=False, email="quality@example.com")
|
||||
setup.add(user)
|
||||
setup.flush()
|
||||
# A *calibrated* model name, deliberately. Semantic admission is
|
||||
# per-model (`classes.SEMANTIC_CALIBRATION`), and the stub below
|
||||
# is built to model this model's similarity distribution, so the
|
||||
# fixture must name it or the suite would silently exercise the
|
||||
# uncalibrated lexical-only path instead.
|
||||
setup.add(models.Settings(
|
||||
user_id=user.id, model="test-model", embedding_model="nomic-embed-text",
|
||||
context_token_budget=6000, max_output_tokens=400,
|
||||
))
|
||||
setup.commit()
|
||||
user_id = user.id
|
||||
setup.close()
|
||||
|
||||
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
|
||||
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
|
||||
monkeypatch.setattr(memorybank, "embedding_provider", lambda s: RealisticEmbedder())
|
||||
monkeypatch.setattr(memorybank, "summary_provider", lambda s: RealisticEmbedder())
|
||||
app.dependency_overrides[auth.get_current_user] = (
|
||||
lambda db=Depends(get_db): db.get(models.User, user_id)
|
||||
)
|
||||
test_client = TestClient(app)
|
||||
test_client.user_id = user_id
|
||||
try:
|
||||
yield test_client
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
memorybank._vector_cache.clear()
|
||||
embeddings._cache.clear()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- helpers
|
||||
|
||||
def campaign(client, opening, sources):
|
||||
"""A campaign with `opening` as its only turn and `sources` imported."""
|
||||
adventure = client.post("/api/adventures", json={"title": "Q"}).json()
|
||||
adv = adventure["id"]
|
||||
with SessionLocal() as db:
|
||||
row = db.get(models.Adventure, adv)
|
||||
db.add(models.Action(adventure_id=adv, type="start", text=opening,
|
||||
branch_id=row.head_branch_id, depth=0, live=True))
|
||||
row.head_depth = 0
|
||||
db.commit()
|
||||
ids = {}
|
||||
for name, body, kind in sources:
|
||||
response = client.post(
|
||||
f"/api/adventures/{adv}/knowledge",
|
||||
files={"file": (name, body, "text/markdown")},
|
||||
data={"classification": kind, "allow_duplicate": "true"})
|
||||
assert response.status_code == 201, response.text[:200]
|
||||
ids[name] = response.json()["id"]
|
||||
embeddings.forget_cached(adv)
|
||||
return adv, ids
|
||||
|
||||
|
||||
def rank(client, adv):
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, adv)
|
||||
settings = db.execute(select(models.Settings).where(
|
||||
models.Settings.user_id == client.user_id)).scalars().first()
|
||||
return asyncio.run(retrieval.retrieve(adventure, settings))
|
||||
|
||||
|
||||
def table(result):
|
||||
rows = [f" {c.filename:22} {c.classification:12} by={c.admitted_by or 'always':9} "
|
||||
f"lex={c.lexical:.3f} sem={c.semantic:.3f} cos={c.cosine:.3f} "
|
||||
f"score={c.score:.3f} terms={c.matched_terms}"
|
||||
for c in result.candidates]
|
||||
rows += [f" {c.filename:22} SUPPRESSED (duplicate of {c.duplicate_of})"
|
||||
for c in result.suppressed]
|
||||
return (f"generated={result.generated} rejected={result.rejected} "
|
||||
f"floor={result.semantic_floor}\n" + "\n".join(rows) or " (nothing)")
|
||||
|
||||
|
||||
def names(result):
|
||||
return [c.filename for c in result.candidates]
|
||||
|
||||
|
||||
# =================================================== the stub is realistic
|
||||
|
||||
def test_the_stub_models_the_real_problem(client):
|
||||
"""M7-F2's guard: the stub must not be more discriminative than reality.
|
||||
|
||||
If this ever fails because unrelated pairs score near zero, the fixture has
|
||||
drifted back to the one that hid the defect, and every no-match test in this
|
||||
file has quietly stopped proving anything.
|
||||
"""
|
||||
embedder = RealisticEmbedder()
|
||||
query = embedder.vector(CRYPT_SCENE)
|
||||
unrelated = [embedder.vector(t.decode()) for t in
|
||||
(SURGERY_REFERENCE, COMPILER_INSPIRATION, SHIP_CANON)]
|
||||
targeted = embedder.vector(ABBEY_CANON.decode())
|
||||
|
||||
from app.vectors import cosine
|
||||
floor = [cosine(query, v) for v in unrelated]
|
||||
hit = cosine(query, targeted)
|
||||
|
||||
assert min(floor) > 0.10, (
|
||||
f"unrelated pairs score {floor} — the stub has no similarity floor and "
|
||||
"cannot model the real model's behaviour")
|
||||
assert hit > max(floor), f"targeted {hit} vs unrelated {floor}"
|
||||
# Real `nomic-embed-text` puts unrelated pairs around 0.36-0.56 and targeted
|
||||
# matches around 0.55-0.85. The stub need not match those numbers, but it
|
||||
# must have the same shape: a floor well clear of zero, under a clear hit.
|
||||
assert hit - max(floor) < 0.9, "the stub separates far more cleanly than reality"
|
||||
|
||||
|
||||
def test_a_relative_only_floor_would_admit_the_irrelevant_set(client):
|
||||
"""The old rule, run against this fixture, still fails — as it must.
|
||||
|
||||
This is what makes the suite able to detect M7-F1. It reproduces the
|
||||
superseded admission rule (a share of the best candidate) on the same
|
||||
vectors the corrected code sees, and shows it admitting the whole
|
||||
irrelevant library.
|
||||
"""
|
||||
embedder = RealisticEmbedder()
|
||||
from app.vectors import cosine
|
||||
query = embedder.vector(OFF_TOPIC_SCENE)
|
||||
raw = {name: cosine(query, embedder.vector(body.decode())) for name, body in (
|
||||
("abbey", ABBEY_CANON), ("crypt-ref", CRYPT_REFERENCE),
|
||||
("mood", CRYPT_MOOD), ("ship", SHIP_CANON))}
|
||||
best = max(raw.values())
|
||||
old_floor = max(0.02, best * 0.25) # the superseded rule
|
||||
admitted_by_old_rule = [n for n, c in raw.items() if c / best >= old_floor / best]
|
||||
assert len(admitted_by_old_rule) == len(raw), (
|
||||
f"the old relative-only rule admitted {admitted_by_old_rule} of {raw} — "
|
||||
"this fixture must be able to fool it, or it cannot prove the fix")
|
||||
# ...and every one of them is below the absolute floor the fix uses.
|
||||
assert all(c < classes.SEMANTIC_FLOOR for c in raw.values()), raw
|
||||
|
||||
|
||||
# ======================================= the four hybrid cases, A B C D
|
||||
|
||||
def test_case_a_strong_semantic_weak_lexical_still_retrieves(client):
|
||||
"""A conceptual match with almost no shared vocabulary must survive."""
|
||||
adv, _ = campaign(client, CRYPT_SCENE, [
|
||||
("ossuary.md", OSSUARY, "reference"),
|
||||
("ship.md", SHIP_CANON, "canon"),
|
||||
])
|
||||
result = rank(client, adv)
|
||||
found = next((c for c in result.candidates if c.filename == "ossuary.md"), None)
|
||||
assert found is not None, table(result)
|
||||
assert found.admitted_by == "semantic", table(result)
|
||||
assert found.cosine >= classes.SEMANTIC_FLOOR, table(result)
|
||||
assert not found.matched_terms, table(result)
|
||||
assert "ship.md" not in names(result), table(result)
|
||||
|
||||
|
||||
def test_case_b_strong_lexical_weak_semantic_still_retrieves(client):
|
||||
"""A distinctive exact term must retrieve even with embeddings unavailable."""
|
||||
adv, _ = campaign(client, "Aldric asks about Westhaven and the broken circle.", [
|
||||
("abbey.md", ABBEY_CANON, "canon"),
|
||||
("surgery.md", SURGERY_REFERENCE, "reference"),
|
||||
])
|
||||
with SessionLocal() as db:
|
||||
row = db.execute(select(models.Settings).where(
|
||||
models.Settings.user_id == client.user_id)).scalars().first()
|
||||
row.embedding_model = ""
|
||||
db.commit()
|
||||
result = rank(client, adv)
|
||||
assert result.semantic_used is False
|
||||
assert "abbey.md" in names(result), table(result)
|
||||
found = next(c for c in result.candidates if c.filename == "abbey.md")
|
||||
assert found.admitted_by == "lexical", table(result)
|
||||
assert len(found.matched_terms) >= classes.LEXICAL_MIN_TERMS, table(result)
|
||||
assert "surgery.md" not in names(result), table(result)
|
||||
|
||||
|
||||
def test_case_c_both_strong_ranks_once_and_is_not_duplicated(client):
|
||||
adv, _ = campaign(client, CRYPT_SCENE, [
|
||||
("abbey.md", ABBEY_CANON, "canon"),
|
||||
("crypt-ref.md", CRYPT_REFERENCE, "reference"),
|
||||
])
|
||||
result = rank(client, adv)
|
||||
hybrid = [c for c in result.candidates if c.admitted_by == "both"]
|
||||
assert hybrid, table(result)
|
||||
ids = [c.chunk_id for c in result.candidates]
|
||||
assert len(ids) == len(set(ids)), table(result)
|
||||
assert all(c.lexical > 0 and c.semantic > 0 for c in hybrid), table(result)
|
||||
|
||||
|
||||
def test_case_d_neither_strong_retrieves_nothing(client):
|
||||
"""**The mandatory case.** No match on either path means no chunks at all."""
|
||||
adv, _ = campaign(client, OFF_TOPIC_SCENE, [
|
||||
("abbey.md", ABBEY_CANON, "canon"),
|
||||
("crypt-ref.md", CRYPT_REFERENCE, "reference"),
|
||||
("mood.md", CRYPT_MOOD, "inspiration"),
|
||||
("ship.md", SHIP_CANON, "canon"),
|
||||
])
|
||||
result = rank(client, adv)
|
||||
assert result.candidates == [], table(result)
|
||||
assert result.suppressed == [], table(result)
|
||||
assert result.generated > 0, (
|
||||
"nothing was even generated — the test would pass for the wrong reason")
|
||||
assert result.rejected == result.generated, table(result)
|
||||
|
||||
# ...and the assembled prompt carries no imported section at all.
|
||||
report = client.get(f"/api/adventures/{adv}/context").json()
|
||||
assert report["knowledge"]["used"] == []
|
||||
assert not [s for s in report["sections"] if s["label"].startswith("imported_")]
|
||||
assert not [s for s in report["sections"]
|
||||
if s["label"] == classes.SECTION_RULE]
|
||||
|
||||
|
||||
def test_case_d_holds_on_the_lexical_only_path_too(client):
|
||||
adv, _ = campaign(client, OFF_TOPIC_SCENE, [
|
||||
("abbey.md", ABBEY_CANON, "canon"),
|
||||
("ship.md", SHIP_CANON, "canon"),
|
||||
])
|
||||
with SessionLocal() as db:
|
||||
row = db.execute(select(models.Settings).where(
|
||||
models.Settings.user_id == client.user_id)).scalars().first()
|
||||
row.embedding_model = ""
|
||||
db.commit()
|
||||
result = rank(client, adv)
|
||||
assert result.candidates == [], table(result)
|
||||
|
||||
|
||||
# ============================== authority must not rescue irrelevance
|
||||
|
||||
@pytest.mark.parametrize("classification", ["canon", "reference", "inspiration"])
|
||||
def test_irrelevant_material_is_excluded_whatever_its_class(client, classification):
|
||||
"""Each class, alone in the library, with nothing else to compete with.
|
||||
|
||||
The old rule admitted whatever was best; with one source there is nothing
|
||||
else, so "best" and "only" coincide and the failure is unmissable.
|
||||
"""
|
||||
adv, _ = campaign(client, OFF_TOPIC_SCENE, [
|
||||
("lore.md", ABBEY_CANON, classification),
|
||||
])
|
||||
result = rank(client, adv)
|
||||
assert result.candidates == [], table(result)
|
||||
assert result.generated >= 1, "nothing generated; the test proves nothing"
|
||||
|
||||
|
||||
def test_canon_is_excluded_even_though_it_is_the_best_candidate(client):
|
||||
"""Explicitly the shape of M7-F1: best of a bad set is still not relevant."""
|
||||
adv, _ = campaign(client, OFF_TOPIC_SCENE, [
|
||||
("abbey.md", ABBEY_CANON, "canon"),
|
||||
("ship.md", SHIP_CANON, "canon"),
|
||||
("surgery.md", SURGERY_REFERENCE, "reference"),
|
||||
])
|
||||
result = rank(client, adv)
|
||||
assert names(result) == [], table(result)
|
||||
|
||||
|
||||
def test_once_relevant_canon_outranks_relevant_reference_and_inspiration(client):
|
||||
"""Authority still orders what did match — the other half of §30."""
|
||||
adv, _ = campaign(client, CRYPT_SCENE, [
|
||||
("abbey.md", ABBEY_CANON, "canon"),
|
||||
("crypt-ref.md", CRYPT_REFERENCE, "reference"),
|
||||
("mood.md", CRYPT_MOOD, "inspiration"),
|
||||
])
|
||||
result = rank(client, adv)
|
||||
by = {c.filename: c for c in result.candidates}
|
||||
assert "abbey.md" in by, table(result)
|
||||
for lower in ("crypt-ref.md", "mood.md"):
|
||||
if lower in by:
|
||||
assert by["abbey.md"].score > by[lower].score, table(result)
|
||||
# and the class is what did it, at comparable relevance
|
||||
equal = 0.5
|
||||
assert (equal * classes.CLASS_WEIGHTS[classes.CANON]
|
||||
> equal * classes.CLASS_WEIGHTS[classes.REFERENCE]
|
||||
> equal * classes.CLASS_WEIGHTS[classes.INSPIRATION])
|
||||
|
||||
|
||||
def test_relevant_reference_outranks_irrelevant_canon(client):
|
||||
adv, _ = campaign(client, CRYPT_SCENE, [
|
||||
("crypt-ref.md", CRYPT_REFERENCE, "reference"),
|
||||
("ship.md", SHIP_CANON, "canon"),
|
||||
])
|
||||
result = rank(client, adv)
|
||||
assert "crypt-ref.md" in names(result), table(result)
|
||||
assert "ship.md" not in names(result), table(result)
|
||||
|
||||
|
||||
# ================================================ the surviving mechanics
|
||||
|
||||
def test_near_duplicates_are_suppressed_before_the_cut(client):
|
||||
adv, _ = campaign(client, CRYPT_SCENE, [
|
||||
("abbey.md", ABBEY_CANON, "canon"),
|
||||
("abbey-copy.md", ABBEY_COPY, "canon"),
|
||||
])
|
||||
result = rank(client, adv)
|
||||
kept = [c for c in result.candidates if c.filename.startswith("abbey")]
|
||||
assert kept, table(result)
|
||||
assert len(kept) == 1, table(result)
|
||||
assert result.suppressed, table(result)
|
||||
assert all(c.duplicate_of is not None for c in result.suppressed)
|
||||
|
||||
|
||||
def test_suppression_never_crosses_a_class(client):
|
||||
adv, _ = campaign(client, CRYPT_SCENE, [
|
||||
("abbey.md", ABBEY_CANON, "canon"),
|
||||
("abbey-copy.md", ABBEY_COPY, "reference"),
|
||||
])
|
||||
result = rank(client, adv)
|
||||
by_id = {c.chunk_id: c for c in result.candidates}
|
||||
for suppressed in result.suppressed:
|
||||
keeper = by_id.get(suppressed.duplicate_of)
|
||||
assert keeper is not None
|
||||
assert keeper.classification == suppressed.classification, table(result)
|
||||
|
||||
|
||||
def test_a_disabled_source_is_excluded_before_admission(client):
|
||||
adv, ids = campaign(client, CRYPT_SCENE, [("abbey.md", ABBEY_CANON, "canon")])
|
||||
assert "abbey.md" in names(rank(client, adv))
|
||||
client.patch(f"/api/adventures/{adv}/knowledge/{ids['abbey.md']}",
|
||||
json={"enabled": False})
|
||||
embeddings.forget_cached(adv)
|
||||
after = rank(client, adv)
|
||||
assert after.candidates == []
|
||||
assert after.generated == 0, "a disabled source still reached candidate generation"
|
||||
|
||||
|
||||
def test_a_source_in_another_campaign_cannot_win(client):
|
||||
adv_a, _ = campaign(client, CRYPT_SCENE, [("abbey.md", ABBEY_CANON, "canon")])
|
||||
adv_b, _ = campaign(client, CRYPT_SCENE, [])
|
||||
result = rank(client, adv_b)
|
||||
assert result.candidates == [] and result.generated == 0
|
||||
assert "abbey.md" in names(rank(client, adv_a))
|
||||
|
||||
|
||||
def test_every_score_and_reason_is_recorded(client):
|
||||
adv, _ = campaign(client, CRYPT_SCENE, [("abbey.md", ABBEY_CANON, "canon")])
|
||||
result = rank(client, adv)
|
||||
assert result.candidates, table(result)
|
||||
for candidate in result.candidates:
|
||||
record = candidate.as_record()
|
||||
for field in ("chunk_id", "source_id", "filename", "classification",
|
||||
"mode", "lexical", "semantic", "cosine", "score",
|
||||
"admitted_by", "matched_terms"):
|
||||
assert field in record, field
|
||||
assert record["mode"] in ("lexical", "semantic", "hybrid", "always")
|
||||
assert record["admitted_by"] in ("lexical", "semantic", "both")
|
||||
assert result.semantic_floor == classes.SEMANTIC_FLOOR
|
||||
assert result.generated >= len(result.candidates)
|
||||
Reference in New Issue
Block a user