v1.1 WP-B.1: diagnose independent long-term memory retention
Diagnostic only; no memory behaviour changes. - tools/memory_diagnostic.py: planted-fact isolation checks, the four-stage diagnosis (created / retained / ranked / injected) with a verdict, a production-ranking replica, deterministic summariser/embedder/narrator stubs and seven scenarios (default, past capacity, pinned, low top_k, long-block early/late, lineage control) - tools/v11_b1_memory.py: CLI for the scenarios and for diagnosing a copy of a finished real campaign - tools/m11_long_run.py: opt-in --independent-fact mode with per-turn isolation tracking and the recovered_through_memory_independent verdict; M04 verdicts unchanged - tests: diagnostic stages, eviction, creation window, ranking, lineage and authority controls; two strict xfails record the diagnosed retention and creation defects for WP-B.2 to flip - planning/reports/v1.1/V1.1-WP-B1-REPORT.md First failing stage: ranking (real model); retention past capacity and creation for early facts in long blocks (deterministic, same on v1.0.0). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VvegagkhuCZoFPdv4M1egY
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 5
Claude Opus 5
parent
d63804f22e
commit
beb17ada10
@@ -0,0 +1,839 @@
|
||||
"""v1.1 WP-B.1: where an early story fact is lost on its way to the narrator.
|
||||
|
||||
One planted fact **F** has four stages to survive before the narrator can use
|
||||
it from memory, and this module reports each one separately:
|
||||
|
||||
created a memory whose `source_start`..`source_end` covers the planting
|
||||
depth carries F
|
||||
retained that memory is not `forgotten`
|
||||
ranked it is eligible on the active lineage and embedded, and where it
|
||||
scores for the recall query against `memory_top_k`
|
||||
injected the recall turn's own stored `memories.used` names it, and its text
|
||||
is in that turn's `used_memories` section
|
||||
|
||||
A fact is only evidence about memory if memory is the **only** thing carrying it.
|
||||
`isolation()` checks every other layer: the authoritative document, per-node state
|
||||
snapshots, the active summary, imported knowledge, the narration after the
|
||||
planting block, and the recent-history window. A run where any of those carries F
|
||||
is reported as a failed precondition, never as a memory result.
|
||||
|
||||
**Nothing here changes behaviour.**
|
||||
- It reads rows.
|
||||
- It reuses production's own pure helpers (`memorybank._drop_redundant`,
|
||||
`memorybank.classify_authority`, `vectors.cosine`, `lineage.path_of`), so its
|
||||
ranking is production's ranking, not a second opinion.
|
||||
- It checks itself against what the recall turn actually recorded.
|
||||
- The only computed fields are ephemeral report data. No column or table is
|
||||
added.
|
||||
|
||||
The deterministic stubs at the bottom stand in for the models when a test needs a
|
||||
fixed answer. **Read what they model before reading any result they produce:**
|
||||
|
||||
- `BestCaseSummariser` keeps F if and only if F is in the excerpt it is given.
|
||||
It is the ideal summariser, so a creation failure under it is the
|
||||
application's, not the model's.
|
||||
- `ConceptEmbedder` maps words to a small concept table, so that "the brass dial
|
||||
that tells the hour" lands near "sundial". It models what an embedding is
|
||||
supposed to do. It says nothing about how well `nomic-embed-text` does it,
|
||||
which is what the real-model run is for.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app import memorybank, models, summaries, vectors
|
||||
from app.context import builder, history, lineage
|
||||
from app.knowledge import classes as knowledge_classes
|
||||
|
||||
VERDICTS = (
|
||||
"not_created",
|
||||
"created_but_evicted",
|
||||
"retained_but_not_ranked",
|
||||
"ranked_but_not_selected",
|
||||
"selected_but_not_injected",
|
||||
"injected",
|
||||
)
|
||||
|
||||
#: Section labels in a stored context snapshot. Copied from the builder's
|
||||
#: vocabulary so a renamed section fails loudly here.
|
||||
HISTORY_LABELS = ("history", "recent_history")
|
||||
SUMMARY_LABEL = "story_summary"
|
||||
MEMORIES_LABEL = "used_memories"
|
||||
STATE_LABEL = "narrative_state"
|
||||
KNOWLEDGE_LABELS = (
|
||||
knowledge_classes.SECTION_CANON,
|
||||
knowledge_classes.SECTION_REFERENCE,
|
||||
knowledge_classes.SECTION_INSPIRATION,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Fact:
|
||||
"""A planted fact, and how to recognise it in a text.
|
||||
|
||||
`carry_groups`: a text carries the fact when every group matches, where a
|
||||
group matches when any one of its terms appears as a whole word. A memory has
|
||||
to name both the thing and where it is to carry "where the thing is".
|
||||
|
||||
`leak_terms`: any one of these in another layer means that layer carries the
|
||||
fact. This is deliberately looser than `carry_groups`. For isolation, a
|
||||
mention is enough to disqualify.
|
||||
"""
|
||||
|
||||
fact_id: str
|
||||
sentence: str
|
||||
carry_groups: tuple[tuple[str, ...], ...]
|
||||
leak_terms: tuple[str, ...]
|
||||
|
||||
def carried_by(self, text: str | None) -> bool:
|
||||
low = (text or "").lower()
|
||||
return all(any(_has_word(low, term) for term in group) for group in self.carry_groups)
|
||||
|
||||
def mentioned_by(self, text: str | None) -> bool:
|
||||
low = (text or "").lower()
|
||||
return any(_has_word(low, term) for term in self.leak_terms)
|
||||
|
||||
|
||||
def _has_word(low: str, term: str) -> bool:
|
||||
return re.search(rf"(?<![a-z]){re.escape(term.lower())}(?![a-z])", low) is not None
|
||||
|
||||
|
||||
#: The fixture's planted fact. Chosen to be natural in a tavern scene and absent
|
||||
#: from every existing fixture: no "sundial" or "teapot" appears anywhere in the
|
||||
#: Westhaven campaign, its knowledge files or its beats.
|
||||
FACT_F = Fact(
|
||||
fact_id="F-amber-sundial",
|
||||
sentence="Mara slipped the amber sundial inside the cracked teapot on the tavern's top shelf.",
|
||||
carry_groups=(("sundial",), ("teapot",)),
|
||||
leak_terms=("sundial", "teapot"),
|
||||
)
|
||||
#: The abandoned-line control fact.
|
||||
FACT_G = Fact(
|
||||
fact_id="G-iron-weathervane",
|
||||
sentence="Edrin buried the iron weathervane beneath the mill's broken waterwheel.",
|
||||
carry_groups=(("weathervane",), ("waterwheel",)),
|
||||
leak_terms=("weathervane", "waterwheel"),
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ reading
|
||||
|
||||
def _lineage_actions(db, adventure):
|
||||
path = lineage.path_of(db, adventure)
|
||||
return (
|
||||
db.query(models.Action)
|
||||
.filter(models.Action.adventure_id == adventure.id, path.clause(models.Action))
|
||||
.order_by(models.Action.depth, models.Action.id)
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
def covering_memories(db, adventure, depth: int, *, any_branch: bool = False):
|
||||
"""Memories whose source range covers `depth`, oldest first."""
|
||||
query = select(models.Memory).where(
|
||||
models.Memory.adventure_id == adventure.id,
|
||||
models.Memory.source_start <= depth,
|
||||
models.Memory.source_end >= depth,
|
||||
)
|
||||
if not any_branch:
|
||||
query = query.where(lineage.path_of(db, adventure).clause(models.Memory))
|
||||
return db.execute(query.order_by(models.Memory.id)).scalars().all()
|
||||
|
||||
|
||||
def planting_block_end(db, adventure, plant_depth: int) -> int:
|
||||
"""The last depth of the memory block holding the planted turn.
|
||||
|
||||
Taken from the memory that covers it where one exists. Before one exists it
|
||||
is the furthest a block could reach, so a later-narration check never counts
|
||||
a turn inside the planting block as a repetition.
|
||||
"""
|
||||
rows = covering_memories(db, adventure, plant_depth)
|
||||
if rows:
|
||||
return max(row.source_end for row in rows)
|
||||
return plant_depth + memorybank.MEMORY_INTERVAL
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- isolation
|
||||
|
||||
def isolation(db, adventure, fact: Fact, plant_depth: int, *,
|
||||
recall_snapshot: dict | None = None,
|
||||
recall_depth: int | None = None) -> dict:
|
||||
"""Every layer other than memory that could carry F, checked.
|
||||
|
||||
Returns `{check: {"ok": bool, "detail": str}}` and `ok` over all of them.
|
||||
With `recall_snapshot`, the recall turn's stored context, the prompt-level
|
||||
checks (history window, summary section, knowledge sections) are made
|
||||
against what the narrator was actually given.
|
||||
"""
|
||||
checks: dict[str, dict] = {}
|
||||
|
||||
document = adventure.narrative_state or {}
|
||||
hits = [key for key in ("entities", "facts", "relationships", "threads", "scene",
|
||||
"possessions")
|
||||
if fact.mentioned_by(json.dumps(document.get(key), default=str))]
|
||||
checks["state_document"] = {
|
||||
"ok": not hits and not fact.mentioned_by(json.dumps(document, default=str)),
|
||||
"detail": f"mentioned in {hits}" if hits else "absent",
|
||||
}
|
||||
|
||||
snapshot_hits = []
|
||||
later_hits = []
|
||||
block_end = planting_block_end(db, adventure, plant_depth)
|
||||
for action in _lineage_actions(db, adventure):
|
||||
if fact.mentioned_by(json.dumps(action.narrative_state_after, default=str)):
|
||||
snapshot_hits.append(action.depth)
|
||||
if (action.type == "ai" and action.depth is not None and action.depth > block_end
|
||||
and (recall_depth is None or action.depth < recall_depth)
|
||||
and fact.mentioned_by(action.text)):
|
||||
later_hits.append(action.depth)
|
||||
checks["state_snapshots"] = {
|
||||
"ok": not snapshot_hits,
|
||||
"detail": f"mentioned in snapshots at depths {snapshot_hits[:10]}" if snapshot_hits
|
||||
else "absent from every node's narrative_state_after on the active lineage",
|
||||
}
|
||||
checks["later_narration"] = {
|
||||
"ok": not later_hits,
|
||||
"detail": (f"narration after the planting block (ends at depth {block_end}) "
|
||||
f"mentions the fact at depths {later_hits[:10]}") if later_hits
|
||||
else f"no narrator turn after depth {block_end} mentions the fact",
|
||||
}
|
||||
|
||||
active = summaries.current(db, adventure)
|
||||
summary_text = active.text if active is not None else ""
|
||||
if recall_snapshot is not None:
|
||||
summary_text += "\n" + _section(recall_snapshot, SUMMARY_LABEL)
|
||||
checks["summary"] = {
|
||||
"ok": not fact.mentioned_by(summary_text),
|
||||
"detail": "the active summary mentions the fact" if fact.mentioned_by(summary_text)
|
||||
else ("absent from the active summary" if active is not None else "no summary yet"),
|
||||
}
|
||||
|
||||
sources = db.execute(
|
||||
select(models.KnowledgeSource.content).where(
|
||||
models.KnowledgeSource.adventure_id == adventure.id)
|
||||
).scalars().all()
|
||||
knowledge_text = "\n".join(s or "" for s in sources)
|
||||
if recall_snapshot is not None:
|
||||
knowledge_text += "\n" + "\n".join(_section(recall_snapshot, l) for l in KNOWLEDGE_LABELS)
|
||||
checks["knowledge"] = {
|
||||
"ok": not fact.mentioned_by(knowledge_text),
|
||||
"detail": "imported knowledge mentions the fact" if fact.mentioned_by(knowledge_text)
|
||||
else f"absent from {len(sources)} imported source(s)",
|
||||
}
|
||||
|
||||
if recall_snapshot is not None:
|
||||
hist = recall_snapshot.get("history") or {}
|
||||
floor = hist.get("floor_depth")
|
||||
history_text = "\n".join(_section(recall_snapshot, l) for l in HISTORY_LABELS)
|
||||
outside = floor is not None and plant_depth < floor
|
||||
checks["recent_history"] = {
|
||||
"ok": outside and not fact.carried_by(history_text),
|
||||
"detail": (f"history window starts at depth {floor}; planted at {plant_depth}; "
|
||||
f"fact text in history sections: {fact.carried_by(history_text)}"),
|
||||
}
|
||||
checks["state_section"] = {
|
||||
"ok": not fact.mentioned_by(_section(recall_snapshot, STATE_LABEL)),
|
||||
"detail": "the recall prompt's narrative_state section "
|
||||
+ ("mentions the fact" if fact.mentioned_by(_section(recall_snapshot, STATE_LABEL))
|
||||
else "does not mention the fact"),
|
||||
}
|
||||
|
||||
return {"ok": all(c["ok"] for c in checks.values()), "checks": checks}
|
||||
|
||||
|
||||
def _section(snapshot: dict, label: str) -> str:
|
||||
return "\n".join(s.get("text", "") for s in (snapshot.get("sections") or [])
|
||||
if s.get("label") == label)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- stages
|
||||
|
||||
async def rank_bank(db, adventure, settings, query: str, embed) -> dict:
|
||||
"""Production's ranking, recomputed for `query`, for every eligible memory.
|
||||
|
||||
The same catalogue clause, the same cosine, the same pin rule, the same
|
||||
redundancy suppression helper. Returns every scored row, not just the top-k,
|
||||
because "where did F rank" is the question.
|
||||
"""
|
||||
catalogue = db.execute(
|
||||
select(models.Memory.id, models.Memory.pinned, models.Memory.authority,
|
||||
models.Memory.embedding_blob).where(
|
||||
models.Memory.adventure_id == adventure.id,
|
||||
lineage.path_of(db, adventure).clause(models.Memory),
|
||||
models.Memory.forgotten.is_(False),
|
||||
models.Memory.embedded.is_(True),
|
||||
)
|
||||
).all()
|
||||
if not catalogue or not query.strip():
|
||||
return {"query": query, "scored": [], "selected": [], "top_k": settings.memory_top_k}
|
||||
[query_vec] = await embed([query])
|
||||
held = {row.id: vectors.unpack(row.embedding_blob) for row in catalogue if row.embedding_blob}
|
||||
authority_of = {row.id: row.authority for row in catalogue}
|
||||
scored = sorted(
|
||||
((vectors.cosine(query_vec, held[row.id]), row.id, row.pinned)
|
||||
for row in catalogue if row.id in held),
|
||||
key=lambda r: r[0], reverse=True,
|
||||
)
|
||||
top_k = max(1, settings.memory_top_k)
|
||||
used = [r for r in scored if r[2]]
|
||||
remaining = max(0, top_k - len(used))
|
||||
candidates = [r for r in scored if not r[2]]
|
||||
kept, suppressed = memorybank._drop_redundant(candidates, held, authority_of, remaining)
|
||||
selected = {r[1] for r in used + kept}
|
||||
suppressed_by = dict(suppressed)
|
||||
return {
|
||||
"query": query,
|
||||
"top_k": top_k,
|
||||
"scored": [
|
||||
{"rank": i + 1, "memory_id": memory_id, "similarity": round(score, 4),
|
||||
"pinned": pinned, "selected": memory_id in selected,
|
||||
"suppressed_as_duplicate_of": suppressed_by.get(memory_id)}
|
||||
for i, (score, memory_id, pinned) in enumerate(scored)
|
||||
],
|
||||
"selected": sorted(selected),
|
||||
}
|
||||
|
||||
|
||||
def production_query(adventure, exclude_action_id: int | None) -> str:
|
||||
"""The retrieval query a turn used: its newest actions, as `retrieve_memories` builds it."""
|
||||
recent = history.tail(adventure, memorybank.RETRIEVAL_WINDOW_ACTIONS, exclude_action_id)
|
||||
return builder.truncate_to_last_tokens(
|
||||
"\n\n".join(a.text for a in recent), memorybank.RETRIEVAL_WINDOW_TOKENS)
|
||||
|
||||
|
||||
def eviction_order(db, adventure) -> list[int]:
|
||||
"""The order `_evict_over_capacity` would take unpinned active memories in."""
|
||||
from sqlalchemy import func
|
||||
return db.execute(
|
||||
select(models.Memory.id).where(
|
||||
models.Memory.adventure_id == adventure.id,
|
||||
models.Memory.forgotten.is_(False),
|
||||
models.Memory.pinned.is_(False),
|
||||
).order_by(func.coalesce(models.Memory.last_used_at, models.Memory.created_at),
|
||||
models.Memory.use_count)
|
||||
).scalars().all()
|
||||
|
||||
|
||||
async def diagnose(db, adventure, settings, fact: Fact, plant_depth: int, *,
|
||||
recall_action: models.Action, embed) -> dict:
|
||||
"""The four stages for `fact`, judged at `recall_action`, the recall turn's AI node.
|
||||
|
||||
Ranking is recomputed with the query that turn used, and checked against the
|
||||
turn's own stored `memories.used`. Injection is read from that snapshot, so
|
||||
it reports what the narrator was actually given, not a re-run.
|
||||
"""
|
||||
snapshot = recall_action.context_snapshot or {}
|
||||
out: dict = {"fact_id": fact.fact_id, "plant_depth": plant_depth,
|
||||
"recall_depth": recall_action.depth}
|
||||
|
||||
covering = covering_memories(db, adventure, plant_depth)
|
||||
carrying = [m for m in covering if fact.carried_by(m.text)]
|
||||
elsewhere = [m for m in db.execute(select(models.Memory).where(
|
||||
models.Memory.adventure_id == adventure.id)).scalars().all()
|
||||
if fact.carried_by(m.text) and m not in carrying]
|
||||
creation_input = []
|
||||
for memory in covering:
|
||||
block = memorybank.source_block(db, memory)
|
||||
raw = "\n\n".join(a.text for a in block)
|
||||
excerpt = builder.truncate_to_last_tokens(raw, memorybank.MEMORY_EXCERPT_TOKENS)
|
||||
creation_input.append({
|
||||
"memory_id": memory.id, "source_start": memory.source_start,
|
||||
"source_end": memory.source_end, "block_tokens": builder.count_tokens(raw),
|
||||
"fact_in_block": fact.carried_by(raw),
|
||||
"fact_in_summariser_excerpt": fact.carried_by(excerpt),
|
||||
"memory_text": memory.text,
|
||||
})
|
||||
memory = carrying[0] if carrying else None
|
||||
out["created"] = {
|
||||
"yes": memory is not None,
|
||||
"memory_id": getattr(memory, "id", None),
|
||||
"source_start": getattr(memory, "source_start", None),
|
||||
"source_end": getattr(memory, "source_end", None),
|
||||
"memory_text": getattr(memory, "text", None),
|
||||
"covering_memories": creation_input,
|
||||
"no_covering_memory": not covering,
|
||||
"carried_by_other_memories": [
|
||||
{"memory_id": m.id, "source_start": m.source_start, "source_end": m.source_end}
|
||||
for m in elsewhere],
|
||||
}
|
||||
|
||||
if memory is None:
|
||||
out["verdict"] = "not_created"
|
||||
return out
|
||||
|
||||
order = eviction_order(db, adventure)
|
||||
active = db.execute(select(models.Memory.id).where(
|
||||
models.Memory.adventure_id == adventure.id,
|
||||
models.Memory.forgotten.is_(False))).scalars().all()
|
||||
on_lineage = db.execute(select(models.Memory.id).where(
|
||||
models.Memory.id == memory.id,
|
||||
lineage.path_of(db, adventure).clause(models.Memory))).scalar() is not None
|
||||
out["retained"] = {
|
||||
"yes": not memory.forgotten,
|
||||
"forgotten": memory.forgotten,
|
||||
"pinned": memory.pinned,
|
||||
"embedded": memory.embedded,
|
||||
"on_active_lineage": on_lineage,
|
||||
"use_count": memory.use_count,
|
||||
"last_used_at": str(memory.last_used_at) if memory.last_used_at else None,
|
||||
"created_at": str(memory.created_at),
|
||||
"active_memories": len(active),
|
||||
"memory_bank_capacity": settings.memory_bank_capacity,
|
||||
"eviction_position": (order.index(memory.id) + 1) if memory.id in order else None,
|
||||
"reason": ("evicted: marked forgotten by capacity eviction" if memory.forgotten
|
||||
else "active"),
|
||||
}
|
||||
if memory.forgotten:
|
||||
out["verdict"] = "created_but_evicted"
|
||||
return out
|
||||
|
||||
query = production_query(adventure, recall_action.id)
|
||||
ranking = await rank_bank(db, adventure, settings, query, embed)
|
||||
row = next((r for r in ranking["scored"] if r["memory_id"] == memory.id), None)
|
||||
stored_used = [m.get("id") for m in (snapshot.get("memories") or {}).get("used") or []]
|
||||
out["ranked"] = {
|
||||
"yes": row is not None and row["rank"] <= ranking["top_k"],
|
||||
"eligible": row is not None,
|
||||
"lexical_score": None, # memory ranking has no lexical term (CONTEXT-AND-MEMORY §20)
|
||||
"semantic_score": row["similarity"] if row else None,
|
||||
"final_score": row["similarity"] if row else None,
|
||||
"pin_effect": "always selected" if memory.pinned else "none",
|
||||
"rank": row["rank"] if row else None,
|
||||
"of": len(ranking["scored"]),
|
||||
"top_k_cutoff": ranking["top_k"],
|
||||
"selected": bool(row and row["selected"]),
|
||||
"suppressed_as_duplicate_of": row["suppressed_as_duplicate_of"] if row else None,
|
||||
"query": query,
|
||||
"replica_matches_stored_selection": sorted(stored_used) == ranking["selected"],
|
||||
}
|
||||
if row is None or row["rank"] > ranking["top_k"] and not row["selected"]:
|
||||
out["verdict"] = "retained_but_not_ranked"
|
||||
return out
|
||||
if not row["selected"]:
|
||||
out["verdict"] = "ranked_but_not_selected"
|
||||
return out
|
||||
|
||||
section = _section(snapshot, MEMORIES_LABEL)
|
||||
injected = memory.id in stored_used and memory.text in section
|
||||
out["injected"] = {
|
||||
"yes": injected,
|
||||
"context_component": MEMORIES_LABEL,
|
||||
"in_stored_memories_used": memory.id in stored_used,
|
||||
"text_in_section": memory.text in section,
|
||||
"token_count": builder.count_tokens(section) if section else 0,
|
||||
}
|
||||
out["verdict"] = "injected" if injected else "selected_but_not_injected"
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the stubs
|
||||
|
||||
@dataclass
|
||||
class BestCaseSummariser:
|
||||
"""The ideal memory writer: F survives if, and only if, F reached it.
|
||||
|
||||
A memory keeps every sentence of the excerpt that carries a planted fact, and
|
||||
adds one sentence naming the block's own distinct detail so memories differ.
|
||||
Summary updates never repeat a planted fact, so the summary layer stays out of
|
||||
the experiment. Every excerpt it was given is kept, for the creation-window
|
||||
diagnostic.
|
||||
"""
|
||||
|
||||
facts: tuple[Fact, ...] = (FACT_F, FACT_G)
|
||||
excerpts: list = field(default_factory=list)
|
||||
|
||||
async def complete(self, system, user, *, temperature=0.3, max_tokens=400):
|
||||
if "Current story summary:" in user:
|
||||
return "The travellers kept moving through the country around Westhaven."
|
||||
excerpt = user.split("Story excerpt:\n\n", 1)[-1].rsplit("\n\nMemory:", 1)[0]
|
||||
self.excerpts.append(excerpt)
|
||||
kept = [s.strip() for s in re.split(r"(?<=[.!?])\s+", excerpt)
|
||||
if any(f.carried_by(s) for f in self.facts)]
|
||||
detail = re.findall(r"\bat the ([a-z]+ [a-z]+)\b", excerpt.lower())
|
||||
tail = f"The travellers spent time at the {detail[-1]}." if detail else \
|
||||
"The travellers pressed on."
|
||||
return " ".join(dict.fromkeys(kept + [tail]))
|
||||
|
||||
|
||||
#: Words that mean the same thing to `ConceptEmbedder`. The point is only that a
|
||||
#: paraphrase lands near the original; the table is the model of that.
|
||||
CONCEPTS = {
|
||||
"timepiece": ("sundial", "dial", "hour", "hours", "clock", "timepiece"),
|
||||
"vessel": ("teapot", "pot", "kettle", "tea", "jar"),
|
||||
"hid": ("hid", "hide", "hidden", "slipped", "tucked", "put", "stashed"),
|
||||
"weathervane": ("weathervane", "vane"),
|
||||
"waterwheel": ("waterwheel", "wheel", "mill"),
|
||||
}
|
||||
_WORD_TO_CONCEPT = {w: c for c, words in CONCEPTS.items() for w in words}
|
||||
DIMENSIONS = 96
|
||||
|
||||
|
||||
@dataclass
|
||||
class ConceptEmbedder:
|
||||
"""A deterministic embedding: concepts in fixed dimensions, other words hashed."""
|
||||
|
||||
calls: int = 0
|
||||
|
||||
async def embed(self, texts):
|
||||
self.calls += 1
|
||||
return [self.vector(t) for t in texts]
|
||||
|
||||
@staticmethod
|
||||
def vector(text: str) -> list[float]:
|
||||
v = [0.0] * DIMENSIONS
|
||||
v[0] = 0.2 # every text shares a little, as real embeddings do
|
||||
concept_names = list(CONCEPTS)
|
||||
for word in re.findall(r"[a-z]+", text.lower()):
|
||||
concept = _WORD_TO_CONCEPT.get(word)
|
||||
if concept is not None:
|
||||
v[1 + concept_names.index(concept)] += 3.0
|
||||
elif len(word) > 3:
|
||||
bucket = int(hashlib.sha256(word.encode()).hexdigest(), 16)
|
||||
v[1 + len(concept_names) + bucket % (DIMENSIONS - 1 - len(concept_names))] += 1.0
|
||||
norm = math.sqrt(sum(x * x for x in v)) or 1.0
|
||||
return [x / norm for x in v]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- scenarios
|
||||
|
||||
#: Filler places. No word here is in `CONCEPTS`, and none names a planted fact.
|
||||
PLACES = (
|
||||
"north gate", "salt market", "ferry landing", "chapel steps", "rope walk",
|
||||
"fish stalls", "old bridge", "tanner yard", "lamp street", "weir path",
|
||||
"grain store", "boat yard", "watch house", "cloth hall", "eel traps",
|
||||
"sheep fold", "smith forge", "stone quay", "reed beds", "toll booth",
|
||||
)
|
||||
PARAPHRASE_QUERY = "I ask Mara where she tucked the little brass dial that tells the hour."
|
||||
UNRELATED_QUERY = "I ask the ferryman what rope costs at the landing this season."
|
||||
|
||||
|
||||
def filler_prose(index: int, words: int) -> str:
|
||||
"""Narration that moves on and never touches a planted fact."""
|
||||
place = PLACES[index % len(PLACES)]
|
||||
sentence = (f"At the {place} the travellers stopped, listened to the gulls over the "
|
||||
f"grey water, and talked about the long road north.")
|
||||
reps = max(1, round(words / len(sentence.split())))
|
||||
return " ".join([sentence] * reps)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Scenario:
|
||||
"""One deterministic campaign. Depths: the opening is 0, turn *n*'s player
|
||||
action is 2n-1 and its reply 2n."""
|
||||
|
||||
name: str
|
||||
turns: int = 52
|
||||
capacity: int = 80
|
||||
top_k: int = 5
|
||||
budget: int = 4096
|
||||
prose_words: int = 60
|
||||
plant_turn: int = 1
|
||||
recall_text: str = "I ask Mara where she hid the amber sundial."
|
||||
pin_first_memory: bool = False
|
||||
lineage_control: bool = False
|
||||
diagnose_recall: bool = True
|
||||
|
||||
|
||||
SCENARIOS = {
|
||||
"independent_default": Scenario("independent_default"),
|
||||
"past_capacity": Scenario("past_capacity", capacity=6),
|
||||
"past_capacity_pinned": Scenario("past_capacity_pinned", capacity=6, pin_first_memory=True),
|
||||
# Closer to the shipped ratio (memory_top_k 5 against capacity 80): most of
|
||||
# the bank is not retrieved on a given turn.
|
||||
"past_capacity_low_top_k": Scenario("past_capacity_low_top_k", capacity=8, top_k=2),
|
||||
"long_block_fact_early": Scenario("long_block_fact_early", turns=10, prose_words=850,
|
||||
plant_turn=1, budget=16384),
|
||||
"long_block_fact_late": Scenario("long_block_fact_late", turns=10, prose_words=850,
|
||||
plant_turn=3, budget=16384),
|
||||
"lineage_control": Scenario("lineage_control", lineage_control=True),
|
||||
}
|
||||
|
||||
|
||||
class ScriptNarrator:
|
||||
"""Stands in for the narrator: returns `next_reply`, with an empty state block."""
|
||||
|
||||
next_reply = ""
|
||||
last_usage = None
|
||||
prompts: list = []
|
||||
|
||||
def __init__(self, *a, **k):
|
||||
pass
|
||||
|
||||
async def generate(self, parts, *, temperature, max_tokens):
|
||||
ScriptNarrator.prompts.append((parts.system, parts.story))
|
||||
yield ("text", ScriptNarrator.next_reply)
|
||||
|
||||
|
||||
def run_scenario(scenario: Scenario) -> dict:
|
||||
"""Plays `scenario` through the real turn route and returns everything measured.
|
||||
|
||||
Uses the database `app.database` is already bound to, creating and dropping
|
||||
its tables, the way the suite's fixtures do. Patches are applied here and
|
||||
removed before returning, so this runs the same under pytest and from the CLI.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
from fastapi import Depends
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy.orm import undefer
|
||||
|
||||
from app import auth, limits
|
||||
from app.database import Base, SessionLocal, engine, get_db
|
||||
from app.main import app
|
||||
from app.routers import adventures as adventure_routes
|
||||
|
||||
summariser = BestCaseSummariser()
|
||||
embedder = ConceptEmbedder()
|
||||
patches = [
|
||||
(memorybank, "summary_provider", lambda s: summariser),
|
||||
(memorybank, "embedding_provider", lambda s: embedder),
|
||||
# Post-turn work is settled explicitly after each turn, so eviction
|
||||
# happens at a known point rather than whenever a background task runs.
|
||||
(memorybank, "schedule_post_turn", lambda adventure: None),
|
||||
(adventure_routes.turns, "OpenAICompatibleProvider", ScriptNarrator),
|
||||
(limits, "check_row_cap", lambda *a, **k: None),
|
||||
]
|
||||
saved = [(obj, name, getattr(obj, name)) for obj, name, _ in patches]
|
||||
for obj, name, value in patches:
|
||||
setattr(obj, name, value)
|
||||
ScriptNarrator.prompts = []
|
||||
|
||||
Base.metadata.create_all(bind=engine)
|
||||
memorybank._vector_cache.clear()
|
||||
with SessionLocal() as db:
|
||||
user = models.User(is_guest=False, email=f"b1-{scenario.name}@example.com")
|
||||
db.add(user)
|
||||
db.flush()
|
||||
db.add(models.Settings(
|
||||
user_id=user.id, model="script", endpoint_url="http://127.0.0.1:9/v1",
|
||||
embedding_model="concept-embed", context_token_budget=scenario.budget,
|
||||
max_output_tokens=500, memory_bank_capacity=scenario.capacity,
|
||||
memory_top_k=scenario.top_k,
|
||||
))
|
||||
adventure = models.Adventure(
|
||||
user_id=user.id, title=f"B.1 {scenario.name}", memory_bank_enabled=True,
|
||||
auto_summarize=True, persona_name="Aldric",
|
||||
)
|
||||
db.add(adventure)
|
||||
db.flush()
|
||||
db.add(models.Action(adventure_id=adventure.id, type="start",
|
||||
text="Rain over Westhaven, and the tavern door banging in the wind."))
|
||||
db.commit()
|
||||
adv, user_id = adventure.id, user.id
|
||||
app.dependency_overrides[auth.get_current_user] = (
|
||||
lambda db=Depends(get_db): db.get(models.User, user_id)
|
||||
)
|
||||
client = TestClient(app)
|
||||
result: dict = {"scenario": scenario.__dict__.copy(), "trace": []}
|
||||
|
||||
def call(method, path, body=None, expect=200):
|
||||
response = client.request(method, f"/api/adventures/{adv}{path}", json=body)
|
||||
assert response.status_code == expect, (path, response.status_code, response.text[:300])
|
||||
return response.json() if response.content else None
|
||||
|
||||
def marks():
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(select(models.Memory.id, models.Memory.forgotten,
|
||||
models.Memory.embedded).where(
|
||||
models.Memory.adventure_id == adv)).all()
|
||||
summaries_n = db.query(models.Summary).filter_by(adventure_id=adv).count()
|
||||
return tuple(sorted(rows)), summaries_n
|
||||
|
||||
def settle():
|
||||
for _ in range(12):
|
||||
before = marks()
|
||||
asyncio.run(memorybank.run_post_turn(adv))
|
||||
if marks() == before:
|
||||
return
|
||||
|
||||
def memories():
|
||||
with SessionLocal() as db:
|
||||
return [dict(row._mapping) for row in db.execute(select(
|
||||
models.Memory.id, models.Memory.text, models.Memory.source_start,
|
||||
models.Memory.source_end, models.Memory.forgotten, models.Memory.pinned,
|
||||
models.Memory.use_count, models.Memory.last_used_at, models.Memory.branch_id,
|
||||
models.Memory.created_at).where(models.Memory.adventure_id == adv)
|
||||
.order_by(models.Memory.id)).all()]
|
||||
|
||||
def turn(kind, text, reply):
|
||||
ScriptNarrator.next_reply = f"{reply}\n```state\n{{\"events\": []}}\n```"
|
||||
response = client.post(f"/api/adventures/{adv}/actions", json={"type": kind, "text": text})
|
||||
assert response.status_code == 200, response.text[:300]
|
||||
assert '"type": "error"' not in response.text, response.text[-300:]
|
||||
|
||||
plant_depth = None
|
||||
f_memory_id = None
|
||||
pinned_id = None
|
||||
known: dict[int, dict] = {}
|
||||
g: dict = {}
|
||||
try:
|
||||
for n in range(1, scenario.turns + 1):
|
||||
if n == scenario.plant_turn:
|
||||
turn("story", FACT_F.sentence, filler_prose(n, scenario.prose_words))
|
||||
with SessionLocal() as db:
|
||||
plant_depth = db.query(models.Action.depth).filter_by(
|
||||
adventure_id=adv, text=FACT_F.sentence).scalar()
|
||||
elif scenario.lineage_control and n == 21:
|
||||
call("POST", "/checkpoints", {"name": "before the mill"}, expect=201)
|
||||
turn("story", FACT_G.sentence, filler_prose(n, scenario.prose_words))
|
||||
with SessionLocal() as db:
|
||||
g["plant_depth"] = db.query(models.Action.depth).filter_by(
|
||||
adventure_id=adv, text=FACT_G.sentence).scalar()
|
||||
elif scenario.lineage_control and n == 30:
|
||||
# Line A carries G's memory. Mark it, then abandon it: Undo back
|
||||
# to before G was planted and write something else.
|
||||
g["line_a"] = call("POST", "/checkpoints", {"name": "line A, after the mill"},
|
||||
expect=201)["id"]
|
||||
with SessionLocal() as db:
|
||||
g_rows = [m for m in db.execute(select(models.Memory).where(
|
||||
models.Memory.adventure_id == adv)).scalars() if FACT_G.carried_by(m.text)]
|
||||
g["memory_ids"] = [m.id for m in g_rows]
|
||||
with SessionLocal() as db:
|
||||
g["last_action_id_before_divergence"] = db.query(models.Action.id).filter_by(
|
||||
adventure_id=adv).order_by(models.Action.id.desc()).limit(1).scalar()
|
||||
for _ in range(9):
|
||||
call("POST", "/undo")
|
||||
turn("do", f"I turn away from the mill and walk to the {PLACES[n % len(PLACES)]}.",
|
||||
filler_prose(n + 100, scenario.prose_words))
|
||||
g["diverged_at_turn"] = n
|
||||
else:
|
||||
turn("do", f"I walk on to the {PLACES[n % len(PLACES)]}.",
|
||||
filler_prose(n, scenario.prose_words))
|
||||
settle()
|
||||
|
||||
rows = memories()
|
||||
created = [r["id"] for r in rows if r["id"] not in known]
|
||||
newly_forgotten = [r["id"] for r in rows
|
||||
if r["forgotten"] and not known.get(r["id"], {}).get("forgotten")]
|
||||
for r in rows:
|
||||
known[r["id"]] = r
|
||||
if f_memory_id is None and plant_depth is not None:
|
||||
for r in rows:
|
||||
if (r["source_start"] is not None and r["source_start"] <= plant_depth
|
||||
<= r["source_end"] and FACT_F.carried_by(r["text"])):
|
||||
f_memory_id = r["id"]
|
||||
if scenario.pin_first_memory and pinned_id is None:
|
||||
candidate = next((r for r in rows if r["id"] != f_memory_id), None)
|
||||
if candidate is not None:
|
||||
call("PATCH", f"/memories/{candidate['id']}", {"pinned": True})
|
||||
pinned_id = candidate["id"]
|
||||
f_row = known.get(f_memory_id) if f_memory_id else None
|
||||
result["trace"].append({
|
||||
"turn": n,
|
||||
"active": sum(1 for r in rows if not r["forgotten"]),
|
||||
"total": len(rows),
|
||||
"created": created,
|
||||
"evicted": newly_forgotten,
|
||||
"created_and_evicted_same_turn": sorted(set(created) & set(newly_forgotten)),
|
||||
"f_memory_id": f_memory_id,
|
||||
"f_forgotten": bool(f_row and f_row["forgotten"]),
|
||||
"f_use_count": f_row["use_count"] if f_row else None,
|
||||
})
|
||||
|
||||
turn("do", scenario.recall_text, filler_prose(999, scenario.prose_words))
|
||||
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, adv)
|
||||
settings = db.query(models.Settings).filter_by(user_id=user_id).first()
|
||||
recall_action = (db.query(models.Action)
|
||||
.filter(models.Action.adventure_id == adv,
|
||||
models.Action.type == "ai")
|
||||
.options(undefer(models.Action.context_snapshot))
|
||||
.order_by(models.Action.id.desc()).first())
|
||||
result["plant_depth"] = plant_depth
|
||||
result["recall_depth"] = recall_action.depth
|
||||
result["isolation"] = isolation(
|
||||
db, adventure, FACT_F, plant_depth,
|
||||
recall_snapshot=recall_action.context_snapshot,
|
||||
recall_depth=recall_action.depth)
|
||||
result["diagnosis"] = asyncio.run(diagnose(
|
||||
db, adventure, settings, FACT_F, plant_depth,
|
||||
recall_action=recall_action, embed=embedder.embed))
|
||||
result["summariser_excerpts"] = len(summariser.excerpts)
|
||||
|
||||
memory_id = result["diagnosis"]["created"]["memory_id"]
|
||||
if memory_id is not None and not result["diagnosis"]["retained"]["forgotten"]:
|
||||
variants = {}
|
||||
for label, query in (("direct", scenario.recall_text),
|
||||
("paraphrase", PARAPHRASE_QUERY),
|
||||
("unrelated", UNRELATED_QUERY)):
|
||||
ranking = asyncio.run(rank_bank(db, adventure, settings, query, embedder.embed))
|
||||
row = next((r for r in ranking["scored"] if r["memory_id"] == memory_id), None)
|
||||
variants[label] = {"query": query, "rank": row and row["rank"],
|
||||
"of": len(ranking["scored"]),
|
||||
"similarity": row and row["similarity"],
|
||||
"selected": bool(row and row["selected"]),
|
||||
"top_k": ranking["top_k"]}
|
||||
result["ranking_variants"] = variants
|
||||
if memory_id is not None:
|
||||
result["f_first_used_turn"] = next(
|
||||
(t["turn"] for t in result["trace"] if (t["f_use_count"] or 0) > 0), None)
|
||||
result["f_last_use_increase_turn"] = max(
|
||||
(b["turn"] for a, b in zip(result["trace"], result["trace"][1:])
|
||||
if (b["f_use_count"] or 0) > (a["f_use_count"] or 0)), default=None)
|
||||
|
||||
evicted_turn = next((t["turn"] for t in result["trace"] if t["f_forgotten"]), None)
|
||||
first_evictions = next((t["evicted"] for t in result["trace"] if t["evicted"]), [])
|
||||
result["eviction"] = {
|
||||
"capacity": scenario.capacity,
|
||||
"f_evicted_at_turn": evicted_turn,
|
||||
"f_use_count_when_evicted": next(
|
||||
(t["f_use_count"] for t in result["trace"] if t["f_forgotten"]), None),
|
||||
"first_eviction_turn": next(
|
||||
(t["turn"] for t in result["trace"] if t["evicted"]), None),
|
||||
"first_evicted_ids": first_evictions,
|
||||
"f_memory_was_first_evicted": bool(f_memory_id and f_memory_id in first_evictions),
|
||||
"created_and_evicted_same_turn": sorted(
|
||||
{i for t in result["trace"] for i in t["created_and_evicted_same_turn"]}),
|
||||
"pinned_memory_id": pinned_id,
|
||||
"pinned_memory_forgotten": bool(pinned_id and known[pinned_id]["forgotten"]),
|
||||
}
|
||||
|
||||
if scenario.lineage_control:
|
||||
path_clause = lineage.path_of(db, adventure).clause(models.Memory)
|
||||
stored = db.execute(select(models.Memory.id).where(
|
||||
models.Memory.id.in_(g.get("memory_ids") or [-1]))).scalars().all()
|
||||
eligible = db.execute(select(models.Memory.id).where(
|
||||
models.Memory.id.in_(g.get("memory_ids") or [-1]), path_clause)).scalars().all()
|
||||
used_after = set()
|
||||
injected_text = False
|
||||
# Only turns played after the divergence. Before it, G was on the
|
||||
# active line, and a memory of it being used then is correct.
|
||||
for action in (db.query(models.Action)
|
||||
.filter(models.Action.adventure_id == adv,
|
||||
models.Action.type == "ai",
|
||||
models.Action.id > g["last_action_id_before_divergence"])
|
||||
.options(undefer(models.Action.context_snapshot))):
|
||||
snap = action.context_snapshot or {}
|
||||
for m in (snap.get("memories") or {}).get("used") or []:
|
||||
if m.get("id") in (g.get("memory_ids") or []):
|
||||
used_after.add(action.id)
|
||||
if FACT_G.mentioned_by(_section(snap, MEMORIES_LABEL)):
|
||||
injected_text = True
|
||||
g.update(stored=stored, eligible_on_active_line=eligible,
|
||||
turns_whose_memories_used_named_g=sorted(used_after),
|
||||
g_text_ever_in_used_memories=injected_text)
|
||||
if scenario.lineage_control:
|
||||
call("POST", f"/checkpoints/{g['line_a']}/restore")
|
||||
with SessionLocal() as db:
|
||||
adventure = db.get(models.Adventure, adv)
|
||||
eligible = db.execute(select(models.Memory.id).where(
|
||||
models.Memory.id.in_(g.get("memory_ids") or [-1]),
|
||||
lineage.path_of(db, adventure).clause(models.Memory))).scalars().all()
|
||||
g["eligible_after_returning_to_line_a"] = eligible
|
||||
result["lineage_control"] = g
|
||||
return result
|
||||
finally:
|
||||
for obj, name, value in saved:
|
||||
setattr(obj, name, value)
|
||||
app.dependency_overrides.clear()
|
||||
adventure_routes.turns._active_turns.clear()
|
||||
memorybank._vector_cache.clear()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
Reference in New Issue
Block a user