diff --git a/backend/app/memorybank.py b/backend/app/memorybank.py index 9f740e1..e66e6d3 100644 --- a/backend/app/memorybank.py +++ b/backend/app/memorybank.py @@ -63,6 +63,14 @@ MAX_CAST_MEMBERS = 8 CAST_ENTRY_CHARS = 240 SETTING_TOKENS = 300 # of `adventure.memory`, the plot essentials +# A ceiling on one memory, in words. Measured, not guessed: with only +# "1-2 plain sentences" to go on, a real model wrote 34 words for one block and +# 105 for the next, and a 105-word memory is a paragraph. Five of those are +# injected per turn at the default `memory_top_k`, so the bank's cost is set +# here. A stated number also holds the length steady between memories, which is +# the same consistency the framing rule buys for the wording. +MEMORY_MAX_WORDS = 50 + # The framing rule is the larger half of this prompt, and it is worth the # tokens. Without it the model chooses a person per call, so one bank ends up # holding "You entered the crypt", "The player entered the crypt" and "He @@ -77,7 +85,9 @@ SETTING_TOKENS = 300 # of `adventure.memory`, the plot essentials MEMORY_SYSTEM_PROMPT = ( "You compress interactive-fiction story excerpts into memories. Respond with " "1-2 plain sentences in past tense stating the concrete facts and events " - "(names, places, items, promises, injuries).\n\n" + f"(names, places, items, promises, injuries), in at most {MEMORY_MAX_WORDS} " + "words. Keep the details a later scene could turn on — a name, a promise, an " + "injury, where something is — and drop the ones it could not.\n\n" "Write in the third person. The excerpt is written in the second person: " '"you" is the protagonist, who is named in the Cast. Refer to the ' 'protagonist by that name, never as "you". If the Cast gives no name for ' diff --git a/plan/18-persona-and-memory-quality.md b/plan/18-persona-and-memory-quality.md index d3025be..216b9ce 100644 --- a/plan/18-persona-and-memory-quality.md +++ b/plan/18-persona-and-memory-quality.md @@ -5,9 +5,8 @@ along with the cast, to fix the memories. Phase 1 is worth shipping on its own; Phase 2 depends on it and is much smaller once it lands. **Both phases are built and green (610 backend tests). Phase 1 was driven in a -browser (21/21 checks). Phase 2's prompts were verified against the real seeded -scenario — see "Phase 2 as built" — but no memory has been generated by a real -model yet, because that needs an API key.** +browser (21/21 checks). Phase 2 was run end to end against a real model, as a +controlled A/B on one story — see "Run with a real model".** **Last updated: 2026-08-31.** @@ -459,15 +458,65 @@ person and named. Embeddings handle paraphrase well, so this is speculative. If retrieval quality visibly dips, prepending the same brief to the query text is the first thing to try. -## Still to check with a real model +## Run with a real model, and what it changed -Everything above is the prompt, not the output. Nobody has yet run a real -provider over it and read the memories that come back. That needs an API key, -and it is the only way to know whether the framing rule actually holds across a -whole adventure. What to look for: +Run end to end against a real model, using the `claude` CLI as the provider: one +story generated through the app's own `build_context` a turn at a time, then +**both** memory prompts run over the **same** blocks, so the story is held +constant and the prompt is the only variable. Every call is a fresh process, so +no arm can see the other's output and the model is never told what is being +tested. The control is the exact prompt from commit `9cdcb55`, not a paraphrase. -1. Every new memory names the protagonist and uses no bare pronouns. -2. The framing is the same across memories written many turns apart. -3. The rewritten story summary inherits it. -4. Memories do not get noticeably longer — the brief is context, not content to - be repeated back. +### The reported fault reproduced, and the fix held + +| | words | framing | +|---|---|---| +| memory 1, before | 34 | second person — "**You** crept low through the mist…" | +| memory 2, before | 105 | third person — "**The player** asked Gwen to…" | +| memory 1, after | 72 | "**Kaelen** and Gwen crouched at dawn…" | +| memory 2, after | 68 | "**Kaelen** and Gwen infiltrated the bandit camp…" | + +Two consecutive memories in one bank, written from the same story minutes apart, +in two different persons. That is the complaint, reproduced under controlled +conditions rather than argued from the prompt text. + +The control also got a fact wrong that the treatment did not. The player's move +was `grab her wrist and pull her down behind the woodpile`; the before-memory +recorded "The player asked Gwen to grab her wrist and pull her down behind the +woodpile", inverting who acted. One sample, so this is an observation rather +than a claim — but it is the failure mode the brief predicts, since without a +cast there is no way to tell whose wrist "her wrist" is. + +### It also found a real problem, which is now fixed + +"1-2 plain sentences" is not a length. The same model wrote 34 words for one +block and 105 for the next. A 105-word memory is a paragraph, and +`memory_top_k` injects five of them every turn, so the bank's running cost is +set by a number nobody had ever stated. + +`MEMORY_MAX_WORDS = 50` now states it, and the prompt says which details to keep +when trimming: the ones a later scene could turn on. Re-run over the identical +story, the same two blocks came back at **32 and 58 words** (mean 45, down from +70), still naming Kaelen, still third person, and still carrying every +load-bearing fact — the camp map, the strongbox behind the second tent, the +strap frayed near through. Overshooting 50 slightly is expected: models exceed +word budgets, which is why `builder.length_hint` carries a `LENGTH_BUFFER` for +the same reason. + +The variance is the real gain. Before: 34 to 105. After: 32 to 58. + +### What this does not show + +The model behind the run is a Claude model. The app talks to an +OpenAI-compatible endpoint, and the parser in `worldstate/parse.py` exists +because weaker free models emit trailing commas and leading `+`. So this shows +the prompt is followable and that the brief supplies the missing information. +It does not show that a weaker production model complies as well. An explicit +framing rule is usually worth *more* on a weaker model, but that is an +expectation, not a measurement. + +Two side observations from the same run, both pre-existing behavior working +correctly: the model sent `{"milestones.strongbox_found": false}`, `apply_delta` +refused it ("a milestone is sticky, so only true is accepted"), the refusal +reached the model through `render_refusals`, and the next turn sent `true`. The +Phase 1 persona also held across all six generated turns.