71b24b622910bcf86dcf1f3c08f6ea71d760900b
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Claude
|
71b24b6229 |
Replicate the A/B on a fresh story, and narrow one claim
The checked-in harness was run end to end through the shim on a newly generated story, which is both a check that it works and an independent replication. The finding the change is for held: across two runs, none of the four control memories names the protagonist and all four treatment memories do. The length finding held too — controls at 34, 89, 105 and 107 words, a four-fold spread with no budget stated anywhere, against 54 and 55 under MEMORY_MAX_WORDS. One claim did not replicate, and the writeups now say so. Run 1 produced two control memories in two different persons, which is the reported complaint exactly, and plan/18 presented that as reproduced. Run 2's controls were both "the player", consistently. Drifting between second and third person is therefore something a model sometimes does, observed once, not something it does every time. The naming gap is the durable result and the writeups now lead with that instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NPyQN926gkZTAYfgugcaok |
||
|
Claude
|
712ef44f57 |
Keep the A/B, and the harness that produced it
The run that justified MEMORY_MAX_WORDS lived in a scratch directory and would have been gone with the container. The numbers in plan/18 were therefore assertions nobody could check. plan/18-appendix-memory-ab-run.md now carries the whole transcript: both memories, both summaries, and the thirteen-action story they were written from. backend/tools/memory_ab.py reproduces it. It replaces the throwaway script the first run used, and differs in two ways that matter. It goes through OpenAICompatibleProvider rather than calling a model directly, so a run exercises the provider, the streaming path and complete() instead of a stub. And it reads the control prompt out of git at the commit given to --before, so the thing being compared against cannot drift from what actually shipped. There was already a claude_shim.py serving an OpenAI-compatible endpoint backed by the CLI, which is exactly what the throwaway script had reinvented. memory_ab.py points at it by default, so a run spends a Claude subscription rather than API credit, and --endpoint aims it at the provider the deployed app really uses — which is the one question this whole exercise could not answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NPyQN926gkZTAYfgugcaok |
||
|
Claude
|
07192767d8 |
Give a memory a word ceiling, after measuring one
Ran the memory prompt end to end against a real model as a controlled A/B: one story generated through the app's own build_context a turn at a time, then both prompts run over the same blocks, so the story is held constant and the prompt is the only variable. Fresh process per call, so neither arm sees the other and the model is never told what is being tested. The control is the exact prompt from 9cdcb55. The reported fault reproduced. Two consecutive memories written from one story minutes apart came back in two different persons — "You crept low through the mist" and "The player asked Gwen to". With the cast brief both named Kaelen. The control also inverted who acted on a move whose player text was "grab her wrist and pull her down", which is the failure the brief predicts: with no cast there is nothing to say whose wrist "her wrist" was. It also found something the prompt review had not. "1-2 plain sentences" is not a length, and the same model wrote 34 words for one block and 105 for the next. A 105-word memory is a paragraph, and `memory_top_k` injects five every turn, so the bank's running cost was set by a number nobody had ever stated. MEMORY_MAX_WORDS states it, and the prompt now says which details to keep when trimming: the ones a later scene could turn on. Re-run over the identical story, the same two blocks came back at 32 and 58 words, still named, still third person, still carrying the camp map, the strongbox behind the second tent, and the strap frayed near through. Variance is the real gain — 34..105 became 32..58. Overshooting 50 slightly is expected. Models exceed word budgets, which is why builder.length_hint already carries a buffer for the same reason. What this does not show: the run used a Claude model, and the app talks to an OpenAI-compatible endpoint whose weaker models are why worldstate/parse.py tolerates trailing commas. The prompt is followable and the brief supplies the missing information; a weaker model is not proven to comply as well. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NPyQN926gkZTAYfgugcaok |
||
|
Claude
|
1b5e4c56cd |
Tell the summarizer who the characters are
`_create_due_memories` sent six actions of second-person prose and nothing else — no protagonist, no cast, no setting, and no instruction about what person to write in. So for `You push the door open. She grabs your arm.` the only honest memory was "You entered a room and she stopped you", which names nobody when it is retrieved forty turns later. The framing wandered too: with no rule, the model picked a person per call, and one bank ended up holding "You entered the crypt", "The player entered the crypt" and "He entered the crypt" for the same kind of event. Both prompts now carry a cast brief and a framing rule: third person, the protagonist by name, other characters named rather than left as bare pronouns. The rule states its reason, because a memory really is read in isolation much later and a model told why complies far more consistently than one handed a bare instruction. The cast comes from the story cards, not from `stat_schema`. Every schema NPC is already turned into a card at adventure creation, deduplicated against the hand-written ones by name, so the cards cover schema NPCs, an author's own cards, and an adventure with no RPG layer at all through one path instead of three. Keyword matching alone was not enough, and finding that out changed the design. Built that way first, the brief for "She grabs your arm" listed the protagonist and nobody else: the block that most needs a cast is exactly the one written in bare pronouns, and Gwen's trigger keys include "her" but the text says "she". So matched cards come first and the remaining slots are filled with the other character cards. Places and items are not topped up — an unmentioned tavern is not who "she" was — though a place that is mentioned still matches normally. The asymmetry with the turn prompt is deliberate: an untriggered card is wrong as lore and right in a roster, because the roster answers "who could these pronouns be" rather than "what is on stage". Fixed descriptions only, never live values. `Gwen: trust 40 (wary)` in the brief would make the same event summarized at two different times come out framed differently, which is the fault this removes. An adventure with no persona still gets the cast and the setting, and the model is told to write "the player". One with nothing to say sends byte for byte the prompt it sent before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NPyQN926gkZTAYfgugcaok |
||
|
Claude
|
dfb56d4569 |
Record that the persona was driven in a browser
Chromium against a fresh database with the demo scenarios seeded. No API key needed: Insights assembles the prompt without calling a model, so the checks run on the real assembled context rather than a stub. 21/21. The two request failures in the run were the sandbox — Google Fonts is blocked by the egress policy, and the analytics beacon is aborted on unload — not the app. The case worth naming: a blank adventure with no RPG layer produced sections `['narrator', 'persona', 'length_hint']`. That is what the persona was added for, and it is the one thing no unit test in this repo would have caught if the section had been gated behind `has_ws`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NPyQN926gkZTAYfgugcaok |
||
|
Claude
|
9ee052c51e |
Give the adventure a persona, so the protagonist has a name
The player had stats but no identity. `stat_schema.player` carried hp and
mana beside `npc.gwen.trust`, but where an NPC has a name and a
description the player had neither, so the block rendered as
`You: hp 100/100` and nothing in the prompt said who "you" was.
Three columns on `adventures`: name, pronouns, description. All
user-only, all optional, and an empty name means the app behaves exactly
as it did before — no backfill, no special case for an adventure that
predates the migration.
They are adventure columns rather than part of `stat_schema` for two
reasons. An adventure with no RPG layer still has a protagonist, and
that is the case this was added for. And `worldstate.schema._initials`
treats every dict inside a stat section as a stat definition, so a
persona placed there would be instantiated, rendered in the guide, and
handed an `initial` value as though it were one.
The paths do not change. `player.hp` stays `player.hp`; only the label
moves, to `Kaelen (player): hp 100/100`, the same way NPC lines already
print a display name beside the id. A path carrying the persona's name
would break the moment a player renamed their character, because
`_history_text` replays every past turn's stored delta into the prompt
and those blobs hold literal `player.hp` strings.
The section sits in the system block. Only the user can edit it, so it
never changes mid-story and stays inside the cached prefix. That is what
makes it free, and it is why the AI must not be able to move it — a
delta aimed at `persona.*` is already refused by `_resolve`, and there
is now a test holding that in place.
The modal that used to appear only for scenarios with `${Placeholder}`
tokens now always opens, and is where the character is named. Persona
and placeholders stay independent: a scenario asking for `${Name}` is
asking its own question. No scenario in the repo uses placeholders at
all, so the overlap is hypothetical.
Phase 2, which feeds the persona and the cast to the summarizer, is
written up in plan/18 and not started. That is where the memory-quality
problem actually gets fixed; this change is what gives it a name to use.
Not yet driven in a browser — plan/18 lists what to check by hand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NPyQN926gkZTAYfgugcaok
|