Files

T

JesseMarkowitzandClaude Opus 5 a6e9c7a32b M6: branch-safe context, summaries and long-term story memory

Aligns the inherited AI-DnD memory and context foundation with the history,
authority and state model M3-M5 established. Long stories now reach the narrator
through a bounded, lineage-safe, inspectable context rather than a growing
transcript.

This commit includes the corrective work that followed the independent review in
planning/reports/M6-IMPLEMENTATION-REPORT.md. The first implementation reported
E03 as passing and it was not; the report records that history rather than
hiding it.

What was already correct, and was kept rather than rebuilt

  Memory lineage. Memories already carried (branch_id, depth) and retrieval
  already filtered through the capped-path clause; the ten-step negative control
  was measured passing against b7005e6 before any change here. M6 adds the
  regression tests that pin it, plus provenance and authority on the result.

Summary lineage — both halves

  A summary is a row carrying the coordinate of the last node it covers, and
  eligibility is the same head-capped lineage clause memories use. That alone
  was not enough: generation was seeded from adventures.story_summary, a
  campaign-global column with no lineage, so after a divergence the summariser
  was handed the abandoned line's prose and asked to update it. The row it
  produced was correctly anchored and therefore looked safe while its sentences
  described a story the reader had left.

  Generation is now seeded from summaries.current — the same question the
  context builder asks — so the input and the output are scoped by one rule.
  adventures.story_summary remains a reader-facing mirror for the Plot panel and
  the export bundle, kept in step when a summary is written and when the head
  moves, and nothing authoritative reads it.

Retrieval redundancy

  With a real embedding model, four near-identical memories crowded out the one
  distinctive clue, which survived only because the default memory_top_k is 5.
  Retrieval now drops a candidate that repeats one already chosen, never across
  authority classes, at a threshold measured against the configured embedding
  model. The clue is retrieved at top_k 5, 4 and 3. Ranking itself is unchanged;
  the further factors CONTEXT-AND-MEMORY §20 contemplates remain unimplemented
  and are recorded as such.

Memory authority, budgeting, observability

  Memory.authority is accepted_story or heuristic, classified by the application
  and marked in the prompt; retrieval never writes state. The reply is reserved
  out of the context budget, and an impossible configuration fails clearly
  instead of overflowing. Each derived pass records ok/idle/failed per campaign,
  served by GET /adventures/{id}/derived and shown in Insights, so the M2
  failure — a dead memory bank with a green suite — is visible if it recurs.
  Provider-wiring tests mock no factory.

Also: two pre-existing test-suite leaks fixed; two fixtures that stored one
vector in every memory now use distinct ones, so lineage assertions stay
readable alongside redundancy suppression.

Planning: CONTEXT-AND-MEMORY, TECHNICAL-DESIGN, DATA-MODEL, V1-ACCEPTANCE-TESTS,
BUILD-MILESTONES, VERSION and planning/README updated to describe what exists,
including that a valid E03 test must regenerate a summary after diverging. The
M5 report was rotated to planning/archive/milestone-reports/. No new ADR — every
choice implements a decision the package had already settled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PWU4gTfLYY6Qq9U7aa9Qw2

2026-09-06 03:00:33 -04:00

19 KiB

Raw Blame History

Planning Package Version

Package: Adventure Storyteller Planning Package v2.7
Revision date: 2026-09-06
Status: Phase 0 complete; architecture selected; Milestones M1-M6 implemented and accepted; M7 is next to brief.

v2.7 — M5 and M6 Closeout (2026-09-06)

Two milestones, and one lesson they share: an independent review found a real defect in each after the implementation reported success, and in both cases the defect was invisible to the tests the implementation had written for itself.

M5 — Genre-Neutral Authoritative Narrative State. Accepted 2026-09-04. Its review returned PASS WITH CORRECTIVE WORK REQUIRED: editing a narrator turn rewound the campaign's live state while the head stayed at the tip, breaking the transcript position == head == authoritative state invariant. The corrective pass rebuilt narrator editing on the §§14-15 fork semantics. Recorded here late: M5's closeout did not add a package revision entry, and this one covers it.

M6 — Branch-Safe Context, Summaries, and Long-Term Story Memory. Accepted 2026-09-06. Sequence:

Implementation. Summaries moved onto lineage-anchored rows; memory authority, provenance, an output reserve, and observable derived-work failure were added. Reported as passing, including E03.
Independent review — E03 still failed. Summary rows were anchored, but generation was seeded from adventures.story_summary, a campaign-global column with no lineage. After a divergence the summariser was handed the abandoned line's prose and asked to update it, so the new summary carried abandoned content inside a correctly anchored row. The review also found F02 passing by a single retrieval slot, two test defects, and a misleading derived-work status.
Corrective pass. Generation is seeded from summaries.current; redundancy suppression was added before the retrieval cut; the test fixture no longer makes real network calls; the browser suite gained a scenario that regenerates a summary after divergence.

What the planning package learned from M6, recorded in the active documents:

CONTEXT-AND-MEMORY.md §11 — lineage safety has two halves. Anchoring the output row is not enough; the input to the summarizer must be scoped by the same rule.
V1-ACCEPTANCE-TESTS.md E03 — a valid test must regenerate a summary after diverging. Checking only that the old row went ineligible passes while the defect is live.
CONTEXT-AND-MEMORY.md §20/§22 — what M6 implements (redundancy suppression between memories) is now distinguished from what it does not (importance, entity, recency and thread ranking; cross-layer duplication).

v2.6 — M4 Closeout (2026-09-03)

M4 is accepted. Its review returned PASS WITH CORRECTIVE WORK REQUIRED; the corrective work is done, and the browser condition that M3 and M4 both carried is closed.

The three review findings, fixed:

B-1 — the Save Point list was an N+1. It resolved each Save Point with its own query and loaded whole Action rows, narration included, to answer "does a row exist here". It is now one bulk two-column coordinate query plus one lineage computation: 53 SELECTs for 25 Save Points became 5, and the count no longer grows with the list. Guarded by three tests, including one proving the coordinate is matched as a pair — an IN-list version would report a Save Point resolved because another branch has a live row at the same depth.
B-2 — deleting a branch silently deleted its Save Points. Fixed as a behaviour defect rather than a missing warning, because STORY-BRANCH-SEMANTICS.md §19 says a named checkpoint remains until explicitly deleted and §28 already required future cleanup to retain checkpoint-referenced paths. A branch a Save Point names can no longer be deleted. The request is refused with the offending Save Points named; the user deletes them explicitly, which deletes no story, and the branch then goes. Both delete controls disable and explain. Recorded as a new §19.1.
B-3 — the D11/L03 automation never left one process. A new module spawns real server processes, kills the first, and reads the campaign back with the second.

Also fixed (review §S C-5): creating a Save Point now takes the campaign's turn lock, so "save where I am" cannot read a head a turn in flight is about to move. Rename and Delete deliberately do not take it, and a test pins that decision.

Real-browser verification — the first in this project. A Firefox 154.0.1 driven through geckodriver over the W3C WebDriver protocol exercised the rendered DOM for both milestones: 44/44 checks passed, no console errors. It covered M3's Undo/Redo enable states, transcript movement, Retry and the take pager, and divergence retiring Redo; and M4's whole Save Point lifecycle including both confirmations and the new branch-delete warning. The outstanding M3 browser condition is therefore closed as well. No dependency was added to the repository: the WebDriver client for the run was written against stdlib HTTP.

Also corrected, found while fixing B-2: models.py, TECHNICAL-DESIGN.md §8.8 and DATA-MODEL.md §8 all described the cascade as the durability rule. They now describe the refusal, and record that checkpoints.branch_id's cascade survives as referential integrity that the application no longer reaches.

Documents corrected by this closeout:

V1-ACCEPTANCE-TESTS.md records results for D11-D14, I04, L03 and the E-series, and states that the browser-level condition is satisfied. No pass condition was weakened — and D11/L03 now note that the automation crosses a genuine OS process boundary, which is the standard later milestones should meet.
DATA-MODEL.md §8 records the coordinate as implemented, with the retry measurement that settles coordinate-versus-turn-id, and three decisions that were previously implicit: names are not unique, several Save Points may name one position, and the list is newest-created first.
STORY-BRANCH-SEMANTICS.md gains §19.1 — a checkpoint protects the history it names. This is the only behavioural specification change in the closeout, and it strengthens §19 rather than weakening anything.
BROWSER-UX-SPEC.md §25 rules for the implemented vocabulary: Moment N, not Turn N, because the branch panel and tree overlay already count in moments. Vocabulary only; no behaviour changes.
BUILD-MILESTONES.md marks M4 COMPLETE, records the fixes and the browser result, and warns M5 that the instrumentation to move is now 55 tests.
README.md records M1-M4 accepted and M5 as next to brief.
PROJECT-SOURCES.md and project-sources.txt point at the current report; project-sources.txt still named the archived M3 report and was corrected.

Report rotation happened in the reporting pass that preceded this closeout: M3-IMPLEMENTATION-REPORT.md moved to archive/milestone-reports/ as a pure rename, and reports/ now holds M4's report, whose §W is this closeout's evidence record.

No new ADR. ADR 012 already decides the architecture, and the corrective work forced no new architectural decision. SPECIFICATION.md, STORY-BRANCH-SEMANTICS.md, SECURITY-THREAT-MODEL.md, CONTEXT-AND-MEMORY.md, IMPORTED-KNOWLEDGE-DESIGN.md and ADRs 003, 005 and 012 are unchanged.

M5 readiness: ready. Save Points store no state and no checkpoint code reads any, so M5 can change what a snapshot contains without touching what a Save Point is — provided it keeps state recoverable at a position without replay (TECHNICAL-DESIGN.md §10.4).

v2.5 — M4 Implementation (2026-09-03)

M4 added durable named Save Points. This revision records only what the implementation established as fact; no product requirement changed, and the milestone is not marked accepted — its review has not been written.

TECHNICAL-DESIGN.md gains §8.8 and §9.2: the Save Point as a name plus a coordinate holding no story, restore as head movement with a bounds check, the rule that the branch half of the head moves only when the coordinate is off the path being read, restore never forking, and the bundle carrying Save Points independently of the head.
DATA-MODEL.md §8 records the pointer as implemented — (branch, depth) rather than a turn id, with the reason: one coordinate holds every attempt at a turn and exactly one is live, so a coordinate follows a retry where a row id would pin a superseded take. §29 records the checkpoints in the export.
BUILD-MILESTONES.md M4 gains a status block: what shipped, the one architectural decision the milestone had to make and why it needed no new ADR, the test count, and the outstanding browser condition.
README.md describes Save Points as a user-facing capability.

No ADR was created. ADR 012 already decides the architecture M4 needed — restore reuses active-head movement — and a table is not a decision. The one question ADR 012 does not answer, whether restore moves the branch half of the head, is that same mechanism applied to a coordinate ADR 012 already defines; TECHNICAL-DESIGN.md §8.8 records the answer rather than a new ADR asserting it.

SPECIFICATION.md, SECURITY-THREAT-MODEL.md, STORY-BRANCH-SEMANTICS.md and V1-ACCEPTANCE-TESTS.md are unchanged. M4 altered no product requirement, added no outbound path, and implemented the checkpoint semantics STORY-BRANCH-SEMANTICS.md §18-25 already specified rather than amending them.

The browser smoke test remains unperformed, now for both M3 and M4. No session has had a usable browser. See BUILD-MILESTONES.md M3 and M4.

v2.4 — Documentation Consolidation (2026-09-03)

No product requirement, architecture decision or milestone status changed in this revision. It reorganises the documentation so that a new coding agent can tell authoritative material from evidence at a glance.

planning/archive/ is created and is non-authoritative by declaration (archive/README.md). It holds phase0/ — the research that chose AI-DnD — milestone-reports/ — the completed M1 and M2 reports — and decisions/, which now holds ADR 008, the Phase-0-before-build process gate that Phase 0 satisfied. ADR numbering continues from 012; 008 is not reused.
planning/reports/ now holds only the current milestone's report, M3-IMPLEMENTATION-REPORT.md, because M4 planning has to consult it. It moves to the archive when M4's report replaces it.
The Phase 0B execution prompts and handoff/status/summary documents (CODEX-HANDOFF-NOTE.md, PHASE-0B-CODEX-BRIEF.md, PHASE-0B-CODEX-HANDOFF.md, PLANNING-UPDATE-SUMMARY.md) and the Phase 0A discovery and triage reports were deleted: intermediate working documents whose conclusions all reached the two recommendation reports, and which remain in Git history.
Upstream AI-DnD's inherited plan/ build log and docs/ project site (guides, generated HTML, screenshots) were deleted. They documented a hosted, scripted, multi-user product with accounts — every screenshot showed a Scripts tab and a Sign up button — which M2 removed. Both trees remain in Git history and in upstream.
README.md, DEVELOPMENT.md and PROVENANCE.md are corrected where they pointed at the removed trees or described removed capability as present. DEVELOPMENT.md's "things M1 did not touch" section had gone stale at M2 and now says what is actually still inherited.
planning/README.md is rewritten as the documentation index: current milestone, the three-way active/ADR/archive split, the authority order, reading order, where reports live, and what comes next. The milestone correction tables are preserved unchanged.
New: planning/PROJECT-SOURCES.md and planning/project-sources.txt, the manifest of files to upload as ChatGPT Project Sources.

v2.3 — Post-M3 Closeout (2026-09-03)

M3 replaced destructive Undo with a stored active head. Its review is archive/milestone-reports/M3-IMPLEMENTATION-REPORT.md, which is also M3's primary evidence record — no separate baseline report was produced — and whose §W records this closeout.

In summary:

the architecture is recorded as ADR 012 — Active-Head Non-Destructive History: the head is stored rather than derived, every read of the story is capped at it in one place, one mechanism moves it, state comes from the node rather than from a replay, the first write below a moved-back head is the divergence, and Redo is decided by the lineage rather than by a flag. ADR 005 is unchanged: it states the product requirement, and ADR 012 states the architecture chosen to implement it,
two history semantics are ratified in STORY-BRANCH-SEMANTICS.md: Undo crosses fork points to the campaign opening (§5), and the system refuses to switch which take is live while a later story is off screen (§10),
a new §14A records the interim in-place-editing rule — refuse when story descends from the turn and is not on screen — and states explicitly that §14-15's full narrator-edit requirement stands and is completed in M5,
TECHNICAL-DESIGN.md gains §8.7 and §9.1 recording the implemented model and bundle behaviour as fact, and a constraint on §10.4: the snapshot half of the hybrid state model is a requirement, because head movement must not become proportional to campaign length,
DATA-MODEL.md records the head as campaign-stored, the branch disposition as implemented and deliberately advisory, and the export as carrying a chosen position rather than a derived one,
BUILD-MILESTONES.md marks M3 complete, states the one outstanding condition (the browser smoke test), tells M4 to reuse M3's head movement rather than build a second restore path, and gives M5 three constraints,
V1-ACCEPTANCE-TESTS.md records D03's full pass, states D10's milestone ownership without weakening any pass condition, adds I07's pre-M3 bundle clause, and resolves the apparent L01/A05 conflict,
README.md is corrected to describe the current local-only single-user application rather than the upstream hosted one.

SPECIFICATION.md and SECURITY-THREAT-MODEL.md are unchanged: M3 altered no product requirement and touched no path in the threat model.

v2.2 — Post-M2 Closeout (2026-09-03)

M2 removed the hosted, cloud, account and scripting surface and added the inference endpoint policy. Its review recommended six planning changes and reported rather than applied them; all six are applied in this revision, listed in README.md § Post-M2 corrections applied, with the evidence in archive/milestone-reports/M2-BASELINE-REPORT.md and archive/milestone-reports/M2-IMPLEMENTATION-REPORT.md.

In summary:

the inference endpoint policy is recorded as implemented — an address allowlist of explicit local-network CIDRs, every resolved address checked, enforced when settings are saved and again before every outbound request, with TLS verification never traded against it (new ADR 011, SECURITY-THREAT-MODEL.md §10A),
its two residual limits are stated rather than mitigated: a hostile host already on the trusted LAN, and the DNS-rebinding interval between the policy's resolution and the client's connection,
TECHNICAL-DESIGN.md §5.1 items 3 and 4 are resolved, and a new §5.2 records the M1/M2 production architecture as fact,
a wiring rule is added (§18.1): removing a setting requires testing a real consumer path, and adding one requires proving it reaches its component — M2 shipped two defects behind a 604-test green suite because the tests at that boundary were mocks,
BUILD-MILESTONES.md records M2 complete, warns M5 that eight rollback tests use the world-state engine as instrumentation rather than as architecture, and requires M6 to make background memory failure observable,
the security acceptance contract is strengthened: H10 now names the wildcard-origin and /api 404 conditions, and new H12 covers inference endpoint enforcement including the database-edited-behind-the-API case.

SPECIFICATION.md is unchanged: M2 altered no product requirement. Nothing in the architecture selected in v2 was reversed.

v2.1 — Post-M1 Corrections (2026-09-02)

M1 implementation evidence contradicted or under-specified parts of v2. The corrections are recorded in the documents themselves and listed in README.md § Post-M1 corrections applied; the evidence behind them is in archive/milestone-reports/M1-BASELINE-REPORT.md and archive/milestone-reports/M1-IMPLEMENTATION-REPORT.md.

In summary:

a trusted-LAN Ollama may be HTTPS with a privately issued certificate; clients verify against the operating system's CA store, with full certificate and hostname verification and no bypass option (ADR 002, TECHNICAL-DESIGN.md §5, A06),
offline claims require a fresh cache and no route out to be evidence at all, and vendored runtime artifacts should be integrity-verifiable (ADR 004),
A05's invariant is about accepted history; the user's submitted text is deliberately retained on a failed turn,
A06 requires a real second machine and an HTTPS endpoint; a plain-HTTP LAN test is no longer sufficient evidence,
the standard test environment records CPU/GPU/RAM, because cold model load on a CPU-only host exceeded the inherited 120 s client timeout,
BUILD-MILESTONES.md records M1 as complete and reframes M2's endpoint work as narrowing an existing capability rather than inventing it,
SECURITY-THREAT-MODEL.md §53 distinguishes inbound TLS (still deferred) from outbound certificate verification (required, done in M1).

Nothing in the architecture selected in v2 was reversed.

v2 — Post Phase 0B Revision (2026-09-01)

This v2 package supersedes the earlier planning package produced before the final Phase 0B review and the trusted-LAN Ollama deployment clarification.

Key v2 changes include:

AI-DnD selected as the production base at the pinned Phase 0B commit.
Non-destructive head-cursor Undo/Redo design selected.
Explicit typed/absolute narrative-state events selected for production state handling.
The authoritative narrative-state document — its shape, its authority/provenance fields, and the event/document/snapshot split — recorded in ADR 013 after M5 implemented it.
Imported knowledge separated from AI-DnD Story Cards.
Trusted-LAN Ollama inference supported in v1 while the storyteller UI/API remains loopback-bound by default.
Offline first-use dependencies and runtime remote assets identified as M1 hardening work.
Production implementation divided into milestones M1-M11.

Historical Phase 0 prompts/reports are retained as evidence and should not be treated as current implementation instructions unless a current milestone prompt explicitly refers to them.

19 KiB Raw Blame History

Planning Package Version

v2.7 — M5 and M6 Closeout (2026-09-06)

v2.6 — M4 Closeout (2026-09-03)

v2.5 — M4 Implementation (2026-09-03)

v2.4 — Documentation Consolidation (2026-09-03)

v2.3 — Post-M3 Closeout (2026-09-03)

v2.2 — Post-M2 Closeout (2026-09-03)

v2.1 — Post-M1 Corrections (2026-09-02)

v2 — Post Phase 0B Revision (2026-09-01)

19 KiB

Raw Blame History