JesseMarkowitzandClaude Opus 5
ba737de9b4
Validates the three finalists by clone, build, test run and live local Ollama inference, then answers the fork question with measurements rather than static review. Recommendation: fork AI-DnD, confidence high. The Phase 0A call holds, but it was wrong that AI-DnD's undo is non-destructive — retry preserves the replaced take, undo hard-deletes it. A follow-up spike fixed that in 3 files (+130/-31): undo now moves a head cursor, redo round-trips, writing below a moved-back head forks and keeps the abandoned line, branch-scoped memory isolation survives, suite 627/632 with all 5 failures asserting the deleted-row behaviour that was replaced. Findings that change the plan: - AI-DnD cannot take a turn air-gapped as shipped; tiktoken fetches its encoding from a CDN. Proven on an internal Docker network, proven fixed by vendoring the file. - ai-adventure needs zero code for Ollama — two config lines — and its turn/head/checkpoint schema is the target model to build to. - Open Dungeon has zero automated tests and a positional summary watermark, making its branch retrofit larger than Phase 0A costed. - The world-state referee takes relative deltas; a 3B model sent absolute values under full context, so a wounded player ended at full health. Validation cannot catch this, so prefer ai-adventure's typed-event vocabulary when generalising narrative state. - Export/import recomputes head depth, so a round-trip silently undoes an undo. Must be fixed alongside the undo work. Docs only; no production code. Working tree from the runs stays untracked under phase0b/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015gUPLuxLs8wypxZPEmccJu
165 lines
7.6 KiB
Markdown
165 lines
7.6 KiB
Markdown
# Phase 0B — Follow-Up Checks
|
||
|
||
Four open questions settled after the undo/redo spike. Ordered by how much each
|
||
changes the plan.
|
||
|
||
---
|
||
|
||
## 1. The world-state referee under a local model
|
||
|
||
**Why it mattered:** the referee is the closest thing AI-DnD has to
|
||
specification §2's "the model proposes, the application owns authoritative
|
||
state", and §7's structured narrative state. It was never exercised in the main
|
||
round, because the RPG path was deliberately left empty to prove genre-neutrality.
|
||
If it only works with a cloud-class model, the principle the architecture rests
|
||
on is at risk locally.
|
||
|
||
**How it works.** `EMIT_RULE` asks for a fenced ` ```state ` block of **relative
|
||
deltas**, with an example (`{"player.hp": -15, ...}`) and a recency reminder
|
||
appended at the end of the prompt. `apply.py` computes `new = old + delta`, then
|
||
clamps by `max_delta_per_turn`, then by `min`/`max`, and reports every change as
|
||
applied / clamped / rejected with corrective text fed back to the model.
|
||
|
||
### Result A — in the full application, `qwen2.5:3b-instruct` sent absolute values
|
||
|
||
Playing the seeded "Bandit Camp (RPG world state)" scenario:
|
||
|
||
```
|
||
turn 1 proposed: {"player.hp": 92, "npc.gwen.trust": 25,
|
||
"player.outfit": "...arrow in shoulder", "npc.gwen.health": 90}
|
||
applied : player.hp 100 -> 100 "did not move. It is already at its
|
||
maximum of 100 ... at most 35 per turn"
|
||
gwen.trust 20 -> 40 (clamped from +25 to +20)
|
||
gwen.health 100 -> 100 (clamped)
|
||
outfit updated correctly
|
||
|
||
turn 2 proposed: {"npc.gwen.trust": 30}
|
||
applied : gwen.trust 40 -> 60 (clamped from +30 to +20)
|
||
```
|
||
|
||
The model meant "hp is now 92". The engine read `+92`, clamped it to `+35`, then
|
||
to the ceiling of 100. **The player took an arrow to the shoulder and finished
|
||
the turn at full health**, with the prose describing a wound.
|
||
|
||
This is the failure mode that matters: it is not corruption, and no validator can
|
||
catch it, because `+92` is a perfectly legal proposal. The application stayed
|
||
authoritative — which is the point of the referee — but the authoritative state
|
||
silently stopped tracking the narration.
|
||
|
||
The free-text stat (`player.outfit`) was handled correctly, because it is marked
|
||
`(free text)` and takes a whole value rather than a delta.
|
||
|
||
### Result B — in isolation, the same model gets it right
|
||
|
||
An isolated probe using AI-DnD's own `EMIT_RULE` and `extract_delta` verbatim, so
|
||
that only the model varies. A short stat guide, live values, one wound scene,
|
||
3 samples each:
|
||
|
||
| Model | Emitted a parsable state block | Correct relative (negative) hp |
|
||
|---|---|---|
|
||
| `qwen2.5:0.5b` | 0/3 | 0/3 |
|
||
| `qwen2.5:3b-instruct` | 3/3 | **3/3** (−20, −20, −15) |
|
||
| `qwen2.5:7b-instruct` | 3/3 | **3/3** (−15, −30, −15) |
|
||
|
||
So the delta protocol is **not** beyond a 3B model. The difference between A and
|
||
B is context: the full application prompt carries scenario prose, persona, a
|
||
stat guide, live values, story cards and history, and under that load the 3B
|
||
model degraded to absolute values. In a short prompt it followed the same
|
||
instruction perfectly.
|
||
|
||
> Recorded because it nearly became a wrong finding: the first run of this probe
|
||
> reported 0/3 for both models. That was a bug in the probe, not the models —
|
||
> `extract_delta` returns `(clean_text, delta)` and the harness was reading
|
||
> element 0. The table above is the corrected run.
|
||
|
||
Note also `qwen2.5:7b-instruct` sample 2 proposing `npc.gwen.trust: +75`, which
|
||
the referee would clamp to +20. Even a capable model proposes disproportionate
|
||
values, which is precisely what the clamp is for.
|
||
|
||
### What this means for the design
|
||
|
||
1. **Not a blocker for the fork.** The referee is per-scenario and opt-in, and we
|
||
are generalizing it into narrative state rather than adopting it as-is.
|
||
2. **Prefer an unambiguous state vocabulary.** A relative-delta protocol has a
|
||
failure mode that validation cannot detect, and it degrades with context
|
||
length. ai-adventure's typed events (`set_flag key=… value=…`) are explicit
|
||
and absolute, so the same mistake is impossible to express. **This is a
|
||
concrete argument for taking ai-adventure's event vocabulary rather than
|
||
AI-DnD's delta vocabulary when generalizing to genre-neutral narrative state.**
|
||
3. **Keep the emission rule close to generation.** AI-DnD already appends
|
||
`EMIT_REMINDER` at the end of the prompt for exactly this reason; whatever we
|
||
build should keep that property and be tested at realistic context length,
|
||
not just in a clean harness.
|
||
4. **Sample sizes are small** (3 per model; 2 parsed in-app turns). Enough to
|
||
establish the failure mode exists and that it is context-dependent, not enough
|
||
to rank models. A proper capability matrix belongs in the build phase.
|
||
|
||
---
|
||
|
||
## 2. Export/import silently redoes an undone story
|
||
|
||
**Status: a real defect the spike introduces into the export path. One-field fix.**
|
||
|
||
`bundle.py:_point_the_head` sets the head on import by recomputing it:
|
||
|
||
```python
|
||
depths = [n["depth"] for n in story["nodes"] if n["branch"] == head]
|
||
if depths:
|
||
adventure.head_depth = max(depths)
|
||
```
|
||
|
||
The export format carries `headBranch` (line 120) but **not** the head depth.
|
||
|
||
With the destructive undo this was correct — undone rows did not exist, so
|
||
`max(depths)` was the head. With the spike's non-destructive undo, an adventure
|
||
exported while its head sits behind the tip comes back with every abandoned turn
|
||
restored. The undo is silently undone by a round-trip.
|
||
|
||
**Fix:** add `headDepth` to the bundle (optional field, or a `v3`), and honour it
|
||
in `_point_the_head`, falling back to `max(depths)` when absent so that existing
|
||
`ai-dnd-adventure-v2` files still import. Small, but it must land in the same
|
||
change as the non-destructive undo, not after it.
|
||
|
||
---
|
||
|
||
## 3. Story cards are not lineage-scoped
|
||
|
||
**Status: a design constraint for imported knowledge, not a present defect.**
|
||
|
||
`models.StoryCard` is keyed by `scenario_id` **or** `adventure_id`, and carries
|
||
**no `branch_id` and no `depth`** — unlike `Memory`, which carries both and is
|
||
therefore filtered by the same capped lineage clause that gives memories their
|
||
branch isolation.
|
||
|
||
`context.builder.match_cards` is pure keyword matching over the adventure's whole
|
||
card list; `routers/story_cards.py` contains no lineage reference at all.
|
||
|
||
This is harmless today, because cards are authored by the user or copied from a
|
||
scenario — nothing derives a card from story content, so an abandoned branch
|
||
cannot invent one.
|
||
|
||
It stops being harmless the moment imported knowledge or any derived-card feature
|
||
lands. **Implication for `IMPORTED-KNOWLEDGE-DESIGN.md`:** any knowledge source
|
||
that can be created or updated from story content must carry `(branch_id, depth)`
|
||
like `Memory` does, and be read through `lineage.Path.clause`. Do that and it
|
||
inherits both the branch isolation and the undo isolation for free — which is the
|
||
strongest argument for a new table rather than extending `story_cards`.
|
||
|
||
---
|
||
|
||
## 4. Postgres/psycopg is cleanly removable
|
||
|
||
**Status: resolved, low effort.**
|
||
|
||
The entire Postgres surface is three places:
|
||
|
||
- `database.py` — the `AIDND_DATABASE_URL` / `DATABASE_URL` branch and
|
||
`_normalize_url`. Deleting the branch leaves the SQLite path, which is already
|
||
the default when neither variable is set.
|
||
- `migrations.py` — two helpers that tolerate SQLite returning a raw JSON string
|
||
where psycopg returns a parsed list. They simplify rather than disappear.
|
||
- `analytics.py` — `from sqlalchemy.dialects.postgresql import insert as pg_insert`,
|
||
which goes with analytics, already scheduled for removal.
|
||
|
||
`psycopg[binary]` then drops out of `requirements.txt`. No blockers.
|