Add Phase 0B local validation findings and recommendation
Validates the three finalists by clone, build, test run and live local Ollama inference, then answers the fork question with measurements rather than static review. Recommendation: fork AI-DnD, confidence high. The Phase 0A call holds, but it was wrong that AI-DnD's undo is non-destructive — retry preserves the replaced take, undo hard-deletes it. A follow-up spike fixed that in 3 files (+130/-31): undo now moves a head cursor, redo round-trips, writing below a moved-back head forks and keeps the abandoned line, branch-scoped memory isolation survives, suite 627/632 with all 5 failures asserting the deleted-row behaviour that was replaced. Findings that change the plan: - AI-DnD cannot take a turn air-gapped as shipped; tiktoken fetches its encoding from a CDN. Proven on an internal Docker network, proven fixed by vendoring the file. - ai-adventure needs zero code for Ollama — two config lines — and its turn/head/checkpoint schema is the target model to build to. - Open Dungeon has zero automated tests and a positional summary watermark, making its branch retrofit larger than Phase 0A costed. - The world-state referee takes relative deltas; a 3B model sent absolute values under full context, so a wounded player ended at full health. Validation cannot catch this, so prefer ai-adventure's typed-event vocabulary when generalising narrative state. - Export/import recomputes head depth, so a round-trip silently undoes an undo. Must be fixed alongside the undo work. Docs only; no production code. Working tree from the runs stays untracked under phase0b/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015gUPLuxLs8wypxZPEmccJu
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 5
Claude Opus 5
parent
f011362494
commit
ba737de9b4
@@ -0,0 +1,164 @@
|
||||
# Phase 0B — Follow-Up Checks
|
||||
|
||||
Four open questions settled after the undo/redo spike. Ordered by how much each
|
||||
changes the plan.
|
||||
|
||||
---
|
||||
|
||||
## 1. The world-state referee under a local model
|
||||
|
||||
**Why it mattered:** the referee is the closest thing AI-DnD has to
|
||||
specification §2's "the model proposes, the application owns authoritative
|
||||
state", and §7's structured narrative state. It was never exercised in the main
|
||||
round, because the RPG path was deliberately left empty to prove genre-neutrality.
|
||||
If it only works with a cloud-class model, the principle the architecture rests
|
||||
on is at risk locally.
|
||||
|
||||
**How it works.** `EMIT_RULE` asks for a fenced ` ```state ` block of **relative
|
||||
deltas**, with an example (`{"player.hp": -15, ...}`) and a recency reminder
|
||||
appended at the end of the prompt. `apply.py` computes `new = old + delta`, then
|
||||
clamps by `max_delta_per_turn`, then by `min`/`max`, and reports every change as
|
||||
applied / clamped / rejected with corrective text fed back to the model.
|
||||
|
||||
### Result A — in the full application, `qwen2.5:3b-instruct` sent absolute values
|
||||
|
||||
Playing the seeded "Bandit Camp (RPG world state)" scenario:
|
||||
|
||||
```
|
||||
turn 1 proposed: {"player.hp": 92, "npc.gwen.trust": 25,
|
||||
"player.outfit": "...arrow in shoulder", "npc.gwen.health": 90}
|
||||
applied : player.hp 100 -> 100 "did not move. It is already at its
|
||||
maximum of 100 ... at most 35 per turn"
|
||||
gwen.trust 20 -> 40 (clamped from +25 to +20)
|
||||
gwen.health 100 -> 100 (clamped)
|
||||
outfit updated correctly
|
||||
|
||||
turn 2 proposed: {"npc.gwen.trust": 30}
|
||||
applied : gwen.trust 40 -> 60 (clamped from +30 to +20)
|
||||
```
|
||||
|
||||
The model meant "hp is now 92". The engine read `+92`, clamped it to `+35`, then
|
||||
to the ceiling of 100. **The player took an arrow to the shoulder and finished
|
||||
the turn at full health**, with the prose describing a wound.
|
||||
|
||||
This is the failure mode that matters: it is not corruption, and no validator can
|
||||
catch it, because `+92` is a perfectly legal proposal. The application stayed
|
||||
authoritative — which is the point of the referee — but the authoritative state
|
||||
silently stopped tracking the narration.
|
||||
|
||||
The free-text stat (`player.outfit`) was handled correctly, because it is marked
|
||||
`(free text)` and takes a whole value rather than a delta.
|
||||
|
||||
### Result B — in isolation, the same model gets it right
|
||||
|
||||
An isolated probe using AI-DnD's own `EMIT_RULE` and `extract_delta` verbatim, so
|
||||
that only the model varies. A short stat guide, live values, one wound scene,
|
||||
3 samples each:
|
||||
|
||||
| Model | Emitted a parsable state block | Correct relative (negative) hp |
|
||||
|---|---|---|
|
||||
| `qwen2.5:0.5b` | 0/3 | 0/3 |
|
||||
| `qwen2.5:3b-instruct` | 3/3 | **3/3** (−20, −20, −15) |
|
||||
| `qwen2.5:7b-instruct` | 3/3 | **3/3** (−15, −30, −15) |
|
||||
|
||||
So the delta protocol is **not** beyond a 3B model. The difference between A and
|
||||
B is context: the full application prompt carries scenario prose, persona, a
|
||||
stat guide, live values, story cards and history, and under that load the 3B
|
||||
model degraded to absolute values. In a short prompt it followed the same
|
||||
instruction perfectly.
|
||||
|
||||
> Recorded because it nearly became a wrong finding: the first run of this probe
|
||||
> reported 0/3 for both models. That was a bug in the probe, not the models —
|
||||
> `extract_delta` returns `(clean_text, delta)` and the harness was reading
|
||||
> element 0. The table above is the corrected run.
|
||||
|
||||
Note also `qwen2.5:7b-instruct` sample 2 proposing `npc.gwen.trust: +75`, which
|
||||
the referee would clamp to +20. Even a capable model proposes disproportionate
|
||||
values, which is precisely what the clamp is for.
|
||||
|
||||
### What this means for the design
|
||||
|
||||
1. **Not a blocker for the fork.** The referee is per-scenario and opt-in, and we
|
||||
are generalizing it into narrative state rather than adopting it as-is.
|
||||
2. **Prefer an unambiguous state vocabulary.** A relative-delta protocol has a
|
||||
failure mode that validation cannot detect, and it degrades with context
|
||||
length. ai-adventure's typed events (`set_flag key=… value=…`) are explicit
|
||||
and absolute, so the same mistake is impossible to express. **This is a
|
||||
concrete argument for taking ai-adventure's event vocabulary rather than
|
||||
AI-DnD's delta vocabulary when generalizing to genre-neutral narrative state.**
|
||||
3. **Keep the emission rule close to generation.** AI-DnD already appends
|
||||
`EMIT_REMINDER` at the end of the prompt for exactly this reason; whatever we
|
||||
build should keep that property and be tested at realistic context length,
|
||||
not just in a clean harness.
|
||||
4. **Sample sizes are small** (3 per model; 2 parsed in-app turns). Enough to
|
||||
establish the failure mode exists and that it is context-dependent, not enough
|
||||
to rank models. A proper capability matrix belongs in the build phase.
|
||||
|
||||
---
|
||||
|
||||
## 2. Export/import silently redoes an undone story
|
||||
|
||||
**Status: a real defect the spike introduces into the export path. One-field fix.**
|
||||
|
||||
`bundle.py:_point_the_head` sets the head on import by recomputing it:
|
||||
|
||||
```python
|
||||
depths = [n["depth"] for n in story["nodes"] if n["branch"] == head]
|
||||
if depths:
|
||||
adventure.head_depth = max(depths)
|
||||
```
|
||||
|
||||
The export format carries `headBranch` (line 120) but **not** the head depth.
|
||||
|
||||
With the destructive undo this was correct — undone rows did not exist, so
|
||||
`max(depths)` was the head. With the spike's non-destructive undo, an adventure
|
||||
exported while its head sits behind the tip comes back with every abandoned turn
|
||||
restored. The undo is silently undone by a round-trip.
|
||||
|
||||
**Fix:** add `headDepth` to the bundle (optional field, or a `v3`), and honour it
|
||||
in `_point_the_head`, falling back to `max(depths)` when absent so that existing
|
||||
`ai-dnd-adventure-v2` files still import. Small, but it must land in the same
|
||||
change as the non-destructive undo, not after it.
|
||||
|
||||
---
|
||||
|
||||
## 3. Story cards are not lineage-scoped
|
||||
|
||||
**Status: a design constraint for imported knowledge, not a present defect.**
|
||||
|
||||
`models.StoryCard` is keyed by `scenario_id` **or** `adventure_id`, and carries
|
||||
**no `branch_id` and no `depth`** — unlike `Memory`, which carries both and is
|
||||
therefore filtered by the same capped lineage clause that gives memories their
|
||||
branch isolation.
|
||||
|
||||
`context.builder.match_cards` is pure keyword matching over the adventure's whole
|
||||
card list; `routers/story_cards.py` contains no lineage reference at all.
|
||||
|
||||
This is harmless today, because cards are authored by the user or copied from a
|
||||
scenario — nothing derives a card from story content, so an abandoned branch
|
||||
cannot invent one.
|
||||
|
||||
It stops being harmless the moment imported knowledge or any derived-card feature
|
||||
lands. **Implication for `IMPORTED-KNOWLEDGE-DESIGN.md`:** any knowledge source
|
||||
that can be created or updated from story content must carry `(branch_id, depth)`
|
||||
like `Memory` does, and be read through `lineage.Path.clause`. Do that and it
|
||||
inherits both the branch isolation and the undo isolation for free — which is the
|
||||
strongest argument for a new table rather than extending `story_cards`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Postgres/psycopg is cleanly removable
|
||||
|
||||
**Status: resolved, low effort.**
|
||||
|
||||
The entire Postgres surface is three places:
|
||||
|
||||
- `database.py` — the `AIDND_DATABASE_URL` / `DATABASE_URL` branch and
|
||||
`_normalize_url`. Deleting the branch leaves the SQLite path, which is already
|
||||
the default when neither variable is set.
|
||||
- `migrations.py` — two helpers that tolerate SQLite returning a raw JSON string
|
||||
where psycopg returns a parsed list. They simplify rather than disappear.
|
||||
- `analytics.py` — `from sqlalchemy.dialects.postgresql import insert as pg_insert`,
|
||||
which goes with analytics, already scheduled for removal.
|
||||
|
||||
`psycopg[binary]` then drops out of `requirements.txt`. No blockers.
|
||||
Reference in New Issue
Block a user