Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Keep the local Claude test rig, and record what it found

Browse Source 
`backend/tools/claude_shim.py` serves an OpenAI-compatible endpoint backed by
the local `claude` command line tool, so a demo can be played against a real
model with no API key. Each request spawns one `claude --print` process, which
suits the turn engine: the app assembles the whole prompt every turn and
expects a stateless endpoint.

Playing the Pokemon demo through it made all five of the world-state changes in
`plan/16` work, and the refusal loop ran end to end for the first time: turn 3
clamped to nothing, turn 4's assembled prompt carried the correction verbatim,
and the model's next delta was right. Two failures previously blamed on the
code were the demo model. One is a schema fault and is still open: Milo's three
Pokemon share one `active_hp` stat, so a switch leaves the newcomer at 0 HP.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PacdRuPXSkQQy4ZYdH32hF
This commit is contained in:
parththakkar106
2026-08-28 18:42:24 +05:30
co-authored by Claude Opus 5
parent 6cbf6d996f
commit ef4da7fda2
5 changed files with 470 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan/16-world-state-refusals.md
+92
View File
@@ -231,3 +231,95 @@ old cap made the reset unreachable in one turn.
**Not fixed:** the `initial == max` shape itself. It is now loud rather than
silent, and the wording removes the usual cause, but the shape is still there.
---
## Driven against Claude, 2026-08-28
The four turns above ran on the free demo model, which ignored two instructions
that were present and correct in the prompt. To separate model behavior from
code, the same demo was played again through `backend/tools/claude_shim.py`, an
OpenAI-compatible endpoint backed by the local `claude` command line tool. The
model was `sonnet`. See the README for how to run it.
Every one of the five changes worked:
| Turn | Chips |
|---|---|
| 1 | `turn +1`, `active pokemon → Wartortle`, `wartortle hp -8`, `milo active hp -46`, dashed `type advantage used refused — no such flag` |
| 2 | `turn +1`, `milo active hp -52`, `milo pokemon fainted +1`, `milo active pokemon → Onix`, `✓ graveler defeated` |
| 3 | `turn +1 (limited)`, dashed `milo active hp no change — at its limit` |
| 4 | `turn +1`, `milo active hp +32`, `✓ type advantage used` |
`graveler_defeated` fired for the first time in five playtests, `world.turn`
moved every turn, `pokemon_fainted` went 0 to 1 at the faint, and every HP number
was a signed change rather than a total. The two failures left open above were
the demo model, not the instructions.
**The refusal loop is verified end to end.** Turn 3 clamped to nothing. Turn 4's
assembled prompt, read from `GET /api/adventures/1/actions/9/context`, carried
the correction verbatim:
```
[Part of your last state block was not applied. Correct it in this turn's block:
- `npc.milo.active_hp` did not move. It is already at its minimum of 0 (it runs
from 0 to 98; it moves at most 98 per turn).]
```
The model's next delta was `{"world.turn": 1, "npc.milo.active_hp": 32,
"milestones.type_advantage_used": true}`. That is the loop the whole change
exists for, and it had never been observed running.
### The one bug that is still a schema fault
At the faint the model set `npc.milo.active_pokemon` to `Onix` and sent no
positive `active_hp` change, so Onix arrived on the field at 0 of 98 and every
later hit was refused. Sonnet followed the rest of the scenario closely, so this
is the schema rather than the model: `active_hp` is one stat with one `max` of
98, shared by Graveler at 98, Onix at 90, and Kabutops at 88. A switch has to
raise it, and nothing in the schema can raise it on the referee's side.
The fix is to give Milo's three Pokemon their own NPC entries, which also removes
the `initial == max` shape from this demo. Not done: the session's remaining time
went to the guest starter instead.
### Cost
About $0.04 a turn, billed against the Claude subscription rather than a card.
Roughly 13k of each request's 20k prompt tokens is the command line tool's own
overhead; the app's prompt is about 7k.
---
## What shipped alongside it, 2026-08-28
**A permanent local test rig.** `backend/tools/claude_shim.py` is now in the
repo. It finds the `claude` binary through `AIDND_CLAUDE_BIN`, then `PATH`, then
the per-user install, and takes `--port` and `--claude`.
**The demo says Pokemon in its title, and has a Pokeball for cover art.**
`05-league-championship.json` is now `[Demo] Pokemon League Championship: Round
One`. Renaming a seed exposed a fault: `seed.py` matches a seed to its row by
title, so a rename inserted a second public scenario and stranded the first,
which is the same failure this file records for "Road to the Champion". Seeds
now carry `previous_titles`, and `_find_renamed` lands the rename on the
existing row. The scenario kept its id, and no orphan appeared.
The cover art is a 3.2 kB PNG data URI. An SVG one was tried first and stored as
an empty `image_url`: `app/images.py` accepts raster formats only, on purpose,
because SVG can carry script and these bytes are served from the app's own
origin. `backend/tools/make_pokeball.py` draws the ball with `zlib` alone, so
regenerating it needs no image library.
**Every new guest gets the played adventure.** `app/starter.py` copies a shipped
export bundle into each new guest account, from the guest mint in
`routers/auth.py`. The bundle is this session's adventure trimmed to its first
two exchanges, which ends on the knockout and shows an applied change, a refused
one, and a milestone. It stops before the Onix bug above, and the state it
leaves has Onix at its own 90 HP so a guest can play on from it.
The row building that `POST /adventures/import` did inline moved into
`bundle.materialize`, which both callers now use. The limit and rate checks
stayed in the endpoint: the starter writes a file the server ships, so it has no
untrusted list to cap. `tests/test_starter_adventure.py` covers the file, the
copy, the chips, the playable end state, and the two failure paths.
plan/STATUS.md
+68 -3
View File
@@ -86,8 +86,12 @@ nothing still reads `variant_count`/`variant_index` before dropping them, and no
is the migration shape that rewrites toasted values, so it owes one `VACUUM FULL actions;`
on the direct (non-`-pooler`) endpoint afterwards.
Ahead of that, `plan/16-world-state-refusals.md` records a set of world-state fixes that
are merged but **never driven in a browser**. See that file for what to test.
Ahead of that, `plan/16-world-state-refusals.md` is closed. Its fixes were driven in a
browser on 2026-08-28, first on the demo model and then on Claude Sonnet through the
local shim, and all five changes work. One item in it is still open and is the next
world-state job: **Milo's three Pokemon share one `active_hp` stat**, so a switch leaves
the newcomer at 0 HP and every later hit is refused. Give them their own NPC entries,
which also removes the `initial == max` shape from that demo.
**SP9 and SP10 are both on `main`, despite what earlier notes here said.** The branches
`sp7b-take-pager` and `sp10-memory-bank-eviction` still exist and still read as unmerged
@@ -206,6 +210,47 @@ an automated version; see SP7's entry in `plan/14`.
---
## What happened on 2026-08-28 — world state against a real model
**The refusal loop ran end to end for the first time.** `plan/16` shipped a correction
that is injected into the next turn's prompt when the engine clamps or rejects a change,
and nothing had ever observed it working. Turn 3 of the Pokemon demo clamped to nothing,
turn 4's assembled prompt carried the note verbatim, and the model's next delta was
correct. The full record, with the four turns and their chips, is at the end of
`plan/16-world-state-refusals.md`.
**Two failures blamed on the code were the model.** `graveler_defeated` never firing and
`world.turn` never moving both survived four playtests on the free demo model, with the
instructions present and correct in the assembled prompt. Both worked on the first try
against Sonnet. When an instruction is provably in the prompt, the next thing to change
is the model, not the wording.
**There is now a permanent way to test against a real model.**
`backend/tools/claude_shim.py` serves an OpenAI-compatible endpoint backed by the local
`claude` command line tool, so a demo can be played on a real model with no API key.
About $0.04 a turn against the subscription. The README documents it under **Connect a
model**.
**Renaming a seed scenario used to strand its old row.** `seed.py` matches a seed file to
its scenario by title, so changing a title inserted a second public scenario and left the
first orphaned and public forever. This is the same failure this file already records for
"Road to the Champion". Seed files now carry `previous_titles`, and the rename lands on
the existing row. Verified: the Pokemon demo kept its id.
**Every new guest is given a pre-played adventure.** `app/starter.py` copies a shipped
export bundle into each new guest account. The point is the first screen: real turns with
their world-state chips, including a refused change and a milestone, before spending any
of the daily demo turns. The row building inside `POST /adventures/import` moved to
`bundle.materialize` so both callers share one writer.
**An SVG data URI is not usable as scenario art.** `app/images.py` accepts raster formats
only, deliberately, because SVG can carry script and the bytes are served from the app's
own origin. A rejected value is stored but yields an empty `image_url`, which fails
quietly. The Pokeball is a 3.2 kB PNG, drawn by `backend/tools/make_pokeball.py` with
`zlib` alone.
---
## What happened on 2026-08-21 — visit analytics
The hosted demo can now answer whether anyone is using it. `/analytics` is a dashboard —
@@ -899,7 +944,7 @@ the SQLite dev parity this codebase protects on purpose).
```
cd backend
.venv/Scripts/python.exe -m pytest tests/ # 365 tests (~100s)
.venv/Scripts/python.exe -m pytest tests/ # 539 tests (~180s)
.venv/Scripts/python.exe -m tools.stress_session # egress report (SQLite)
# Same harness against a real Postgres. The target must be a THROWAWAY database
@@ -945,6 +990,26 @@ holds the egress ceilings.
On Windows the report's box-drawing characters crash the default cp1252 console;
prefix with `PYTHONIOENCODING=utf-8`.
**A real model, without an API key.** `tools/claude_shim.py` serves an OpenAI-compatible
endpoint backed by the local `claude` command line tool. Point Settings at
`http://127.0.0.1:8787/v1`, put any string in the API key field, and pick `sonnet`. Set
the reasoning budget to `0` or `-1`: a positive budget sends `reasoning.max_tokens`,
which Claude 5 models reject with a 400. It costs about $0.04 a turn against the
subscription, and it does not serve embeddings.
```
cd backend
.venv/Scripts/python.exe tools/claude_shim.py # 127.0.0.1:8787
```
**To exercise the guest path**, which is off in single-user mode:
```
cd backend
AIDND_MULTI_USER=1 AIDND_SECRET_KEY=throwaway AIDND_DB_PATH=$PWD/guest_scratch.db .venv/Scripts/python.exe -m uvicorn app.main:app --port 8001
curl -c jar.txt http://127.0.0.1:8001/api/auth/me # mints a guest and its starter
```
Port 8000 is shared with the job-pipeline app, which will squat it and silently shadow
the AI-DnD API — free it before running the backend, or move the vite proxy with
`AIDND_API_PORT`, which is what the `--keep` recipe above does.