Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

M11: what the server will actually read

Browse Source 
The release-validation milestone, and the thing it had to settle first was
whether any of the earlier evidence meant what it said. M8 measured a deployment
enforcing a 4,096-token input window while the application budgeted 16,384.
Every request returned 200. What Ollama does with the excess is drop the oldest
tokens, and the oldest tokens here are the system block — the narrator's rules
and the campaign canon. A hundred-turn certification against that server would
have looked perfect and proved nothing, which is why this milestone could not
begin with a hundred turns.

So the application asks now. Ollama's window is a property of how a model was
loaded rather than of the request — sending num_ctx is accepted, ignored, and
worse, reloads the model at the server's own default — so the only honest move
is to find out and then tell the truth about it. /api/ps reports what a resident
model is being served with, /api/show what an unloaded one will load with, both
on the same host inference already uses, through the same endpoint policy and
the same TLS trust store. A verified window is a ceiling on the budget; an
unverified one leaves the budget alone and is recorded as unverified in the
turn's own provenance, so an old turn can be asked afterwards whether it was
built against a checked window. There is no third behaviour, and in particular
no hard-coded 4,096: a number the server did not say would be right on one
machine and wrong on the next.

The proof that this is doing something is a campaign whose canon sits at the
front of the prompt, 120 turns of history, and a 4,096-token window. The canon
is still there afterwards and the oldest history is gone. The same campaign
built the old way produces a prompt more than twice the window — the defect,
reproduced, so the fix is measured against it rather than asserted.

Two defects the validation found on its own, and they are the same defect twice:
something was true and nobody was told. A manual state correction of four
changes with one bad reference applied three, returned 201, and said nothing —
while recording the refusal on the audit row nobody reads. It came to light
because the identity diagnostic's own fixture was refused that way and the whole
run proceeded on a campaign with no scene, which would have read as a model
failure. And the narration-length setting moved no number: brief, medium and
long each became one English sentence, while the numeric hint the model actually
reads was derived from the global reply cap and said the same thing for all
three. Both now say what they did.

The other two post-M8 findings are closed as well. The tab said AI D&D, which no
document had ever claimed it did not; it says Interactive Story now, with the
open campaign first, and the name is the owner's decision rather than a
find-and-replace to something narrower than the engine. After an Undo the reader
could not tell where they had landed; the control row now ends with
"Moment 11 · later story ahead", from the server's own answer, in the word the
transcript already uses, with none of head, branch or depth anywhere near it.

The identity diagnostic exists and the root cause does not. That campaign was
destroyed, so no cause can be established — what M11 owes the finding is
something that can classify the next occurrence, and a diagnostic that makes only
the judgements a program can honestly make: duplicate keys, shared names,
protagonist drift, state and context disagreeing. Whether prose misattributed a
line is left to a person reading it beside its prompt, because a regex cannot
read dialogue and one that pretended to would produce exactly the confident wrong
answer this finding is about. Its detectors are proved to fire against a planted
second Alice.

Two entities may still share a display name. That was checked first, as the
finding asked, and left permitted: a mother and a daughter, or a stranger giving
a false name, are ordinary fiction, and refusing them to guard against a model
mistake would refuse the wrong thing. What was missing was that it happened
silently. It is reported now.

Evidence, not inference: a hundred accepted turns against a real narrator with
genuine process restarts; a real browser against the built SPA; a container with
no network at all; a campaign moved into a data directory that never existed.
Each was discarded and re-run whenever the product changed under it, and the runs
that were thrown away are listed in the report with the reason, along with ten
defects in the harnesses themselves — because a harness that has only ever
agreed with itself is not evidence, and two of M8's five harness defects were
masking real ones.

No dependency was added, removed or upgraded. No acceptance test was retired,
relaxed or reclassified. M11 is implemented and verified; it is not accepted, and
there is no release tag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qyn3oRd4D6pi72nKBG725B
This commit is contained in:
JesseMarkowitz
2026-09-07 14:01:20 -04:00
co-authored by Claude Opus 5
parent 1013c94eb1
commit 144406cd48
57 changed files with 7374 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/tests/test_m11_findings.py
+358
View File
@@ -0,0 +1,358 @@
"""M11: the post-M8 playtest findings, on the backend side.
Findings A and B are browser-only and are tested in `frontend/src/m11.test.jsx`.
This file covers finding C, which is half a browser change and half a prompt
change, and the structural fact finding D asks M11 to check first.
**Finding C, in one sentence:** the campaign's narration-length choice became an
English sentence in the instructions and moved no number, while the numeric hint
the model actually reads was derived from the *global* reply cap and therefore
said the same thing — "must not exceed 506 words, and it should not stop short of
about 177" — whether the reader chose brief, medium or long.
python -m pytest tests/test_m11_findings.py -v
"""
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from app import auth, bundle, limits, models
from app.context import builder
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
from app.narrative import model as nmodel
from app.routers import adventures
from fakes import ScriptedProvider
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
user = models.User(is_guest=False, email="m11f@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(
user_id=user.id, model="test-model", embedding_model="",
context_token_budget=16384, max_output_tokens=800,
))
setup.commit()
user_id = user.id
setup.close()
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
app.dependency_overrides[auth.get_current_user] = (
lambda db=Depends(get_db): db.get(models.User, user_id)
)
test_client = TestClient(app)
try:
yield test_client
finally:
app.dependency_overrides.clear()
adventures.turns._active_turns.clear()
Base.metadata.drop_all(bind=engine)
def _campaign(client, **fields):
body = {"title": "Length", "opening": "Rain over Westhaven."} | fields
response = client.post("/api/adventures", json=body)
assert response.status_code == 201, response.text[:300]
return response.json()
def _hint_for(length, cap=800):
return builder.length_hint(cap, length)
def _numbers(hint):
import re
return [int(n) for n in re.findall(r"\b(\d+)\b", hint)]
# ------------------------------------------------ finding C: the defect itself
def test_the_three_lengths_no_longer_say_the_same_thing():
"""The finding, as a test that would have failed before M11.
At the default 800-token cap every length produced the identical sentence.
Now each produces a different ceiling, and they are ordered the way the
words are.
"""
brief, medium, long = (_hint_for(x) for x in ("brief", "medium", "long"))
assert brief != medium != long
assert brief != long
ceilings = [_numbers(h)[0] for h in (brief, medium, long)]
assert ceilings == sorted(ceilings), ceilings
assert len(set(ceilings)) == 3
def test_a_campaign_with_no_preference_reads_exactly_as_it_did_before():
"""No existing campaign's prompt changes under the migration.
The empty value is the pre-M11 behaviour, unchanged — which is what makes a
backfill unnecessary rather than merely inconvenient.
"""
assert _hint_for("") == builder.length_hint(800)
def test_the_reply_cap_still_wins_over_the_band():
"""A long campaign on a small cap gets the cap's number, not the band's.
The cap is what the endpoint will actually emit, so a hint that asked for
more would be asking for a truncated turn — and the state block is emitted
last, so a truncated turn loses its state.
"""
long_on_small_cap = _hint_for("long", cap=300)
assert _numbers(long_on_small_cap)[0] <= _numbers(builder.length_hint(300))[0]
def test_the_band_narrows_rather_than_widens_the_cap():
for length in ("brief", "medium", "long"):
banded = _numbers(_hint_for(length, cap=800))[0]
unbanded = _numbers(builder.length_hint(800))[0]
assert banded <= unbanded, length
def test_every_hint_still_protects_the_state_block():
"""The invariant the old hint had, kept by the new one."""
for length in ("", "brief", "medium", "long"):
assert "state block" in _hint_for(length)
def test_an_unknown_length_falls_back_rather_than_inventing_a_band():
assert _hint_for("epic") == builder.length_hint(800)
# ------------------------------------------- finding C: it reaches the prompt
def test_the_choice_is_stored_and_returned(client):
campaign = _campaign(client, narration_length="brief")
assert campaign["narration_length"] == "brief"
assert client.get(f"/api/adventures/{campaign['id']}").json()[
"narration_length"] == "brief"
def test_the_choice_can_be_changed_afterwards(client):
campaign = _campaign(client, narration_length="brief")
updated = client.patch(f"/api/adventures/{campaign['id']}",
json={"narration_length": "long"})
assert updated.status_code == 200, updated.text[:300]
assert updated.json()["narration_length"] == "long"
def test_a_length_the_builder_cannot_serve_is_refused(client):
"""A closed set, because an unknown value would silently mean 'no effect'."""
response = client.post("/api/adventures", json={
"title": "Bad", "opening": "x", "narration_length": "epic"})
assert response.status_code == 422
def test_the_stored_prompt_carries_the_campaigns_own_range(client):
"""End to end: two campaigns, two choices, two different prompts."""
from sqlalchemy.orm import undefer
seen = {}
for length in ("brief", "long"):
campaign = _campaign(client, narration_length=length)
ScriptedProvider.replies = ["The rain does not let up."]
assert client.post(f"/api/adventures/{campaign['id']}/actions",
json={"type": "do", "text": "look"}).status_code == 200
with SessionLocal() as db:
action = (
db.query(models.Action)
.filter(models.Action.adventure_id == campaign["id"],
models.Action.type == "ai")
.options(undefer(models.Action.context_snapshot))
.order_by(models.Action.id.desc()).first()
)
hint = next(s for s in action.context_snapshot["sections"]
if s["label"] == "length_hint")
seen[length] = _numbers(hint["text"])[0]
assert seen["brief"] < seen["long"], seen
def test_the_choice_travels_in_the_bundle(client):
campaign = _campaign(client, narration_length="long")
exported = client.get(f"/api/adventures/{campaign['id']}/export").json()
assert exported["narrationLength"] == "long"
copy_id = client.post("/api/adventures/import", json=exported).json()["id"]
assert client.get(f"/api/adventures/{copy_id}").json()["narration_length"] == "long"
def test_a_bundle_naming_a_length_this_build_cannot_serve_drops_it(client):
campaign = _campaign(client, narration_length="long")
payload = client.get(f"/api/adventures/{campaign['id']}/export").json()
payload["narrationLength"] = "cinematic"
copy_id = client.post("/api/adventures/import", json=payload).json()["id"]
# Empty rather than stored: a preference the builder ignores is
# indistinguishable from the defect this milestone fixed.
assert client.get(f"/api/adventures/{copy_id}").json()["narration_length"] == ""
def test_an_older_bundle_with_no_length_imports_unchanged(client):
campaign = _campaign(client, narration_length="long")
payload = client.get(f"/api/adventures/{campaign['id']}/export").json()
del payload["narrationLength"]
copy_id = client.post("/api/adventures/import", json=payload).json()["id"]
assert client.get(f"/api/adventures/{copy_id}").json()["narration_length"] == ""
# ------------------- C04: a correction that is partly refused says so (M11-1)
def test_a_partly_refused_correction_reports_what_did_not_apply(client):
"""The defect the identity diagnostic surfaced, as a regression.
A correction of two changes where one names a location that does not exist:
the good one lands, the bad one does not, and **before M11 the answer was an
unqualified 201**. The reader was told nothing, and went on believing they
had set a scene they had not.
`validate.py` already said this must not happen — "what is never allowed is
a rejected event mutating anything, or **a rejection being silent**" — and
the refusal was recorded on the proposal for the audit trail. What was
missing was telling the person who made the correction. Partial application
itself is deliberate and is unchanged: losing three good changes to one typo
would be worse.
"""
campaign = _campaign(client)
response = client.post(f"/api/adventures/{campaign['id']}/state/corrections", json={
"events": [
{"type": "create_entity", "entity": "mara",
"entity_type": "character", "name": "Mara"},
{"type": "set_scene", "summary": "In the hall.",
"location": "nowhere", "present": ["mara"]},
],
"note": "one good, one bad",
})
assert response.status_code == 201, response.text[:300]
body = response.json()
# The good change landed.
assert "mara" in body["document"]["entities"]
# The bad one did not, and the caller is told which and why.
assert body["document"].get("scene") in ({}, None)
assert len(body["refused"]) == 1, body["refused"]
refusal = body["refused"][0]
assert refusal["event"]["type"] == "set_scene"
assert refusal["reason"] == "unknown_reference"
assert "nowhere" in refusal["detail"]
def test_a_correction_that_fully_applies_reports_nothing_refused(client):
"""The control: `refused` is empty when nothing was refused."""
campaign = _campaign(client)
response = client.post(f"/api/adventures/{campaign['id']}/state/corrections", json={
"events": [{"type": "create_entity", "entity": "mara",
"entity_type": "character", "name": "Mara"}],
"note": "",
})
assert response.status_code == 201
assert response.json()["refused"] == []
def test_a_wholly_refused_correction_is_still_a_400(client):
"""Unchanged: nothing applied is an error, not a success with a note."""
campaign = _campaign(client)
response = client.post(f"/api/adventures/{campaign['id']}/state/corrections", json={
"events": [{"type": "set_scene", "summary": "x", "location": "nowhere"}],
"note": "",
})
assert response.status_code == 400
assert "nowhere" in response.json()["detail"]
def test_the_refusal_is_still_recorded_for_the_audit_trail(client):
"""The half that already worked keeps working: §8's proposal record."""
campaign = _campaign(client)
client.post(f"/api/adventures/{campaign['id']}/state/corrections", json={
"events": [
{"type": "create_entity", "entity": "mara",
"entity_type": "character", "name": "Mara"},
{"type": "set_scene", "summary": "In the hall.", "location": "nowhere"},
],
"note": "",
})
with SessionLocal() as db:
# `detail` is deferred, so it is read inside the session — reading it
# after the session closed is how the first version of this test failed.
rows = [
(p.status, repr(p.detail))
for p in db.query(models.StateProposal).filter(
models.StateProposal.adventure_id == campaign["id"]).all()
]
partial = [row for row in rows if row[0] == "partially_accepted"]
assert partial, [row[0] for row in rows]
# And the reason is on the record, not only the verdict.
assert "nowhere" in partial[0][1]
# --------------------------------- finding D: the structural fact to check first
def test_two_entities_may_still_share_a_display_name(client):
"""Recorded, not fixed — and the distinction matters.
The finding says to check this first: the narrative state keys entities by
the model-supplied id and `DUPLICATE_ENTITY` rejects only a repeated *key*,
so two characters can be created with the same `name` and nothing says so.
That is one of the finding's candidate failure modes.
It is **not** made an error here. Two people called Alice is an ordinary
thing for a story to contain, and refusing it would refuse legitimate
fiction to guard against a model mistake. What M11 adds instead is
*detection*: `nmodel.duplicate_names` reports it, the identity diagnostic
(`tools/m11_identity.py`) reads that report, and the reader's State panel
can show it. This test pins the permissive behaviour so a later milestone
changes it deliberately rather than by accident.
"""
campaign = _campaign(client)
response = client.post(f"/api/adventures/{campaign['id']}/state/corrections", json={
"events": [
{"type": "create_entity", "entity": "alice_1",
"entity_type": "character", "name": "Alice"},
{"type": "create_entity", "entity": "alice_2",
"entity_type": "character", "name": "Alice"},
],
"note": "two people, one name",
})
assert response.status_code == 201, response.text[:300]
document = client.get(f"/api/adventures/{campaign['id']}/state").json()["document"]
assert set(document["entities"]) >= {"alice_1", "alice_2"}
def test_the_state_reports_a_shared_display_name(client):
"""M11 adds the detection the finding asks for, without adding a refusal."""
campaign = _campaign(client)
client.post(f"/api/adventures/{campaign['id']}/state/corrections", json={
"events": [
{"type": "create_entity", "entity": "alice_1",
"entity_type": "character", "name": "Alice"},
{"type": "create_entity", "entity": "alice_2",
"entity_type": "character", "name": "alice "},
{"type": "create_entity", "entity": "roger",
"entity_type": "character", "name": "Roger"},
],
"note": "",
})
with SessionLocal() as db:
adventure = db.get(models.Adventure, campaign["id"])
clashes = nmodel.duplicate_names(adventure.narrative_state)
# Case and surrounding space do not make two people different.
assert clashes == {"alice": ["alice_1", "alice_2"]}
def test_a_campaign_with_distinct_names_reports_nothing(client):
campaign = _campaign(client)
client.post(f"/api/adventures/{campaign['id']}/state/corrections", json={
"events": [
{"type": "create_entity", "entity": "a", "entity_type": "character",
"name": "Alice"},
{"type": "create_entity", "entity": "r", "entity_type": "character",
"name": "Roger"},
],
"note": "",
})
with SessionLocal() as db:
adventure = db.get(models.Adventure, campaign["id"])
assert nmodel.duplicate_names(adventure.narrative_state) == {}