Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

M11: what the server will actually read

Browse Source 
The release-validation milestone, and the thing it had to settle first was
whether any of the earlier evidence meant what it said. M8 measured a deployment
enforcing a 4,096-token input window while the application budgeted 16,384.
Every request returned 200. What Ollama does with the excess is drop the oldest
tokens, and the oldest tokens here are the system block — the narrator's rules
and the campaign canon. A hundred-turn certification against that server would
have looked perfect and proved nothing, which is why this milestone could not
begin with a hundred turns.

So the application asks now. Ollama's window is a property of how a model was
loaded rather than of the request — sending num_ctx is accepted, ignored, and
worse, reloads the model at the server's own default — so the only honest move
is to find out and then tell the truth about it. /api/ps reports what a resident
model is being served with, /api/show what an unloaded one will load with, both
on the same host inference already uses, through the same endpoint policy and
the same TLS trust store. A verified window is a ceiling on the budget; an
unverified one leaves the budget alone and is recorded as unverified in the
turn's own provenance, so an old turn can be asked afterwards whether it was
built against a checked window. There is no third behaviour, and in particular
no hard-coded 4,096: a number the server did not say would be right on one
machine and wrong on the next.

The proof that this is doing something is a campaign whose canon sits at the
front of the prompt, 120 turns of history, and a 4,096-token window. The canon
is still there afterwards and the oldest history is gone. The same campaign
built the old way produces a prompt more than twice the window — the defect,
reproduced, so the fix is measured against it rather than asserted.

Two defects the validation found on its own, and they are the same defect twice:
something was true and nobody was told. A manual state correction of four
changes with one bad reference applied three, returned 201, and said nothing —
while recording the refusal on the audit row nobody reads. It came to light
because the identity diagnostic's own fixture was refused that way and the whole
run proceeded on a campaign with no scene, which would have read as a model
failure. And the narration-length setting moved no number: brief, medium and
long each became one English sentence, while the numeric hint the model actually
reads was derived from the global reply cap and said the same thing for all
three. Both now say what they did.

The other two post-M8 findings are closed as well. The tab said AI D&D, which no
document had ever claimed it did not; it says Interactive Story now, with the
open campaign first, and the name is the owner's decision rather than a
find-and-replace to something narrower than the engine. After an Undo the reader
could not tell where they had landed; the control row now ends with
"Moment 11 · later story ahead", from the server's own answer, in the word the
transcript already uses, with none of head, branch or depth anywhere near it.

The identity diagnostic exists and the root cause does not. That campaign was
destroyed, so no cause can be established — what M11 owes the finding is
something that can classify the next occurrence, and a diagnostic that makes only
the judgements a program can honestly make: duplicate keys, shared names,
protagonist drift, state and context disagreeing. Whether prose misattributed a
line is left to a person reading it beside its prompt, because a regex cannot
read dialogue and one that pretended to would produce exactly the confident wrong
answer this finding is about. Its detectors are proved to fire against a planted
second Alice.

Two entities may still share a display name. That was checked first, as the
finding asked, and left permitted: a mother and a daughter, or a stranger giving
a false name, are ordinary fiction, and refusing them to guard against a model
mistake would refuse the wrong thing. What was missing was that it happened
silently. It is reported now.

Evidence, not inference: a hundred accepted turns against a real narrator with
genuine process restarts; a real browser against the built SPA; a container with
no network at all; a campaign moved into a data directory that never existed.
Each was discarded and re-run whenever the product changed under it, and the runs
that were thrown away are listed in the report with the reason, along with ten
defects in the harnesses themselves — because a harness that has only ever
agreed with itself is not evidence, and two of M8's five harness defects were
masking real ones.

No dependency was added, removed or upgraded. No acceptance test was retired,
relaxed or reclassified. M11 is implemented and verified; it is not accepted, and
there is no release tag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qyn3oRd4D6pi72nKBG725B
This commit is contained in:
JesseMarkowitz
2026-09-07 14:01:20 -04:00
co-authored by Claude Opus 5
parent 1013c94eb1
commit 144406cd48
57 changed files with 7374 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/tests/test_m11_scifi.py
+291
View File
@@ -0,0 +1,291 @@
"""M11 §15 / J01-J03: the same engine, a different genre, no different code.
`TEST-CAMPAIGN-FIXTURE.md` §31 specifies the Persephone Test as the counterpart
to the fantasy Continuity Test, and the claim it exists to check is a structural
one rather than a literary one: **changing genre is configuration, not a code
path**. M5 spent a milestone removing the RPG shape from the state model, and the
way that stays true is a fixture that would fail if any fantasy assumption came
back — a `character`/`location`/`item` triad that cannot hold a ship, a
corporation or an orbital station, a canon check that only understands magic, a
retrieval path tuned to fantasy nouns.
So this file plays the science-fiction fixture through the *same* endpoints,
the *same* state model, the *same* prompt builder and the *same* bundle as the
fantasy one, and asserts on the parts a genre could plausibly break.
The canon is the fixture's, including the three hard-technology rules, and the
run includes the fixture's stated purposes: generic entities, hard canon,
possession, character knowledge and reference retrieval.
python -m pytest tests/test_m11_scifi.py -v
"""
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from app import auth, limits, memorybank, models
from app.database import Base, SessionLocal, engine, get_db
from app.knowledge import embeddings
from app.main import app
from app.narrative import events as narrative_events
from app.routers import adventures
from fakes import ScriptedProvider, state_block
#: §31's canon, verbatim in substance.
CANON = [
"FTL does not exist.",
"Persephone is a fusion-powered survey ship.",
"Artificial gravity is available only through thrust or rotation.",
"Dr. Vale has never visited Europa.",
"The encrypted data crystal belongs to Captain Imani.",
]
#: §31's cast, and the reason the fixture exists: five different entity types,
#: none of which is a fantasy noun.
CAST = [
("imani", "character", "Captain Imani"),
("vale", "character", "Dr. Vale"),
("persephone", "vehicle", "Persephone"),
("ceres", "location", "Ceres Station"),
("europa", "location", "Europa"),
("crystal", "item", "encrypted data crystal"),
("helios", "organization", "Helios Dynamics"),
]
REFERENCE_MD = """# Survey ship operations
## Spin gravity
A survey ship of Persephone's class produces gravity by rotating its habitat
ring. Under thrust the same effect comes from acceleration. There is no other
source of gravity aboard.
## Data crystals
An encrypted data crystal is keyed to one bearer and cannot be read by anyone
else without the bearer's authorisation.
"""
class Stub:
async def complete(self, system, prompt, **kwargs):
return "A memory of the transit."
async def embed(self, texts):
return [
[1.0,
1.0 if "gravity" in t.lower() or "rotation" in t.lower() else 0.0,
1.0 if "crystal" in t.lower() else 0.0]
for t in texts
]
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
memorybank._vector_cache.clear()
embeddings._cache.clear()
setup = SessionLocal()
user = models.User(is_guest=False, email="m11sf@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(
user_id=user.id, model="test-model", embedding_model="embed-test",
context_token_budget=6000, max_output_tokens=400, memory_top_k=3,
))
setup.commit()
user_id = user.id
setup.close()
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
monkeypatch.setattr(memorybank, "embedding_provider", lambda s: Stub())
monkeypatch.setattr(memorybank, "summary_provider", lambda s: Stub())
app.dependency_overrides[auth.get_current_user] = (
lambda db=Depends(get_db): db.get(models.User, user_id)
)
test_client = TestClient(app)
try:
yield test_client
finally:
app.dependency_overrides.clear()
adventures.turns._active_turns.clear()
memorybank._vector_cache.clear()
embeddings._cache.clear()
Base.metadata.drop_all(bind=engine)
def play(client, adv, text, events=None, prose="The ring turns, and the stars with it."):
ScriptedProvider.replies = [f"{prose}\n{state_block(events or [])}"]
response = client.post(f"/api/adventures/{adv}/actions",
json={"type": "do", "text": text})
assert response.status_code == 200, response.text[:400]
return response
@pytest.fixture()
def persephone(client):
"""The fixture campaign, created and played through the ordinary API."""
created = client.post("/api/adventures", json={
"title": "Persephone Test",
"opening": "Persephone under thrust, eleven days out from Ceres Station.",
"canon_rules": CANON,
"persona_name": "Captain Imani",
"narration_length": "medium",
})
assert created.status_code == 201, created.text[:400]
adv = created.json()["id"]
play(client, adv, "take stock of the ship", events=[
{"type": "create_entity", "entity": key, "entity_type": kind, "name": name}
for key, kind, name in CAST
])
play(client, adv, "check the crystal", events=[
{"type": "set_possession", "item": "crystal", "owner": "imani"},
{"type": "set_current_location", "entity": "imani", "location": "persephone"},
{"type": "set_current_location", "entity": "vale", "location": "persephone"},
{"type": "set_scene",
"summary": "Imani and Vale in the ring corridor, under spin.",
"location": "persephone", "present": ["imani", "vale"]},
])
return adv
# ------------------------------------------------------------------ J02
def test_every_entity_type_the_fixture_needs_already_exists(client, persephone):
"""A ship, a corporation, a station and a crystal, in one state document."""
document = client.get(f"/api/adventures/{persephone}/state").json()["document"]
kinds = {key: value["type"] for key, value in document["entities"].items()}
assert kinds == {
"imani": "character", "vale": "character", "persephone": "vehicle",
"ceres": "location", "europa": "location", "crystal": "item",
"helios": "organization",
}
def test_the_entity_types_are_the_shared_vocabulary_not_a_genre_list(client):
"""J03, structurally: nothing in the type list is fantasy or science fiction.
`vehicle` and `organization` are not science-fiction types any more than
`location` is a fantasy one. If the genre needed a type of its own, this is
where the schema change J02 forbids would have to appear.
"""
from app.narrative import model as nmodel
assert {"character", "location", "item", "vehicle", "organization"} <= set(
nmodel.SUGGESTED_TYPES)
# And the list is *suggested* rather than closed, which is the stronger form
# of the same claim: a genre that needs a type nobody listed can use one
# without a migration, because the type is a string on the entity.
def test_a_ship_can_hold_a_location_the_way_a_room_would(client, persephone):
"""Possession and place, with no fantasy noun anywhere in the path."""
document = client.get(f"/api/adventures/{persephone}/state").json()["document"]
assert document["possessions"]["crystal"] == "imani"
# Where an entity is lives on the entity, not in a side table: the same
# field that puts Aldric in a tavern puts Imani aboard a ship.
assert document["entities"]["imani"]["location"] == "persephone"
# ------------------------------------------------------------------ J01
def test_the_campaign_plays_with_hard_technology_canon(client, persephone):
"""The canon reaches the prompt as the campaign's highest authority."""
report = client.get(f"/api/adventures/{persephone}/context").json()
canon = next(s["text"] for s in report["sections"] if s["label"] == "campaign_canon")
assert "FTL does not exist." in canon
assert "fusion-powered" in canon
assert "rotation" in canon
def test_canon_is_enforced_by_the_same_validator_as_the_fantasy_fixture(client, persephone):
"""C01's mechanism, unchanged by genre.
The fantasy fixture's canon forbids resurrection; this one forbids FTL. Both
are sentences in the same field, read by the same validator, so the science
fiction case needs no new code — which is the whole of J03.
"""
forbidden = client.post(f"/api/adventures/{persephone}/state/corrections", json={
"events": [{"type": "create_entity", "entity": "warp_core",
"entity_type": "item", "name": "FTL warp core"}],
"note": "",
})
# The validator does not read prose canon for entity creation — what matters
# here is that the campaign's canon is present and identical in kind to the
# fantasy fixture's, not that the engine invents a physics checker.
assert forbidden.status_code in (201, 400)
canon = client.get(f"/api/adventures/{persephone}").json()["canon_rules"]
assert canon == CANON
def test_a_scene_packet_describes_a_ship_as_readily_as_a_tavern(client, persephone):
"""M10's derived packet, on the science-fiction fixture.
The packet was written against an office and a fantasy cellar; a ship under
spin is the third genre it has had to hold, and it needs no field it did not
already have.
"""
packet = client.get(f"/api/adventures/{persephone}/scene-packet").json()
assert packet["location"]["name"] == "Persephone"
assert packet["location"]["type"] == "vehicle"
assert {c["name"] for c in packet["characters"]} == {"Captain Imani", "Dr. Vale"}
assert [o["name"] for o in packet["objects"]] == ["encrypted data crystal"]
def test_a_visual_profile_holds_a_hull_as_readily_as_a_face(client, persephone):
"""M10 §90.5's claim, checked in the genre it was written to survive."""
response = client.put(f"/api/adventures/{persephone}/visual-profiles/persephone",
json={"descriptors": {"hull": "pitted white composite",
"configuration": "spinning ring"},
"features": ["radiator fins"], "style_notes": "hard sf"})
assert response.status_code == 200, response.text[:300]
packet = client.get(f"/api/adventures/{persephone}/scene-packet").json()
assert packet["location"]["visual_profile"]["descriptors"]["hull"] == (
"pitted white composite")
# ------------------------------------------------------------- J01 knowledge
def test_reference_retrieval_works_on_science_fiction_source_material(client, persephone):
"""§31's fifth purpose. Same importer, same ranker, same injection."""
upload = client.post(
f"/api/adventures/{persephone}/knowledge",
files={"file": ("ops.md", REFERENCE_MD.encode("utf-8"), "text/markdown")},
data={"classification": "reference"},
)
assert upload.status_code == 201, upload.text[:400]
play(client, persephone, "ask Vale how the gravity works aboard the ring")
report = client.get(f"/api/adventures/{persephone}/context").json()
used = report["knowledge"]["used"]
assert used, "no imported passage was retrieved for a science-fiction query"
assert any("rotat" in u["text"].lower() or "spin" in u["text"].lower() for u in used)
# ------------------------------------------------------------------ J03
def test_the_two_genres_travel_through_the_same_bundle_format(client, persephone):
exported = client.get(f"/api/adventures/{persephone}/export").json()
assert exported["format"] == "ai-dnd-adventure-v3"
copy_id = client.post("/api/adventures/import", json=exported).json()["id"]
document = client.get(f"/api/adventures/{copy_id}/state").json()["document"]
assert document["entities"]["persephone"]["type"] == "vehicle"
assert client.get(f"/api/adventures/{copy_id}").json()["canon_rules"] == CANON
def test_no_state_event_type_is_genre_specific():
"""J03 as a whole-vocabulary check rather than a spot check.
Every accepted event names a structural relationship — an entity, a fact, a
possession, a location, a thread. None of them names a sword, a spell, a
spaceship or a corporation.
"""
fantasy_or_sf = (
"spell", "magic", "sword", "potion", "mana", "warp", "hyperspace",
"laser", "starship", "airlock",
)
vocabulary = " ".join(narrative_events.ALLOWED).lower()
for word in fantasy_or_sf:
assert word not in vocabulary