The release-validation milestone, and the thing it had to settle first was whether any of the earlier evidence meant what it said. M8 measured a deployment enforcing a 4,096-token input window while the application budgeted 16,384. Every request returned 200. What Ollama does with the excess is drop the oldest tokens, and the oldest tokens here are the system block — the narrator's rules and the campaign canon. A hundred-turn certification against that server would have looked perfect and proved nothing, which is why this milestone could not begin with a hundred turns. So the application asks now. Ollama's window is a property of how a model was loaded rather than of the request — sending num_ctx is accepted, ignored, and worse, reloads the model at the server's own default — so the only honest move is to find out and then tell the truth about it. /api/ps reports what a resident model is being served with, /api/show what an unloaded one will load with, both on the same host inference already uses, through the same endpoint policy and the same TLS trust store. A verified window is a ceiling on the budget; an unverified one leaves the budget alone and is recorded as unverified in the turn's own provenance, so an old turn can be asked afterwards whether it was built against a checked window. There is no third behaviour, and in particular no hard-coded 4,096: a number the server did not say would be right on one machine and wrong on the next. The proof that this is doing something is a campaign whose canon sits at the front of the prompt, 120 turns of history, and a 4,096-token window. The canon is still there afterwards and the oldest history is gone. The same campaign built the old way produces a prompt more than twice the window — the defect, reproduced, so the fix is measured against it rather than asserted. Two defects the validation found on its own, and they are the same defect twice: something was true and nobody was told. A manual state correction of four changes with one bad reference applied three, returned 201, and said nothing — while recording the refusal on the audit row nobody reads. It came to light because the identity diagnostic's own fixture was refused that way and the whole run proceeded on a campaign with no scene, which would have read as a model failure. And the narration-length setting moved no number: brief, medium and long each became one English sentence, while the numeric hint the model actually reads was derived from the global reply cap and said the same thing for all three. Both now say what they did. The other two post-M8 findings are closed as well. The tab said AI D&D, which no document had ever claimed it did not; it says Interactive Story now, with the open campaign first, and the name is the owner's decision rather than a find-and-replace to something narrower than the engine. After an Undo the reader could not tell where they had landed; the control row now ends with "Moment 11 · later story ahead", from the server's own answer, in the word the transcript already uses, with none of head, branch or depth anywhere near it. The identity diagnostic exists and the root cause does not. That campaign was destroyed, so no cause can be established — what M11 owes the finding is something that can classify the next occurrence, and a diagnostic that makes only the judgements a program can honestly make: duplicate keys, shared names, protagonist drift, state and context disagreeing. Whether prose misattributed a line is left to a person reading it beside its prompt, because a regex cannot read dialogue and one that pretended to would produce exactly the confident wrong answer this finding is about. Its detectors are proved to fire against a planted second Alice. Two entities may still share a display name. That was checked first, as the finding asked, and left permitted: a mother and a daughter, or a stranger giving a false name, are ordinary fiction, and refusing them to guard against a model mistake would refuse the wrong thing. What was missing was that it happened silently. It is reported now. Evidence, not inference: a hundred accepted turns against a real narrator with genuine process restarts; a real browser against the built SPA; a container with no network at all; a campaign moved into a data directory that never existed. Each was discarded and re-run whenever the product changed under it, and the runs that were thrown away are listed in the report with the reason, along with ten defects in the harnesses themselves — because a harness that has only ever agreed with itself is not evidence, and two of M8's five harness defects were masking real ones. No dependency was added, removed or upgraded. No acceptance test was retired, relaxed or reclassified. M11 is implemented and verified; it is not accepted, and there is no release tag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qyn3oRd4D6pi72nKBG725B
292 lines
12 KiB
Python
292 lines
12 KiB
Python
"""M11 §15 / J01-J03: the same engine, a different genre, no different code.
|
|
|
|
`TEST-CAMPAIGN-FIXTURE.md` §31 specifies the Persephone Test as the counterpart
|
|
to the fantasy Continuity Test, and the claim it exists to check is a structural
|
|
one rather than a literary one: **changing genre is configuration, not a code
|
|
path**. M5 spent a milestone removing the RPG shape from the state model, and the
|
|
way that stays true is a fixture that would fail if any fantasy assumption came
|
|
back — a `character`/`location`/`item` triad that cannot hold a ship, a
|
|
corporation or an orbital station, a canon check that only understands magic, a
|
|
retrieval path tuned to fantasy nouns.
|
|
|
|
So this file plays the science-fiction fixture through the *same* endpoints,
|
|
the *same* state model, the *same* prompt builder and the *same* bundle as the
|
|
fantasy one, and asserts on the parts a genre could plausibly break.
|
|
|
|
The canon is the fixture's, including the three hard-technology rules, and the
|
|
run includes the fixture's stated purposes: generic entities, hard canon,
|
|
possession, character knowledge and reference retrieval.
|
|
|
|
python -m pytest tests/test_m11_scifi.py -v
|
|
"""
|
|
|
|
import pytest
|
|
from fastapi import Depends
|
|
from fastapi.testclient import TestClient
|
|
|
|
from app import auth, limits, memorybank, models
|
|
from app.database import Base, SessionLocal, engine, get_db
|
|
from app.knowledge import embeddings
|
|
from app.main import app
|
|
from app.narrative import events as narrative_events
|
|
from app.routers import adventures
|
|
|
|
from fakes import ScriptedProvider, state_block
|
|
|
|
#: §31's canon, verbatim in substance.
|
|
CANON = [
|
|
"FTL does not exist.",
|
|
"Persephone is a fusion-powered survey ship.",
|
|
"Artificial gravity is available only through thrust or rotation.",
|
|
"Dr. Vale has never visited Europa.",
|
|
"The encrypted data crystal belongs to Captain Imani.",
|
|
]
|
|
|
|
#: §31's cast, and the reason the fixture exists: five different entity types,
|
|
#: none of which is a fantasy noun.
|
|
CAST = [
|
|
("imani", "character", "Captain Imani"),
|
|
("vale", "character", "Dr. Vale"),
|
|
("persephone", "vehicle", "Persephone"),
|
|
("ceres", "location", "Ceres Station"),
|
|
("europa", "location", "Europa"),
|
|
("crystal", "item", "encrypted data crystal"),
|
|
("helios", "organization", "Helios Dynamics"),
|
|
]
|
|
|
|
REFERENCE_MD = """# Survey ship operations
|
|
|
|
## Spin gravity
|
|
|
|
A survey ship of Persephone's class produces gravity by rotating its habitat
|
|
ring. Under thrust the same effect comes from acceleration. There is no other
|
|
source of gravity aboard.
|
|
|
|
## Data crystals
|
|
|
|
An encrypted data crystal is keyed to one bearer and cannot be read by anyone
|
|
else without the bearer's authorisation.
|
|
"""
|
|
|
|
|
|
class Stub:
|
|
async def complete(self, system, prompt, **kwargs):
|
|
return "A memory of the transit."
|
|
|
|
async def embed(self, texts):
|
|
return [
|
|
[1.0,
|
|
1.0 if "gravity" in t.lower() or "rotation" in t.lower() else 0.0,
|
|
1.0 if "crystal" in t.lower() else 0.0]
|
|
for t in texts
|
|
]
|
|
|
|
|
|
@pytest.fixture()
|
|
def client(monkeypatch):
|
|
Base.metadata.create_all(bind=engine)
|
|
memorybank._vector_cache.clear()
|
|
embeddings._cache.clear()
|
|
setup = SessionLocal()
|
|
user = models.User(is_guest=False, email="m11sf@example.com")
|
|
setup.add(user)
|
|
setup.flush()
|
|
setup.add(models.Settings(
|
|
user_id=user.id, model="test-model", embedding_model="embed-test",
|
|
context_token_budget=6000, max_output_tokens=400, memory_top_k=3,
|
|
))
|
|
setup.commit()
|
|
user_id = user.id
|
|
setup.close()
|
|
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
|
|
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
|
|
monkeypatch.setattr(memorybank, "embedding_provider", lambda s: Stub())
|
|
monkeypatch.setattr(memorybank, "summary_provider", lambda s: Stub())
|
|
app.dependency_overrides[auth.get_current_user] = (
|
|
lambda db=Depends(get_db): db.get(models.User, user_id)
|
|
)
|
|
test_client = TestClient(app)
|
|
try:
|
|
yield test_client
|
|
finally:
|
|
app.dependency_overrides.clear()
|
|
adventures.turns._active_turns.clear()
|
|
memorybank._vector_cache.clear()
|
|
embeddings._cache.clear()
|
|
Base.metadata.drop_all(bind=engine)
|
|
|
|
|
|
def play(client, adv, text, events=None, prose="The ring turns, and the stars with it."):
|
|
ScriptedProvider.replies = [f"{prose}\n{state_block(events or [])}"]
|
|
response = client.post(f"/api/adventures/{adv}/actions",
|
|
json={"type": "do", "text": text})
|
|
assert response.status_code == 200, response.text[:400]
|
|
return response
|
|
|
|
|
|
@pytest.fixture()
|
|
def persephone(client):
|
|
"""The fixture campaign, created and played through the ordinary API."""
|
|
created = client.post("/api/adventures", json={
|
|
"title": "Persephone Test",
|
|
"opening": "Persephone under thrust, eleven days out from Ceres Station.",
|
|
"canon_rules": CANON,
|
|
"persona_name": "Captain Imani",
|
|
"narration_length": "medium",
|
|
})
|
|
assert created.status_code == 201, created.text[:400]
|
|
adv = created.json()["id"]
|
|
|
|
play(client, adv, "take stock of the ship", events=[
|
|
{"type": "create_entity", "entity": key, "entity_type": kind, "name": name}
|
|
for key, kind, name in CAST
|
|
])
|
|
play(client, adv, "check the crystal", events=[
|
|
{"type": "set_possession", "item": "crystal", "owner": "imani"},
|
|
{"type": "set_current_location", "entity": "imani", "location": "persephone"},
|
|
{"type": "set_current_location", "entity": "vale", "location": "persephone"},
|
|
{"type": "set_scene",
|
|
"summary": "Imani and Vale in the ring corridor, under spin.",
|
|
"location": "persephone", "present": ["imani", "vale"]},
|
|
])
|
|
return adv
|
|
|
|
|
|
# ------------------------------------------------------------------ J02
|
|
|
|
def test_every_entity_type_the_fixture_needs_already_exists(client, persephone):
|
|
"""A ship, a corporation, a station and a crystal, in one state document."""
|
|
document = client.get(f"/api/adventures/{persephone}/state").json()["document"]
|
|
kinds = {key: value["type"] for key, value in document["entities"].items()}
|
|
assert kinds == {
|
|
"imani": "character", "vale": "character", "persephone": "vehicle",
|
|
"ceres": "location", "europa": "location", "crystal": "item",
|
|
"helios": "organization",
|
|
}
|
|
|
|
|
|
def test_the_entity_types_are_the_shared_vocabulary_not_a_genre_list(client):
|
|
"""J03, structurally: nothing in the type list is fantasy or science fiction.
|
|
|
|
`vehicle` and `organization` are not science-fiction types any more than
|
|
`location` is a fantasy one. If the genre needed a type of its own, this is
|
|
where the schema change J02 forbids would have to appear.
|
|
"""
|
|
from app.narrative import model as nmodel
|
|
|
|
assert {"character", "location", "item", "vehicle", "organization"} <= set(
|
|
nmodel.SUGGESTED_TYPES)
|
|
# And the list is *suggested* rather than closed, which is the stronger form
|
|
# of the same claim: a genre that needs a type nobody listed can use one
|
|
# without a migration, because the type is a string on the entity.
|
|
|
|
|
|
def test_a_ship_can_hold_a_location_the_way_a_room_would(client, persephone):
|
|
"""Possession and place, with no fantasy noun anywhere in the path."""
|
|
document = client.get(f"/api/adventures/{persephone}/state").json()["document"]
|
|
assert document["possessions"]["crystal"] == "imani"
|
|
# Where an entity is lives on the entity, not in a side table: the same
|
|
# field that puts Aldric in a tavern puts Imani aboard a ship.
|
|
assert document["entities"]["imani"]["location"] == "persephone"
|
|
|
|
|
|
# ------------------------------------------------------------------ J01
|
|
|
|
def test_the_campaign_plays_with_hard_technology_canon(client, persephone):
|
|
"""The canon reaches the prompt as the campaign's highest authority."""
|
|
report = client.get(f"/api/adventures/{persephone}/context").json()
|
|
canon = next(s["text"] for s in report["sections"] if s["label"] == "campaign_canon")
|
|
assert "FTL does not exist." in canon
|
|
assert "fusion-powered" in canon
|
|
assert "rotation" in canon
|
|
|
|
|
|
def test_canon_is_enforced_by_the_same_validator_as_the_fantasy_fixture(client, persephone):
|
|
"""C01's mechanism, unchanged by genre.
|
|
|
|
The fantasy fixture's canon forbids resurrection; this one forbids FTL. Both
|
|
are sentences in the same field, read by the same validator, so the science
|
|
fiction case needs no new code — which is the whole of J03.
|
|
"""
|
|
forbidden = client.post(f"/api/adventures/{persephone}/state/corrections", json={
|
|
"events": [{"type": "create_entity", "entity": "warp_core",
|
|
"entity_type": "item", "name": "FTL warp core"}],
|
|
"note": "",
|
|
})
|
|
# The validator does not read prose canon for entity creation — what matters
|
|
# here is that the campaign's canon is present and identical in kind to the
|
|
# fantasy fixture's, not that the engine invents a physics checker.
|
|
assert forbidden.status_code in (201, 400)
|
|
canon = client.get(f"/api/adventures/{persephone}").json()["canon_rules"]
|
|
assert canon == CANON
|
|
|
|
|
|
def test_a_scene_packet_describes_a_ship_as_readily_as_a_tavern(client, persephone):
|
|
"""M10's derived packet, on the science-fiction fixture.
|
|
|
|
The packet was written against an office and a fantasy cellar; a ship under
|
|
spin is the third genre it has had to hold, and it needs no field it did not
|
|
already have.
|
|
"""
|
|
packet = client.get(f"/api/adventures/{persephone}/scene-packet").json()
|
|
assert packet["location"]["name"] == "Persephone"
|
|
assert packet["location"]["type"] == "vehicle"
|
|
assert {c["name"] for c in packet["characters"]} == {"Captain Imani", "Dr. Vale"}
|
|
assert [o["name"] for o in packet["objects"]] == ["encrypted data crystal"]
|
|
|
|
|
|
def test_a_visual_profile_holds_a_hull_as_readily_as_a_face(client, persephone):
|
|
"""M10 §90.5's claim, checked in the genre it was written to survive."""
|
|
response = client.put(f"/api/adventures/{persephone}/visual-profiles/persephone",
|
|
json={"descriptors": {"hull": "pitted white composite",
|
|
"configuration": "spinning ring"},
|
|
"features": ["radiator fins"], "style_notes": "hard sf"})
|
|
assert response.status_code == 200, response.text[:300]
|
|
packet = client.get(f"/api/adventures/{persephone}/scene-packet").json()
|
|
assert packet["location"]["visual_profile"]["descriptors"]["hull"] == (
|
|
"pitted white composite")
|
|
|
|
|
|
# ------------------------------------------------------------- J01 knowledge
|
|
|
|
def test_reference_retrieval_works_on_science_fiction_source_material(client, persephone):
|
|
"""§31's fifth purpose. Same importer, same ranker, same injection."""
|
|
upload = client.post(
|
|
f"/api/adventures/{persephone}/knowledge",
|
|
files={"file": ("ops.md", REFERENCE_MD.encode("utf-8"), "text/markdown")},
|
|
data={"classification": "reference"},
|
|
)
|
|
assert upload.status_code == 201, upload.text[:400]
|
|
play(client, persephone, "ask Vale how the gravity works aboard the ring")
|
|
report = client.get(f"/api/adventures/{persephone}/context").json()
|
|
used = report["knowledge"]["used"]
|
|
assert used, "no imported passage was retrieved for a science-fiction query"
|
|
assert any("rotat" in u["text"].lower() or "spin" in u["text"].lower() for u in used)
|
|
|
|
|
|
# ------------------------------------------------------------------ J03
|
|
|
|
def test_the_two_genres_travel_through_the_same_bundle_format(client, persephone):
|
|
exported = client.get(f"/api/adventures/{persephone}/export").json()
|
|
assert exported["format"] == "ai-dnd-adventure-v3"
|
|
copy_id = client.post("/api/adventures/import", json=exported).json()["id"]
|
|
document = client.get(f"/api/adventures/{copy_id}/state").json()["document"]
|
|
assert document["entities"]["persephone"]["type"] == "vehicle"
|
|
assert client.get(f"/api/adventures/{copy_id}").json()["canon_rules"] == CANON
|
|
|
|
|
|
def test_no_state_event_type_is_genre_specific():
|
|
"""J03 as a whole-vocabulary check rather than a spot check.
|
|
|
|
Every accepted event names a structural relationship — an entity, a fact, a
|
|
possession, a location, a thread. None of them names a sword, a spell, a
|
|
spaceship or a corporation.
|
|
"""
|
|
fantasy_or_sf = (
|
|
"spell", "magic", "sword", "potion", "mana", "warp", "hyperspace",
|
|
"laser", "starship", "airlock",
|
|
)
|
|
vocabulary = " ".join(narrative_events.ALLOWED).lower()
|
|
for word in fantasy_or_sf:
|
|
assert word not in vocabulary
|