JesseMarkowitz
8c65ae99de
94 files, +1,395 -6,578. Three files are new; twenty-four are gone. The milestone is subtraction, and what is left is the single-user local storyteller the specification describes. Removed in full: campaign scripting and its QuickJS sandbox; multi-user accounts, guest sessions, login, registration and the shared demo key; the visitor-analytics tables, dashboard and page beacon; the access log of sign-ins, addresses and devices; per-IP and per-user rate limiting and quotas; Render deployment config; Postgres and psycopg; cloud inference providers, the API-key field and the key encryption that existed to store it; session-cookie signing. None of it was hidden behind a flag — the routes are gone and answer 404. Two things were kept that the brief allowed keeping. The `users` table and its foreign keys stay as an internal ownership detail, because rewriting them out means a migration across most of the schema to delete a column that costs nothing; nothing creates a second user and no request carries an identity. Five inert tables and four inert columns stay for the same reason, so an M1 campaign database opens unchanged. The one addition is app/endpoints.py, which decides where a story may be sent. Loopback, RFC1918, link-local, unique-local and CGNAT — an explicit allowlist of networks, not a guess at what `ipaddress` means by "private", which calls the documentation ranges private and IPv6 loopback reserved. Every address a hostname resolves to must be in it, so a split answer does not squeak through, and the rule runs both when the endpoint is saved and before every outbound request, because a name that resolved to the LAN this morning can resolve elsewhere this afternoon. Known cloud hosts are named in the refusal so the error says why rather than looking like broken DNS. TLS is never traded against it: M1's shared trust context is intact on all four clients and there is no way to skip verification. The hardcoded 120-second model timeout is now a setting. That was not theoretical — on this GPU-less four-core host a cold load of qwen2.5:3b-instruct took 648.9 seconds to produce the first turn, while turns 2 to 5 of the same campaign took 3.6 to 13.1. Connect stays short at 10s so a wrong address still fails fast; the read timeout defaults to 300s and is bounded at 3600, because "wait longer" must stay a number. Two defects found while testing and fixed here. An unknown /api path fell through the SPA catch-all and came back as HTML with status 200, so a client asking for JSON parsed a web page instead of learning the route was gone. And AIDND_CORS_ORIGINS accepted "*", which on an unauthenticated loopback API would hand every page on the Internet a write handle on the campaign database; it now refuses to start. Verified rather than assumed. Offline, on a network with no route out and no DNS: five turns, retry with both takes retained, restart with an identical transcript digest, a failed model call leaving the accepted AI-turn count untouched, and a capture with zero non-loopback unicast packets. Against a real second machine on the LAN over HTTPS with a private CA: four turns, restart, and a capture showing 289 packets to the approved host, 344 loopback, zero anywhere else, zero DNS queries. Cloud and public endpoints refused with their reasons; no API key settable; every removed route 404. 604 backend tests pass, down from 648 by the fifteen retired with the subsystems they tested and up by the twenty-nine added for the endpoint policy and the removed surface. The scripting tests were not deleted: eight files used a JavaScript counter as instrumentation for the state snapshot and rollback machinery, which M2 does not touch, so the counter moved to the world-state engine and those tests still assert what they always did. Frontend lint and build are clean; the image builds, and its wheel-building stage is gone with quickjs. No M3 work. Undo is still destructive and there is still no Redo.
208 lines
8.5 KiB
Python
208 lines
8.5 KiB
Python
"""Prompt caching: the prompt has to start with the same bytes every turn.
|
|
|
|
Every endpoint that caches prompts caches a prefix. It reuses the request
|
|
up to the first byte that differs from last time, and no further. The cost
|
|
of a turn is therefore decided by layout: one section that changes each
|
|
turn, placed near the top, re-prices everything underneath it, and
|
|
underneath it is the story history, which makes up most of the prompt.
|
|
|
|
Three things must hold, and each is easy to undo by accident:
|
|
|
|
* The static block is byte-identical across turns. Adding a section that
|
|
moves (live stats, retrieved memories, a rewritten summary) to
|
|
`system_sections` is the mistake this file exists to catch.
|
|
* The sections that move sit after the history, but still before the tail
|
|
that is last for its own reasons: front memory, the length hint, and
|
|
`EMIT_REMINDER`, which is what keeps the state block emitted at all.
|
|
* Moving a section out of the system block does not drop it from the
|
|
token budget. It is still in the prompt.
|
|
|
|
This file also covers one request-level concern: reading back the usage
|
|
the endpoint reports, so the hit rate is measurable rather than assumed.
|
|
It covered OpenRouter upstream pinning too, until M2 removed cloud
|
|
provider support.
|
|
|
|
python -m pytest tests/test_prompt_caching.py -v
|
|
"""
|
|
import os
|
|
|
|
|
|
import pytest
|
|
|
|
from app import models, worldstate
|
|
from app.context import builder
|
|
from app.database import Base, SessionLocal, engine
|
|
from app.providers.openai_compatible import OpenAICompatibleProvider
|
|
|
|
SCHEMA = {
|
|
"player": {"hp": {"min": 0, "max": 100, "initial": 100, "desc": "Health"}},
|
|
"world": {"alarm": {"min": 0, "max": 10, "initial": 0, "desc": "Alarm level"}},
|
|
}
|
|
|
|
|
|
def test_usage_is_recorded_from_a_final_chunk():
|
|
"""In a stream, the usage block arrives in a final chunk that carries
|
|
no choices, which is why it is read separately from the text
|
|
extraction."""
|
|
provider = OpenAICompatibleProvider("http://127.0.0.1:11434/v1", "m")
|
|
assert provider.last_usage is None
|
|
provider._record_usage({"choices": [{"delta": {"content": "hi"}}]})
|
|
assert provider.last_usage is None
|
|
provider._record_usage(
|
|
{"choices": [], "usage": {"prompt_tokens": 900,
|
|
"prompt_tokens_details": {"cached_tokens": 768}}}
|
|
)
|
|
assert provider.last_usage["prompt_tokens_details"]["cached_tokens"] == 768
|
|
|
|
|
|
def test_a_later_chunk_without_usage_does_not_erase_it():
|
|
provider = OpenAICompatibleProvider("http://127.0.0.1:11434/v1", "m")
|
|
provider._record_usage({"usage": {"prompt_tokens": 5}})
|
|
provider._record_usage({"choices": [{"delta": {"content": "x"}}]})
|
|
provider._record_usage({"usage": {}})
|
|
assert provider.last_usage == {"prompt_tokens": 5}
|
|
|
|
|
|
# ------------------------------------------------------------ prompt layout
|
|
|
|
def _with_hp(world_state, hp):
|
|
"""`world_state` is nested by group, and the JSON column only detects a
|
|
whole new object. Build a new one instead of mutating in place."""
|
|
return {**world_state, "player": {**world_state["player"], "hp": hp}}
|
|
|
|
|
|
@pytest.fixture()
|
|
def story():
|
|
Base.metadata.create_all(bind=engine)
|
|
db = SessionLocal()
|
|
user = models.User(is_guest=False, email="cache@example.com")
|
|
db.add(user)
|
|
db.flush()
|
|
settings = models.Settings(user_id=user.id, api_key="enc:dummy", model="m")
|
|
db.add(settings)
|
|
scenario = models.Scenario(
|
|
user_id=user.id, title="S", prompt="A road.", stat_schema=SCHEMA
|
|
)
|
|
db.add(scenario)
|
|
db.flush()
|
|
adventure = models.Adventure(
|
|
user_id=user.id, title="A", scenario_id=scenario.id, script_state={},
|
|
memory="The hero hunts bandits.",
|
|
ai_instructions="Write in second person.",
|
|
story_summary="The hero left the village.",
|
|
world_state=worldstate.instantiate(SCHEMA),
|
|
# Phase 18. Set here so that every test in this file runs with a
|
|
# persona present: it is user-only, so it belongs in the static block,
|
|
# and this is the file that guards what may live there.
|
|
persona_name="Kaelen",
|
|
persona_pronouns="he/him",
|
|
persona_desc="A half-elf ranger.",
|
|
)
|
|
db.add(adventure)
|
|
db.flush()
|
|
for i in range(6):
|
|
db.add(models.Action(adventure_id=adventure.id,
|
|
type="ai" if i % 2 else "do",
|
|
text=f"[{i}] The road bends onward past the treeline."))
|
|
db.commit()
|
|
db.expire_all()
|
|
adventure = db.get(models.Adventure, adventure.id)
|
|
settings = db.get(models.Settings, settings.id)
|
|
try:
|
|
yield db, adventure, settings
|
|
finally:
|
|
db.close()
|
|
Base.metadata.drop_all(bind=engine)
|
|
|
|
|
|
def test_changing_a_stat_leaves_the_static_block_untouched(story):
|
|
"""The whole point. Live values used to sit third from the top, so a
|
|
single point of damage re-priced the instructions, the plot, and the
|
|
history."""
|
|
db, adventure, settings = story
|
|
before, _, _ = builder.build_context(adventure, settings)
|
|
adventure.world_state = _with_hp(adventure.world_state, 40)
|
|
db.commit()
|
|
after, story_text, _ = builder.build_context(adventure, settings)
|
|
assert before == after
|
|
assert "hp 40/100" in story_text, "the new value still has to reach the model"
|
|
|
|
|
|
def test_the_static_block_holds_the_things_that_do_not_move(story):
|
|
db, adventure, settings = story
|
|
system_text, story_text, _ = builder.build_context(adventure, settings)
|
|
for fixed in ("Write in second person.", "The hero hunts bandits.",
|
|
"You are Kaelen (he/him).", "A half-elf ranger."):
|
|
assert fixed in system_text
|
|
# The stat guide is derived from the schema, so it is fixed. The live
|
|
# values it describes are not fixed, and belong to the story text.
|
|
assert "Stat guide" in system_text
|
|
for moves in ("The hero left the village.", "hp 100/100"):
|
|
assert moves not in system_text
|
|
assert moves in story_text
|
|
|
|
|
|
def test_volatile_sections_sit_after_the_history(story):
|
|
db, adventure, settings = story
|
|
_, story_text, _ = builder.build_context(adventure, settings)
|
|
history_at = story_text.index("[5] The road bends")
|
|
for label in ("Story summary:", "World state"):
|
|
assert story_text.index(label) > history_at, label
|
|
|
|
|
|
def test_the_tail_stays_the_tail(story):
|
|
"""Front memory, the length hint, and `EMIT_REMINDER` are last for
|
|
reasons of their own, and the live sections must not have displaced
|
|
them."""
|
|
db, adventure, settings = story
|
|
_, story_text, report = builder.build_context(adventure, settings)
|
|
labels = [s["label"] for s in report["sections"]]
|
|
assert labels[-1] == "world_state_reminder"
|
|
assert labels[-2] == "length_hint"
|
|
assert labels.index("world_state") < labels.index("length_hint")
|
|
assert story_text.rstrip().endswith(worldstate.EMIT_REMINDER.rstrip())
|
|
|
|
|
|
def test_live_sections_are_still_charged_to_the_budget(story):
|
|
"""They moved out of `system_sections`, so it would be easy to stop
|
|
counting them in `reserved`. If that happened, the history, which is
|
|
budgeted with what is left over, would quietly overrun."""
|
|
db, adventure, settings = story
|
|
for i in range(6, 90):
|
|
db.add(models.Action(
|
|
adventure_id=adventure.id, type="do",
|
|
text=f"[{i}] " + "The road bends onward past the treeline. " * 6,
|
|
))
|
|
settings.context_token_budget = 4000
|
|
db.commit()
|
|
db.expire_all()
|
|
adventure = db.get(models.Adventure, adventure.id)
|
|
settings = db.get(models.Settings, settings.id)
|
|
|
|
_, _, lean = builder.build_context(adventure, settings)
|
|
adventure.story_summary = "The hero left the village. " * 150
|
|
db.commit()
|
|
_, _, fat = builder.build_context(adventure, settings)
|
|
|
|
assert fat["history"]["included"] < lean["history"]["included"], (
|
|
"a bigger summary has to leave less room for history"
|
|
)
|
|
|
|
|
|
def test_a_new_turn_only_appends_to_the_cached_prefix(story):
|
|
"""Playing on must extend the previous prompt, not rewrite it: the shared
|
|
prefix has to still contain the whole static block and the older history."""
|
|
db, adventure, settings = story
|
|
system_a, story_a, _ = builder.build_context(adventure, settings)
|
|
db.add(models.Action(adventure_id=adventure.id, type="do",
|
|
text="[6] You step into the clearing."))
|
|
db.commit()
|
|
db.expire_all()
|
|
adventure = db.get(models.Adventure, adventure.id)
|
|
system_b, story_b, _ = builder.build_context(adventure, settings)
|
|
|
|
assert system_a == system_b
|
|
shared = os.path.commonprefix([story_a, story_b])
|
|
assert "[0] The road bends" in shared
|
|
assert "[5] The road bends" in shared
|