Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v1.1: harden context window and narrator protocol boundary

Browse Source 
WP-A1 and WP-A2, implemented in sequence, plus the corrective work the owner
asked for at review. Reported in
planning/reports/v1.1/V1.1-WP-A1-A2-REPORT.md (corrective addendum §R).
Planning package v4.2.

WP-A1: context-window safety reserve
- The prompt leaves max(256, ceil(5% of the effective window)) tokens free
  beside the reply. That is 256 at 4,096 and 820 at 16,384. The value is fixed,
  not a setting, and not calibrated per model.
- M6's 64-token margin is gone. Separators and the chat hint are priced
  exactly; tokenizer drift is the reserve's job.
- Protected context that cannot fit raises ContextOverflow before the model
  is called.
- Streams set stream_options.include_usage. Measured on Ollama 0.33, a stream
  sent no usage without it.
- Each sent turn records fits, exceeded, truncation_suspected or unknown.
  The status is returned on the done event, logged when bad, and shown in the
  context inspector. The turn is always kept.
- Accounting is per-attempt data (attempts.ATTEMPT_KEYS).
- Corrective: a cold model is loaded before its turn is built. When the
  window is unverified but the server answered, contextwindow.ensure_window
  makes one bounded POST /api/generate naming only the model. It sends no
  prompt, generates nothing and writes nothing. It then probes again, and the
  turn is built to that answer. If the load fails, or the window is still
  unknown, the turn falls back to the old behaviour.
- Real host, 4,096 window:
  - v1 cold turn: sent 13,875, the server read 2,050.
  - Same turn after the correction: the window was verified, 3,082 sent,
    3,097 read, fits, 499 tokens left beside the reply.
  - Verified turns elsewhere left 275-2,297 tokens against v1's 23-42.

WP-A2: protocol echo and genre-neutral state prompting
- The vocabulary is shown as the JSON object the model sends, not as
  name(field, ...). This costs 121 tokens.
- The example uses character-1, item-1 and location-1.
- The extractor removes shapes anchored to application-owned text:
  - a vocabulary call line;
  - an echoed length hint;
  - the renderer's scene line left last;
  - an empty fence opener.
- Corrective R5: the echoed continue hint is recognised by its own sentence
  ("Output only story text"). A Hard-limit-opened bracket is removed only
  directly above an echo already cut from the same reply.
- Replay of all 518 real v1 replies: 9 changed, 0 flagged, and no story prose
  removed. That is unchanged by R5.
- Replay of 64 v1.1 replies: 3 changed, 0 flagged. The depth-16 instruction
  tail is removed.
- Identity diagnostic after the correction:
  - 0 identity signals;
  - 0 prompt example identifiers proposed;
  - 0/10 stored turns with protocol or instruction shapes.
- 50-turn run: 51 accepted, 0 of 54 stored turns carry protocol.
- SPECS, render.py and validate.py are identical to v1.0.0.

Compatibility: a real v1.0.0 database reads identically on v1.0.0 and v1.1,
field for field, with schema and user_version 94 unchanged. Undo, redo, Save
Point restore, export and import all work on it. There is no schema,
migration or bundle-format change.

Verification: the backend suite passes 1,534 with 17 skipped and 0 failed.
The frontend passes 165/165, and lint and the build are clean. The offline
container and the browser regression were re-run on this tree (see §R.3).

One test was re-calibrated, not weakened: test_history_block_trim's prefix
test had assumed which turn holds the floor at a 2,048 budget.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VvegagkhuCZoFPdv4M1egY
This commit is contained in:
JesseMarkowitz
2026-09-14 16:35:05 -04:00
co-authored by Claude Opus 5
parent ac465ed867
commit d63804f22e
26 changed files with 3903 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/contextwindow.py
+208
View File
@@ -86,6 +86,7 @@ one.
from __future__ import annotations
import logging
import math
import re
import time
from dataclasses import dataclass
@@ -132,6 +133,11 @@ class Window:
model_max: int | None = None
#: Why the window is unknown, or how it was found. Shown to the user.
detail: str = ""
#: v1.1: the server answered a discovery request at all, whatever it said.
#: A server that answered but could not report a window may simply not have
#: the model loaded yet, which `ensure_window` can fix; one that did not
#: answer cannot be helped by asking it to load anything.
reachable: bool = False
@property
def verified(self) -> bool:
@@ -180,6 +186,116 @@ def effective_budget(configured: int, window: Window | int | None) -> int:
return min(configured, tokens)
#: v1.1 WP-A1: the tokens kept free below the effective window, beyond the reply.
#:
#: The builder counts with `cl100k_base`; the narrator counts with its own
#: tokenizer. The v1 evidence put the largest prompts 23-42 real tokens from the
#: edge of a 16,384 window, and Ollama does not refuse a prompt past the edge —
#: measured on Ollama 0.33 at a 4,096 window, a 6,316-token prompt came back 200
#: with `prompt_tokens` 2,050. So the reserve is deliberate and sized to the
#: window: the larger of a floor and a share, **rounded up to a whole token**.
#:
#: 4,096 -> 256 8,192 -> 410 16,384 -> 820
#:
#: A fixed, documented tolerance, owner-chosen for v1.1. It is not a setting and
#: it is not calibrated per model.
SAFETY_RESERVE_FLOOR = 256
SAFETY_RESERVE_PERCENT = 5
def safety_reserve(effective_window: int) -> int:
"""`max(256, ceil(5% of the effective window))`, in tokens.
The effective window is the budget the prompt is actually built to — the
verified or declared window when there is one, the configured budget
otherwise — so a 16,384 setting against a 4,096 server reserves 256, not 820.
Integer arithmetic, so the rounding is exact rather than a float's.
"""
share = math.ceil(max(0, effective_window) * SAFETY_RESERVE_PERCENT / 100)
return max(SAFETY_RESERVE_FLOOR, share)
#: v1.1 WP-A1: what the server's own count says about a turn that was sent.
FITS = "fits"
EXCEEDED = "exceeded"
TRUNCATION_SUSPECTED = "truncation_suspected"
#: `UNKNOWN` above: the server reported no usable count.
def classify_usage(usage: dict | None, *, estimate: int, budget: int,
max_output_tokens: int, window_verified: bool) -> dict:
"""Sets the server's reported prompt count against what the application sent.
The order of the checks is the order of what they prove:
``unknown``
No positive integer `prompt_tokens`. Nothing can be said, and nothing
is claimed: an absent count is never read as a prompt that fitted.
``truncation_suspected``
The server read fewer tokens than were sent by more than the safety
reserve. A tokenizer thriftier than `cl100k_base` may honestly count a
little less; a shortfall larger than the tolerance the application keeps
for drift is the signature of a server that cut the prompt — the real
shape was 6,316 sent and 2,050 read.
``exceeded``
The server's count plus the reply allocation is more than the window
the prompt was built for. The drift was larger than the whole reserve,
so the reply may have been cut short.
``fits``
Otherwise.
`observed_margin` is what was left beside the reply by the server's count:
`budget - max_output_tokens - server_prompt_tokens`. The safety reserve is
the tolerance, so a margin between 0 and the reserve is still `fits`.
A discrepancy is recorded, never acted on: the reply has already streamed
to the reader and is accepted story.
"""
prompt = usage.get("prompt_tokens") if isinstance(usage, dict) else None
reserve = safety_reserve(budget)
verified_note = "" if window_verified else (
" The window itself was not verified for this turn.")
record = {
"status": UNKNOWN,
"server_prompt_tokens": None,
"estimate": estimate,
"difference": None,
"budget": budget,
"max_output_tokens": max_output_tokens,
"safety_reserve": reserve,
"observed_margin": None,
"window_verified": bool(window_verified),
"detail": "",
}
if type(prompt) is not int or prompt <= 0:
record["detail"] = ("The server reported no prompt token count, so nothing "
"confirms the whole prompt was read." + verified_note)
return record
record["server_prompt_tokens"] = prompt
record["difference"] = prompt - estimate
record["observed_margin"] = budget - max_output_tokens - prompt
if prompt + reserve < estimate:
record["status"] = TRUNCATION_SUSPECTED
record["detail"] = (
f"The server read {prompt:,} prompt tokens of the {estimate:,} sent, a "
f"shortfall larger than the {reserve:,}-token safety reserve. A server "
"that cuts an over-window prompt reports exactly this, and what it cuts "
"is the start: the narrator's rules and the canon." + verified_note)
elif prompt + max_output_tokens > budget:
record["status"] = EXCEEDED
record["detail"] = (
f"The server counted {prompt:,} prompt tokens; with {max_output_tokens:,} "
f"for the reply that is more than the {budget:,}-token window the prompt "
"was built for, so the reply may have been cut short." + verified_note)
else:
record["status"] = FITS
record["detail"] = (
f"The server read {prompt:,} prompt tokens, leaving "
f"{record['observed_margin']:,} beside the reply." + verified_note)
return record
def cache_clear() -> None:
"""Forgets what was learned. Called when the endpoint or model changes."""
_cache.clear()
@@ -223,6 +339,7 @@ def _declared_or(declared: int | None, discovered: Window) -> Window:
declared, DECLARED, discovered.model_max,
f"{declared:,} tokens, declared in settings — the server was not able "
f"to say ({discovered.detail})",
reachable=discovered.reachable,
)
@@ -266,6 +383,7 @@ async def _ask(endpoint_url: str, model: str) -> Window:
return Window(
tokens, LOADED, ceiling,
f"{tokens:,} tokens, reported by the running model",
reachable=True,
)
return await _declared_window(client, base, model)
except (httpx.HTTPError, ValueError, TypeError, KeyError) as exc:
@@ -293,6 +411,7 @@ async def _declared_window(client, base: str, model: str) -> Window:
return Window(
None, UNKNOWN,
detail=f"the server did not describe the model (HTTP {resp.status_code})",
reachable=True,
)
body = resp.json() or {}
ceiling = _architecture_ceiling(body.get("model_info") or {})
@@ -304,14 +423,103 @@ async def _declared_window(client, base: str, model: str) -> Window:
"the model sets no num_ctx, so the server will load it at its own "
"default — which is 4,096 where there is no VRAM"
),
reachable=True,
)
tokens = min(declared, ceiling) if ceiling else declared
return Window(
tokens, PARAMETERS, ceiling,
f"{tokens:,} tokens, from the model's own num_ctx",
reachable=True,
)
#: v1.1 WP-A1 corrective: loading the configured model so its window can be read.
#:
#: The first real turn of the A1 evidence found a cold model: `/api/ps` knew
#: nothing, `/api/show` found no `num_ctx`, so the window was unverified and the
#: prompt was built to the configured 16,384. Ollama loaded the model at its own
#: 4,096 default, kept 2,050 of 13,875 tokens and answered 200. That case is
#: preventable, because the window becomes readable the moment the model is
#: resident. Ollama's native `POST /api/generate` with a model and **no prompt**
#: loads the model and generates nothing — measured on Ollama 0.33: HTTP 200,
#: `"response": ""`, `"done_reason": "load"`, and `/api/ps` then reported the
#: window. The OpenAI-compatible request that followed did not reload it.
WARM_PATH = "/api/generate"
async def warm(endpoint_url: str, model: str, *, timeout: float) -> tuple[bool, str]:
"""Asks the configured server to load `model`. One request, no story text.
Held to the same endpoint policy and TLS trust as inference and the probe, and
sent to the same host the probe asks. The body names the model and nothing
else: no prompt, so nothing is generated, and no `options` or `keep_alive`, so
the model loads the way the server would load it for the turn itself.
Returns `(loaded, detail)`. Every failure is `(False, why)` and never raises:
a server that will not load the model on request will fail the turn's own
call the ordinary way, which is where that failure belongs.
"""
reason = endpoints.rejection_reason(endpoint_url)
if reason is not None:
return False, f"endpoint not allowed — {reason}"
base = native_base(endpoint_url)
try:
async with httpx.AsyncClient(
timeout=httpx.Timeout(timeout, connect=CONNECT_TIMEOUT),
verify=tlstrust.ssl_context(),
) as client:
resp = await client.post(f"{base}{WARM_PATH}", json={"model": model})
except httpx.HTTPError as exc:
log.debug("model warm-up failed for %s: %s", base, exc)
return False, f"could not ask the server to load the model ({type(exc).__name__})"
if resp.status_code != 200:
return False, f"the server did not load the model (HTTP {resp.status_code})"
try:
body = resp.json() or {}
except ValueError:
return False, "the server answered the load request with something that was not JSON"
return True, f"the server loaded the model ({body.get('done_reason') or 'done'})"
async def ensure_window(endpoint_url: str, model: str, *, declared: int | None = None,
warm_timeout: float = 300.0) -> tuple[Window, dict]:
"""The window for a turn about to be generated, loading the model once if that is what it takes.
1. Probe as before.
2. If the window is not verified, the server answered, and there is a model to
load: one bounded `warm` request.
3. If the model loaded, probe again, bypassing the cache that still holds the
unverified answer.
Whatever the second probe says is the answer. There is no retry loop, no
guessed window, and no hard-coded 4,096: a window still unverified leaves the
configured budget standing, exactly as before, and the turn's accounting
still catches a server that cut the prompt.
Returns the window and a `preflight` record for the turn's provenance.
Not used by the context dry run: loading a model is a side effect, and
opening a panel should not cause one.
"""
window = await probe(endpoint_url, model, declared=declared)
preflight = {"attempted": False, "loaded": None, "verified_before": window.verified,
"verified_after": window.verified, "detail": ""}
if window.verified:
preflight["detail"] = "the window was already verified"
return window, preflight
if not (endpoint_url and model):
preflight["detail"] = "no endpoint or model configured"
return window, preflight
if not window.reachable:
preflight["detail"] = "the server did not answer, so no model was loaded"
return window, preflight
loaded, detail = await warm(endpoint_url, model, timeout=warm_timeout)
preflight.update(attempted=True, loaded=loaded, detail=detail)
if loaded:
window = await probe(endpoint_url, model, declared=declared, use_cache=False)
preflight["verified_after"] = window.verified
return window, preflight
def _num_ctx(parameters) -> int | None:
"""Reads `num_ctx` out of the plain-text parameter block Ollama returns."""
if not isinstance(parameters, str):