Rewrite Python comments in Google developer documentation style (#12)
* Rewrite comments in Google developer documentation style Rewrite the comments and docstrings across the backend core modules so they read plainly. The previous prose was accurate but dense and figurative, which made it slow to skim. Applies the Google developer documentation style guide: short sentences, active voice, present tense, American spelling, and no metaphors, idioms, or rhetorical asides. Replaces em-dash chains with separate sentences.
This commit is contained in:
+45
-37
@@ -3,25 +3,25 @@
|
|||||||
The deliberate opposite of analytics.py. That module counts and stores nothing
|
The deliberate opposite of analytics.py. That module counts and stores nothing
|
||||||
that points at a person; this one records addresses, email addresses and
|
that points at a person; this one records addresses, email addresses and
|
||||||
devices, because an access log that cannot identify the access is not an access
|
devices, because an access log that cannot identify the access is not an access
|
||||||
log. They are kept in separate modules and separate tables on purpose — the
|
log. The two live in separate modules and separate tables on purpose, so that
|
||||||
anonymity of the counters is then a property of the code rather than of a
|
the anonymity of the counters is a property of the code rather than a convention
|
||||||
convention someone has to remember.
|
someone has to remember.
|
||||||
|
|
||||||
Owner-only, and never shown to the people it records.
|
Owner-only, and never shown to the people it records.
|
||||||
|
|
||||||
Four kinds of row:
|
Four kinds of row:
|
||||||
|
|
||||||
- `session` a browser that has a session made a request — for a guest,
|
- `session` A browser that has a session made a request. For a guest, this
|
||||||
their first visit;
|
is their first visit.
|
||||||
- `login` an existing account signed in;
|
- `login` An existing account signed in.
|
||||||
- `register` a guest upgraded to an account;
|
- `register` A guest upgraded to an account.
|
||||||
- `login_failed` a password attempt that didn't match, with the address tried.
|
- `login_failed` A password attempt that did not match, with the address tried.
|
||||||
|
|
||||||
Session rows are the only ones that need thinning: `/auth/me` runs on every
|
Session rows are the only ones that need thinning. `/auth/me` runs on every page
|
||||||
page load, and a row per load would be noise rather than a log. One is written
|
load, and one row per load would be noise rather than a log. A row is written
|
||||||
when the day or the address changes for that user, which is the granularity a
|
when the day or the address changes for that user. That is the granularity a log
|
||||||
log is actually read at — "seen on the 3rd from 1.2.3.4" — and it still catches
|
is read at, such as seen on the 3rd from 1.2.3.4, and it still records someone
|
||||||
someone moving networks mid-day.
|
moving networks during a day.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
@@ -50,31 +50,36 @@ _MAX_TRACKED = 10_000
|
|||||||
|
|
||||||
|
|
||||||
def _client_ip(request) -> str:
|
def _client_ip(request) -> str:
|
||||||
# Deferred: limits imports auth, which is imported by the routers that call
|
# This import is deferred. `limits` imports `auth`, which the routers that
|
||||||
# this, so a module-level import here would close the loop. The spoof
|
# call this function import, so a module-level import here would create a
|
||||||
# resistance lives there and must not be reimplemented — a second, laxer
|
# cycle. The spoof resistance lives in `limits` and must not be
|
||||||
# copy of "what is the client's address" is exactly how one of them ends up
|
# reimplemented. A second, looser answer to which address belongs to the
|
||||||
# trusting a header it shouldn't.
|
# client is how one of them ends up trusting a header it should not.
|
||||||
from . import limits
|
from . import limits
|
||||||
|
|
||||||
return limits.client_ip(request)
|
return limits.client_ip(request)
|
||||||
|
|
||||||
|
|
||||||
def describe(user: models.User) -> str:
|
def describe(user: models.User) -> str:
|
||||||
"""How a user is named in the log. Guests have no email, and their id is
|
"""Returns how a user is named in the log.
|
||||||
the only handle anyone has for them. The third case is a local install's
|
|
||||||
implicit single user, which is also email-less but is the operator rather
|
A guest has no email, and their id is the only handle anyone has for them.
|
||||||
than a visitor — calling that one "Guest #1" would be a small lie in the
|
The third case is a local install's implicit single user, who also has no
|
||||||
one row they are certain to read."""
|
email but is the operator rather than a visitor. Naming that user "Guest #1"
|
||||||
|
would be wrong in the one row they are certain to read.
|
||||||
|
"""
|
||||||
if user.email:
|
if user.email:
|
||||||
return user.email
|
return user.email
|
||||||
return f"Guest #{user.id}" if user.is_guest else f"Local user #{user.id}"
|
return f"Guest #{user.id}" if user.is_guest else f"Local user #{user.id}"
|
||||||
|
|
||||||
|
|
||||||
def _country(request) -> str:
|
def _country(request) -> str:
|
||||||
"""The edge's country header, or "" when there isn't one. Blank rather than
|
"""Returns the edge's country header, or "" when there is none.
|
||||||
the counters' "(unknown)" label: a table column reads better as an em dash
|
|
||||||
than as a word, and an empty string is the honest value for "not known"."""
|
The blank differs from the counters' "(unknown)" label. A table column reads
|
||||||
|
better as a dash than as a word, and an empty string is the correct value for
|
||||||
|
a country that is not known.
|
||||||
|
"""
|
||||||
country = analytics.country_of(request.headers)
|
country = analytics.country_of(request.headers)
|
||||||
return "" if country == analytics.UNKNOWN else country
|
return "" if country == analytics.UNKNOWN else country
|
||||||
|
|
||||||
@@ -87,8 +92,11 @@ def record(
|
|||||||
user: models.User | None = None,
|
user: models.User | None = None,
|
||||||
who: str | None = None,
|
who: str | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Write one row. Never raises: the log watches sign-in, it doesn't guard
|
"""Writes one row.
|
||||||
it, and a logging failure must not be able to lock anyone out."""
|
|
||||||
|
This function never raises. The log observes sign-in rather than guarding it,
|
||||||
|
and a logging failure must not lock anyone out.
|
||||||
|
"""
|
||||||
try:
|
try:
|
||||||
event = models.AccessEvent(
|
event = models.AccessEvent(
|
||||||
kind=kind,
|
kind=kind,
|
||||||
@@ -108,7 +116,7 @@ def record(
|
|||||||
|
|
||||||
|
|
||||||
def note_session(db: Session, user: models.User, request) -> None:
|
def note_session(db: Session, user: models.User, request) -> None:
|
||||||
"""A session made a request. Thinned to one row per day per address."""
|
"""Records that a session made a request, at most one row per day per address."""
|
||||||
try:
|
try:
|
||||||
today = analytics._today()
|
today = analytics._today()
|
||||||
ip = _client_ip(request)
|
ip = _client_ip(request)
|
||||||
@@ -117,8 +125,8 @@ def note_session(db: Session, user: models.User, request) -> None:
|
|||||||
return
|
return
|
||||||
_last_session[user.id] = (today, ip)
|
_last_session[user.id] = (today, ip)
|
||||||
if len(_last_session) > _MAX_TRACKED:
|
if len(_last_session) > _MAX_TRACKED:
|
||||||
# Nothing here is worth persisting; dropping the map costs at
|
# Nothing here needs to persist. Clearing the map costs at most
|
||||||
# most one extra row per active user.
|
# one extra row per active user.
|
||||||
_last_session.clear()
|
_last_session.clear()
|
||||||
_last_session[user.id] = (today, ip)
|
_last_session[user.id] = (today, ip)
|
||||||
except Exception: # pragma: no cover - defensive
|
except Exception: # pragma: no cover - defensive
|
||||||
@@ -135,11 +143,11 @@ def recent(
|
|||||||
kind: str | None = None,
|
kind: str | None = None,
|
||||||
query: str | None = None,
|
query: str | None = None,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""A page of the log, newest first.
|
"""Returns a page of the log, newest first.
|
||||||
|
|
||||||
Anchored on a row id rather than an offset, like the story pager: rows keep
|
The page is anchored on a row id rather than an offset, as the story pager
|
||||||
arriving while it is being read, and an offset would shift the page under
|
is. Rows keep arriving while the log is read, and an offset would shift the
|
||||||
whoever is reading it.
|
page under whoever is reading it.
|
||||||
"""
|
"""
|
||||||
statement = select(models.AccessEvent).order_by(desc(models.AccessEvent.id))
|
statement = select(models.AccessEvent).order_by(desc(models.AccessEvent.id))
|
||||||
if before_id is not None:
|
if before_id is not None:
|
||||||
@@ -153,8 +161,8 @@ def recent(
|
|||||||
models.AccessEvent.ip.ilike(like),
|
models.AccessEvent.ip.ilike(like),
|
||||||
models.AccessEvent.country.ilike(like),
|
models.AccessEvent.country.ilike(like),
|
||||||
))
|
))
|
||||||
# One extra row answers "is there more" without a second COUNT over the
|
# Requesting one extra row reports whether more rows exist, without a
|
||||||
# whole table.
|
# second COUNT over the whole table.
|
||||||
rows = list(db.scalars(statement.limit(limit + 1)))
|
rows = list(db.scalars(statement.limit(limit + 1)))
|
||||||
has_more = len(rows) > limit
|
has_more = len(rows) > limit
|
||||||
return {"events": rows[:limit], "has_more": has_more}
|
return {"events": rows[:limit], "has_more": has_more}
|
||||||
|
|||||||
+149
-122
@@ -1,34 +1,38 @@
|
|||||||
"""Visit analytics for the hosted demo.
|
"""Visit analytics for the hosted demo.
|
||||||
|
|
||||||
A small self-hosted counter answering "did anyone visit, and did they play?",
|
This is a small self-hosted counter that answers whether anyone visited and
|
||||||
built into the app rather than bolted on with a third-party script: the CSP in
|
whether they played. It is built into the app rather than added with a
|
||||||
main.py allows scripts from 'self' only, adblockers eat the popular trackers,
|
third-party script, because the CSP in `main.py` allows scripts from 'self'
|
||||||
and none of them can see the things actually worth knowing here (turns taken,
|
only, ad blockers block the popular trackers, and none of those trackers can see
|
||||||
demo-key spend, which seeded scenario people pick).
|
what is worth knowing here: turns taken, demo-key spend, and which seeded
|
||||||
|
scenario people pick.
|
||||||
|
|
||||||
Three rules shaped it:
|
Three rules shape the design:
|
||||||
|
|
||||||
1. **Nothing personal is stored.** No IP addresses, no user agents, no user
|
1. It stores nothing personal. It records no IP addresses, no user agents, no
|
||||||
ids, no title of anything a player wrote. A visitor appears only as an HMAC
|
user ids, and no title of anything a player wrote. A visitor appears only as
|
||||||
of their user id — one-way and salted with the app's secret key, so these
|
an HMAC of their user id, which is one-way and salted with the app's secret
|
||||||
tables cannot be joined back to an account even by someone holding the
|
key, so these tables cannot be joined back to an account even by someone
|
||||||
database. Story content never reaches this module at all. What a *specific*
|
holding the database. Story content never reaches this module. What one
|
||||||
person did is deliberately unanswerable; only totals are.
|
specific person did is unanswerable by design, and only totals are
|
||||||
2. **Egress is the budget.** Neon bills for bytes leaving the database and this
|
available.
|
||||||
project has already paid for forgetting that once. So counts are aggregated
|
2. Egress is the budget. Neon bills for bytes leaving the database, and this
|
||||||
in memory and flushed as UPSERTs — a visit is a write, never a read — and
|
project has already paid for forgetting that once. Counts are therefore
|
||||||
every dashboard query is a GROUP BY returning tens of rows, never per-visit
|
aggregated in memory and flushed as UPSERTs, so a visit is a write and never
|
||||||
rows. A month of traffic costs a few kilobytes to read back.
|
a read, and every dashboard query is a GROUP BY that returns tens of rows
|
||||||
3. **The numbers are the server's, not the browser's.** The client reports one
|
rather than per-visit rows. A month of traffic costs a few kilobytes to read
|
||||||
thing: which page was viewed. Everything that *means* something ("a turn
|
back.
|
||||||
happened", "an account was created") is recorded by the code that does it,
|
3. The numbers come from the server, not from the browser. The client reports
|
||||||
where it can be neither faked by a stranger nor blocked by an extension.
|
one thing, which is the page that was viewed. Everything with meaning, such
|
||||||
|
as a turn happening or an account being created, is recorded by the code that
|
||||||
|
performs it, where a stranger cannot fake it and an extension cannot block
|
||||||
|
it.
|
||||||
|
|
||||||
Storage is two tables, both bounded. `analytics_daily` is one row per (day,
|
Storage is two tables, both bounded. `analytics_daily` holds one counter row per
|
||||||
metric, label) counter — a few dozen a day. `analytics_visitor_days` is one row
|
day, metric, and label, which is a few dozen rows a day.
|
||||||
per visitor per day carrying the funnel flags, which is what makes the funnel
|
`analytics_visitor_days` holds one row per visitor per day carrying the funnel
|
||||||
count *people* rather than clicks; it is the only table that grows with traffic
|
flags, which is what makes the funnel count people rather than clicks. It is the
|
||||||
and cleanup ages it out.
|
only table that grows with traffic, and cleanup ages it out.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import hmac
|
import hmac
|
||||||
@@ -50,31 +54,31 @@ from . import models, security
|
|||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
# ---------- Metrics ----------
|
# ---------- Metrics ----------
|
||||||
# `metric` is the family, `label` the bucket within it. One generic counter
|
# `metric` is the family, and `label` is the bucket within it. One generic
|
||||||
# table beats a column per thing measured: adding a new question later is a
|
# counter table is better than a column per measurement, because adding a new
|
||||||
# constant, not a migration.
|
# question later costs nothing rather than a migration.
|
||||||
|
|
||||||
M_PAGE = "pageview"
|
M_PAGE = "pageview"
|
||||||
M_EVENT = "event"
|
M_EVENT = "event"
|
||||||
M_REFERRER = "referrer"
|
M_REFERRER = "referrer"
|
||||||
M_DEVICE = "device"
|
M_DEVICE = "device"
|
||||||
M_COUNTRY = "country"
|
M_COUNTRY = "country"
|
||||||
M_SCENARIO = "scenario" # which seeded/public scenario got played
|
M_SCENARIO = "scenario" # Which seeded or public scenario was played.
|
||||||
M_ERROR = "error" # "<status> <route>" for 4xx/5xx on /api
|
M_ERROR = "error" # "<status> <route>" for a 4xx or 5xx on /api.
|
||||||
|
|
||||||
EV_SCENARIO_OPEN = "scenario_opened"
|
EV_SCENARIO_OPEN = "scenario_opened"
|
||||||
EV_ADVENTURE = "adventure_created"
|
EV_ADVENTURE = "adventure_created"
|
||||||
EV_IMPORT = "adventure_imported"
|
EV_IMPORT = "adventure_imported"
|
||||||
EV_TURN = "turn"
|
EV_TURN = "turn"
|
||||||
EV_DEMO_TURN = "demo_turn" # a turn billed to the shared demo key
|
EV_DEMO_TURN = "demo_turn" # A turn billed to the shared demo key.
|
||||||
EV_TURN_ERROR = "turn_error"
|
EV_TURN_ERROR = "turn_error"
|
||||||
EV_SIGNUP = "signup"
|
EV_SIGNUP = "signup"
|
||||||
EV_LOGIN = "login"
|
EV_LOGIN = "login"
|
||||||
|
|
||||||
# Events that are also funnel steps: recording one flips a flag on the
|
# Events that are also funnel steps. Recording one sets a flag on the visitor's
|
||||||
# visitor's row for the day, so the funnel counts distinct people-days instead
|
# row for the day, so the funnel counts distinct visitor-days rather than repeat
|
||||||
# of repeat clicks. The name -> column map is the whole definition of the
|
# clicks. This name-to-column map is the whole definition of the funnel, and the
|
||||||
# funnel; the dashboard reads it back in this order.
|
# dashboard reads it back in this order.
|
||||||
FUNNEL_FLAGS = {
|
FUNNEL_FLAGS = {
|
||||||
EV_SCENARIO_OPEN: "opened",
|
EV_SCENARIO_OPEN: "opened",
|
||||||
EV_ADVENTURE: "created",
|
EV_ADVENTURE: "created",
|
||||||
@@ -87,44 +91,45 @@ NONE_LABEL = "(direct)"
|
|||||||
UNKNOWN = "(unknown)"
|
UNKNOWN = "(unknown)"
|
||||||
|
|
||||||
# ---------- Bounds ----------
|
# ---------- Bounds ----------
|
||||||
# Everything below exists so a hostile visitor can add rows to these tables no
|
# These bounds exist so that a hostile visitor can add rows to these tables no
|
||||||
# faster than an honest one. The only label a client can influence is the
|
# faster than an honest one. The only label a client can influence is the
|
||||||
# referrer, and these caps together mean the worst it can do is fill one day's
|
# referrer, and together these caps mean the worst it can do is fill one day's
|
||||||
# referrer list with junk and then be folded into "(other)".
|
# referrer list and then be folded into "(other)".
|
||||||
|
|
||||||
MAX_LABEL_LEN = 80
|
MAX_LABEL_LEN = 80
|
||||||
MAX_LABELS_PER_METRIC = 200 # distinct labels per metric per day, then OTHER
|
MAX_LABELS_PER_METRIC = 200 # Distinct labels per metric per day, then OTHER.
|
||||||
MAX_PENDING = 4000 # buffered entries before an inline flush
|
MAX_PENDING = 4000 # Buffered entries before an inline flush.
|
||||||
FLUSH_INTERVAL_SECONDS = 60
|
FLUSH_INTERVAL_SECONDS = 60
|
||||||
|
|
||||||
# How long the per-visitor-day rows are kept. The daily counters are tiny and
|
# How long the per-visitor-day rows are kept. The daily counters are small and
|
||||||
# kept forever; these are the ones that scale with traffic. A visitor whose
|
# are kept indefinitely. These rows are the ones that scale with traffic. A
|
||||||
# last visit falls off the end counts as new again — a fair trade at this
|
# visitor whose last visit ages out counts as new again, which is an acceptable
|
||||||
# horizon, and it keeps the table from being a permanent record of anyone.
|
# trade at this horizon and keeps the table from being a permanent record of
|
||||||
|
# anyone.
|
||||||
RETENTION_DAYS = int(os.environ.get("AIDND_ANALYTICS_RETENTION_DAYS", "400") or 400)
|
RETENTION_DAYS = int(os.environ.get("AIDND_ANALYTICS_RETENTION_DAYS", "400") or 400)
|
||||||
|
|
||||||
_HOST_OK = re.compile(r"^[a-z0-9.-]+$")
|
_HOST_OK = re.compile(r"^[a-z0-9.-]+$")
|
||||||
_COUNTRY_OK = re.compile(r"^[A-Z]{2}$")
|
_COUNTRY_OK = re.compile(r"^[A-Z]{2}$")
|
||||||
_NUMERIC_SEGMENT = re.compile(r"^\d+$")
|
_NUMERIC_SEGMENT = re.compile(r"^\d+$")
|
||||||
|
|
||||||
# SPA routes, in the shape the dashboard should show them. Anything else a
|
# SPA routes, in the form the dashboard shows them. Any other path a client
|
||||||
# client claims to have viewed becomes OTHER, so the page list can neither be
|
# reports becomes OTHER, so the page list cannot be filled with junk and cannot
|
||||||
# polluted nor accidentally record which adventure someone is reading.
|
# record which adventure someone is reading.
|
||||||
KNOWN_ROUTES = {
|
KNOWN_ROUTES = {
|
||||||
"/", "/adventures", "/scenarios", "/scenarios/:id", "/play/:id",
|
"/", "/adventures", "/scenarios", "/scenarios/:id", "/play/:id",
|
||||||
"/scripts", "/scripts/:id", "/settings", "/chat", "/analytics",
|
"/scripts", "/scripts/:id", "/settings", "/chat", "/analytics",
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---------- In-process buffer ----------
|
# ---------- In-process buffer ----------
|
||||||
# Single-process deployment (same assumption as limits.py), so a plain dict
|
# The deployment is a single process, which is the same assumption `limits.py`
|
||||||
# under a lock is the whole design. Losing up to a minute of counts to a hard
|
# makes, so a plain dict under a lock is the whole design. Losing up to a minute
|
||||||
# restart is acceptable for traffic numbers, and the flusher also runs on
|
# of counts to a hard restart is acceptable for traffic numbers, and the flusher
|
||||||
# shutdown; on Render's free tier the service is idle when it sleeps, so the
|
# also runs on shutdown. On Render's free tier the service is idle when it
|
||||||
# buffer it sleeps on is empty anyway.
|
# sleeps, so the buffer it sleeps on is empty.
|
||||||
|
|
||||||
_counts: dict[tuple[str, str, str], int] = {}
|
_counts: dict[tuple[str, str, str], int] = {}
|
||||||
_visits: dict[tuple[str, str], set[str]] = {} # (day, visitor) -> flags
|
_visits: dict[tuple[str, str], set[str]] = {} # (day, visitor) -> flags.
|
||||||
_labels_seen: dict[tuple[str, str], set[str]] = {} # (day, metric) -> labels
|
_labels_seen: dict[tuple[str, str], set[str]] = {} # (day, metric) -> labels.
|
||||||
_guard = threading.Lock()
|
_guard = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
@@ -133,8 +138,11 @@ def _today() -> str:
|
|||||||
|
|
||||||
|
|
||||||
def record(metric: str, label: str = "", *, n: int = 1) -> None:
|
def record(metric: str, label: str = "", *, n: int = 1) -> None:
|
||||||
"""Add `n` to one counter. Never raises: analytics must not be able to
|
"""Adds `n` to one counter.
|
||||||
fail a request it is only watching."""
|
|
||||||
|
This function never raises. Analytics must not fail a request that it is
|
||||||
|
only observing.
|
||||||
|
"""
|
||||||
try:
|
try:
|
||||||
day = _today()
|
day = _today()
|
||||||
label = (label or "").strip()[:MAX_LABEL_LEN]
|
label = (label or "").strip()[:MAX_LABEL_LEN]
|
||||||
@@ -156,22 +164,26 @@ def record(metric: str, label: str = "", *, n: int = 1) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def visitor_id(user: models.User) -> str:
|
def visitor_id(user: models.User) -> str:
|
||||||
"""A stable but one-way handle for one visitor.
|
"""Returns a stable, one-way handle for one visitor.
|
||||||
|
|
||||||
HMAC of the user id under the app's secret key. Stable, so a returning
|
The handle is an HMAC of the user id under the app's secret key. It is
|
||||||
visitor can be told from a new one; one-way, so nothing in the analytics
|
stable, so a returning visitor can be distinguished from a new one. It is
|
||||||
tables points back at an account; keyed, so a client cannot compute one and
|
one-way, so nothing in the analytics tables points back at an account. It is
|
||||||
claim to be somebody else. One consequence worth knowing: rotating
|
keyed, so a client cannot compute one and claim to be someone else. One
|
||||||
AIDND_SECRET_KEY makes every returning visitor look new again.
|
consequence follows: rotating `AIDND_SECRET_KEY` makes every returning
|
||||||
|
visitor look new.
|
||||||
"""
|
"""
|
||||||
digest = hmac.new(security.SECRET_KEY, f"visitor:{user.id}".encode(), sha256)
|
digest = hmac.new(security.SECRET_KEY, f"visitor:{user.id}".encode(), sha256)
|
||||||
return digest.hexdigest()[:32]
|
return digest.hexdigest()[:32]
|
||||||
|
|
||||||
|
|
||||||
def record_visit(user: models.User | None, *, flag: str | None = None) -> None:
|
def record_visit(user: models.User | None, *, flag: str | None = None) -> None:
|
||||||
"""Note that this visitor was here today, optionally flipping one funnel
|
"""Records that this visitor was here today, and optionally sets one funnel
|
||||||
flag. A no-op without a user: a page loaded before a session exists still
|
flag.
|
||||||
counts as a pageview, just not as a person."""
|
|
||||||
|
Without a user the call does nothing. A page loaded before a session exists
|
||||||
|
still counts as a pageview, but not as a person.
|
||||||
|
"""
|
||||||
if user is None:
|
if user is None:
|
||||||
return
|
return
|
||||||
try:
|
try:
|
||||||
@@ -184,8 +196,10 @@ def record_visit(user: models.User | None, *, flag: str | None = None) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def record_event(name: str, user: models.User | None = None) -> None:
|
def record_event(name: str, user: models.User | None = None) -> None:
|
||||||
"""One thing that happened: counted, and — if it is a funnel step —
|
"""Records one event, and credits the visitor's day if it is a funnel step.
|
||||||
credited to the visitor's day. This is the call sites' whole interface."""
|
|
||||||
|
This is the whole interface the call sites use.
|
||||||
|
"""
|
||||||
record(M_EVENT, name)
|
record(M_EVENT, name)
|
||||||
record_visit(user, flag=FUNNEL_FLAGS.get(name))
|
record_visit(user, flag=FUNNEL_FLAGS.get(name))
|
||||||
|
|
||||||
@@ -193,10 +207,10 @@ def record_event(name: str, user: models.User | None = None) -> None:
|
|||||||
# ---------- Normalizing what the browser reports ----------
|
# ---------- Normalizing what the browser reports ----------
|
||||||
|
|
||||||
def normalize_route(path: str) -> str:
|
def normalize_route(path: str) -> str:
|
||||||
"""A client-reported path, reduced to one of KNOWN_ROUTES.
|
"""Reduces a client-reported path to one of `KNOWN_ROUTES`.
|
||||||
|
|
||||||
Numeric segments become ":id" — both to bound the label count and because
|
Numeric segments become ":id". That bounds the label count, and it keeps
|
||||||
*which* adventure someone opened is their business, not a statistic.
|
which adventure someone opened out of the statistics.
|
||||||
"""
|
"""
|
||||||
path = (path or "/").split("?")[0].split("#")[0]
|
path = (path or "/").split("?")[0].split("#")[0]
|
||||||
if not path.startswith("/"):
|
if not path.startswith("/"):
|
||||||
@@ -209,8 +223,11 @@ def normalize_route(path: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def normalize_referrer(referrer: str, own_host: str = "") -> str:
|
def normalize_referrer(referrer: str, own_host: str = "") -> str:
|
||||||
"""The sending site as a bare host. Our own host means an internal
|
"""Returns the sending site as a bare host.
|
||||||
navigation, which is not a referral — "" tells the caller to skip it."""
|
|
||||||
|
This app's own host means an internal navigation, which is not a referral.
|
||||||
|
In that case the function returns "", which tells the caller to skip it.
|
||||||
|
"""
|
||||||
if not referrer:
|
if not referrer:
|
||||||
return NONE_LABEL
|
return NONE_LABEL
|
||||||
host = (urlsplit(referrer).hostname or "").lower().lstrip(".")
|
host = (urlsplit(referrer).hostname or "").lower().lstrip(".")
|
||||||
@@ -222,21 +239,23 @@ def normalize_referrer(referrer: str, own_host: str = "") -> str:
|
|||||||
|
|
||||||
|
|
||||||
def api_route_label(scope: dict, status: int) -> str:
|
def api_route_label(scope: dict, status: int) -> str:
|
||||||
"""An error bucket like "500 /api/adventures/{adventure_id}".
|
"""Returns an error bucket such as "500 /api/adventures/{adventure_id}".
|
||||||
|
|
||||||
The route *template* is used, never the request path: it keeps one bucket
|
The label uses the route template, never the request path. That keeps one
|
||||||
per endpoint instead of one per adventure id, and — the reason it is not
|
bucket per endpoint rather than one per adventure id. It also bounds the
|
||||||
merely tidier — an unmatched path is entirely attacker-chosen, so labelling
|
table: an unmatched path is chosen entirely by the caller, so labeling by it
|
||||||
by it would let anyone mint rows by requesting nonsense.
|
would let anyone create rows by requesting arbitrary paths.
|
||||||
"""
|
"""
|
||||||
template = getattr(scope.get("route"), "path", None)
|
template = getattr(scope.get("route"), "path", None)
|
||||||
return f"{status} {template}" if template else f"{status} (unmatched)"
|
return f"{status} {template}" if template else f"{status} (unmatched)"
|
||||||
|
|
||||||
|
|
||||||
def device_of(user_agent: str) -> str:
|
def device_of(user_agent: str) -> str:
|
||||||
"""Mobile / tablet / desktop, and nothing finer. The UA string itself is
|
"""Returns "mobile", "tablet", or "desktop", and nothing more specific.
|
||||||
never stored — it is a fingerprint, and the answer worth having is one
|
|
||||||
word."""
|
The user-agent string itself is never stored, because it is a fingerprint
|
||||||
|
and the useful answer is one word.
|
||||||
|
"""
|
||||||
ua = (user_agent or "").lower()
|
ua = (user_agent or "").lower()
|
||||||
if not ua:
|
if not ua:
|
||||||
return UNKNOWN
|
return UNKNOWN
|
||||||
@@ -249,10 +268,10 @@ def device_of(user_agent: str) -> str:
|
|||||||
return "desktop"
|
return "desktop"
|
||||||
|
|
||||||
|
|
||||||
# Geo headers an edge may add. Render fronts services with a CDN that can set
|
# Geo headers an edge network may add. Render fronts services with a CDN that
|
||||||
# cf-ipcountry; the others cost nothing to look for. A value is trusted only if
|
# can set `cf-ipcountry`, and the others cost nothing to check. A value is
|
||||||
# it looks like an ISO code, since a client can send any header it likes — the
|
# trusted only if it looks like an ISO code, because a client can send any
|
||||||
# worst case is therefore a wrong country, never an unbounded label.
|
# header, so the worst case is a wrong country rather than an unbounded label.
|
||||||
_GEO_HEADERS = ("cf-ipcountry", "x-vercel-ip-country", "x-geo-country", "x-country-code")
|
_GEO_HEADERS = ("cf-ipcountry", "x-vercel-ip-country", "x-geo-country", "x-country-code")
|
||||||
|
|
||||||
|
|
||||||
@@ -275,8 +294,8 @@ def _drain() -> tuple[dict, dict]:
|
|||||||
counts, visits = _counts.copy(), _visits.copy()
|
counts, visits = _counts.copy(), _visits.copy()
|
||||||
_counts.clear()
|
_counts.clear()
|
||||||
_visits.clear()
|
_visits.clear()
|
||||||
# The label sets only bound cardinality within a day, so let yesterday's
|
# The label sets bound cardinality within one day, so drop the
|
||||||
# go rather than growing a map that never shrinks.
|
# previous day's rather than grow a map that never shrinks.
|
||||||
today = _today()
|
today = _today()
|
||||||
for key in [k for k in _labels_seen if k[0] != today]:
|
for key in [k for k in _labels_seen if k[0] != today]:
|
||||||
del _labels_seen[key]
|
del _labels_seen[key]
|
||||||
@@ -284,7 +303,7 @@ def _drain() -> tuple[dict, dict]:
|
|||||||
|
|
||||||
|
|
||||||
def _restore(counts: dict, visits: dict) -> None:
|
def _restore(counts: dict, visits: dict) -> None:
|
||||||
"""Put a failed flush's work back so the next one retries it."""
|
"""Returns a failed flush's work to the buffer, so the next flush retries it."""
|
||||||
with _guard:
|
with _guard:
|
||||||
for key, n in counts.items():
|
for key, n in counts.items():
|
||||||
_counts[key] = _counts.get(key, 0) + n
|
_counts[key] = _counts.get(key, 0) + n
|
||||||
@@ -293,7 +312,7 @@ def _restore(counts: dict, visits: dict) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def flush(db: Session | None = None) -> None:
|
def flush(db: Session | None = None) -> None:
|
||||||
"""Write the buffer out. Safe to call from anywhere; never raises."""
|
"""Writes the buffer out. This is safe to call from anywhere and never raises."""
|
||||||
counts, visits = _drain()
|
counts, visits = _drain()
|
||||||
if not counts and not visits:
|
if not counts and not visits:
|
||||||
return
|
return
|
||||||
@@ -334,9 +353,9 @@ def _write_visits(db: Session, visits: dict) -> None:
|
|||||||
return
|
return
|
||||||
table = models.AnalyticsVisitorDay.__table__
|
table = models.AnalyticsVisitorDay.__table__
|
||||||
ids = {visitor for _, visitor in visits}
|
ids = {visitor for _, visitor in visits}
|
||||||
# One indexed lookup settles new-vs-returning for the whole batch. It is
|
# One indexed lookup decides new against returning for the whole batch. It
|
||||||
# the only read this module does off the dashboard, and it returns short
|
# is the only read this module makes outside the dashboard, and it returns
|
||||||
# hashes for visitors who are active right now — bounded by the batch.
|
# short hashes for the visitors active right now, so the batch bounds it.
|
||||||
known = set(db.scalars(
|
known = set(db.scalars(
|
||||||
select(models.AnalyticsVisitorDay.visitor)
|
select(models.AnalyticsVisitorDay.visitor)
|
||||||
.where(models.AnalyticsVisitorDay.visitor.in_(ids))
|
.where(models.AnalyticsVisitorDay.visitor.in_(ids))
|
||||||
@@ -354,8 +373,8 @@ def _write_visits(db: Session, visits: dict) -> None:
|
|||||||
stmt = _insert(db)(table).values(rows)
|
stmt = _insert(db)(table).values(rows)
|
||||||
db.execute(stmt.on_conflict_do_update(
|
db.execute(stmt.on_conflict_do_update(
|
||||||
index_elements=["day", "visitor"],
|
index_elements=["day", "visitor"],
|
||||||
# Flags only ever turn on, and `is_new` is deliberately absent: the
|
# Flags only turn on, and `is_new` is absent on purpose. The first
|
||||||
# first write of a visitor's first day is the one that decided it.
|
# write of a visitor's first day is what decided it.
|
||||||
set_={
|
set_={
|
||||||
column: or_(table.c[column], stmt.excluded[column])
|
column: or_(table.c[column], stmt.excluded[column])
|
||||||
for column in FUNNEL_FLAGS.values()
|
for column in FUNNEL_FLAGS.values()
|
||||||
@@ -364,9 +383,11 @@ def _write_visits(db: Session, visits: dict) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def purge_old_visitor_days(db: Session) -> int:
|
def purge_old_visitor_days(db: Session) -> int:
|
||||||
"""Drop visitor-day rows past the retention horizon. Called by the cleanup
|
"""Deletes visitor-day rows past the retention horizon.
|
||||||
sweeper; the daily counters are never purged — they are aggregate, tiny,
|
|
||||||
and a portfolio project wants to keep its history."""
|
The cleanup sweeper calls this. The daily counters are never purged, because
|
||||||
|
they are aggregates, they are small, and this project keeps its history.
|
||||||
|
"""
|
||||||
if RETENTION_DAYS <= 0:
|
if RETENTION_DAYS <= 0:
|
||||||
return 0
|
return 0
|
||||||
cutoff = (models.utcnow().date() - timedelta(days=RETENTION_DAYS)).isoformat()
|
cutoff = (models.utcnow().date() - timedelta(days=RETENTION_DAYS)).isoformat()
|
||||||
@@ -378,9 +399,9 @@ def purge_old_visitor_days(db: Session) -> int:
|
|||||||
|
|
||||||
|
|
||||||
# ---------- Reading it back ----------
|
# ---------- Reading it back ----------
|
||||||
# Every query below is an aggregate: the database does the counting and ships
|
# Every query below is an aggregate. The database does the counting and returns
|
||||||
# back tens of rows, whatever the traffic behind them. Nothing here can return
|
# tens of rows, however much traffic is behind them. No query here can return a
|
||||||
# a row that belongs to one visitor.
|
# row that belongs to one visitor.
|
||||||
|
|
||||||
TOP_N = 12
|
TOP_N = 12
|
||||||
|
|
||||||
@@ -390,17 +411,20 @@ def _top(rows: list[dict], limit: int = TOP_N) -> list[dict]:
|
|||||||
|
|
||||||
|
|
||||||
def summary(db: Session, days: int = 30) -> dict:
|
def summary(db: Session, days: int = 30) -> dict:
|
||||||
"""Everything the dashboard shows, for the last `days` days (today
|
"""Returns everything the dashboard shows for the last `days` days, including
|
||||||
included). Flushes first so the numbers include the last minute."""
|
today.
|
||||||
|
|
||||||
|
The function flushes first, so the numbers include the last minute.
|
||||||
|
"""
|
||||||
flush(db)
|
flush(db)
|
||||||
today = models.utcnow().date()
|
today = models.utcnow().date()
|
||||||
since = (today - timedelta(days=days - 1)).isoformat()
|
since = (today - timedelta(days=days - 1)).isoformat()
|
||||||
daily = models.AnalyticsDaily
|
daily = models.AnalyticsDaily
|
||||||
visitor = models.AnalyticsVisitorDay
|
visitor = models.AnalyticsVisitorDay
|
||||||
|
|
||||||
# 1. Every counter in the window, folded to (metric, label) totals: the
|
# 1. Every counter in the window, reduced to (metric, label) totals. The
|
||||||
# page/referrer/country/device/scenario/error tables all come from this
|
# page, referrer, country, device, scenario, and error tables all come
|
||||||
# one pass rather than a query each.
|
# from this one pass rather than from a query each.
|
||||||
by_metric: dict[str, list[dict]] = {}
|
by_metric: dict[str, list[dict]] = {}
|
||||||
for metric, label, hits in db.execute(
|
for metric, label, hits in db.execute(
|
||||||
select(daily.metric, daily.label, func.sum(daily.hits))
|
select(daily.metric, daily.label, func.sum(daily.hits))
|
||||||
@@ -412,7 +436,7 @@ def summary(db: Session, days: int = 30) -> dict:
|
|||||||
rows.sort(key=lambda row: -row["hits"])
|
rows.sort(key=lambda row: -row["hits"])
|
||||||
events = {row["label"]: row["hits"] for row in by_metric.get(M_EVENT, [])}
|
events = {row["label"]: row["hits"] for row in by_metric.get(M_EVENT, [])}
|
||||||
|
|
||||||
# 2. Two per-day series worth drawing.
|
# 2. The two per-day series the dashboard draws.
|
||||||
pageviews_by_day = {
|
pageviews_by_day = {
|
||||||
day: int(hits)
|
day: int(hits)
|
||||||
for day, hits in db.execute(
|
for day, hits in db.execute(
|
||||||
@@ -430,8 +454,8 @@ def summary(db: Session, days: int = 30) -> dict:
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
# 3. People, per day. One row per visitor per day means COUNT(*) is already
|
# 3. People, per day. There is one row per visitor per day, so COUNT(*) is
|
||||||
# the day's unique visitors — no DISTINCT needed here.
|
# already the day's unique visitors and no DISTINCT is needed.
|
||||||
visitors_by_day: dict[str, dict] = {}
|
visitors_by_day: dict[str, dict] = {}
|
||||||
for day, total, fresh in db.execute(
|
for day, total, fresh in db.execute(
|
||||||
select(
|
select(
|
||||||
@@ -444,9 +468,9 @@ def summary(db: Session, days: int = 30) -> dict:
|
|||||||
):
|
):
|
||||||
visitors_by_day[day] = {"visitors": int(total), "new": int(fresh or 0)}
|
visitors_by_day[day] = {"visitors": int(total), "new": int(fresh or 0)}
|
||||||
|
|
||||||
# 4. The funnel, over the whole window, counting *people* once each:
|
# 4. The funnel over the whole window, counting each person once.
|
||||||
# COUNT(DISTINCT CASE WHEN flag THEN visitor END) ignores the NULLs the
|
# COUNT(DISTINCT CASE WHEN flag THEN visitor END) ignores the NULLs the
|
||||||
# CASE leaves for everyone who didn't reach that step.
|
# CASE leaves for everyone who did not reach that step.
|
||||||
unique, unique_new, *reached = db.execute(
|
unique, unique_new, *reached = db.execute(
|
||||||
select(
|
select(
|
||||||
func.count(func.distinct(visitor.visitor)),
|
func.count(func.distinct(visitor.visitor)),
|
||||||
@@ -480,9 +504,9 @@ def summary(db: Session, days: int = 30) -> dict:
|
|||||||
"until": today.isoformat(),
|
"until": today.isoformat(),
|
||||||
"generated_at": models.utcnow().isoformat(),
|
"generated_at": models.utcnow().isoformat(),
|
||||||
"totals": {
|
"totals": {
|
||||||
# `visitors` counts each person once for the window; `visits` counts
|
# `visitors` counts each person once for the window. `visits`
|
||||||
# them once per day they came back, which is the closest honest
|
# counts them once per day they returned, which is the closest
|
||||||
# thing to "sessions" without tracking sessions.
|
# measure to "sessions" that does not track sessions.
|
||||||
"visitors": int(unique),
|
"visitors": int(unique),
|
||||||
"new_visitors": int(unique_new),
|
"new_visitors": int(unique_new),
|
||||||
"visits": visits,
|
"visits": visits,
|
||||||
@@ -498,8 +522,8 @@ def summary(db: Session, days: int = 30) -> dict:
|
|||||||
"pages_per_visit": round(pageviews / visits, 1) if visits else 0,
|
"pages_per_visit": round(pageviews / visits, 1) if visits else 0,
|
||||||
},
|
},
|
||||||
"series": series,
|
"series": series,
|
||||||
# Step 0 is everyone who showed up, so the drop-off between it and
|
# Step 0 is everyone who arrived, so the drop-off between it and
|
||||||
# "Opened a scenario" is visible as a step like any other.
|
# "Opened a scenario" appears as a step like any other.
|
||||||
"funnel": [{"step": "Visited", "count": int(unique)}] + [
|
"funnel": [{"step": "Visited", "count": int(unique)}] + [
|
||||||
{"step": step, "count": int(count)}
|
{"step": step, "count": int(count)}
|
||||||
for step, count in zip(
|
for step, count in zip(
|
||||||
@@ -518,8 +542,9 @@ def summary(db: Session, days: int = 30) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
# ---------- Background flusher ----------
|
# ---------- Background flusher ----------
|
||||||
# Mirrors cleanup's start/stop pair so main.py's lifespan reads the same way
|
# This matches the start and stop pair in `cleanup`, so the lifespan in
|
||||||
# for both. The interval is what bounds how much a hard restart can lose.
|
# `main.py` reads the same way for both. The interval bounds how much a hard
|
||||||
|
# restart can lose.
|
||||||
|
|
||||||
async def _flush_loop() -> None:
|
async def _flush_loop() -> None:
|
||||||
import asyncio
|
import asyncio
|
||||||
@@ -528,8 +553,8 @@ async def _flush_loop() -> None:
|
|||||||
|
|
||||||
while True:
|
while True:
|
||||||
await asyncio.sleep(FLUSH_INTERVAL_SECONDS)
|
await asyncio.sleep(FLUSH_INTERVAL_SECONDS)
|
||||||
# Blocking DB work: keep it off the event loop, which is also serving
|
# This is blocking database work, so keep it off the event loop, which
|
||||||
# SSE turn streams.
|
# is also serving SSE turn streams.
|
||||||
await run_in_threadpool(flush)
|
await run_in_threadpool(flush)
|
||||||
|
|
||||||
|
|
||||||
@@ -540,9 +565,11 @@ def start_flusher():
|
|||||||
|
|
||||||
|
|
||||||
async def stop_flusher(task) -> None:
|
async def stop_flusher(task) -> None:
|
||||||
"""Cancel the loop and write out whatever it was holding — a deploy is the
|
"""Cancels the loop and writes out whatever it was holding.
|
||||||
one restart that is both frequent and predictable, so it should not be the
|
|
||||||
thing that loses a minute of counts."""
|
A deploy is the one restart that is both frequent and predictable, so it
|
||||||
|
should not be what loses a minute of counts.
|
||||||
|
"""
|
||||||
import asyncio
|
import asyncio
|
||||||
|
|
||||||
from starlette.concurrency import run_in_threadpool
|
from starlette.concurrency import run_in_threadpool
|
||||||
|
|||||||
+97
-94
@@ -1,33 +1,33 @@
|
|||||||
"""Phase 14, SP4 — the attempts at one turn.
|
"""Phase 14, SP4: the attempts at one turn.
|
||||||
|
|
||||||
A retry used to rewrite the AI action in place and push the discarded take into
|
A retry used to rewrite the AI action in place and append the discarded attempt
|
||||||
a JSON list on the same row. That is where seven separate bugs came from: the
|
to a JSON list on the same row. Seven separate bugs came from that arrangement.
|
||||||
row's `text` mirrored one entry of a repeating group, `variant_count` mirrored
|
The row's `text` duplicated one entry of a repeating group, `variant_count`
|
||||||
its length, and every reader that touched the story during a retry had to be
|
duplicated its length, and every reader that touched the story during a retry
|
||||||
told to pretend the row was not there.
|
had to be told to ignore the row.
|
||||||
|
|
||||||
Now an attempt is a **node**. Retry writes a sibling at the same
|
Now an attempt is a node. A retry writes a sibling at the same `(branch_id,
|
||||||
`(branch_id, depth)` and marks it live; the previous one stays exactly as it
|
depth)` and marks it live. The previous attempt stays as it was written, at the
|
||||||
was written, at the same coordinate, `live = False`. Nothing is mirrored, so
|
same coordinate, with `live = False`. Nothing is duplicated, so nothing can
|
||||||
nothing can drift.
|
diverge.
|
||||||
|
|
||||||
Two invariants hold the arrangement together, and this module is the only place
|
Two invariants hold the arrangement together, and this module is the only place
|
||||||
that maintains either:
|
that maintains either one:
|
||||||
|
|
||||||
* **Exactly one sibling in a group is live.** `lineage.Path.clause` selects on
|
* Exactly one sibling in a group is live. `lineage.Path.clause` selects on it, so
|
||||||
it, so the losing attempts are invisible to every read of the story without
|
the other attempts are invisible to every read of the story, and none of those
|
||||||
any of those reads knowing that attempts exist.
|
reads has to know that attempts exist.
|
||||||
* **The assembled prompt is stored once per turn, on the live sibling.**
|
* The assembled prompt is stored once per turn, on the live sibling. A
|
||||||
A `context_snapshot` is ~163 kB of prompt that every attempt at a turn
|
`context_snapshot` is about 163 kB of prompt that every attempt at a turn
|
||||||
shares, plus a few hundred bytes that differ (`ATTEMPT_KEYS`). Giving each
|
shares, plus a few hundred bytes that differ, listed in `ATTEMPT_KEYS`. Giving
|
||||||
sibling its own copy would have made retry a permanent multiplier on the
|
each sibling its own copy would make a retry a permanent multiplier on the
|
||||||
biggest column in the database — the thing the JSON list was invented to
|
largest column in the database, which is what the JSON list was invented to
|
||||||
avoid. So the prompt moves with the live flag, and a superseded sibling keeps
|
avoid. The prompt therefore moves with the live flag, and a superseded sibling
|
||||||
only its own slices.
|
keeps only its own slices.
|
||||||
|
|
||||||
Ordering inside a group is `variant_index`, an explicit ordinal, not
|
Ordering inside a group comes from `variant_index`, which is an explicit ordinal
|
||||||
`created_at`. Two attempts made in the same second must still page in the order
|
rather than `created_at`. Two attempts made in the same second still have to page
|
||||||
they were made, and the migration that split the old JSON lists had to be able
|
in the order they were made, and the migration that split the old JSON lists had
|
||||||
to state the order rather than reconstruct it.
|
to state the order rather than reconstruct it.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -38,41 +38,42 @@ from sqlalchemy.orm import Session, undefer
|
|||||||
from . import models
|
from . import models
|
||||||
from .context import lineage
|
from .context import lineage
|
||||||
|
|
||||||
# The slices of a context snapshot that belong to one attempt rather than to
|
# The slices of a context snapshot that belong to one attempt rather than to the
|
||||||
# the turn: the world-state delta it proposed and what the referee did with it,
|
# turn. They are the world-state delta the attempt proposed and what the engine
|
||||||
# the script report, the model's literal reply, and the endpoint's token
|
# did with it, the script report, the model's literal reply, and the endpoint's
|
||||||
# accounting — each attempt is its own API call, and a retry is precisely the
|
# token accounting. Each attempt is its own API call, and a retry is the call
|
||||||
# call expected to read the prompt back out of cache. Everything else in a
|
# most likely to read the prompt back out of cache. Everything else in a snapshot
|
||||||
# snapshot is the prompt, which is assembled once per turn.
|
# is the prompt, which is assembled once per turn.
|
||||||
ATTEMPT_KEYS = ("world_state", "script", "raw_output", "usage")
|
ATTEMPT_KEYS = ("world_state", "script", "raw_output", "usage")
|
||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------ reading
|
# ------------------------------------------------------------------ reading
|
||||||
|
|
||||||
def group(db: Session, action: models.Action) -> list[models.Action]:
|
def group(db: Session, action: models.Action) -> list[models.Action]:
|
||||||
"""Every attempt at `action`'s turn, oldest first.
|
"""Returns every attempt at `action`'s turn, oldest first.
|
||||||
|
|
||||||
Keyed on the **parent**, not on the coordinate (SP9). The two agree right up
|
The query keys on the parent rather than on the coordinate (SP9). The two
|
||||||
until a take is forked onto its own branch: it keeps its parent but leaves
|
agree until an attempt is forked onto its own branch. That attempt keeps its
|
||||||
the (branch, depth) its siblings are still at, so a coordinate would report
|
parent but leaves the `(branch, depth)` its siblings are still at, so a
|
||||||
it as the only take of its turn — `1/1` where the player is owed `1/3`.
|
coordinate would report it as the only attempt at its turn, showing `1/1`
|
||||||
|
where the player should see `1/3`.
|
||||||
|
|
||||||
The parent also gets the nesting right without being asked. Takes under C1
|
The parent also nests groups correctly without extra work. Attempts under C1
|
||||||
and takes under C2 share a depth and, until one of them forks, a branch;
|
and attempts under C2 share a depth, and until one of them forks they share a
|
||||||
only the parent separates them, which is what makes a pager under C2 read
|
branch. Only the parent separates them, which is what makes a pager under C2
|
||||||
`2/2` instead of counting C1's three as well.
|
read `2/2` rather than count C1's three as well.
|
||||||
|
|
||||||
Two fallbacks, both meaning "this row predates the key being asked about":
|
There are two fallbacks, and both mean the row predates the key being asked
|
||||||
a node with no branch is a pre-tree row no path contains, and a node with no
|
about. A node with no branch is a pre-tree row that no path contains, and a
|
||||||
parent is a pre-SP9 row the backfill could not place. Both are their own
|
node with no parent is a pre-SP9 row the backfill could not place. Under the
|
||||||
only attempt under the rule they were written with.
|
rule each was written with, both are the only attempt at their turn.
|
||||||
"""
|
"""
|
||||||
if action.branch_id is None or action.depth is None:
|
if action.branch_id is None or action.depth is None:
|
||||||
return [action]
|
return [action]
|
||||||
if action.parent_id is None:
|
if action.parent_id is None:
|
||||||
# Pre-SP9, and the coordinate is the key those rows were written under.
|
# This row is pre-SP9, and the coordinate is the key those rows were
|
||||||
# Root nodes land here too and are genuinely alone: nothing is a take of
|
# written under. A root node also reaches this branch and is genuinely
|
||||||
# the opening of a story.
|
# alone, because nothing is an attempt at the opening of a story.
|
||||||
return (
|
return (
|
||||||
db.query(models.Action)
|
db.query(models.Action)
|
||||||
.filter(
|
.filter(
|
||||||
@@ -96,18 +97,18 @@ def group(db: Session, action: models.Action) -> list[models.Action]:
|
|||||||
|
|
||||||
|
|
||||||
def on_branch(rows: list[models.Action], node: models.Action) -> list[models.Action]:
|
def on_branch(rows: list[models.Action], node: models.Action) -> list[models.Action]:
|
||||||
"""The takes in `rows` that sit on `node`'s own branch.
|
"""Returns the attempts in `rows` that are on `node`'s own branch.
|
||||||
|
|
||||||
`group` answers "which takes are of this turn", and since SP9 that spans
|
`group` reports which attempts belong to this turn, and since SP9 that spans
|
||||||
branches — a take forked onto its own line is still a take of the same turn,
|
branches. An attempt forked onto its own line is still an attempt at the same
|
||||||
which is the whole point of keying on the parent.
|
turn, which is the reason for keying on the parent.
|
||||||
|
|
||||||
Deleting is the one caller that must not follow it there. A take on another
|
Deletion is the one caller that must not follow a group across branches. An
|
||||||
branch is reachable through that branch and belongs to the story somebody is
|
attempt on another branch is reachable through that branch and belongs to the
|
||||||
telling on it; removing it because a turn was undone over here would delete
|
story someone is telling there. Removing it because a turn was undone here
|
||||||
a line nobody asked about. Same parent *and* same branch is the coordinate,
|
would delete a line nobody asked about. The same parent and the same branch
|
||||||
which is what "every attempt at this turn" meant before a fork could move
|
together are the coordinate, which is what every attempt at this turn meant
|
||||||
one out of it.
|
before a fork could move one out of it.
|
||||||
"""
|
"""
|
||||||
return [row for row in rows if row.branch_id == node.branch_id]
|
return [row for row in rows if row.branch_id == node.branch_id]
|
||||||
|
|
||||||
@@ -122,12 +123,12 @@ def live_in(rows: list[models.Action]) -> models.Action | None:
|
|||||||
def preceding(
|
def preceding(
|
||||||
db: Session, adventure: models.Adventure, node: models.Action
|
db: Session, adventure: models.Adventure, node: models.Action
|
||||||
) -> models.Action | None:
|
) -> models.Action | None:
|
||||||
"""The node the story tells immediately before `node`.
|
"""Returns the node the story tells immediately before `node`.
|
||||||
|
|
||||||
"Before this turn" as a fact about the path rather than as a snapshot taken
|
This reads "before this turn" as a fact about the path rather than as a
|
||||||
from inside the turn — which is what makes the after-snapshots enough on
|
snapshot taken from inside the turn, which is what makes the after-snapshots
|
||||||
their own. Undefers both of them because the only reason to ask for this
|
sufficient on their own. The query undefers both of them, because the only
|
||||||
row is to put back what it left behind.
|
reason to fetch this row is to restore what it left behind.
|
||||||
"""
|
"""
|
||||||
if node.depth is None:
|
if node.depth is None:
|
||||||
return None
|
return None
|
||||||
@@ -150,12 +151,12 @@ def preceding(
|
|||||||
# ------------------------------------------------------------------ writing
|
# ------------------------------------------------------------------ writing
|
||||||
|
|
||||||
def restore_state(adventure: models.Adventure, node: models.Action | None) -> None:
|
def restore_state(adventure: models.Adventure, node: models.Action | None) -> None:
|
||||||
"""Put back the script scoreboard and world state `node` left behind.
|
"""Restores the script state and world state that `node` left behind.
|
||||||
|
|
||||||
A NULL snapshot means "leave the live state alone", never "reset it": rows
|
A NULL snapshot means leave the live state as it is, never reset it. Rows
|
||||||
written before SP4 that the migration could not derive an outcome for carry
|
written before SP4 that the migration could not derive an outcome for carry
|
||||||
NULLs, and clobbering a running adventure's scoreboard with an empty dict
|
NULLs, and overwriting a running adventure's state with an empty dict would
|
||||||
would be a far worse answer than doing nothing.
|
be worse than doing nothing.
|
||||||
"""
|
"""
|
||||||
if node is None:
|
if node is None:
|
||||||
return
|
return
|
||||||
@@ -166,7 +167,7 @@ def restore_state(adventure: models.Adventure, node: models.Action | None) -> No
|
|||||||
|
|
||||||
|
|
||||||
def snapshot_outcome(adventure: models.Adventure, node: models.Action) -> None:
|
def snapshot_outcome(adventure: models.Adventure, node: models.Action) -> None:
|
||||||
"""Record on `node` what the adventure looks like now that it has played."""
|
"""Records on `node` the state of the adventure now that the node has played."""
|
||||||
state = adventure.script_state if isinstance(adventure.script_state, dict) else {}
|
state = adventure.script_state if isinstance(adventure.script_state, dict) else {}
|
||||||
world = adventure.world_state if isinstance(adventure.world_state, dict) else {}
|
world = adventure.world_state if isinstance(adventure.world_state, dict) else {}
|
||||||
node.state_after = copy.deepcopy(state)
|
node.state_after = copy.deepcopy(state)
|
||||||
@@ -176,7 +177,7 @@ def snapshot_outcome(adventure: models.Adventure, node: models.Action) -> None:
|
|||||||
def roll_back_before(
|
def roll_back_before(
|
||||||
db: Session, adventure: models.Adventure, node: models.Action
|
db: Session, adventure: models.Adventure, node: models.Action
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Rewind the shared state to before `node` was played."""
|
"""Rewinds the shared state to what it was before `node` was played."""
|
||||||
restore_state(adventure, preceding(db, adventure, node))
|
restore_state(adventure, preceding(db, adventure, node))
|
||||||
|
|
||||||
|
|
||||||
@@ -186,26 +187,28 @@ def add_attempt(
|
|||||||
previous: models.Action,
|
previous: models.Action,
|
||||||
replacement: models.Action,
|
replacement: models.Action,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Put `replacement` beside `previous` as the newer attempt at that turn.
|
"""Places `replacement` next to `previous` as the newer attempt at that turn.
|
||||||
|
|
||||||
Placed by hand rather than through `tree.place_action`, which would read the
|
The placement is done here rather than through `tree.place_action`, which
|
||||||
depth off the legacy `index` and move the head: a sibling is not a new turn,
|
would read the depth from the legacy `index` and move the head. A sibling is
|
||||||
it is another take on the one the head is already standing on.
|
not a new turn. It is another attempt at the turn the head is already on.
|
||||||
"""
|
"""
|
||||||
replacement.branch_id = previous.branch_id
|
replacement.branch_id = previous.branch_id
|
||||||
replacement.depth = previous.depth
|
replacement.depth = previous.depth
|
||||||
# Copied, never resolved from the path: a take belongs to the turn it is a
|
# Copy the parent rather than resolve it from the path. An attempt belongs
|
||||||
# take *of*, and that is what `group` keys on. Resolving it here would ask
|
# to the turn it is an attempt at, and that is what `group` keys on.
|
||||||
# what is live one depth back, which is the same node right now and stops
|
# Resolving it here would ask which node is live one depth back. That is the
|
||||||
# being once this turn is forked away from.
|
# same node right now, and it stops being the same node once the story forks
|
||||||
|
# away from this turn.
|
||||||
replacement.parent_id = previous.parent_id
|
replacement.parent_id = previous.parent_id
|
||||||
replacement.live = True
|
replacement.live = True
|
||||||
# The end of the group, not one past `previous` — which is only the same
|
# Use the end of the group rather than one past `previous`. The two match
|
||||||
# thing when `previous` is the newest take. Switch a three-take turn back to
|
# only when `previous` is the newest attempt. Switch a three-attempt turn
|
||||||
# take 1 and retry, and `previous.variant_index + 1` collides with take 2;
|
# back to attempt 1 and retry, and `previous.variant_index + 1` collides with
|
||||||
# `renumber` then breaks the tie by id and files the new attempt *between*
|
# attempt 2. `renumber` then breaks the tie by id and places the new attempt
|
||||||
# takes 2 and 3, so the pager walks the takes in an order they were not made
|
# between attempts 2 and 3, so the pager walks the attempts in an order they
|
||||||
# in. `group` is oldest-first, and `replacement` is not in it yet.
|
# were not made in. `group` returns oldest first, and `replacement` is not in
|
||||||
|
# it yet.
|
||||||
siblings = group(db, previous)
|
siblings = group(db, previous)
|
||||||
replacement.variant_index = 1 + max(
|
replacement.variant_index = 1 + max(
|
||||||
(s.variant_index for s in siblings if s.variant_index is not None),
|
(s.variant_index for s in siblings if s.variant_index is not None),
|
||||||
@@ -213,18 +216,18 @@ def add_attempt(
|
|||||||
)
|
)
|
||||||
previous.live = False
|
previous.live = False
|
||||||
# The replacement was assembled with a fresh snapshot, so the prompt for
|
# The replacement was assembled with a fresh snapshot, so the prompt for
|
||||||
# this turn is now the one it carries; the superseded attempt keeps only
|
# this turn is now the one it carries. The superseded attempt keeps only the
|
||||||
# what was its own.
|
# slices that were its own.
|
||||||
keep_own_slices(previous)
|
keep_own_slices(previous)
|
||||||
|
|
||||||
|
|
||||||
def make_live(
|
def make_live(
|
||||||
db: Session, adventure: models.Adventure, node: models.Action
|
db: Session, adventure: models.Adventure, node: models.Action
|
||||||
) -> list[models.Action]:
|
) -> list[models.Action]:
|
||||||
"""Make `node` the attempt the story tells, and put its outcome back.
|
"""Makes `node` the attempt the story tells, and restores its outcome.
|
||||||
|
|
||||||
Returns the group, renumbered, so a caller that wants to report on it does
|
Returns the group, renumbered, so that a caller reporting on it does not read
|
||||||
not read it twice.
|
it twice.
|
||||||
"""
|
"""
|
||||||
rows = group(db, node)
|
rows = group(db, node)
|
||||||
previous = live_in(rows)
|
previous = live_in(rows)
|
||||||
@@ -238,11 +241,11 @@ def make_live(
|
|||||||
|
|
||||||
|
|
||||||
def renumber(rows: list[models.Action]) -> None:
|
def renumber(rows: list[models.Action]) -> None:
|
||||||
"""Refresh the group-shape cache the page response reads.
|
"""Refreshes the group-shape cache that the page response reads.
|
||||||
|
|
||||||
`variant_count` is 0 rather than 1 for a turn nobody retried, because the
|
`variant_count` is 0 rather than 1 for a turn nobody retried, because the
|
||||||
pager's question is "is there anything to page through?" and the answer for
|
pager asks whether there is anything to page through, and for a single
|
||||||
a single attempt is no.
|
attempt the answer is no.
|
||||||
"""
|
"""
|
||||||
count = len(rows) if len(rows) > 1 else 0
|
count = len(rows) if len(rows) > 1 else 0
|
||||||
for i, row in enumerate(rows):
|
for i, row in enumerate(rows):
|
||||||
@@ -253,7 +256,7 @@ def renumber(rows: list[models.Action]) -> None:
|
|||||||
# ------------------------------------------------- the prompt, stored once
|
# ------------------------------------------------- the prompt, stored once
|
||||||
|
|
||||||
def keep_own_slices(node: models.Action) -> None:
|
def keep_own_slices(node: models.Action) -> None:
|
||||||
"""Strip `node`'s snapshot back to what is only its own."""
|
"""Reduces `node`'s snapshot to the slices that are only its own."""
|
||||||
snapshot = node.context_snapshot
|
snapshot = node.context_snapshot
|
||||||
if not isinstance(snapshot, dict):
|
if not isinstance(snapshot, dict):
|
||||||
return
|
return
|
||||||
@@ -263,11 +266,11 @@ def keep_own_slices(node: models.Action) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def hand_over_the_prompt(giver: models.Action, taker: models.Action) -> None:
|
def hand_over_the_prompt(giver: models.Action, taker: models.Action) -> None:
|
||||||
"""Move the turn's assembled prompt from one attempt to another.
|
"""Moves the turn's assembled prompt from one attempt to another.
|
||||||
|
|
||||||
Called when the live flag moves, so the row in the story is always the row
|
The caller runs this when the live flag moves, so that the row in the story
|
||||||
the Insights viewer can explain. Nothing is copied — the prompt exists once
|
is always the row the Insights viewer can explain. Nothing is copied. The
|
||||||
before and once after, on whichever sibling is being read.
|
prompt exists once before and once after, on whichever sibling is being read.
|
||||||
"""
|
"""
|
||||||
held = giver.context_snapshot if isinstance(giver.context_snapshot, dict) else {}
|
held = giver.context_snapshot if isinstance(giver.context_snapshot, dict) else {}
|
||||||
shared = {k: v for k, v in held.items() if k not in ATTEMPT_KEYS}
|
shared = {k: v for k, v in held.items() if k not in ATTEMPT_KEYS}
|
||||||
|
|||||||
+87
-63
@@ -1,18 +1,19 @@
|
|||||||
"""Phase 8 — user resolution, sessions, and the shared demo key.
|
"""Phase 8: user resolution, sessions, and the shared demo key.
|
||||||
|
|
||||||
Two modes, chosen by the AIDND_MULTI_USER env var:
|
The `AIDND_MULTI_USER` environment variable selects one of two modes:
|
||||||
|
|
||||||
- Local mode (default): every request resolves to one auto-created "local
|
* Local mode, the default. Every request resolves to one automatically created
|
||||||
user". No cookies, no login UI — a clone/docker-compose behaves exactly
|
local user. There are no cookies and no login UI, so a clone or a
|
||||||
like the pre-Phase-8 single-user app.
|
docker-compose run behaves like the single-user app from before Phase 8.
|
||||||
- Multi-user mode (hosted): requests carry a signed session cookie. GET
|
* Multi-user mode, used for hosted deployments. Requests carry a signed session
|
||||||
/api/auth/me creates a guest user on first visit; registering upgrades the
|
cookie. `GET /api/auth/me` creates a guest user on the first visit, and
|
||||||
guest in place so their data survives. Requests without a valid session get
|
registering upgrades that guest in place so their data survives. A request
|
||||||
401 and the frontend re-establishes via /me.
|
without a valid session gets a 401, and the frontend re-establishes the
|
||||||
|
session through `/me`.
|
||||||
|
|
||||||
The shared demo key (BYOK fallback) is also configured here: users whose
|
The shared demo key, which is the fallback when a user brings no key of their
|
||||||
settings have no API key are routed to a server-funded endpoint with a model
|
own, is also configured here. A user whose settings hold no API key is routed to
|
||||||
whitelist and a per-day turn cap.
|
a server-funded endpoint with a model allowlist and a per-day turn cap.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
@@ -33,9 +34,10 @@ def _env_flag(name: str) -> bool:
|
|||||||
MULTI_USER = _env_flag("AIDND_MULTI_USER")
|
MULTI_USER = _env_flag("AIDND_MULTI_USER")
|
||||||
|
|
||||||
SESSION_COOKIE = "aidnd_session"
|
SESSION_COOKIE = "aidnd_session"
|
||||||
# Secure cookies default on in multi-user (hosted = HTTPS; browsers also
|
# Secure cookies are on by default in multi-user mode, because a hosted
|
||||||
# accept Secure on http://localhost). AIDND_COOKIE_SECURE=0/1 overrides —
|
# deployment serves HTTPS and browsers also accept Secure on http://localhost.
|
||||||
# e.g. 0 when testing multi-user over plain http on a LAN address.
|
# `AIDND_COOKIE_SECURE` overrides the default with 0 or 1. Use 0 when testing
|
||||||
|
# multi-user mode over plain HTTP on a LAN address.
|
||||||
_cookie_secure_env = os.environ.get("AIDND_COOKIE_SECURE", "").strip().lower()
|
_cookie_secure_env = os.environ.get("AIDND_COOKIE_SECURE", "").strip().lower()
|
||||||
COOKIE_SECURE = (
|
COOKIE_SECURE = (
|
||||||
_cookie_secure_env in ("1", "true", "yes", "on")
|
_cookie_secure_env in ("1", "true", "yes", "on")
|
||||||
@@ -58,18 +60,19 @@ DEMO_MODELS = [
|
|||||||
] or ["google/gemma-4-26b-a4b-it:free"]
|
] or ["google/gemma-4-26b-a4b-it:free"]
|
||||||
DEMO_TURNS_PER_DAY = int(os.environ.get("AIDND_DEMO_TURNS_PER_DAY", "20") or 20)
|
DEMO_TURNS_PER_DAY = int(os.environ.get("AIDND_DEMO_TURNS_PER_DAY", "20") or 20)
|
||||||
|
|
||||||
# Trusted testers (by email) who bypass the daily demo cap — unmetered turns on
|
# Trusted testers, listed by email, who bypass the daily demo cap and take
|
||||||
# the shared demo key. Comma-separated emails; matched case-insensitively.
|
# unmetered turns on the shared demo key. The list is comma-separated, and the
|
||||||
|
# match ignores case.
|
||||||
POWER_USERS = {
|
POWER_USERS = {
|
||||||
e.strip().lower()
|
e.strip().lower()
|
||||||
for e in os.environ.get("AIDND_POWER_USERS", "").split(",")
|
for e in os.environ.get("AIDND_POWER_USERS", "").split(",")
|
||||||
if e.strip()
|
if e.strip()
|
||||||
}
|
}
|
||||||
|
|
||||||
# Who can see the visit analytics. Deliberately its own list rather than
|
# Who can see the visit analytics. This is a separate list from `POWER_USERS` on
|
||||||
# POWER_USERS: a trusted tester gets unmetered turns and the AI Chat page,
|
# purpose. A trusted tester gets unmetered turns and the AI Chat page, which is
|
||||||
# which is not a reason to hand them the site's traffic numbers. Empty (the
|
# not a reason to give them the site's traffic numbers. An empty list, which is
|
||||||
# default) means nobody sees the dashboard in a hosted deployment.
|
# the default, means nobody sees the dashboard in a hosted deployment.
|
||||||
ANALYTICS_EMAILS = {
|
ANALYTICS_EMAILS = {
|
||||||
e.strip().lower()
|
e.strip().lower()
|
||||||
for e in os.environ.get("AIDND_ANALYTICS_EMAILS", "").split(",")
|
for e in os.environ.get("AIDND_ANALYTICS_EMAILS", "").split(",")
|
||||||
@@ -83,15 +86,19 @@ DEMO_CAP_MESSAGE = (
|
|||||||
|
|
||||||
|
|
||||||
def demo_enabled() -> bool:
|
def demo_enabled() -> bool:
|
||||||
# The demo key is a hosted-deployment feature; local installs talk to
|
# The demo key is a hosted-deployment feature. A local install talks to
|
||||||
# whatever endpoint Settings points at, even with no API key (Ollama).
|
# whatever endpoint Settings points at, even with no API key, such as
|
||||||
|
# Ollama.
|
||||||
return MULTI_USER and bool(DEMO_API_KEY)
|
return MULTI_USER and bool(DEMO_API_KEY)
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class ProviderConfig:
|
class ProviderConfig:
|
||||||
"""What the turn engine should actually connect with, after the
|
"""What the turn engine connects with, after the decision between a
|
||||||
BYOK-vs-demo decision. Build these with resolve_provider_config()."""
|
user-supplied key and the demo key.
|
||||||
|
|
||||||
|
Build one of these with `resolve_provider_config()`.
|
||||||
|
"""
|
||||||
|
|
||||||
endpoint_url: str
|
endpoint_url: str
|
||||||
api_key: str
|
api_key: str
|
||||||
@@ -99,19 +106,20 @@ class ProviderConfig:
|
|||||||
using_demo: bool
|
using_demo: bool
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
def __post_init__(self) -> None:
|
||||||
# Belt and braces around server-funded turns: resolve_provider_config()
|
# A second guard around server-funded turns. `resolve_provider_config()`
|
||||||
# already pins the model, and this makes it a property of the config
|
# already pins the model, and this makes the pin a property of the config
|
||||||
# object too, so a future caller can't construct an unpinned one.
|
# object too, so a later caller cannot construct an unpinned one. This
|
||||||
# Unreachable by design — a raise here means a new code path bypassed
|
# raise is unreachable by design. Reaching it means a new code path
|
||||||
# the pinning, which is worth failing loudly rather than billing.
|
# bypassed the pinning, which is worth failing on rather than billing
|
||||||
|
# for.
|
||||||
#
|
#
|
||||||
# The test is `using_demo`, NOT `api_key == DEMO_API_KEY`. Keying it on
|
# The test is `using_demo`, not `api_key == DEMO_API_KEY`. Keying on the
|
||||||
# the key value looks stricter but is wrong: the demo key is a normal
|
# key value looks stricter and is wrong. The demo key is an ordinary
|
||||||
# OpenRouter key, so a user can legitimately paste that same key into
|
# OpenRouter key, so a user can legitimately paste that same key into
|
||||||
# their own Settings as BYOK — and then every resolution raised, 500ing
|
# their own Settings. Every resolution then raised, which returned a 500
|
||||||
# even GET /auth/me and taking the whole SPA down with it. `using_demo`
|
# even from `GET /auth/me` and took the whole SPA down. `using_demo` is
|
||||||
# is what actually means "the server is paying", and only the demo
|
# what means the server is paying, and only the demo branch below sets
|
||||||
# branch below sets it.
|
# it.
|
||||||
if self.using_demo and self.model not in DEMO_MODELS:
|
if self.using_demo and self.model not in DEMO_MODELS:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"Refusing to use the shared demo key with non-whitelisted model {self.model!r}"
|
f"Refusing to use the shared demo key with non-whitelisted model {self.model!r}"
|
||||||
@@ -121,20 +129,23 @@ class ProviderConfig:
|
|||||||
def resolve_provider_config(
|
def resolve_provider_config(
|
||||||
settings: models.Settings, *, model_override: str | None = None
|
settings: models.Settings, *, model_override: str | None = None
|
||||||
) -> ProviderConfig:
|
) -> ProviderConfig:
|
||||||
"""BYOK when the user has their own key, the shared demo key otherwise.
|
"""Returns the user's own key when they have one, and the shared demo key
|
||||||
|
otherwise.
|
||||||
|
|
||||||
THE security-relevant branch is the demo one, and it is the only place the
|
The demo branch is the security-relevant one, and it is the only place the
|
||||||
whitelist rule lives — every caller must come through here rather than
|
allowlist rule lives. Every caller has to come through this function rather
|
||||||
building a ProviderConfig itself. On the demo key:
|
than build a `ProviderConfig` itself. On the demo key:
|
||||||
|
|
||||||
- the model is pinned to DEMO_MODELS, so a caller-supplied override (the AI
|
* The model is pinned to `DEMO_MODELS`, so a caller-supplied override from
|
||||||
Chat page) or a hand-edited Settings row cannot aim a server-funded key at
|
the AI Chat page, or a hand-edited Settings row, cannot point a
|
||||||
a paid model; anything unrecognised falls back to DEMO_MODELS[0];
|
server-funded key at a paid model. An unrecognized model falls back to
|
||||||
- the endpoint is pinned to DEMO_ENDPOINT_URL, so the key itself can't be
|
`DEMO_MODELS[0]`.
|
||||||
redirected to a URL the user controls and harvested.
|
* The endpoint is pinned to `DEMO_ENDPOINT_URL`, so the key cannot be
|
||||||
|
redirected to a URL the user controls and captured there.
|
||||||
|
|
||||||
`model_override` is a per-request preference (never a grant): it's honoured
|
`model_override` is a per-request preference and never a grant. It is used
|
||||||
verbatim under BYOK, and only if whitelisted on the demo key.
|
verbatim with the user's own key, and on the demo key only when the model is
|
||||||
|
on the allowlist.
|
||||||
"""
|
"""
|
||||||
key = settings.api_key_plain
|
key = settings.api_key_plain
|
||||||
requested = (model_override or "").strip() or settings.model
|
requested = (model_override or "").strip() or settings.model
|
||||||
@@ -149,27 +160,33 @@ def _today() -> str:
|
|||||||
|
|
||||||
|
|
||||||
def is_power_user(user: models.User) -> bool:
|
def is_power_user(user: models.User) -> bool:
|
||||||
"""Trusted testers: unmetered demo turns, plus tooling that isn't part of
|
"""Returns whether this user is a trusted tester.
|
||||||
the game (the AI Chat scratchpad). Local installs are always trusted — it's
|
|
||||||
the operator's own machine and their own API key, same reasoning as the
|
A trusted tester gets unmetered demo turns, plus tooling that is not part of
|
||||||
provider debug log being local-only."""
|
the game, such as the AI Chat scratchpad. A local install is always trusted,
|
||||||
|
because it runs on the operator's own machine with their own API key. The
|
||||||
|
provider debug log is local-only for the same reason.
|
||||||
|
"""
|
||||||
if not MULTI_USER:
|
if not MULTI_USER:
|
||||||
return True
|
return True
|
||||||
return bool(user.email) and user.email.lower() in POWER_USERS
|
return bool(user.email) and user.email.lower() in POWER_USERS
|
||||||
|
|
||||||
|
|
||||||
def is_owner(user: models.User) -> bool:
|
def is_owner(user: models.User) -> bool:
|
||||||
"""May this user see the visit analytics? Local installs always can — it is
|
"""Returns whether this user may see the visit analytics.
|
||||||
the operator's own machine and their own visits, same reasoning as the
|
|
||||||
provider debug log; hosted deployments check AIDND_ANALYTICS_EMAILS."""
|
A local install always may, because it runs on the operator's own machine and
|
||||||
|
shows their own visits. The provider debug log follows the same reasoning. A
|
||||||
|
hosted deployment checks `AIDND_ANALYTICS_EMAILS`.
|
||||||
|
"""
|
||||||
if not MULTI_USER:
|
if not MULTI_USER:
|
||||||
return True
|
return True
|
||||||
return bool(user.email) and user.email.lower() in ANALYTICS_EMAILS
|
return bool(user.email) and user.email.lower() in ANALYTICS_EMAILS
|
||||||
|
|
||||||
|
|
||||||
def demo_turns_left(user: models.User) -> int:
|
def demo_turns_left(user: models.User) -> int:
|
||||||
# Power users are never capped; report the full cap so the banner reads
|
# A power user is never capped, so report the full cap and let the banner
|
||||||
# "N of N" rather than a decrementing count.
|
# read "N of N" rather than count down.
|
||||||
if is_power_user(user):
|
if is_power_user(user):
|
||||||
return DEMO_TURNS_PER_DAY
|
return DEMO_TURNS_PER_DAY
|
||||||
used = user.demo_turns_used if user.demo_turns_date == _today() else 0
|
used = user.demo_turns_used if user.demo_turns_date == _today() else 0
|
||||||
@@ -177,9 +194,9 @@ def demo_turns_left(user: models.User) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def count_demo_turn(user: models.User) -> None:
|
def count_demo_turn(user: models.User) -> None:
|
||||||
"""Record one demo turn; the caller's commit persists it."""
|
"""Records one demo turn. The caller's commit stores it."""
|
||||||
if is_power_user(user):
|
if is_power_user(user):
|
||||||
return # unmetered — power users don't count against the cap
|
return # A power user's turns do not count against the cap.
|
||||||
today = _today()
|
today = _today()
|
||||||
if user.demo_turns_date != today:
|
if user.demo_turns_date != today:
|
||||||
user.demo_turns_date = today
|
user.demo_turns_date = today
|
||||||
@@ -190,8 +207,11 @@ def count_demo_turn(user: models.User) -> None:
|
|||||||
# ---------- User resolution ----------
|
# ---------- User resolution ----------
|
||||||
|
|
||||||
def local_user(db: Session) -> models.User:
|
def local_user(db: Session) -> models.User:
|
||||||
"""The single implicit user in local mode (owns pre-Phase-8 data via
|
"""Returns the single implicit user used in local mode.
|
||||||
migration; created lazily on a fresh database)."""
|
|
||||||
|
A migration gives this user ownership of data written before Phase 8. On a
|
||||||
|
fresh database the user is created on first use.
|
||||||
|
"""
|
||||||
user = (
|
user = (
|
||||||
db.query(models.User)
|
db.query(models.User)
|
||||||
.filter(models.User.email.is_(None), models.User.is_guest.is_(False))
|
.filter(models.User.email.is_(None), models.User.is_guest.is_(False))
|
||||||
@@ -209,7 +229,8 @@ def _touch(user: models.User, db: Session) -> None:
|
|||||||
now = models.utcnow()
|
now = models.utcnow()
|
||||||
last = user.last_seen_at
|
last = user.last_seen_at
|
||||||
if last is not None and last.tzinfo is None:
|
if last is not None and last.tzinfo is None:
|
||||||
# SQLite hands DateTime columns back naive; they were stored as UTC.
|
# SQLite returns DateTime columns without a timezone. They were stored
|
||||||
|
# as UTC.
|
||||||
last = last.replace(tzinfo=timezone.utc)
|
last = last.replace(tzinfo=timezone.utc)
|
||||||
if last is None or (now - last).total_seconds() > 3600:
|
if last is None or (now - last).total_seconds() > 3600:
|
||||||
user.last_seen_at = now
|
user.last_seen_at = now
|
||||||
@@ -227,8 +248,11 @@ def resolve_session_user(request: Request, db: Session) -> models.User | None:
|
|||||||
|
|
||||||
|
|
||||||
def get_current_user(request: Request, db: Session = Depends(get_db)) -> models.User:
|
def get_current_user(request: Request, db: Session = Depends(get_db)) -> models.User:
|
||||||
"""Dependency used by every router. 401 in multi-user mode means the
|
"""The dependency every router uses to resolve the current user.
|
||||||
frontend must (re)establish a session via GET /api/auth/me."""
|
|
||||||
|
In multi-user mode a 401 means the frontend has to establish a session again
|
||||||
|
through `GET /api/auth/me`.
|
||||||
|
"""
|
||||||
if not MULTI_USER:
|
if not MULTI_USER:
|
||||||
user = local_user(db)
|
user = local_user(db)
|
||||||
else:
|
else:
|
||||||
|
|||||||
+156
-139
@@ -1,51 +1,52 @@
|
|||||||
"""Phase 14, SP6 — the export bundle, as a tree.
|
"""Phase 14, SP6: the export bundle, as a tree.
|
||||||
|
|
||||||
A bundle is the one place a story leaves the database, and the only part of the
|
A bundle is the one place a story leaves the database, and the only part of the
|
||||||
tree no migration can ever reach: a file downloaded today has to still import
|
tree no migration can reach. A file downloaded today has to still import into a
|
||||||
into a build shipped next year. So the format is versioned, both versions live
|
build shipped next year. The format is therefore versioned, both versions are
|
||||||
here, and nothing else in the app knows either of them.
|
defined here, and nothing else in the app knows either of them.
|
||||||
|
|
||||||
**v1** is a flat list of turns, each with an optional `variants` array — the
|
Version 1 is a flat list of turns, each with an optional `variants` array, which
|
||||||
repeating group SP4 unpacked into rows. Nothing writes that shape any more.
|
is the repeating group SP4 unpacked into rows. Nothing writes that shape now.
|
||||||
The *reader* stays, because bundles already on people's disks still have it and
|
The reader stays, because bundles already on people's disks still use it, and a
|
||||||
a backup that stops importing is not a backup.
|
backup that stops importing is not a backup.
|
||||||
|
|
||||||
**v2** carries the tree. Three things it holds that v1 could not, each
|
Version 2 carries the tree. It holds three things version 1 could not, and each
|
||||||
load-bearing:
|
one is required:
|
||||||
|
|
||||||
* **the branches**, because a forked adventure is two stories and a flat list
|
* The branches, because a forked adventure is two stories and a flat list holds
|
||||||
can hold one — v1 export interleaved them by `index`, which read as a mangled
|
one. A version 1 export interleaved them by `index`, which read as a garbled
|
||||||
story rather than as lost data;
|
story rather than as lost data.
|
||||||
* **`live`**, because a coordinate can hold several attempts at one turn and
|
* `live`, because a coordinate can hold several attempts at one turn and exactly
|
||||||
exactly one of them is the story;
|
one of them is the story.
|
||||||
* **both after-snapshots**, because they are what a branch switch and an undo
|
* Both after-snapshots, because they are what a branch switch and an undo
|
||||||
put back. A bundle carrying the actions but not the outcomes would import a
|
restore. A bundle carrying the actions but not the outcomes would import a
|
||||||
tree nobody could switch inside.
|
tree nobody could switch inside.
|
||||||
|
|
||||||
## The rule about what a bundle carries
|
## The rule about what a bundle carries
|
||||||
|
|
||||||
**What was chosen, never what is derived.** The head *branch*, the fork points,
|
A bundle carries what was chosen, never what is derived. The head branch, the
|
||||||
the live flags and the anchors are decisions somebody made; they are in the
|
fork points, the live flags, and the anchors are decisions somebody made, so
|
||||||
file. `lineage`, the head *depth*, `index` and the variant ordinals are all
|
they are in the file. `lineage`, the head depth, `index`, and the variant
|
||||||
computed from those, and they are recomputed on import instead:
|
ordinals are all computed from those, and the import recomputes them:
|
||||||
|
|
||||||
* `lineage` is a cache of `parent` + `fork_depth`. Shipping it too would put a
|
* `lineage` is a cache of `parent` plus `fork_depth`. Shipping it as well would
|
||||||
second source of truth for one fact in a file anybody can hand-edit, and the
|
put a second source of truth for one fact into a file anyone can hand-edit,
|
||||||
two could then disagree in a way no read would ever report.
|
and the two could then disagree without any read reporting it.
|
||||||
* the head depth is the tip of the head branch, which is a fact about the nodes
|
* The head depth is the tip of the head branch, which is a fact about the nodes
|
||||||
that arrived with it.
|
that arrived with it.
|
||||||
* `index` is the legacy column SP8 drops. Its one remaining job is to hand the
|
* `index` is the legacy column SP8 drops. Its one remaining job is to give the
|
||||||
next row a number nothing else holds — a fact about the *adventure*, not
|
next row a number nothing else holds, which is a fact about the adventure
|
||||||
about a path — so `depth` cannot be it: two branches have a node at depth 4.
|
rather than about a path, so `depth` cannot serve: two branches each have a
|
||||||
The import allocates one per turn instead, which keeps `max_action_index`
|
node at depth 4. The import allocates one index per turn instead, which keeps
|
||||||
honest and keeps siblings sharing an index the way SP4 leaves them.
|
`max_action_index` correct and keeps siblings sharing an index the way SP4
|
||||||
* the variant ordinals are `attempts.renumber`'s to maintain, and it is the
|
leaves them.
|
||||||
only place allowed to.
|
* `attempts.renumber` maintains the variant ordinals, and it is the only place
|
||||||
|
allowed to.
|
||||||
|
|
||||||
Every hand-editable coordinate is therefore checked before a row is written
|
Every hand-editable coordinate is therefore checked before a row is written, in
|
||||||
(`plan`), not fixed up afterwards: an import that fails halfway leaves an
|
`plan`, rather than repaired afterwards. An import that fails partway leaves an
|
||||||
adventure holding half a tree, and a tree missing a branch is a story that
|
adventure holding half a tree, and a tree missing a branch is a story that stops
|
||||||
silently stops rather than one that reports.
|
without reporting anything.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
@@ -60,24 +61,24 @@ from .context import cursors, lineage
|
|||||||
FORMAT = "ai-dnd-adventure-v2"
|
FORMAT = "ai-dnd-adventure-v2"
|
||||||
LEGACY_FORMAT = "ai-dnd-adventure-v1"
|
LEGACY_FORMAT = "ai-dnd-adventure-v1"
|
||||||
|
|
||||||
# VARCHAR(20) on `actions.type`; a raw-dict import bypasses the schemas.
|
# `actions.type` is VARCHAR(20), and a raw-dict import bypasses the schemas.
|
||||||
TYPE_MAX = 20
|
TYPE_MAX = 20
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- exporting
|
# ---------------------------------------------------------------- exporting
|
||||||
|
|
||||||
def export(db: Session, adventure: models.Adventure) -> dict:
|
def export(db: Session, adventure: models.Adventure) -> dict:
|
||||||
"""The whole adventure as a v2 bundle.
|
"""Returns the whole adventure as a version 2 bundle.
|
||||||
|
|
||||||
Deliberately un-pathed: a backup wants the entire tree, not the branch its
|
The export is not scoped to a path. A backup holds the entire tree, not the
|
||||||
owner happens to be standing on. Both after-snapshots are undeferred in the
|
branch its owner is currently reading. Both after-snapshots are undeferred in
|
||||||
one query — they are per-node columns nothing else reads in bulk, and asking
|
the one query, because they are per-node columns nothing else reads in bulk
|
||||||
for them a row at a time would be a query per turn.
|
and requesting them a row at a time would cost one query per turn.
|
||||||
|
|
||||||
No context snapshots. A bundle has never carried the assembled prompts and
|
The bundle carries no context snapshots. It has never carried the assembled
|
||||||
still does not: they are ~163 kB a turn, they are an explanation of a
|
prompts, and it still does not. They run about 163 kB per turn, they explain
|
||||||
generation rather than part of the story, and the Insights viewer they feed
|
a generation rather than form part of the story, and the Insights viewer they
|
||||||
is reading the adventure it came from.
|
feed reads the adventure they came from.
|
||||||
"""
|
"""
|
||||||
branches = (
|
branches = (
|
||||||
db.query(models.Branch)
|
db.query(models.Branch)
|
||||||
@@ -85,8 +86,9 @@ def export(db: Session, adventure: models.Adventure) -> dict:
|
|||||||
.order_by(models.Branch.id)
|
.order_by(models.Branch.id)
|
||||||
.all()
|
.all()
|
||||||
)
|
)
|
||||||
# Branch ids are local to the file — positions in this list — because the
|
# Branch ids are local to the file and are positions in this list, because
|
||||||
# database ids they had here are already taken over there.
|
# the database ids they hold here are already in use on the importing
|
||||||
|
# side.
|
||||||
local = {branch.id: i for i, branch in enumerate(branches)}
|
local = {branch.id: i for i, branch in enumerate(branches)}
|
||||||
nodes = (
|
nodes = (
|
||||||
db.query(models.Action)
|
db.query(models.Action)
|
||||||
@@ -112,9 +114,9 @@ def export(db: Session, adventure: models.Adventure) -> dict:
|
|||||||
"worldState": adventure.world_state,
|
"worldState": adventure.world_state,
|
||||||
"autoSummarize": adventure.auto_summarize,
|
"autoSummarize": adventure.auto_summarize,
|
||||||
"memoryBankEnabled": adventure.memory_bank_enabled,
|
"memoryBankEnabled": adventure.memory_bank_enabled,
|
||||||
# A root entry even for an adventure whose branch row was never created
|
# Write a root entry even for an adventure whose branch row was never
|
||||||
# — a story with no branch is a pre-tree one, and the tree it belongs to
|
# created. A story with no branch is a pre-tree story, and the tree it
|
||||||
# is the root. `_local` puts its nodes there.
|
# belongs to is the root. `_local` places its nodes there.
|
||||||
"branches": [_exported_branch(b, local) for b in branches] or [_ROOT],
|
"branches": [_exported_branch(b, local) for b in branches] or [_ROOT],
|
||||||
"headBranch": local.get(adventure.head_branch_id, 0),
|
"headBranch": local.get(adventure.head_branch_id, 0),
|
||||||
"memoryCursor": _exported_anchor(adventure, cursors.MEMORY, local),
|
"memoryCursor": _exported_anchor(adventure, cursors.MEMORY, local),
|
||||||
@@ -154,10 +156,10 @@ def _exported_branch(branch: models.Branch, local: dict[int, int]) -> dict:
|
|||||||
dict(_ROOT) if parent is None
|
dict(_ROOT) if parent is None
|
||||||
else {"parent": parent, "forkDepth": branch.fork_depth}
|
else {"parent": parent, "forkDepth": branch.fork_depth}
|
||||||
)
|
)
|
||||||
# A name is something a player chose, so it travels — the same rule that
|
# A player chose the name, so it goes into the file. That is the same rule
|
||||||
# puts the fork points in the file and leaves `lineage` out. An unnamed
|
# that puts the fork points in the file and leaves `lineage` out. An unnamed
|
||||||
# branch omits the key rather than carrying a null, which keeps the file
|
# branch omits the key rather than carry a null, which keeps the file for an
|
||||||
# for a tree nobody has named byte-identical to the one SP6 wrote.
|
# unnamed tree byte-identical to the one SP6 wrote.
|
||||||
if branch.name:
|
if branch.name:
|
||||||
out["name"] = branch.name
|
out["name"] = branch.name
|
||||||
return out
|
return out
|
||||||
@@ -166,7 +168,7 @@ def _exported_branch(branch: models.Branch, local: dict[int, int]) -> dict:
|
|||||||
def _exported_node(action: models.Action, local: dict[int, int]) -> dict:
|
def _exported_node(action: models.Action, local: dict[int, int]) -> dict:
|
||||||
node = {
|
node = {
|
||||||
"branch": _local(action.branch_id, local),
|
"branch": _local(action.branch_id, local),
|
||||||
# A pre-tree row's depth is the number `index` already held.
|
# A pre-tree row's depth is the number `index` already holds.
|
||||||
"depth": action.depth if action.depth is not None else action.index,
|
"depth": action.depth if action.depth is not None else action.index,
|
||||||
"live": bool(action.live),
|
"live": bool(action.live),
|
||||||
"type": action.type,
|
"type": action.type,
|
||||||
@@ -175,11 +177,11 @@ def _exported_node(action: models.Action, local: dict[int, int]) -> dict:
|
|||||||
}
|
}
|
||||||
if action.reasoning:
|
if action.reasoning:
|
||||||
node["reasoning"] = action.reasoning
|
node["reasoning"] = action.reasoning
|
||||||
# `{}` and absent mean different things — "this node left an empty
|
# `{}` and an absent key mean different things. `{}` means the node left an
|
||||||
# scoreboard behind" against "nobody knows, leave the live state alone" —
|
# empty state behind, and an absent key means the state is unknown and the
|
||||||
# so an empty snapshot is written out rather than trimmed. It costs about
|
# live state stays as it is. An empty snapshot is therefore written rather
|
||||||
# eighteen bytes a row and it is the difference between an undo that clears
|
# than omitted. It costs about eighteen bytes per row, and it decides
|
||||||
# a score and one that leaves it standing.
|
# whether an undo clears a score or leaves it in place.
|
||||||
if action.state_after is not None:
|
if action.state_after is not None:
|
||||||
node["stateAfter"] = action.state_after
|
node["stateAfter"] = action.state_after
|
||||||
if action.world_state_after is not None:
|
if action.world_state_after is not None:
|
||||||
@@ -194,8 +196,9 @@ def _exported_memory(memory: models.Memory, local: dict[int, int]) -> dict:
|
|||||||
"text": memory.text, "pinned": memory.pinned, "forgotten": memory.forgotten,
|
"text": memory.text, "pinned": memory.pinned, "forgotten": memory.forgotten,
|
||||||
"sourceStart": memory.source_start, "sourceEnd": memory.source_end,
|
"sourceStart": memory.source_start, "sourceEnd": memory.source_end,
|
||||||
"useCount": memory.use_count,
|
"useCount": memory.use_count,
|
||||||
# The node it hangs off. A hand-written memory summarises no node, so it
|
# The node this memory is attached to. A hand-written memory summarizes
|
||||||
# has a branch and no depth, and keeps that shape here.
|
# no node, so it has a branch and no depth, and it keeps that shape
|
||||||
|
# here.
|
||||||
"branch": _local(memory.branch_id, local),
|
"branch": _local(memory.branch_id, local),
|
||||||
"depth": memory.depth,
|
"depth": memory.depth,
|
||||||
}
|
}
|
||||||
@@ -214,7 +217,7 @@ def _exported_anchor(
|
|||||||
# ---------------------------------------------------------------- importing
|
# ---------------------------------------------------------------- importing
|
||||||
|
|
||||||
def check_format(bundle: dict) -> str:
|
def check_format(bundle: dict) -> str:
|
||||||
"""The bundle's version, or 400."""
|
"""Returns the bundle's version, or raises a 400."""
|
||||||
fmt = bundle.get("format")
|
fmt = bundle.get("format")
|
||||||
if fmt in (FORMAT, LEGACY_FORMAT):
|
if fmt in (FORMAT, LEGACY_FORMAT):
|
||||||
return fmt
|
return fmt
|
||||||
@@ -225,16 +228,16 @@ def check_format(bundle: dict) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def plan(bundle: dict, version: str) -> dict:
|
def plan(bundle: dict, version: str) -> dict:
|
||||||
"""The bundle's tree, checked and normalised, before a row is written.
|
"""Returns the bundle's tree, checked and normalized, before a row is written.
|
||||||
|
|
||||||
Pure: no session, no adventure, nothing created. Everything a hand-edited
|
The function has no side effects. It opens no session, needs no adventure,
|
||||||
file can get wrong about the *shape* of a tree is caught here, because the
|
and creates nothing. It catches everything a hand-edited file can get wrong
|
||||||
alternative is an import that fails partway and leaves an adventure holding
|
about the shape of a tree, because the alternative is an import that fails
|
||||||
a story with a hole in it.
|
partway and leaves an adventure holding a story with a gap in it.
|
||||||
|
|
||||||
Both versions land in the same shape, so `write` never learns there are two
|
Both versions produce the same shape, so `write` never learns that there are
|
||||||
formats: a v1 bundle is a linear story, which is a tree with one branch, and
|
two formats. A version 1 bundle is a linear story, which is a tree with one
|
||||||
its `variants` array is a sibling group written the old way.
|
branch, and its `variants` array is a sibling group written the old way.
|
||||||
"""
|
"""
|
||||||
branches = (
|
branches = (
|
||||||
_planned_branches(bundle) if version == FORMAT else [dict(_ROOT)]
|
_planned_branches(bundle) if version == FORMAT else [dict(_ROOT)]
|
||||||
@@ -248,8 +251,9 @@ def plan(bundle: dict, version: str) -> dict:
|
|||||||
"nodes": nodes,
|
"nodes": nodes,
|
||||||
"memories": _planned_memories(bundle, len(branches)),
|
"memories": _planned_memories(bundle, len(branches)),
|
||||||
"head": _as_index(bundle.get("headBranch"), len(branches), default=0),
|
"head": _as_index(bundle.get("headBranch"), len(branches), default=0),
|
||||||
# v2 knows where the derived work got to; v1 counted it, and a count
|
# Version 2 records where the derived work reached. Version 1 counted
|
||||||
# cannot be turned into a node until the nodes exist (see `settle`).
|
# it, and a count cannot become a node until the nodes exist. See
|
||||||
|
# `settle`.
|
||||||
"anchors": _planned_anchors(bundle, len(branches)) if version == FORMAT else None,
|
"anchors": _planned_anchors(bundle, len(branches)) if version == FORMAT else None,
|
||||||
"positions": None if version == FORMAT else {
|
"positions": None if version == FORMAT else {
|
||||||
"memory": _as_int(bundle.get("memoryCursor"), 0),
|
"memory": _as_int(bundle.get("memoryCursor"), 0),
|
||||||
@@ -270,11 +274,12 @@ def _planned_branches(bundle: dict) -> list[dict]:
|
|||||||
if parent is None:
|
if parent is None:
|
||||||
specs.append(dict(_ROOT, **({"name": name} if name else {})))
|
specs.append(dict(_ROOT, **({"name": name} if name else {})))
|
||||||
continue
|
continue
|
||||||
# A branch may only fork from one listed before it. That is how the
|
# A branch may fork only from a branch listed before it. The export
|
||||||
# export writes them — branches are numbered in creation order and a
|
# writes them that way, because branches are numbered in creation order
|
||||||
# parent always exists first — and requiring it here buys acyclicity for
|
# and a parent always exists first. Requiring it here guarantees the
|
||||||
# the price of a comparison: a lineage is computed by walking to the
|
# graph is acyclic for the cost of one comparison. A lineage is computed
|
||||||
# parent, and a cycle would be an import that never returns.
|
# by walking to the parent, so a cycle would be an import that never
|
||||||
|
# returns.
|
||||||
if not _is_int(parent) or not 0 <= parent < i:
|
if not _is_int(parent) or not 0 <= parent < i:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
400,
|
400,
|
||||||
@@ -296,11 +301,12 @@ def _planned_branches(bundle: dict) -> list[dict]:
|
|||||||
|
|
||||||
|
|
||||||
def _planned_branch_name(entry: dict, i: int) -> str | None:
|
def _planned_branch_name(entry: dict, i: int) -> str | None:
|
||||||
"""The name a branch entry carries, or None for one nobody named.
|
"""Returns the name a branch entry carries, or `None` if nobody named it.
|
||||||
|
|
||||||
Checked before the row is created rather than left to the column, for the
|
The check runs before the row is created rather than being left to the
|
||||||
reason the whole planner exists: a 400 from a pure function beats a half
|
column, for the reason the planner exists. A 400 from a function with no side
|
||||||
written adventure and a database error from three branches in.
|
effects is better than a half-written adventure and a database error three
|
||||||
|
branches in.
|
||||||
"""
|
"""
|
||||||
raw = entry.get("name")
|
raw = entry.get("name")
|
||||||
if raw is None:
|
if raw is None:
|
||||||
@@ -351,15 +357,16 @@ def _planned_nodes(bundle: dict, branches: int) -> list[dict]:
|
|||||||
|
|
||||||
|
|
||||||
def _planned_v1_nodes(bundle: dict) -> list[dict]:
|
def _planned_v1_nodes(bundle: dict) -> list[dict]:
|
||||||
"""A v1 bundle's turns as the nodes they describe: one per attempt.
|
"""Returns a version 1 bundle's turns as nodes, one node per attempt.
|
||||||
|
|
||||||
The `variants` array is the repeating group SP4 unpacked, so reading one is
|
The `variants` array is the repeating group SP4 unpacked, so reading one
|
||||||
the same split migration 60 does — every attempt becomes a row at the turn's
|
performs the same split that migration 60 does. Every attempt becomes a row
|
||||||
coordinate and `variantIndex` picks which is live. Clamped, because a
|
at the turn's coordinate, and `variantIndex` selects the live one. The index
|
||||||
hand-edited bundle can name an attempt its own list does not have, and a
|
is clamped, because a hand-edited bundle can name an attempt its own list
|
||||||
turn with no live node is a turn no read can see.
|
does not contain, and a turn with no live node is a turn no read can see.
|
||||||
|
|
||||||
The depth is the bundle's `index`: v1 is one branch, where the two agree.
|
The depth is the bundle's `index`. A version 1 bundle has one branch, where
|
||||||
|
the two numbers agree.
|
||||||
"""
|
"""
|
||||||
raw = bundle.get("actions")
|
raw = bundle.get("actions")
|
||||||
nodes: list[dict] = []
|
nodes: list[dict] = []
|
||||||
@@ -404,9 +411,10 @@ def _planned_memories(bundle: dict, branches: int) -> list[dict]:
|
|||||||
"sourceStart": entry.get("sourceStart"),
|
"sourceStart": entry.get("sourceStart"),
|
||||||
"sourceEnd": entry.get("sourceEnd"),
|
"sourceEnd": entry.get("sourceEnd"),
|
||||||
"useCount": _as_int(entry.get("useCount"), 0),
|
"useCount": _as_int(entry.get("useCount"), 0),
|
||||||
# Out of range rather than absent means a file that disagrees with
|
# A value that is out of range, rather than absent, means the file
|
||||||
# itself; the root is the safe reading, because a memory on a branch
|
# disagrees with itself. The root is the safe reading, because a
|
||||||
# nothing can see is a memory that never reaches a prompt again.
|
# memory on a branch nothing can see never reaches a prompt
|
||||||
|
# again.
|
||||||
"branch": _as_index(entry.get("branch"), branches, default=0),
|
"branch": _as_index(entry.get("branch"), branches, default=0),
|
||||||
"depth": entry.get("depth") if _is_int(entry.get("depth")) else None,
|
"depth": entry.get("depth") if _is_int(entry.get("depth")) else None,
|
||||||
})
|
})
|
||||||
@@ -429,10 +437,10 @@ def _planned_anchors(bundle: dict, branches: int) -> dict:
|
|||||||
# ------------------------------------------------------------------ writing
|
# ------------------------------------------------------------------ writing
|
||||||
|
|
||||||
def write(db: Session, adventure: models.Adventure, story: dict) -> None:
|
def write(db: Session, adventure: models.Adventure, story: dict) -> None:
|
||||||
"""Write a planned tree onto a freshly created adventure.
|
"""Writes a planned tree onto a newly created adventure.
|
||||||
|
|
||||||
Order matters and is not negotiable: branches first, because a node needs an
|
The order is fixed. Branches come first, because a node needs a branch id.
|
||||||
id to hang off; then the nodes, because the head and the anchors name one.
|
The nodes come next, because the head and the anchors name a node.
|
||||||
"""
|
"""
|
||||||
ids = _write_branches(db, adventure, story["branches"])
|
ids = _write_branches(db, adventure, story["branches"])
|
||||||
_write_nodes(db, adventure, story["nodes"], ids)
|
_write_nodes(db, adventure, story["nodes"], ids)
|
||||||
@@ -444,13 +452,13 @@ def write(db: Session, adventure: models.Adventure, story: dict) -> None:
|
|||||||
def _write_branches(
|
def _write_branches(
|
||||||
db: Session, adventure: models.Adventure, specs: list[dict]
|
db: Session, adventure: models.Adventure, specs: list[dict]
|
||||||
) -> list[int]:
|
) -> list[int]:
|
||||||
"""One row per branch, lineage computed rather than read.
|
"""Writes one row per branch, computing the lineage rather than reading it.
|
||||||
|
|
||||||
Inserted through Core and the lineage written second, for the reason
|
The rows are inserted through Core and the lineage is written second, for the
|
||||||
`tree.root_branch` spells out: the lineage names the row's own id. The
|
reason `tree.root_branch` gives: the lineage names the row's own id. The
|
||||||
parent's cached ancestry is capped at this fork, which is the same
|
parent's cached ancestry is capped at this fork, which is the arithmetic
|
||||||
arithmetic `tree.fork` does — a fork made now and a fork made a year ago and
|
`tree.fork` performs. A fork made now and a fork made a year ago and then
|
||||||
exported must produce the same rows.
|
exported have to produce the same rows.
|
||||||
"""
|
"""
|
||||||
ids: list[int] = []
|
ids: list[int] = []
|
||||||
lineages: list[list[list]] = []
|
lineages: list[list[list]] = []
|
||||||
@@ -486,14 +494,14 @@ def _write_branches(
|
|||||||
def _write_nodes(
|
def _write_nodes(
|
||||||
db: Session, adventure: models.Adventure, specs: list[dict], ids: list[int]
|
db: Session, adventure: models.Adventure, specs: list[dict], ids: list[int]
|
||||||
) -> None:
|
) -> None:
|
||||||
"""The nodes, grouped into the turns they are attempts at.
|
"""Writes the nodes, grouped into the turns they are attempts at.
|
||||||
|
|
||||||
Two things are allocated here rather than trusted from the file. `index` is
|
Two values are allocated here rather than read from the file. `index` is
|
||||||
handed out one per *turn*, in the order the bundle lists them, so siblings
|
issued once per turn, in the order the bundle lists them, so siblings share
|
||||||
share one and no two coordinates do — which is what `max_action_index` needs
|
one index and no two coordinates do. `max_action_index` needs that to keep
|
||||||
to keep issuing numbers nothing holds. And exactly one attempt in each group
|
issuing numbers nothing holds. Exactly one attempt in each group is also made
|
||||||
is made live: a file can name none or several, and a turn with no live node
|
live, because a file can name none or several, and a turn with no live node
|
||||||
is a turn that vanishes from the story.
|
disappears from the story.
|
||||||
"""
|
"""
|
||||||
groups: dict[tuple[int, int], list[models.Action]] = {}
|
groups: dict[tuple[int, int], list[models.Action]] = {}
|
||||||
indices: dict[tuple[int, int], int] = {}
|
indices: dict[tuple[int, int], int] = {}
|
||||||
@@ -541,18 +549,19 @@ def _write_memories(
|
|||||||
branch_id=ids[spec["branch"]],
|
branch_id=ids[spec["branch"]],
|
||||||
depth=spec["depth"],
|
depth=spec["depth"],
|
||||||
)
|
)
|
||||||
# A v1 memory has no depth of its own; `source_end` is the index of the
|
# A version 1 memory has no depth of its own. `source_end` is the index
|
||||||
# last action it summarises, which on one branch is that node's depth.
|
# of the last action it summarizes, which on one branch is that node's
|
||||||
|
# depth.
|
||||||
if memory.depth is None and memory.source_end is not None:
|
if memory.depth is None and memory.source_end is not None:
|
||||||
memory.depth = memory.source_end
|
memory.depth = memory.source_end
|
||||||
# A v1 memory that summarises nothing — one the player typed — has no
|
# A version 1 memory that summarizes nothing, which means one the player
|
||||||
# depth to derive, and leaving it NULL here would rebuild by import the
|
# typed, has no depth to derive. Leaving it NULL here would recreate on
|
||||||
# exact state migration 62 exists to end: `Path._entry_clause` compares
|
# import the state migration 62 exists to end. `Path._entry_clause`
|
||||||
# `depth <= max_depth`, which a NULL fails, so the memory would vanish
|
# compares `depth <= max_depth`, which a NULL fails, so the memory would
|
||||||
# from every branch the moment the imported adventure was forked. The
|
# disappear from every branch as soon as the imported adventure was
|
||||||
# root is the same answer the migration gives, and for the same reason
|
# forked. The root is the answer the migration gives, for the same
|
||||||
# — 0 is at or before every fork point, so it is visible from every
|
# reason: 0 is at or before every fork point, so the memory is visible
|
||||||
# path this adventure can grow.
|
# from every path this adventure can grow.
|
||||||
if memory.depth is None:
|
if memory.depth is None:
|
||||||
memory.depth = lineage.ROOT_DEPTH
|
memory.depth = lineage.ROOT_DEPTH
|
||||||
db.add(memory)
|
db.add(memory)
|
||||||
@@ -561,12 +570,13 @@ def _write_memories(
|
|||||||
def _point_the_head(
|
def _point_the_head(
|
||||||
adventure: models.Adventure, story: dict, ids: list[int]
|
adventure: models.Adventure, story: dict, ids: list[int]
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Where the story is being played, and how deep it goes.
|
"""Sets which branch the story is played on, and how deep it goes.
|
||||||
|
|
||||||
The branch comes from the file and the depth does not: the tip of a branch
|
The branch comes from the file and the depth does not. The tip of a branch is
|
||||||
is whatever arrived on it, and a branch with nothing of its own sits at its
|
whatever arrived on it, and a branch with no nodes of its own sits at its
|
||||||
fork point — the last node its story contains, borrowed but the tip all the
|
fork point, which is the last node its story contains. That node is borrowed
|
||||||
same. The same rule as `tree.refresh_head`, applied before a flush.
|
but it is still the tip. This is the rule `tree.refresh_head` applies, run
|
||||||
|
here before a flush.
|
||||||
"""
|
"""
|
||||||
head = story["head"]
|
head = story["head"]
|
||||||
adventure.head_branch_id = ids[head]
|
adventure.head_branch_id = ids[head]
|
||||||
@@ -581,10 +591,11 @@ def _point_the_head(
|
|||||||
def _write_anchors(
|
def _write_anchors(
|
||||||
adventure: models.Adventure, story: dict, ids: list[int]
|
adventure: models.Adventure, story: dict, ids: list[int]
|
||||||
) -> None:
|
) -> None:
|
||||||
"""How far the memories and the summary have read — v2 only.
|
"""Records how far the memories and the summary have read. Version 2 only.
|
||||||
|
|
||||||
A v1 bundle counts instead, and a count cannot be resolved to a node until
|
A version 1 bundle stores a count instead, and a count cannot be resolved to
|
||||||
the nodes are in the database; `settle` does that half afterwards.
|
a node until the nodes are in the database. `settle` does that part
|
||||||
|
afterwards.
|
||||||
"""
|
"""
|
||||||
anchors = story["anchors"]
|
anchors = story["anchors"]
|
||||||
if anchors is None:
|
if anchors is None:
|
||||||
@@ -595,15 +606,17 @@ def _write_anchors(
|
|||||||
|
|
||||||
|
|
||||||
def settle(db: Session, adventure: models.Adventure, story: dict) -> None:
|
def settle(db: Session, adventure: models.Adventure, story: dict) -> None:
|
||||||
"""Line up the two coordinate systems, once the nodes exist.
|
"""Aligns the two coordinate systems, once the nodes exist.
|
||||||
|
|
||||||
The anchors and the legacy counts describe the same boundary in different
|
The anchors and the legacy counts describe the same boundary in different
|
||||||
words, and each version of the bundle brings one of them. A v1 file brings
|
terms, and each version of the bundle carries one of them. A version 1 file
|
||||||
the count, so the anchor is found by counting that far along the story; a v2
|
carries the count, so the anchor is found by counting that far along the
|
||||||
file brings the anchor, so the count is read back off it. The legacy columns
|
story. A version 2 file carries the anchor, so the count is read back from
|
||||||
are still written either way — they are what a rolled-back build reads.
|
it. The legacy columns are written either way, because a rolled-back build
|
||||||
|
reads them.
|
||||||
|
|
||||||
Called after the flush, because both directions need the actions queryable.
|
The caller runs this after the flush, because both directions need the
|
||||||
|
actions to be queryable.
|
||||||
"""
|
"""
|
||||||
positions = story["positions"]
|
positions = story["positions"]
|
||||||
for cursor in cursors.ALL:
|
for cursor in cursors.ALL:
|
||||||
@@ -618,11 +631,12 @@ def settle(db: Session, adventure: models.Adventure, story: dict) -> None:
|
|||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------ reading
|
# ------------------------------------------------------------------ reading
|
||||||
# Small coercions. A raw-dict import bypasses the schemas entirely, so
|
# Small coercions. A raw-dict import bypasses the schemas, so every value from a
|
||||||
# everything out of a bundle is whatever JSON happened to hold.
|
# bundle is whatever the JSON held.
|
||||||
|
|
||||||
def _is_int(value) -> bool:
|
def _is_int(value) -> bool:
|
||||||
"""`True` is an `int` in Python, and is not one in a coordinate."""
|
"""Returns whether `value` is an integer. Python counts `True` as an `int`,
|
||||||
|
and a coordinate does not."""
|
||||||
return isinstance(value, int) and not isinstance(value, bool)
|
return isinstance(value, int) and not isinstance(value, bool)
|
||||||
|
|
||||||
|
|
||||||
@@ -631,9 +645,12 @@ def _as_int(value, default: int) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def _as_index(value, count: int, default: int | None = None) -> int | None:
|
def _as_index(value, count: int, default: int | None = None) -> int | None:
|
||||||
"""A local branch number, or `default` when the file names one that is not
|
"""Returns a local branch number, or `default` when the file names a branch
|
||||||
there. Out of range is a file disagreeing with itself, not a shape a read
|
that is not present.
|
||||||
can be handed."""
|
|
||||||
|
An out-of-range value means the file disagrees with itself, and it is not a
|
||||||
|
value any read can be given.
|
||||||
|
"""
|
||||||
return value if _is_int(value) and 0 <= value < count else default
|
return value if _is_int(value) and 0 <= value < count else default
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+35
-31
@@ -1,10 +1,11 @@
|
|||||||
"""Retention policy for throwaway guest accounts.
|
"""Retention policy for throwaway guest accounts.
|
||||||
|
|
||||||
In multi-user mode every first visit mints a `users` row (GET /api/auth/me),
|
In multi-user mode every first visit creates a `users` row through
|
||||||
so a public demo accumulates one account per curious visitor — most of whom
|
`GET /api/auth/me`, so a public demo accumulates one account per visitor. Most
|
||||||
never come back, each leaving behind whatever scenarios, adventures, actions
|
of those visitors never return, and each one leaves behind whatever scenarios,
|
||||||
and memories they generated. This drops guests that have gone quiet for
|
adventures, actions, and memories they generated. This module deletes guests
|
||||||
AIDND_GUEST_RETENTION_DAYS (default 5) along with everything they made.
|
that have been inactive for `AIDND_GUEST_RETENTION_DAYS`, which defaults to 5,
|
||||||
|
along with everything they made.
|
||||||
|
|
||||||
Why this is safe to run unattended:
|
Why this is safe to run unattended:
|
||||||
|
|
||||||
@@ -12,25 +13,26 @@ Why this is safe to run unattended:
|
|||||||
clauses are checked rather than either alone. Registering upgrades the row
|
clauses are checked rather than either alone. Registering upgrades the row
|
||||||
in place (is_guest -> False), so a guest who signs up keeps everything;
|
in place (is_guest -> False), so a guest who signs up keeps everything;
|
||||||
local mode's implicit single user is also is_guest=False.
|
local mode's implicit single user is also is_guest=False.
|
||||||
- Idle time is COALESCE(last_seen_at, created_at). `auth._touch` only writes
|
- Idle time is `COALESCE(last_seen_at, created_at)`. `auth._touch` writes
|
||||||
last_seen_at once an hour, and a guest minted by /auth/me has NULL until
|
`last_seen_at` at most once an hour, and a guest created by `/auth/me` has
|
||||||
its *second* request, so created_at is the honest floor for a brand-new
|
NULL there until its second request, so `created_at` is the correct floor for
|
||||||
visitor — without the coalesce those rows look infinitely old.
|
a new visitor. Without the coalesce, those rows look arbitrarily old.
|
||||||
- Nothing a guest owns is reachable by anyone else: `is_public` is an
|
- Nothing a guest owns is reachable by anyone else. `is_public` is an
|
||||||
output-only field (see schemas.ScenarioBase), so only seeded scenarios —
|
output-only field, as `schemas.ScenarioBase` shows, so the only shared
|
||||||
which have user_id NULL and are therefore outside this filter entirely —
|
scenarios are the seeded ones, which have a NULL `user_id` and are outside
|
||||||
are shared. Deleting a guest can't take content away from another user.
|
this filter. Deleting a guest cannot remove content from another user.
|
||||||
|
|
||||||
Why one Core DELETE instead of an ORM cascade: `db.delete(user)` would SELECT
|
The sweep uses one Core DELETE rather than an ORM cascade. `db.delete(user)`
|
||||||
every adventure, action, memory and story card into Python purely to delete
|
would SELECT every adventure, action, memory, and story card into Python only to
|
||||||
them, which on Neon is exactly the egress pattern that has already cost this
|
delete them, which on Neon is the egress pattern that has already cost this
|
||||||
project once. Every foreign key from users downwards is ON DELETE CASCADE
|
project once. Every foreign key from `users` downward is ON DELETE CASCADE, from
|
||||||
(users -> scenarios/adventures/scripts/settings -> actions/memories/cards), so
|
users to scenarios, adventures, scripts, and settings, and from those to actions,
|
||||||
the database does the whole graph in one statement and ships back a row count.
|
memories, and cards, so the database deletes the whole graph in one statement and
|
||||||
|
returns a row count.
|
||||||
|
|
||||||
No index is added for the scan: the sweep runs a handful of times a day
|
The scan gets no index. The sweep runs a few times a day against a table holding
|
||||||
against a table with at most a few thousand rows, which is not worth a
|
at most a few thousand rows, which does not justify a migration and the schema
|
||||||
migration and the schema surface that comes with it.
|
surface it adds.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
@@ -56,8 +58,8 @@ def _int_env(name: str, default: int) -> int:
|
|||||||
return default
|
return default
|
||||||
|
|
||||||
|
|
||||||
# Days of inactivity before a guest account is dropped. 0 or less disables the
|
# Days of inactivity before a guest account is deleted. A value of 0 or less
|
||||||
# policy entirely, for a deployment that would rather keep everything.
|
# disables the policy, for a deployment that keeps everything.
|
||||||
RETENTION_DAYS = _int_env("AIDND_GUEST_RETENTION_DAYS", 5)
|
RETENTION_DAYS = _int_env("AIDND_GUEST_RETENTION_DAYS", 5)
|
||||||
|
|
||||||
# How often a long-lived process re-checks. Hours, not minutes: nothing here is
|
# How often a long-lived process re-checks. Hours, not minutes: nothing here is
|
||||||
@@ -86,10 +88,11 @@ def delete_stale_guests(db: Session, *, now: datetime | None = None) -> int:
|
|||||||
"""
|
"""
|
||||||
if RETENTION_DAYS <= 0:
|
if RETENTION_DAYS <= 0:
|
||||||
return 0
|
return 0
|
||||||
# Stored timestamps are naive UTC on both backends (SQLite drops tzinfo;
|
# Stored timestamps are UTC without a timezone on both backends. SQLite
|
||||||
# Postgres columns are TIMESTAMP WITHOUT TIME ZONE with the session pinned
|
# drops the timezone, and the Postgres columns are TIMESTAMP WITHOUT TIME
|
||||||
# to UTC in database.py). Match that exactly so the comparison can't hinge
|
# ZONE with the session pinned to UTC in `database.py`. Match that, so the
|
||||||
# on how a given dialect renders an aware value.
|
# comparison does not depend on how a dialect renders a value that carries a
|
||||||
|
# timezone.
|
||||||
reference = now or models.utcnow()
|
reference = now or models.utcnow()
|
||||||
cutoff = reference.replace(tzinfo=None) - timedelta(days=RETENTION_DAYS)
|
cutoff = reference.replace(tzinfo=None) - timedelta(days=RETENTION_DAYS)
|
||||||
|
|
||||||
@@ -100,9 +103,10 @@ def delete_stale_guests(db: Session, *, now: datetime | None = None) -> int:
|
|||||||
models.User.email.is_(None),
|
models.User.email.is_(None),
|
||||||
func.coalesce(models.User.last_seen_at, models.User.created_at) < cutoff,
|
func.coalesce(models.User.last_seen_at, models.User.created_at) < cutoff,
|
||||||
)
|
)
|
||||||
# Without this, "auto" can't evaluate coalesce in Python and falls back
|
# Without this option, the "auto" strategy cannot evaluate coalesce in
|
||||||
# to fetching every matching primary key first — a second round trip
|
# Python and falls back to fetching every matching primary key first.
|
||||||
# for nothing, since this session holds no User objects to synchronize.
|
# That is a second round trip for no benefit, because this session holds
|
||||||
|
# no User objects to synchronize.
|
||||||
.execution_options(synchronize_session=False)
|
.execution_options(synchronize_session=False)
|
||||||
)
|
)
|
||||||
removed = db.execute(stmt).rowcount or 0
|
removed = db.execute(stmt).rowcount or 0
|
||||||
|
|||||||
+11
-11
@@ -1,18 +1,18 @@
|
|||||||
"""Storing a JSON column compressed.
|
"""Storing a JSON column compressed.
|
||||||
|
|
||||||
`actions.context_snapshot` holds the entire assembled prompt for a turn. It is
|
`actions.context_snapshot` holds the entire assembled prompt for a turn. It is
|
||||||
89% of the database — 150.8 MB of JSON across 944 actions on production, and
|
89% of the database, which is 150.8 MB of JSON across 944 actions in production
|
||||||
232 KB a row on the longest adventure — and the free tier this deploys to
|
and 232 kB per row on the longest adventure, and the free tier this deploys to
|
||||||
allows 512 MB. Reads are not the problem: the column is deferred, so a page
|
allows 512 MB. Reads are not the problem. The column is deferred, so a page load
|
||||||
load never touches it and exactly one endpoint fetches one row of it at a
|
never touches it and exactly one endpoint fetches one row of it at a time.
|
||||||
time. Storage is the problem, and storage has a cliff.
|
Storage is the problem, and storage has a hard limit.
|
||||||
|
|
||||||
Postgres already compresses it. TOAST brings 150.8 MB down to ~89 MB, a factor
|
Postgres already compresses the column. TOAST brings 150.8 MB down to about
|
||||||
of 1.7 — pglz is chosen for decompression speed on data a query might filter
|
89 MB, a factor of 1.7. Postgres chooses pglz for decompression speed on data a
|
||||||
on, which this never is. Nothing filters on a prompt; it is written once and
|
query might filter on, and no query filters on this column. A prompt is written
|
||||||
read whole, occasionally, by one screen. zlib at the application layer gets
|
once and read whole, occasionally, by one screen. zlib at the application layer
|
||||||
three to four times on the same text, and the cost is a decompress on a
|
reaches three to four times on the same text, and the cost is one decompression
|
||||||
request that already costs an LLM call.
|
on a request that already makes an LLM call.
|
||||||
|
|
||||||
Doing it as a TypeDecorator rather than a second column keeps every call site
|
Doing it as a TypeDecorator rather than a second column keeps every call site
|
||||||
writing `action.context_snapshot = {...}` and reading a dict back, and keeps
|
writing `action.context_snapshot = {...}` and reading a dict back, and keeps
|
||||||
|
|||||||
+112
-98
@@ -10,11 +10,11 @@
|
|||||||
[Author's Note] injected AUTHORS_NOTE_DEPTH actions before the end of history
|
[Author's Note] injected AUTHORS_NOTE_DEPTH actions before the end of history
|
||||||
[Latest player action] (+ script frontMemory right after it, Phase 4)
|
[Latest player action] (+ script frontMemory right after it, Phase 4)
|
||||||
|
|
||||||
Which components are present is AI Dungeon's design, above. The *order* they
|
The list above comes from AI Dungeon's design. The order does not. This module
|
||||||
are laid out in is not: everything fixed is emitted first and everything that
|
emits every fixed section first and every changing section after the history,
|
||||||
moves after the history, because prompt caching bills on a shared prefix and
|
because prompt caching bills on a shared prefix. A section that changes near the
|
||||||
one mutable section high up re-prices the whole prompt under it. See the two
|
top of the prompt re-prices everything below it. See the comments on the static
|
||||||
"static block" / "live sections" comments in `build_context`.
|
block and the live sections in `build_context`.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import functools
|
import functools
|
||||||
@@ -30,33 +30,33 @@ CARD_BUDGET_SHARE = 0.4 # max share of non-reserved budget that story cards may
|
|||||||
NPC_WINDOW = 6 # actions of story searched for NPC trigger words ("in scene")
|
NPC_WINDOW = 6 # actions of story searched for NPC trigger words ("in scene")
|
||||||
SEPARATOR = "\n\n"
|
SEPARATOR = "\n\n"
|
||||||
|
|
||||||
# Output-length guidance. max_output_tokens is a hard wall the endpoint enforces
|
# Output-length guidance. The endpoint enforces `max_output_tokens` as a hard
|
||||||
# mid-sentence: hitting it truncates whatever is being written, and since the
|
# limit, and it truncates the reply mid-sentence when the model reaches it. The
|
||||||
# state block is emitted last, it is what gets lost. Asking the model to land
|
# state block is emitted last, so truncation removes it. Asking the model to
|
||||||
# just inside the wall keeps the cut from happening in the first place.
|
# finish inside the limit prevents the truncation.
|
||||||
LENGTH_HEADROOM = 50 # tokens held back from the cap for the state block itself
|
LENGTH_HEADROOM = 50 # Tokens reserved from the cap for the state block.
|
||||||
# Models cannot count their own tokens, but they do follow a word budget, so the
|
# Models cannot count their own tokens, but they do follow a word budget, so the
|
||||||
# reserved budget is stated in words. ~0.75 words per token for English prose.
|
# hint states a number of words. English prose averages 0.75 words per token.
|
||||||
WORDS_PER_TOKEN = 0.75
|
WORDS_PER_TOKEN = 0.75
|
||||||
# A word budget is a suggestion the model routinely overshoots, and the cap it is
|
# Models regularly exceed a word budget, and the cap it protects is a hard
|
||||||
# protecting is a hard wall — so aim 10% short of the real ceiling and let the
|
# limit. Aiming 10% below the real ceiling leaves room for that overshoot, so it
|
||||||
# overshoot land in the slack instead of in the state block.
|
# does not consume the state block.
|
||||||
LENGTH_BUFFER = 0.90
|
LENGTH_BUFFER = 0.90
|
||||||
MIN_LENGTH_HINT_WORDS = 40 # below this the hint is noise; a tiny cap speaks for itself
|
MIN_LENGTH_HINT_WORDS = 40 # Below this, the hint adds nothing useful.
|
||||||
# A ceiling alone is a one-sided instruction, and models read it very differently:
|
# A ceiling on its own gives one-sided guidance, and models respond to it
|
||||||
# a verbose one is held back by it, while a terse one has nothing to act on except
|
# differently. A verbose model treats it as a limit. A terse model has only the
|
||||||
# the "write only as much as the moment needs" clause and collapses to two
|
# instruction to write as much as the moment needs, and it produces two
|
||||||
# paragraphs. Stating a floor as well turns the guidance into a band, so the same
|
# paragraphs. Adding a floor turns the guidance into a range, so the same prompt
|
||||||
# prompt lands in the same place regardless of which way the model leans. Set as a
|
# produces a similar length from either model. The floor is a share of the
|
||||||
# share of the ceiling so the floor can never approach it.
|
# ceiling so that it can never approach the ceiling.
|
||||||
LENGTH_FLOOR_SHARE = 0.35
|
LENGTH_FLOOR_SHARE = 0.35
|
||||||
# Below this a floor is meaningless — at a tight cap a short turn is the correct
|
# Below this word count, a floor means nothing, because a short turn is the
|
||||||
# turn — and the tight-cap wording is the one measured to keep the state block
|
# correct turn at a tight cap. The wording used at a tight cap is also the
|
||||||
# alive, so it is left exactly as it was.
|
# wording that was measured to preserve the state block, so it is unchanged.
|
||||||
MIN_LENGTH_FLOOR_WORDS = 60
|
MIN_LENGTH_FLOOR_WORDS = 60
|
||||||
# The floor exists to stop a collapse to two paragraphs, not to demand an essay:
|
# The floor prevents a collapse to two paragraphs. It does not ask for an essay.
|
||||||
# at a 2400-token cap the share alone would ask for 555 words *minimum*. Past this
|
# At a 2400-token cap, the share alone would request a minimum of 555 words. A
|
||||||
# point a reader wanting more length can say so in the author's note.
|
# reader who wants longer turns can ask for them in the author's note.
|
||||||
MAX_LENGTH_FLOOR_WORDS = 300
|
MAX_LENGTH_FLOOR_WORDS = 300
|
||||||
|
|
||||||
|
|
||||||
@@ -89,9 +89,9 @@ class Section:
|
|||||||
def length_hint(max_output_tokens: int, *, has_ws: bool) -> str:
|
def length_hint(max_output_tokens: int, *, has_ws: bool) -> str:
|
||||||
"""Ask for a turn that fits inside the output cap, stated as a word budget.
|
"""Ask for a turn that fits inside the output cap, stated as a word budget.
|
||||||
|
|
||||||
Returns "" when the cap is too small to phrase usefully — the hint is a
|
Returns an empty string when the cap is too small to state usefully. The
|
||||||
suggestion the model can drift past, so it only earns its tokens when there
|
model can exceed the hint, so the hint earns its tokens only when there is
|
||||||
is enough room for the drift to still land inside the wall.
|
enough room for that overshoot to stay inside the cap.
|
||||||
"""
|
"""
|
||||||
words = int((max_output_tokens - LENGTH_HEADROOM) * WORDS_PER_TOKEN * LENGTH_BUFFER)
|
words = int((max_output_tokens - LENGTH_HEADROOM) * WORDS_PER_TOKEN * LENGTH_BUFFER)
|
||||||
if words < MIN_LENGTH_HINT_WORDS:
|
if words < MIN_LENGTH_HINT_WORDS:
|
||||||
@@ -102,25 +102,27 @@ def length_hint(max_output_tokens: int, *, has_ws: bool) -> str:
|
|||||||
else " Bring the turn to a close well inside the limit rather than "
|
else " Bring the turn to a close well inside the limit rather than "
|
||||||
"stopping mid-sentence."
|
"stopping mid-sentence."
|
||||||
)
|
)
|
||||||
# Phrased as a ceiling, never as a budget. Measured against this model, "keep
|
# State the number as a ceiling, never as a budget. In measurements, the
|
||||||
# this turn under about N words" reads as a target to fill: it moved a 174-word
|
# wording "keep this turn under about N words" read to the model as a target
|
||||||
# average to 246 (n=5, every run longer than every unhinted one), i.e. the hint
|
# to fill. It raised the average from 174 words to 246 across five runs, and
|
||||||
# pushed turns toward the very wall it exists to keep them away from. Naming
|
# every hinted run was longer than every unhinted run. The hint therefore
|
||||||
# the number as a limit, plus saying a typical turn is far shorter, left the
|
# pushed turns toward the limit it exists to avoid. Naming the number as a
|
||||||
# average at 170 while still rescuing the state block at tight caps.
|
# limit, and adding that a typical turn is much shorter, held the average at
|
||||||
|
# 170 while still preserving the state block at tight caps.
|
||||||
floor = min(int(words * LENGTH_FLOOR_SHARE), MAX_LENGTH_FLOOR_WORDS)
|
floor = min(int(words * LENGTH_FLOOR_SHARE), MAX_LENGTH_FLOOR_WORDS)
|
||||||
if floor < MIN_LENGTH_FLOOR_WORDS:
|
if floor < MIN_LENGTH_FLOOR_WORDS:
|
||||||
return (
|
return (
|
||||||
f"[Hard limit: this turn must not exceed {words} words. Write only as "
|
f"[Hard limit: this turn must not exceed {words} words. Write only as "
|
||||||
f"much as the moment needs — a typical turn is much shorter.{tail}]"
|
f"much as the moment needs — a typical turn is much shorter.{tail}]"
|
||||||
)
|
)
|
||||||
# Both numbers are bounds, and deliberately asymmetric ones: "must not exceed"
|
# Both numbers are bounds, and the wording is deliberately asymmetric. The
|
||||||
# for the wall the endpoint enforces, "should not stop short of" for the floor.
|
# ceiling uses "must not exceed", because the endpoint enforces it. The floor
|
||||||
# Neither is a target, which is what the measurement above says matters. The
|
# uses "should not stop short of". Neither reads as a target, which the
|
||||||
# "prefer the lower end" clause does the job the old "a typical turn is much
|
# measurement above shows is what matters. The clause that asks the model to
|
||||||
# shorter" line did — holding a verbose model off the wall — but now with a
|
# prefer the lower end does the job the earlier wording did, which was to
|
||||||
# number under it, so a terse model reading the same clause lands on the floor
|
# keep a verbose model away from the ceiling. It now has a number beneath it,
|
||||||
# instead of at forty words.
|
# so a terse model reading the same clause stops at the floor rather than at
|
||||||
|
# forty words.
|
||||||
return (
|
return (
|
||||||
f"[Hard limit: this turn must not exceed {words} words, and it should not "
|
f"[Hard limit: this turn must not exceed {words} words, and it should not "
|
||||||
f"stop short of about {floor}. Prefer the lower end of that range unless "
|
f"stop short of about {floor}. Prefer the lower end of that range unless "
|
||||||
@@ -136,16 +138,19 @@ def _script_memory(adventure: models.Adventure) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
def _history_text(action: models.Action) -> str:
|
def _history_text(action: models.Action) -> str:
|
||||||
"""An AI turn as the model should see it in replayed history: its narration
|
"""Returns an AI turn as the model should see it in replayed history.
|
||||||
with the state block it emitted re-appended (reconstructed from the stored
|
|
||||||
delta). The block is stripped before storage/UI, so without this every past
|
|
||||||
AI turn would look like one that emitted nothing — biasing the model, by
|
|
||||||
imitation, to stop emitting too. Player turns and blockless turns are
|
|
||||||
returned unchanged.
|
|
||||||
|
|
||||||
Reads `world_delta`, not `context_snapshot`: this runs for every action in
|
The result is the narration with its state block appended again,
|
||||||
the replayed history, and the snapshot is deferred precisely so a turn
|
reconstructed from the stored delta. The app strips that block before
|
||||||
never drags the prompt archive out of the database."""
|
storing and displaying the turn. Without this function, every past AI turn
|
||||||
|
would appear to have emitted no state, and the model would copy that pattern
|
||||||
|
and stop emitting state itself. Player turns and turns with no block pass
|
||||||
|
through unchanged.
|
||||||
|
|
||||||
|
This function reads `world_delta` rather than `context_snapshot`. It runs
|
||||||
|
for every action in the replayed history, and `context_snapshot` is deferred
|
||||||
|
so that a turn never loads the prompt archive from the database.
|
||||||
|
"""
|
||||||
text = action.text
|
text = action.text
|
||||||
wd = action.world_delta if isinstance(action.world_delta, dict) else None
|
wd = action.world_delta if isinstance(action.world_delta, dict) else None
|
||||||
if wd:
|
if wd:
|
||||||
@@ -156,10 +161,13 @@ def _history_text(action: models.Action) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def _visible_npcs(actions: list[models.Action], stat_schema: dict) -> dict[str, str]:
|
def _visible_npcs(actions: list[models.Action], stat_schema: dict) -> dict[str, str]:
|
||||||
"""Defined NPCs whose trigger words appear in the recent story — the ones
|
"""Returns the NPCs whose trigger words appear in the recent story.
|
||||||
"in scene", so only their stats get injected. Maps npc id -> display name.
|
|
||||||
|
|
||||||
`actions` is already the last handful (see NPC_WINDOW)."""
|
These are the NPCs in scene, and the prompt includes stats for them only.
|
||||||
|
The result maps an NPC id to its display name.
|
||||||
|
|
||||||
|
`actions` holds only the most recent actions. See `NPC_WINDOW`.
|
||||||
|
"""
|
||||||
recent = SEPARATOR.join(a.text for a in actions).lower()
|
recent = SEPARATOR.join(a.text for a in actions).lower()
|
||||||
visible: dict[str, str] = {}
|
visible: dict[str, str] = {}
|
||||||
for npc_key, ndef in (stat_schema.get("npcs") or {}).items():
|
for npc_key, ndef in (stat_schema.get("npcs") or {}).items():
|
||||||
@@ -171,8 +179,11 @@ def _visible_npcs(actions: list[models.Action], stat_schema: dict) -> dict[str,
|
|||||||
|
|
||||||
|
|
||||||
def _match_cards(cards: list[models.StoryCard], window_text: str) -> list[dict]:
|
def _match_cards(cards: list[models.StoryCard], window_text: str) -> list[dict]:
|
||||||
"""AI Dungeon trigger rules: case-insensitive, space-sensitive, partial-word
|
"""Returns one record per matched story card, naming the keyword that matched.
|
||||||
('boat' triggers on 'boats'). Returns one record per card with the keyword that fired."""
|
|
||||||
|
Matching follows AI Dungeon's rules. It ignores case, respects spaces, and
|
||||||
|
matches partial words, so "boat" matches "boats".
|
||||||
|
"""
|
||||||
haystack = window_text.lower()
|
haystack = window_text.lower()
|
||||||
matched = []
|
matched = []
|
||||||
for card in cards:
|
for card in cards:
|
||||||
@@ -196,20 +207,19 @@ def build_context(
|
|||||||
`exclude_action_id` omits one action from the story (see history.py)."""
|
`exclude_action_id` omits one action from the story (see history.py)."""
|
||||||
script_mem = _script_memory(adventure)
|
script_mem = _script_memory(adventure)
|
||||||
|
|
||||||
# ----- The static block: byte-for-byte the same prompt every turn -----
|
# ----- The static block, which is identical on every turn -----
|
||||||
# The ordering here is a billing decision, not a stylistic one. Prompt
|
# This ordering exists to reduce cost. Prompt caching matches a prefix. The
|
||||||
# caching matches a *prefix*: an endpoint reuses the prompt up to the first
|
# endpoint reuses the prompt up to the first byte that differs from the
|
||||||
# byte that differs from last time and no further. So one mutable section
|
# previous request, and no further. A section that changes near the top
|
||||||
# near the top re-prices everything below it, and what is below it is the
|
# therefore re-prices everything below it, and what sits below it is the
|
||||||
# story history, which is the bulk of the prompt. Anything that changes
|
# story history, which is most of the prompt. Sections that change from turn
|
||||||
# turn to turn therefore goes *after* the history, in the live sections —
|
# to turn go after the history, among the live sections. Placing them there
|
||||||
# which is also where recency serves it best, the same reasoning that
|
# also gives them the most recency, which is why `EMIT_REMINDER` goes last.
|
||||||
# already puts EMIT_REMINDER last.
|
|
||||||
system_sections: list[Section] = [Section("narrator", settings.narrator_prompt.strip())]
|
system_sections: list[Section] = [Section("narrator", settings.narrator_prompt.strip())]
|
||||||
|
|
||||||
# RPG world state (Phase 12): how to report changes. The live values are a
|
# RPG world state (Phase 12): the instructions for reporting changes. The
|
||||||
# live section below; the schema-derived guide and the emit rule are fixed
|
# live values go into a live section below. The guide derived from the
|
||||||
# for as long as the scenario is.
|
# schema and the emit rule do not change while the scenario is unchanged.
|
||||||
stat_schema = adventure.scenario.stat_schema if adventure.scenario else None
|
stat_schema = adventure.scenario.stat_schema if adventure.scenario else None
|
||||||
has_ws = worldstate.has_schema(stat_schema)
|
has_ws = worldstate.has_schema(stat_schema)
|
||||||
if has_ws:
|
if has_ws:
|
||||||
@@ -227,13 +237,14 @@ def build_context(
|
|||||||
Section("plot_essentials", f"Plot essentials:\n{adventure.memory.strip()}")
|
Section("plot_essentials", f"Plot essentials:\n{adventure.memory.strip()}")
|
||||||
)
|
)
|
||||||
|
|
||||||
# ----- Live sections: everything that moves, built here, placed after the
|
# ----- Live sections, which hold everything that changes -----
|
||||||
# history further down. Ordered least-volatile first, so a turn that
|
# This code builds them here and places them after the history further down.
|
||||||
# changes only the fastest-moving one keeps the others cached too: the
|
# They are ordered from least to most volatile, so a turn that changes only
|
||||||
# summary is rewritten every few turns, lore turns over with the scene, the
|
# the fastest-moving section leaves the others cached. The summary is
|
||||||
# retrieved memories change on most turns, the stat values on nearly all.
|
# rewritten every few turns. Lore changes with the scene. The retrieved
|
||||||
# `world_lore` joins them below — it is the history window that triggers
|
# memories change on most turns, and the stat values change on nearly every
|
||||||
# the cards, so it cannot be known yet.
|
# turn. `world_lore` is added below, because the history window determines
|
||||||
|
# which cards trigger and that window is not known yet.
|
||||||
summary_section = (
|
summary_section = (
|
||||||
Section("story_summary", f"Story summary:\n{adventure.story_summary.strip()}")
|
Section("story_summary", f"Story summary:\n{adventure.story_summary.strip()}")
|
||||||
if adventure.story_summary.strip()
|
if adventure.story_summary.strip()
|
||||||
@@ -265,9 +276,9 @@ def build_context(
|
|||||||
|
|
||||||
length_note = length_hint(settings.max_output_tokens, has_ws=has_ws)
|
length_note = length_hint(settings.max_output_tokens, has_ws=has_ws)
|
||||||
|
|
||||||
# The live sections moved below the history but they are still in the
|
# The live sections sit below the history, but they are still part of the
|
||||||
# prompt, so they are still reserved against the budget. (`world_lore` is
|
# prompt, so they still count against the budget. `world_lore` is the
|
||||||
# not: it is budgeted out of `available` further down, as it always was.)
|
# exception, because the code below budgets it out of `available`.
|
||||||
reserved = (
|
reserved = (
|
||||||
sum(s.tokens for s in system_sections)
|
sum(s.tokens for s in system_sections)
|
||||||
+ sum(
|
+ sum(
|
||||||
@@ -282,10 +293,11 @@ def build_context(
|
|||||||
)
|
)
|
||||||
available = max(256, settings.context_token_budget - reserved)
|
available = max(256, settings.context_token_budget - reserved)
|
||||||
|
|
||||||
# Only the newest actions can reach the prompt: everything below is either
|
# Only the newest actions can reach the prompt, because the code below
|
||||||
# truncated to `available` tokens or stops at the budget. Fetch a window
|
# either truncates the text to `available` tokens or stops at the budget.
|
||||||
# that is provably larger than that and no more — a long adventure would
|
# Fetch a window that is provably larger than that and no larger. Otherwise
|
||||||
# otherwise read its entire history every turn to use the tail of it.
|
# a long adventure reads its whole history on every turn and uses only the
|
||||||
|
# end of it.
|
||||||
actions = history.window_covering(
|
actions = history.window_covering(
|
||||||
adventure, available, count_tokens, exclude_action_id
|
adventure, available, count_tokens, exclude_action_id
|
||||||
)
|
)
|
||||||
@@ -319,8 +331,8 @@ def build_context(
|
|||||||
spent = 0
|
spent = 0
|
||||||
oldest_truncated = False
|
oldest_truncated = False
|
||||||
for action in reversed(actions):
|
for action in reversed(actions):
|
||||||
# Budget on the text as it will actually appear — with the re-attached
|
# Budget against the text as it appears in the prompt, which includes
|
||||||
# state block (B) when this adventure tracks world state.
|
# the state block when this adventure tracks world state.
|
||||||
rendered = _history_text(action) if has_ws else action.text
|
rendered = _history_text(action) if has_ws else action.text
|
||||||
tokens = count_tokens(rendered) + count_tokens(SEPARATOR)
|
tokens = count_tokens(rendered) + count_tokens(SEPARATOR)
|
||||||
if spent + tokens > history_budget:
|
if spent + tokens > history_budget:
|
||||||
@@ -339,9 +351,9 @@ def build_context(
|
|||||||
spent += tokens
|
spent += tokens
|
||||||
included_actions.reverse()
|
included_actions.reverse()
|
||||||
|
|
||||||
# ----- Assemble story text with author's note near the end -----
|
# ----- Assemble the story text, with the author's note near the end -----
|
||||||
# Re-attach each AI turn's state block (stripped before storage) so recent
|
# Append each AI turn's state block again. The app strips it before storage,
|
||||||
# history shows the model its own emit pattern to imitate.
|
# and the recent history has to show the model the pattern to follow.
|
||||||
texts = [_history_text(a) if has_ws else a.text for a in included_actions]
|
texts = [_history_text(a) if has_ws else a.text for a in included_actions]
|
||||||
note_sections: list[Section] = []
|
note_sections: list[Section] = []
|
||||||
if authors_note:
|
if authors_note:
|
||||||
@@ -353,21 +365,22 @@ def build_context(
|
|||||||
note_sections.append(Section("recent_history", SEPARATOR.join(after)))
|
note_sections.append(Section("recent_history", SEPARATOR.join(after)))
|
||||||
else:
|
else:
|
||||||
note_sections.append(Section("history", SEPARATOR.join(texts)))
|
note_sections.append(Section("history", SEPARATOR.join(texts)))
|
||||||
# The live sections, least volatile first (see where they are built). They
|
# The live sections, ordered from least to most volatile. See the comment
|
||||||
# sit below the history so the history caches, and above the tail so the
|
# where they are built. They go below the history so that the history stays
|
||||||
# three sections that are last for a reason stay last.
|
# cached, and above the final sections so that those stay last.
|
||||||
for live in (summary_section, lore_section, memories_section, world_state_section):
|
for live in (summary_section, lore_section, memories_section, world_state_section):
|
||||||
if live is not None:
|
if live is not None:
|
||||||
note_sections.append(live)
|
note_sections.append(live)
|
||||||
if front_memory:
|
if front_memory:
|
||||||
note_sections.append(Section("front_memory", front_memory))
|
note_sections.append(Section("front_memory", front_memory))
|
||||||
# Sits just above the emit reminder, which keeps the strongest recency slot:
|
# Place the length hint just above the emit reminder, which keeps the last
|
||||||
# the length budget is about the narration, the reminder about the block that
|
# position. The length budget applies to the narration, and the reminder
|
||||||
# comes after it, so this is also the order the model has to act in.
|
# applies to the block that follows it, so this is also the order in which
|
||||||
|
# the model acts.
|
||||||
note_sections.append(Section("length_hint", length_note))
|
note_sections.append(Section("length_hint", length_note))
|
||||||
if has_ws:
|
if has_ws:
|
||||||
# Terminal reminder: the emit rule sits up in the system block, far from
|
# The emit rule sits in the system block, far from where the model
|
||||||
# where the model generates; repeat it last, in the strongest recency slot.
|
# generates text, so repeat it last where it has the most effect.
|
||||||
note_sections.append(Section("world_state_reminder", worldstate.EMIT_REMINDER))
|
note_sections.append(Section("world_state_reminder", worldstate.EMIT_REMINDER))
|
||||||
|
|
||||||
story_sections = [s for s in note_sections if s.text]
|
story_sections = [s for s in note_sections if s.text]
|
||||||
@@ -388,8 +401,9 @@ def build_context(
|
|||||||
"memories": memory_bank,
|
"memories": memory_bank,
|
||||||
"history": {
|
"history": {
|
||||||
"included": len(included_actions),
|
"included": len(included_actions),
|
||||||
# The whole story, not just the window fetched above — Insights
|
# The count covers the whole story rather than the window fetched
|
||||||
# reports "N of M actions included" and M is the real total.
|
# above. Insights reports how many of the total actions it
|
||||||
|
# included, so this number must be the real total.
|
||||||
"total": history.count(adventure, exclude_action_id),
|
"total": history.count(adventure, exclude_action_id),
|
||||||
"oldest_truncated": oldest_truncated,
|
"oldest_truncated": oldest_truncated,
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -1,31 +1,32 @@
|
|||||||
"""Phase 14 — how far along a story the derived work has got.
|
"""Phase 14: tracks how far along a story the derived work has reached.
|
||||||
|
|
||||||
Two things are built from the story and stored beside it: the memories, and the
|
Two things are built from the story and stored beside it: the memories and the
|
||||||
Story Summary. Both need to know where they left off, and that mark used to be
|
story summary. Both need to record where they stopped.
|
||||||
a *count* — "the first 12 story actions are covered". A count is a position in
|
|
||||||
a list, and this list moves: delete an action from in front of the mark and
|
|
||||||
every later action slides down a slot, so the mark now covers one it has never
|
|
||||||
seen. Every rule in `memorybank` about sliding cursors, rewinding them and
|
|
||||||
translating between positions and `Action.index` existed to patch that up, and
|
|
||||||
each was a separate chance to get it wrong in a way nothing reports.
|
|
||||||
|
|
||||||
A cursor here is an **anchor**: `(branch_id, depth)`, the node up to and
|
That mark used to be a count, such as "the first 12 story actions are covered".
|
||||||
including which the work is done. Deleting an action does not move it, because
|
A count is a position in a list, and this list changes. If you delete an action
|
||||||
a depth is not a position — it is a coordinate along a path. "What is not
|
in front of the mark, every later action moves down one slot, so the mark now
|
||||||
covered yet" becomes `history.count_after(anchor)`, which is a question about
|
covers an action it never read. The rules in `memorybank` for sliding cursors,
|
||||||
the story rather than about a list index, and it answers correctly whatever has
|
rewinding them, and converting between positions and `Action.index` all existed
|
||||||
been deleted from in front of it.
|
to correct for that, and each rule was a chance to introduce a silent error.
|
||||||
|
|
||||||
The branch half is what makes it survive forking. A depth alone is ambiguous
|
A cursor here is an anchor instead. It stores `(branch_id, depth)`, naming the
|
||||||
once two branches have a node 41; the anchor says which one, and
|
node up to and including which the work is done. Deleting an action does not
|
||||||
`Path.depth_on` reads it back as a depth on whatever story is being played —
|
move it, because a depth is a coordinate along a path rather than a position in
|
||||||
capped at the fork, or "nothing covered" if the anchor sits on ground this path
|
a list. The question "what is not covered yet" becomes
|
||||||
never travelled. Until forking ships there is one branch and that is always a
|
`history.count_after(anchor)`, which asks about the story rather than about a
|
||||||
no-op, which is the point: the coordinate system is right before anything needs
|
list index, and it stays correct no matter what is deleted in front of it.
|
||||||
it to be.
|
|
||||||
|
|
||||||
`NO_DEPTH` (-1) is "nothing covered", so a fresh adventure needs no special
|
The branch half of the anchor is what makes it survive forking. A depth alone is
|
||||||
case: every node is deeper than -1.
|
ambiguous once two branches both hold a node at depth 41. The anchor names the
|
||||||
|
branch, and `Path.depth_on` reads it back as a depth on whichever story is being
|
||||||
|
played. That read caps the depth at the fork, or reports nothing covered if the
|
||||||
|
anchor sits on a branch this path does not contain. Until forking ships there is
|
||||||
|
one branch, so this always returns the stored depth. That is the point: the
|
||||||
|
coordinate system is correct before anything depends on it.
|
||||||
|
|
||||||
|
`NO_DEPTH`, which is -1, means nothing is covered. A new adventure therefore
|
||||||
|
needs no special case, because every node is deeper than -1.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
@@ -37,12 +38,11 @@ NO_DEPTH = lineage.NO_DEPTH
|
|||||||
|
|
||||||
|
|
||||||
class Cursor:
|
class Cursor:
|
||||||
"""One anchor on the adventure row: the memory bank's, or the summary's.
|
"""One anchor on the adventure row, for either the memory bank or the summary.
|
||||||
|
|
||||||
A pair of columns rather than a foreign key to the node. The node can be
|
The anchor is a pair of columns rather than a foreign key to the node. The
|
||||||
deleted — that is most of what undo does — and the boundary is still
|
node can be deleted, which is what undo does, and the boundary still means
|
||||||
meaningful afterwards, so a pointer that has to resolve would be a pointer
|
something afterwards. A foreign key would repeatedly fail to resolve.
|
||||||
that keeps not resolving.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, name: str):
|
def __init__(self, name: str):
|
||||||
@@ -53,14 +53,14 @@ class Cursor:
|
|||||||
# ------------------------------------------------------------- reading
|
# ------------------------------------------------------------- reading
|
||||||
|
|
||||||
def stored(self, adventure: models.Adventure) -> tuple[int | None, int]:
|
def stored(self, adventure: models.Adventure) -> tuple[int | None, int]:
|
||||||
"""The anchor exactly as written, unread by any path."""
|
"""Returns the anchor as written, without resolving it against a path."""
|
||||||
depth = getattr(adventure, self.depth_field)
|
depth = getattr(adventure, self.depth_field)
|
||||||
return getattr(adventure, self.branch_field), (
|
return getattr(adventure, self.branch_field), (
|
||||||
NO_DEPTH if depth is None else depth
|
NO_DEPTH if depth is None else depth
|
||||||
)
|
)
|
||||||
|
|
||||||
def depth(self, db: Session, adventure: models.Adventure) -> int:
|
def depth(self, db: Session, adventure: models.Adventure) -> int:
|
||||||
"""The anchor as a depth on the story currently being played."""
|
"""Returns the anchor as a depth on the story being played now."""
|
||||||
branch_id, depth = self.stored(adventure)
|
branch_id, depth = self.stored(adventure)
|
||||||
return lineage.path_of(db, adventure).depth_on(branch_id, depth)
|
return lineage.path_of(db, adventure).depth_on(branch_id, depth)
|
||||||
|
|
||||||
@@ -69,44 +69,47 @@ class Cursor:
|
|||||||
def anchor(
|
def anchor(
|
||||||
self, adventure: models.Adventure, branch_id: int | None, depth: int
|
self, adventure: models.Adventure, branch_id: int | None, depth: int
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Put the anchor at a coordinate given outright.
|
"""Sets the anchor to a coordinate supplied directly.
|
||||||
|
|
||||||
The plain setter under `anchor_at`. Only an import has a coordinate
|
This is the plain setter beneath `anchor_at`. Only an import supplies a
|
||||||
without a node to read it off — a v2 bundle carries the anchor itself
|
coordinate with no node to read it from. A v2 bundle carries the anchor
|
||||||
(`app/bundle.py`), and the node it named lives in another database.
|
itself, as described in `app/bundle.py`, and the node it named lives in
|
||||||
|
a different database.
|
||||||
"""
|
"""
|
||||||
setattr(adventure, self.branch_field, branch_id)
|
setattr(adventure, self.branch_field, branch_id)
|
||||||
setattr(adventure, self.depth_field, max(depth, NO_DEPTH))
|
setattr(adventure, self.depth_field, max(depth, NO_DEPTH))
|
||||||
|
|
||||||
def anchor_at(self, adventure: models.Adventure, node: models.Action) -> None:
|
def anchor_at(self, adventure: models.Adventure, node: models.Action) -> None:
|
||||||
"""Mark the work done up to and including `node`.
|
"""Marks the work done up to and including `node`.
|
||||||
|
|
||||||
Takes the node's own branch, not the adventure's head: a block of six
|
This uses the node's own branch rather than the adventure's head. A block
|
||||||
actions can end before the fork this branch was made at, and the
|
of six actions can end before the fork that created the current branch,
|
||||||
coverage belongs where the ground is.
|
and the coverage belongs where those actions are.
|
||||||
"""
|
"""
|
||||||
self.anchor(adventure, node.branch_id, lineage.NO_DEPTH
|
self.anchor(adventure, node.branch_id, lineage.NO_DEPTH
|
||||||
if node.depth is None else node.depth)
|
if node.depth is None else node.depth)
|
||||||
|
|
||||||
def clear(self, adventure: models.Adventure) -> None:
|
def clear(self, adventure: models.Adventure) -> None:
|
||||||
"""Forget the anchor entirely: nothing is covered.
|
"""Clears the anchor, so that nothing counts as covered.
|
||||||
|
|
||||||
For when the ground the anchor stood on is gone — a deleted branch. On
|
Call this when the branch the anchor referred to is deleted. On Postgres
|
||||||
Postgres a stale branch id would simply never resolve, but SQLite hands
|
a stale branch id would never resolve. On SQLite the next fork can reuse
|
||||||
a freed id to the next fork, and an anchor that resolves onto a branch
|
an id that was just freed, and a stale anchor would then resolve onto a
|
||||||
it has never seen would report a stretch of story as already
|
branch it never saw and report that stretch of story as summarized.
|
||||||
summarized. Clearing costs a re-summarize, which is the safe direction.
|
Clearing the anchor costs one re-summarize, which is the safe direction
|
||||||
|
to be wrong in.
|
||||||
"""
|
"""
|
||||||
self.anchor(adventure, None, NO_DEPTH)
|
self.anchor(adventure, None, NO_DEPTH)
|
||||||
|
|
||||||
def rewind_to(
|
def rewind_to(
|
||||||
self, adventure: models.Adventure, branch_id: int | None, depth: int
|
self, adventure: models.Adventure, branch_id: int | None, depth: int
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Move the anchor back to `depth` if it is past it; never forward.
|
"""Moves the anchor back to `depth` if it is past that depth.
|
||||||
|
|
||||||
The one direction that is safe without knowing what else has happened:
|
The anchor never moves forward here. Moving backward is the only
|
||||||
re-covering ground costs a summarizer call, skipping it loses a stretch
|
direction that is safe without knowing what else changed. Covering
|
||||||
of story out of the memories for good.
|
ground twice costs one summarizer call. Skipping ground removes a
|
||||||
|
stretch of story from the memories permanently.
|
||||||
"""
|
"""
|
||||||
_, current = self.stored(adventure)
|
_, current = self.stored(adventure)
|
||||||
if current <= depth:
|
if current <= depth:
|
||||||
@@ -123,12 +126,12 @@ ALL = (MEMORY, SUMMARY)
|
|||||||
def rewind_all(
|
def rewind_all(
|
||||||
adventure: models.Adventure, branch_id: int | None, depth: int
|
adventure: models.Adventure, branch_id: int | None, depth: int
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Hand a stretch of story back to *both* passes.
|
"""Returns a stretch of story to both the memory pass and the summary pass.
|
||||||
|
|
||||||
They move together because they cover the same ground from different sides:
|
The two move together because they cover the same actions from different
|
||||||
the summary folds in the memories, so a memory withdrawn without rewinding
|
directions. The summary folds in the memories, so withdrawing a memory
|
||||||
the summary leaves the summary claiming to have read something no longer
|
without rewinding the summary would leave the summary claiming to have read
|
||||||
there.
|
something that no longer exists.
|
||||||
"""
|
"""
|
||||||
for cursor in ALL:
|
for cursor in ALL:
|
||||||
cursor.rewind_to(adventure, branch_id, depth)
|
cursor.rewind_to(adventure, branch_id, depth)
|
||||||
@@ -137,18 +140,19 @@ def rewind_all(
|
|||||||
def anchor_at_position(
|
def anchor_at_position(
|
||||||
adventure: models.Adventure, cursor: Cursor, position: int
|
adventure: models.Adventure, cursor: Cursor, position: int
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Set `cursor` from a count of covered story actions — a v1 bundle's mark,
|
"""Sets `cursor` from a count of covered story actions.
|
||||||
or a database written before the anchors existed.
|
|
||||||
|
|
||||||
The position-th story action in depth order is the node that says the same
|
A v1 bundle stores its mark as a count, and so does a database written
|
||||||
thing, and goes on saying it once something in front of it is deleted. A
|
before the anchors existed.
|
||||||
position past the end of the story is not a bad value: an adventure caught
|
|
||||||
up under the older rule can carry one, and it means the same thing the tip
|
|
||||||
does, so that is where it lands.
|
|
||||||
|
|
||||||
The SQL half of this rule is `migrations._backfill_cursor_anchors`, which
|
The action at `position` in depth order is the node that carries the same
|
||||||
has to do it for every adventure at once without loading any of them; the
|
meaning, and it keeps that meaning after something in front of it is
|
||||||
two must agree.
|
deleted. A position past the end of the story is not an invalid value. An
|
||||||
|
adventure that was fully caught up under the old rule can hold one, and it
|
||||||
|
means the same thing as the tip, so this function anchors at the tip.
|
||||||
|
|
||||||
|
`migrations._backfill_cursor_anchors` implements this rule in SQL for every
|
||||||
|
adventure at once, without loading any of them. The two must agree.
|
||||||
"""
|
"""
|
||||||
if position <= 0:
|
if position <= 0:
|
||||||
return
|
return
|
||||||
@@ -158,11 +162,11 @@ def anchor_at_position(
|
|||||||
|
|
||||||
|
|
||||||
def position_of(adventure: models.Adventure, depth: int) -> int:
|
def position_of(adventure: models.Adventure, depth: int) -> int:
|
||||||
"""How many story actions lie at or before `depth` — an anchor read back as
|
"""Returns how many story actions lie at or before `depth`.
|
||||||
a count.
|
|
||||||
|
|
||||||
The v1 export bundle stores the cursors as positions, and a v1 bundle is
|
This reads an anchor back as a count. The v1 export bundle stores cursors as
|
||||||
read by builds that have never heard of a depth. This is the one place that
|
counts, and builds that have never used depths read v1 bundles. This
|
||||||
still speaks that coordinate system, and SP6's v2 format retires it.
|
function is the only remaining code that speaks that coordinate system. The
|
||||||
|
v2 format introduced in SP6 replaces it.
|
||||||
"""
|
"""
|
||||||
return max(history.count(adventure) - history.count_after(adventure, depth), 0)
|
return max(history.count(adventure) - history.count_after(adventure, depth), 0)
|
||||||
|
|||||||
+140
-124
@@ -1,42 +1,44 @@
|
|||||||
"""Reading the story without reading all of it.
|
"""Reads part of a story without loading all of it.
|
||||||
|
|
||||||
`story_actions()` walked `adventure.actions`, which loads every row of the
|
`story_actions()` used to walk `adventure.actions`, which loads every row of the
|
||||||
adventure — then every caller threw almost all of it away. The context builder
|
adventure. Every caller then discarded nearly all of those rows. The context
|
||||||
concatenates the story and immediately cuts it back to the token budget; the
|
builder joins the story and immediately trims it to the token budget. The
|
||||||
NPC-in-scene check looks at the last 6; memory retrieval looks at the last 4;
|
in-scene NPC check reads the last 6 actions. Memory retrieval reads the last 4.
|
||||||
the post-turn cursor clamp only wants a count. So a turn on a 200-action
|
The post-turn cursor clamp needs only a count. A turn on a 200-action adventure
|
||||||
adventure read ~840 KB to use maybe 70 KB of it, and the cost grew with every
|
read about 840 KB in order to use about 70 KB, and the cost grew with every
|
||||||
turn played.
|
turn.
|
||||||
|
|
||||||
This module serves those shapes directly from SQL — a tail, a slice, a count —
|
This module serves those shapes from SQL directly, as a tail, a slice, or a
|
||||||
so the read is bounded by the context budget instead of by the length of the
|
count. A read is therefore bounded by the context budget rather than by the
|
||||||
story.
|
length of the story.
|
||||||
|
|
||||||
Three rules hold everything together:
|
Three rules hold the module together:
|
||||||
|
|
||||||
* **One definition of "story action".** Membership decides what a reader sees
|
- There is one definition of a story action. That definition decides both what
|
||||||
and what the summarizer is handed, so SQL and Python must agree on it
|
a reader sees and what the summarizer receives, so the SQL and the Python must
|
||||||
exactly. `_STORY_TEXT` and `is_story_text()` are that one definition, written
|
agree exactly. `_STORY_TEXT` and `is_story_text()` express the same rule
|
||||||
twice; keep them in step.
|
twice. Keep them in step.
|
||||||
* **Never load twice.** If `adventure.actions` is already in memory (the
|
- No caller loads the same rows twice. If `adventure.actions` is already in
|
||||||
scripting pipeline hands the whole history to user scripts, as AI Dungeon
|
memory, every helper here slices that collection instead of running a query.
|
||||||
does), every helper here slices that instead of issuing a query, so a
|
The scripting pipeline hands the whole history to user scripts, as AI Dungeon
|
||||||
scripted adventure pays what it always paid and nothing more.
|
does, so a scripted adventure costs no more than it did before.
|
||||||
* **Every read goes through the branch clause** (Phase 14). `adventure.actions`
|
- Every read applies the branch clause, as of Phase 14. `adventure.actions`
|
||||||
is every branch's actions, not the story being played — so the shortcut above
|
holds the actions of every branch rather than the story being played, so the
|
||||||
cuts the loaded collection down to the path before slicing it, exactly as the
|
in-memory path filters the collection down to the path before slicing it, in
|
||||||
SQL does. This is the line that would silently assemble a prompt out of two
|
the same way the SQL does. Skipping that filter would build a prompt from two
|
||||||
different stories, which is why `lineage.Path` owns both halves of it.
|
different stories without reporting an error, which is why `lineage.Path`
|
||||||
|
owns both forms of the rule.
|
||||||
|
|
||||||
Ordering is by `depth` now, not `index`. Since SP4 the two can hold the same
|
Reads order by `depth` rather than `index`. Since SP4, two different rows can
|
||||||
number on *different rows* — attempts at one turn share both — so only `depth`
|
hold the same value for both columns, because the attempts at one turn share
|
||||||
plus the branch clause's `live` test says which of them the story is.
|
them. Only `depth` together with the `live` test in the branch clause
|
||||||
|
identifies the row the story uses.
|
||||||
|
|
||||||
SP3 added the reads that count *from a node* rather than from the start —
|
SP3 added the reads that count from a node rather than from the start:
|
||||||
`count_after`, `after`, `newest`. The memory bank used to ask for
|
`count_after`, `after`, and `newest`. The memory bank used to ask for positions
|
||||||
"positions 12 to 18 of the story", which is a question whose answer moves when
|
12 through 18 of the story, and the answer to that question changes when an
|
||||||
an action is deleted from in front of it. It now asks for "the six actions
|
action in front of those positions is deleted. It now asks for the six actions
|
||||||
after depth 41", which is the same question a fork has to answer anyway.
|
after depth 41, which is the question that forking requires in any case.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from sqlalchemy import func, inspect as sa_inspect
|
from sqlalchemy import func, inspect as sa_inspect
|
||||||
@@ -45,17 +47,17 @@ from sqlalchemy.orm import Session, defer, object_session
|
|||||||
from .. import models
|
from .. import models
|
||||||
from . import lineage
|
from . import lineage
|
||||||
|
|
||||||
# How many of the newest actions to read before checking whether the token
|
# How many of the newest actions to read before checking whether they cover the
|
||||||
# budget is covered. When it isn't, the next size is worked out from the
|
# token budget. If they do not, the next size comes from the average action
|
||||||
# average action length just measured rather than by blind doubling — guessing
|
# length just measured rather than from doubling the previous size. Doubling
|
||||||
# high means reading hundreds of actions to use sixty of them.
|
# overshoots, which means reading hundreds of actions in order to use sixty.
|
||||||
WINDOW_START = 32
|
WINDOW_START = 32
|
||||||
WINDOW_MARGIN = 0.15 # aim this far past the budget, so one more round is rare
|
WINDOW_MARGIN = 0.15 # Aim this far past the budget, so a second round is rare.
|
||||||
WINDOW_STEP = 8 # ...and at least this many more actions each round
|
WINDOW_STEP = 8 # Read at least this many more actions in each round.
|
||||||
|
|
||||||
# Depth is the ordering key; id breaks the tie that a pre-tree row (depth NULL,
|
# `depth` is the ordering key, and `id` breaks ties. Without `id`, the database
|
||||||
# and so invisible anyway) or a future sibling pair would otherwise leave to
|
# would choose the order. Two rows can share a depth: a pre-tree row, which has
|
||||||
# the database's mood.
|
# a NULL depth and is invisible to reads, or a pair of sibling attempts.
|
||||||
_OLDEST_FIRST = (models.Action.depth, models.Action.id)
|
_OLDEST_FIRST = (models.Action.depth, models.Action.id)
|
||||||
_NEWEST_FIRST = (models.Action.depth.desc(), models.Action.id.desc())
|
_NEWEST_FIRST = (models.Action.depth.desc(), models.Action.id.desc())
|
||||||
|
|
||||||
@@ -63,12 +65,13 @@ _NEWEST_FIRST = (models.Action.depth.desc(), models.Action.id.desc())
|
|||||||
def _sql_stripped(column):
|
def _sql_stripped(column):
|
||||||
"""`column` with leading/trailing whitespace removed, portably.
|
"""`column` with leading/trailing whitespace removed, portably.
|
||||||
|
|
||||||
SQLite and Postgres both accept single-argument `trim()`, but it strips
|
SQLite and Postgres both accept `trim()` with a single argument, but that
|
||||||
spaces only — Python's `.strip()` also drops newlines and tabs, and an
|
form removes spaces only. Python's `str.strip()` also removes newlines and
|
||||||
action of nothing but a newline would otherwise count as story text here
|
tabs. Without this helper, an action containing only a newline would count
|
||||||
and not in Python. `replace()` and `trim()` are the two string functions
|
as story text in SQL but not in Python. Both dialects spell `replace()` and
|
||||||
both dialects spell identically, so fold the other whitespace into spaces
|
`trim()` the same way, so this function converts the other whitespace to
|
||||||
first. (Form feed and vertical tab are not covered; nothing produces them.)
|
spaces first. It does not handle form feed or vertical tab, because nothing
|
||||||
|
produces them.
|
||||||
"""
|
"""
|
||||||
folded = column
|
folded = column
|
||||||
for char in ("\n", "\r", "\t"):
|
for char in ("\n", "\r", "\t"):
|
||||||
@@ -80,15 +83,18 @@ _STORY_TEXT = _sql_stripped(models.Action.text) != ""
|
|||||||
|
|
||||||
|
|
||||||
def is_story_text(text: str) -> bool:
|
def is_story_text(text: str) -> bool:
|
||||||
"""The Python half of `_STORY_TEXT` — keep the two in step."""
|
"""Returns whether `text` counts as story text.
|
||||||
|
|
||||||
|
This is the Python form of `_STORY_TEXT`. Keep the two in step.
|
||||||
|
"""
|
||||||
return bool(text.strip())
|
return bool(text.strip())
|
||||||
|
|
||||||
|
|
||||||
def _loaded_actions(adventure: models.Adventure) -> list[models.Action] | None:
|
def _loaded_actions(adventure: models.Adventure) -> list[models.Action] | None:
|
||||||
"""The adventure's actions if they are already in memory, else None.
|
"""The adventure's actions if they are already in memory, else None.
|
||||||
|
|
||||||
Slicing an already-loaded collection is free; issuing a query beside it
|
Slicing a collection that is already loaded costs nothing, and running a
|
||||||
would mean paying for the same rows twice.
|
query beside it would fetch the same rows a second time.
|
||||||
"""
|
"""
|
||||||
state = sa_inspect(adventure)
|
state = sa_inspect(adventure)
|
||||||
if state.detached or "actions" in state.unloaded:
|
if state.detached or "actions" in state.unloaded:
|
||||||
@@ -101,11 +107,14 @@ def _from_memory(
|
|||||||
) -> list[models.Action] | None:
|
) -> list[models.Action] | None:
|
||||||
"""The story, from the already-loaded collection, or None to go to SQL.
|
"""The story, from the already-loaded collection, or None to go to SQL.
|
||||||
|
|
||||||
The collection is the *adventure's* actions — every branch of it. Cutting
|
The collection holds the adventure's actions, which means the actions of
|
||||||
it down to the path here is the same filter the SQL applies, and skipping
|
every branch. Filtering it down to the path here applies the same rule that
|
||||||
it would hand the context builder a prompt assembled from siblings of the
|
the SQL applies. Without that filter, the context builder would receive a
|
||||||
story being played. The path needs a session to read the branch row from;
|
prompt built from siblings of the story being played.
|
||||||
without one there is no answer to give, so say so rather than guess.
|
|
||||||
|
Resolving the path requires a session to read the branch row from. If there
|
||||||
|
is no session, this function returns None so that the caller falls back to
|
||||||
|
SQL rather than guessing.
|
||||||
"""
|
"""
|
||||||
loaded = _loaded_actions(adventure)
|
loaded = _loaded_actions(adventure)
|
||||||
if loaded is None:
|
if loaded is None:
|
||||||
@@ -131,11 +140,10 @@ def _filters(
|
|||||||
exclude_action_id: int | None,
|
exclude_action_id: int | None,
|
||||||
entries: int | None = None,
|
entries: int | None = None,
|
||||||
) -> list:
|
) -> list:
|
||||||
# adventure_id is redundant beside the branch clause — branch ids are
|
# `adventure_id` is redundant beside the branch clause, because branch ids
|
||||||
# unique, so a branch already names one adventure. It stays because it is
|
# are unique and a branch already identifies one adventure. The filter
|
||||||
# the cheap half of the check that catches a node written onto the wrong
|
# remains because it costs little, it catches a node written onto another
|
||||||
# adventure's branch, and because a clause nobody can read is a clause
|
# adventure's branch, and it makes the query easier to read.
|
||||||
# nobody maintains.
|
|
||||||
conditions = [
|
conditions = [
|
||||||
models.Action.adventure_id == adventure.id,
|
models.Action.adventure_id == adventure.id,
|
||||||
path.clause(models.Action, count=entries),
|
path.clause(models.Action, count=entries),
|
||||||
@@ -171,11 +179,11 @@ def _count_query(
|
|||||||
):
|
):
|
||||||
"""A real `SELECT count(...)`.
|
"""A real `SELECT count(...)`.
|
||||||
|
|
||||||
Deliberately not `_query(...).count()`: that wraps the entity select in a
|
This function deliberately avoids `_query(...).count()`. That form wraps the
|
||||||
subquery, so the emitted SQL names every column — including the deferred
|
entity select in a subquery, so the emitted SQL names every column,
|
||||||
ones this whole design exists to keep off the wire. No bytes come back
|
including the deferred columns that this design keeps off the wire. Neither
|
||||||
either way, but the database still has to read them, and an egress guard
|
form returns those bytes to the client, but the database still reads them,
|
||||||
that greps the SQL cannot tell the two apart.
|
and an egress guard that inspects the SQL cannot tell the two forms apart.
|
||||||
"""
|
"""
|
||||||
return db.query(func.count(models.Action.id)).filter(
|
return db.query(func.count(models.Action.id)).filter(
|
||||||
*_filters(adventure, path, exclude_action_id, entries)
|
*_filters(adventure, path, exclude_action_id, entries)
|
||||||
@@ -197,14 +205,14 @@ def story_actions(
|
|||||||
) -> list[models.Action]:
|
) -> list[models.Action]:
|
||||||
"""Every story action, oldest first.
|
"""Every story action, oldest first.
|
||||||
|
|
||||||
Still the right call where the whole story is genuinely wanted — user
|
Call this function when you need the whole story. User scripts receive it,
|
||||||
scripts receive it, per AI Dungeon's scripting API. Prefer `tail`, `slice_`
|
which matches AI Dungeon's scripting API. Use `tail`, `slice_`, or `count`
|
||||||
or `count` anywhere the caller only needs part of it.
|
when you need only part of the story.
|
||||||
|
|
||||||
`exclude_action_id` drops one action from the story — used by retry, where
|
`exclude_action_id` removes one action from the result. Retry uses it. The
|
||||||
the attempt being replaced is still the live node of its turn (it stays
|
attempt being replaced is still the live node of its turn, because it stays
|
||||||
live until a replacement exists) but must not appear in the context
|
live until a replacement exists, but it must not appear in the context that
|
||||||
assembled to replace it.
|
is assembled to replace it.
|
||||||
"""
|
"""
|
||||||
in_memory = _from_memory(adventure, exclude_action_id)
|
in_memory = _from_memory(adventure, exclude_action_id)
|
||||||
if in_memory is not None:
|
if in_memory is not None:
|
||||||
@@ -241,18 +249,20 @@ def tail_range(
|
|||||||
) -> list[models.Action]:
|
) -> list[models.Action]:
|
||||||
"""`limit` story actions ending `skip` actions before the end, oldest first.
|
"""`limit` story actions ending `skip` actions before the end, oldest first.
|
||||||
|
|
||||||
`skip=0` is the newest slice; `skip=32, limit=16` is the 16 actions just
|
Passing `skip=0` returns the newest slice. Passing `skip=32` and `limit=16`
|
||||||
older than the newest 32. Lets a growing window fetch only the part it
|
returns the 16 actions immediately older than the newest 32. A growing
|
||||||
doesn't already have.
|
window therefore fetches only the actions it does not already hold.
|
||||||
|
|
||||||
This is the read the lineage window exists for. The path's ranges are
|
This read is the reason the lineage window exists. The path's ranges do not
|
||||||
disjoint and descending, so the newest N nodes come from the newest few
|
overlap and they descend, so the newest N nodes come from the newest few
|
||||||
lineage entries and the rest of the ancestry need not be named at all: a
|
lineage entries and the query never has to name the rest of the ancestry. A
|
||||||
story forked two hundred times reads its tail with as few clauses as one
|
story that has forked 200 times reads its tail with as few clauses as one
|
||||||
forked never. `prefix_covering` estimates how many entries that takes from
|
that has never forked.
|
||||||
depth arithmetic alone; the estimate is only ever short where a middle
|
|
||||||
action was deleted, and then the read widens to the whole lineage and pays
|
`prefix_covering` estimates how many entries that takes, using depth
|
||||||
one more query.
|
arithmetic alone. The estimate falls short only when an action was deleted
|
||||||
|
from the middle of the story. In that case this function widens the read to
|
||||||
|
the whole lineage, at the cost of one more query.
|
||||||
"""
|
"""
|
||||||
if limit <= 0 or skip < 0:
|
if limit <= 0 or skip < 0:
|
||||||
return []
|
return []
|
||||||
@@ -295,11 +305,12 @@ def slice_(
|
|||||||
) -> list[models.Action]:
|
) -> list[models.Action]:
|
||||||
"""Story actions at positions [start, start + length), oldest first.
|
"""Story actions at positions [start, start + length), oldest first.
|
||||||
|
|
||||||
Positions are into the same filtered, depth-ordered list the memory cursors
|
Positions index into the same filtered, depth-ordered list that the memory
|
||||||
count in, which is why the filter has to match Python's exactly.
|
cursors count in, which is why the SQL filter must match the Python filter
|
||||||
|
exactly.
|
||||||
|
|
||||||
Counts from the oldest end, so it names the whole lineage: there is no
|
This function counts from the oldest end, so it names the whole lineage. No
|
||||||
prefix of the ancestry that holds "the story's first ten actions".
|
prefix of the ancestry contains the first ten actions of the story.
|
||||||
"""
|
"""
|
||||||
if length <= 0 or start < 0:
|
if length <= 0 or start < 0:
|
||||||
return []
|
return []
|
||||||
@@ -321,9 +332,9 @@ def slice_(
|
|||||||
def depth_of(action: models.Action) -> int:
|
def depth_of(action: models.Action) -> int:
|
||||||
"""`action.depth`, with the no-depth case spelled once.
|
"""`action.depth`, with the no-depth case spelled once.
|
||||||
|
|
||||||
A row with no depth is a pre-tree row, which no path contains — so it can
|
A row with no depth predates the tree. No path contains such a row, so it
|
||||||
only turn up in an already-loaded collection, and it sorts before the story
|
appears only in a collection that is already loaded. It sorts before the
|
||||||
rather than after it.
|
story rather than after it.
|
||||||
"""
|
"""
|
||||||
return action.depth if action.depth is not None else lineage.NO_DEPTH
|
return action.depth if action.depth is not None else lineage.NO_DEPTH
|
||||||
|
|
||||||
@@ -333,14 +344,14 @@ def count_after(
|
|||||||
) -> int:
|
) -> int:
|
||||||
"""How many story actions lie past `depth` on the path.
|
"""How many story actions lie past `depth` on the path.
|
||||||
|
|
||||||
The node-anchored replacement for "the story is N long and the cursor is at
|
This replaces the older calculation, which compared the length of the story
|
||||||
M". Deleting an action from in front of the boundary makes this number
|
with the position of the cursor. Deleting an action in front of the boundary
|
||||||
smaller, which is true; it does not make the boundary point somewhere else,
|
makes this number smaller, which is correct. It does not move the boundary
|
||||||
which is the bug the positions had.
|
to a different action, which is the error that positions produced.
|
||||||
|
|
||||||
`covering_after` says exactly which lineage entries can hold a node deeper
|
`covering_after` reports which lineage entries can hold a node deeper than
|
||||||
than the boundary, so a cursor near the tip names one branch however many
|
the boundary, so a cursor near the tip names one branch however many forks
|
||||||
forks are below it.
|
lie below it.
|
||||||
"""
|
"""
|
||||||
in_memory = _from_memory(adventure, exclude_action_id)
|
in_memory = _from_memory(adventure, exclude_action_id)
|
||||||
if in_memory is not None:
|
if in_memory is not None:
|
||||||
@@ -367,8 +378,9 @@ def after(
|
|||||||
) -> list[models.Action]:
|
) -> list[models.Action]:
|
||||||
"""The oldest `limit` story actions past `depth`, oldest first.
|
"""The oldest `limit` story actions past `depth`, oldest first.
|
||||||
|
|
||||||
"The next block the summarizer has not seen", asked as a fact about the
|
This returns the next block that the summarizer has not read. It asks a
|
||||||
story rather than as an offset into a list that shifts underneath it.
|
question about the story rather than using an offset into a list whose
|
||||||
|
entries move.
|
||||||
"""
|
"""
|
||||||
if limit <= 0:
|
if limit <= 0:
|
||||||
return []
|
return []
|
||||||
@@ -391,14 +403,15 @@ def after(
|
|||||||
def newest(adventure: models.Adventure) -> models.Action | None:
|
def newest(adventure: models.Adventure) -> models.Action | None:
|
||||||
"""The newest story action, or None on an empty story.
|
"""The newest story action, or None on an empty story.
|
||||||
|
|
||||||
A row, not a count and an offset: this is the node an anchor moves to when
|
This returns a row rather than a count and an offset, because it names the
|
||||||
derived work catches up with the end of the story.
|
node an anchor moves to when derived work reaches the end of the story.
|
||||||
|
|
||||||
It used to be the *second* newest — the memory bank held one action back
|
It used to return the second newest action. The memory bank held one action
|
||||||
because retry rewrote a row, so a memory covering the newest action could
|
back because retry rewrote a row, so a memory that covered the newest action
|
||||||
end up describing narration the player had retried away. Since SP4 a retry
|
could describe narration the player had already replaced. Since SP4, a retry
|
||||||
writes a sibling instead, and the coordinate's derived work is withdrawn
|
writes a sibling row instead, and the derived work at a coordinate is
|
||||||
when the story at it changes, so there is nothing left to hold back.
|
withdrawn when the story at that coordinate changes. There is nothing left
|
||||||
|
to hold back.
|
||||||
"""
|
"""
|
||||||
rows = tail(adventure, 1)
|
rows = tail(adventure, 1)
|
||||||
return rows[0] if rows else None
|
return rows[0] if rows else None
|
||||||
@@ -407,11 +420,12 @@ def newest(adventure: models.Adventure) -> models.Action | None:
|
|||||||
def max_action_index(adventure: models.Adventure) -> int:
|
def max_action_index(adventure: models.Adventure) -> int:
|
||||||
"""Highest `Action.index` in the adventure, story text or not. -1 if empty.
|
"""Highest `Action.index` in the adventure, story text or not. -1 if empty.
|
||||||
|
|
||||||
The one read here that is deliberately *not* path-scoped. `index` is the
|
This is the only read in the module that is deliberately not scoped to a
|
||||||
legacy column, kept unread until SP8 drops it, and its only remaining job
|
path. `index` is a legacy column that remains unread until SP8 drops it. Its
|
||||||
is to hand the next row a number nothing else holds — which is a fact about
|
one remaining job is to give the next row a number that no other row holds,
|
||||||
the adventure, not about the story being played. Scoping it to a branch
|
which is a fact about the adventure rather than about the story being
|
||||||
would let two branches issue the same index.
|
played. Scoping the query to a branch would let two branches issue the same
|
||||||
|
index.
|
||||||
"""
|
"""
|
||||||
loaded = _loaded_actions(adventure)
|
loaded = _loaded_actions(adventure)
|
||||||
if loaded is not None:
|
if loaded is not None:
|
||||||
@@ -433,16 +447,18 @@ def window_covering(
|
|||||||
token_counter,
|
token_counter,
|
||||||
exclude_action_id: int | None = None,
|
exclude_action_id: int | None = None,
|
||||||
) -> list[models.Action]:
|
) -> list[models.Action]:
|
||||||
"""The newest story actions whose combined text exceeds `budget_tokens` —
|
"""Returns the newest story actions whose combined text exceeds `budget_tokens`.
|
||||||
i.e. more than the context builder can possibly include, and never less.
|
|
||||||
|
|
||||||
Measures rather than guesses a chars-per-token ratio, so the prompt is
|
The result always holds at least as much text as the context builder can
|
||||||
byte-for-byte what loading the whole story would have produced. Budgets on
|
include, and never less.
|
||||||
the raw text, which is never longer than the rendered history text, so
|
|
||||||
erring here can only mean fetching slightly too much.
|
|
||||||
|
|
||||||
Each round fetches only the actions it doesn't already hold, so no row is
|
This function counts tokens rather than estimating a characters-per-token
|
||||||
ever read twice however many rounds it takes.
|
ratio, so the prompt matches what loading the whole story would produce. It
|
||||||
|
budgets against the raw text, which is never longer than the rendered
|
||||||
|
history text, so any error causes it to fetch slightly more than needed.
|
||||||
|
|
||||||
|
Each round fetches only the actions it does not already hold, so no row is
|
||||||
|
read twice however many rounds the loop takes.
|
||||||
"""
|
"""
|
||||||
actions: list[models.Action] = []
|
actions: list[models.Action] = []
|
||||||
tokens = 0
|
tokens = 0
|
||||||
@@ -452,15 +468,15 @@ def window_covering(
|
|||||||
adventure, len(actions), size - len(actions), exclude_action_id
|
adventure, len(actions), size - len(actions), exclude_action_id
|
||||||
)
|
)
|
||||||
if not older:
|
if not older:
|
||||||
return actions # already holding the whole story
|
return actions # The result already holds the whole story.
|
||||||
actions = older + actions
|
actions = older + actions
|
||||||
tokens += sum(token_counter(a.text) for a in older)
|
tokens += sum(token_counter(a.text) for a in older)
|
||||||
if len(actions) < size:
|
if len(actions) < size:
|
||||||
return actions # that was the whole story
|
return actions # That was the whole story.
|
||||||
if tokens > budget_tokens:
|
if tokens > budget_tokens:
|
||||||
return actions
|
return actions
|
||||||
# Short. Project how many actions the budget takes at the length these
|
# The window is still short. Estimate how many actions the budget needs
|
||||||
# ones turned out to be, and go straight there.
|
# at the average length just measured, then read that many.
|
||||||
average = tokens / len(actions)
|
average = tokens / len(actions)
|
||||||
projected = int(budget_tokens / average * (1 + WINDOW_MARGIN)) + WINDOW_STEP
|
projected = int(budget_tokens / average * (1 + WINDOW_MARGIN)) + WINDOW_STEP
|
||||||
size = max(projected, size + WINDOW_STEP)
|
size = max(projected, size + WINDOW_STEP)
|
||||||
|
|||||||
+116
-106
@@ -1,35 +1,35 @@
|
|||||||
"""Phase 14 — which nodes are "this story".
|
"""Phase 14: decides which nodes make up one story.
|
||||||
|
|
||||||
`tree.py` decides where a node is written. This module is the other half: it
|
`tree.py` decides where a node is written. This module decides which nodes a
|
||||||
decides which nodes a read can see, and it is the **only** place that knows.
|
read can see, and it is the only place that makes that decision.
|
||||||
|
|
||||||
A branch owns the nodes played on it and *borrows* everything before its fork
|
A branch owns the nodes played on it and inherits everything before its fork
|
||||||
point from its ancestors, so "the story on branch C" is not a column you can
|
point from its ancestors. "The story on branch C" is therefore not a value you
|
||||||
filter on — it is an OR of ranges::
|
can filter a column on. It is an OR of ranges::
|
||||||
|
|
||||||
(branch_id = C) -- C's own nodes, to the tip
|
(branch_id = C) -- C's own nodes, through to the tip
|
||||||
OR (branch_id = B AND depth <= 5)
|
OR (branch_id = B AND depth <= 5)
|
||||||
OR (branch_id = A AND depth <= 3)
|
OR (branch_id = A AND depth <= 3)
|
||||||
|
|
||||||
which is exactly what `branches.lineage` spells out, newest first, computed
|
The `branches.lineage` column records exactly that list, newest first. The fork
|
||||||
once when the fork happens. Reads never walk parent pointers to rebuild it.
|
computes it once, so no read walks parent pointers to rebuild it.
|
||||||
|
|
||||||
Two properties fall out of the shape, and both are load-bearing:
|
The shape of the list gives two properties that the windowed reads depend on:
|
||||||
|
|
||||||
* **The ranges are disjoint and descending.** A branch's own nodes always sit
|
- The ranges do not overlap, and they descend. A branch's own nodes always sit
|
||||||
deeper than its fork point, and each lineage entry is capped at the fork
|
deeper than its fork point, and each lineage entry is capped at the fork depth
|
||||||
depth of the branch beneath it. So ordering the whole clause by `depth`
|
of the branch below it. Ordering the whole clause by `depth` descending
|
||||||
descending is the same as reading entry 0's nodes, then entry 1's, then
|
therefore reads entry 0's nodes, then entry 1's, then entry 2's. A tail read
|
||||||
entry 2's — which is what lets a tail read use only the newest few entries
|
can use the newest few entries and stop.
|
||||||
and stop.
|
- The number of clauses depends on the size of the context window, not on how
|
||||||
* **Clause count is bounded by the context window, not by fork count.** A
|
many times the story has forked. A story with 200 forks whose newest branch
|
||||||
200-fork story whose newest branch is 40 turns long reads with one clause,
|
runs 40 turns reads with a single clause, because the window is covered before
|
||||||
because the window is covered before the second entry is reached. That is
|
the second entry is reached. `prefix_covering` implements this, and it is why
|
||||||
`prefix_covering`, and it is why `history.window_covering` can keep its shape.
|
`history.window_covering` can keep its current shape.
|
||||||
|
|
||||||
Everything here is a read. Nothing in this module creates a branch or writes a
|
Everything in this module reads. Nothing here creates a branch or writes a row.
|
||||||
row: an adventure with no branch has no story, and healing that is the write
|
An adventure with no branch has no story, and the write side repairs that. See
|
||||||
side's job (`tree.place_action`, and the flush guard in `models.py` behind it).
|
`tree.place_action` and the flush listener in `models.py`.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from sqlalchemy import and_, false, or_
|
from sqlalchemy import and_, false, or_
|
||||||
@@ -37,31 +37,33 @@ from sqlalchemy.orm import Session
|
|||||||
|
|
||||||
from .. import models
|
from .. import models
|
||||||
|
|
||||||
# The depth of an adventure with no actions. Mirrors tree.NO_DEPTH; kept
|
# The depth of an adventure that has no actions. This mirrors `tree.NO_DEPTH`.
|
||||||
# separately so a read never has to import the write half.
|
# It is duplicated here so that a read never has to import the write half.
|
||||||
NO_DEPTH = -1
|
NO_DEPTH = -1
|
||||||
|
|
||||||
# The opening node of an adventure. Depth 0 exists only on the root branch — a
|
# The opening node of an adventure. Depth 0 exists only on the root branch,
|
||||||
# fork starts its own nodes after the depth it forked at — so this names one
|
# because a fork starts its own nodes after the depth it forked at. This
|
||||||
# node per adventure, not one per branch. It is also where migration 62 parked
|
# constant therefore names one node per adventure rather than one per branch.
|
||||||
# every memory written before memories had coordinates, which is why the two
|
#
|
||||||
# places that can retire a memory (`memorybank.forget_node`, and a v1 import
|
# Migration 62 also placed every memory written before memories had coordinates
|
||||||
# with no depth to read) both have to say something about it.
|
# at depth 0. That is why both places that can retire a memory must handle this
|
||||||
|
# depth: `memorybank.forget_node`, and a v1 import that has no depth to read.
|
||||||
ROOT_DEPTH = 0
|
ROOT_DEPTH = 0
|
||||||
|
|
||||||
|
|
||||||
def entries_of(branch: models.Branch) -> list[tuple[int, int | None]]:
|
def entries_of(branch: models.Branch) -> list[tuple[int, int | None]]:
|
||||||
"""`branch.lineage` as (branch_id, max_depth) pairs, newest first.
|
"""Returns `branch.lineage` as (branch_id, max_depth) pairs, newest first.
|
||||||
|
|
||||||
An empty lineage reads as "this branch alone, to its tip" rather than as an
|
An empty lineage means this branch alone, through to its tip. That is not an
|
||||||
error. That is what a root branch's lineage means, and it is what a branch
|
error. It is what a root branch's lineage means, and it is also what a
|
||||||
row looks like in the moment between being inserted and having its own id
|
branch row holds between the moment it is inserted and the moment its own id
|
||||||
to name — so the fallback is the truth, not a guess.
|
is written into the column. The fallback is therefore correct rather than a
|
||||||
|
guess.
|
||||||
"""
|
"""
|
||||||
raw = branch.lineage if isinstance(branch.lineage, list) else []
|
raw = branch.lineage if isinstance(branch.lineage, list) else []
|
||||||
entries: list[tuple[int, int | None]] = []
|
entries: list[tuple[int, int | None]] = []
|
||||||
for item in raw:
|
for item in raw:
|
||||||
# JSON round-trips lists; a hand-written row might hold tuples.
|
# JSON round-trips lists, but a hand-written row might hold tuples.
|
||||||
if not isinstance(item, (list, tuple)) or not item:
|
if not isinstance(item, (list, tuple)) or not item:
|
||||||
continue
|
continue
|
||||||
branch_id = item[0]
|
branch_id = item[0]
|
||||||
@@ -73,10 +75,10 @@ def entries_of(branch: models.Branch) -> list[tuple[int, int | None]]:
|
|||||||
|
|
||||||
|
|
||||||
class Path:
|
class Path:
|
||||||
"""One story, as a clause and as a predicate.
|
"""One story, expressed as a SQL clause and as a Python predicate.
|
||||||
|
|
||||||
Holds the lineage entries newest first, plus the depth of the tip, which is
|
The object holds the lineage entries newest first, plus the depth of the
|
||||||
only used to estimate how much story each entry covers.
|
tip. The tip is used only to estimate how much story each entry covers.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, entries: list[tuple[int, int | None]], tip: int | None = None):
|
def __init__(self, entries: list[tuple[int, int | None]], tip: int | None = None):
|
||||||
@@ -96,28 +98,29 @@ class Path:
|
|||||||
model=models.Action,
|
model=models.Action,
|
||||||
count: int | None = None,
|
count: int | None = None,
|
||||||
):
|
):
|
||||||
"""The branch clause, over `model` (`Action` or `Memory`).
|
"""Returns the branch clause over `model`, which is `Action` or `Memory`.
|
||||||
|
|
||||||
`count` limits it to the newest `count` lineage entries — the windowed
|
`count` limits the clause to the newest `count` lineage entries, which
|
||||||
read. `None` is the whole lineage, which is what anything counting from
|
produces a windowed read. Pass `None` for the whole lineage. Any caller
|
||||||
the *oldest* end (a slice, a total) has to use.
|
that counts from the oldest end, such as a slice or a total, must pass
|
||||||
|
`None`.
|
||||||
|
|
||||||
Every row this reads has a depth. Memories used to be the exception —
|
Every row this clause selects has a depth. Memories were once an
|
||||||
a hand-written one had a branch and no depth, and needed an escape
|
exception, because a hand-written memory had a branch but no depth and
|
||||||
clause here to survive being capped at a fork. SP7 anchors them at the
|
needed an escape clause here to avoid being capped at a fork. SP7
|
||||||
head instead (`tree.place_memory`), which is a better answer to the same
|
anchors those memories at the head instead. See `tree.place_memory`. A
|
||||||
problem: the memory is not exempt from the path, it is *on* one. A row
|
memory is now on a path rather than exempt from one, and a row with no
|
||||||
with no depth is now a pre-tree leftover that no read should see.
|
depth is a pre-tree leftover that no read should return.
|
||||||
|
|
||||||
Actions also have to be *live* (SP4). A coordinate can hold several
|
Actions must also be live, as of SP4. One coordinate can hold several
|
||||||
attempts at the same turn, and the story tells one of them; the losing
|
attempts at a turn, and the story uses one of them. The other attempts
|
||||||
siblings sit at the same branch and depth and are excluded here, once,
|
sit at the same branch and depth, and this clause excludes them once, so
|
||||||
so that no read of the story has to know that retries exist. Only
|
that no read of the story has to account for retries. Only
|
||||||
`app/attempts.py` looks past this.
|
`app/attempts.py` looks past this filter.
|
||||||
|
|
||||||
An empty path yields `false`, not "no filter": an adventure whose nodes
|
An empty path returns `false` rather than no filter at all. An adventure
|
||||||
carry no branch has no story, and the loud version of that is an empty
|
whose nodes carry no branch has no story, and the correct way to show
|
||||||
page, not every branch at once.
|
that is an empty page rather than every branch at once.
|
||||||
"""
|
"""
|
||||||
entries = self.entries if count is None else self.entries[:count]
|
entries = self.entries if count is None else self.entries[:count]
|
||||||
if not entries:
|
if not entries:
|
||||||
@@ -136,14 +139,16 @@ class Path:
|
|||||||
# ------------------------------------------------------------- Python
|
# ------------------------------------------------------------- Python
|
||||||
|
|
||||||
def contains(self, node) -> bool:
|
def contains(self, node) -> bool:
|
||||||
"""The Python half of `clause()` — keep the two in step.
|
"""Returns whether `node` is on this path.
|
||||||
|
|
||||||
Used where the rows are already in memory (the scripting pipeline hands
|
This is the Python equivalent of `clause()`. Keep the two in step.
|
||||||
user scripts the whole history), so an already-loaded collection can be
|
|
||||||
cut down to the path without a second read.
|
|
||||||
|
|
||||||
`live` is checked first, and only on rows that have the attribute:
|
Callers use it where the rows are already in memory. The scripting
|
||||||
memories have no siblings to lose to.
|
pipeline hands user scripts the whole history, so a loaded collection
|
||||||
|
can be reduced to the path without a second query.
|
||||||
|
|
||||||
|
The method checks `live` first, and only on rows that define the
|
||||||
|
attribute, because memories have no siblings.
|
||||||
"""
|
"""
|
||||||
if getattr(node, "live", True) is False:
|
if getattr(node, "live", True) is False:
|
||||||
return False
|
return False
|
||||||
@@ -157,22 +162,25 @@ class Path:
|
|||||||
return False
|
return False
|
||||||
|
|
||||||
def sort_key(self, node) -> tuple[int, int]:
|
def sort_key(self, node) -> tuple[int, int]:
|
||||||
"""Oldest-first ordering. `depth` is the ordering key; `id` breaks the
|
"""Returns a sort key that orders nodes from oldest to newest.
|
||||||
tie a pre-tree row (depth NULL) or a future sibling pair would leave."""
|
|
||||||
|
`depth` is the ordering key. `id` breaks ties, which a pre-tree row with
|
||||||
|
a NULL depth or a pair of siblings can produce.
|
||||||
|
"""
|
||||||
return (node.depth if node.depth is not None else NO_DEPTH, node.id or 0)
|
return (node.depth if node.depth is not None else NO_DEPTH, node.id or 0)
|
||||||
|
|
||||||
# ------------------------------------------------------------ windowing
|
# ------------------------------------------------------------ windowing
|
||||||
|
|
||||||
def prefix_covering(self, rows: int) -> int:
|
def prefix_covering(self, rows: int) -> int:
|
||||||
"""How many lineage entries it takes to hold the newest `rows` nodes.
|
"""Returns how many lineage entries hold the newest `rows` nodes.
|
||||||
|
|
||||||
An estimate from depth arithmetic, not a query: entry *i* covers the
|
The result is an estimate from depth arithmetic rather than a query.
|
||||||
depths between its own cap and the cap of the entry below it, and there
|
Entry *i* covers the depths between its own cap and the cap of the entry
|
||||||
is at most one node per depth on a path. So the count it returns is
|
below it, and a path holds at most one node per depth. The estimate is
|
||||||
never too many, and is too few only where the story has gaps — an
|
therefore never too large. It is too small only when the story has gaps,
|
||||||
action deleted from the middle. The caller widens to the full lineage
|
which happens after an action is deleted from the middle. In that case
|
||||||
if the window comes up short, which costs a second query on a story
|
the caller widens the read to the whole lineage, which costs one extra
|
||||||
somebody has deleted from, and nothing at all otherwise.
|
query.
|
||||||
"""
|
"""
|
||||||
total = len(self.entries)
|
total = len(self.entries)
|
||||||
if rows <= 0 or total == 0:
|
if rows <= 0 or total == 0:
|
||||||
@@ -182,9 +190,9 @@ class Path:
|
|||||||
top = self.tip if max_depth is None else max_depth
|
top = self.tip if max_depth is None else max_depth
|
||||||
below = self.entries[i + 1][1] if i + 1 < total else NO_DEPTH
|
below = self.entries[i + 1][1] if i + 1 < total else NO_DEPTH
|
||||||
if top is None or below is None:
|
if top is None or below is None:
|
||||||
# No tip recorded, or a cap missing from a hand-written row:
|
# Either no tip was recorded, or a hand-written row is missing a
|
||||||
# nothing to estimate from, so read the lot rather than guess
|
# cap. There is nothing to estimate from, so return every entry.
|
||||||
# short and hide the older half of the story.
|
# Guessing low would hide the older half of the story.
|
||||||
return total
|
return total
|
||||||
covered += max(top - below, 0)
|
covered += max(top - below, 0)
|
||||||
if covered >= rows:
|
if covered >= rows:
|
||||||
@@ -192,15 +200,15 @@ class Path:
|
|||||||
return total
|
return total
|
||||||
|
|
||||||
def covering_after(self, depth: int) -> int:
|
def covering_after(self, depth: int) -> int:
|
||||||
"""How many lineage entries can hold a node deeper than `depth`.
|
"""Returns how many lineage entries can hold a node deeper than `depth`.
|
||||||
|
|
||||||
The counterpart to `prefix_covering`, and unlike it this is exact
|
This is the counterpart to `prefix_covering`, and it is exact rather than
|
||||||
rather than an estimate: entry *i* holds nothing deeper than its own
|
an estimate. Entry *i* holds nothing deeper than its own cap, and the
|
||||||
cap, and the caps descend, so the first entry capped at or below
|
caps descend, so the first entry capped at or below `depth` ends the
|
||||||
`depth` ends the search — it and everything older is behind the
|
search. That entry and every older one fall behind the boundary. As a
|
||||||
boundary. Reading "the story after the cursor" therefore names one
|
result, reading the story after the cursor touches one branch on any
|
||||||
branch on any story whose cursor is on its newest branch, however
|
story whose cursor sits on its newest branch, however many times the
|
||||||
often it has forked.
|
story has forked.
|
||||||
"""
|
"""
|
||||||
for i, (_, max_depth) in enumerate(self.entries):
|
for i, (_, max_depth) in enumerate(self.entries):
|
||||||
if max_depth is not None and max_depth <= depth:
|
if max_depth is not None and max_depth <= depth:
|
||||||
@@ -208,27 +216,28 @@ class Path:
|
|||||||
return len(self.entries)
|
return len(self.entries)
|
||||||
|
|
||||||
def depth_on(self, branch_id: int | None, depth: int) -> int:
|
def depth_on(self, branch_id: int | None, depth: int) -> int:
|
||||||
"""A stored `(branch_id, depth)` anchor, read as a depth on *this* path.
|
"""Reads a stored `(branch_id, depth)` anchor as a depth on this path.
|
||||||
|
|
||||||
An anchor is how far along a story some derived work has got — which
|
An anchor records how far along a story some derived work reached, such
|
||||||
memories cover, what the summary has folded in. It names a node, so
|
as which actions the memories cover or what the summary folded in. The
|
||||||
moving to another path has to be answered rather than assumed:
|
anchor names a node, so moving to a different path needs an explicit
|
||||||
|
answer. There are two cases:
|
||||||
|
|
||||||
* the anchor's branch is on this path — the depth stands, capped at the
|
- The anchor's branch is on this path. The depth stands, capped at the
|
||||||
fork the path takes off that branch, because nothing past the fork is
|
fork where this path leaves that branch, because nothing past the fork
|
||||||
on this story;
|
belongs to this story.
|
||||||
* the branch is not on this path at all — the work was done on ground
|
- The anchor's branch is not on this path. The work was done on a branch
|
||||||
this story never travelled, so nothing here is covered.
|
this story does not contain, so nothing here counts as covered.
|
||||||
|
|
||||||
The second case cannot arise while an adventure has one branch: the
|
The second case cannot occur while an adventure has one branch, because
|
||||||
anchor is always set from a node on it. It exists because the fallback
|
the anchor is always set from a node on it. It exists because the safe
|
||||||
for "I don't know" must be to redo the work, not to skip it.
|
answer to an unknown anchor is to redo the work rather than skip it.
|
||||||
"""
|
"""
|
||||||
if depth <= NO_DEPTH:
|
if depth <= NO_DEPTH:
|
||||||
return NO_DEPTH
|
return NO_DEPTH
|
||||||
if branch_id is None:
|
if branch_id is None:
|
||||||
# A pre-tree anchor, or one set by hand. There is one story, so the
|
# The anchor predates the tree, or someone set it by hand. There is
|
||||||
# depth is a position in it and means what it says.
|
# only one story, so the depth is a position in it.
|
||||||
return depth
|
return depth
|
||||||
for entry_branch, max_depth in self.entries:
|
for entry_branch, max_depth in self.entries:
|
||||||
if entry_branch == branch_id:
|
if entry_branch == branch_id:
|
||||||
@@ -237,18 +246,19 @@ class Path:
|
|||||||
|
|
||||||
|
|
||||||
def branch_of(db: Session, adventure: models.Adventure) -> models.Branch | None:
|
def branch_of(db: Session, adventure: models.Adventure) -> models.Branch | None:
|
||||||
"""The branch this adventure is being read at, or None if it has none.
|
"""Returns the branch this adventure is read at, or None if it has none.
|
||||||
|
|
||||||
Deliberately not `tree.head_branch`, which creates one: a GET must not
|
This function is deliberately not `tree.head_branch`, which creates a branch.
|
||||||
write. An adventure with no branch row also has no nodes carrying a branch,
|
A GET request must not write. An adventure with no branch row also has no
|
||||||
so the two agree — both say "no story here".
|
nodes that carry a branch, so both answers agree that there is no story.
|
||||||
"""
|
"""
|
||||||
if adventure.head_branch_id is not None:
|
if adventure.head_branch_id is not None:
|
||||||
branch = db.get(models.Branch, adventure.head_branch_id)
|
branch = db.get(models.Branch, adventure.head_branch_id)
|
||||||
if branch is not None:
|
if branch is not None:
|
||||||
return branch
|
return branch
|
||||||
# A head naming a branch that is gone: fall through to the root, the
|
# The head points at a branch that no longer exists. Fall through to the
|
||||||
# same recovery `tree.head_branch` makes on the write side.
|
# root, which is the same recovery that `tree.head_branch` performs on
|
||||||
|
# the write side.
|
||||||
return (
|
return (
|
||||||
db.query(models.Branch)
|
db.query(models.Branch)
|
||||||
.filter(
|
.filter(
|
||||||
@@ -261,7 +271,7 @@ def branch_of(db: Session, adventure: models.Adventure) -> models.Branch | None:
|
|||||||
|
|
||||||
|
|
||||||
def path_of(db: Session, adventure: models.Adventure) -> Path:
|
def path_of(db: Session, adventure: models.Adventure) -> Path:
|
||||||
"""The story the adventure's head is currently on."""
|
"""Returns the story that the adventure's head currently sits on."""
|
||||||
branch = branch_of(db, adventure)
|
branch = branch_of(db, adventure)
|
||||||
if branch is None:
|
if branch is None:
|
||||||
return Path([], adventure.head_depth)
|
return Path([], adventure.head_depth)
|
||||||
|
|||||||
@@ -15,9 +15,10 @@ DB_PATH = (
|
|||||||
else Path(__file__).resolve().parent.parent / "data.db"
|
else Path(__file__).resolve().parent.parent / "data.db"
|
||||||
)
|
)
|
||||||
|
|
||||||
# AIDND_DATABASE_URL (or the platform-conventional DATABASE_URL) switches the
|
# `AIDND_DATABASE_URL`, or the conventional `DATABASE_URL`, switches the app to
|
||||||
# app to a server database — any SQLAlchemy URL works, but Postgres is what
|
# a server database. Any SQLAlchemy URL works, and hosted deploys use Postgres,
|
||||||
# hosted deploys use (Phase 9 decision: Neon). Unset = SQLite, as always.
|
# which Phase 9 settled on Neon for. If neither variable is set, the app uses
|
||||||
|
# SQLite.
|
||||||
DATABASE_URL = (
|
DATABASE_URL = (
|
||||||
os.environ.get("AIDND_DATABASE_URL", "").strip()
|
os.environ.get("AIDND_DATABASE_URL", "").strip()
|
||||||
or os.environ.get("DATABASE_URL", "").strip()
|
or os.environ.get("DATABASE_URL", "").strip()
|
||||||
|
|||||||
@@ -54,10 +54,13 @@ def finish_entry(
|
|||||||
error: str | None = None,
|
error: str | None = None,
|
||||||
usage: dict | None = None,
|
usage: dict | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""`usage` is the endpoint's own token accounting when it reported any.
|
"""Finishes a log entry.
|
||||||
On OpenRouter it carries `prompt_tokens_details.cached_tokens`, which is
|
|
||||||
the only direct read on whether the prompt prefix is actually being
|
`usage` is the endpoint's own token accounting, when it reported any. On
|
||||||
cached — a number worth seeing beside the request that produced it."""
|
OpenRouter it carries `prompt_tokens_details.cached_tokens`, which is the
|
||||||
|
only direct measure of whether the prompt prefix is being cached, and it is
|
||||||
|
worth seeing next to the request that produced it.
|
||||||
|
"""
|
||||||
entry["response"] = _clip(response)
|
entry["response"] = _clip(response)
|
||||||
entry["usage"] = usage
|
entry["usage"] = usage
|
||||||
entry["error"] = error
|
entry["error"] = error
|
||||||
|
|||||||
@@ -22,9 +22,9 @@ DATA_URI_RE = re.compile(
|
|||||||
def public_url(scenario_id: int, image: str, version: object) -> str:
|
def public_url(scenario_id: int, image: str, version: object) -> str:
|
||||||
"""The URL a client should load for this scenario's art ("" if none).
|
"""The URL a client should load for this scenario's art ("" if none).
|
||||||
|
|
||||||
`version` (any object with a stable repr — normally the row's updated_at)
|
`version` is any object with a stable repr, and is normally the row's
|
||||||
becomes a cache-buster, letting the image response be marked immutable
|
`updated_at`. It becomes a cache-buster, so the image response can be marked
|
||||||
while still refreshing the moment the author swaps the picture.
|
immutable and still refresh as soon as the author replaces the picture.
|
||||||
"""
|
"""
|
||||||
if not image:
|
if not image:
|
||||||
return ""
|
return ""
|
||||||
@@ -40,9 +40,9 @@ def public_url(scenario_id: int, image: str, version: object) -> str:
|
|||||||
def sanitize(value: object, max_length: int) -> str:
|
def sanitize(value: object, max_length: int) -> str:
|
||||||
"""Coerce an untrusted `image` value from an import bundle to a safe one.
|
"""Coerce an untrusted `image` value from an import bundle to a safe one.
|
||||||
|
|
||||||
Anything that isn't a supported data URI or an https URL — or that is too
|
A value that is not a supported data URI or an https URL, or that is too
|
||||||
large to store — becomes "", so a hostile or merely foreign bundle can't
|
large to store, becomes "". A hostile or merely unfamiliar bundle therefore
|
||||||
smuggle in a `javascript:` URI or blow past the column cap.
|
cannot pass in a `javascript:` URI or exceed the column cap.
|
||||||
"""
|
"""
|
||||||
if not isinstance(value, str) or not value or len(value) > max_length:
|
if not isinstance(value, str) or not value or len(value) > max_length:
|
||||||
return ""
|
return ""
|
||||||
@@ -62,5 +62,6 @@ def decode(image: str) -> tuple[bytes, str] | None:
|
|||||||
payload = re.sub(r"\s+", "", match.group(2))
|
payload = re.sub(r"\s+", "", match.group(2))
|
||||||
return base64.b64decode(payload, validate=True), match.group(1).lower()
|
return base64.b64decode(payload, validate=True), match.group(1).lower()
|
||||||
except (binascii.Error, ValueError):
|
except (binascii.Error, ValueError):
|
||||||
# Truncated or hand-edited base64 — treat as "no image" rather than 500.
|
# The base64 is truncated or hand-edited. Treat it as no image rather
|
||||||
|
# than return a 500.
|
||||||
return None
|
return None
|
||||||
|
|||||||
+102
-77
@@ -1,9 +1,10 @@
|
|||||||
"""Phase 9 — abuse guards for hosted (multi-user) deployments.
|
"""Phase 9: abuse guards for hosted, multi-user deployments.
|
||||||
|
|
||||||
Rate limits and row caps are no-ops in local mode: a single local player
|
Rate limits and row caps do nothing in local mode, because a single local player
|
||||||
should never be throttled by their own app. Values are hardcoded on purpose —
|
should never be throttled by their own app. The values are hardcoded on purpose.
|
||||||
generous enough that a legitimate player never notices, tight enough that a
|
They are generous enough that a legitimate player never notices them, and tight
|
||||||
hostile visitor can't burn the demo key, peg the CPU, or bloat the database.
|
enough that a hostile visitor cannot exhaust the demo key, saturate the CPU, or
|
||||||
|
fill the database.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
@@ -19,22 +20,22 @@ from sqlalchemy.orm import Session
|
|||||||
from . import auth, models
|
from . import auth, models
|
||||||
|
|
||||||
# ---------- Rate limiting ----------
|
# ---------- Rate limiting ----------
|
||||||
# Fixed windows per (scope, caller). In-memory: fine for the single-process
|
# Fixed windows per scope and caller. The windows live in memory, which is
|
||||||
# deployment this app targets (and the worst case after a restart is a brief
|
# enough for the single-process deployment this app targets. The worst case
|
||||||
# extra allowance).
|
# after a restart is a brief extra allowance.
|
||||||
|
|
||||||
# scope -> (max requests, window seconds)
|
# Maps a scope to (max requests, window seconds).
|
||||||
RATE_LIMITS: dict[str, tuple[int, int]] = {
|
RATE_LIMITS: dict[str, tuple[int, int]] = {
|
||||||
"turn": (10, 60), # AI turn generation (demo key also has a daily cap)
|
"turn": (10, 60), # AI turn generation. The demo key also has a daily cap.
|
||||||
"chat": (30, 60), # AI Chat scratchpad (power users only)
|
"chat": (30, 60), # The AI Chat scratchpad, for power users.
|
||||||
"script-test": (30, 60), # sandboxed, but each run costs up to 2s CPU
|
"script-test": (30, 60), # Sandboxed, but each run costs up to 2s of CPU.
|
||||||
"connection-test": (10, 60), # outbound HTTP to a user-supplied URL
|
"connection-test": (10, 60), # Outbound HTTP to a user-supplied URL.
|
||||||
"import": (30, 60), # large writes
|
"import": (30, 60), # Large writes.
|
||||||
"auth": (10, 300), # register/login attempts, per IP
|
"auth": (10, 300), # Register and login attempts, per IP.
|
||||||
"guest": (30, 300), # new guest users, per IP (each is a DB row)
|
"guest": (30, 300), # New guest users, per IP. Each one is a database row.
|
||||||
# Pageview beacons. Generous — a real reader clicking around a SPA fires a
|
# Pageview beacons. The limit is generous, because a real reader clicking
|
||||||
# handful a minute — but low enough that nobody can inflate the traffic
|
# around a SPA sends a handful a minute, and it is low enough that nobody
|
||||||
# numbers faster than they could by actually reloading the page.
|
# can inflate the traffic numbers faster than by reloading the page.
|
||||||
"analytics": (120, 60),
|
"analytics": (120, 60),
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -42,26 +43,30 @@ _windows: dict[tuple[str, str], deque] = defaultdict(deque)
|
|||||||
_windows_guard = threading.Lock()
|
_windows_guard = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
# How many proxy hops sit between the app and the real client. On Render (and
|
# How many proxy hops sit between the app and the real client. On Render, and on
|
||||||
# most PaaS) that's one: the platform's edge appends the connecting IP to the
|
# most platforms, that is one, because the platform's edge appends the connecting
|
||||||
# RIGHT of X-Forwarded-For. A client can prepend anything it likes to the left,
|
# IP to the right of `X-Forwarded-For`. A client can prepend any value on the
|
||||||
# but it cannot push a value past the edge's own append — so the trustworthy
|
# left, but it cannot push a value past the edge's own append, so the trustworthy
|
||||||
# client IP is the (hops)-th entry from the right, NOT uvicorn's leftmost pick.
|
# client IP is the entry that many places from the right rather than uvicorn's
|
||||||
# Trusting the leftmost let anyone rotate X-Forwarded-For to mint a fresh
|
# leftmost choice. Trusting the leftmost entry let anyone rotate
|
||||||
# rate-limit bucket per request and bypass the auth/guest limits entirely.
|
# `X-Forwarded-For` to get a fresh rate-limit bucket per request and bypass the
|
||||||
# Override with AIDND_TRUSTED_PROXY_HOPS if the deployment adds more hops.
|
# auth and guest limits. If the deployment adds more hops, set
|
||||||
|
# `AIDND_TRUSTED_PROXY_HOPS`.
|
||||||
TRUSTED_PROXY_HOPS = max(1, int(os.environ.get("AIDND_TRUSTED_PROXY_HOPS", "1") or 1))
|
TRUSTED_PROXY_HOPS = max(1, int(os.environ.get("AIDND_TRUSTED_PROXY_HOPS", "1") or 1))
|
||||||
|
|
||||||
|
|
||||||
def client_ip(request: Request) -> str:
|
def client_ip(request: Request) -> str:
|
||||||
"""The real client IP, resistant to a spoofed X-Forwarded-For. Takes the
|
"""Returns the real client IP, resisting a spoofed `X-Forwarded-For`.
|
||||||
hop the trusted edge appended (rightmost minus any extra trusted hops);
|
|
||||||
falls back to the socket peer when no forwarded header is present
|
|
||||||
(local/dev, or a direct connection).
|
|
||||||
|
|
||||||
Public because the access log needs the same answer, and two functions that
|
The function reads the hop the trusted edge appended, which is the rightmost
|
||||||
both decide "which address is the caller's" is how one of them ends up
|
entry minus any extra trusted hops. If no forwarded header is present, which
|
||||||
trusting a header it shouldn't."""
|
happens locally, in development, and on a direct connection, it falls back to
|
||||||
|
the socket peer.
|
||||||
|
|
||||||
|
The function is public because the access log needs the same answer. Two
|
||||||
|
functions that each decide which address belongs to the caller is how one of
|
||||||
|
them ends up trusting a header it should not.
|
||||||
|
"""
|
||||||
forwarded = request.headers.get("x-forwarded-for")
|
forwarded = request.headers.get("x-forwarded-for")
|
||||||
if forwarded:
|
if forwarded:
|
||||||
parts = [p.strip() for p in forwarded.split(",") if p.strip()]
|
parts = [p.strip() for p in forwarded.split(",") if p.strip()]
|
||||||
@@ -71,8 +76,11 @@ def client_ip(request: Request) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def rate_limit(scope: str, request: Request, user: models.User | None = None) -> None:
|
def rate_limit(scope: str, request: Request, user: models.User | None = None) -> None:
|
||||||
"""429 when the caller exceeds the scope's window. Keyed per user when one
|
"""Raises a 429 when the caller exceeds the scope's window.
|
||||||
is known (accounts survive IP changes), per IP otherwise."""
|
|
||||||
|
The window is keyed per user when a user is known, because an account
|
||||||
|
survives an IP change, and per IP otherwise.
|
||||||
|
"""
|
||||||
if not auth.MULTI_USER:
|
if not auth.MULTI_USER:
|
||||||
return
|
return
|
||||||
limit, window_seconds = RATE_LIMITS[scope]
|
limit, window_seconds = RATE_LIMITS[scope]
|
||||||
@@ -92,25 +100,30 @@ def rate_limit(scope: str, request: Request, user: models.User | None = None) ->
|
|||||||
|
|
||||||
|
|
||||||
# ---------- Per-account login throttle ----------
|
# ---------- Per-account login throttle ----------
|
||||||
# Defense in depth beside the per-IP `auth` limit: that one can be diluted by a
|
# This is defense in depth next to the per-IP `auth` limit. A botnet dilutes
|
||||||
# botnet (many real source IPs, one bucket each), so it can't by itself stop a
|
# that limit, because many real source IPs each get their own bucket, so it
|
||||||
# distributed guessing run against a single account. This cap keys on the target
|
# cannot by itself stop a distributed guessing run against one account. This cap
|
||||||
# email instead of the caller, so guessing ONE account's password stays
|
# keys on the target email rather than on the caller, so guessing one account's
|
||||||
# expensive regardless of how many addresses it comes from. Failures only — a
|
# password stays expensive however many addresses the guesses come from.
|
||||||
# correct password clears the record — and it's a short sliding window, not a
|
#
|
||||||
# hard lock, so a user mistyping a few times recovers on their own in minutes.
|
# Only failures count, and a correct password clears the record. The window
|
||||||
# Tradeoff: an attacker can keep a known account throttled (a nuisance), which
|
# slides over a short period rather than locking the account, so a user who
|
||||||
# is strictly preferable to letting it be brute-forced.
|
# mistypes a few times recovers within minutes. The trade-off is that an
|
||||||
LOGIN_FAIL_LIMIT = 8 # failed attempts per account...
|
# attacker can keep a known account throttled, which is an inconvenience and is
|
||||||
LOGIN_FAIL_WINDOW = 900 # ...within this many seconds (15 min)
|
# preferable to letting the account be brute-forced.
|
||||||
|
LOGIN_FAIL_LIMIT = 8 # Failed attempts per account.
|
||||||
|
LOGIN_FAIL_WINDOW = 900 # The window in seconds, which is 15 minutes.
|
||||||
|
|
||||||
_login_fails: dict[str, deque] = defaultdict(deque)
|
_login_fails: dict[str, deque] = defaultdict(deque)
|
||||||
_login_guard = threading.Lock()
|
_login_guard = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
def check_login_allowed(email: str) -> None:
|
def check_login_allowed(email: str) -> None:
|
||||||
"""429 when an account has too many recent failed logins. Call before
|
"""Raises a 429 when an account has too many recent failed logins.
|
||||||
verifying the password so guesses don't even reach the hash."""
|
|
||||||
|
Call this before verifying the password, so that a guess never reaches the
|
||||||
|
hash.
|
||||||
|
"""
|
||||||
if not auth.MULTI_USER:
|
if not auth.MULTI_USER:
|
||||||
return
|
return
|
||||||
now = time.time()
|
now = time.time()
|
||||||
@@ -127,13 +140,13 @@ def check_login_allowed(email: str) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def note_login_failure(email: str) -> None:
|
def note_login_failure(email: str) -> None:
|
||||||
"""Record one failed attempt against `email`."""
|
"""Records one failed attempt against `email`."""
|
||||||
if not auth.MULTI_USER:
|
if not auth.MULTI_USER:
|
||||||
return
|
return
|
||||||
now = time.time()
|
now = time.time()
|
||||||
with _login_guard:
|
with _login_guard:
|
||||||
_login_fails[email].append(now)
|
_login_fails[email].append(now)
|
||||||
if len(_login_fails) > 10_000: # bound the map on a flood of unique emails
|
if len(_login_fails) > 10_000: # Bound the map against a flood of unique emails.
|
||||||
stale = [
|
stale = [
|
||||||
key for key, window in _login_fails.items()
|
key for key, window in _login_fails.items()
|
||||||
if not window or window[-1] < now - LOGIN_FAIL_WINDOW
|
if not window or window[-1] < now - LOGIN_FAIL_WINDOW
|
||||||
@@ -143,14 +156,16 @@ def note_login_failure(email: str) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def note_login_success(email: str) -> None:
|
def note_login_success(email: str) -> None:
|
||||||
"""A correct password wipes the account's failure streak."""
|
"""Clears the account's failure record after a correct password."""
|
||||||
with _login_guard:
|
with _login_guard:
|
||||||
_login_fails.pop(email, None)
|
_login_fails.pop(email, None)
|
||||||
|
|
||||||
|
|
||||||
def _prune(now: float) -> None:
|
def _prune(now: float) -> None:
|
||||||
"""Drop callers whose whole window has expired (call with guard held) so
|
"""Drops callers whose whole window has expired, so the per-IP dict stays bounded.
|
||||||
the per-IP dict can't grow without bound."""
|
|
||||||
|
Call this with the guard held.
|
||||||
|
"""
|
||||||
longest = max(seconds for _, seconds in RATE_LIMITS.values())
|
longest = max(seconds for _, seconds in RATE_LIMITS.values())
|
||||||
stale = [key for key, window in _windows.items()
|
stale = [key for key, window in _windows.items()
|
||||||
if not window or window[-1] < now - longest]
|
if not window or window[-1] < now - longest]
|
||||||
@@ -163,13 +178,14 @@ def _prune(now: float) -> None:
|
|||||||
MAX_ADVENTURES_PER_USER = 100
|
MAX_ADVENTURES_PER_USER = 100
|
||||||
MAX_SCENARIOS_PER_USER = 200
|
MAX_SCENARIOS_PER_USER = 200
|
||||||
MAX_SCRIPTS_PER_USER = 200
|
MAX_SCRIPTS_PER_USER = 200
|
||||||
MAX_STORY_CARDS_PER_OWNER = 200 # per scenario or adventure
|
MAX_STORY_CARDS_PER_OWNER = 200 # Per scenario or per adventure.
|
||||||
MAX_MEMORIES_PER_ADVENTURE = 1000
|
MAX_MEMORIES_PER_ADVENTURE = 1000
|
||||||
MAX_ACTIONS_PER_ADVENTURE = 5000
|
MAX_ACTIONS_PER_ADVENTURE = 5000
|
||||||
# Phase 14, SP6. A branch per divergence somebody built a story on, so a tree
|
# Phase 14, SP6. A tree holds one branch per divergence somebody built a story
|
||||||
# with more of them than a story has turns is a file, not a game. Import-only
|
# on, so a tree with more branches than the story has turns came from a file
|
||||||
# for now: forking is a POST that adds one row and has no cap of its own, and
|
# rather than from play. The cap applies to imports only. Forking is a POST that
|
||||||
# the cap that matters there is `MAX_ACTIONS_PER_ADVENTURE` above it.
|
# adds one row and has no cap of its own, and the cap that matters there is
|
||||||
|
# `MAX_ACTIONS_PER_ADVENTURE` above.
|
||||||
MAX_BRANCHES_PER_ADVENTURE = 1000
|
MAX_BRANCHES_PER_ADVENTURE = 1000
|
||||||
|
|
||||||
|
|
||||||
@@ -182,9 +198,11 @@ def check_row_cap(
|
|||||||
scenario_id: int | None = None,
|
scenario_id: int | None = None,
|
||||||
adventure_id: int | None = None,
|
adventure_id: int | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""409 with a friendly message when creating one more row of `kind` would
|
"""Raises a 409 when creating one more row of `kind` would exceed its cap.
|
||||||
exceed its cap. Ownership of the passed scenario/adventure has already
|
|
||||||
been checked by the caller."""
|
The caller has already checked ownership of the scenario or adventure passed
|
||||||
|
in.
|
||||||
|
"""
|
||||||
if not auth.MULTI_USER:
|
if not auth.MULTI_USER:
|
||||||
return
|
return
|
||||||
if kind == "adventures":
|
if kind == "adventures":
|
||||||
@@ -220,18 +238,18 @@ def check_row_cap(
|
|||||||
"delete some to make room",
|
"delete some to make room",
|
||||||
)
|
)
|
||||||
elif kind == "actions":
|
elif kind == "actions":
|
||||||
# Every action of the adventure — the whole tree, not the path being
|
# Count every action in the adventure, which is the whole tree rather
|
||||||
# played. That is the number that costs storage, and nothing is ever
|
# than the path being played. That number is what costs storage, and
|
||||||
# auto-pruned, so it is the right one to cap on. It does mean a heavily
|
# nothing is pruned automatically, so it is the right one to cap. It does
|
||||||
# branched adventure reaches the cap while its *story* is shorter than
|
# mean a heavily branched adventure reaches the cap while its story is
|
||||||
# the cap, which is why the message counts "actions in this adventure"
|
# shorter than the cap, which is why the message counts "actions in this
|
||||||
# rather than turns.
|
# adventure" rather than turns.
|
||||||
count = _count(db, models.Action, models.Action.adventure_id == adventure.id)
|
count = _count(db, models.Action, models.Action.adventure_id == adventure.id)
|
||||||
cap, subject, hint = (
|
cap, subject, hint = (
|
||||||
MAX_ACTIONS_PER_ADVENTURE, "actions in this adventure",
|
MAX_ACTIONS_PER_ADVENTURE, "actions in this adventure",
|
||||||
"export it and continue in a new adventure",
|
"export it and continue in a new adventure",
|
||||||
)
|
)
|
||||||
else: # pragma: no cover — programming error, not user input
|
else: # pragma: no cover. This is a programming error, not user input.
|
||||||
raise ValueError(f"Unknown row cap kind: {kind}")
|
raise ValueError(f"Unknown row cap kind: {kind}")
|
||||||
if count >= cap:
|
if count >= cap:
|
||||||
raise HTTPException(409, f"You've reached the limit of {cap} {subject} — {hint}.")
|
raise HTTPException(409, f"You've reached the limit of {cap} {subject} — {hint}.")
|
||||||
@@ -250,8 +268,11 @@ _BUNDLE_LIST_CAPS = {
|
|||||||
|
|
||||||
|
|
||||||
def check_bundle_lists(**lists) -> None:
|
def check_bundle_lists(**lists) -> None:
|
||||||
"""409 when an import bundle's lists exceed the same caps live creation
|
"""Raises a 409 when an import bundle's lists exceed the caps live creation uses.
|
||||||
enforces (kwargs: story_cards=, memories=, actions=, branches=)."""
|
|
||||||
|
The keyword arguments are `story_cards`, `memories`, `actions`, and
|
||||||
|
`branches`.
|
||||||
|
"""
|
||||||
if not auth.MULTI_USER:
|
if not auth.MULTI_USER:
|
||||||
return
|
return
|
||||||
for name, value in lists.items():
|
for name, value in lists.items():
|
||||||
@@ -264,18 +285,22 @@ def check_bundle_lists(**lists) -> None:
|
|||||||
|
|
||||||
|
|
||||||
# ---------- Request body size ----------
|
# ---------- Request body size ----------
|
||||||
# Generous enough for the biggest legitimate payload (an adventure export with
|
# The limit is generous enough for the largest legitimate payload, which is an
|
||||||
# thousands of actions), applied in every mode — no honest request comes close.
|
# adventure export holding thousands of actions. It applies in every mode, and no
|
||||||
|
# honest request approaches it.
|
||||||
|
|
||||||
MAX_BODY_BYTES = 2 * 1024 * 1024
|
MAX_BODY_BYTES = 2 * 1024 * 1024
|
||||||
MAX_IMPORT_BODY_BYTES = 20 * 1024 * 1024
|
MAX_IMPORT_BODY_BYTES = 20 * 1024 * 1024
|
||||||
|
|
||||||
|
|
||||||
class BodySizeLimitMiddleware:
|
class BodySizeLimitMiddleware:
|
||||||
"""Rejects oversized request bodies by declared Content-Length. Pure ASGI
|
"""Rejects oversized request bodies by their declared `Content-Length`.
|
||||||
(not BaseHTTPMiddleware) so SSE responses stream through untouched.
|
|
||||||
Chunked uploads without a length are refused — every real client of this
|
This is pure ASGI rather than `BaseHTTPMiddleware`, so SSE responses stream
|
||||||
API (browser fetch, curl with a file) sends Content-Length."""
|
through unchanged. A chunked upload with no length is refused, because every
|
||||||
|
real client of this API sends `Content-Length`, including browser fetch and
|
||||||
|
curl with a file.
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(self, app):
|
def __init__(self, app):
|
||||||
self.app = app
|
self.app = app
|
||||||
|
|||||||
+6
-6
@@ -106,12 +106,12 @@ class SecurityHeadersMiddleware:
|
|||||||
class ApiErrorMiddleware:
|
class ApiErrorMiddleware:
|
||||||
"""Counts failed API responses for the analytics dashboard.
|
"""Counts failed API responses for the analytics dashboard.
|
||||||
|
|
||||||
Here rather than in an exception handler because it sees what the client
|
This is middleware rather than an exception handler, because it observes
|
||||||
actually got: a 429 from a rate limiter, a 404 from routing, a 500 from a
|
what the client received. A 429 from a rate limiter, a 404 from routing, and
|
||||||
handler that never returned, all the same way. Pure ASGI for the same
|
a 500 from a handler that never returned all reach it the same way. It is
|
||||||
reason as the headers above — an SSE turn must not be buffered on its way
|
pure ASGI for the same reason as the headers above: an SSE turn must not be
|
||||||
out. Only /api is watched; a 404 on the SPA mount is a page load, not a
|
buffered on its way out. It watches `/api` only, because a 404 on the SPA
|
||||||
fault.
|
mount is a page load rather than a fault.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, app):
|
def __init__(self, app):
|
||||||
|
|||||||
+186
-151
@@ -1,20 +1,28 @@
|
|||||||
"""Phase 6 — auto summarization + embedding memory bank
|
"""Phase 6: automatic summarization and the embedding memory bank.
|
||||||
(per help.aidungeon.com/faq/the-memory-system).
|
|
||||||
|
|
||||||
After each turn, a fire-and-forget task (`run_post_turn`) runs with its own DB
|
This follows AI Dungeon's memory system. See
|
||||||
session:
|
help.aidungeon.com/faq/the-memory-system.
|
||||||
- every MEMORY_INTERVAL actions (starting at MEMORY_START), each uncovered
|
|
||||||
block of actions is summarized into a short "memory";
|
|
||||||
- every SUMMARY_INTERVAL actions, the Story Summary is rewritten folding in
|
|
||||||
the new memories (the user-edited text is always the base, never clobbered);
|
|
||||||
- new memories are embedded (OpenAI-compatible /v1/embeddings) and the bank
|
|
||||||
is evicted down to capacity ("forgotten" memories are kept for the UI).
|
|
||||||
|
|
||||||
At generation time, `retrieve_memories` embeds the recent story text and ranks
|
After each turn, `run_post_turn` runs as a fire-and-forget task with its own
|
||||||
the bank by cosine similarity; the top-K become the "Memories" context section.
|
database session. It does three things:
|
||||||
|
|
||||||
All AI calls here are best-effort: failures are logged (debug page) and retried
|
- Every `MEMORY_INTERVAL` actions, starting once the adventure reaches
|
||||||
on a later turn because the cursors only advance on success.
|
`MEMORY_START` actions, it summarizes each uncovered block of actions into a
|
||||||
|
short memory.
|
||||||
|
- Every `SUMMARY_INTERVAL` actions, it rewrites the story summary to include the
|
||||||
|
new memories. The rewrite always starts from the text the user edited and
|
||||||
|
never discards it.
|
||||||
|
- It embeds new memories through an OpenAI-compatible `/v1/embeddings` endpoint,
|
||||||
|
then evicts the bank down to its capacity. Evicted memories are marked as
|
||||||
|
forgotten and kept so that the UI can still show them.
|
||||||
|
|
||||||
|
When the app generates a turn, `retrieve_memories` embeds the recent story text
|
||||||
|
and ranks the bank by cosine similarity. The highest-ranked memories become the
|
||||||
|
Memories section of the context.
|
||||||
|
|
||||||
|
Every AI call in this module is best-effort. A failure is logged to the debug
|
||||||
|
page and retried on a later turn, because the cursors advance only after a call
|
||||||
|
succeeds.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
@@ -53,17 +61,19 @@ SUMMARY_SYSTEM_PROMPT = (
|
|||||||
|
|
||||||
# Adventures with a post-turn task currently running (single-process app).
|
# Adventures with a post-turn task currently running (single-process app).
|
||||||
_running: set[int] = set()
|
_running: set[int] = set()
|
||||||
# Strong refs to in-flight tasks — the event loop only keeps weak references,
|
# Strong references to tasks that are still running. The event loop holds only
|
||||||
# so a fire-and-forget task can otherwise be garbage-collected mid-run.
|
# weak references, so without this set a fire-and-forget task can be garbage
|
||||||
|
# collected before it finishes.
|
||||||
_tasks: set[asyncio.Task] = set()
|
_tasks: set[asyncio.Task] = set()
|
||||||
|
|
||||||
|
|
||||||
# BYOK-only by construction: both factories below take the user's own
|
# Both factories below use the user's own key by construction. They read the
|
||||||
# endpoint/key straight from Settings and never auth.DEMO_*, so summarization
|
# endpoint and key from `Settings` and never from `auth.DEMO_*`, so
|
||||||
# and embedding can't spend the shared demo key (their call sites are also
|
# summarization and embedding cannot spend the shared demo key. Their call sites
|
||||||
# skipped when using_demo). Don't "fix" this by passing a ProviderConfig in —
|
# are also skipped when `using_demo` is true.
|
||||||
# summary_model/embedding_model are free-form user input and are not on the
|
#
|
||||||
# demo whitelist.
|
# Do not change these to accept a `ProviderConfig`. `summary_model` and
|
||||||
|
# `embedding_model` are free-form user input and are not on the demo allowlist.
|
||||||
def summary_provider(settings: models.Settings) -> OpenAICompatibleProvider:
|
def summary_provider(settings: models.Settings) -> OpenAICompatibleProvider:
|
||||||
return OpenAICompatibleProvider(
|
return OpenAICompatibleProvider(
|
||||||
settings.endpoint_url,
|
settings.endpoint_url,
|
||||||
@@ -83,15 +93,15 @@ def embedding_provider(settings: models.Settings) -> OpenAICompatibleProvider:
|
|||||||
def set_vector(memory: models.Memory, vector: list[float] | None) -> None:
|
def set_vector(memory: models.Memory, vector: list[float] | None) -> None:
|
||||||
"""Store (or clear) a memory's embedding.
|
"""Store (or clear) a memory's embedding.
|
||||||
|
|
||||||
Every column that describes the vector moves together: `embedding_blob` is
|
This function updates both columns that describe the vector. The ranking
|
||||||
what the ranking reads and `embedded` is the flag everything else reads.
|
reads `embedding_blob`, and everything else reads the `embedded` flag.
|
||||||
Going through one function is what keeps them in step — and it is also the
|
Routing every write through one function keeps the two in step, and it makes
|
||||||
only place a stored vector can change, which is what makes the cache below
|
this the only place a stored vector changes. That is why the cache below can
|
||||||
safe to invalidate here and nowhere else.
|
be invalidated here and nowhere else.
|
||||||
|
|
||||||
The one caller that legitimately cannot come through here is the bulk
|
One caller cannot use this function: the bulk clear in
|
||||||
clear in `routers/settings.py` when the embedding model changes. It has to
|
`routers/settings.py` that runs when the embedding model changes. It sets
|
||||||
set the same two columns by hand; see the note there.
|
the same two columns directly. See the comment there.
|
||||||
"""
|
"""
|
||||||
memory.embedding_blob = None if vector is None else vectors.pack(vector)
|
memory.embedding_blob = None if vector is None else vectors.pack(vector)
|
||||||
memory.embedded = vector is not None
|
memory.embedded = vector is not None
|
||||||
@@ -102,31 +112,36 @@ def set_vector(memory: models.Memory, vector: list[float] | None) -> None:
|
|||||||
|
|
||||||
# ---------- The vector cache ----------
|
# ---------- The vector cache ----------
|
||||||
|
|
||||||
# adventure id -> {memory id: vector}, most-recently-used last.
|
# Maps an adventure id to a dict of memory id to vector, with the
|
||||||
|
# most recently used adventure last.
|
||||||
#
|
#
|
||||||
# Turns for one adventure arrive back to back, and the bank barely changes
|
# Turns for one adventure arrive one after another, and the bank changes little
|
||||||
# between them, so re-reading every vector each turn is the same 600 KB over
|
# between them. Reading every vector on each turn fetches the same 600 KB
|
||||||
# and over. Vectors are held as array("f") — 4 bytes a component, the same
|
# repeatedly. This cache stores vectors as `array("f")`, which uses 4 bytes per
|
||||||
# 6 KB the column holds. A list of Python floats would be eight times that.
|
# component and matches the 6 KB the column holds. A list of Python floats would
|
||||||
|
# use eight times as much.
|
||||||
#
|
#
|
||||||
# Correctness rests on two things. Anything that *changes* a vector goes
|
# Two rules keep the cache correct. Any code that changes a vector calls
|
||||||
# through set_vector, which drops that one entry. Anything that *removes* a
|
# `set_vector`, which removes that entry. Any code that removes a memory from
|
||||||
# memory from play — eviction, deletion, pruning, an edit clearing the vector —
|
# play removes it from the catalogue query below, and the next read discards
|
||||||
# takes it out of the catalogue query below, and entries missing from the
|
# entries that the catalogue no longer lists. Eviction, deletion, pruning, and
|
||||||
# catalogue are dropped on the next read. So nothing has to remember to call an
|
# an edit that clears the vector all work this way. No code path has to remember
|
||||||
# invalidate, which is the failure this design is chosen to avoid.
|
# to invalidate the cache, which is the error this design avoids.
|
||||||
#
|
#
|
||||||
# In-process, so it assumes one worker. That is what the deploy runs; a second
|
# The cache lives in the process, so it assumes one worker, which is what the
|
||||||
# worker would each keep their own copy and both would still be correct on
|
# deploy runs. With two workers, each keeps its own copy. Both stay correct
|
||||||
# eviction and deletion, but a vector rewritten by one could go stale in the
|
# about eviction and deletion, but a vector rewritten by one worker can remain
|
||||||
# other until that memory next leaves the catalogue.
|
# stale in the other until that memory leaves the catalogue.
|
||||||
_vector_cache: OrderedDict[int, dict[int, array]] = OrderedDict()
|
_vector_cache: OrderedDict[int, dict[int, array]] = OrderedDict()
|
||||||
VECTOR_CACHE_ADVENTURES = 8 # ~600 KB each at a 100-memory bank
|
VECTOR_CACHE_ADVENTURES = 8 # ~600 KB each at a 100-memory bank
|
||||||
|
|
||||||
|
|
||||||
def forget_cached_vectors(adventure_id: int) -> None:
|
def forget_cached_vectors(adventure_id: int) -> None:
|
||||||
"""Drop an adventure's cached vectors. Only needed when the adventure
|
"""Drops an adventure's cached vectors.
|
||||||
itself goes away — everything else self-corrects (see above)."""
|
|
||||||
|
Call this only when the adventure itself is deleted. Every other case
|
||||||
|
corrects itself, as described in the comment above.
|
||||||
|
"""
|
||||||
_vector_cache.pop(adventure_id, None)
|
_vector_cache.pop(adventure_id, None)
|
||||||
|
|
||||||
|
|
||||||
@@ -157,27 +172,27 @@ def _vectors_for(db: Session, adventure_id: int, ids: list[int]) -> dict[int, ar
|
|||||||
def forget_node(db: Session, adventure: models.Adventure, action: models.Action) -> int:
|
def forget_node(db: Session, adventure: models.Adventure, action: models.Action) -> int:
|
||||||
"""Withdraw what a node produced, because the node is being removed.
|
"""Withdraw what a node produced, because the node is being removed.
|
||||||
|
|
||||||
Call it before deleting `action` (undo, delete-an-action). A memory hangs
|
Call this before deleting `action`, which undo and the delete-action
|
||||||
off the node whose block it ends on, so "which memories described this?" is
|
endpoint both do. A memory attaches to the node its block ends on, so
|
||||||
a lookup on `(branch_id, depth)` rather than a scan for rows whose covered
|
finding the memories that describe a node is a lookup on `(branch_id,
|
||||||
range has fallen off the end of the story — which is what
|
depth)`. The earlier `prune_dangling_memories` instead scanned for rows
|
||||||
`prune_dangling_memories` did, and it could only ever notice the damage
|
whose covered range no longer existed, so it could only detect the problem
|
||||||
after the fact.
|
after it occurred.
|
||||||
|
|
||||||
Discarding the memory is half of it. The stretch of story it covered is
|
Deleting the memory is half the work. The stretch of story it covered still
|
||||||
still behind the cursors, so without a rewind those actions read as
|
sits behind the cursors. Without a rewind, those actions count as
|
||||||
summarized with nothing describing them, silently, for the rest of the
|
summarized while nothing describes them, and nothing reports the problem for
|
||||||
adventure. `source_start` is where that stretch began; the anchor goes to
|
the rest of the adventure. `source_start` records where that stretch began,
|
||||||
the node before it, which is a depth whether or not anything still sits
|
so the anchor moves to the node before it. That depth is valid whether or
|
||||||
there.
|
not a node still occupies it.
|
||||||
|
|
||||||
The opening node is the one exception, because migration 62 parked the
|
The opening node is the one exception, because migration 62 placed the whole
|
||||||
whole pre-coordinate bank on it — see the comment on `lineage.ROOT_DEPTH`.
|
pre-coordinate bank on it. See the comment on `lineage.ROOT_DEPTH`.
|
||||||
|
|
||||||
Returns how many memories were withdrawn.
|
Returns the number of memories withdrawn.
|
||||||
"""
|
"""
|
||||||
if action.branch_id is None or action.depth is None:
|
if action.branch_id is None or action.depth is None:
|
||||||
return 0 # a pre-tree row: no path contains it, so nothing hangs off it
|
return 0 # A pre-tree row. No path contains it, so nothing refers to it.
|
||||||
doomed = (
|
doomed = (
|
||||||
db.query(models.Memory)
|
db.query(models.Memory)
|
||||||
.filter(
|
.filter(
|
||||||
@@ -188,15 +203,16 @@ def forget_node(db: Session, adventure: models.Adventure, action: models.Action)
|
|||||||
.all()
|
.all()
|
||||||
)
|
)
|
||||||
if action.depth == lineage.ROOT_DEPTH:
|
if action.depth == lineage.ROOT_DEPTH:
|
||||||
# The opening node is special, and only for memories that describe no
|
# Special case for the opening node, and only for memories that
|
||||||
# stretch of story. Migration 62 parked every memory written before
|
# describe no stretch of story. Migration 62 placed every memory written
|
||||||
# memories had coordinates at depth 0 — that was the choice that took
|
# before memories had coordinates at depth 0. That choice preserved
|
||||||
# nothing away from anybody, but it also collected them all onto one
|
# every memory, but it also placed them all on one node, so withdrawing
|
||||||
# node, so withdrawing that node would retire a player's whole bank in
|
# that node would delete a player's entire bank in one action.
|
||||||
# a single click. A memory with no `source_start` was typed (or
|
#
|
||||||
# migrated), describes nothing that can fall off the end, and so has
|
# A memory with no `source_start` was either typed by the player or
|
||||||
# nothing to be withdrawn *from*: it stays. A summary that genuinely
|
# migrated. It describes no actions, so no deletion can invalidate it,
|
||||||
# ends here is still withdrawn, because the text it describes is going.
|
# and it stays. A memory that genuinely summarizes a block ending here is
|
||||||
|
# still withdrawn, because the text it describes is being deleted.
|
||||||
doomed = [m for m in doomed if m.source_start is not None]
|
doomed = [m for m in doomed if m.source_start is not None]
|
||||||
if not doomed:
|
if not doomed:
|
||||||
return 0
|
return 0
|
||||||
@@ -217,11 +233,19 @@ async def retrieve_memories(
|
|||||||
update_stats: bool,
|
update_stats: bool,
|
||||||
exclude_action_id: int | None = None,
|
exclude_action_id: int | None = None,
|
||||||
) -> dict | None:
|
) -> dict | None:
|
||||||
"""Returns {"used": [{id, text, similarity, pinned}], "error": str|None},
|
"""Returns the memories to inject, or None when the bank is off.
|
||||||
or None when the memory bank is off for this adventure. `update_stats`
|
|
||||||
bumps use counters (real turns only, not Insights dry runs).
|
The result is a dict of the form
|
||||||
`exclude_action_id` drops the action being retried from the similarity
|
`{"used": [{id, text, similarity, pinned}], "error": str | None}`. It is
|
||||||
query, so the discarded attempt can't steer which memories come back."""
|
None when the memory bank is disabled for this adventure.
|
||||||
|
|
||||||
|
Set `update_stats` to True to increment the use counters. Only real turns
|
||||||
|
should do this, not the dry runs that Insights performs.
|
||||||
|
|
||||||
|
`exclude_action_id` removes the action being retried from the similarity
|
||||||
|
query, so that a discarded attempt cannot influence which memories are
|
||||||
|
returned.
|
||||||
|
"""
|
||||||
if not adventure.memory_bank_enabled:
|
if not adventure.memory_bank_enabled:
|
||||||
return None
|
return None
|
||||||
if not settings.embedding_model.strip():
|
if not settings.embedding_model.strip():
|
||||||
@@ -230,16 +254,17 @@ async def retrieve_memories(
|
|||||||
if db is None:
|
if db is None:
|
||||||
return {"used": [], "error": None}
|
return {"used": [], "error": None}
|
||||||
|
|
||||||
# Which memories are in play, and nothing else about them. This used to
|
# Select which memories are in play, and nothing else about them. This code
|
||||||
# walk adventure.memories, which loaded every row of the bank *including
|
# used to walk `adventure.memories`, which loaded every row of the bank,
|
||||||
# its vector* — ~31 KB a memory, three megabytes a turn, 96% of everything
|
# including its vector. That cost about 31 KB per memory and about 3 MB per
|
||||||
# a turn read. Two ids and a flag per row is about eight bytes.
|
# turn, which was 96% of everything a turn read. An id and a flag come to
|
||||||
|
# about eight bytes per row.
|
||||||
#
|
#
|
||||||
# The branch clause is the *whole* lineage here, not the window the story
|
# The branch clause uses the whole lineage here rather than the window the
|
||||||
# is read through: retrieval is long-range recall, and a memory of what
|
# story is read through. Retrieval exists to recall events from far back in
|
||||||
# happened forty turns ago is exactly what it exists to find. It stays
|
# the story, such as what happened forty turns ago. The full lineage stays
|
||||||
# affordable because memories are sparse — one per six actions — so the
|
# affordable because memories are sparse, at roughly one per six actions, so
|
||||||
# ancestry of even a heavily forked story returns tens of tiny rows.
|
# even a heavily forked story returns only tens of small rows.
|
||||||
catalogue = db.execute(
|
catalogue = db.execute(
|
||||||
select(models.Memory.id, models.Memory.pinned).where(
|
select(models.Memory.id, models.Memory.pinned).where(
|
||||||
models.Memory.adventure_id == adventure.id,
|
models.Memory.adventure_id == adventure.id,
|
||||||
@@ -273,8 +298,9 @@ async def retrieve_memories(
|
|||||||
key=lambda row: row[0],
|
key=lambda row: row[0],
|
||||||
reverse=True,
|
reverse=True,
|
||||||
)
|
)
|
||||||
# Pinned memories are always used and count toward top_k, so the injected
|
# Pinned memories are always used, and they count toward `top_k`, so the
|
||||||
# set never exceeds the configured budget (unless pinned alone exceed it).
|
# injected set stays within the budget unless the pinned memories alone
|
||||||
|
# exceed it.
|
||||||
top_k = max(1, settings.memory_top_k)
|
top_k = max(1, settings.memory_top_k)
|
||||||
used = [row for row in scored if row[2]]
|
used = [row for row in scored if row[2]]
|
||||||
remaining = max(0, top_k - len(used))
|
remaining = max(0, top_k - len(used))
|
||||||
@@ -283,7 +309,7 @@ async def retrieve_memories(
|
|||||||
if not used:
|
if not used:
|
||||||
return {"used": [], "error": None}
|
return {"used": [], "error": None}
|
||||||
|
|
||||||
# Only now, for at most top_k rows, is the text worth fetching.
|
# Fetch the text only now, and only for the `top_k` rows that were chosen.
|
||||||
used_ids = [memory_id for _, memory_id, _ in used]
|
used_ids = [memory_id for _, memory_id, _ in used]
|
||||||
texts = dict(
|
texts = dict(
|
||||||
db.execute(
|
db.execute(
|
||||||
@@ -293,9 +319,10 @@ async def retrieve_memories(
|
|||||||
)
|
)
|
||||||
|
|
||||||
if update_stats:
|
if update_stats:
|
||||||
# synchronize_session=False: nothing in this request reads the counters
|
# Pass `synchronize_session=False` because nothing in this request
|
||||||
# back, and matching the UPDATE against loaded objects would mean having
|
# reads the counters back. Matching the UPDATE against loaded objects
|
||||||
# loaded them, which is the cost this whole path exists to avoid.
|
# would require loading those objects, which is the cost this code path
|
||||||
|
# exists to avoid.
|
||||||
db.execute(
|
db.execute(
|
||||||
update(models.Memory)
|
update(models.Memory)
|
||||||
.where(models.Memory.id.in_(used_ids))
|
.where(models.Memory.id.in_(used_ids))
|
||||||
@@ -343,14 +370,16 @@ async def run_post_turn(adventure_id: int) -> None:
|
|||||||
)
|
)
|
||||||
if settings is None:
|
if settings is None:
|
||||||
return
|
return
|
||||||
# No cursor clamp here any more. Undo can leave the story shorter than
|
# This code no longer clamps the cursors. Undo can leave the story
|
||||||
# the mark, and a *position* past the end of the list was a stalled
|
# shorter than the mark. When the mark was a position, a value past the
|
||||||
# pass until the story grew back past it — hence a clamp on every
|
# end of the list stalled the pass until the story grew back, so every
|
||||||
# post-turn run, which had its own trap (clamping to the settled count
|
# post-turn run clamped it. That clamp introduced its own error, because
|
||||||
# rewound a caught-up adventure a step and re-covered an action). An
|
# clamping to the settled count rewound a caught-up adventure by one
|
||||||
# anchor past the tip is not a broken value: `settled_after` just
|
# step and covered an action twice.
|
||||||
# reports nothing to do, and the story growing back past it resumes
|
#
|
||||||
# exactly where it left off.
|
# An anchor past the tip is not an invalid value. `settled_after`
|
||||||
|
# reports that there is nothing to do, and once the story grows past the
|
||||||
|
# anchor the pass resumes where it stopped.
|
||||||
if adventure.auto_summarize:
|
if adventure.auto_summarize:
|
||||||
await _create_due_memories(adventure, settings, db)
|
await _create_due_memories(adventure, settings, db)
|
||||||
await _update_story_summary(adventure, settings, db)
|
await _update_story_summary(adventure, settings, db)
|
||||||
@@ -367,16 +396,17 @@ async def _create_due_memories(
|
|||||||
) -> None:
|
) -> None:
|
||||||
provider = summary_provider(settings)
|
provider = summary_provider(settings)
|
||||||
for _ in range(MAX_MEMORIES_PER_RUN):
|
for _ in range(MAX_MEMORIES_PER_RUN):
|
||||||
# Re-read each pass: a memory just committed doesn't change the story,
|
# Re-read the anchor on every pass. Committing a memory does not change
|
||||||
# but this loop is the only thing that moves the anchor, so both
|
# the story, but this loop is the only code that moves the anchor, so
|
||||||
# numbers have to be current.
|
# both numbers must be current.
|
||||||
anchor = cursors.MEMORY.depth(db, adventure)
|
anchor = cursors.MEMORY.depth(db, adventure)
|
||||||
if history.count_after(adventure, anchor) < MEMORY_INTERVAL:
|
if history.count_after(adventure, anchor) < MEMORY_INTERVAL:
|
||||||
return # no full block of story past the mark
|
return # No full block of story sits past the mark.
|
||||||
if history.count(adventure) < MEMORY_START:
|
if history.count(adventure) < MEMORY_START:
|
||||||
return # ...and the adventure is too short to have started at all
|
return # The adventure is too short to have started summarizing.
|
||||||
# (that order on purpose: the common answer is "nothing due", and the
|
# The order of those two checks is deliberate. The usual answer is that
|
||||||
# first question answers it without asking how long the story is)
|
# no memory is due, and the first check settles that without measuring
|
||||||
|
# the length of the whole story.
|
||||||
block = history.after(adventure, anchor, MEMORY_INTERVAL)
|
block = history.after(adventure, anchor, MEMORY_INTERVAL)
|
||||||
if len(block) < MEMORY_INTERVAL:
|
if len(block) < MEMORY_INTERVAL:
|
||||||
return
|
return
|
||||||
@@ -386,7 +416,8 @@ async def _create_due_memories(
|
|||||||
MEMORY_SYSTEM_PROMPT, f"Story excerpt:\n\n{excerpt}\n\nMemory:"
|
MEMORY_SYSTEM_PROMPT, f"Story excerpt:\n\n{excerpt}\n\nMemory:"
|
||||||
)
|
)
|
||||||
except ProviderError:
|
except ProviderError:
|
||||||
return # logged in the debug page; cursor unchanged → retried next turn
|
return # Logged on the debug page. The cursor is unchanged, so the
|
||||||
|
# next turn retries this block.
|
||||||
if not text:
|
if not text:
|
||||||
return
|
return
|
||||||
memory = models.Memory(
|
memory = models.Memory(
|
||||||
@@ -395,11 +426,11 @@ async def _create_due_memories(
|
|||||||
source_start=block[0].depth,
|
source_start=block[0].depth,
|
||||||
source_end=block[-1].depth,
|
source_end=block[-1].depth,
|
||||||
)
|
)
|
||||||
# Hang it off the node it summarised, so a fork inherits the memories of
|
# Attach the memory to the node it summarizes, so that a fork inherits
|
||||||
# the path it forked from and nothing else — and move the mark to that
|
# the memories of the path it forked from and no others. Then move the
|
||||||
# same node. The two are one statement about where this pass has got to,
|
# mark to that same node. Both values record how far this pass has
|
||||||
# and writing them from the same row is what keeps them in step however
|
# reached, and taking them from one row keeps them in step even when the
|
||||||
# gappy the depths underneath are.
|
# depths have gaps.
|
||||||
tree.attach_memory(memory, block[-1])
|
tree.attach_memory(memory, block[-1])
|
||||||
db.add(memory)
|
db.add(memory)
|
||||||
cursors.MEMORY.anchor_at(adventure, block[-1])
|
cursors.MEMORY.anchor_at(adventure, block[-1])
|
||||||
@@ -413,18 +444,19 @@ async def _update_story_summary(
|
|||||||
uncovered = history.count_after(adventure, anchor)
|
uncovered = history.count_after(adventure, anchor)
|
||||||
if uncovered < SUMMARY_INTERVAL:
|
if uncovered < SUMMARY_INTERVAL:
|
||||||
return
|
return
|
||||||
# Where the summary will stand once this run succeeds. Read before the AI
|
# Where the summary stands once this run succeeds. Read this before the AI
|
||||||
# call, not after: the mark is the end of the story as this pass saw it,
|
# call rather than after it. The mark records the end of the story as this
|
||||||
# and a turn landing meanwhile must not be quietly claimed as read.
|
# pass saw it, and a turn that arrives during the call must not be counted
|
||||||
|
# as read.
|
||||||
caught_up = history.newest(adventure)
|
caught_up = history.newest(adventure)
|
||||||
if caught_up is None:
|
if caught_up is None:
|
||||||
return
|
return
|
||||||
|
|
||||||
# Fold in the memories of the stretch the summary has not read — every
|
# Include the memories for the stretch that the summary has not read, which
|
||||||
# memory hanging off a node past the anchor. Both marks and every memory
|
# means every memory attached to a node past the anchor. The marks and the
|
||||||
# are now depths on one path, so there is no translation between coordinate
|
# memories are now depths on one path, so no coordinate conversion remains.
|
||||||
# systems left to get wrong. Falls back to raw story text if memory
|
# If memory creation has fallen behind, for example because the last attempt
|
||||||
# creation is lagging (e.g. it just failed).
|
# failed, this falls back to the raw story text.
|
||||||
new_events = db.execute(
|
new_events = db.execute(
|
||||||
select(models.Memory.text)
|
select(models.Memory.text)
|
||||||
.where(
|
.where(
|
||||||
@@ -462,15 +494,17 @@ async def _update_story_summary(
|
|||||||
async def _embed_pending(
|
async def _embed_pending(
|
||||||
adventure: models.Adventure, settings: models.Settings, db: Session
|
adventure: models.Adventure, settings: models.Settings, db: Session
|
||||||
) -> None:
|
) -> None:
|
||||||
# A query, not a walk of adventure.memories: this ran every turn and pulled
|
# Use a query rather than walking `adventure.memories`. That walk ran on
|
||||||
# the whole bank's vectors to find the handful that had none.
|
# every turn and loaded the whole bank's vectors in order to find the few
|
||||||
|
# rows with none.
|
||||||
#
|
#
|
||||||
# No branch clause, deliberately, here and in the eviction below. Being
|
# Neither this query nor the eviction below applies a branch clause, and
|
||||||
# embedded is a fact about the row, not about the path being played:
|
# that is deliberate. Whether a row is embedded is a fact about the row, not
|
||||||
# skipping a sibling's memories would only mean embedding them later, at
|
# about the path being played. Skipping a sibling branch's memories would
|
||||||
# the moment somebody switched branches and wanted them ranked. Capacity is
|
# only postpone the work until someone switched branches and needed them
|
||||||
# the same — the bank belongs to the adventure, and evicting the memories
|
# ranked. Capacity works the same way. The bank belongs to the adventure,
|
||||||
# of a story nobody is reading is exactly the right thing to evict first.
|
# and the memories of a branch nobody is reading are the right ones to evict
|
||||||
|
# first.
|
||||||
pending = (
|
pending = (
|
||||||
db.query(models.Memory)
|
db.query(models.Memory)
|
||||||
.filter(
|
.filter(
|
||||||
@@ -496,9 +530,10 @@ async def _embed_pending(
|
|||||||
def _evict_over_capacity(
|
def _evict_over_capacity(
|
||||||
adventure: models.Adventure, settings: models.Settings, db: Session
|
adventure: models.Adventure, settings: models.Settings, db: Session
|
||||||
) -> None:
|
) -> None:
|
||||||
# Counting and ranking are both things the database does without sending
|
# The database performs both the count and the ranking, and returns neither
|
||||||
# anything back. Walking adventure.memories to count them fetched every
|
# the rows nor the vectors. Counting by walking `adventure.memories` fetched
|
||||||
# vector in the bank, every turn, whether or not anything was over capacity.
|
# every vector in the bank on every turn, whether or not the bank was over
|
||||||
|
# capacity.
|
||||||
in_this_bank = (models.Memory.adventure_id == adventure.id,
|
in_this_bank = (models.Memory.adventure_id == adventure.id,
|
||||||
models.Memory.forgotten.is_(False))
|
models.Memory.forgotten.is_(False))
|
||||||
active = db.execute(
|
active = db.execute(
|
||||||
@@ -507,23 +542,23 @@ def _evict_over_capacity(
|
|||||||
overflow = active - max(1, settings.memory_bank_capacity)
|
overflow = active - max(1, settings.memory_bank_capacity)
|
||||||
if overflow <= 0:
|
if overflow <= 0:
|
||||||
return
|
return
|
||||||
# Least recently touched goes first, and how often it was used only breaks
|
# Evict the least recently used memory first, and use the use count only to
|
||||||
# a tie. The other way round — use_count first — shut the bank. A memory
|
# break ties.
|
||||||
# written this turn has never been used, so once every survivor had been
|
|
||||||
# retrieved even once the newborn was the lowest row in the bank and was
|
|
||||||
# evicted by the same post-turn run that wrote it, in the pass right after
|
|
||||||
# the one that embedded it. Counts only ever go up, so that state never
|
|
||||||
# ends: the bank an adventure happened to hold when it first filled is the
|
|
||||||
# bank it keeps for good, and everything the story does afterwards is
|
|
||||||
# summarized, marked forgotten, and never ranked.
|
|
||||||
#
|
#
|
||||||
# Recency does not have that hole, because a new memory carries the newest
|
# Ordering by use count first froze the bank. A memory written on this turn
|
||||||
# timestamp there is — it is the safest row in the bank rather than the
|
# has never been used, so once every other memory had been retrieved at
|
||||||
# most doomed, and it gets the whole span until something outlives it to
|
# least once, the new memory held the lowest count in the bank. The same
|
||||||
# prove itself. Little is given up by demoting the count: a memory that is
|
# post-turn run that wrote it then evicted it, one pass after embedding it.
|
||||||
# genuinely used stays recently-used by being retrieved, so the two only
|
# Use counts only increase, so the bank never recovered. An adventure kept
|
||||||
# disagree about memories that mattered once and have not been wanted
|
# whatever memories it held when the bank first filled, and every later
|
||||||
# since, which is what a full bank should be dropping anyway.
|
# memory was summarized, marked as forgotten, and never ranked.
|
||||||
|
#
|
||||||
|
# Ordering by recency avoids that. A new memory carries the newest
|
||||||
|
# timestamp, so it is the last row to be evicted rather than the first, and
|
||||||
|
# it remains until other memories are used. Demoting the use count costs
|
||||||
|
# little, because retrieving a useful memory also makes it recent. The two
|
||||||
|
# orderings differ only for memories that were used once and have not been
|
||||||
|
# retrieved since, which are the rows a full bank should evict.
|
||||||
doomed = db.execute(
|
doomed = db.execute(
|
||||||
select(models.Memory.id)
|
select(models.Memory.id)
|
||||||
.where(*in_this_bank, models.Memory.pinned.is_(False))
|
.where(*in_this_bank, models.Memory.pinned.is_(False))
|
||||||
@@ -534,7 +569,7 @@ def _evict_over_capacity(
|
|||||||
.limit(overflow)
|
.limit(overflow)
|
||||||
).scalars().all()
|
).scalars().all()
|
||||||
if not doomed:
|
if not doomed:
|
||||||
return # every active memory is pinned; capacity yields to the pins
|
return # Every active memory is pinned, so the pins override capacity.
|
||||||
db.execute(
|
db.execute(
|
||||||
update(models.Memory)
|
update(models.Memory)
|
||||||
.where(models.Memory.id.in_(doomed))
|
.where(models.Memory.id.in_(doomed))
|
||||||
@@ -542,6 +577,6 @@ def _evict_over_capacity(
|
|||||||
.execution_options(synchronize_session=False)
|
.execution_options(synchronize_session=False)
|
||||||
)
|
)
|
||||||
db.commit()
|
db.commit()
|
||||||
# The bulk UPDATE went around any loaded objects, so anything still holding
|
# The bulk UPDATE bypassed the loaded objects, so code that still holds the
|
||||||
# the collection would see the evicted memories as active.
|
# collection would otherwise see the evicted memories as active.
|
||||||
db.expire(adventure, ["memories"])
|
db.expire(adventure, ["memories"])
|
||||||
|
|||||||
+384
-349
File diff suppressed because it is too large
Load Diff
+247
-223
@@ -15,15 +15,18 @@ def utcnow() -> datetime:
|
|||||||
|
|
||||||
|
|
||||||
class User(Base):
|
class User(Base):
|
||||||
"""Phase 8 — optional accounts.
|
"""Phase 8: optional accounts.
|
||||||
|
|
||||||
Three kinds of rows share this table:
|
Three kinds of row share this table:
|
||||||
- the "local user" (email NULL, is_guest False): auto-created in
|
|
||||||
single-user/local mode; owns everything a pre-Phase-8 DB had;
|
- The local user has a NULL email and `is_guest` set to False. Single-user
|
||||||
- guests (email NULL, is_guest True): created on first visit in
|
mode creates this row automatically. It owns everything that a database
|
||||||
multi-user mode, identified only by their session cookie;
|
from before Phase 8 contained.
|
||||||
- registered users (email set): a guest upgraded in place, so their
|
- Guests have a NULL email and `is_guest` set to True. Multi-user mode
|
||||||
data survives registration with no re-parenting.
|
creates one on a visitor's first visit and identifies it only by the
|
||||||
|
session cookie.
|
||||||
|
- Registered users have an email. Registration upgrades a guest row in
|
||||||
|
place, so the guest's data survives without being reassigned.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
__tablename__ = "users"
|
__tablename__ = "users"
|
||||||
@@ -64,18 +67,20 @@ class Scenario(Base):
|
|||||||
authors_note: Mapped[str] = mapped_column(Text, default="")
|
authors_note: Mapped[str] = mapped_column(Text, default="")
|
||||||
ai_instructions: Mapped[str] = mapped_column(Text, default="")
|
ai_instructions: Mapped[str] = mapped_column(Text, default="")
|
||||||
tags: Mapped[str] = mapped_column(String(500), default="")
|
tags: Mapped[str] = mapped_column(String(500), default="")
|
||||||
# Cover art. Either an external "https://…" URL or an inline
|
# Cover art. The value is either an "https://" URL or an inline
|
||||||
# "data:image/…;base64,…" URI (the editor downscales uploads before storing
|
# "data:image/...;base64,..." URI. The editor downscales uploads before
|
||||||
# one). Empty means the UI falls back to an emoji sigil or generated art.
|
# storing them. An empty value tells the UI to fall back to an emoji sigil
|
||||||
# Kept in the row rather than on disk because Render's free tier has no
|
# or to generated art. The image is stored in the row rather than on disk,
|
||||||
# persistent volume, and it makes export bundles self-contained.
|
# because Render's free tier provides no persistent volume. Storing it here
|
||||||
|
# also keeps export bundles self-contained.
|
||||||
image: Mapped[str] = mapped_column(Text, default="")
|
image: Mapped[str] = mapped_column(Text, default="")
|
||||||
# A single emoji or glyph used when there's no `image` — cheap art for
|
# A single emoji or glyph, used when `image` is empty. This is a separate
|
||||||
# scenarios nobody wants to find a picture for. Separate from `image`
|
# column because the value is a character rather than a location, so
|
||||||
# because it's a character, not a locator: no fetch, no cache, no bytes.
|
# nothing needs to fetch or cache it.
|
||||||
icon: Mapped[str] = mapped_column(String(16), default="")
|
icon: Mapped[str] = mapped_column(String(16), default="")
|
||||||
# Phase 12: RPG world-state template — stat definitions (bands, rules) and
|
# Phase 12: the RPG world-state template. It holds stat definitions, which
|
||||||
# milestones. NULL/empty means this scenario has no RPG layer.
|
# include bands and rules, and milestones. A NULL or empty value means the
|
||||||
|
# scenario has no RPG layer.
|
||||||
stat_schema: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
stat_schema: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||||
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
|
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
|
||||||
updated_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow, onupdate=utcnow)
|
updated_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow, onupdate=utcnow)
|
||||||
@@ -113,31 +118,36 @@ class Adventure(Base):
|
|||||||
# Phase 6: opt-in per adventure (extra AI calls)
|
# Phase 6: opt-in per adventure (extra AI calls)
|
||||||
auto_summarize: Mapped[bool] = mapped_column(Boolean, default=False)
|
auto_summarize: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
memory_bank_enabled: Mapped[bool] = mapped_column(Boolean, default=False)
|
memory_bank_enabled: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
# LEGACY (Phase 6): how many actions had been folded into memories / the
|
# Legacy columns from Phase 6. Each holds a count of the actions that were
|
||||||
# story summary, as a *position* in the story. Unread since SP3, and
|
# folded into the memories or the story summary, expressed as a position in
|
||||||
# unwritten except by a v1 import which is handed one; kept for one release
|
# the story. Nothing has read them since SP3, and nothing writes them except
|
||||||
# so a rollback resumes from a real number, and dropped in SP8 beside
|
# a v1 import. They remain for one release so that a rollback resumes from a
|
||||||
# `actions.index`. The live mark is the anchor pair below.
|
# real number. SP8 drops them along with `actions.index`. The columns below
|
||||||
|
# hold the marks that this code actually uses.
|
||||||
memory_cursor: Mapped[int] = mapped_column(Integer, default=0)
|
memory_cursor: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
summary_cursor: Mapped[int] = mapped_column(Integer, default=0)
|
summary_cursor: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
# Phase 14, SP3: the same two marks as nodes — (branch, depth) of the last
|
# Phase 14, SP3: the same two marks expressed as coordinates. Each pair
|
||||||
# action each pass covered. A position slides when an action in front of it
|
# holds the branch and depth of the last action that pass covered. A
|
||||||
# is deleted and silently starts covering one it has never read; a depth
|
# position moves when an action in front of it is deleted, so the mark
|
||||||
# does not move, because it is a coordinate along a path rather than an
|
# silently starts covering an action it never read. A depth is a coordinate
|
||||||
# offset into a list. NO_DEPTH (-1) is "nothing covered yet", so the first
|
# along a path, so deleting an action does not move it. NO_DEPTH, which is
|
||||||
# block needs no special case. Plain integers, not foreign keys, for the
|
# -1, means that nothing is covered yet, so the first block needs no special
|
||||||
# same reason `head_branch_id` below is one. See `context/cursors.py`.
|
# case. These are plain integers rather than foreign keys, for the reason
|
||||||
|
# given on `head_branch_id` below. See `context/cursors.py`.
|
||||||
memory_cursor_branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
memory_cursor_branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
memory_cursor_depth: Mapped[int] = mapped_column(Integer, default=-1)
|
memory_cursor_depth: Mapped[int] = mapped_column(Integer, default=-1)
|
||||||
summary_cursor_branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
summary_cursor_branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
summary_cursor_depth: Mapped[int] = mapped_column(Integer, default=-1)
|
summary_cursor_depth: Mapped[int] = mapped_column(Integer, default=-1)
|
||||||
# Phase 14: where the story is being played — which branch, and the depth of
|
# Phase 14: where the story is being played. `head_branch_id` names the
|
||||||
# its newest node. Deliberately NOT a ForeignKey: branches.adventure_id
|
# branch, and `head_depth` gives the depth of its newest node.
|
||||||
# already points this way, and a second constraint back would make the two
|
#
|
||||||
# tables a cycle that create_all cannot order (the fix for that is
|
# `head_branch_id` is deliberately not a ForeignKey. `branches.adventure_id`
|
||||||
# use_alter, which SQLite has no ALTER for). It is a cache of a pointer, and
|
# already points from branches to adventures, so a constraint in this
|
||||||
# `tree.head_branch` treats a head naming a branch that no longer exists as
|
# direction would make the two tables a cycle that `create_all` cannot
|
||||||
# a bug to recover from rather than a state to honour.
|
# order. The usual fix is `use_alter`, which needs an ALTER statement that
|
||||||
|
# SQLite does not provide. The column caches a pointer, and
|
||||||
|
# `tree.head_branch` treats a head that names a missing branch as a bug to
|
||||||
|
# recover from rather than a state to preserve.
|
||||||
head_branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
head_branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
# The depth of the tip, so the next node is always head_depth + 1.
|
# The depth of the tip, so the next node is always head_depth + 1.
|
||||||
# NO_DEPTH (-1) for an adventure with no actions yet.
|
# NO_DEPTH (-1) for an adventure with no actions yet.
|
||||||
@@ -149,12 +159,12 @@ class Adventure(Base):
|
|||||||
story_cards: Mapped[list["StoryCard"]] = relationship(
|
story_cards: Mapped[list["StoryCard"]] = relationship(
|
||||||
back_populates="adventure", cascade="all, delete-orphan"
|
back_populates="adventure", cascade="all, delete-orphan"
|
||||||
)
|
)
|
||||||
# Every action of the adventure — that is, every *branch's*. Not the story
|
# Every action in the adventure, across all branches. This collection is
|
||||||
# being played, and re-ordering it by depth would not make it one: the
|
# the tree, not the story being played. Ordering it by depth does not make
|
||||||
# collection is the tree, and a path is a selection out of it. Anything
|
# it a story, because a path is a selection out of the tree. Code that shows
|
||||||
# showing a reader a story goes through `context.history`, which goes
|
# a story to a reader goes through `context.history`, which applies the
|
||||||
# through the branch clause. What is left here is ownership and the
|
# branch clause. This relationship exists for ownership and for the
|
||||||
# delete-orphan cascade, which are facts about the adventure.
|
# delete-orphan cascade.
|
||||||
actions: Mapped[list["Action"]] = relationship(
|
actions: Mapped[list["Action"]] = relationship(
|
||||||
back_populates="adventure",
|
back_populates="adventure",
|
||||||
cascade="all, delete-orphan",
|
cascade="all, delete-orphan",
|
||||||
@@ -173,24 +183,25 @@ class Adventure(Base):
|
|||||||
|
|
||||||
|
|
||||||
class Branch(Base):
|
class Branch(Base):
|
||||||
"""Phase 14 — one path through an adventure's story tree.
|
"""Phase 14: one path through an adventure's story tree.
|
||||||
|
|
||||||
A branch does not own a copy of the story: it holds the nodes played on it
|
A branch does not own a copy of the story. It holds the nodes played on it,
|
||||||
and *borrows* everything before its fork point from its ancestors. Reading
|
and it inherits everything before its fork point from its ancestors. Reading
|
||||||
branch C means reading C's nodes, plus B's up to where C left it, plus A's
|
branch C means reading C's nodes, then B's nodes up to the depth where C
|
||||||
up to where B left it — which is what `lineage` spells out, so a read is an
|
forked, then A's nodes up to the depth where B forked. The `lineage` column
|
||||||
OR-clause per entry instead of a walk up parent pointers.
|
records that list, so a read becomes one OR clause per entry instead of a
|
||||||
|
walk up parent pointers.
|
||||||
|
|
||||||
Until forking ships there is exactly one root branch per adventure and
|
Until forking ships, each adventure has one root branch and every node
|
||||||
every node hangs off it. That is not a half-migrated state: a linear story
|
belongs to it. This is not a partly migrated state. A linear story is a tree
|
||||||
*is* a tree with one branch, which is why writing these columns changes
|
with one branch, which is why writing these columns changes nothing that a
|
||||||
nothing anyone can observe.
|
reader can observe.
|
||||||
|
|
||||||
No ORM relationships on purpose. `actions.branch_id` and `memories
|
This class defines no ORM relationships, by design. `actions.branch_id` and
|
||||||
.branch_id` carry ON DELETE CASCADE, so the database removes a deleted
|
`memories.branch_id` both use ON DELETE CASCADE, so the database removes a
|
||||||
branch's nodes; a relationship would have SQLAlchemy load them all to do
|
deleted branch's nodes. A relationship would make SQLAlchemy load those rows
|
||||||
the same thing, and loading every action of a branch is the exact cost the
|
first, and loading every action of a branch is what the windowed reads exist
|
||||||
windowed reads exist to avoid.
|
to avoid.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
__tablename__ = "branches"
|
__tablename__ = "branches"
|
||||||
@@ -201,22 +212,23 @@ class Branch(Base):
|
|||||||
parent_branch_id: Mapped[int | None] = mapped_column(
|
parent_branch_id: Mapped[int | None] = mapped_column(
|
||||||
ForeignKey("branches.id", ondelete="CASCADE"), nullable=True
|
ForeignKey("branches.id", ondelete="CASCADE"), nullable=True
|
||||||
)
|
)
|
||||||
# The depth this branch left its parent at, stored when the fork happens and
|
# The depth at which this branch left its parent. The fork records this
|
||||||
# never inferred afterwards. Inferring it from where two branches' nodes
|
# value, and no code infers it later. Deriving it from the first depth where
|
||||||
# first differ would be a guess about how the story was played — and a wrong
|
# two branches' nodes differ would produce a wrong answer whenever an
|
||||||
# one as soon as an attempt happens to repeat its parent's text.
|
# attempt repeats its parent's text.
|
||||||
fork_depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
fork_depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
# The ancestry, newest first: [[branch_id, max_depth], ...] where max_depth
|
# The ancestry, newest first, as [[branch_id, max_depth], ...]. A NULL
|
||||||
# is NULL for "to the tip" and otherwise the fork_depth of the branch
|
# `max_depth` means the entry extends to the tip of that branch. Any other
|
||||||
# beneath it, inclusive. Computed once at fork from the parent's lineage
|
# value is the fork depth of the branch below it, inclusive. The fork
|
||||||
# plus one entry, so no read ever reconstructs it.
|
# computes this list once from the parent's lineage plus one entry, so no
|
||||||
|
# read has to reconstruct it.
|
||||||
lineage: Mapped[list] = mapped_column(JSON, default=list)
|
lineage: Mapped[list] = mapped_column(JSON, default=list)
|
||||||
# What the player called this line of the story, or NULL for one nobody has
|
# The name the player gave this line of the story, or NULL if no one named
|
||||||
# named. NULL rather than a generated "branch 4", because a generated name
|
# it. The column stores NULL rather than a generated name such as
|
||||||
# is derived and this column is for what was chosen — the same rule the v2
|
# "branch 4", because it records what the player chose rather than what the
|
||||||
# bundle is built on. A stored default would also become a lie the moment a
|
# app derived. A stored default would also become wrong as soon as an
|
||||||
# branch before it is deleted and the ordinals shift under it; an unnamed
|
# earlier branch is deleted and the ordinals shift. The UI labels an unnamed
|
||||||
# branch is drawn from its fork depth instead, which nothing can shift.
|
# branch by its fork depth, which deleting a branch does not change.
|
||||||
name: Mapped[str | None] = mapped_column(String(80), nullable=True)
|
name: Mapped[str | None] = mapped_column(String(80), nullable=True)
|
||||||
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
|
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
|
||||||
|
|
||||||
@@ -228,11 +240,10 @@ class Memory(Base):
|
|||||||
NULL until embedded, which also marks it for backfill when an embedding
|
NULL until embedded, which also marks it for backfill when an embedding
|
||||||
model becomes available.
|
model becomes available.
|
||||||
|
|
||||||
Cosine ranking happens in Python, which means the vectors cross the wire.
|
Cosine ranking runs in Python, so the vectors travel over the wire. Measure
|
||||||
The original comment here sized that by count — "fine at a few hundred" —
|
that cost in bytes rather than in rows. A few hundred vectors stored as JSON
|
||||||
and it was wrong by the only measure that mattered: a few hundred JSON
|
come to about 10 MB, fetched again on every turn. Size any new column by the
|
||||||
vectors is ten megabytes, fetched fresh every turn. Weigh new columns in
|
bytes it adds, not by the number of rows.
|
||||||
bytes.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
__tablename__ = "memories"
|
__tablename__ = "memories"
|
||||||
@@ -240,39 +251,42 @@ class Memory(Base):
|
|||||||
id: Mapped[int] = mapped_column(primary_key=True)
|
id: Mapped[int] = mapped_column(primary_key=True)
|
||||||
adventure_id: Mapped[int] = mapped_column(ForeignKey("adventures.id", ondelete="CASCADE"))
|
adventure_id: Mapped[int] = mapped_column(ForeignKey("adventures.id", ondelete="CASCADE"))
|
||||||
text: Mapped[str] = mapped_column(Text, default="")
|
text: Mapped[str] = mapped_column(Text, default="")
|
||||||
# The vector, little-endian float32. Deferred because it is wider than the
|
# The vector, stored as little-endian float32. This column is deferred
|
||||||
# rest of the row put together and exactly one code path wants it: anything
|
# because it is wider than the rest of the row combined and only one code
|
||||||
# bulk-loading memories (the Memories drawer, eviction, the embed queue)
|
# path reads it. Code that loads memories in bulk, such as the Memories
|
||||||
# must project the columns it needs rather than load whole entities.
|
# drawer, eviction, and the embed queue, must select the columns it needs
|
||||||
|
# instead of loading whole entities.
|
||||||
embedding_blob: Mapped[bytes | None] = mapped_column(
|
embedding_blob: Mapped[bytes | None] = mapped_column(
|
||||||
LargeBinary, nullable=True, deferred=True
|
LargeBinary, nullable=True, deferred=True
|
||||||
)
|
)
|
||||||
# The stretch of story this memory summarizes, as depths on `branch_id`
|
# The stretch of story this memory summarizes, given as depths on
|
||||||
# (null for a hand-written memory, which summarizes nothing). Written as
|
# `branch_id`. Both are NULL for a hand-written memory, which summarizes no
|
||||||
# `Action.index` values before SP3, which held the same numbers.
|
# actions. Before SP3 these columns held `Action.index` values, which were
|
||||||
# `source_end` is the depth of the node the memory hangs off, mirrored into
|
# the same numbers. `source_end` is the depth of the node the memory
|
||||||
# `depth` below; `source_start` is where it began, which is where the
|
# attaches to, and `depth` below mirrors it. `source_start` is where the
|
||||||
# summarizer has to resume from if the memory is ever withdrawn.
|
# stretch begins, which is where the summarizer resumes if the memory is
|
||||||
|
# withdrawn.
|
||||||
source_start: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
source_start: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
source_end: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
source_end: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
# Phase 14: the node that produced this memory — the last action it
|
# Phase 14: the node that produced this memory, meaning the last action the
|
||||||
# summarises. Anything derived attaches to the node it came from, which is
|
# memory summarizes. Derived data attaches to the node it came from, which
|
||||||
# what makes a fork free: a shared ancestor's memories are shared
|
# is what makes forking cheap. Memories on a shared ancestor are shared
|
||||||
# automatically, and a memory covering a stretch of branch B is invisible
|
# automatically, and a memory that covers part of branch B is not visible
|
||||||
# from any path that does not go through B.
|
# from any path that does not go through B.
|
||||||
#
|
#
|
||||||
# Every memory has one, including a hand-written one: it takes the head at
|
# Every memory has a coordinate, including a hand-written one, which takes
|
||||||
# the moment it was written (SP7). A NULL depth used to mean "belongs to the
|
# the head as of the moment it was written (SP7). A NULL depth used to mean
|
||||||
# adventure, not to a path", which is a category no fork could cap — the
|
# that the memory belonged to the adventure rather than to a path. A fork
|
||||||
# memory followed the reader onto branches whose story it never described.
|
# cannot cap a NULL, so such a memory followed the reader onto branches
|
||||||
|
# whose story it did not describe.
|
||||||
branch_id: Mapped[int | None] = mapped_column(
|
branch_id: Mapped[int | None] = mapped_column(
|
||||||
ForeignKey("branches.id", ondelete="CASCADE"), nullable=True
|
ForeignKey("branches.id", ondelete="CASCADE"), nullable=True
|
||||||
)
|
)
|
||||||
depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
# Whether embedding_blob is set. Maintained on write by memorybank
|
# Whether `embedding_blob` is set. `memorybank.set_vector` keeps this
|
||||||
# .set_vector, for the same reason actions.variant_count exists beside
|
# column current, for the same reason that `actions.variant_count` sits
|
||||||
# actions.variants: every reader wants the one-bit answer and none of them
|
# beside `actions.variants`. Readers need only the yes-or-no answer, and
|
||||||
# should have to fetch six kilobytes of vector to get it.
|
# fetching six kilobytes of vector to get it is too expensive.
|
||||||
embedded: Mapped[bool] = mapped_column(Boolean, default=False)
|
embedded: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
pinned: Mapped[bool] = mapped_column(Boolean, default=False)
|
pinned: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
forgotten: Mapped[bool] = mapped_column(Boolean, default=False) # evicted, kept for UI
|
forgotten: Mapped[bool] = mapped_column(Boolean, default=False) # evicted, kept for UI
|
||||||
@@ -300,10 +314,12 @@ class StoryCard(Base):
|
|||||||
keys: Mapped[str] = mapped_column(Text, default="") # comma-separated triggers
|
keys: Mapped[str] = mapped_column(Text, default="") # comma-separated triggers
|
||||||
entry: Mapped[str] = mapped_column(Text, default="")
|
entry: Mapped[str] = mapped_column(Text, default="")
|
||||||
notes: Mapped[str] = mapped_column(Text, default="")
|
notes: Mapped[str] = mapped_column(Text, default="")
|
||||||
# Adventure copies only: which piece of the scenario this card came from —
|
# Set on adventure copies only. It records which piece of the scenario the
|
||||||
# "card:<scenario_card_id>" or "npc:<npc_key>". "Update from scenario"
|
# card came from, as either "card:<scenario_card_id>" or "npc:<npc_key>".
|
||||||
# refreshes/removes exactly these; NULL means player-authored (left alone),
|
# The "Update from scenario" action refreshes or removes exactly these
|
||||||
# or a copy predating the column (matched by name, then adopted).
|
# cards. A NULL value means the player wrote the card, so the update leaves
|
||||||
|
# it alone, or that the copy predates this column, in which case the update
|
||||||
|
# matches it by name and then sets this value.
|
||||||
source_ref: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
source_ref: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||||
|
|
||||||
scenario: Mapped[Scenario | None] = relationship(back_populates="story_cards")
|
scenario: Mapped[Scenario | None] = relationship(back_populates="story_cards")
|
||||||
@@ -312,72 +328,76 @@ class StoryCard(Base):
|
|||||||
|
|
||||||
class Action(Base):
|
class Action(Base):
|
||||||
__tablename__ = "actions"
|
__tablename__ = "actions"
|
||||||
# Phase 14: every read of a story is "this branch up to this depth, or that
|
# Phase 14: every story read selects one branch up to one depth, then
|
||||||
# branch up to that depth, ...", so (branch_id, depth) is the shape every
|
# another branch up to another depth, and so on. The pair (branch_id, depth)
|
||||||
# one of those clauses wants an index on.
|
# is the index those clauses need.
|
||||||
__table_args__ = (Index("ix_actions_branch_depth", "branch_id", "depth"),)
|
__table_args__ = (Index("ix_actions_branch_depth", "branch_id", "depth"),)
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(primary_key=True)
|
id: Mapped[int] = mapped_column(primary_key=True)
|
||||||
adventure_id: Mapped[int] = mapped_column(ForeignKey("adventures.id", ondelete="CASCADE"))
|
adventure_id: Mapped[int] = mapped_column(ForeignKey("adventures.id", ondelete="CASCADE"))
|
||||||
index: Mapped[int] = mapped_column(Integer)
|
index: Mapped[int] = mapped_column(Integer)
|
||||||
# Phase 14: the node's place in the tree. `depth` is a position along *a*
|
# Phase 14: the node's place in the tree. `depth` is a position along one
|
||||||
# path, not a global turn number — A4 and B4 are alternatives, not
|
# path rather than a global turn number. Node A4 and node B4 are
|
||||||
# duplicates — and it replaces `index` as the ordering key.
|
# alternatives, not duplicates. `depth` replaces `index` as the ordering
|
||||||
|
# key.
|
||||||
#
|
#
|
||||||
# Nullable because ALTER TABLE cannot add a NOT NULL column with no
|
# Both columns are nullable because ALTER TABLE cannot add a NOT NULL column
|
||||||
# default and there is no sensible default for "which branch": the
|
# without a default, and no default makes sense for a branch. The migration
|
||||||
# migration fills them, `tree.place_action` fills them for new nodes, and
|
# fills these columns for existing rows, and `tree.place_action` fills them
|
||||||
# from SP2 on a NULL branch_id is a row no read can see. Legacy `index`
|
# for new rows. From SP2 onward, a NULL `branch_id` marks a row that no read
|
||||||
# stays beside them, unread, until the tree is proven live (SP8 drops it).
|
# can see. The legacy `index` column stays alongside, unread, until the tree
|
||||||
|
# is proven in production. SP8 drops it.
|
||||||
branch_id: Mapped[int | None] = mapped_column(
|
branch_id: Mapped[int | None] = mapped_column(
|
||||||
ForeignKey("branches.id", ondelete="CASCADE"), nullable=True
|
ForeignKey("branches.id", ondelete="CASCADE"), nullable=True
|
||||||
)
|
)
|
||||||
depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
# Phase 14, SP9: the node this one was played after — the take that was
|
# Phase 14, SP9: the node that this node was played after. It names the take
|
||||||
# live when it was written, not merely whatever sits at depth - 1 now.
|
# that was live when this row was written, not whatever sits at depth - 1
|
||||||
|
# now.
|
||||||
#
|
#
|
||||||
# It exists for one question: which takes belong to the same turn. A
|
# This column answers one question: which takes belong to the same turn. A
|
||||||
# coordinate cannot answer it, because a take that gets forked onto its own
|
# coordinate cannot answer it. A take that is forked onto its own branch
|
||||||
# branch leaves the coordinate its siblings are still at and would read
|
# leaves the coordinate that its siblings still occupy, so the pager would
|
||||||
# `1/1` next to their `1/3`. A parent does not move when a branch does.
|
# show it as 1/1 next to their 1/3. Forking a branch does not change a
|
||||||
|
# node's parent.
|
||||||
#
|
#
|
||||||
# Read only to group takes — one indexed lookup, never a walk. Paths still
|
# Code reads this column only to group takes, using one indexed lookup
|
||||||
# resolve through `lineage`, which is why this column can be added without
|
# rather than a walk. Paths still resolve through `lineage`, which is why
|
||||||
# touching a single read of the story.
|
# adding this column required no change to any read of the story.
|
||||||
#
|
#
|
||||||
# NULL on a root node, and on every pre-SP9 row the migration could not
|
# The value is NULL on a root node, and on pre-SP9 rows that the migration
|
||||||
# place: `attempts.group` falls back to the coordinate there, which is what
|
# could not place. For those rows, `attempts.group` falls back to the
|
||||||
# those rows were written under.
|
# coordinate, which is how they were written.
|
||||||
parent_id: Mapped[int | None] = mapped_column(
|
parent_id: Mapped[int | None] = mapped_column(
|
||||||
ForeignKey("actions.id", ondelete="SET NULL"), nullable=True, index=True
|
ForeignKey("actions.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
)
|
)
|
||||||
# Phase 14, SP4: whether this node is the one the story tells at its
|
# Phase 14, SP4: whether this node is the one the story uses at its
|
||||||
# coordinate. Retry no longer rewrites a row — it writes a *sibling* at the
|
# coordinate. Retry no longer rewrites a row. It writes a sibling at the
|
||||||
# same (branch, depth), so a coordinate can hold several attempts and
|
# same branch and depth, so one coordinate can hold several attempts while
|
||||||
# exactly one of them is on the path. `lineage.Path.clause` is the only
|
# exactly one of them is on the path. `lineage.Path.clause` is the only
|
||||||
# place that reads this, for the same reason it is the only place that
|
# place that reads this column, for the same reason it is the only place
|
||||||
# knows about branches: an attempt leaking into a read is a story quietly
|
# that knows about branches. If a discarded attempt reaches a read, the page
|
||||||
# telling itself twice.
|
# renders the same turn twice.
|
||||||
#
|
#
|
||||||
# A node with no siblings is live, which is why the default is True and why
|
# A node with no siblings is live, so the default is True and every pre-SP4
|
||||||
# every pre-SP4 row is correct without being visited.
|
# row is already correct. The migration does not need to visit them.
|
||||||
live: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
live: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||||
type: Mapped[str] = mapped_column(String(20)) # start|do|say|story|continue|ai
|
type: Mapped[str] = mapped_column(String(20)) # start|do|say|story|continue|ai
|
||||||
text: Mapped[str] = mapped_column(Text, default="")
|
text: Mapped[str] = mapped_column(Text, default="")
|
||||||
# Reasoning-model "thinking" that preceded the text (AI actions only).
|
# Reasoning-model "thinking" that preceded the text (AI actions only).
|
||||||
reasoning: Mapped[str | None] = mapped_column(Text, nullable=True)
|
reasoning: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||||
# The full assembled prompt for this turn, for the Insights viewer. By far
|
# The full assembled prompt for this turn, used by the Insights viewer.
|
||||||
# the biggest column in the database — 163 KB a row averaged over
|
# This is the largest column in the database. It averages 163 KB per row in
|
||||||
# production and 232 KB on the longest adventure, 89% of everything stored
|
# production and 232 KB on the longest adventure, and it accounts for 89% of
|
||||||
# — and needed by exactly one endpoint, one action at a time.
|
# everything stored. Only one endpoint reads it, one action at a time.
|
||||||
#
|
#
|
||||||
# Two separate defences, because it is expensive in two separate ways.
|
# The column is expensive in two ways, so it has two protections.
|
||||||
# `deferred=True` is the read defence: never loaded unless something
|
# `deferred=True` protects reads, because SQLAlchemy loads the column only
|
||||||
# touches the attribute, so a page load pays nothing for it. Bulk readers
|
# when code touches the attribute. A page load therefore costs nothing. Code
|
||||||
# must NOT touch it; that is what `world_delta` below exists for.
|
# that reads actions in bulk must not touch this attribute, which is why
|
||||||
# CompressedJSON is the *storage* defence: this is the column that decides
|
# `world_delta` below exists. `CompressedJSON` protects storage, because
|
||||||
# when the free tier's 512 MB runs out. Still a dict either way — see
|
# this column determines when the free tier's 512 MB limit is reached. The
|
||||||
# compression.py.
|
# attribute behaves like a plain dict in both cases. See compression.py.
|
||||||
context_snapshot: Mapped[dict | None] = mapped_column(
|
context_snapshot: Mapped[dict | None] = mapped_column(
|
||||||
CompressedJSON, nullable=True, deferred=True
|
CompressedJSON, nullable=True, deferred=True
|
||||||
)
|
)
|
||||||
@@ -386,58 +406,58 @@ class Action(Base):
|
|||||||
# and for re-attaching the emit block when replaying history to the model.
|
# and for re-attaching the emit block when replaying history to the model.
|
||||||
# Mirrors the active variant, same as text/reasoning/context_snapshot.
|
# Mirrors the active variant, same as text/reasoning/context_snapshot.
|
||||||
world_delta: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
world_delta: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||||
# LEGACY (SP4): Adventure.script_state / world_state as they were
|
# Legacy columns from before SP4. They hold `Adventure.script_state` and
|
||||||
# immediately BEFORE this action's script hooks ran. Unwritten since SP4
|
# `Adventure.world_state` as they were immediately before this action's
|
||||||
# and read by nothing — the *after* pair below replaced them, because a
|
# script hooks ran. Nothing has written or read them since SP4. The pair of
|
||||||
# sibling attempt needs its own outcome and a "before" picture is shared by
|
# "after" columns below replaced them, because each sibling attempt needs
|
||||||
# every attempt at the turn. Kept for one release so a rolled-back build
|
# its own outcome and every attempt at a turn shares the same starting
|
||||||
# still finds a real snapshot on every row it wrote itself; SP8 drops them
|
# state. These columns remain for one release so that a rolled-back build
|
||||||
# beside `index` and `variants`.
|
# still finds a real snapshot on the rows it wrote. SP8 drops them along
|
||||||
|
# with `index` and `variants`.
|
||||||
state_before: Mapped[dict | None] = mapped_column(
|
state_before: Mapped[dict | None] = mapped_column(
|
||||||
JSON, nullable=True, deferred=True
|
JSON, nullable=True, deferred=True
|
||||||
)
|
)
|
||||||
world_state_before: Mapped[dict | None] = mapped_column(
|
world_state_before: Mapped[dict | None] = mapped_column(
|
||||||
JSON, nullable=True, deferred=True
|
JSON, nullable=True, deferred=True
|
||||||
)
|
)
|
||||||
# Phase 14, SP4: the shared script scoreboard and the RPG world state as
|
# Phase 14, SP4: the shared script state and the RPG world state as they
|
||||||
# they stood once this node had been played — *its* outcome, not its
|
# stood after this node was played. These columns record the node's outcome
|
||||||
# starting position.
|
# rather than its starting position.
|
||||||
#
|
#
|
||||||
# Two things want this and neither can use a "before" picture. Switching
|
# Two operations need this outcome, and neither can use a snapshot taken
|
||||||
# between siblings has to put back the state the chosen attempt produced,
|
# before the turn. Switching between siblings must restore the state that
|
||||||
# and the attempts differ precisely in what they produced. And rolling back
|
# the chosen attempt produced, and the attempts differ in exactly that.
|
||||||
# to before a turn is "the state the node in front of it left behind",
|
# Rolling back to before a turn means restoring the state that the preceding
|
||||||
# which is one lookup on the path rather than a snapshot that has to be
|
# node left behind, which is one lookup along the path.
|
||||||
# taken from inside the turn being rolled back.
|
|
||||||
#
|
#
|
||||||
# NULL on rows written before SP4 that the migration could not derive one
|
# The value is NULL on pre-SP4 rows for which the migration could not derive
|
||||||
# for, and tolerated everywhere: a missing snapshot means "leave the live
|
# one. Every caller tolerates that. A missing snapshot means that the caller
|
||||||
# state alone", never "reset it".
|
# leaves the live state unchanged. It never means reset the state.
|
||||||
#
|
#
|
||||||
# Deferred: only ever read for the one node being switched to, undone or
|
# These columns are deferred, because code reads them only for the single
|
||||||
# retried past.
|
# node being switched to, undone, or retried past.
|
||||||
state_after: Mapped[dict | None] = mapped_column(
|
state_after: Mapped[dict | None] = mapped_column(
|
||||||
JSON, nullable=True, deferred=True
|
JSON, nullable=True, deferred=True
|
||||||
)
|
)
|
||||||
world_state_after: Mapped[dict | None] = mapped_column(
|
world_state_after: Mapped[dict | None] = mapped_column(
|
||||||
JSON, nullable=True, deferred=True
|
JSON, nullable=True, deferred=True
|
||||||
)
|
)
|
||||||
# LEGACY (SP4): retry history as a JSON repeating group. Every attempt at
|
# A legacy column from before SP4. It holds the retry history as a repeating
|
||||||
# this turn is its own row now — see `live` above and `app/attempts.py` —
|
# group inside a JSON list. Every attempt at a turn is now its own row, as
|
||||||
# so nothing reads this. Kept until SP8 for the same reason `index` is, and
|
# described on `live` above and in `app/attempts.py`, so nothing reads this
|
||||||
# read exactly once more on the way out: migration 60 is what turns each
|
# column. It remains until SP8 for the same reason `index` does. Migration
|
||||||
# entry into the sibling row it should always have been.
|
# 60 reads it once more, to turn each entry into a sibling row.
|
||||||
variants: Mapped[list | None] = mapped_column(JSON, nullable=True, deferred=True)
|
variants: Mapped[list | None] = mapped_column(JSON, nullable=True, deferred=True)
|
||||||
# Where the row sits in its sibling group: `variant_index` is this
|
# Where the row sits in its sibling group. `variant_index` is this
|
||||||
# attempt's ordinal, oldest first, and `variant_count` is how many attempts
|
# attempt's ordinal, counting from the oldest. `variant_count` is the number
|
||||||
# the group holds (0, not 1, when the turn was never retried — the pager
|
# of attempts in the group. It is 0 rather than 1 when the turn was never
|
||||||
# reads that as "nothing to page through").
|
# retried, which the pager treats as having nothing to page through.
|
||||||
#
|
#
|
||||||
# A cache of two facts about the group, maintained in one place
|
# Both columns are caches, and `attempts.renumber` is the only place that
|
||||||
# (`attempts.renumber`) for the same reason it used to be a cache of
|
# maintains them. The reason matches why `variant_count` once cached
|
||||||
# `len(variants)`: a page response wants them for every row and must not
|
# `len(variants)`: a page response needs both numbers for every row and
|
||||||
# pay a query per turn to get them. SP7 replaces the pager with the branch
|
# cannot afford one query per turn. SP7 replaces the pager with the branch
|
||||||
# view and SP8 drops both columns.
|
# view, and SP8 drops both columns.
|
||||||
variant_count: Mapped[int] = mapped_column(Integer, default=0)
|
variant_count: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
variant_index: Mapped[int] = mapped_column(Integer, default=0)
|
variant_index: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
|
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
|
||||||
@@ -450,9 +470,9 @@ class Action(Base):
|
|||||||
under an AI message. Labels are path-based (no schema needed):
|
under an AI message. Labels are path-based (no schema needed):
|
||||||
`npc.gwen.trust` -> "gwen trust".
|
`npc.gwen.trust` -> "gwen trust".
|
||||||
|
|
||||||
Reads `world_delta`, never `context_snapshot` — this runs for every
|
Reads `world_delta`, never `context_snapshot`. This runs for every
|
||||||
action in a list response, and touching the deferred snapshot here
|
action in a list response, and touching the deferred snapshot here would
|
||||||
would drag the whole prompt archive out of the database."""
|
drag the entire prompt archive out of the database."""
|
||||||
wd = self.world_delta if isinstance(self.world_delta, dict) else None
|
wd = self.world_delta if isinstance(self.world_delta, dict) else None
|
||||||
if wd is None:
|
if wd is None:
|
||||||
return []
|
return []
|
||||||
@@ -500,10 +520,10 @@ class AdventureScript(Base):
|
|||||||
|
|
||||||
id: Mapped[int] = mapped_column(primary_key=True)
|
id: Mapped[int] = mapped_column(primary_key=True)
|
||||||
adventure_id: Mapped[int] = mapped_column(ForeignKey("adventures.id", ondelete="CASCADE"))
|
adventure_id: Mapped[int] = mapped_column(ForeignKey("adventures.id", ondelete="CASCADE"))
|
||||||
# The library Script this copy was made from, so it can be re-synced on
|
# The library Script that this copy was made from, which lets the player
|
||||||
# demand. NULL for legacy copies (predate this column) and demo-derived
|
# re-sync it on demand. The value is NULL for legacy copies that predate
|
||||||
# ones whose source isn't owned by the player — those fall back to a
|
# this column, and for demo-derived copies whose source the player does not
|
||||||
# name match, or simply aren't syncable.
|
# own. Those copies fall back to matching by name, or cannot be synced.
|
||||||
source_script_id: Mapped[int | None] = mapped_column(
|
source_script_id: Mapped[int | None] = mapped_column(
|
||||||
ForeignKey("scripts.id", ondelete="SET NULL"), nullable=True
|
ForeignKey("scripts.id", ondelete="SET NULL"), nullable=True
|
||||||
)
|
)
|
||||||
@@ -529,7 +549,8 @@ class Settings(Base):
|
|||||||
ForeignKey("users.id", ondelete="CASCADE"), nullable=True, unique=True
|
ForeignKey("users.id", ondelete="CASCADE"), nullable=True, unique=True
|
||||||
)
|
)
|
||||||
endpoint_url: Mapped[str] = mapped_column(String(500), default="http://localhost:11434/v1")
|
endpoint_url: Mapped[str] = mapped_column(String(500), default="http://localhost:11434/v1")
|
||||||
# Fernet-encrypted at rest ("enc:..." — see security.py); use api_key_plain.
|
# Encrypted at rest with Fernet, which produces a value that starts with
|
||||||
|
# "enc:". See security.py. To read the key, use `api_key_plain`.
|
||||||
api_key: Mapped[str] = mapped_column(String(500), default="")
|
api_key: Mapped[str] = mapped_column(String(500), default="")
|
||||||
model: Mapped[str] = mapped_column(String(200), default="")
|
model: Mapped[str] = mapped_column(String(200), default="")
|
||||||
api_mode: Mapped[str] = mapped_column(String(20), default="chat") # chat|completion
|
api_mode: Mapped[str] = mapped_column(String(20), default="chat") # chat|completion
|
||||||
@@ -557,10 +578,10 @@ class Settings(Base):
|
|||||||
# Phase 6: auto-summarization + memory bank
|
# Phase 6: auto-summarization + memory bank
|
||||||
summary_model: Mapped[str] = mapped_column(String(200), default="") # "" = main model
|
summary_model: Mapped[str] = mapped_column(String(200), default="") # "" = main model
|
||||||
embedding_model: Mapped[str] = mapped_column(String(200), default="") # "" = bank disabled
|
embedding_model: Mapped[str] = mapped_column(String(200), default="") # "" = bank disabled
|
||||||
# Was 200. Lowered on retrieval-quality grounds first: ranking two hundred
|
# This was 200. It was lowered mainly to improve retrieval quality. Ranking
|
||||||
# memories to pick five means the five are chosen out of a lot of noise,
|
# 200 memories to choose 5 selects from a large amount of noise, and the
|
||||||
# and older memories describe a story the player has moved on from. That it
|
# oldest memories describe a part of the story that the player has left
|
||||||
# also cuts what the bank costs to read is the smaller reason.
|
# behind. Cheaper reads are a secondary benefit rather than the reason.
|
||||||
memory_bank_capacity: Mapped[int] = mapped_column(Integer, default=80)
|
memory_bank_capacity: Mapped[int] = mapped_column(Integer, default=80)
|
||||||
memory_top_k: Mapped[int] = mapped_column(Integer, default=5)
|
memory_top_k: Mapped[int] = mapped_column(Integer, default=5)
|
||||||
|
|
||||||
@@ -576,18 +597,19 @@ class Settings(Base):
|
|||||||
|
|
||||||
|
|
||||||
# ---------- Visit analytics (see analytics.py) ----------
|
# ---------- Visit analytics (see analytics.py) ----------
|
||||||
# Two deliberately dumb tables. Neither can hold anything a player wrote, and
|
# Two intentionally simple tables. Neither can hold text that a player wrote,
|
||||||
# neither can be joined back to a `users` row: the visitor column is an HMAC,
|
# and neither can be joined back to a `users` row, because the visitor column
|
||||||
# with no foreign key, so guest cleanup deleting an account leaves the history
|
# holds an HMAC and has no foreign key. When guest cleanup deletes an account,
|
||||||
# it contributed to intact and anonymous.
|
# the history that account contributed remains intact and anonymous.
|
||||||
|
|
||||||
|
|
||||||
class AnalyticsDaily(Base):
|
class AnalyticsDaily(Base):
|
||||||
"""One counter: how many times `label` happened within `metric` on `day`.
|
"""One counter: how many times `label` happened within `metric` on `day`.
|
||||||
|
|
||||||
A generic (metric, label, hits) triple rather than a column per statistic,
|
The table stores a generic triple of metric, label, and hits rather than one
|
||||||
so measuring something new later costs a constant instead of a migration.
|
column per statistic. Measuring something new therefore costs a constant
|
||||||
Written only by UPSERT, from a buffer — see analytics.flush.
|
rather than a migration. The only writer is an UPSERT that runs from a
|
||||||
|
buffer. See `analytics.flush`.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
__tablename__ = "analytics_daily"
|
__tablename__ = "analytics_daily"
|
||||||
@@ -607,10 +629,10 @@ class AnalyticsDaily(Base):
|
|||||||
class AnalyticsVisitorDay(Base):
|
class AnalyticsVisitorDay(Base):
|
||||||
"""One visitor, one day, and which funnel steps they reached on it.
|
"""One visitor, one day, and which funnel steps they reached on it.
|
||||||
|
|
||||||
Exists so the funnel counts people rather than clicks — a player who starts
|
This table exists so that the funnel counts people rather than clicks. A
|
||||||
six adventures is one person who started an adventure. `is_new` is set when
|
player who starts six adventures counts as one person who started an
|
||||||
the visitor has no earlier row, which is also why the visitor column is
|
adventure. `is_new` is set when the visitor has no earlier row, which is why
|
||||||
indexed on its own.
|
the visitor column also has an index of its own.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
__tablename__ = "analytics_visitor_days"
|
__tablename__ = "analytics_visitor_days"
|
||||||
@@ -634,15 +656,16 @@ class AnalyticsVisitorDay(Base):
|
|||||||
class AccessEvent(Base):
|
class AccessEvent(Base):
|
||||||
"""One sign-in, registration, failed attempt, or session first-seen.
|
"""One sign-in, registration, failed attempt, or session first-seen.
|
||||||
|
|
||||||
The counterpart to the two tables above, and deliberately not mixed in with
|
This table is the counterpart to the two above, and it is kept separate from
|
||||||
them: this one identifies people on purpose — address, email, device — so
|
them on purpose. It identifies people by design, recording address, email,
|
||||||
keeping it in its own table (and its own module) means the anonymity of the
|
and device. Keeping it in its own table and its own module means that the
|
||||||
counters stays a property of the code rather than of a convention.
|
structure enforces the anonymity of the counters rather than a convention.
|
||||||
|
|
||||||
`user_id` is a plain integer with no foreign key. An access log that
|
`user_id` is a plain integer with no foreign key. An access log that
|
||||||
disappeared when the account did would not be an access log, and guest
|
disappeared when the account did would not serve its purpose, and guest
|
||||||
cleanup deletes accounts on a schedule; `who` and `is_guest` are snapshots
|
cleanup deletes accounts on a schedule. `who` and `is_guest` are snapshots
|
||||||
for the same reason, so a row still reads correctly afterwards.
|
for the same reason, so a row still reads correctly after the account is
|
||||||
|
gone.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
__tablename__ = "access_events"
|
__tablename__ = "access_events"
|
||||||
@@ -652,8 +675,9 @@ class AccessEvent(Base):
|
|||||||
# session | login | register | login_failed
|
# session | login | register | login_failed
|
||||||
kind: Mapped[str] = mapped_column(String(16))
|
kind: Mapped[str] = mapped_column(String(16))
|
||||||
user_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
user_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
# Email for a registered account, "Guest #12" otherwise; for a failed
|
# The email for a registered account, or a label such as "Guest #12"
|
||||||
# sign-in, the address that was tried — which is the point of the row.
|
# otherwise. For a failed sign-in, this holds the address that was tried,
|
||||||
|
# which is the reason the row exists.
|
||||||
who: Mapped[str] = mapped_column(String(320), default="")
|
who: Mapped[str] = mapped_column(String(320), default="")
|
||||||
is_guest: Mapped[bool] = mapped_column(Boolean, default=False)
|
is_guest: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
ip: Mapped[str] = mapped_column(String(45), default="") # 45 = max IPv6
|
ip: Mapped[str] = mapped_column(String(45), default="") # 45 = max IPv6
|
||||||
@@ -662,19 +686,19 @@ class AccessEvent(Base):
|
|||||||
user_agent: Mapped[str] = mapped_column(String(200), default="")
|
user_agent: Mapped[str] = mapped_column(String(200), default="")
|
||||||
|
|
||||||
|
|
||||||
# Phase 14 — the floor under `tree.place_action`.
|
# Phase 14: the fallback under `tree.place_action`.
|
||||||
#
|
#
|
||||||
# From SP2 a read selects on (branch_id, depth): a node written without them is
|
# Since SP2, reads filter on `branch_id` and `depth`. A node written without
|
||||||
# a node no page, no context build and no memory pass can see, and it fails by
|
# them is invisible to every page, every context build, and every memory pass.
|
||||||
# disappearing rather than by raising. The writers all place their nodes
|
# The failure is silent, because nothing raises an error. Every current writer
|
||||||
# explicitly, but "all the writers remember" is a promise that has to hold for
|
# places its nodes explicitly, but relying on that would also mean relying on
|
||||||
# every fixture, script and test written from here on, so the session enforces
|
# every fixture, script, and test written from now on. The session therefore
|
||||||
# it on the way to the database instead.
|
# enforces the rule as rows travel to the database.
|
||||||
#
|
#
|
||||||
# Registered here rather than in tree.py so that importing the models is enough
|
# This listener is registered here rather than in tree.py so that importing the
|
||||||
# to arm it — the invariant belongs to the rows, not to the module that usually
|
# models is enough to enable it. The invariant belongs to the rows, not to the
|
||||||
# writes them. The import is deferred into the callback because tree.py imports
|
# module that usually writes them. The import sits inside the callback because
|
||||||
# this module.
|
# tree.py imports this module.
|
||||||
@event.listens_for(Session, "before_flush")
|
@event.listens_for(Session, "before_flush")
|
||||||
def _place_new_nodes_on_the_tree(session, flush_context, instances):
|
def _place_new_nodes_on_the_tree(session, flush_context, instances):
|
||||||
from . import tree
|
from . import tree
|
||||||
|
|||||||
@@ -1,14 +1,15 @@
|
|||||||
"""SSRF guard for the one place the server makes an outbound request to a
|
"""SSRF guard for the one place the server makes an outbound request to a
|
||||||
user-supplied address: the BYOK `endpoint_url` (connection test + turns/chat).
|
user-supplied address: the BYOK `endpoint_url` (connection test + turns/chat).
|
||||||
|
|
||||||
Without this, a hosted user could point endpoint_url at an internal service or
|
Without this guard, a hosted user could point `endpoint_url` at an internal
|
||||||
the cloud metadata endpoint (169.254.169.254) and have the server fetch it —
|
service or at the cloud metadata endpoint, 169.254.169.254, and have the server
|
||||||
the connection test even echoes part of the response back. We refuse any URL
|
fetch it. The connection test even returns part of the response. The guard
|
||||||
that resolves to a non-public address.
|
therefore refuses any URL that resolves to a non-public address.
|
||||||
|
|
||||||
No-op in local mode: a local install talking to http://localhost:11434 (Ollama)
|
The guard does nothing in local mode. A local install talking to
|
||||||
is the normal, intended case — the guard only applies to the hosted, multi-user
|
http://localhost:11434, which is Ollama, is the intended case. The guard applies
|
||||||
deployment where the endpoint comes from an untrusted visitor.
|
only to a hosted, multi-user deployment, where the endpoint comes from an
|
||||||
|
untrusted visitor.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import ipaddress
|
import ipaddress
|
||||||
|
|||||||
@@ -17,10 +17,10 @@ class ProviderError(Exception):
|
|||||||
|
|
||||||
class Provider(ABC):
|
class Provider(ABC):
|
||||||
# The endpoint's own token accounting for the most recent call, when it
|
# The endpoint's own token accounting for the most recent call, when it
|
||||||
# reported any — notably `prompt_tokens_details.cached_tokens`, which is
|
# reported any. It notably carries `prompt_tokens_details.cached_tokens`,
|
||||||
# the only direct read on whether the prompt prefix is being cached.
|
# which is the only direct measure of whether the prompt prefix is being
|
||||||
# Callers read it after the call they made; one provider is built per
|
# cached. A caller reads it after the call it made, and one provider is built
|
||||||
# request, so there is nothing to race.
|
# per request, so nothing races.
|
||||||
last_usage: dict | None = None
|
last_usage: dict | None = None
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
|
|||||||
@@ -6,32 +6,33 @@ import httpx
|
|||||||
from .. import debuglog, netguard
|
from .. import debuglog, netguard
|
||||||
from .base import PromptParts, Provider, ProviderError
|
from .base import PromptParts, Provider, ProviderError
|
||||||
|
|
||||||
# Framing appended after the story text in chat mode, so chat-tuned models keep
|
# Appended after the story text in chat mode, so a chat-tuned model continues
|
||||||
# continuing prose instead of replying conversationally.
|
# the prose rather than replying conversationally.
|
||||||
CHAT_CONTINUE_HINT = "\n\n[Continue the story directly. Output only story text.]"
|
CHAT_CONTINUE_HINT = "\n\n[Continue the story directly. Output only story text.]"
|
||||||
|
|
||||||
# OpenRouter serves one model from whichever upstream is available, and every
|
# OpenRouter serves one model from whichever upstream is available, and every
|
||||||
# upstream holds its own prompt cache — so a request that lands somewhere new
|
# upstream holds its own prompt cache, so a request routed somewhere new starts
|
||||||
# starts cold however stable the prompt is. Naming a preferred upstream makes
|
# with a cold cache however stable the prompt is. Naming a preferred upstream
|
||||||
# routing deterministic, which is what lets a cache be hit at all.
|
# makes routing deterministic, which is what allows a cache hit at all.
|
||||||
#
|
#
|
||||||
# `allow_fallbacks` is deliberately left at its default of true: this is a
|
# `allow_fallbacks` stays at its default of true on purpose, because this is a
|
||||||
# preference, not a restriction. If the named upstream is down the request
|
# preference rather than a restriction. If the named upstream is down, the
|
||||||
# still goes through somewhere else and merely misses the cache, which is the
|
# request still goes elsewhere and only misses the cache, which is the behavior
|
||||||
# behaviour we had anyway.
|
# without this setting.
|
||||||
#
|
#
|
||||||
# A whitelist rather than a derivation from the model slug. The vendor half of
|
# This is a list rather than a value derived from the model slug. The vendor half
|
||||||
# a slug is *usually* the provider slug ("deepseek/..." -> "deepseek", verified
|
# of a slug is usually the provider slug, such as "deepseek/..." mapping to
|
||||||
# against /api/v1/providers) but not reliably: Google's models are served by
|
# "deepseek", which was verified against /api/v1/providers, but not reliably.
|
||||||
# "google-ai-studio" and "google-vertex", and there is no "google". Look a
|
# Google's models are served by "google-ai-studio" and "google-vertex", and there
|
||||||
# vendor up on the model's Providers tab before adding it here — a slug that
|
# is no "google". Look a vendor up on the model's Providers tab before adding it
|
||||||
# does not exist is a routing preference that silently does nothing at best.
|
# here. A slug that does not exist is a routing preference that, at best, does
|
||||||
|
# nothing.
|
||||||
_OPENROUTER_HOST = "openrouter.ai"
|
_OPENROUTER_HOST = "openrouter.ai"
|
||||||
_PREFERRED_UPSTREAM = {"deepseek": "deepseek"}
|
_PREFERRED_UPSTREAM = {"deepseek": "deepseek"}
|
||||||
|
|
||||||
|
|
||||||
# Completion endpoints have no roles, so a plain chat has to be flattened into
|
# Completion endpoints have no roles, so a chat has to be flattened into one
|
||||||
# one labelled transcript that trails off on "Assistant:" for the model to continue.
|
# labeled transcript that ends on "Assistant:" for the model to continue.
|
||||||
_ROLE_LABELS = {"system": "System", "user": "User", "assistant": "Assistant"}
|
_ROLE_LABELS = {"system": "System", "user": "User", "assistant": "Assistant"}
|
||||||
|
|
||||||
|
|
||||||
@@ -43,7 +44,11 @@ def flatten_messages(messages: list[dict]) -> str:
|
|||||||
|
|
||||||
|
|
||||||
class OpenAICompatibleProvider(Provider):
|
class OpenAICompatibleProvider(Provider):
|
||||||
"""Adapter for any /v1-style endpoint: Ollama, LM Studio, OpenAI, OpenRouter, vLLM, Groq…"""
|
"""Adapter for any /v1-style endpoint.
|
||||||
|
|
||||||
|
This covers Ollama, LM Studio, OpenAI, OpenRouter, vLLM, and Groq, among
|
||||||
|
others.
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
@@ -56,17 +61,18 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
self.base_url = endpoint_url.rstrip("/")
|
self.base_url = endpoint_url.rstrip("/")
|
||||||
self.api_key = api_key
|
self.api_key = api_key
|
||||||
self.model = model
|
self.model = model
|
||||||
self.api_mode = api_mode # "chat" | "completion"
|
self.api_mode = api_mode # Either "chat" or "completion".
|
||||||
# Thinking budget for reasoning models, on top of max_tokens. 0 = the
|
# The thinking budget for reasoning models, on top of `max_tokens`. A
|
||||||
# `reasoning` param is not sent (endpoints that don't know it may
|
# value of 0 means the `reasoning` parameter is not sent, because an
|
||||||
# reject unknown fields); negative = explicitly ask the endpoint to
|
# endpoint that does not know the field may reject it. A negative value
|
||||||
# turn reasoning off.
|
# asks the endpoint to turn reasoning off.
|
||||||
self.reasoning_max_tokens = reasoning_max_tokens
|
self.reasoning_max_tokens = reasoning_max_tokens
|
||||||
# Token accounting from the last call, when the endpoint reported any:
|
# The token accounting from the last call, when the endpoint reported
|
||||||
# prompt/completion counts plus, on OpenRouter, `prompt_tokens_details.
|
# any. It holds the prompt and completion counts, plus, on OpenRouter,
|
||||||
# cached_tokens` — the number of prompt tokens read from cache instead
|
# `prompt_tokens_details.cached_tokens`, which is the number of prompt
|
||||||
# of billed in full. Written by every request method, so a caller reads
|
# tokens read from cache rather than billed in full. Every request method
|
||||||
# it after the call it made; one provider is built per request.
|
# writes it, so a caller reads it after the call it made. One provider is
|
||||||
|
# built per request.
|
||||||
self.last_usage: dict | None = None
|
self.last_usage: dict | None = None
|
||||||
|
|
||||||
def _headers(self) -> dict:
|
def _headers(self) -> dict:
|
||||||
@@ -76,14 +82,16 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
return headers
|
return headers
|
||||||
|
|
||||||
def _apply_reasoning_budget(self, body: dict) -> None:
|
def _apply_reasoning_budget(self, body: dict) -> None:
|
||||||
"""Give reasoning models their own thinking budget (OpenRouter-style),
|
"""Gives reasoning models their own thinking budget, in the OpenRouter style.
|
||||||
raising max_tokens so the actual output keeps its full budget.
|
|
||||||
|
|
||||||
A negative budget means the opposite: send `effort: "none"` to switch
|
The method raises `max_tokens`, so the output keeps its full budget.
|
||||||
reasoning off on models that do it by default (DeepSeek V4 Flash, say).
|
|
||||||
That's distinct from `exclude: true`, which still thinks — and bills —
|
A negative budget does the opposite. It sends `effort: "none"` to turn
|
||||||
but hides the trace. Zero stays "send nothing at all" so endpoints that
|
reasoning off on a model that reasons by default, such as DeepSeek V4
|
||||||
reject unknown fields (Ollama) keep working."""
|
Flash. That differs from `exclude: true`, which still reasons and still
|
||||||
|
bills for it while hiding the trace. Zero still means send nothing, so an
|
||||||
|
endpoint that rejects unknown fields, such as Ollama, keeps working.
|
||||||
|
"""
|
||||||
if self.api_mode != "chat":
|
if self.api_mode != "chat":
|
||||||
return
|
return
|
||||||
if self.reasoning_max_tokens < 0:
|
if self.reasoning_max_tokens < 0:
|
||||||
@@ -93,11 +101,12 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
body["max_tokens"] += self.reasoning_max_tokens
|
body["max_tokens"] += self.reasoning_max_tokens
|
||||||
|
|
||||||
def _apply_provider_routing(self, body: dict) -> None:
|
def _apply_provider_routing(self, body: dict) -> None:
|
||||||
"""Prefer one upstream on OpenRouter, so the prompt cache is warm.
|
"""Prefers one upstream on OpenRouter, so the prompt cache stays warm.
|
||||||
|
|
||||||
Silent no-op everywhere else: `provider` is an OpenRouter extension and
|
The method does nothing anywhere else. `provider` is an OpenRouter
|
||||||
Ollama and friends reject fields they do not know — the same trap the
|
extension, and Ollama and similar servers reject fields they do not know.
|
||||||
`reasoning` param above is written around."""
|
The `reasoning` parameter above is written around the same constraint.
|
||||||
|
"""
|
||||||
if _OPENROUTER_HOST not in self.base_url:
|
if _OPENROUTER_HOST not in self.base_url:
|
||||||
return
|
return
|
||||||
upstream = _PREFERRED_UPSTREAM.get(self.model.split("/", 1)[0].lower())
|
upstream = _PREFERRED_UPSTREAM.get(self.model.split("/", 1)[0].lower())
|
||||||
@@ -105,12 +114,13 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
body["provider"] = {"order": [upstream]}
|
body["provider"] = {"order": [upstream]}
|
||||||
|
|
||||||
def _record_usage(self, payload: dict) -> None:
|
def _record_usage(self, payload: dict) -> None:
|
||||||
"""Record the endpoint's own token accounting, if it reported any.
|
"""Records the endpoint's own token accounting, if it reported any.
|
||||||
|
|
||||||
OpenRouter always reports usage now (`usage: {include: true}` and
|
OpenRouter now always reports usage, and `usage: {include: true}` and
|
||||||
`stream_options` are deprecated no-ops), and in a stream it rides on a
|
`stream_options` are deprecated and do nothing. In a stream the usage
|
||||||
final chunk that carries no choices — which is why this is read
|
arrives on a final chunk that carries no choices, which is why this is
|
||||||
separately from the text extraction rather than beside it."""
|
read separately from the text extraction.
|
||||||
|
"""
|
||||||
usage = payload.get("usage")
|
usage = payload.get("usage")
|
||||||
if isinstance(usage, dict) and usage:
|
if isinstance(usage, dict) and usage:
|
||||||
self.last_usage = usage
|
self.last_usage = usage
|
||||||
@@ -147,8 +157,8 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
if not choices:
|
if not choices:
|
||||||
return ""
|
return ""
|
||||||
choice = choices[0]
|
choice = choices[0]
|
||||||
# chat stream → delta.content; completion stream → text;
|
# A chat stream uses `delta.content`, and a completion stream uses
|
||||||
# non-stream fallbacks → message.content / text
|
# `text`. The non-stream fallbacks are `message.content` and `text`.
|
||||||
delta = choice.get("delta") or {}
|
delta = choice.get("delta") or {}
|
||||||
return (
|
return (
|
||||||
delta.get("content")
|
delta.get("content")
|
||||||
@@ -159,8 +169,11 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _extract_reasoning(payload: dict) -> str:
|
def _extract_reasoning(payload: dict) -> str:
|
||||||
"""Reasoning-model thinking: OpenRouter normalizes to `reasoning`;
|
"""Returns a reasoning model's thinking text.
|
||||||
DeepSeek-style servers use `reasoning_content`."""
|
|
||||||
|
OpenRouter normalizes it to `reasoning`, and DeepSeek-style servers use
|
||||||
|
`reasoning_content`.
|
||||||
|
"""
|
||||||
choices = payload.get("choices") or []
|
choices = payload.get("choices") or []
|
||||||
if not choices:
|
if not choices:
|
||||||
return ""
|
return ""
|
||||||
@@ -178,7 +191,7 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
async def generate(
|
async def generate(
|
||||||
self, parts: PromptParts, *, temperature: float, max_tokens: int
|
self, parts: PromptParts, *, temperature: float, max_tokens: int
|
||||||
) -> AsyncIterator[tuple[str, str]]:
|
) -> AsyncIterator[tuple[str, str]]:
|
||||||
"""Yields ("text" | "reasoning", chunk) pairs."""
|
"""Yields `("text", chunk)` and `("reasoning", chunk)` pairs."""
|
||||||
if not self.model:
|
if not self.model:
|
||||||
raise ProviderError("No model configured — set one in Settings.")
|
raise ProviderError("No model configured — set one in Settings.")
|
||||||
url, body = self._request(parts, temperature, max_tokens)
|
url, body = self._request(parts, temperature, max_tokens)
|
||||||
@@ -188,9 +201,12 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
async def chat(
|
async def chat(
|
||||||
self, messages: list[dict], *, temperature: float, max_tokens: int
|
self, messages: list[dict], *, temperature: float, max_tokens: int
|
||||||
) -> AsyncIterator[tuple[str, str]]:
|
) -> AsyncIterator[tuple[str, str]]:
|
||||||
"""Plain multi-turn chat — no story framing, no context assembly. Takes
|
"""Runs a plain multi-turn chat, with no story framing and no context
|
||||||
[{"role", "content"}, ...] straight to the endpoint. Used by the AI Chat
|
assembly.
|
||||||
scratchpad; the turn engine uses generate()."""
|
|
||||||
|
The method sends `[{"role", "content"}, ...]` straight to the endpoint.
|
||||||
|
The AI Chat scratchpad uses it, and the turn engine uses `generate()`.
|
||||||
|
"""
|
||||||
if not self.model:
|
if not self.model:
|
||||||
raise ProviderError("No model configured — set one in Settings.")
|
raise ProviderError("No model configured — set one in Settings.")
|
||||||
if self.api_mode == "completion":
|
if self.api_mode == "completion":
|
||||||
@@ -217,10 +233,14 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
yield event
|
yield event
|
||||||
|
|
||||||
async def _stream(self, url: str, body: dict) -> AsyncIterator[tuple[str, str]]:
|
async def _stream(self, url: str, body: dict) -> AsyncIterator[tuple[str, str]]:
|
||||||
"""Shared SSE plumbing for generate()/chat(): POST a streaming request
|
"""Runs the shared SSE request for `generate()` and `chat()`.
|
||||||
and yield ("text" | "reasoning", chunk) pairs, logging the exchange."""
|
|
||||||
# SSRF guard (hosted mode): a user-supplied endpoint_url must not point
|
The method POSTs a streaming request, yields `("text", chunk)` and
|
||||||
# at an internal/metadata address. No-op for local installs.
|
`("reasoning", chunk)` pairs, and logs the exchange.
|
||||||
|
"""
|
||||||
|
# SSRF guard for hosted mode. A user-supplied `endpoint_url` must not
|
||||||
|
# point at an internal or metadata address. This does nothing for a
|
||||||
|
# local install.
|
||||||
reason = netguard.endpoint_block_reason(url)
|
reason = netguard.endpoint_block_reason(url)
|
||||||
if reason:
|
if reason:
|
||||||
raise ProviderError(f"This endpoint can't be used — {reason}.")
|
raise ProviderError(f"This endpoint can't be used — {reason}.")
|
||||||
@@ -232,8 +252,8 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
if resp.status_code != 200:
|
if resp.status_code != 200:
|
||||||
detail = (await resp.aread()).decode(errors="replace")[:500]
|
detail = (await resp.aread()).decode(errors="replace")[:500]
|
||||||
raise ProviderError(self._friendly_http_error(resp.status_code, detail))
|
raise ProviderError(self._friendly_http_error(resp.status_code, detail))
|
||||||
# Some servers ignore stream=true and return one plain JSON
|
# Some servers ignore `stream=true` and return one plain
|
||||||
# body; buffer non-SSE lines so we can fall back to it.
|
# JSON body, so buffer the non-SSE lines to fall back to.
|
||||||
saw_sse = False
|
saw_sse = False
|
||||||
raw_lines: list[str] = []
|
raw_lines: list[str] = []
|
||||||
async for line in resp.aiter_lines():
|
async for line in resp.aiter_lines():
|
||||||
@@ -301,8 +321,11 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
async def complete(
|
async def complete(
|
||||||
self, system: str, user: str, *, temperature: float = 0.3, max_tokens: int = 400
|
self, system: str, user: str, *, temperature: float = 0.3, max_tokens: int = 400
|
||||||
) -> str:
|
) -> str:
|
||||||
"""Single non-streaming completion for background calls (summarization).
|
"""Runs a single non-streaming completion, for background calls such as
|
||||||
Unlike generate(), no story-continuation framing is added."""
|
summarization.
|
||||||
|
|
||||||
|
Unlike `generate()`, this adds no story-continuation framing.
|
||||||
|
"""
|
||||||
if not self.model:
|
if not self.model:
|
||||||
raise ProviderError("No model configured — set one in Settings.")
|
raise ProviderError("No model configured — set one in Settings.")
|
||||||
if self.api_mode == "completion":
|
if self.api_mode == "completion":
|
||||||
@@ -351,7 +374,7 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
return text.strip()
|
return text.strip()
|
||||||
|
|
||||||
async def embed(self, texts: list[str]) -> list[list[float]]:
|
async def embed(self, texts: list[str]) -> list[list[float]]:
|
||||||
"""POST /v1/embeddings; self.model is the embedding model here."""
|
"""POSTs to /v1/embeddings. Here `self.model` is the embedding model."""
|
||||||
if not self.model:
|
if not self.model:
|
||||||
raise ProviderError("No embedding model configured — set one in Settings.")
|
raise ProviderError("No embedding model configured — set one in Settings.")
|
||||||
url = f"{self.base_url}/embeddings"
|
url = f"{self.base_url}/embeddings"
|
||||||
@@ -388,8 +411,9 @@ class OpenAICompatibleProvider(Provider):
|
|||||||
f"model '{self.model}' exists. {detail}"
|
f"model '{self.model}' exists. {detail}"
|
||||||
)
|
)
|
||||||
if status == 429:
|
if status == 429:
|
||||||
# OpenRouter's shared free tier has a per-day cap; distinguish it
|
# OpenRouter's shared free tier has a per-day cap. Distinguish it
|
||||||
# from a short-term burst limit so the message is actionable.
|
# from a short-term burst limit, so the message tells the reader what
|
||||||
|
# to do.
|
||||||
if "free-models-per-day" in detail:
|
if "free-models-per-day" in detail:
|
||||||
return (
|
return (
|
||||||
"The free demo has hit its daily request limit (resets at "
|
"The free demo has hit its daily request limit (resets at "
|
||||||
|
|||||||
+591
-524
File diff suppressed because it is too large
Load Diff
@@ -1,10 +1,10 @@
|
|||||||
"""Visit analytics: one endpoint the browser writes to, one the owner reads.
|
"""Visit analytics: one endpoint the browser writes to, one the owner reads.
|
||||||
|
|
||||||
The split matters. `/collect` is public and takes exactly one fact — which
|
The split matters. `/collect` is public and accepts one fact, which page was
|
||||||
page was viewed — because anything a stranger can POST is a number a stranger
|
viewed, because anything a stranger can POST is a number a stranger can invent.
|
||||||
can invent. Everything the dashboard actually relies on (turns, adventures,
|
Everything the dashboard relies on, meaning turns, adventures, sign-ups, demo
|
||||||
sign-ups, demo spend, errors) is recorded server-side by the code performing
|
spend, and errors, is recorded on the server by the code that performs it, so
|
||||||
it, so those counts are as trustworthy as the app itself.
|
those counts are as trustworthy as the app itself.
|
||||||
|
|
||||||
The two reading endpoints are owner-only and 404 for everyone else, the same
|
The two reading endpoints are owner-only and 404 for everyone else, the same
|
||||||
way the AI Chat router does: a feature nobody else can use is better off not
|
way the AI Chat router does: a feature nobody else can use is better off not
|
||||||
@@ -27,7 +27,8 @@ def owner(
|
|||||||
db: Session = Depends(get_db),
|
db: Session = Depends(get_db),
|
||||||
user: models.User = Depends(auth.get_current_user),
|
user: models.User = Depends(auth.get_current_user),
|
||||||
) -> models.User:
|
) -> models.User:
|
||||||
"""Gate for the reading half. 404, not 403 — see the module docstring."""
|
"""Gates the reading half. It returns 404 rather than 403. See the module
|
||||||
|
docstring."""
|
||||||
if not auth.is_owner(user):
|
if not auth.is_owner(user):
|
||||||
raise HTTPException(404, "Not found")
|
raise HTTPException(404, "Not found")
|
||||||
return user
|
return user
|
||||||
@@ -39,10 +40,10 @@ Owner = Depends(owner)
|
|||||||
class Pageview(BaseModel):
|
class Pageview(BaseModel):
|
||||||
"""What the SPA reports on a page load or a route change.
|
"""What the SPA reports on a page load or a route change.
|
||||||
|
|
||||||
`first` marks a real page load rather than a client-side navigation: the
|
`first` marks a real page load rather than a client-side navigation. The
|
||||||
things that describe a *visit* rather than a *view* — where it came from,
|
facts that describe a visit rather than a view, which are where it came from,
|
||||||
on what kind of device, from which country — are recorded only then, so a
|
on what kind of device, and from which country, are recorded only on a page
|
||||||
visitor who clicks around five pages is still one referral.
|
load, so a visitor who clicks through five pages is still one referral.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
path: str = Field("", max_length=300)
|
path: str = Field("", max_length=300)
|
||||||
@@ -67,9 +68,9 @@ def collect(
|
|||||||
if auth.MULTI_USER
|
if auth.MULTI_USER
|
||||||
else auth.local_user(db)
|
else auth.local_user(db)
|
||||||
)
|
)
|
||||||
# The operator's own clicking is not traffic. Only in multi-user mode —
|
# The operator's own clicks are not traffic. This applies only in
|
||||||
# locally everyone is the owner, and excluding them would leave the
|
# multi-user mode. Locally every user is the owner, and excluding them would
|
||||||
# dashboard permanently empty on the machine it is developed on.
|
# leave the dashboard empty on the machine the app is developed on.
|
||||||
if auth.MULTI_USER and user is not None and auth.is_owner(user):
|
if auth.MULTI_USER and user is not None and auth.is_owner(user):
|
||||||
return Response(status_code=204)
|
return Response(status_code=204)
|
||||||
|
|
||||||
@@ -95,8 +96,10 @@ def summary(
|
|||||||
db: Session = Depends(get_db),
|
db: Session = Depends(get_db),
|
||||||
_user: models.User = Owner,
|
_user: models.User = Owner,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""The whole dashboard in one aggregate response — a few kilobytes however
|
"""Returns the whole dashboard in one aggregate response.
|
||||||
much traffic sits behind it."""
|
|
||||||
|
The response is a few kilobytes however much traffic is behind it.
|
||||||
|
"""
|
||||||
return analytics.summary(db, days)
|
return analytics.summary(db, days)
|
||||||
|
|
||||||
|
|
||||||
@@ -111,9 +114,9 @@ def access_log(
|
|||||||
) -> dict:
|
) -> dict:
|
||||||
"""A page of the access log, newest first.
|
"""A page of the access log, newest first.
|
||||||
|
|
||||||
Unlike /summary this returns rows about people, which is the whole point of
|
Unlike `/summary`, this returns rows about people, which is what it is for.
|
||||||
it — so it is behind the same owner gate, paged rather than dumped, and has
|
It is therefore behind the same owner gate, it is paged rather than returned
|
||||||
no counterpart the people it describes can reach.
|
in full, and the people it describes have no endpoint that reaches it.
|
||||||
"""
|
"""
|
||||||
page = accesslog.recent(
|
page = accesslog.recent(
|
||||||
db, limit=limit, before_id=before_id, kind=kind, query=q
|
db, limit=limit, before_id=before_id, kind=kind, query=q
|
||||||
|
|||||||
@@ -53,15 +53,19 @@ def me_payload(user: models.User, db: Session) -> dict:
|
|||||||
|
|
||||||
@router.get("/me")
|
@router.get("/me")
|
||||||
def me(request: Request, response: Response, db: Session = Depends(get_db)):
|
def me(request: Request, response: Response, db: Session = Depends(get_db)):
|
||||||
"""Who am I? In multi-user mode this also bootstraps the session: with no
|
"""Returns the current user.
|
||||||
(or an invalid) cookie it creates a guest user and sets one — the
|
|
||||||
frontend calls this on load and after any 401."""
|
In multi-user mode this also establishes the session. If the cookie is
|
||||||
|
missing or invalid, the endpoint creates a guest user and sets a cookie. The
|
||||||
|
frontend calls it on load and after any 401.
|
||||||
|
"""
|
||||||
if not auth.MULTI_USER:
|
if not auth.MULTI_USER:
|
||||||
user = auth.local_user(db)
|
user = auth.local_user(db)
|
||||||
else:
|
else:
|
||||||
user = auth.resolve_session_user(request, db)
|
user = auth.resolve_session_user(request, db)
|
||||||
if user is None:
|
if user is None:
|
||||||
# Each new guest is a database row — cap how fast one IP can mint them.
|
# Each new guest is a database row, so cap how fast one IP can
|
||||||
|
# create them.
|
||||||
limits.rate_limit("guest", request)
|
limits.rate_limit("guest", request)
|
||||||
user = models.User(is_guest=True)
|
user = models.User(is_guest=True)
|
||||||
db.add(user)
|
db.add(user)
|
||||||
@@ -80,8 +84,11 @@ def register(
|
|||||||
db: Session = Depends(get_db),
|
db: Session = Depends(get_db),
|
||||||
user: models.User = Depends(auth.get_current_user),
|
user: models.User = Depends(auth.get_current_user),
|
||||||
):
|
):
|
||||||
"""Upgrade the current guest in place — same user_id, so every adventure,
|
"""Upgrades the current guest in place.
|
||||||
scenario, script and setting they created as a guest is kept."""
|
|
||||||
|
The `user_id` does not change, so every adventure, scenario, script, and
|
||||||
|
setting they created as a guest is kept.
|
||||||
|
"""
|
||||||
if not auth.MULTI_USER:
|
if not auth.MULTI_USER:
|
||||||
raise HTTPException(400, "Accounts are disabled in local mode.")
|
raise HTTPException(400, "Accounts are disabled in local mode.")
|
||||||
limits.rate_limit("auth", request)
|
limits.rate_limit("auth", request)
|
||||||
|
|||||||
+28
-18
@@ -1,14 +1,15 @@
|
|||||||
"""AI Chat — a plain scratchpad for talking to a model directly.
|
"""AI Chat: a plain scratchpad for talking to a model directly.
|
||||||
|
|
||||||
Power users only (the AIDND_POWER_USERS email allowlist). Deliberately thin:
|
Power users reach it, which means the `AIDND_POWER_USERS` email allowlist. It is
|
||||||
no story context, no scripts, no world state, and nothing persisted — the
|
deliberately thin. It adds no story context, no scripts, and no world state, and
|
||||||
conversation lives in the browser and is posted up whole on each turn. It
|
it persists nothing. The conversation lives in the browser and is posted in full
|
||||||
exists to poke at models, prompts and endpoints without starting an adventure.
|
on each turn. It exists for testing models, prompts, and endpoints without
|
||||||
|
starting an adventure.
|
||||||
|
|
||||||
Model choice is free-form when the user brought their own API key. On the
|
Model choice is free when the user brought their own API key. On the shared demo
|
||||||
shared demo key it stays pinned to the AIDND_DEMO_MODELS whitelist, exactly as
|
key the model stays pinned to the `AIDND_DEMO_MODELS` allowlist, exactly as it is
|
||||||
turns are: the server funds that key, so it must not be able to reach paid
|
for turns. The server funds that key, so this page must not let it reach paid
|
||||||
models by way of this page.
|
models.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||||
@@ -41,10 +42,13 @@ PowerUser = Depends(power_user)
|
|||||||
def _resolve_model(
|
def _resolve_model(
|
||||||
settings: models.Settings, requested: str | None
|
settings: models.Settings, requested: str | None
|
||||||
) -> tuple[auth.ProviderConfig, str | None]:
|
) -> tuple[auth.ProviderConfig, str | None]:
|
||||||
"""Provider config for this chat, plus a note when the requested model was
|
"""Returns the provider config for this chat, plus a note when the requested
|
||||||
not honoured. The pinning rule itself lives in resolve_provider_config —
|
model was not used.
|
||||||
this only reports the substitution it made, so there is exactly one place
|
|
||||||
that decides what the demo key is allowed to talk to."""
|
The pinning rule lives in `resolve_provider_config`. This function only
|
||||||
|
reports the substitution that call made, so one place decides what the demo
|
||||||
|
key may talk to.
|
||||||
|
"""
|
||||||
cfg = auth.resolve_provider_config(settings, model_override=requested)
|
cfg = auth.resolve_provider_config(settings, model_override=requested)
|
||||||
wanted = (requested or "").strip()
|
wanted = (requested or "").strip()
|
||||||
if wanted and wanted != cfg.model:
|
if wanted and wanted != cfg.model:
|
||||||
@@ -60,9 +64,12 @@ async def chat_config(
|
|||||||
db: Session = Depends(get_db),
|
db: Session = Depends(get_db),
|
||||||
user: models.User = PowerUser,
|
user: models.User = PowerUser,
|
||||||
):
|
):
|
||||||
"""What this page can talk to: the resolved endpoint/model, whether model
|
"""Returns what this page can talk to.
|
||||||
choice is pinned to the demo whitelist, and the endpoint's model listing
|
|
||||||
(best effort — an unreachable endpoint just yields an empty list)."""
|
The response holds the resolved endpoint and model, whether model choice is
|
||||||
|
pinned to the demo allowlist, and the endpoint's model listing. The listing
|
||||||
|
is best effort, and an unreachable endpoint returns an empty list.
|
||||||
|
"""
|
||||||
settings = get_settings(db, user)
|
settings = get_settings(db, user)
|
||||||
cfg = auth.resolve_provider_config(settings)
|
cfg = auth.resolve_provider_config(settings)
|
||||||
listing = await list_endpoint_models(cfg)
|
listing = await list_endpoint_models(cfg)
|
||||||
@@ -82,8 +89,11 @@ async def chat_config(
|
|||||||
|
|
||||||
async def run_chat(cfg: auth.ProviderConfig, settings: models.Settings, payload: schemas.ChatRequest,
|
async def run_chat(cfg: auth.ProviderConfig, settings: models.Settings, payload: schemas.ChatRequest,
|
||||||
note: str | None, db: Session, user: models.User):
|
note: str | None, db: Session, user: models.User):
|
||||||
"""SSE generator mirroring the turn stream's event shape: reasoning/chunk
|
"""Streams the reply as SSE, using the turn stream's event shape.
|
||||||
while generating, then done — so the frontend reuses the same plumbing."""
|
|
||||||
|
The generator emits `reasoning` and `chunk` events while generating and then
|
||||||
|
a `done` event, so the frontend reuses the same code.
|
||||||
|
"""
|
||||||
if note:
|
if note:
|
||||||
yield sse({"type": "note", "detail": note})
|
yield sse({"type": "note", "detail": note})
|
||||||
provider = OpenAICompatibleProvider(
|
provider = OpenAICompatibleProvider(
|
||||||
|
|||||||
@@ -9,9 +9,11 @@ router = APIRouter(prefix="/api/debug", tags=["debug"])
|
|||||||
def recent_requests():
|
def recent_requests():
|
||||||
"""Most-recent-first log of provider requests/responses (no API keys).
|
"""Most-recent-first log of provider requests/responses (no API keys).
|
||||||
|
|
||||||
The log is a single process-wide ring buffer with no per-user
|
The log is a single process-wide ring buffer with no per-user attribution,
|
||||||
attribution, so in multi-user (hosted) mode it would leak other players'
|
so in multi-user mode, which is how a hosted deployment runs, it would expose
|
||||||
prompts — disabled there, available on local installs."""
|
other players' prompts. It is disabled there and available on a local
|
||||||
|
install.
|
||||||
|
"""
|
||||||
if auth.MULTI_USER:
|
if auth.MULTI_USER:
|
||||||
raise HTTPException(403, "The debug log is only available on local installs.")
|
raise HTTPException(403, "The debug log is only available on local installs.")
|
||||||
return debuglog.recent()
|
return debuglog.recent()
|
||||||
|
|||||||
@@ -54,9 +54,10 @@ def get_scenario(
|
|||||||
user: models.User = Depends(auth.get_current_user),
|
user: models.User = Depends(auth.get_current_user),
|
||||||
):
|
):
|
||||||
scenario = get_scenario_or_404(scenario_id, db, user)
|
scenario = get_scenario_or_404(scenario_id, db, user)
|
||||||
# Funnel step. Only for shared scenarios: opening one is the first sign a
|
# A funnel step, recorded for shared scenarios only. Opening one is the
|
||||||
# visitor is interested, whereas someone editing their own is already past
|
# first sign that a visitor is interested, and someone editing their own
|
||||||
# this point — and their titles are theirs, not a statistic.
|
# scenario is already past this point. Their titles are theirs rather than a
|
||||||
|
# statistic.
|
||||||
if scenario.is_public:
|
if scenario.is_public:
|
||||||
analytics.record_event(analytics.EV_SCENARIO_OPEN, user)
|
analytics.record_event(analytics.EV_SCENARIO_OPEN, user)
|
||||||
return scenario
|
return scenario
|
||||||
@@ -204,8 +205,9 @@ def import_scenario(
|
|||||||
if isinstance(schema, dict):
|
if isinstance(schema, dict):
|
||||||
fields["stat_schema"] = schema
|
fields["stat_schema"] = schema
|
||||||
|
|
||||||
# AI Dungeon bundles carry an `image` too, so this is worth honouring — but
|
# An AI Dungeon bundle also carries an `image`, so this reads it. The value
|
||||||
# it's untrusted input, hence sanitize() rather than a straight assignment.
|
# is untrusted input, so it goes through `sanitize()` rather than a direct
|
||||||
|
# assignment.
|
||||||
image = images.sanitize(bundle.get("image"), schemas.IMAGE_MAX)
|
image = images.sanitize(bundle.get("image"), schemas.IMAGE_MAX)
|
||||||
if image:
|
if image:
|
||||||
fields["image"] = image
|
fields["image"] = image
|
||||||
@@ -216,10 +218,10 @@ def import_scenario(
|
|||||||
scenario = models.Scenario(**fields, user_id=user.id)
|
scenario = models.Scenario(**fields, user_id=user.id)
|
||||||
if not scenario.title:
|
if not scenario.title:
|
||||||
scenario.title = "Imported Scenario"
|
scenario.title = "Imported Scenario"
|
||||||
# Raw-dict import bypasses the schemas — clamp to VARCHAR widths
|
# A raw-dict import bypasses the schemas, so truncate to the VARCHAR widths.
|
||||||
# (Postgres enforces them; see schemas.py). Column defaults haven't been
|
# Postgres enforces them. See `schemas.py`. Column defaults have not been
|
||||||
# applied yet at this point (that happens at flush), so a bundle with no
|
# applied yet, because that happens at flush, so a bundle with no `tags` key
|
||||||
# `tags` key leaves the attribute None — hence the `or ""`.
|
# leaves the attribute None, which is why the code says `or ""`.
|
||||||
scenario.title = scenario.title[:schemas.NAME_MAX]
|
scenario.title = scenario.title[:schemas.NAME_MAX]
|
||||||
scenario.tags = (scenario.tags or "")[:schemas.TAGS_MAX]
|
scenario.tags = (scenario.tags or "")[:schemas.TAGS_MAX]
|
||||||
db.add(scenario)
|
db.add(scenario)
|
||||||
|
|||||||
@@ -84,7 +84,7 @@ def test_script(
|
|||||||
db: Session = Depends(get_db),
|
db: Session = Depends(get_db),
|
||||||
user: models.User = Depends(auth.get_current_user),
|
user: models.User = Depends(auth.get_current_user),
|
||||||
):
|
):
|
||||||
"""Dry-run one hook against sample text — no AI call, no persistence."""
|
"""Runs one hook against sample text, making no AI call and storing nothing."""
|
||||||
script = get_script_or_404(script_id, db, user)
|
script = get_script_or_404(script_id, db, user)
|
||||||
limits.rate_limit("script-test", request, user)
|
limits.rate_limit("script-test", request, user)
|
||||||
result = run_hook(
|
result = run_hook(
|
||||||
@@ -145,7 +145,8 @@ def import_script(
|
|||||||
|
|
||||||
script = models.Script(
|
script = models.Script(
|
||||||
user_id=user.id,
|
user_id=user.id,
|
||||||
# Raw-dict import bypasses the schemas — clamp to the VARCHAR width.
|
# A raw-dict import bypasses the schemas, so truncate to the VARCHAR
|
||||||
|
# width.
|
||||||
name=(pick("name") or "Imported Script")[:schemas.NAME_MAX],
|
name=(pick("name") or "Imported Script")[:schemas.NAME_MAX],
|
||||||
description=pick("description"),
|
description=pick("description"),
|
||||||
library_js=pick("library", "library_js", "sharedLibrary"),
|
library_js=pick("library", "library_js", "sharedLibrary"),
|
||||||
|
|||||||
@@ -10,8 +10,11 @@ router = APIRouter(prefix="/api/settings", tags=["settings"])
|
|||||||
|
|
||||||
|
|
||||||
def get_settings(db: Session, user: models.User) -> models.Settings:
|
def get_settings(db: Session, user: models.User) -> models.Settings:
|
||||||
"""Per-user settings row, created on first access (Phase 8: settings —
|
"""Returns the user's settings row, creating it on first access.
|
||||||
endpoint, key, models, memory config — are per user, not global)."""
|
|
||||||
|
Phase 8 made settings per user rather than global. They cover the endpoint,
|
||||||
|
the key, the models, and the memory configuration.
|
||||||
|
"""
|
||||||
settings = (
|
settings = (
|
||||||
db.query(models.Settings).filter(models.Settings.user_id == user.id).first()
|
db.query(models.Settings).filter(models.Settings.user_id == user.id).first()
|
||||||
)
|
)
|
||||||
@@ -50,14 +53,16 @@ def update_settings(
|
|||||||
if embedding_model_changed:
|
if embedding_model_changed:
|
||||||
# Vectors from the old model have a different dimensionality/space;
|
# Vectors from the old model have a different dimensionality/space;
|
||||||
# clear them so the post-turn task re-embeds with the new model.
|
# clear them so the post-turn task re-embeds with the new model.
|
||||||
# (This user's adventures only — settings are per-user now.)
|
# This covers only this user's adventures, because settings are per
|
||||||
|
# user now.
|
||||||
#
|
#
|
||||||
# Both columns, and the flag. This is the one place that clears vectors
|
# Both columns, and the flag. This is the one place that clears vectors
|
||||||
# in bulk rather than through memorybank.set_vector, and when the
|
# in bulk rather than through memorybank.set_vector, and when the
|
||||||
# vectors moved to embedding_blob it kept nulling the old JSON column
|
# vectors moved to embedding_blob it kept nulling the old JSON column
|
||||||
# alone: the blob survived, `embedded` stayed true, and _embed_pending
|
# alone. The blob survived, `embedded` stayed true, and
|
||||||
# — which looks for embedded IS FALSE — never picked the rows up. The
|
# `_embed_pending`, which selects rows where `embedded IS FALSE`, never
|
||||||
# bank went on ranking against the previous model's vectors forever.
|
# found the rows. The bank kept ranking against the previous model's
|
||||||
|
# vectors.
|
||||||
owned = (
|
owned = (
|
||||||
db.query(models.Adventure.id)
|
db.query(models.Adventure.id)
|
||||||
.filter(models.Adventure.user_id == user.id)
|
.filter(models.Adventure.user_id == user.id)
|
||||||
@@ -70,15 +75,18 @@ def update_settings(
|
|||||||
# `embedded` drops these rows out of the catalogue query, so retrieval
|
# `embedded` drops these rows out of the catalogue query, so retrieval
|
||||||
# stops asking for them, and by the time _embed_pending puts one back
|
# stops asking for them, and by the time _embed_pending puts one back
|
||||||
# it has gone through set_vector, which evicts that entry. The rule
|
# it has gone through set_vector, which evicts that entry. The rule
|
||||||
# holds — anything that removes a memory from play self-corrects.
|
# holds: anything that removes a memory from play corrects itself.
|
||||||
db.commit()
|
db.commit()
|
||||||
return settings
|
return settings
|
||||||
|
|
||||||
|
|
||||||
async def list_endpoint_models(cfg: auth.ProviderConfig) -> dict:
|
async def list_endpoint_models(cfg: auth.ProviderConfig) -> dict:
|
||||||
"""GET the endpoint's /models listing. Doubles as a connectivity check, so
|
"""Fetches the endpoint's /models listing.
|
||||||
failures come back as {"ok": False, "detail": ...} rather than raising."""
|
|
||||||
# SSRF guard: never probe a non-public address the user pointed us at.
|
The call also serves as a connectivity check, so a failure returns
|
||||||
|
`{"ok": False, "detail": ...}` rather than raising.
|
||||||
|
"""
|
||||||
|
# SSRF guard. Never probe a non-public address the user supplied.
|
||||||
reason = await run_in_threadpool(netguard.endpoint_block_reason, cfg.endpoint_url)
|
reason = await run_in_threadpool(netguard.endpoint_block_reason, cfg.endpoint_url)
|
||||||
if reason:
|
if reason:
|
||||||
return {"ok": False, "detail": f"Can't reach that endpoint — {reason}."}
|
return {"ok": False, "detail": f"Can't reach that endpoint — {reason}."}
|
||||||
@@ -100,7 +108,8 @@ async def list_endpoint_models(cfg: auth.ProviderConfig) -> dict:
|
|||||||
data = resp.json()
|
data = resp.json()
|
||||||
models_available = [m.get("id", "?") for m in data.get("data", [])]
|
models_available = [m.get("id", "?") for m in data.get("data", [])]
|
||||||
except (ValueError, AttributeError, TypeError):
|
except (ValueError, AttributeError, TypeError):
|
||||||
pass # non-JSON or unexpected shape — connectivity is still confirmed
|
pass # The body is not JSON or has an unexpected shape. The endpoint
|
||||||
|
# is still reachable.
|
||||||
return {"ok": True, "models": models_available}
|
return {"ok": True, "models": models_available}
|
||||||
|
|
||||||
|
|
||||||
@@ -110,8 +119,11 @@ async def test_connection(
|
|||||||
db: Session = Depends(get_db),
|
db: Session = Depends(get_db),
|
||||||
user: models.User = Depends(auth.get_current_user),
|
user: models.User = Depends(auth.get_current_user),
|
||||||
):
|
):
|
||||||
"""Cheap connectivity check against whatever the turn engine would actually
|
"""Runs a cheap connectivity check against whatever the turn engine would use.
|
||||||
use — including the shared demo endpoint when the user has no key."""
|
|
||||||
|
That includes the shared demo endpoint, when the user has no key of their
|
||||||
|
own.
|
||||||
|
"""
|
||||||
limits.rate_limit("connection-test", request, user)
|
limits.rate_limit("connection-test", request, user)
|
||||||
settings = get_settings(db, user)
|
settings = get_settings(db, user)
|
||||||
return await list_endpoint_models(auth.resolve_provider_config(settings))
|
return await list_endpoint_models(auth.resolve_provider_config(settings))
|
||||||
|
|||||||
@@ -6,15 +6,19 @@ from ..database import get_db
|
|||||||
|
|
||||||
router = APIRouter(prefix="/api/story-cards", tags=["story-cards"])
|
router = APIRouter(prefix="/api/story-cards", tags=["story-cards"])
|
||||||
|
|
||||||
# AI Dungeon world-info / story-card array format. Its field names differ from
|
# The AI Dungeon world-info and story-card array format. Its field names differ
|
||||||
# our columns: value<->entry, title<->name, description<->notes. The extra
|
# from this app's columns: `value` maps to `entry`, `title` maps to `name`, and
|
||||||
# `useForCharacterCreation` flag has no equivalent here — ignored on import,
|
# `description` maps to `notes`. The extra `useForCharacterCreation` flag has no
|
||||||
# emitted as false on export so round-tripping through AI Dungeon stays valid.
|
# equivalent here. It is ignored on import and written as false on export, so a
|
||||||
|
# round trip through AI Dungeon stays valid.
|
||||||
|
|
||||||
|
|
||||||
def _visible_owner(scenario_id, adventure_id, db, user):
|
def _visible_owner(scenario_id, adventure_id, db, user):
|
||||||
"""Resolve the scenario/adventure a caller may *read* cards from (public
|
"""Resolves the scenario or adventure a caller may read cards from.
|
||||||
demo scenarios included), or 404/422."""
|
|
||||||
|
Public demo scenarios are included. The function raises a 404 or a 422 when
|
||||||
|
it cannot resolve one.
|
||||||
|
"""
|
||||||
if (scenario_id is None) == (adventure_id is None):
|
if (scenario_id is None) == (adventure_id is None):
|
||||||
raise HTTPException(422, "Provide exactly one of scenario_id or adventure_id")
|
raise HTTPException(422, "Provide exactly one of scenario_id or adventure_id")
|
||||||
if scenario_id is not None:
|
if scenario_id is not None:
|
||||||
|
|||||||
+110
-97
@@ -5,24 +5,25 @@ from pydantic import BaseModel, ConfigDict, Field, computed_field
|
|||||||
|
|
||||||
from . import images
|
from . import images
|
||||||
|
|
||||||
# Length caps (Phase 9). The VARCHAR ones are correctness, not just abuse
|
# Length caps (Phase 9). The VARCHAR caps are a correctness requirement rather
|
||||||
# limits: Postgres enforces column lengths (SQLite never did), so anything
|
# than only an abuse limit. Postgres enforces column lengths and SQLite never
|
||||||
# longer must be a 422 here rather than a 500 at INSERT. Text-column caps are
|
# did, so a longer value has to be a 422 here rather than a 500 at INSERT. The
|
||||||
# generous abuse ceilings a legitimate player won't hit.
|
# text-column caps are generous abuse ceilings that a legitimate player does not
|
||||||
NAME_MAX = 200 # titles/names — VARCHAR(200)
|
# reach.
|
||||||
TAGS_MAX = 500 # VARCHAR(500)
|
NAME_MAX = 200 # Titles and names. VARCHAR(200).
|
||||||
CARD_TYPE_MAX = 100 # VARCHAR(100)
|
TAGS_MAX = 500 # VARCHAR(500).
|
||||||
PROSE_MAX = 50_000 # memory, author's note, prompts, entries, notes...
|
CARD_TYPE_MAX = 100 # VARCHAR(100).
|
||||||
SCRIPT_MAX = 200_000 # one JS source
|
PROSE_MAX = 50_000 # Memory, author's note, prompts, entries, and notes.
|
||||||
ACTION_MAX = 20_000 # one player action
|
SCRIPT_MAX = 200_000 # One JavaScript source file.
|
||||||
|
ACTION_MAX = 20_000 # One player action.
|
||||||
MEMORY_TEXT_MAX = 5_000
|
MEMORY_TEXT_MAX = 5_000
|
||||||
# A scenario cover image, stored inline as a base64 data URI. 400x300 WebP at
|
# A scenario cover image, stored inline as a base64 data URI. A 400x300 WebP at
|
||||||
# the quality the editor encodes lands around 20-40 KB; 400 KB leaves room for
|
# the quality the editor encodes runs about 20 to 40 kB. A cap of 400 kB leaves
|
||||||
# a client that downscales less aggressively without letting anyone park a
|
# room for a client that downscales less aggressively, and it stops anyone from
|
||||||
# multi-megabyte PNG in a row that gets read on every list request.
|
# storing a multi-megabyte PNG in a row that every list request reads.
|
||||||
IMAGE_MAX = 400_000
|
IMAGE_MAX = 400_000
|
||||||
ICON_MAX = 16 # one emoji/glyph — VARCHAR(16)
|
ICON_MAX = 16 # One emoji or glyph. VARCHAR(16).
|
||||||
BRANCH_NAME_MAX = 80 # what a player called one line of the story — VARCHAR(80)
|
BRANCH_NAME_MAX = 80 # What a player called one line of the story. VARCHAR(80).
|
||||||
|
|
||||||
Name = Annotated[str, Field(max_length=NAME_MAX)]
|
Name = Annotated[str, Field(max_length=NAME_MAX)]
|
||||||
Tags = Annotated[str, Field(max_length=TAGS_MAX)]
|
Tags = Annotated[str, Field(max_length=TAGS_MAX)]
|
||||||
@@ -77,12 +78,12 @@ class ScenarioBase(BaseModel):
|
|||||||
authors_note: Prose = ""
|
authors_note: Prose = ""
|
||||||
ai_instructions: Prose = ""
|
ai_instructions: Prose = ""
|
||||||
tags: Tags = ""
|
tags: Tags = ""
|
||||||
# Cover art — an https URL or a base64 data URI. See app/images.py.
|
# Cover art, either an https URL or a base64 data URI. See `app/images.py`.
|
||||||
image: Image = ""
|
image: Image = ""
|
||||||
# Emoji/glyph shown when `image` is empty.
|
# The emoji or glyph shown when `image` is empty.
|
||||||
icon: Icon = ""
|
icon: Icon = ""
|
||||||
# Phase 12: RPG world-state template (stat defs, bands, rules, milestones).
|
# Phase 12: the RPG world-state template, holding stat definitions, bands,
|
||||||
# None means no RPG layer.
|
# rules, and milestones. `None` means the scenario has no RPG layer.
|
||||||
stat_schema: dict | None = None
|
stat_schema: dict | None = None
|
||||||
|
|
||||||
|
|
||||||
@@ -106,7 +107,7 @@ class ScenarioUpdate(BaseModel):
|
|||||||
|
|
||||||
class ScenarioOut(ORMModel, ScenarioBase):
|
class ScenarioOut(ORMModel, ScenarioBase):
|
||||||
id: int
|
id: int
|
||||||
is_public: bool = False # shared demo content — read-only for everyone
|
is_public: bool = False # Shared demo content, read-only for everyone.
|
||||||
created_at: datetime
|
created_at: datetime
|
||||||
updated_at: datetime
|
updated_at: datetime
|
||||||
story_cards: list[StoryCardOut] = []
|
story_cards: list[StoryCardOut] = []
|
||||||
@@ -120,8 +121,9 @@ class ScenarioListItem(ORMModel):
|
|||||||
tags: str
|
tags: str
|
||||||
is_public: bool = False
|
is_public: bool = False
|
||||||
updated_at: datetime
|
updated_at: datetime
|
||||||
# Read off the row so `image_url` can be derived, but excluded from the
|
# Read from the row so that `image_url` can be derived, and excluded from
|
||||||
# response: a list of base64 data URIs would be megabytes of JSON.
|
# the response, because a list of base64 data URIs would be megabytes of
|
||||||
|
# JSON.
|
||||||
image: str = Field("", exclude=True)
|
image: str = Field("", exclude=True)
|
||||||
icon: str = ""
|
icon: str = ""
|
||||||
|
|
||||||
@@ -136,7 +138,8 @@ class ScenarioListItem(ORMModel):
|
|||||||
class AdventureCreate(BaseModel):
|
class AdventureCreate(BaseModel):
|
||||||
scenario_id: int | None = None
|
scenario_id: int | None = None
|
||||||
title: Name | None = None
|
title: Name | None = None
|
||||||
# ${Placeholder} values collected from the player at start (AI Dungeon behavior).
|
# The `${Placeholder}` values collected from the player at the start, which
|
||||||
|
# is the AI Dungeon behavior.
|
||||||
placeholders: dict[str, str] = {}
|
placeholders: dict[str, str] = {}
|
||||||
|
|
||||||
|
|
||||||
@@ -151,27 +154,32 @@ class AdventureUpdate(BaseModel):
|
|||||||
|
|
||||||
|
|
||||||
class AdventureRefresh(BaseModel):
|
class AdventureRefresh(BaseModel):
|
||||||
"""Body for "Update from scenario". `placeholders` supplies answers the
|
"""The body for "Update from scenario".
|
||||||
adventure has no stored value for (see AdventureCreate.placeholders); they
|
|
||||||
are merged over the stored ones and saved."""
|
`placeholders` supplies answers the adventure has no stored value for. See
|
||||||
|
`AdventureCreate.placeholders`. The answers are merged over the stored ones
|
||||||
|
and saved.
|
||||||
|
"""
|
||||||
|
|
||||||
placeholders: dict[str, str] = {}
|
placeholders: dict[str, str] = {}
|
||||||
|
|
||||||
|
|
||||||
class RefreshPlan(BaseModel):
|
class RefreshPlan(BaseModel):
|
||||||
"""What a refresh would change — drives the confirm dialog."""
|
"""What a refresh would change. The confirm dialog is built from this."""
|
||||||
|
|
||||||
scenario_id: int
|
scenario_id: int
|
||||||
scenario_title: str
|
scenario_title: str
|
||||||
has_changes: bool
|
has_changes: bool
|
||||||
# field name -> {"old": ..., "new": ...}, only for fields that differ.
|
# Maps a field name to `{"old": ..., "new": ...}`, for differing fields
|
||||||
|
# only.
|
||||||
fields: dict[str, dict] = {}
|
fields: dict[str, dict] = {}
|
||||||
# {"added"|"updated"|"removed": [card name, ...]}
|
# Maps "added", "updated", or "removed" to a list of card names.
|
||||||
cards: dict[str, list[str]] = {}
|
cards: dict[str, list[str]] = {}
|
||||||
# {"added"|"removed": [stat path, ...]} — live values are otherwise kept.
|
# Maps "added" or "removed" to a list of stat paths. Live values are
|
||||||
|
# otherwise kept.
|
||||||
world_state: dict[str, list[str]] = {}
|
world_state: dict[str, list[str]] = {}
|
||||||
# ${Placeholder} names the scenario asks for that the adventure has no
|
# The `${Placeholder}` names the scenario asks for that the adventure has
|
||||||
# stored answer to; the client must collect these and send them back.
|
# no stored answer to. The client collects these and sends them back.
|
||||||
placeholders_needed: list[str] = []
|
placeholders_needed: list[str] = []
|
||||||
|
|
||||||
|
|
||||||
@@ -182,42 +190,44 @@ class ActionOut(ORMModel):
|
|||||||
type: str
|
type: str
|
||||||
text: str
|
text: str
|
||||||
reasoning: str | None = None
|
reasoning: str | None = None
|
||||||
# Phase 12: compact RPG state changes for this turn (from the model property).
|
# Phase 12: the compact RPG state changes for this turn, read from the
|
||||||
|
# model property.
|
||||||
world_changes: list[dict] = []
|
world_changes: list[dict] = []
|
||||||
# Retry history: how many attempts exist for this turn (0 = never retried)
|
# Retry history: how many attempts exist for this turn, where 0 means the
|
||||||
# and which one is live. The attempts themselves come from
|
# turn was never retried, and which attempt is live. The attempts themselves
|
||||||
# GET /actions/{id}/variants so this payload stays small.
|
# come from `GET /actions/{id}/variants`, so this payload stays small.
|
||||||
variant_count: int = 0
|
variant_count: int = 0
|
||||||
variant_index: int = 0
|
variant_index: int = 0
|
||||||
# SP9: the pager. How many takes this turn has, and which one is on screen —
|
# SP9: the pager, such as `2/4`. It reports how many attempts this turn has
|
||||||
# `2/4`. Keyed on the parent, so it counts the takes of *this* turn and not
|
# and which one is on screen. It is keyed on the parent, so it counts the
|
||||||
# every node that happens to share a depth, and so it keeps counting them
|
# attempts of this turn rather than every node that shares a depth, and it
|
||||||
# after one has been forked onto a branch of its own.
|
# keeps counting them after one has been forked onto its own branch.
|
||||||
#
|
#
|
||||||
# 1/1 for a turn nobody has retaken, which is most of them; the client draws
|
# A turn nobody has retaken reads 1/1, which is most turns, and the client
|
||||||
# no pager for a count of one. That is a different convention from
|
# draws no pager for a count of one. `variant_count` uses a different
|
||||||
# `variant_count`, which says 0 for the same case — those two are the
|
# convention and reports 0 for the same case. Those two fields are the
|
||||||
# pre-SP9 pair and SP8 drops them.
|
# pre-SP9 pair, and SP8 drops them.
|
||||||
take_count: int = 1
|
take_count: int = 1
|
||||||
take_index: int = 0
|
take_index: int = 0
|
||||||
# Which line this node is on, so the pager can tell the two kinds of step
|
# Which line this node is on, so the pager can distinguish the two kinds of
|
||||||
# apart without asking the server first: a take on this branch is a leaf
|
# step without asking the server. An attempt on this branch is a leaf with
|
||||||
# with nothing under it, and showing it is a local matter; a take on another
|
# nothing below it, so showing it is a local change. An attempt on another
|
||||||
# branch has a story of its own, and going there is a branch switch.
|
# branch has a story of its own, so moving to it is a branch switch.
|
||||||
branch_id: int | None = None
|
branch_id: int | None = None
|
||||||
created_at: datetime
|
created_at: datetime
|
||||||
|
|
||||||
|
|
||||||
class VariantOut(BaseModel):
|
class VariantOut(BaseModel):
|
||||||
# Since SP4 every attempt is its own node, so each one has an id — and the
|
# Since SP4 every attempt is its own node, so each one has an id, and the
|
||||||
# client needs it: forking is addressed by the attempt being taken, not by
|
# client needs that id. A fork is addressed by the attempt being promoted,
|
||||||
# its ordinal in a group that renumbers whenever one is added.
|
# not by its position in a group that renumbers whenever an attempt is
|
||||||
|
# added.
|
||||||
id: int
|
id: int
|
||||||
index: int
|
index: int
|
||||||
text: str
|
text: str
|
||||||
reasoning: str | None = None
|
reasoning: str | None = None
|
||||||
# See ActionOut.branch_id: it decides whether choosing this take is a local
|
# See `ActionOut.branch_id`. It decides whether choosing this attempt is a
|
||||||
# step or a branch switch.
|
# local step or a branch switch.
|
||||||
branch_id: int | None = None
|
branch_id: int | None = None
|
||||||
created_at: str | None = None
|
created_at: str | None = None
|
||||||
active: bool = False
|
active: bool = False
|
||||||
@@ -230,12 +240,11 @@ class VariantSelect(BaseModel):
|
|||||||
class BranchOut(ORMModel):
|
class BranchOut(ORMModel):
|
||||||
"""One line through the story tree (Phase 14, SP5).
|
"""One line through the story tree (Phase 14, SP5).
|
||||||
|
|
||||||
Enough to draw the tree and nothing more: `fork_depth` is where this line
|
This carries enough to draw the tree and nothing more. `fork_depth` is where
|
||||||
leaves its parent and `depth` is where it currently ends, so a fork is two
|
this line leaves its parent, and `depth` is where it currently ends, so a
|
||||||
numbers rather than a walk. `own_actions` counts the turns played on this
|
fork is two numbers rather than a walk. `own_actions` counts the turns played
|
||||||
branch itself — the rest of its story is borrowed from its ancestors, which
|
on this branch itself. The rest of its story is borrowed from its ancestors,
|
||||||
is the whole point and also why the number is smaller than the reader
|
which is why the number is smaller than a reader expects.
|
||||||
expects.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
id: int
|
id: int
|
||||||
@@ -244,14 +253,14 @@ class BranchOut(ORMModel):
|
|||||||
depth: int
|
depth: int
|
||||||
own_actions: int = 0
|
own_actions: int = 0
|
||||||
is_head: bool = False
|
is_head: bool = False
|
||||||
# NULL for a branch nobody has named. The client draws those from the fork
|
# NULL for a branch nobody has named. The client labels those from the fork
|
||||||
# depth rather than the server inventing one — see the column comment.
|
# depth rather than the server inventing a name. See the column comment.
|
||||||
name: str | None = None
|
name: str | None = None
|
||||||
created_at: datetime
|
created_at: datetime
|
||||||
|
|
||||||
|
|
||||||
class BranchRename(BaseModel):
|
class BranchRename(BaseModel):
|
||||||
"""A name a player chose, or `null` to go back to being unnamed."""
|
"""A name a player chose, or `null` to make the branch unnamed again."""
|
||||||
|
|
||||||
name: Annotated[str, Field(max_length=BRANCH_NAME_MAX)] | None = None
|
name: Annotated[str, Field(max_length=BRANCH_NAME_MAX)] | None = None
|
||||||
|
|
||||||
@@ -263,23 +272,23 @@ class ActionUpdate(BaseModel):
|
|||||||
class ActionCreate(BaseModel):
|
class ActionCreate(BaseModel):
|
||||||
type: Literal["do", "say", "story", "continue"]
|
type: Literal["do", "say", "story", "continue"]
|
||||||
text: ActionText = ""
|
text: ActionText = ""
|
||||||
# The node this action is played after (SP9). Omitted means "the tip",
|
# The node this action is played after (SP9). Omitting it means the tip,
|
||||||
# which is every ordinary turn.
|
# which is what every ordinary turn uses.
|
||||||
#
|
#
|
||||||
# Naming a take the story moved past is how a branch gets made: stepping
|
# Naming an attempt the story moved past is what creates a branch. Stepping
|
||||||
# between takes costs nothing and creates nothing, and the fork happens on
|
# between attempts costs nothing and creates nothing, and the fork happens
|
||||||
# the first thing written below one. That is the only moment the player has
|
# on the first text written below one. That is the first moment the player
|
||||||
# said which line they mean — before it, they were reading.
|
# states which line they mean. Before it, they were reading.
|
||||||
after_id: int | None = None
|
after_id: int | None = None
|
||||||
|
|
||||||
|
|
||||||
class TakeCreate(BaseModel):
|
class TakeCreate(BaseModel):
|
||||||
"""Another take of a turn (SP9).
|
"""Another attempt at a turn (SP9).
|
||||||
|
|
||||||
`text` is what the player is saying instead, and is theirs to write only
|
`text` is what the player says instead, and it applies only when the turn was
|
||||||
when the turn was theirs. An AI turn's other take is generated, so the field
|
the player's. An AI turn's other attempt is generated, so the field is
|
||||||
is ignored there rather than refused — the client asks the same way for both
|
ignored there rather than rejected. The client makes the same request for
|
||||||
and the node type decides what happens.
|
both, and the node type decides what happens.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
text: ActionText = ""
|
text: ActionText = ""
|
||||||
@@ -298,9 +307,9 @@ class AdventureOut(ORMModel):
|
|||||||
created_at: datetime
|
created_at: datetime
|
||||||
updated_at: datetime
|
updated_at: datetime
|
||||||
story_cards: list[StoryCardOut] = []
|
story_cards: list[StoryCardOut] = []
|
||||||
# The NEWEST window of the story, not all of it — older pages arrive from
|
# The newest window of the story, not all of it. Older pages arrive from
|
||||||
# GET /{id}/actions as the reader scrolls up. `action_count` is the whole
|
# `GET /{id}/actions` as the reader scrolls up. `action_count` is the whole
|
||||||
# story's length, which is how the client knows there is more above.
|
# story's length, which is how the client knows more actions exist above.
|
||||||
actions: list[ActionOut] = []
|
actions: list[ActionOut] = []
|
||||||
action_count: int = 0
|
action_count: int = 0
|
||||||
|
|
||||||
@@ -310,7 +319,7 @@ class ActionPage(BaseModel):
|
|||||||
|
|
||||||
actions: list[ActionOut] = []
|
actions: list[ActionOut] = []
|
||||||
total: int = 0
|
total: int = 0
|
||||||
# Whether anything older than this slice exists. Computed server-side so
|
# Whether anything older than this slice exists. The server computes it, so
|
||||||
# the client never has to do arithmetic on positions to find the end.
|
# the client never has to do arithmetic on positions to find the end.
|
||||||
has_more: bool = False
|
has_more: bool = False
|
||||||
|
|
||||||
@@ -348,10 +357,10 @@ class AdventureListItem(ORMModel):
|
|||||||
title: str
|
title: str
|
||||||
updated_at: datetime
|
updated_at: datetime
|
||||||
action_count: int = 0
|
action_count: int = 0
|
||||||
# "Where you left off" — the tail of the most recent narrative beat, so a
|
# The end of the most recent narration, so a Continue card can show the
|
||||||
# Continue card can show the story instead of just a turn count.
|
# story rather than only a turn count.
|
||||||
snippet: str = ""
|
snippet: str = ""
|
||||||
# Cover art inherited from the parent scenario (see app/images.py).
|
# Cover art inherited from the parent scenario. See `app/images.py`.
|
||||||
image_url: str = ""
|
image_url: str = ""
|
||||||
icon: str = ""
|
icon: str = ""
|
||||||
|
|
||||||
@@ -403,8 +412,9 @@ class AdventureScriptOut(ORMModel):
|
|||||||
input_js: str
|
input_js: str
|
||||||
context_js: str
|
context_js: str
|
||||||
output_js: str
|
output_js: str
|
||||||
# Set by the router (not stored): True when a syncable library version
|
# The router sets this field, which is not stored. It is `True` when a
|
||||||
# exists whose code differs from this copy; None when nothing to sync.
|
# syncable library version exists whose code differs from this copy, and
|
||||||
|
# `None` when there is nothing to sync from.
|
||||||
out_of_date: bool | None = None
|
out_of_date: bool | None = None
|
||||||
|
|
||||||
|
|
||||||
@@ -419,9 +429,9 @@ class AdventureScriptUpdate(BaseModel):
|
|||||||
# ---------- Auth (Phase 8) ----------
|
# ---------- Auth (Phase 8) ----------
|
||||||
|
|
||||||
class AuthCredentials(BaseModel):
|
class AuthCredentials(BaseModel):
|
||||||
email: Annotated[str, Field(max_length=320)] # VARCHAR(320)
|
email: Annotated[str, Field(max_length=320)] # VARCHAR(320).
|
||||||
# Upper bound keeps scrypt cost flat — hashing megabyte "passwords" is CPU
|
# The upper bound keeps the scrypt cost constant. Without it, hashing a
|
||||||
# an attacker would otherwise get for free.
|
# megabyte password would give an attacker free CPU time.
|
||||||
password: Annotated[str, Field(max_length=128)]
|
password: Annotated[str, Field(max_length=128)]
|
||||||
|
|
||||||
|
|
||||||
@@ -429,7 +439,8 @@ class AuthCredentials(BaseModel):
|
|||||||
|
|
||||||
class SettingsOut(ORMModel):
|
class SettingsOut(ORMModel):
|
||||||
endpoint_url: str
|
endpoint_url: str
|
||||||
# The key itself is never echoed back (encrypted at rest, write-only).
|
# The key itself is never returned. It is encrypted at rest and
|
||||||
|
# write-only.
|
||||||
has_api_key: bool
|
has_api_key: bool
|
||||||
model: str
|
model: str
|
||||||
api_mode: str
|
api_mode: str
|
||||||
@@ -449,12 +460,12 @@ ScenarioOut.model_rebuild()
|
|||||||
|
|
||||||
|
|
||||||
# ---------- AI Chat (power users) ----------
|
# ---------- AI Chat (power users) ----------
|
||||||
# A scratchpad for talking to a model directly, with no story framing. Nothing
|
# A scratchpad for talking to a model directly, with no story framing. The
|
||||||
# is persisted server-side, so these caps are purely per-request abuse limits.
|
# server persists nothing, so these caps are per-request abuse limits only.
|
||||||
|
|
||||||
CHAT_MESSAGE_MAX = 100_000 # one message
|
CHAT_MESSAGE_MAX = 100_000 # One message.
|
||||||
CHAT_TOTAL_MAX = 400_000 # whole conversation sent up per request
|
CHAT_TOTAL_MAX = 400_000 # The whole conversation sent per request.
|
||||||
CHAT_MESSAGES_MAX = 200 # turns per request
|
CHAT_MESSAGES_MAX = 200 # Turns per request.
|
||||||
|
|
||||||
|
|
||||||
class ChatMessage(BaseModel):
|
class ChatMessage(BaseModel):
|
||||||
@@ -464,22 +475,24 @@ class ChatMessage(BaseModel):
|
|||||||
|
|
||||||
class ChatRequest(BaseModel):
|
class ChatRequest(BaseModel):
|
||||||
messages: Annotated[list[ChatMessage], Field(min_length=1, max_length=CHAT_MESSAGES_MAX)]
|
messages: Annotated[list[ChatMessage], Field(min_length=1, max_length=CHAT_MESSAGES_MAX)]
|
||||||
# Empty/omitted = fall back to the user's configured model.
|
# If this field is empty or omitted, the user's configured model is used.
|
||||||
model: Name | None = None
|
model: Name | None = None
|
||||||
temperature: Annotated[float, Field(ge=0, le=5)] | None = None
|
temperature: Annotated[float, Field(ge=0, le=5)] | None = None
|
||||||
max_tokens: Annotated[int, Field(ge=1, le=100_000)] | None = None
|
max_tokens: Annotated[int, Field(ge=1, le=100_000)] | None = None
|
||||||
|
|
||||||
|
|
||||||
class SettingsUpdate(BaseModel):
|
class SettingsUpdate(BaseModel):
|
||||||
endpoint_url: Annotated[str, Field(max_length=500)] | None = None # VARCHAR(500)
|
endpoint_url: Annotated[str, Field(max_length=500)] | None = None # VARCHAR(500).
|
||||||
# Encryption expands the stored value ~4/3 into the same VARCHAR(500):
|
# Encryption expands the stored value by about four thirds into the same
|
||||||
# 256 plaintext chars is the largest safe input ("enc:" + Fernet + base64).
|
# VARCHAR(500), so 256 plaintext characters is the largest safe input. The
|
||||||
|
# stored form is "enc:" plus Fernet plus base64.
|
||||||
api_key: Annotated[str, Field(max_length=256)] | None = None
|
api_key: Annotated[str, Field(max_length=256)] | None = None
|
||||||
model: Name | None = None
|
model: Name | None = None
|
||||||
api_mode: Annotated[str, Field(max_length=20)] | None = None
|
api_mode: Annotated[str, Field(max_length=20)] | None = None
|
||||||
temperature: Annotated[float, Field(ge=0, le=5)] | None = None
|
temperature: Annotated[float, Field(ge=0, le=5)] | None = None
|
||||||
max_output_tokens: Annotated[int, Field(ge=1, le=100_000)] | None = None
|
max_output_tokens: Annotated[int, Field(ge=1, le=100_000)] | None = None
|
||||||
# -1 = explicitly off (sends `reasoning: {effort: none}`); 0 = send nothing.
|
# A value of -1 turns reasoning off explicitly, which sends
|
||||||
|
# `reasoning: {effort: none}`. A value of 0 sends nothing.
|
||||||
reasoning_max_tokens: Annotated[int, Field(ge=-1, le=100_000)] | None = None
|
reasoning_max_tokens: Annotated[int, Field(ge=-1, le=100_000)] | None = None
|
||||||
context_token_budget: Annotated[int, Field(ge=256, le=200_000)] | None = None
|
context_token_budget: Annotated[int, Field(ge=256, le=200_000)] | None = None
|
||||||
narrator_prompt: Prose | None = None
|
narrator_prompt: Prose | None = None
|
||||||
|
|||||||
@@ -94,8 +94,9 @@ def run_hook(
|
|||||||
story_cards: list[dict],
|
story_cards: list[dict],
|
||||||
info: dict,
|
info: dict,
|
||||||
) -> HookResult:
|
) -> HookResult:
|
||||||
"""Run one modifier hook. Never raises: failures come back as .error with
|
"""Run one modifier hook. This function never raises. Failures return as
|
||||||
text/state/cards unchanged, so a bad script can't break a turn."""
|
`.error` with text, state, and cards unchanged, so a bad script cannot
|
||||||
|
break a turn."""
|
||||||
unchanged = HookResult(text=text, state=state, story_cards=story_cards)
|
unchanged = HookResult(text=text, state=state, story_cards=story_cards)
|
||||||
source = f"{library_js}\n;\n{hook_js}" if library_js.strip() else hook_js
|
source = f"{library_js}\n;\n{hook_js}" if library_js.strip() else hook_js
|
||||||
if not source.strip():
|
if not source.strip():
|
||||||
|
|||||||
@@ -24,19 +24,20 @@ class ScriptPipeline:
|
|||||||
return msg if isinstance(msg, str) and msg.strip() else None
|
return msg if isinstance(msg, str) and msg.strip() else None
|
||||||
|
|
||||||
def _history(self) -> list[dict]:
|
def _history(self) -> list[dict]:
|
||||||
# The path, not `adventure.actions` — that collection is every branch's
|
# Read the path rather than `adventure.actions`. That collection holds
|
||||||
# actions, and this is the documented history API a user script reads.
|
# every branch's actions, and this is the documented history API a user
|
||||||
# Handing a script the siblings of the turn it is running on would be
|
# script reads. Giving a script the siblings of the turn it is running on
|
||||||
# the same bug as building a prompt from them, only user-visible.
|
# would be the same bug as building a prompt from them, and visible to
|
||||||
|
# the user.
|
||||||
#
|
#
|
||||||
# `story_actions` also drops blank-text rows, which `adventure.actions`
|
# `story_actions` also drops rows with blank text, which
|
||||||
# kept, so this array is shorter than it used to be for an adventure
|
# `adventure.actions` kept, so this array is shorter than it was for an
|
||||||
# that has any — and `info.actionCount` counts the same way. That is
|
# adventure that has any such rows. `info.actionCount` counts the same
|
||||||
# deliberate: a row with no text is this app's bookkeeping, it has no
|
# way. That is intended. A row with no text is this app's bookkeeping, it
|
||||||
# counterpart in the AI Dungeon history a ported script was written
|
# has no counterpart in the AI Dungeon history a ported script was
|
||||||
# against, and the prompt has never included one. A script keyed on
|
# written against, and the prompt has never included one. A script keyed
|
||||||
# "every N actions" will land on different turns than it did before
|
# on every N actions lands on different turns than it did before phase
|
||||||
# phase 14; there is no reading of this that is compatible with both.
|
# 14, and no reading of this is compatible with both.
|
||||||
return [
|
return [
|
||||||
{"text": a.text, "rawText": a.text, "type": a.type}
|
{"text": a.text, "rawText": a.text, "type": a.type}
|
||||||
for a in context_history.story_actions(self.adventure)
|
for a in context_history.story_actions(self.adventure)
|
||||||
|
|||||||
+15
-13
@@ -1,18 +1,19 @@
|
|||||||
"""Phase 8 — secrets and crypto primitives for optional accounts.
|
"""Phase 8: secrets and crypto primitives for optional accounts.
|
||||||
|
|
||||||
Everything keys off one server-side secret:
|
Everything derives from one server-side secret:
|
||||||
- session cookies are HMAC-signed with it,
|
|
||||||
- stored LLM API keys are Fernet-encrypted with a key derived from it.
|
|
||||||
|
|
||||||
The secret comes from AIDND_SECRET_KEY, or is auto-generated once into
|
* Session cookies are HMAC-signed with it.
|
||||||
`secret.key` next to the database so local installs and Docker volumes work
|
* Stored LLM API keys are Fernet-encrypted with a key derived from it.
|
||||||
with zero configuration (losing the file logs everyone out and orphans
|
|
||||||
stored API keys — users just re-enter them). Multi-user deploys must set the
|
|
||||||
env var: hosted filesystems are ephemeral, and a secret.key regenerated on
|
|
||||||
every deploy would silently log out all users each time.
|
|
||||||
|
|
||||||
Passwords use hashlib.scrypt (stdlib, OpenSSL-backed) so we don't need a
|
The secret comes from `AIDND_SECRET_KEY`, or it is generated once into
|
||||||
separate hashing dependency.
|
`secret.key` next to the database, so a local install and a Docker volume work
|
||||||
|
with no configuration. Losing that file logs everyone out and makes the stored
|
||||||
|
API keys unreadable, and users then re-enter them. A multi-user deployment has
|
||||||
|
to set the environment variable, because a hosted filesystem is ephemeral and a
|
||||||
|
`secret.key` regenerated on every deploy would log out every user each time.
|
||||||
|
|
||||||
|
Passwords use `hashlib.scrypt`, which is in the standard library and backed by
|
||||||
|
OpenSSL, so this needs no separate hashing dependency.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import base64
|
import base64
|
||||||
@@ -81,7 +82,8 @@ def verify_password(password: str, stored: str) -> bool:
|
|||||||
|
|
||||||
|
|
||||||
# ---------- Session tokens ----------
|
# ---------- Session tokens ----------
|
||||||
# "v1.<user_id>.<hmac>" — no expiry (long-lived guest sessions are the point).
|
# The token is "v1.<user_id>.<hmac>". It does not expire, because a long-lived
|
||||||
|
# guest session is what this is for.
|
||||||
|
|
||||||
def sign_session(user_id: int) -> str:
|
def sign_session(user_id: int) -> str:
|
||||||
payload = f"v1.{user_id}"
|
payload = f"v1.{user_id}"
|
||||||
|
|||||||
+8
-7
@@ -8,11 +8,11 @@ adventure copies the scenario's story cards and scripts into the adventure, so
|
|||||||
the seeded scripts run for guests too.
|
the seeded scripts run for guests too.
|
||||||
|
|
||||||
Seed files are the source of truth for demo content: a scenario is inserted if
|
Seed files are the source of truth for demo content: a scenario is inserted if
|
||||||
missing, and reconciled in place when a seed file's content changes (so edits
|
missing, and reconciled in place when a seed file's content changes, so an edit
|
||||||
ship on the next deploy). When a seed already matches, nothing is written, so
|
ships on the next deploy. When a seed already matches, nothing is written, so
|
||||||
this stays cheap to run on every boot. Existing adventures already started from
|
this stays cheap to run on every boot. An adventure already started from a demo
|
||||||
a demo keep their own copied cards/scripts and are unaffected — only new
|
keeps its own copied cards and scripts and is unchanged. Only a new adventure
|
||||||
adventures pick up the updated content.
|
picks up the updated content.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
@@ -122,8 +122,9 @@ def _insert_scenario(db, data: dict) -> None:
|
|||||||
|
|
||||||
def _update_scenario(db, scenario: models.Scenario, data: dict) -> None:
|
def _update_scenario(db, scenario: models.Scenario, data: dict) -> None:
|
||||||
_apply_scalars(scenario, data)
|
_apply_scalars(scenario, data)
|
||||||
# Replace child content wholesale — demo content is server-owned and cheap
|
# Replace the child content in full. Demo content is owned by the server and
|
||||||
# to rebuild, and this keeps the scenario row (and adventure FKs) intact.
|
# cheap to rebuild, and replacing it this way keeps the scenario row, and the
|
||||||
|
# adventure foreign keys that point at it, intact.
|
||||||
for card in list(scenario.story_cards):
|
for card in list(scenario.story_cards):
|
||||||
db.delete(card)
|
db.delete(card)
|
||||||
for script in list(scenario.scripts):
|
for script in list(scenario.scripts):
|
||||||
|
|||||||
+144
-142
@@ -1,25 +1,24 @@
|
|||||||
"""Phase 14 — putting nodes on the story tree.
|
"""Phase 14: writes nodes onto the story tree.
|
||||||
|
|
||||||
The write half of the tree. Which branch a new node hangs off, what depth it
|
This module is the write half of the tree. It decides which branch a new node
|
||||||
gets, and where an adventure's head points all live here, because every one of
|
goes on, what depth the node gets, and where the adventure's head points. All
|
||||||
them is the kind of thing that is silently wrong when it is spread across four
|
three decisions live here because a mistake in any of them is silent. A node
|
||||||
call sites: a node written without a branch is a node no read can see, and it
|
written without a branch is invisible to every read, and nothing raises an
|
||||||
fails by disappearing rather than by raising.
|
error.
|
||||||
|
|
||||||
The read half — the lineage clause that turns a branch into "this story" —
|
The read half is `context/lineage.py`. It turns a branch into the set of nodes
|
||||||
lives beside it in `context/lineage.py`.
|
that make up one story.
|
||||||
|
|
||||||
Until forking ships there is exactly one branch per adventure and `depth` is
|
Until forking ships, each adventure has one branch and `depth` mirrors `index`,
|
||||||
the number `index` already held, so everything in this module is bookkeeping
|
so nothing here changes observable behavior yet. That is intentional. By the
|
||||||
that changes nothing observable. That is the point: by the time a read depends
|
time reads depend on these columns, every row already has them, including the
|
||||||
on these columns, every row has them — including the rows written between the
|
rows written between the two deploys that no migration visits.
|
||||||
two deploys, which no migration will ever visit.
|
|
||||||
|
|
||||||
SP2 added `place_new_nodes`, which the session calls on every flush. Wiring the
|
SP2 added `place_new_nodes`, which runs on every flush. Wiring up individual
|
||||||
call sites was enough while nothing read the columns; now that reads select on
|
call sites worked while nothing read the columns. Now that reads filter on them,
|
||||||
them, "every writer remembers" is a promise that has to hold for every fixture,
|
relying on each writer to remember would also mean relying on every fixture,
|
||||||
script and test ever written too, and its breach is a story quietly missing
|
script, and test. A missed call produces a story with missing turns, so the
|
||||||
turns. So the invariant is enforced at the flush instead of asked for.
|
flush enforces the rule instead.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import copy
|
import copy
|
||||||
@@ -30,18 +29,19 @@ from sqlalchemy.orm import Session
|
|||||||
from . import models
|
from . import models
|
||||||
from .context import lineage
|
from .context import lineage
|
||||||
|
|
||||||
# The head depth of an adventure with no actions. Keeps "the next node goes at
|
# Head depth of an adventure that has no actions. Using -1 keeps the rule "the
|
||||||
# head_depth + 1" true with no special case, and mirrors migrations.NO_DEPTH.
|
# next node goes at head_depth + 1" true without a special case. This matches
|
||||||
|
# `migrations.NO_DEPTH`.
|
||||||
NO_DEPTH = -1
|
NO_DEPTH = -1
|
||||||
|
|
||||||
|
|
||||||
def root_branch(db: Session, adventure: models.Adventure) -> models.Branch:
|
def root_branch(db: Session, adventure: models.Adventure) -> models.Branch:
|
||||||
"""The adventure's root branch, created on first use.
|
"""Returns the adventure's root branch, creating it on first use.
|
||||||
|
|
||||||
Get-or-create rather than created-with-the-adventure, because the adventures
|
This function gets or creates the branch instead of creating it alongside
|
||||||
that need one most are the ones that already exist: a bundle being imported,
|
the adventure. The adventures that need a root branch are usually ones that
|
||||||
a fixture built straight through the ORM, or a database whose migration ran
|
already exist: an imported bundle, a fixture built through the ORM, or a
|
||||||
before this code shipped.
|
database migrated before this code shipped.
|
||||||
"""
|
"""
|
||||||
branch = (
|
branch = (
|
||||||
db.query(models.Branch)
|
db.query(models.Branch)
|
||||||
@@ -54,11 +54,10 @@ def root_branch(db: Session, adventure: models.Adventure) -> models.Branch:
|
|||||||
)
|
)
|
||||||
if branch is not None:
|
if branch is not None:
|
||||||
return branch
|
return branch
|
||||||
# Inserted through Core rather than through the unit of work, because this
|
# Use a Core insert instead of the ORM. `place_new_nodes` can call this
|
||||||
# also runs from `place_new_nodes` inside a flush, and a nested ORM flush
|
# during a flush, and a nested ORM flush raises an error. Both paths share
|
||||||
# inside a flush raises. Same transaction either way, so it rolls back with
|
# one transaction. The lineage refers to the branch's own id, so it needs a
|
||||||
# everything else. The lineage names the branch's own id, so it takes a
|
# second statement, which runs once per adventure.
|
||||||
# second statement — once per adventure, ever.
|
|
||||||
new_id = db.execute(
|
new_id = db.execute(
|
||||||
insert(models.Branch).values(
|
insert(models.Branch).values(
|
||||||
adventure_id=adventure.id,
|
adventure_id=adventure.id,
|
||||||
@@ -77,52 +76,50 @@ def root_branch(db: Session, adventure: models.Adventure) -> models.Branch:
|
|||||||
|
|
||||||
|
|
||||||
def head_branch(db: Session, adventure: models.Adventure) -> models.Branch:
|
def head_branch(db: Session, adventure: models.Adventure) -> models.Branch:
|
||||||
"""The branch new nodes are played onto."""
|
"""Returns the branch that new nodes are played onto."""
|
||||||
if adventure.head_branch_id is not None:
|
if adventure.head_branch_id is not None:
|
||||||
branch = db.get(models.Branch, adventure.head_branch_id)
|
branch = db.get(models.Branch, adventure.head_branch_id)
|
||||||
if branch is not None:
|
if branch is not None:
|
||||||
return branch
|
return branch
|
||||||
# A head naming a branch that is gone is a bug somewhere else. Recover
|
# The head points at a branch that no longer exists, which means a bug
|
||||||
# onto the root instead of refusing to play — the alternative is an
|
# elsewhere. Fall back to the root instead of refusing to play,
|
||||||
# adventure nobody can add to.
|
# otherwise the adventure becomes unusable.
|
||||||
branch = root_branch(db, adventure)
|
branch = root_branch(db, adventure)
|
||||||
adventure.head_branch_id = branch.id
|
adventure.head_branch_id = branch.id
|
||||||
return branch
|
return branch
|
||||||
|
|
||||||
|
|
||||||
def fork(db: Session, adventure: models.Adventure, node: models.Action) -> models.Branch:
|
def fork(db: Session, adventure: models.Adventure, node: models.Action) -> models.Branch:
|
||||||
"""Take the story down `node`, on a branch of its own.
|
"""Moves `node` onto a new branch so the story can continue from it.
|
||||||
|
|
||||||
`node` is a discarded attempt at a turn the story has already moved past.
|
`node` is a discarded attempt at a turn that the story has already moved
|
||||||
Making it live where it stands would orphan every turn played after it —
|
past. Making it live where it stands would orphan every turn played after
|
||||||
they were written as a continuation of the attempt that won — so it moves
|
it, because those turns continue the attempt that won. Instead, `node` moves
|
||||||
onto a new branch instead, forked from the depth just before it. The parent
|
to a new branch that forks from the depth just before it. The parent branch
|
||||||
keeps its story, complete and untouched; the new branch borrows everything
|
keeps its story unchanged. The new branch inherits everything up to the fork
|
||||||
up to the fork and owns exactly one node.
|
and owns this one node.
|
||||||
|
|
||||||
**One row is inserted and one row is moved. Nothing is copied.** That is
|
This function inserts one row and moves one row. It copies nothing, so a
|
||||||
the whole claim of the design: a fork costs a `branches` row and the
|
fork costs one `branches` row plus the ancestry cached on it, regardless of
|
||||||
ancestry cached on it, whatever the story behind it is worth.
|
how long the story is.
|
||||||
|
|
||||||
Nothing derived moves with it, and that is not an omission. A memory hangs
|
Derived data stays on the parent, by design. A memory attaches to the node
|
||||||
off the coordinate its block ends on, and what it describes is whatever
|
its block ends on, and that node does not move. From the new branch, the
|
||||||
attempt was live there — which stays on the parent. From the new branch it
|
lineage caps the parent at `fork_depth`, so the memory sits one depth past
|
||||||
is simply out of range: the lineage caps the parent at `fork_depth`, so the
|
the border. Neither retrieval nor the cursors can see it, and the block is
|
||||||
memory sits one depth past the border and neither the retrieval clause nor
|
summarized again from the text this branch contains.
|
||||||
the cursors can see it. The block is summarized again, from the text this
|
|
||||||
branch actually tells, without a line of bookkeeping.
|
|
||||||
|
|
||||||
One thing does stay behind: the attempts this node leaves. They are still
|
The other attempts at this turn also stay on the parent, because they are
|
||||||
takes on the parent's turn, and one of them has to be the parent's story —
|
still takes on the parent's turn. If none of them is live, the oldest one
|
||||||
the oldest, so the line the parent keeps is the one it was written on.
|
becomes live, so the parent keeps the line it was written on.
|
||||||
"""
|
"""
|
||||||
parent = db.get(models.Branch, node.branch_id)
|
parent = db.get(models.Branch, node.branch_id)
|
||||||
if parent is None or node.depth is None:
|
if parent is None or node.depth is None:
|
||||||
raise ValueError("cannot fork from a node that is not on a branch")
|
raise ValueError("cannot fork from a node that is not on a branch")
|
||||||
fork_depth = node.depth - 1
|
fork_depth = node.depth - 1
|
||||||
# The attempts this node is leaving, read *before* it moves. The session
|
# Read the sibling attempts before moving the node. The session does not
|
||||||
# does not autoflush, so asking afterwards would still find the node here
|
# autoflush, so a later read still finds the node here and renumbers it back
|
||||||
# and renumber it back into the group it just left.
|
# into the group it just left.
|
||||||
remaining = [
|
remaining = [
|
||||||
row for row in db.query(models.Action)
|
row for row in db.query(models.Action)
|
||||||
.filter(
|
.filter(
|
||||||
@@ -134,17 +131,17 @@ def fork(db: Session, adventure: models.Adventure, node: models.Action) -> model
|
|||||||
.all()
|
.all()
|
||||||
if row is not node
|
if row is not node
|
||||||
]
|
]
|
||||||
# The parent's ancestry, every entry capped at the fork. Only the first can
|
# The parent's ancestry, with every entry capped at the fork depth. Only the
|
||||||
# actually move — an older entry is already capped at the fork depth of the
|
# first entry can change in practice, because older entries are already
|
||||||
# branch beneath it, which is shallower than any node on the parent — but
|
# capped at a shallower depth. Capping all of them states the invariant
|
||||||
# capping them all says the invariant instead of relying on it.
|
# directly.
|
||||||
inherited = [
|
inherited = [
|
||||||
[branch_id, fork_depth if cap is None else min(cap, fork_depth)]
|
[branch_id, fork_depth if cap is None else min(cap, fork_depth)]
|
||||||
for branch_id, cap in lineage.entries_of(parent)
|
for branch_id, cap in lineage.entries_of(parent)
|
||||||
]
|
]
|
||||||
# Inserted through Core, and its lineage written second, for the reason
|
# Core insert with the lineage written second, for the reason given in
|
||||||
# `root_branch` spells out: this can run inside a flush, and the lineage
|
# `root_branch`. This code can run inside a flush, and the lineage refers to
|
||||||
# names the row's own id.
|
# the new row's own id.
|
||||||
new_id = db.execute(
|
new_id = db.execute(
|
||||||
insert(models.Branch).values(
|
insert(models.Branch).values(
|
||||||
adventure_id=adventure.id,
|
adventure_id=adventure.id,
|
||||||
@@ -180,26 +177,29 @@ def fork(db: Session, adventure: models.Adventure, node: models.Action) -> model
|
|||||||
def branch_at(
|
def branch_at(
|
||||||
db: Session, adventure: models.Adventure, fork_depth: int
|
db: Session, adventure: models.Adventure, fork_depth: int
|
||||||
) -> models.Branch:
|
) -> models.Branch:
|
||||||
"""An empty branch leaving the path being read at `fork_depth`.
|
"""Creates an empty branch that leaves the current path at `fork_depth`.
|
||||||
|
|
||||||
`fork` moves a node that already exists onto a line of its own. This is the
|
`fork` moves an existing node onto its own branch. This function creates the
|
||||||
same branch with nothing in it yet, for the case where the take that will
|
same kind of branch with no nodes on it yet, for the case where the take
|
||||||
live there has not been written: the player asking for another take of a
|
that will live there does not exist. A player asking for another take of a
|
||||||
turn the story has moved past (SP9). The head lands at `fork_depth`, so the
|
turn the story has moved past reaches this path (SP9).
|
||||||
next node written is the new take, at the same depth as the one it is a take
|
|
||||||
of, with the same parent — `place_action` resolves that from the path, and
|
|
||||||
the path now ends at exactly the node the original hangs off.
|
|
||||||
|
|
||||||
The line being left is not touched at all. It keeps its node at that depth,
|
The head lands at `fork_depth`, so the next node written becomes the new
|
||||||
it keeps that node live, and it keeps everything played after it.
|
take. That node gets the same depth as the original and the same parent,
|
||||||
|
which `place_action` derives from the path.
|
||||||
|
|
||||||
|
This function does not modify the branch being left. That branch keeps its
|
||||||
|
node at that depth, the node stays live, and every turn played after it
|
||||||
|
stays in place.
|
||||||
"""
|
"""
|
||||||
parent = head_branch(db, adventure)
|
parent = head_branch(db, adventure)
|
||||||
inherited = [
|
inherited = [
|
||||||
[branch_id, fork_depth if cap is None else min(cap, fork_depth)]
|
[branch_id, fork_depth if cap is None else min(cap, fork_depth)]
|
||||||
for branch_id, cap in lineage.entries_of(parent)
|
for branch_id, cap in lineage.entries_of(parent)
|
||||||
]
|
]
|
||||||
# Core insert with the lineage written second, for the reason `root_branch`
|
# Core insert with the lineage written second, for the reason given in
|
||||||
# spells out: this can run inside a flush, and the lineage names its own id.
|
# `root_branch`. This code can run inside a flush, and the lineage refers to
|
||||||
|
# the new row's own id.
|
||||||
new_id = db.execute(
|
new_id = db.execute(
|
||||||
insert(models.Branch).values(
|
insert(models.Branch).values(
|
||||||
adventure_id=adventure.id,
|
adventure_id=adventure.id,
|
||||||
@@ -226,20 +226,20 @@ def place_action(
|
|||||||
branch: models.Branch | None = None,
|
branch: models.Branch | None = None,
|
||||||
parent: models.Action | None = None,
|
parent: models.Action | None = None,
|
||||||
) -> models.Branch:
|
) -> models.Branch:
|
||||||
"""Put `action` on the head branch and move the head to it.
|
"""Puts `action` on the head branch and moves the head to it.
|
||||||
|
|
||||||
`depth` follows `index` while the two coexist. They have to agree: a read
|
`depth` follows `index` while both columns exist. The two must agree,
|
||||||
ordering by depth and a cursor counting in index space are describing the
|
because a read that orders by depth and a cursor that counts in index space
|
||||||
same story, and SP2 swaps one for the other under everything at once.
|
describe the same story, and SP2 swaps one for the other in a single step.
|
||||||
|
|
||||||
`branch` is the head, already resolved, for a caller placing several nodes
|
Pass `branch` when you have already resolved the head and are placing
|
||||||
at once — see `place_new_nodes` for why that is worth a parameter.
|
several nodes at once. See `place_new_nodes` for why that is worth doing.
|
||||||
|
|
||||||
`parent` is the take this node was played after (SP9), and is what groups a
|
`parent` is the take that this node was played after (SP9), and it is what
|
||||||
turn's takes. Resolved from the path when the caller does not say, which is
|
groups the takes of one turn. If you omit it, this function derives it from
|
||||||
the honest default: a node written now follows whatever the player is
|
the path, which is the correct default: a node written now follows the story
|
||||||
reading now. A caller placing several nodes in one flush should chain it —
|
the player is reading now. If you place several nodes in one flush, chain
|
||||||
the second node's parent is the first, and the database has not seen either.
|
`parent` explicitly, because the database has not seen any of them yet.
|
||||||
"""
|
"""
|
||||||
branch = branch or head_branch(db, adventure)
|
branch = branch or head_branch(db, adventure)
|
||||||
action.branch_id = branch.id
|
action.branch_id = branch.id
|
||||||
@@ -258,12 +258,12 @@ def place_action(
|
|||||||
def _preceding_id(
|
def _preceding_id(
|
||||||
db: Session, adventure: models.Adventure, branch: models.Branch, depth: int
|
db: Session, adventure: models.Adventure, branch: models.Branch, depth: int
|
||||||
) -> int | None:
|
) -> int | None:
|
||||||
"""The id of the live node one step back along `branch`'s path.
|
"""Returns the id of the live node one step back along `branch`'s path.
|
||||||
|
|
||||||
Asked of the whole lineage rather than of `branch` alone, because a branch
|
The query searches the whole lineage instead of `branch` alone, because a
|
||||||
borrows the story before its fork point: the node in front of a forked
|
branch inherits the story before its fork point. The node in front of a
|
||||||
branch's first turn lives on an ancestor, and that is exactly the parent a
|
forked branch's first turn lives on an ancestor, and that node is the parent
|
||||||
pager needs to find its siblings through.
|
a pager needs in order to find siblings.
|
||||||
"""
|
"""
|
||||||
path = lineage.Path(lineage.entries_of(branch))
|
path = lineage.Path(lineage.entries_of(branch))
|
||||||
return (
|
return (
|
||||||
@@ -286,19 +286,19 @@ def place_memory(
|
|||||||
memory: models.Memory,
|
memory: models.Memory,
|
||||||
branch: models.Branch | None = None,
|
branch: models.Branch | None = None,
|
||||||
) -> models.Branch:
|
) -> models.Branch:
|
||||||
"""Attach a memory to the node it belongs to.
|
"""Attaches a memory to the node it belongs to.
|
||||||
|
|
||||||
`source_end` is the index of the last action the memory summarises, which is
|
`source_end` is the index of the last action the memory summarizes, which is
|
||||||
that node's depth. A hand-written memory summarises nothing, so it takes the
|
that node's depth. A hand-written memory summarizes no actions, so it uses
|
||||||
head instead: **the story you were reading when you wrote it.**
|
the head instead. That records the story the author was reading at the time.
|
||||||
|
|
||||||
That anchor is what makes a memory mean one thing (SP7). Before it, a
|
Giving every memory a coordinate is what makes its scope unambiguous (SP7).
|
||||||
hand-written memory kept a NULL depth and "belonged to the adventure rather
|
Before SP7, hand-written memories had a NULL depth and belonged to the
|
||||||
than to a path" — which sounded harmless and meant it followed you onto
|
adventure rather than to a path. A fork cannot cap a NULL, so those memories
|
||||||
branches whose story it did not describe, because a NULL cannot be capped at
|
followed the reader onto branches whose story they did not describe. Now the
|
||||||
a fork. Every memory now sits at a coordinate, so "is this part of the story
|
question "is this memory part of the story I am reading?" has one answer for
|
||||||
I am reading?" has one answer for every row in the bank, and it is the same
|
every row in the bank, and it is the same answer the lineage gives for
|
||||||
answer the lineage already gives for nodes.
|
nodes.
|
||||||
"""
|
"""
|
||||||
branch = branch or head_branch(db, adventure)
|
branch = branch or head_branch(db, adventure)
|
||||||
memory.branch_id = branch.id
|
memory.branch_id = branch.id
|
||||||
@@ -311,35 +311,37 @@ def place_memory(
|
|||||||
|
|
||||||
|
|
||||||
def attach_memory(memory: models.Memory, node: models.Action) -> None:
|
def attach_memory(memory: models.Memory, node: models.Action) -> None:
|
||||||
"""Hang a memory off the node it was derived from.
|
"""Attaches a memory to the node it was derived from.
|
||||||
|
|
||||||
The general rule, of which the memory bank is the first instance: anything
|
This follows the general rule for derived data, and the memory bank is the
|
||||||
derived from the story attaches to the node that produced it, and is then
|
first case of it. Anything derived from the story attaches to the node that
|
||||||
visible from exactly the paths that node is on. A fork inherits its
|
produced it, and is then visible from exactly the paths that contain that
|
||||||
ancestors' memories because it inherits their nodes — nothing is copied and
|
node. A fork inherits its ancestors' memories because it inherits their
|
||||||
nothing is recreated — and a memory made on a sibling is invisible here
|
nodes, so nothing is copied. A memory made on a sibling branch is not
|
||||||
because that node is not on this path.
|
visible, because that node is not on this path.
|
||||||
|
|
||||||
Not `place_memory`: this takes the branch from the *node*, which is not
|
This function differs from `place_memory` because it takes the branch from
|
||||||
always the head. A block of story can end before the fork the current
|
`node`, which is not always the head. A block of story can end before the
|
||||||
branch was made at, and the memory belongs where the ground is.
|
fork that created the current branch, and the memory belongs where that
|
||||||
|
block is.
|
||||||
"""
|
"""
|
||||||
memory.branch_id = node.branch_id
|
memory.branch_id = node.branch_id
|
||||||
memory.depth = node.depth
|
memory.depth = node.depth
|
||||||
|
|
||||||
|
|
||||||
def stamp_outcome(adventure: models.Adventure, action: models.Action) -> None:
|
def stamp_outcome(adventure: models.Adventure, action: models.Action) -> None:
|
||||||
"""Give a node the state it left behind, if its writer did not.
|
"""Records the state a node left behind, if the writer did not record it.
|
||||||
|
|
||||||
The floor under `attempts.snapshot_outcome`, and it is here for the same
|
This is the fallback under `attempts.snapshot_outcome`, and it exists for
|
||||||
reason `place_action` has one: from SP4 a node with no outcome is a node
|
the same reason `place_action` has one. Since SP4, undo and retry cannot
|
||||||
undo and retry cannot roll back past, and it fails by leaving the
|
roll back past a node that has no outcome, and the failure is silent: the
|
||||||
scoreboard where it was rather than by raising. The turn engine records the
|
script state and world state stay where they were. The turn engine records
|
||||||
outcome itself and this skips those rows; what it catches is every fixture,
|
the outcome itself, so this function skips those rows. It catches fixtures,
|
||||||
script and import that writes a story straight through the ORM.
|
scripts, and imports that write a story directly through the ORM.
|
||||||
|
|
||||||
What it writes is the truth as of the flush: a writer that changes no state
|
The values written are the state as of the flush. That is correct, because a
|
||||||
between two nodes leaves the same state behind both of them.
|
writer that changes no state between two nodes leaves the same state behind
|
||||||
|
both of them.
|
||||||
"""
|
"""
|
||||||
if action.state_after is None:
|
if action.state_after is None:
|
||||||
state = adventure.script_state if isinstance(adventure.script_state, dict) else {}
|
state = adventure.script_state if isinstance(adventure.script_state, dict) else {}
|
||||||
@@ -350,25 +352,25 @@ def stamp_outcome(adventure: models.Adventure, action: models.Action) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def place_new_nodes(session: Session) -> None:
|
def place_new_nodes(session: Session) -> None:
|
||||||
"""Place every unplaced node about to be inserted. Runs on every flush.
|
"""Places every unplaced node that is about to be inserted.
|
||||||
|
|
||||||
The call sites still call `place_action` / `place_memory` themselves, and
|
This runs on every flush. Call sites still call `place_action` and
|
||||||
should: a node placed at the call site is placed *before* the code around
|
`place_memory` directly, and they should. Placing a node at the call site
|
||||||
it reads the row back, and the explicit call is what makes the ordering
|
happens before the surrounding code reads the row back, and the explicit
|
||||||
visible. This is the floor under them — a fixture built straight through
|
call makes that ordering visible. This function is the fallback under those
|
||||||
the ORM, a script, a test, or a call site added next year gets a branch
|
calls, so a fixture, script, test, or a call site added later still gets a
|
||||||
without knowing the tree exists.
|
branch without knowing the tree exists.
|
||||||
|
|
||||||
Nodes whose adventure has not been inserted yet are left alone: there is no
|
Nodes whose adventure has not been inserted yet are skipped, because there
|
||||||
id to hang a branch off, and an Action needs `adventure_id` to be written
|
is no id to attach a branch to. This case does not arise in practice, since
|
||||||
at all, so the case does not arise from any writer we have.
|
an Action needs `adventure_id` before it can be written at all.
|
||||||
|
|
||||||
The head is resolved once per adventure per flush, and held in `heads` for
|
The head is resolved once per adventure per flush and cached in `heads`.
|
||||||
the length of the call. That is not just saving a dictionary lookup: the
|
This saves more than a dictionary lookup. The identity map holds weak
|
||||||
identity map holds *weak* references, so a branch row nobody keeps a strong
|
references, so a branch row that nothing else refers to is collected between
|
||||||
reference to is collected between two nodes and read back from the database
|
two nodes and read from the database again for the next one. Resolving the
|
||||||
for the next one. Resolving per node turned a fixture writing two hundred
|
head per node turned a fixture that wrote 200 actions in one flush into 200
|
||||||
actions in one flush into two hundred SELECTs on `branches`.
|
separate queries.
|
||||||
"""
|
"""
|
||||||
heads: dict[int, models.Branch] = {}
|
heads: dict[int, models.Branch] = {}
|
||||||
for obj in list(session.new):
|
for obj in list(session.new):
|
||||||
@@ -394,11 +396,11 @@ def place_new_nodes(session: Session) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def refresh_head(db: Session, adventure: models.Adventure) -> None:
|
def refresh_head(db: Session, adventure: models.Adventure) -> None:
|
||||||
"""Re-derive the head depth after nodes were removed (undo, delete).
|
"""Recomputes the head depth after nodes are removed by undo or delete.
|
||||||
|
|
||||||
A branch with nothing on it sits at its fork point, because that is the last
|
A branch with no nodes of its own sits at its fork point, because that is
|
||||||
node its story contains — borrowed from the parent, but the tip all the
|
the last node its story contains. The node is inherited from the parent, but
|
||||||
same. A root branch with nothing on it has no story at all.
|
it is still the tip. A root branch with no nodes has no story at all.
|
||||||
"""
|
"""
|
||||||
branch = head_branch(db, adventure)
|
branch = head_branch(db, adventure)
|
||||||
tip = (
|
tip = (
|
||||||
|
|||||||
@@ -1,15 +1,15 @@
|
|||||||
"""Storing and comparing embedding vectors.
|
"""Storing and comparing embedding vectors.
|
||||||
|
|
||||||
A 1536-dimension vector written as a JSON list is about 31 KB, because every
|
A 1536-dimension vector written as a JSON list is about 31 kB, because every
|
||||||
component is spelled out as a decimal string of seventeen-odd digits. The same
|
component is written as a decimal string of about seventeen digits. The same
|
||||||
vector as packed float32 is 6,144 bytes — a straight 5x, and the memory bank is
|
vector as packed float32 is 6,144 bytes, which is five times smaller, and the
|
||||||
read in full on every turn, so those bytes are paid over and over.
|
memory bank is read in full on every turn, so those bytes are paid repeatedly.
|
||||||
|
|
||||||
**Float32 is not an approximation here.** The embedding endpoints return
|
Float32 is not an approximation here. The embedding endpoints return vectors
|
||||||
vectors computed in float32, rendered into JSON as the shortest decimal string
|
computed in float32 and render them into JSON as the shortest decimal string
|
||||||
that round-trips through a double; converting that back to float32 recovers the
|
that round-trips through a double. Converting that back to float32 recovers the
|
||||||
original bits exactly. Nothing is lost that was ever there, which is why the
|
original bits exactly. Nothing that was present is lost, which is why the
|
||||||
conversion needs no re-embedding and carries no retrieval-quality risk.
|
conversion needs no re-embedding and carries no risk to retrieval quality.
|
||||||
|
|
||||||
Dimensions are deliberately unchanged. Dropping to 512 or 768 would have saved
|
Dimensions are deliberately unchanged. Dropping to 512 or 768 would have saved
|
||||||
another 3x and cost an API call per stored memory to re-embed, against a bank
|
another 3x and cost an API call per stored memory to re-embed, against a bank
|
||||||
|
|||||||
@@ -1,5 +1,8 @@
|
|||||||
"""Phase 12 — RPG world state: the AI proposes stat/milestone deltas, this
|
"""Phase 12: RPG world state.
|
||||||
module validates and clamps them against a scenario's stat_schema."""
|
|
||||||
|
The AI proposes stat and milestone deltas, and this module validates and clamps
|
||||||
|
them against a scenario's `stat_schema`.
|
||||||
|
"""
|
||||||
|
|
||||||
from .engine import (
|
from .engine import (
|
||||||
EMIT_REMINDER,
|
EMIT_REMINDER,
|
||||||
|
|||||||
@@ -1,23 +1,25 @@
|
|||||||
"""RPG world-state engine.
|
"""RPG world-state engine.
|
||||||
|
|
||||||
The scenario carries a `stat_schema` (the template: which stats exist, their
|
The scenario carries a `stat_schema`, which is the template: which stats exist,
|
||||||
bands and rules, and the milestones). An adventure carries a live `world_state`
|
their bands and rules, and the milestones. An adventure carries a live
|
||||||
instantiated from it. Each turn the AI proposes a *delta* (only what changed);
|
`world_state` instantiated from it. Each turn the AI proposes a delta holding
|
||||||
`apply_delta` is the referee — it clamps to min/max, caps per-turn change,
|
only what changed. `apply_delta` decides what the delta is allowed to do. It
|
||||||
enforces cooldowns, and marks milestones sticky.
|
clamps values to min and max, caps the change per turn, enforces cooldowns, and
|
||||||
|
makes milestones sticky.
|
||||||
|
|
||||||
Nothing here ever raises on bad AI output: a malformed delta yields `{}` and the
|
Nothing here raises on bad AI output. A malformed delta returns `{}` and the
|
||||||
turn continues, exactly like a broken script never breaks a turn.
|
turn continues, the same way a broken script never breaks a turn.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import copy
|
import copy
|
||||||
import json
|
import json
|
||||||
import re
|
import re
|
||||||
|
|
||||||
# stat_schema top-level sections that hold stat definitions.
|
# The `stat_schema` top-level sections that hold stat definitions.
|
||||||
STAT_SECTIONS = ("world", "player")
|
STAT_SECTIONS = ("world", "player")
|
||||||
|
|
||||||
# Appended once to the system prompt so the model knows how to report changes.
|
# Appended once to the system prompt, so the model knows how to report
|
||||||
|
# changes.
|
||||||
EMIT_RULE = (
|
EMIT_RULE = (
|
||||||
"You maintain a numeric world state. Treat your own narration as authoritative: "
|
"You maintain a numeric world state. Treat your own narration as authoritative: "
|
||||||
"whenever what you write implies a change to any tracked value — health or resources "
|
"whenever what you write implies a change to any tracked value — health or resources "
|
||||||
@@ -45,8 +47,8 @@ EMIT_RULE = (
|
|||||||
'"player.outfit": "torn traveling cloak"}\n```'
|
'"player.outfit": "torn traveling cloak"}\n```'
|
||||||
)
|
)
|
||||||
|
|
||||||
# Short terminal reminder placed at the very end of the prompt (strongest
|
# A short reminder placed at the end of the prompt, which is the strongest
|
||||||
# recency position) so the emit rule is fresh right where the model generates.
|
# recency position, so the emit rule is close to where the model generates.
|
||||||
EMIT_REMINDER = (
|
EMIT_REMINDER = (
|
||||||
"[Reminder: end your reply with a ```state block of the changes this turn "
|
"[Reminder: end your reply with a ```state block of the changes this turn "
|
||||||
"(deltas only), or omit it if truly nothing changed.]"
|
"(deltas only), or omit it if truly nothing changed.]"
|
||||||
@@ -54,9 +56,12 @@ EMIT_REMINDER = (
|
|||||||
|
|
||||||
|
|
||||||
def render_delta_block(delta: dict) -> str:
|
def render_delta_block(delta: dict) -> str:
|
||||||
"""Render a stored delta back into the fenced `state` block the AI emitted,
|
"""Renders a stored delta back into the fenced `state` block the AI emitted.
|
||||||
for re-injecting past turns into context so the model imitates the format.
|
|
||||||
Empty delta -> empty string (the turn legitimately changed nothing)."""
|
The caller replays past turns into the context with this, so the model copies
|
||||||
|
the format. An empty delta returns an empty string, which means the turn
|
||||||
|
changed nothing.
|
||||||
|
"""
|
||||||
if not isinstance(delta, dict) or not delta:
|
if not isinstance(delta, dict) or not delta:
|
||||||
return ""
|
return ""
|
||||||
return "```state\n" + json.dumps(delta, ensure_ascii=False) + "\n```"
|
return "```state\n" + json.dumps(delta, ensure_ascii=False) + "\n```"
|
||||||
@@ -68,7 +73,7 @@ _TRAILING_RE = re.compile(r"(\{[^{}]*\})\s*$", re.DOTALL)
|
|||||||
|
|
||||||
|
|
||||||
def has_schema(stat_schema: dict | None) -> bool:
|
def has_schema(stat_schema: dict | None) -> bool:
|
||||||
"""True when a scenario actually defines an RPG layer."""
|
"""Returns `True` when a scenario defines an RPG layer."""
|
||||||
if not isinstance(stat_schema, dict):
|
if not isinstance(stat_schema, dict):
|
||||||
return False
|
return False
|
||||||
return any(
|
return any(
|
||||||
@@ -83,8 +88,11 @@ def npc_name(ndef: dict, key: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def npc_triggers(ndef: dict, key: str) -> list[str]:
|
def npc_triggers(ndef: dict, key: str) -> list[str]:
|
||||||
"""Lower-cased trigger words for detecting an NPC in scene: its `keys`
|
"""Returns the lowercased trigger words that detect an NPC in a scene.
|
||||||
field, falling back to its display name."""
|
|
||||||
|
The words come from the NPC's `keys` field, or from its display name when
|
||||||
|
`keys` is empty.
|
||||||
|
"""
|
||||||
raw = ndef.get("keys") or npc_name(ndef, key)
|
raw = ndef.get("keys") or npc_name(ndef, key)
|
||||||
return [k.strip().lower() for k in str(raw).split(",") if k.strip()]
|
return [k.strip().lower() for k in str(raw).split(",") if k.strip()]
|
||||||
|
|
||||||
@@ -98,7 +106,7 @@ def _initials(defs: dict) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
def instantiate(stat_schema: dict | None) -> dict:
|
def instantiate(stat_schema: dict | None) -> dict:
|
||||||
"""Build a fresh live world_state from a schema (initial values only)."""
|
"""Builds a fresh live `world_state` from a schema, using initial values only."""
|
||||||
if not has_schema(stat_schema):
|
if not has_schema(stat_schema):
|
||||||
return {}
|
return {}
|
||||||
ws: dict = {}
|
ws: dict = {}
|
||||||
@@ -121,24 +129,26 @@ def instantiate(stat_schema: dict | None) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
def reconcile(world_state: dict | None, stat_schema: dict | None) -> tuple[dict, dict]:
|
def reconcile(world_state: dict | None, stat_schema: dict | None) -> tuple[dict, dict]:
|
||||||
"""Bring a live world_state back in line with an edited schema.
|
"""Brings a live `world_state` back in line with an edited schema.
|
||||||
|
|
||||||
Deliberately NOT `instantiate`: a value the schema still defines keeps
|
This is not `instantiate`. A value the schema still defines keeps whatever it
|
||||||
whatever it has reached in play (re-instantiating would heal the player to
|
reached in play, because re-instantiating would restore the player to full
|
||||||
full and wipe their milestones). Only the difference is applied — stats,
|
health and clear their milestones. Only the difference is applied. Stats,
|
||||||
NPCs, flags and milestones the schema gained appear at their initial value,
|
NPCs, flags, and milestones the schema gained appear at their initial value,
|
||||||
and ones it no longer defines are dropped, along with their `last_changed`
|
and ones it no longer defines are removed along with their `last_changed`
|
||||||
bookkeeping. Returns (new_state, report) where the report lists paths under
|
bookkeeping. The return value is `(new_state, report)`, where the report
|
||||||
`added` / `removed` so the UI can show what a refresh would do.
|
lists paths under `added` and `removed`, so the UI can show what a refresh
|
||||||
|
would do.
|
||||||
|
|
||||||
Note the additions are mostly cosmetic: rendering and delta-application both
|
The additions are mostly cosmetic. Rendering and delta application both fall
|
||||||
fall back to a stat def's `initial` when the live state has no value for it,
|
back to a stat definition's `initial` when the live state has no value for
|
||||||
so a newly added stat already behaves correctly. This materialises it (and,
|
it, so a newly added stat already behaves correctly. This function stores the
|
||||||
unlike those read-through paths, actually cleans up removals).
|
value, and unlike those read-through paths it also removes what the schema
|
||||||
|
dropped.
|
||||||
"""
|
"""
|
||||||
report: dict = {"added": [], "removed": []}
|
report: dict = {"added": [], "removed": []}
|
||||||
if not has_schema(stat_schema):
|
if not has_schema(stat_schema):
|
||||||
# The scenario dropped its RPG layer entirely — so does the adventure.
|
# The scenario dropped its RPG layer, so the adventure drops it too.
|
||||||
stale = bool(world_state)
|
stale = bool(world_state)
|
||||||
if stale:
|
if stale:
|
||||||
report["removed"].append("(all world state)")
|
report["removed"].append("(all world state)")
|
||||||
@@ -199,9 +209,10 @@ def reconcile(world_state: dict | None, stat_schema: dict | None) -> tuple[dict,
|
|||||||
report["removed"].append(f"flags.{name}")
|
report["removed"].append(f"flags.{name}")
|
||||||
ws["flags"] = new_flags
|
ws["flags"] = new_flags
|
||||||
|
|
||||||
# Milestones store only the ones reached, so there is nothing to add here —
|
# Milestones store only the ones reached, so there is nothing to add here.
|
||||||
# an unreached milestone is simply absent. Drop reached ones the scenario
|
# An unreached milestone is absent. Drop reached milestones the scenario no
|
||||||
# no longer defines, or they'd sit in "Achieved" forever with no label.
|
# longer defines, because they would otherwise stay in "Achieved" with no
|
||||||
|
# label.
|
||||||
milestone_defs = stat_schema.get("milestones") or {}
|
milestone_defs = stat_schema.get("milestones") or {}
|
||||||
reached = ws.get("milestones") if isinstance(ws.get("milestones"), dict) else {}
|
reached = ws.get("milestones") if isinstance(ws.get("milestones"), dict) else {}
|
||||||
ws["milestones"] = {k: v for k, v in reached.items() if k in milestone_defs}
|
ws["milestones"] = {k: v for k, v in reached.items() if k in milestone_defs}
|
||||||
@@ -209,8 +220,8 @@ def reconcile(world_state: dict | None, stat_schema: dict | None) -> tuple[dict,
|
|||||||
if k not in milestone_defs:
|
if k not in milestone_defs:
|
||||||
report["removed"].append(f"milestones.{k}")
|
report["removed"].append(f"milestones.{k}")
|
||||||
|
|
||||||
# Cooldown bookkeeping for paths that no longer exist would never be read,
|
# Cooldown bookkeeping for paths that no longer exist is never read, but it
|
||||||
# but it accumulates in every stored snapshot — prune it with the rest.
|
# accumulates in every stored snapshot, so remove it with the rest.
|
||||||
meta = ws.setdefault("_meta", {})
|
meta = ws.setdefault("_meta", {})
|
||||||
last_changed = meta.get("last_changed")
|
last_changed = meta.get("last_changed")
|
||||||
if isinstance(last_changed, dict):
|
if isinstance(last_changed, dict):
|
||||||
@@ -226,10 +237,10 @@ def reconcile(world_state: dict | None, stat_schema: dict | None) -> tuple[dict,
|
|||||||
|
|
||||||
|
|
||||||
def band_label(stat_def: dict, value) -> str | None:
|
def band_label(stat_def: dict, value) -> str | None:
|
||||||
"""The word label for `value` from a stat def's bands, if any.
|
"""Returns the word label for `value` from a stat definition's bands, if any.
|
||||||
|
|
||||||
Bands are [lo, hi, label]; matched as lo <= value < hi, with the top band
|
A band is `[lo, hi, label]` and matches when `lo <= value < hi`. The top band
|
||||||
inclusive of its upper bound so a maxed stat still gets a label.
|
includes its upper bound, so a stat at its maximum still gets a label.
|
||||||
"""
|
"""
|
||||||
bands = stat_def.get("bands")
|
bands = stat_def.get("bands")
|
||||||
if not isinstance(bands, list) or not isinstance(value, (int, float)):
|
if not isinstance(bands, list) or not isinstance(value, (int, float)):
|
||||||
@@ -265,12 +276,12 @@ def _tolerant_load(blob: str) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
def extract_delta(text: str) -> tuple[str, dict]:
|
def extract_delta(text: str) -> tuple[str, dict]:
|
||||||
"""Pull the trailing state block out of an AI response.
|
"""Removes the trailing state block from an AI response.
|
||||||
|
|
||||||
Returns (clean_text, delta). `delta` is `{}` when there is no block or it
|
The return value is `(clean_text, delta)`. `delta` is `{}` when there is no
|
||||||
can't be parsed; `clean_text` has the block removed. Only strips a bare
|
block or the block cannot be parsed, and `clean_text` has the block removed.
|
||||||
trailing object when it actually parses to a delta, so ordinary prose
|
A bare trailing object is stripped only when it parses to a delta, so
|
||||||
ending in `}` is never eaten.
|
ordinary prose that ends in `}` is left alone.
|
||||||
"""
|
"""
|
||||||
matches = list(_FENCE_RE.finditer(text))
|
matches = list(_FENCE_RE.finditer(text))
|
||||||
if matches:
|
if matches:
|
||||||
@@ -293,7 +304,7 @@ def extract_delta(text: str) -> tuple[str, dict]:
|
|||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
def _coerce_number(value):
|
def _coerce_number(value):
|
||||||
if isinstance(value, bool): # bool is an int subclass — reject here
|
if isinstance(value, bool): # `bool` is an `int` subclass, so reject it here.
|
||||||
return None
|
return None
|
||||||
if isinstance(value, (int, float)):
|
if isinstance(value, (int, float)):
|
||||||
return value
|
return value
|
||||||
@@ -349,9 +360,12 @@ def _apply_stat(container: dict, key: str, stat_def: dict, change,
|
|||||||
|
|
||||||
def _apply_text_stat(container: dict, key: str, stat_def: dict, change,
|
def _apply_text_stat(container: dict, key: str, stat_def: dict, change,
|
||||||
path: str, action_index: int, meta: dict, report: dict) -> None:
|
path: str, action_index: int, meta: dict, report: dict) -> None:
|
||||||
"""Free-text stats replace rather than add: the AI sends the new value in
|
"""Applies a free-text stat, which replaces rather than adds.
|
||||||
full, not a delta. No clamping/bands apply — only an optional cooldown and
|
|
||||||
an optional max_length truncation."""
|
The AI sends the new value in full rather than a delta. No clamping and no
|
||||||
|
bands apply. Only an optional cooldown and an optional `max_length`
|
||||||
|
truncation apply.
|
||||||
|
"""
|
||||||
if not isinstance(change, str):
|
if not isinstance(change, str):
|
||||||
report["rejected"].append({"path": path, "reason": "not a string"})
|
report["rejected"].append({"path": path, "reason": "not a string"})
|
||||||
return
|
return
|
||||||
@@ -369,7 +383,7 @@ def _apply_text_stat(container: dict, key: str, stat_def: dict, change,
|
|||||||
|
|
||||||
old = container.get(key, stat_def.get("initial", ""))
|
old = container.get(key, stat_def.get("initial", ""))
|
||||||
if new == old:
|
if new == old:
|
||||||
return # no actual change — silent no-op
|
return # Nothing changed, so do nothing.
|
||||||
|
|
||||||
container[key] = new
|
container[key] = new
|
||||||
meta["last_changed"][path] = action_index
|
meta["last_changed"][path] = action_index
|
||||||
@@ -377,14 +391,17 @@ def _apply_text_stat(container: dict, key: str, stat_def: dict, change,
|
|||||||
|
|
||||||
|
|
||||||
def apply_override(world_state: dict, stat_schema: dict, overrides: dict) -> tuple[dict, dict]:
|
def apply_override(world_state: dict, stat_schema: dict, overrides: dict) -> tuple[dict, dict]:
|
||||||
"""Directly set live values — a manual author/admin edit, not an AI turn.
|
"""Sets live values directly, as a manual author edit rather than an AI turn.
|
||||||
|
|
||||||
Unlike `apply_delta`: numeric stats are SET rather than added to, and
|
This differs from `apply_delta` in three ways. Numeric stats are set rather
|
||||||
`cooldown`/`max_delta_per_turn`/the counter-can't-decrease rule are all
|
than added to. `cooldown`, `max_delta_per_turn`, and the rule that a counter
|
||||||
ignored (a deliberate correction, not an AI move to police). Milestones
|
cannot decrease are all ignored, because this is a deliberate correction
|
||||||
can be toggled either way, not only marked reached. Values are still
|
rather than an AI move to check. Milestones can be toggled in both
|
||||||
validated against the schema (unknown path/type is rejected) and numeric
|
directions rather than only marked reached.
|
||||||
values still clamp to min/max."""
|
|
||||||
|
Values are still validated against the schema, so an unknown path or a wrong
|
||||||
|
type is rejected, and numeric values still clamp to min and max.
|
||||||
|
"""
|
||||||
ws = copy.deepcopy(world_state) if isinstance(world_state, dict) else {}
|
ws = copy.deepcopy(world_state) if isinstance(world_state, dict) else {}
|
||||||
if not ws:
|
if not ws:
|
||||||
ws = instantiate(stat_schema)
|
ws = instantiate(stat_schema)
|
||||||
@@ -492,8 +509,11 @@ def apply_override(world_state: dict, stat_schema: dict, overrides: dict) -> tup
|
|||||||
|
|
||||||
def apply_delta(world_state: dict, stat_schema: dict, delta: dict,
|
def apply_delta(world_state: dict, stat_schema: dict, delta: dict,
|
||||||
action_index: int) -> tuple[dict, dict]:
|
action_index: int) -> tuple[dict, dict]:
|
||||||
"""Validate/clamp `delta` against `stat_schema` and apply to a copy of
|
"""Validates and clamps `delta` against `stat_schema`, then applies it.
|
||||||
`world_state`. Returns (new_world_state, report)."""
|
|
||||||
|
The delta is applied to a copy of `world_state`. The return value is
|
||||||
|
`(new_world_state, report)`.
|
||||||
|
"""
|
||||||
ws = copy.deepcopy(world_state) if isinstance(world_state, dict) else {}
|
ws = copy.deepcopy(world_state) if isinstance(world_state, dict) else {}
|
||||||
if not ws:
|
if not ws:
|
||||||
ws = instantiate(stat_schema)
|
ws = instantiate(stat_schema)
|
||||||
@@ -512,7 +532,7 @@ def apply_delta(world_state: dict, stat_schema: dict, delta: dict,
|
|||||||
path = str(raw_path)
|
path = str(raw_path)
|
||||||
parts = path.split(".")
|
parts = path.split(".")
|
||||||
|
|
||||||
# flags.<name> — free two-way boolean, either value accepted.
|
# `flags.<name>` is a two-way boolean, and either value is accepted.
|
||||||
if parts[0] == "flags" and len(parts) == 2:
|
if parts[0] == "flags" and len(parts) == 2:
|
||||||
fid = parts[1]
|
fid = parts[1]
|
||||||
if fid not in flag_defs:
|
if fid not in flag_defs:
|
||||||
@@ -528,7 +548,7 @@ def apply_delta(world_state: dict, stat_schema: dict, delta: dict,
|
|||||||
report["applied"].append({"path": path, "old": old, "new": change})
|
report["applied"].append({"path": path, "old": old, "new": change})
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# milestones.<id> — sticky boolean, only `true` accepted.
|
# `milestones.<id>` is a sticky boolean, and only `true` is accepted.
|
||||||
if parts[0] == "milestones" and len(parts) == 2:
|
if parts[0] == "milestones" and len(parts) == 2:
|
||||||
mid = parts[1]
|
mid = parts[1]
|
||||||
if mid not in milestones:
|
if mid not in milestones:
|
||||||
@@ -539,7 +559,7 @@ def apply_delta(world_state: dict, stat_schema: dict, delta: dict,
|
|||||||
continue
|
continue
|
||||||
reached = ws.setdefault("milestones", {})
|
reached = ws.setdefault("milestones", {})
|
||||||
if reached.get(mid, {}).get("reached"):
|
if reached.get(mid, {}).get("reached"):
|
||||||
continue # already done — silent no-op
|
continue # Already reached, so do nothing.
|
||||||
reached[mid] = {"reached": True, "at": action_index}
|
reached[mid] = {"reached": True, "at": action_index}
|
||||||
report["applied"].append({"path": path, "old": False, "new": True})
|
report["applied"].append({"path": path, "old": False, "new": True})
|
||||||
continue
|
continue
|
||||||
@@ -559,7 +579,7 @@ def apply_delta(world_state: dict, stat_schema: dict, delta: dict,
|
|||||||
action_index, meta, report)
|
action_index, meta, report)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# npc.<npcId>.<stat> — each NPC has its own stat defs.
|
# `npc.<npcId>.<stat>`. Each NPC has its own stat definitions.
|
||||||
if parts[0] == "npc" and len(parts) == 3:
|
if parts[0] == "npc" and len(parts) == 3:
|
||||||
ndef = npcs.get(parts[1])
|
ndef = npcs.get(parts[1])
|
||||||
if not isinstance(ndef, dict):
|
if not isinstance(ndef, dict):
|
||||||
@@ -608,8 +628,11 @@ def _stat_line(defs: dict, values: dict) -> str:
|
|||||||
|
|
||||||
def render_state_section(world_state: dict, stat_schema: dict,
|
def render_state_section(world_state: dict, stat_schema: dict,
|
||||||
visible_npcs: dict[str, str]) -> str:
|
visible_npcs: dict[str, str]) -> str:
|
||||||
"""Compact, always-included context block. `visible_npcs` maps card-id ->
|
"""Returns the compact context block that every turn includes.
|
||||||
display name for NPCs currently in scene."""
|
|
||||||
|
`visible_npcs` maps a card id to a display name, for the NPCs currently in
|
||||||
|
the scene.
|
||||||
|
"""
|
||||||
ws = world_state if isinstance(world_state, dict) else {}
|
ws = world_state if isinstance(world_state, dict) else {}
|
||||||
lines: list[str] = []
|
lines: list[str] = []
|
||||||
|
|
||||||
@@ -658,13 +681,16 @@ def render_state_section(world_state: dict, stat_schema: dict,
|
|||||||
|
|
||||||
|
|
||||||
def _describe_stat(name: str, d: dict) -> str | None:
|
def _describe_stat(name: str, d: dict) -> str | None:
|
||||||
"""One reference line for a stat. Description and band-ladder are independent —
|
"""Returns one reference line for a stat.
|
||||||
each is included only when present, so a stat may have either, both, or neither."""
|
|
||||||
|
The description and the band ladder are independent, and each is included
|
||||||
|
only when present, so a stat may have either, both, or neither.
|
||||||
|
"""
|
||||||
bits: list[str] = []
|
bits: list[str] = []
|
||||||
desc = d.get("desc")
|
desc = d.get("desc")
|
||||||
if isinstance(desc, str) and desc.strip():
|
if isinstance(desc, str) and desc.strip():
|
||||||
# Fragments are joined with "; " and end with a single ".", so drop any
|
# Fragments are joined with "; " and end with a single ".", so remove
|
||||||
# trailing period the author already put on the description.
|
# any trailing period the author put on the description.
|
||||||
bits.append(desc.strip().rstrip("."))
|
bits.append(desc.strip().rstrip("."))
|
||||||
if d.get("type") == "text":
|
if d.get("type") == "text":
|
||||||
bits.append("free text")
|
bits.append("free text")
|
||||||
@@ -684,8 +710,12 @@ def _describe_stat(name: str, d: dict) -> str | None:
|
|||||||
|
|
||||||
|
|
||||||
def render_reference(stat_schema: dict) -> str:
|
def render_reference(stat_schema: dict) -> str:
|
||||||
"""A fixed, per-scenario legend describing what each stat means (its `desc`)
|
"""Returns a fixed, per-scenario legend for the stats.
|
||||||
and its band ladder. Static across turns — separate from the live values."""
|
|
||||||
|
Each line gives what a stat means, from its `desc`, and its band ladder. The
|
||||||
|
legend does not change from turn to turn, and it is separate from the live
|
||||||
|
values.
|
||||||
|
"""
|
||||||
lines: list[str] = []
|
lines: list[str] = []
|
||||||
for section in STAT_SECTIONS:
|
for section in STAT_SECTIONS:
|
||||||
for name, d in (stat_schema.get(section) or {}).items():
|
for name, d in (stat_schema.get(section) or {}).items():
|
||||||
|
|||||||
@@ -19,7 +19,8 @@ migrations.bootstrap(engine)
|
|||||||
DEMO_PREFIX = "[Demo]"
|
DEMO_PREFIX = "[Demo]"
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Sample scripts — AI Dungeon contract: define modifier(text), call it last.
|
# Sample scripts. The AI Dungeon contract is to define `modifier(text)` and call
|
||||||
|
# it last.
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
DICE_ROLLER = dict(
|
DICE_ROLLER = dict(
|
||||||
@@ -227,7 +228,7 @@ try:
|
|||||||
scripts = [models.Script(**s) for s in SCRIPTS]
|
scripts = [models.Script(**s) for s in SCRIPTS]
|
||||||
db.add_all(scripts)
|
db.add_all(scripts)
|
||||||
|
|
||||||
# Scenario with cards and scripts attached — public starter content.
|
# A scenario with cards and scripts attached, as public starter content.
|
||||||
scenario = models.Scenario(**SCENARIO, is_public=True)
|
scenario = models.Scenario(**SCENARIO, is_public=True)
|
||||||
scenario.scripts = scripts
|
scenario.scripts = scripts
|
||||||
db.add(scenario)
|
db.add(scenario)
|
||||||
|
|||||||
@@ -1,35 +1,35 @@
|
|||||||
"""Make a database look like an older schema version, so a migration can run.
|
"""Make a database look like an older schema version, so a migration can run.
|
||||||
|
|
||||||
`create_all` always builds the *current* schema. A test that wants to watch a
|
`create_all` always builds the current schema. A test that wants to watch a
|
||||||
migration happen therefore has to take the newer columns back off before it
|
migration run must remove the newer columns first and then stamp an older
|
||||||
stamps an older version — otherwise the migration meets a table that already
|
version. Otherwise the migration finds a column that already exists and
|
||||||
has its column and dies on a duplicate.
|
fails on a duplicate.
|
||||||
|
|
||||||
Rewinding the stamp alone was enough for a while, which is why two test files
|
Rewinding the stamp alone worked for a while, so two test files did exactly
|
||||||
did exactly that. It stopped being enough the moment another `ADD COLUMN`
|
that. It stopped working when another `ADD COLUMN` migration landed. Without
|
||||||
landed after theirs: the replay then runs migrations they never meant to
|
this rewind, the replay runs migrations the tests never intended to
|
||||||
exercise, against columns `create_all` had already made. This module is that
|
exercise, against columns `create_all` already added. This module rewinds
|
||||||
rewind done properly, in one place, so appending a migration means adding its
|
properly in one place. Adding a migration now means adding its inverse here,
|
||||||
inverse here rather than discovering three unrelated test failures.
|
instead of tracking down failures in three unrelated test files.
|
||||||
|
|
||||||
SQLite only — every test that replays migrations runs on a temp file, and
|
This module supports SQLite only. Every test that replays migrations runs on
|
||||||
`PRAGMA user_version` is where the stamp lives there. Migrations that change a
|
a temp file, and SQLite stores the stamp in `PRAGMA user_version`.
|
||||||
column's *type* (43–45, JSON to compressed bytes) have no clean inverse and are
|
Migrations that change a column's type (43-45, JSON to compressed bytes)
|
||||||
not listed: they get replayed as-is, which is what the tests using them already
|
have no clean inverse, so this list omits them. Those migrations replay
|
||||||
relied on.
|
as-is, which is what the tests using them already expect.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from sqlalchemy import text
|
from sqlalchemy import text
|
||||||
from sqlalchemy.engine import Engine
|
from sqlalchemy.engine import Engine
|
||||||
|
|
||||||
# (version that added it, statements that take it back off), newest first.
|
# Each entry is (version that added the column, statements that remove it).
|
||||||
|
# The list is ordered newest first.
|
||||||
#
|
#
|
||||||
# Phase 14's `branch_id` columns are deliberately absent: SQLite refuses to drop
|
# Phase 14's `branch_id` columns are missing on purpose. SQLite refuses to
|
||||||
# a column a foreign key names ("unknown column in foreign key definition"), so
|
# drop a column that a foreign key references, so a current-schema database
|
||||||
# a current-schema database cannot be rewound past them at all. That is what
|
# cannot be rewound past them. `migrations._column_already_there` handles
|
||||||
# `migrations._column_already_there` is for — the replay skips DDL that has
|
# this case. It skips DDL that already ran, so the tree migrations run their
|
||||||
# already happened, so the tree migrations run their backfill against a schema
|
# backfill against a schema that already has the columns.
|
||||||
# that already has the columns, which is exactly the situation here.
|
|
||||||
_UNDO: list[tuple[int, tuple[str, ...]]] = [
|
_UNDO: list[tuple[int, tuple[str, ...]]] = [
|
||||||
# Packed float32 vectors and the flag beside them.
|
# Packed float32 vectors and the flag beside them.
|
||||||
(39, ("ALTER TABLE memories DROP COLUMN embedded",)),
|
(39, ("ALTER TABLE memories DROP COLUMN embedded",)),
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
"""The access log — app/accesslog.py and GET /api/analytics/access.
|
"""The access log: app/accesslog.py and GET /api/analytics/access.
|
||||||
|
|
||||||
This is the half of the analytics work that identifies people on purpose, so
|
This is the half of the analytics work that identifies people on purpose,
|
||||||
the things worth pinning are the ones that would quietly make it wrong: that
|
so these tests pin the details that would quietly make it wrong. The
|
||||||
the address recorded is the hardened one and not a header a client chose, that
|
address recorded must be the hardened one, not a header a client chose.
|
||||||
session rows are thinned instead of written per page load, and that a row
|
Session rows must be thinned instead of written on every page load. And a
|
||||||
outlives the account it describes — guest cleanup runs on a schedule, and a log
|
row must outlive the account it describes, because guest cleanup deletes
|
||||||
that deletes itself is not a log.
|
accounts on a schedule, and a log that deletes itself is not a log.
|
||||||
|
|
||||||
python -m pytest tests/test_accesslog.py -v
|
python -m pytest tests/test_accesslog.py -v
|
||||||
"""
|
"""
|
||||||
@@ -57,7 +57,7 @@ def client(monkeypatch):
|
|||||||
monkeypatch.setattr(auth, "ANALYTICS_EMAILS", {"owner@example.com"})
|
monkeypatch.setattr(auth, "ANALYTICS_EMAILS", {"owner@example.com"})
|
||||||
|
|
||||||
# /auth/me resolves its own session, so the cookie flow below is the real
|
# /auth/me resolves its own session, so the cookie flow below is the real
|
||||||
# one; every other endpoint goes through get_current_user, and `act_as`
|
# one. Every other endpoint goes through get_current_user, and `act_as`
|
||||||
# decides who that is.
|
# decides who that is.
|
||||||
acting = {"id": ids["owner"]}
|
acting = {"id": ids["owner"]}
|
||||||
|
|
||||||
@@ -111,9 +111,9 @@ def test_a_new_session_is_logged(client):
|
|||||||
|
|
||||||
def test_the_address_is_the_hardened_one_not_the_clients(client):
|
def test_the_address_is_the_hardened_one_not_the_clients(client):
|
||||||
visit(client)
|
visit(client)
|
||||||
# The client prepended its own value; only the hop the edge appended counts.
|
# The client prepended its own value. Only the hop the edge appended counts.
|
||||||
# Recording the leftmost would make every row forgeable, which for a log is
|
# Recording the leftmost value would make every row forgeable, which is
|
||||||
# worse than having no log.
|
# worse for a log than having no log at all.
|
||||||
assert rows()[0].ip == EDGE
|
assert rows()[0].ip == EDGE
|
||||||
|
|
||||||
|
|
||||||
@@ -141,15 +141,16 @@ def test_sign_in_and_failure_are_both_logged(client):
|
|||||||
assert accesslog.LOGIN_FAILED in kinds and accesslog.LOGIN in kinds
|
assert accesslog.LOGIN_FAILED in kinds and accesslog.LOGIN in kinds
|
||||||
|
|
||||||
failure = rows(accesslog.LOGIN_FAILED)[0]
|
failure = rows(accesslog.LOGIN_FAILED)[0]
|
||||||
# The address tried, not the account that owns it: a run against an address
|
# This records the address that was tried, not the account it belongs to.
|
||||||
# with no account behind it is exactly what this row is for.
|
# A failed attempt against an address with no matching account is
|
||||||
|
# exactly what this row exists to capture.
|
||||||
assert failure.who == "player@example.com"
|
assert failure.who == "player@example.com"
|
||||||
assert failure.user_id is None
|
assert failure.user_id is None
|
||||||
assert rows(accesslog.LOGIN)[0].user_id == client.ids["member"]
|
assert rows(accesslog.LOGIN)[0].user_id == client.ids["member"]
|
||||||
|
|
||||||
|
|
||||||
def test_registering_is_logged_against_the_upgraded_account(client):
|
def test_registering_is_logged_against_the_upgraded_account(client):
|
||||||
visit(client) # mints the guest whose session registers
|
visit(client) # creates the guest whose session then registers
|
||||||
client.act_as(rows()[0].user_id)
|
client.act_as(rows()[0].user_id)
|
||||||
client.post("/api/auth/register", json={"email": "new@example.com", "password": "hunter2long"})
|
client.post("/api/auth/register", json={"email": "new@example.com", "password": "hunter2long"})
|
||||||
entry = rows(accesslog.REGISTER)[0]
|
entry = rows(accesslog.REGISTER)[0]
|
||||||
@@ -165,8 +166,9 @@ def test_a_row_outlives_the_account_it_describes(client):
|
|||||||
db.commit()
|
db.commit()
|
||||||
finally:
|
finally:
|
||||||
db.close()
|
db.close()
|
||||||
# No foreign key, and `who` is a snapshot — guest cleanup deletes accounts
|
# There is no foreign key, and `who` is a snapshot. Guest cleanup deletes
|
||||||
# on a schedule, and a log that vanishes with them is not a log.
|
# accounts on a schedule, and a log that vanishes along with them is not
|
||||||
|
# a log.
|
||||||
survivor = rows()[0]
|
survivor = rows()[0]
|
||||||
assert survivor.who == entry.who and survivor.ip == EDGE
|
assert survivor.who == entry.who and survivor.ip == EDGE
|
||||||
|
|
||||||
@@ -178,7 +180,7 @@ def test_a_long_user_agent_is_truncated(client):
|
|||||||
|
|
||||||
def test_a_logging_failure_does_not_break_the_request(client, monkeypatch):
|
def test_a_logging_failure_does_not_break_the_request(client, monkeypatch):
|
||||||
monkeypatch.setattr(accesslog, "_client_ip", lambda request: 1 / 0)
|
monkeypatch.setattr(accesslog, "_client_ip", lambda request: 1 / 0)
|
||||||
# The log watches sign-in; it must not be able to stand in its way.
|
# The log observes sign-in. A logging failure must not block the request.
|
||||||
assert visit(client).status_code == 200
|
assert visit(client).status_code == 200
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,12 +1,13 @@
|
|||||||
"""Opening an adventure fetches a window, not the whole story.
|
"""Opening an adventure fetches a window, not the whole story.
|
||||||
|
|
||||||
A story only ever gets longer. Production's longest is 607 actions and 589.5 kB
|
A story only ever gets longer. Production's longest is 607 actions and
|
||||||
in one response, and nothing about that curve bends on its own — so the page
|
589.5 kB in one response, and that number never decreases on its own. The
|
||||||
load returns the newest ACTION_PAGE and the reader pages upward.
|
page load returns the newest `ACTION_PAGE` window, and the reader pages
|
||||||
|
upward from there.
|
||||||
|
|
||||||
The paging anchors on an action id rather than an offset, and these tests are
|
The paging anchors on an action id rather than an offset, and these tests
|
||||||
mostly about why. An offset counted back from the newest shifts every older
|
cover why. An offset counted back from the newest shifts every older
|
||||||
position the moment a turn lands, which is precisely when a reader is likely
|
position the moment a turn lands, which is exactly when a reader is likely
|
||||||
to be scrolling. An anchor means the same thing before and after.
|
to be scrolling. An anchor means the same thing before and after.
|
||||||
|
|
||||||
python -m pytest tests/test_action_paging.py -v
|
python -m pytest tests/test_action_paging.py -v
|
||||||
@@ -104,7 +105,7 @@ def test_the_page_load_returns_only_the_newest_window(client):
|
|||||||
body = r.json()
|
body = r.json()
|
||||||
assert len(body["actions"]) == ACTION_PAGE
|
assert len(body["actions"]) == ACTION_PAGE
|
||||||
assert body["action_count"] == TOTAL
|
assert body["action_count"] == TOTAL
|
||||||
# ...and it is the *newest* window, ending on the last action.
|
# It is the newest window, ending on the last action.
|
||||||
assert body["actions"][-1]["index"] == TOTAL - 1
|
assert body["actions"][-1]["index"] == TOTAL - 1
|
||||||
assert body["actions"][0]["index"] == TOTAL - ACTION_PAGE
|
assert body["actions"][0]["index"] == TOTAL - ACTION_PAGE
|
||||||
|
|
||||||
@@ -123,8 +124,8 @@ def test_a_short_story_is_returned_whole(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_page_load_does_not_grow_with_the_story(client):
|
def test_the_page_load_does_not_grow_with_the_story(client):
|
||||||
"""The point of the change. Whatever the story's length, opening it costs
|
"""Confirm that opening a story costs a window, regardless of the
|
||||||
a window."""
|
story's length."""
|
||||||
meter = dbmeter.Meter()
|
meter = dbmeter.Meter()
|
||||||
meter.attach(engine)
|
meter.attach(engine)
|
||||||
try:
|
try:
|
||||||
@@ -134,8 +135,9 @@ def test_the_page_load_does_not_grow_with_the_story(client):
|
|||||||
finally:
|
finally:
|
||||||
meter.detach()
|
meter.detach()
|
||||||
|
|
||||||
# Each action carries ~1 kB of text and there are 187 of them; a window is
|
# Each action carries about 1 KB of text, and there are 187 of them. A
|
||||||
# 60. Generous ceiling, but far below the whole story.
|
# window holds 60 actions. This ceiling is generous but still far below
|
||||||
|
# the size of the whole story.
|
||||||
assert windowed < ACTION_PAGE * 2_000, f"{windowed:,} B for one window"
|
assert windowed < ACTION_PAGE * 2_000, f"{windowed:,} B for one window"
|
||||||
assert windowed < TOTAL * 500, (
|
assert windowed < TOTAL * 500, (
|
||||||
f"{windowed:,} B — that is the whole story, not a window"
|
f"{windowed:,} B — that is the whole story, not a window"
|
||||||
@@ -182,9 +184,10 @@ def test_each_page_is_ordered_oldest_first(client):
|
|||||||
# ------------------------------------------------- the reason for the anchor
|
# ------------------------------------------------- the reason for the anchor
|
||||||
|
|
||||||
def test_a_turn_arriving_mid_scroll_does_not_shift_the_next_page(client):
|
def test_a_turn_arriving_mid_scroll_does_not_shift_the_next_page(client):
|
||||||
"""The failure an offset would have. Read the newest page, let a turn land,
|
"""Reproduce the failure an offset-based scheme would have. Read the
|
||||||
then page up: the reader must get exactly what precedes what they hold —
|
newest page, let a turn land, then page up. The reader must get exactly
|
||||||
no duplicate, no skipped action."""
|
what precedes the actions they already hold, with no duplicate and no
|
||||||
|
skipped action."""
|
||||||
first = page(client)
|
first = page(client)
|
||||||
oldest_held = first["actions"][0]
|
oldest_held = first["actions"][0]
|
||||||
|
|
||||||
@@ -194,13 +197,13 @@ def test_a_turn_arriving_mid_scroll_does_not_shift_the_next_page(client):
|
|||||||
assert older["actions"][-1]["index"] == oldest_held["index"] - 1, \
|
assert older["actions"][-1]["index"] == oldest_held["index"] - 1, \
|
||||||
"the page shifted when a turn landed"
|
"the page shifted when a turn landed"
|
||||||
assert all(a["index"] < oldest_held["index"] for a in older["actions"])
|
assert all(a["index"] < oldest_held["index"] for a in older["actions"])
|
||||||
# The new turn moved the total, which is fine — it must not move the window.
|
# The new turn changes the total, which is expected. It must not move the window.
|
||||||
assert older["total"] == TOTAL + 1
|
assert older["total"] == TOTAL + 1
|
||||||
|
|
||||||
|
|
||||||
def test_a_deleted_anchor_reports_the_end_rather_than_a_duplicate_page(client):
|
def test_a_deleted_anchor_reports_the_end_rather_than_a_duplicate_page(client):
|
||||||
"""Undo can remove the action a slow scroll was anchored to. Better to stop
|
"""Undo can remove the action a slow scroll was anchored to. The endpoint
|
||||||
than to hand back a page the reader already has."""
|
must stop instead of returning a page the reader already has."""
|
||||||
body = page(client)
|
body = page(client)
|
||||||
anchor = body["actions"][0]
|
anchor = body["actions"][0]
|
||||||
|
|
||||||
@@ -220,7 +223,7 @@ def test_a_deleted_anchor_reports_the_end_rather_than_a_duplicate_page(client):
|
|||||||
|
|
||||||
def test_limit_is_honoured_and_capped(client):
|
def test_limit_is_honoured_and_capped(client):
|
||||||
assert len(page(client, limit=5)["actions"]) == 5
|
assert len(page(client, limit=5)["actions"]) == 5
|
||||||
# A client asking for the whole story does not get to undo the paging.
|
# A client that asks for the whole story cannot bypass the paging cap.
|
||||||
assert len(page(client, limit=100_000)["actions"]) <= ACTION_PAGE * 4
|
assert len(page(client, limit=100_000)["actions"]) <= ACTION_PAGE * 4
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
"""Visit analytics — app/analytics.py and the two endpoints in front of it.
|
"""Visit analytics: app/analytics.py and the two endpoints in front of it.
|
||||||
|
|
||||||
Three things are worth testing here and the rest is arithmetic. That the
|
This file tests three things, and the rest is arithmetic. The counters must
|
||||||
counters survive the buffer/UPSERT round trip (a flush must add to what is
|
survive the buffer/UPSERT round trip: a flush adds to what is already
|
||||||
already stored, not replace it, or every number is only ever the last minute).
|
stored instead of replacing it, or every number would show only the last
|
||||||
That the funnel counts *people* rather than clicks, which is the only reason
|
minute. The funnel counts people rather than clicks, which is the only
|
||||||
the visitor-day table exists. And that the gate holds: a stranger cannot read
|
reason the visitor-day table exists. The gate holds: a stranger cannot read
|
||||||
the dashboard, and cannot inflate what it says beyond hitting the page.
|
the dashboard, and cannot inflate what it reports beyond hitting the page.
|
||||||
|
|
||||||
python -m pytest tests/test_analytics.py -v
|
python -m pytest tests/test_analytics.py -v
|
||||||
"""
|
"""
|
||||||
@@ -102,7 +102,7 @@ def test_label_cardinality_is_capped(db):
|
|||||||
analytics.flush(db)
|
analytics.flush(db)
|
||||||
labels = db.query(models.AnalyticsDaily).filter_by(metric=analytics.M_REFERRER).count()
|
labels = db.query(models.AnalyticsDaily).filter_by(metric=analytics.M_REFERRER).count()
|
||||||
# Everything past the cap is folded into one bucket, so a referrer flood
|
# Everything past the cap is folded into one bucket, so a referrer flood
|
||||||
# cannot mint rows without limit.
|
# cannot create unlimited rows.
|
||||||
assert labels == analytics.MAX_LABELS_PER_METRIC + 1
|
assert labels == analytics.MAX_LABELS_PER_METRIC + 1
|
||||||
assert counter(db, analytics.M_REFERRER, analytics.OTHER) == 25
|
assert counter(db, analytics.M_REFERRER, analytics.OTHER) == 25
|
||||||
|
|
||||||
@@ -115,8 +115,9 @@ def test_visitor_id_is_stable_and_keyed(db, monkeypatch):
|
|||||||
assert handle == analytics.visitor_id(user) # a returning visitor
|
assert handle == analytics.visitor_id(user) # a returning visitor
|
||||||
assert handle != analytics.visitor_id(make_user(db)) # is still one visitor
|
assert handle != analytics.visitor_id(make_user(db)) # is still one visitor
|
||||||
assert len(handle) == 32 and int(handle, 16) >= 0 # opaque hex, not an id
|
assert len(handle) == 32 and int(handle, 16) >= 0 # opaque hex, not an id
|
||||||
# Keyed on the app secret, not a bare hash of the user id: otherwise anyone
|
# Keyed on the app secret, not a bare hash of the user id. Otherwise
|
||||||
# holding this table could rebuild the mapping by hashing 1, 2, 3, …
|
# anyone holding this table could rebuild the mapping by hashing
|
||||||
|
# sequential ids.
|
||||||
monkeypatch.setattr(analytics.security, "SECRET_KEY", b"a-different-secret")
|
monkeypatch.setattr(analytics.security, "SECRET_KEY", b"a-different-secret")
|
||||||
assert analytics.visitor_id(user) != handle
|
assert analytics.visitor_id(user) != handle
|
||||||
|
|
||||||
@@ -128,8 +129,8 @@ def test_a_repeat_visitor_is_new_only_once(db):
|
|||||||
rows = db.query(models.AnalyticsVisitorDay).all()
|
rows = db.query(models.AnalyticsVisitorDay).all()
|
||||||
assert len(rows) == 1 and rows[0].is_new
|
assert len(rows) == 1 and rows[0].is_new
|
||||||
|
|
||||||
# Same visitor, a later day: seen before, so not new — and not merged into
|
# Same visitor, a later day: seen before, so not new. The row is not
|
||||||
# the first day's row either.
|
# merged into the first day's row either.
|
||||||
tomorrow = (models.utcnow().date() + timedelta(days=1)).isoformat()
|
tomorrow = (models.utcnow().date() + timedelta(days=1)).isoformat()
|
||||||
analytics._visits[(tomorrow, analytics.visitor_id(user))] = set()
|
analytics._visits[(tomorrow, analytics.visitor_id(user))] = set()
|
||||||
analytics.flush(db)
|
analytics.flush(db)
|
||||||
@@ -234,7 +235,7 @@ def test_summary_counts_people_once_per_step(db):
|
|||||||
steps = {row["step"]: row["count"] for row in result["funnel"]}
|
steps = {row["step"]: row["count"] for row in result["funnel"]}
|
||||||
assert steps["Visited"] == 2
|
assert steps["Visited"] == 2
|
||||||
assert steps["Opened a scenario"] == 2
|
assert steps["Opened a scenario"] == 2
|
||||||
assert steps["Played a turn"] == 1 # not 3 — one person, three turns
|
assert steps["Played a turn"] == 1 # not 3, because one person made three turns
|
||||||
assert steps["Signed up"] == 0
|
assert steps["Signed up"] == 0
|
||||||
# Raw event totals still count every occurrence.
|
# Raw event totals still count every occurrence.
|
||||||
assert result["totals"]["turns"] == 3
|
assert result["totals"]["turns"] == 3
|
||||||
@@ -361,10 +362,11 @@ def test_api_errors_are_counted_by_route(client):
|
|||||||
# ---------- The dialect the tests never run on ----------
|
# ---------- The dialect the tests never run on ----------
|
||||||
|
|
||||||
def test_the_upserts_compile_for_postgres():
|
def test_the_upserts_compile_for_postgres():
|
||||||
"""Prod is Neon; these tests are SQLite, and a failed flush is caught and
|
"""Prod runs on Neon, but these tests run on SQLite, and a failed flush
|
||||||
logged rather than raised. A dialect mistake would therefore be invisible
|
is caught and logged instead of raised. A dialect mistake would
|
||||||
until the dashboard quietly stayed empty — so compile both statements
|
therefore stay invisible until the dashboard quietly stayed empty. This
|
||||||
against Postgres without ever connecting to one.
|
test compiles both statements against Postgres without connecting to
|
||||||
|
one.
|
||||||
"""
|
"""
|
||||||
from sqlalchemy import create_engine
|
from sqlalchemy import create_engine
|
||||||
from sqlalchemy.dialects import postgresql
|
from sqlalchemy.dialects import postgresql
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
"""Phase 14 SP4 — a retry writes a sibling node instead of rewriting a row.
|
"""Phase 14 SP4: a retry writes a sibling node instead of rewriting a row.
|
||||||
|
|
||||||
`test_retry_variants.py` is the behavioural contract, unchanged since before
|
`test_retry_variants.py` is the behavioral contract from before the tree
|
||||||
the tree, and it still passes: the same URLs, the same payload shape, the same
|
existed, and it still passes unchanged: the same URLs, the same payload
|
||||||
outcomes. This file asserts the things that are *only* true of the new storage
|
shape, the same outcomes. This file asserts the things that are true only of
|
||||||
— that a turn can be several rows, that exactly one of them is the story, and
|
the new storage. A turn can be several rows, exactly one of them is the
|
||||||
that the arrangement costs neither an extra prompt nor an extra turn.
|
story, and the arrangement costs neither an extra prompt nor an extra turn.
|
||||||
|
|
||||||
python -m pytest tests/test_attempt_siblings.py -v
|
python -m pytest tests/test_attempt_siblings.py -v
|
||||||
"""
|
"""
|
||||||
@@ -122,8 +122,9 @@ def _page(client) -> dict:
|
|||||||
def _rows(adv_id) -> list[models.Action]:
|
def _rows(adv_id) -> list[models.Action]:
|
||||||
"""Every action row of the adventure, story or not, live or not.
|
"""Every action row of the adventure, story or not, live or not.
|
||||||
|
|
||||||
Undeferred, because the session is closed before the caller looks: the
|
The query undefers these columns because the session closes before the
|
||||||
columns this file is about are exactly the ones a page load never loads.
|
caller reads the result. This file specifically tests the columns that a
|
||||||
|
page load never loads.
|
||||||
"""
|
"""
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
@@ -156,7 +157,7 @@ def test_a_retry_writes_a_second_row_at_the_same_coordinate(client):
|
|||||||
assert [a.text for a in ai] == ["Attempt one.", "Attempt two."]
|
assert [a.text for a in ai] == ["Attempt one.", "Attempt two."]
|
||||||
# Exactly one of them is the story, and it is the newer take.
|
# Exactly one of them is the story, and it is the newer take.
|
||||||
assert [a.live for a in ai] == [False, True]
|
assert [a.live for a in ai] == [False, True]
|
||||||
# ...and the discarded attempt is untouched, not a copy of anything.
|
# The discarded attempt is untouched, not a copy of anything.
|
||||||
assert ai[0].state_after is not None
|
assert ai[0].state_after is not None
|
||||||
|
|
||||||
|
|
||||||
@@ -173,9 +174,10 @@ def test_the_story_shows_and_counts_the_turn_once(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_discarded_attempt_never_reaches_the_prompt(client):
|
def test_a_discarded_attempt_never_reaches_the_prompt(client):
|
||||||
"""The trap the branch clause exists to close, at sibling scale: the losing
|
"""This is the failure case the branch clause exists to prevent, at
|
||||||
attempt sits at the same branch and depth as the live one, so anything
|
sibling scale. The losing attempt sits at the same branch and depth as
|
||||||
reading the story by coordinate alone would replay both."""
|
the live one, so anything reading the story by coordinate alone would
|
||||||
|
replay both."""
|
||||||
ScriptedProvider.replies = ["Attempt one.", "Attempt two.", "Next turn."]
|
ScriptedProvider.replies = ["Attempt one.", "Attempt two.", "Next turn."]
|
||||||
_play(client)
|
_play(client)
|
||||||
_retry(client)
|
_retry(client)
|
||||||
@@ -198,21 +200,22 @@ def test_switching_moves_the_story_onto_the_other_row(client):
|
|||||||
r = client.post(
|
r = client.post(
|
||||||
f"/api/adventures/{client.adv_id}/actions/{newest_id}/variant", json={"index": 0})
|
f"/api/adventures/{client.adv_id}/actions/{newest_id}/variant", json={"index": 0})
|
||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
# A different row answers — that is the whole change.
|
# A different row answers the request. That is the only change.
|
||||||
assert r.json()["id"] != newest_id
|
assert r.json()["id"] != newest_id
|
||||||
assert r.json()["text"].startswith("A scratch")
|
assert r.json()["text"].startswith("A scratch")
|
||||||
|
|
||||||
rows = _rows(client.adv_id)
|
rows = _rows(client.adv_id)
|
||||||
ai = [a for a in rows if a.type == "ai"]
|
ai = [a for a in rows if a.type == "ai"]
|
||||||
assert [a.live for a in ai] == [True, False]
|
assert [a.live for a in ai] == [True, False]
|
||||||
# Both takes are still there, byte for byte.
|
# Both takes remain unchanged in the database.
|
||||||
assert [a.text.split(".")[0] for a in ai] == ["A scratch", "A beating"]
|
assert [a.text.split(".")[0] for a in ai] == ["A scratch", "A beating"]
|
||||||
|
|
||||||
|
|
||||||
def test_the_assembled_prompt_is_stored_once_per_turn(client):
|
def test_the_assembled_prompt_is_stored_once_per_turn(client):
|
||||||
"""A snapshot is ~160 kB of prompt every attempt at a turn shares. Giving
|
"""A snapshot holds about 160 kB of prompt that every attempt at a turn
|
||||||
each sibling a copy would have made retry a permanent multiplier on the
|
shares. Giving each sibling its own copy would make retry multiply the
|
||||||
biggest column in the database, so the prompt moves with the live flag."""
|
size of the largest column in the database. Instead, the prompt moves
|
||||||
|
with the live flag."""
|
||||||
ScriptedProvider.replies = ["Attempt one.", "Attempt two."]
|
ScriptedProvider.replies = ["Attempt one.", "Attempt two."]
|
||||||
_play(client)
|
_play(client)
|
||||||
_retry(client)
|
_retry(client)
|
||||||
@@ -278,12 +281,12 @@ def test_deleting_a_turn_through_a_discarded_attempt_still_takes_the_turn(client
|
|||||||
def test_retrying_withdraws_the_memory_the_turn_produced(client):
|
def test_retrying_withdraws_the_memory_the_turn_produced(client):
|
||||||
"""Why summarization no longer holds the newest action back.
|
"""Why summarization no longer holds the newest action back.
|
||||||
|
|
||||||
A memory covering the newest turn used to be unreachable-by-construction:
|
A memory covering the newest turn used to be unreachable by
|
||||||
the summarizer stopped one action short, because a retry rewrote the row
|
construction. The summarizer stopped one action short, because a retry
|
||||||
under a mark that had already moved past it. Now the mark and the memory
|
rewrote the row under a mark that had already moved past it. Now the
|
||||||
both name the node, and replacing what a node says withdraws them — the
|
mark and the memory both name the node, so replacing what a node says
|
||||||
same repair undo and delete already made, so the holdback was the only
|
withdraws them. Undo and delete already had this repair; the holdback
|
||||||
thing left that a retry needed.
|
was the only gap a retry still needed to close.
|
||||||
"""
|
"""
|
||||||
ScriptedProvider.replies = ["One.", "Two."]
|
ScriptedProvider.replies = ["One.", "Two."]
|
||||||
_play(client)
|
_play(client)
|
||||||
@@ -311,8 +314,8 @@ def test_retrying_withdraws_the_memory_the_turn_produced(client):
|
|||||||
try:
|
try:
|
||||||
adventure = db.get(models.Adventure, client.adv_id)
|
adventure = db.get(models.Adventure, client.adv_id)
|
||||||
assert db.query(models.Memory).count() == 0, "the withdrawn memory is gone"
|
assert db.query(models.Memory).count() == 0, "the withdrawn memory is gone"
|
||||||
# ...and the ground it covered is handed back, so the block is summarized
|
# The depth range it covered is released, so the block is
|
||||||
# again from where it began rather than silently skipped.
|
# summarized again from where it began instead of being skipped.
|
||||||
assert cursors.MEMORY.depth(db, adventure) == 0
|
assert cursors.MEMORY.depth(db, adventure) == 0
|
||||||
assert cursors.SUMMARY.depth(db, adventure) == 0
|
assert cursors.SUMMARY.depth(db, adventure) == 0
|
||||||
assert covered_depth > 0
|
assert covered_depth > 0
|
||||||
@@ -382,12 +385,14 @@ def test_attempts_module_agrees_with_the_endpoint(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_export_carries_every_attempt_as_its_own_node(client):
|
def test_export_carries_every_attempt_as_its_own_node(client):
|
||||||
"""SP6 changed the answer here, and the reason is the whole of that subphase.
|
"""SP6 changed the answer here, for the same reasons as the rest of that
|
||||||
|
subphase.
|
||||||
|
|
||||||
A v1 bundle had one entry per turn and folded the group back into a
|
A v1 bundle had one entry per turn and folded the group back into a
|
||||||
`variants` array, because the format had nowhere else to put a second take.
|
`variants` array, because the format had nowhere else to put a second
|
||||||
A v2 bundle has coordinates, so an attempt is a node in the file exactly as
|
take. A v2 bundle has coordinates, so an attempt is a node in the file
|
||||||
it is a node in the database, and `live` says which one is the story.
|
exactly as it is a node in the database, and `live` says which one is
|
||||||
|
the story.
|
||||||
"""
|
"""
|
||||||
ScriptedProvider.replies = ["One.", "Two."]
|
ScriptedProvider.replies = ["One.", "Two."]
|
||||||
_play(client)
|
_play(client)
|
||||||
@@ -399,7 +404,7 @@ def test_export_carries_every_attempt_as_its_own_node(client):
|
|||||||
assert len({(a["branch"], a["depth"]) for a in ai}) == 1, "one turn, two takes"
|
assert len({(a["branch"], a["depth"]) for a in ai}) == 1, "one turn, two takes"
|
||||||
assert "variants" not in ai[0], "nothing writes the repeating group any more"
|
assert "variants" not in ai[0], "nothing writes the repeating group any more"
|
||||||
|
|
||||||
# ...and importing it puts the group back exactly as it stood.
|
# Importing the bundle puts the group back exactly as it stood.
|
||||||
imported = client.post("/api/adventures/import", json=bundle).json()["id"]
|
imported = client.post("/api/adventures/import", json=bundle).json()["id"]
|
||||||
rows = _rows(imported)
|
rows = _rows(imported)
|
||||||
ai_rows = [a for a in rows if a.type == "ai"]
|
ai_rows = [a for a in rows if a.type == "ai"]
|
||||||
@@ -410,11 +415,12 @@ def test_export_carries_every_attempt_as_its_own_node(client):
|
|||||||
def test_a_retry_after_switching_back_files_the_new_attempt_last(client):
|
def test_a_retry_after_switching_back_files_the_new_attempt_last(client):
|
||||||
"""The group stays in the order the attempts were made.
|
"""The group stays in the order the attempts were made.
|
||||||
|
|
||||||
`add_attempt` used to number a new take one past the take it replaced, which
|
`add_attempt` used to number a new take one past the take it replaced.
|
||||||
is the end of the group only when the story is standing on the newest one.
|
That numbering is correct only when the story is standing on the newest
|
||||||
Switch a three-take turn back to the first and retry, and the new attempt
|
take. Switch a three-take turn back to the first and retry, and the new
|
||||||
collided with take 2 — `renumber` then broke the tie by id and filed it
|
attempt collides with take 2. `renumber` broke the tie by id and placed
|
||||||
*between* takes 2 and 3, so the pager walked them in an order nobody played.
|
the new attempt between takes 2 and 3, so the pager listed them in an
|
||||||
|
order the player never produced.
|
||||||
"""
|
"""
|
||||||
ScriptedProvider.replies = ["One.", "Two.", "Three.", "Four."]
|
ScriptedProvider.replies = ["One.", "Two.", "Three.", "Four."]
|
||||||
_play(client)
|
_play(client)
|
||||||
@@ -439,9 +445,10 @@ def test_a_retry_after_switching_back_files_the_new_attempt_last(client):
|
|||||||
def test_the_adventure_list_quotes_the_take_the_story_tells(client):
|
def test_the_adventure_list_quotes_the_take_the_story_tells(client):
|
||||||
"""The index screen and the story have to agree.
|
"""The index screen and the story have to agree.
|
||||||
|
|
||||||
Siblings share a depth and the newest of them has the highest id, so a
|
Siblings share a depth, and the newest of them has the highest id. A
|
||||||
snippet ordered by `(depth, id)` alone quotes whichever attempt was written
|
snippet ordered by `(depth, id)` alone quotes whichever attempt was
|
||||||
last — which, after switching back, is the one the player threw away.
|
written last. After switching back, that attempt is the one the player
|
||||||
|
discarded.
|
||||||
"""
|
"""
|
||||||
ScriptedProvider.replies = ["One.", "Two."]
|
ScriptedProvider.replies = ["One.", "Two."]
|
||||||
_play(client)
|
_play(client)
|
||||||
|
|||||||
@@ -1,21 +1,23 @@
|
|||||||
"""Phase 14 SP2 — a read sees one story, and knows which one.
|
"""Phase 14 SP2: a read sees one story, and knows which one.
|
||||||
|
|
||||||
These tests build the fork by hand: three branch rows and their nodes, written
|
These tests build the fork by hand. Three branch rows and their nodes are
|
||||||
straight to the database, arranged as the design doc's own worked example. That
|
written straight to the database, arranged as the design doc's own worked
|
||||||
was the only way to build one when this file was written (nothing forked until
|
example. That was the only way to build one when this file was written,
|
||||||
SP5) and it stays that way now that `tree.fork` exists — a fixture that agreed
|
because nothing forked until SP5. It stays that way now that `tree.fork`
|
||||||
with the code under test could not catch it being wrong. The two are checked
|
exists, because a fixture built with the same code under test could not
|
||||||
against each other in `test_branch_forking.py`.
|
catch that code being wrong. `test_branch_forking.py` checks the two
|
||||||
|
against each other.
|
||||||
|
|
||||||
branch C, tip at depth 7, lineage [(C, 7), (B, 5), (A, 3)]
|
branch C, tip at depth 7, lineage [(C, 7), (B, 5), (A, 3)]
|
||||||
→ A0 A1 A2 A3 B4 B5 C6 C7
|
-> A0 A1 A2 A3 B4 B5 C6 C7
|
||||||
|
|
||||||
The point of building it by hand is that every read in the app is supposed to
|
The point of building the fixture by hand is that every read in the app
|
||||||
go through one module, and a forgotten clause does not raise — it quietly shows
|
must go through one module. A forgotten clause does not raise an error. It
|
||||||
a story assembled out of two different ones. So the fixture deliberately leaves
|
silently shows a story assembled out of two different branches. The
|
||||||
nodes lying where a forgotten clause would pick them up: A kept playing past
|
fixture deliberately leaves nodes where a forgotten clause would pick them
|
||||||
the fork (A4, A5), B kept playing past its own (B6), and a second adventure
|
up: A kept playing past the fork (A4, A5), B kept playing past its own
|
||||||
holds a whole story of its own. None of them may appear on C.
|
(B6), and a second adventure holds a whole story of its own. None of these
|
||||||
|
nodes may appear on C.
|
||||||
|
|
||||||
python -m pytest tests/test_branch_clause.py -v
|
python -m pytest tests/test_branch_clause.py -v
|
||||||
"""
|
"""
|
||||||
@@ -46,8 +48,9 @@ from tools import dbmeter
|
|||||||
def make_branch(db, adventure, parent=None, fork_depth=None):
|
def make_branch(db, adventure, parent=None, fork_depth=None):
|
||||||
"""A branch row whose lineage is its parent's, capped, plus itself.
|
"""A branch row whose lineage is its parent's, capped, plus itself.
|
||||||
|
|
||||||
The same computation SP5 will do at fork time; written out here so the
|
This function performs the same computation SP5 does at fork time. It
|
||||||
fixture cannot pass by agreeing with a bug in the code under test.
|
is written out here so the fixture cannot pass by repeating a bug in
|
||||||
|
the code under test.
|
||||||
"""
|
"""
|
||||||
branch = models.Branch(
|
branch = models.Branch(
|
||||||
adventure_id=adventure.id,
|
adventure_id=adventure.id,
|
||||||
@@ -89,7 +92,7 @@ def make_adventure(db, user, title):
|
|||||||
|
|
||||||
@pytest.fixture()
|
@pytest.fixture()
|
||||||
def forked():
|
def forked():
|
||||||
"""The worked example, plus everything a forgotten clause would sweep up."""
|
"""The worked example, plus everything a forgotten clause would expose."""
|
||||||
Base.metadata.create_all(bind=engine)
|
Base.metadata.create_all(bind=engine)
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
user = models.User(is_guest=False, email="branch@example.com")
|
user = models.User(is_guest=False, email="branch@example.com")
|
||||||
@@ -158,9 +161,10 @@ def test_the_worked_example_reads_back_as_the_design_doc_says(forked):
|
|||||||
def test_a_siblings_nodes_are_invisible(forked):
|
def test_a_siblings_nodes_are_invisible(forked):
|
||||||
db, adventure, _ = forked
|
db, adventure, _ = forked
|
||||||
seen = labels(history.story_actions(adventure))
|
seen = labels(history.story_actions(adventure))
|
||||||
# A4/A5 are A's own continuation past B's fork; B6 is B's past C's.
|
# A4 and A5 are A's own continuation past B's fork. B6 is B's own
|
||||||
|
# continuation past C's fork.
|
||||||
assert "A4" not in seen and "A5" not in seen and "B6" not in seen
|
assert "A4" not in seen and "A5" not in seen and "B6" not in seen
|
||||||
# And nothing from the adventure next door.
|
# Nothing from the other adventure appears either.
|
||||||
assert not [text for text in seen if text.startswith("X")]
|
assert not [text for text in seen if text.startswith("X")]
|
||||||
|
|
||||||
|
|
||||||
@@ -213,11 +217,12 @@ def test_a_window_that_reaches_past_the_fork_still_reads_in_order(forked):
|
|||||||
# ------------------------------------------- the loaded-collection short cut
|
# ------------------------------------------- the loaded-collection short cut
|
||||||
|
|
||||||
def test_an_already_loaded_collection_is_cut_down_to_the_path(forked):
|
def test_an_already_loaded_collection_is_cut_down_to_the_path(forked):
|
||||||
"""`history._from_memory`'s shortcut, which was the highest-risk line here.
|
"""Tests `history._from_memory`'s shortcut, the highest-risk line here.
|
||||||
|
|
||||||
`adventure.actions` is every branch's actions. Slicing it without the path
|
`adventure.actions` returns every branch's actions. Slicing it without
|
||||||
would assemble a prompt out of two different stories, and nothing would
|
the path would assemble a prompt out of two different stories, and
|
||||||
raise — so load it deliberately and check the answer is the path anyway.
|
nothing would raise an error. This test loads the collection
|
||||||
|
deliberately and checks that the answer is still the path.
|
||||||
"""
|
"""
|
||||||
db, adventure, _ = forked
|
db, adventure, _ = forked
|
||||||
loaded = list(adventure.actions) # every branch, ordered by index
|
loaded = list(adventure.actions) # every branch, ordered by index
|
||||||
@@ -230,8 +235,8 @@ def test_an_already_loaded_collection_is_cut_down_to_the_path(forked):
|
|||||||
|
|
||||||
|
|
||||||
def test_user_scripts_are_handed_the_path(forked):
|
def test_user_scripts_are_handed_the_path(forked):
|
||||||
"""The same trap, one layer up and user-visible: `pipeline._history()` is
|
"""The same risk one layer up, in code visible to users:
|
||||||
the documented scripting history API."""
|
`pipeline._history()` is the documented scripting history API."""
|
||||||
db, adventure, _ = forked
|
db, adventure, _ = forked
|
||||||
list(adventure.actions) # the pipeline's caller has usually loaded these
|
list(adventure.actions) # the pipeline's caller has usually loaded these
|
||||||
pipeline = ScriptPipeline(adventure, db)
|
pipeline = ScriptPipeline(adventure, db)
|
||||||
@@ -268,8 +273,8 @@ def test_the_page_the_reader_opens_is_the_path(client):
|
|||||||
assert [a["text"] for a in body["actions"]] == [
|
assert [a["text"] for a in body["actions"]] == [
|
||||||
"A0", "A1", "A2", "A3", "B4", "B5", "C6", "C7"
|
"A0", "A1", "A2", "A3", "B4", "B5", "C6", "C7"
|
||||||
]
|
]
|
||||||
# `action_count` is what tells the reader there is more above, so it counts
|
# `action_count` tells the reader whether more actions exist above. It
|
||||||
# the path too — 8, not the 13 rows the adventure holds.
|
# counts the path too: 8, not the 13 rows the adventure holds.
|
||||||
assert body["action_count"] == 8
|
assert body["action_count"] == 8
|
||||||
|
|
||||||
|
|
||||||
@@ -324,11 +329,11 @@ def test_the_index_screen_quotes_the_branch_being_played(client):
|
|||||||
# ------------------------------------------------------- the flush guard
|
# ------------------------------------------------------- the flush guard
|
||||||
|
|
||||||
def test_a_node_written_without_a_branch_is_placed_anyway(forked):
|
def test_a_node_written_without_a_branch_is_placed_anyway(forked):
|
||||||
"""SP1 wired the writers; from SP2 an unplaced node is an invisible one.
|
"""SP1 wired the writers. Since SP2, an unplaced node is an invisible one.
|
||||||
|
|
||||||
This is what lets a fixture, a script or a test built straight through the
|
This behavior lets a fixture, a script, or a test built straight through
|
||||||
ORM keep working — and it is why the baseline contract still passes with
|
the ORM keep working. It is also why the baseline contract still passes
|
||||||
its actions written directly to the database.
|
with its actions written directly to the database.
|
||||||
"""
|
"""
|
||||||
db, adventure, ids = forked
|
db, adventure, ids = forked
|
||||||
written = models.Action(
|
written = models.Action(
|
||||||
@@ -356,11 +361,11 @@ def test_a_memory_written_without_a_branch_is_placed_anyway(forked):
|
|||||||
def test_placing_a_flush_of_nodes_reads_the_branch_once(forked, emitted_sql):
|
def test_placing_a_flush_of_nodes_reads_the_branch_once(forked, emitted_sql):
|
||||||
"""The guard resolves the head once per flush, not once per node.
|
"""The guard resolves the head once per flush, not once per node.
|
||||||
|
|
||||||
The identity map holds weak references, so a branch row nobody keeps a
|
The identity map holds weak references. A branch row with no strong
|
||||||
strong reference to is collected between two nodes and read back for the
|
reference gets collected between two nodes and read back again for the
|
||||||
next one. Writing two hundred actions in one flush was two hundred SELECTs
|
next one. Writing two hundred actions in one flush ran two hundred
|
||||||
on `branches` before the head was hoisted out of the loop, and nothing
|
SELECTs on `branches` before the head lookup moved outside the loop.
|
||||||
about the result would have told you.
|
The test result alone would not have shown this.
|
||||||
"""
|
"""
|
||||||
db, adventure, _ = forked
|
db, adventure, _ = forked
|
||||||
emitted_sql.clear()
|
emitted_sql.clear()
|
||||||
@@ -376,10 +381,11 @@ def test_placing_a_flush_of_nodes_reads_the_branch_once(forked, emitted_sql):
|
|||||||
|
|
||||||
|
|
||||||
def test_an_adventure_with_no_branch_at_all_reads_as_empty(forked):
|
def test_an_adventure_with_no_branch_at_all_reads_as_empty(forked):
|
||||||
"""The loud version of a missing branch: nothing, rather than everything.
|
"""A missing branch must fail loudly: nothing, rather than everything.
|
||||||
|
|
||||||
A row with no branch cannot be shown without guessing which story it is
|
A row with no branch cannot be shown without guessing which story it
|
||||||
on, and a guess here is how a sibling's turns end up in a prompt.
|
belongs to, and a wrong guess here puts a sibling's turns into a
|
||||||
|
prompt.
|
||||||
"""
|
"""
|
||||||
db, adventure, ids = forked
|
db, adventure, ids = forked
|
||||||
stray = make_adventure(db, db.get(models.User, ids["user"]), "Stray")
|
stray = make_adventure(db, db.get(models.User, ids["user"]), "Stray")
|
||||||
@@ -400,9 +406,9 @@ def test_an_adventure_with_no_branch_at_all_reads_as_empty(forked):
|
|||||||
def deeply_forked():
|
def deeply_forked():
|
||||||
"""A story forked twenty times, then played forty turns past the last one.
|
"""A story forked twenty times, then played forty turns past the last one.
|
||||||
|
|
||||||
The shape the design is betting on: reading the tail of this must cost what
|
This shape tests the design's core assumption: reading the tail of this
|
||||||
reading the tail of an unforked story costs, because the window is covered
|
story must cost the same as reading the tail of an unforked story,
|
||||||
long before the ancestry runs out.
|
because the window is covered long before the ancestry runs out.
|
||||||
"""
|
"""
|
||||||
Base.metadata.create_all(bind=engine)
|
Base.metadata.create_all(bind=engine)
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
@@ -477,7 +483,7 @@ def test_a_window_reaching_past_the_forks_names_only_what_it_needs(
|
|||||||
rows = history.tail(adventure, 41) # 40 on the tip branch, one older
|
rows = history.tail(adventure, 41) # 40 on the tip branch, one older
|
||||||
assert len(rows) == 41
|
assert len(rows) == 41
|
||||||
selects = [s for s in emitted_sql if "FROM actions" in s and branch_terms(s)]
|
selects = [s for s in emitted_sql if "FROM actions" in s and branch_terms(s)]
|
||||||
# Two lineage entries reach 41 deep (40 + 2); the other twenty stay
|
# Two lineage entries reach 41 deep (40 + 2). The other twenty stay
|
||||||
# unnamed. Every fork past the window costs the query nothing.
|
# unnamed. Every fork past the window costs the query nothing.
|
||||||
assert max(branch_terms(s) for s in selects) == 2
|
assert max(branch_terms(s) for s in selects) == 2
|
||||||
|
|
||||||
@@ -495,13 +501,13 @@ def test_the_estimate_is_arithmetic_not_a_query(deeply_forked):
|
|||||||
def test_forking_twenty_times_costs_the_same_bytes_as_never_forking(
|
def test_forking_twenty_times_costs_the_same_bytes_as_never_forking(
|
||||||
deeply_forked,
|
deeply_forked,
|
||||||
):
|
):
|
||||||
"""The design's bet, in bytes.
|
"""The design's cost assumption, measured in bytes.
|
||||||
|
|
||||||
Two stories of the same length, one played straight through and one forked
|
Two stories of the same length, one played straight through and one
|
||||||
twenty times, read their newest window for the same money — because the
|
forked twenty times, cost the same to read their newest window. The
|
||||||
window is covered by the newest lineage entry either way, and the ancestry
|
window is covered by the newest lineage entry either way, and the
|
||||||
is never named. The forked read pays for one extra row: the branch it read
|
ancestry is never named. The forked read pays for one extra row: the
|
||||||
the lineage off.
|
branch it reads the lineage from.
|
||||||
"""
|
"""
|
||||||
db, forked_adventure = deeply_forked
|
db, forked_adventure = deeply_forked
|
||||||
flat = make_adventure(db, db.get(models.User, forked_adventure.user_id), "Flat")
|
flat = make_adventure(db, db.get(models.User, forked_adventure.user_id), "Flat")
|
||||||
@@ -511,10 +517,10 @@ def test_forking_twenty_times_costs_the_same_bytes_as_never_forking(
|
|||||||
flat.head_branch_id = branch.id
|
flat.head_branch_id = branch.id
|
||||||
flat.head_depth = 83
|
flat.head_depth = 83
|
||||||
flat_id, forked_id = flat.id, forked_adventure.id
|
flat_id, forked_id = flat.id, forked_adventure.id
|
||||||
# Commit and let go of the connection: the meter wraps the pool's factory,
|
# Commit and release the connection. The meter wraps the connection
|
||||||
# so a connection checked out before it attaches is a connection it never
|
# pool's factory, so a connection checked out before the meter attaches
|
||||||
# sees. Building the fixture is a write path nobody plays, and is not
|
# is never visible to it. Building the fixture is a write path the test
|
||||||
# charged to either scope.
|
# does not measure, and it is not charged to either scope.
|
||||||
db.commit()
|
db.commit()
|
||||||
db.expire_all()
|
db.expire_all()
|
||||||
|
|
||||||
@@ -541,8 +547,8 @@ def test_forking_twenty_times_costs_the_same_bytes_as_never_forking(
|
|||||||
def test_a_gap_in_the_story_widens_the_read_rather_than_shortening_it(
|
def test_a_gap_in_the_story_widens_the_read_rather_than_shortening_it(
|
||||||
deeply_forked, emitted_sql
|
deeply_forked, emitted_sql
|
||||||
):
|
):
|
||||||
"""The estimate counts depths, and a deleted action leaves a depth with no
|
"""The estimate counts depths, and a deleted action leaves a depth with
|
||||||
row behind it. The read has to notice it came up short and widen."""
|
no row behind it. The read must notice it came up short and widen."""
|
||||||
db, adventure = deeply_forked
|
db, adventure = deeply_forked
|
||||||
victim = (
|
victim = (
|
||||||
db.query(models.Action)
|
db.query(models.Action)
|
||||||
|
|||||||
@@ -1,14 +1,14 @@
|
|||||||
"""Phase 14 SP5 — continuing from a discarded attempt forks a branch.
|
"""Phase 14 SP5: continuing from a discarded attempt forks a branch.
|
||||||
|
|
||||||
SP4 made every attempt at a turn a node. While the attempts sit at the tip they
|
SP4 made every attempt at a turn a node. While the attempts sit at the
|
||||||
are leaves and cost nothing: switching between them just moves the `live` flag.
|
tip, they are leaves and cost nothing: switching between them just moves
|
||||||
The moment the player takes the story down one the line has already moved past,
|
the `live` flag. The moment the player continues from an attempt the line
|
||||||
the two futures have to coexist — and that is a branch.
|
has already moved past, the two futures must coexist. That is a branch.
|
||||||
|
|
||||||
What this file is really watching is the claim the whole design rests on: **a
|
This file tests the claim the whole design rests on: a fork inserts one
|
||||||
fork inserts one row and moves one row, whatever the story behind it is worth.**
|
row and moves one row, no matter how large the story behind it is.
|
||||||
Everything before the fork is borrowed, not copied, and the arithmetic that
|
Everything before the fork is borrowed, not copied, and the arithmetic
|
||||||
makes borrowing readable is `lineage`.
|
that makes borrowing possible lives in `lineage`.
|
||||||
|
|
||||||
python -m pytest tests/test_branch_forking.py -v
|
python -m pytest tests/test_branch_forking.py -v
|
||||||
"""
|
"""
|
||||||
@@ -32,8 +32,8 @@ from app.main import app
|
|||||||
from app.providers import PromptParts
|
from app.providers import PromptParts
|
||||||
from app.routers import adventures
|
from app.routers import adventures
|
||||||
|
|
||||||
# `hp` moves freely; `mana` has a cooldown of 2 turns, so a clock that advances
|
# `hp` moves freely. `mana` has a cooldown of 2 turns, so an incorrect
|
||||||
# when it should not shows up as a change the referee should have rejected.
|
# advance shows up as a change the referee should have rejected.
|
||||||
SCHEMA = {
|
SCHEMA = {
|
||||||
"player": {
|
"player": {
|
||||||
"hp": {"min": 0, "max": 100, "initial": 100},
|
"hp": {"min": 0, "max": 100, "initial": 100},
|
||||||
@@ -164,12 +164,12 @@ def _rows(adv_id) -> list[models.Action]:
|
|||||||
|
|
||||||
|
|
||||||
def _divergent_story(client):
|
def _divergent_story(client):
|
||||||
"""A story that retried turn 2, continued from the newer take, and left the
|
"""A story that retried turn 2, continued from the newer take, and left
|
||||||
older one behind as a leaf.
|
the older one behind as a leaf.
|
||||||
|
|
||||||
start · do · [attempt one | ATTEMPT TWO] · do · next turn
|
start > do > [attempt one | ATTEMPT TWO] > do > next turn
|
||||||
|
|
||||||
Returns the id of the attempt nobody built on.
|
Returns the id of the discarded attempt.
|
||||||
"""
|
"""
|
||||||
ScriptedProvider.replies = ["Attempt one.", "Attempt two.", "Next turn."]
|
ScriptedProvider.replies = ["Attempt one.", "Attempt two.", "Next turn."]
|
||||||
_play(client)
|
_play(client)
|
||||||
@@ -194,9 +194,10 @@ def test_a_fork_inserts_one_branch_row_and_copies_no_actions(client):
|
|||||||
forked = [b for b in branches if b["parent_branch_id"] is not None][0]
|
forked = [b for b in branches if b["parent_branch_id"] is not None][0]
|
||||||
assert forked["is_head"] is True
|
assert forked["is_head"] is True
|
||||||
assert forked["own_actions"] == 1, "the promoted attempt, and nothing else"
|
assert forked["own_actions"] == 1, "the promoted attempt, and nothing else"
|
||||||
# The fork point is the depth just before the attempt, stored rather than
|
# The fork point is the depth just before the attempt. The code stores
|
||||||
# inferred: inferring it from where two branches first differ would be a
|
# this value instead of inferring it from where two branches first
|
||||||
# guess, and a wrong one as soon as an attempt repeats its parent's text.
|
# differ, because that inference would guess wrong as soon as an
|
||||||
|
# attempt repeats its parent's text.
|
||||||
assert forked["fork_depth"] == forked["depth"] - 1
|
assert forked["fork_depth"] == forked["depth"] - 1
|
||||||
|
|
||||||
|
|
||||||
@@ -225,7 +226,7 @@ def test_both_branches_read_independently(client):
|
|||||||
assert _texts(client) == [
|
assert _texts(client) == [
|
||||||
"You enter a cave.", "> You look around.", "Attempt one.",
|
"You enter a cave.", "> You look around.", "Attempt one.",
|
||||||
]
|
]
|
||||||
# And the line it left is exactly as it was, turns after the fork included.
|
# The branch it left behind is unchanged, including turns after the fork.
|
||||||
r = client.post(f"/api/adventures/{client.adv_id}/branches/{parent}/switch")
|
r = client.post(f"/api/adventures/{client.adv_id}/branches/{parent}/switch")
|
||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
assert _texts(client) == [
|
assert _texts(client) == [
|
||||||
@@ -235,8 +236,9 @@ def test_both_branches_read_independently(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_parent_keeps_a_live_attempt_where_the_fork_left(client):
|
def test_the_parent_keeps_a_live_attempt_where_the_fork_left(client):
|
||||||
"""Promoting the loser must not leave the parent with a hole in its story:
|
"""Promoting the other attempt must not leave the parent with a gap in
|
||||||
a coordinate with no live node is a turn that disappears from the read."""
|
its story. A coordinate with no live node is a turn that disappears
|
||||||
|
from the read."""
|
||||||
discarded = _divergent_story(client)
|
discarded = _divergent_story(client)
|
||||||
_fork(client, discarded)
|
_fork(client, discarded)
|
||||||
|
|
||||||
@@ -251,8 +253,8 @@ def test_the_parent_keeps_a_live_attempt_where_the_fork_left(client):
|
|||||||
for (branch_id, depth), group in per_coordinate.items():
|
for (branch_id, depth), group in per_coordinate.items():
|
||||||
live = [a for a in group if a.live]
|
live = [a for a in group if a.live]
|
||||||
assert len(live) == 1, f"branch {branch_id} depth {depth}"
|
assert len(live) == 1, f"branch {branch_id} depth {depth}"
|
||||||
# The parent's turn 2 is now a single take, so the pager stops offering
|
# The parent's turn 2 is now a single take, so the pager no longer
|
||||||
# a page through attempts that have gone their own way.
|
# offers a page through attempts that diverged onto another branch.
|
||||||
parent_turn = per_coordinate[(parent_id, 2)]
|
parent_turn = per_coordinate[(parent_id, 2)]
|
||||||
assert len(parent_turn) == 1
|
assert len(parent_turn) == 1
|
||||||
assert parent_turn[0].variant_count == 0
|
assert parent_turn[0].variant_count == 0
|
||||||
@@ -261,9 +263,10 @@ def test_the_parent_keeps_a_live_attempt_where_the_fork_left(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_playing_on_a_fork_continues_that_branchs_depths(client):
|
def test_playing_on_a_fork_continues_that_branchs_depths(client):
|
||||||
"""A depth is a position along *this* story. Numbering the next node from
|
"""A depth is a position along this story. Numbering the next node from
|
||||||
the adventure-wide index would leave a hole where the other branch's turns
|
the adventure-wide index would leave a gap where the other branch's
|
||||||
are, which every windowing estimate then has to work around."""
|
turns are, and every windowing estimate would then have to work around
|
||||||
|
that gap."""
|
||||||
discarded = _divergent_story(client)
|
discarded = _divergent_story(client)
|
||||||
_fork(client, discarded)
|
_fork(client, discarded)
|
||||||
ScriptedProvider.replies = ["Onward."]
|
ScriptedProvider.replies = ["Onward."]
|
||||||
@@ -283,8 +286,8 @@ def test_playing_on_a_fork_continues_that_branchs_depths(client):
|
|||||||
# ------------------------------------------------------------- not a fork
|
# ------------------------------------------------------------- not a fork
|
||||||
|
|
||||||
def test_forking_at_the_tip_switches_without_making_a_branch(client):
|
def test_forking_at_the_tip_switches_without_making_a_branch(client):
|
||||||
"""Attempts nobody has built on stay leaves — that is what keeps the
|
"""Attempts nobody has built on stay leaves. This is what keeps the
|
||||||
lineage a list of divergences rather than of every retry ever."""
|
lineage a list of divergences instead of a list of every retry."""
|
||||||
ScriptedProvider.replies = ["Attempt one.", "Attempt two."]
|
ScriptedProvider.replies = ["Attempt one.", "Attempt two."]
|
||||||
_play(client)
|
_play(client)
|
||||||
_retry(client)
|
_retry(client)
|
||||||
@@ -297,8 +300,8 @@ def test_forking_at_the_tip_switches_without_making_a_branch(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_forking_the_attempt_already_in_the_story_does_nothing(client):
|
def test_forking_the_attempt_already_in_the_story_does_nothing(client):
|
||||||
"""Idempotent, because a client that has lost track of which take is live
|
"""This call is idempotent. A client that has lost track of which take
|
||||||
must not be able to fork a branch per click."""
|
is live must not create a new branch on every click."""
|
||||||
discarded = _divergent_story(client)
|
discarded = _divergent_story(client)
|
||||||
_fork(client, discarded)
|
_fork(client, discarded)
|
||||||
promoted = [a.id for a in _rows(client.adv_id) if a.type == "ai" and a.live
|
promoted = [a.id for a in _rows(client.adv_id) if a.type == "ai" and a.live
|
||||||
@@ -323,10 +326,11 @@ def test_forking_a_turn_that_is_already_the_story_is_a_no_op(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_forking_a_live_node_on_another_branch_is_refused(client):
|
def test_forking_a_live_node_on_another_branch_is_refused(client):
|
||||||
"""A live node off the path is another line's story, not an attempt going
|
"""A live node off the path belongs to another branch's story, not to a
|
||||||
spare — so the refusal names the tool that would actually do it. It used to
|
spare attempt on this one. The refusal names the tool that actually
|
||||||
answer "only one take", which was true of the group and no help at all: the
|
switches branches. It used to answer "only one take", which was true
|
||||||
caller does not want another take, it wants the branch this one is on."""
|
of the attempt group but useless here: the caller does not want
|
||||||
|
another take, it wants the branch this node is on."""
|
||||||
discarded = _divergent_story(client)
|
discarded = _divergent_story(client)
|
||||||
_fork(client, discarded)
|
_fork(client, discarded)
|
||||||
parent_id = [b for b in _branches(client) if b["parent_branch_id"] is None][0]["id"]
|
parent_id = [b for b in _branches(client) if b["parent_branch_id"] is None][0]["id"]
|
||||||
@@ -336,8 +340,8 @@ def test_forking_a_live_node_on_another_branch_is_refused(client):
|
|||||||
r = _fork(client, stranded)
|
r = _fork(client, stranded)
|
||||||
assert r.status_code == 400
|
assert r.status_code == 400
|
||||||
assert "another branch" in r.json()["detail"]
|
assert "another branch" in r.json()["detail"]
|
||||||
# And refusing left the tree alone — the bug this guards is a fork that
|
# Refusing must leave the tree alone. The bug this guards against is a
|
||||||
# promotes a sibling on the branch it was called against.
|
# fork that promotes a sibling on the branch it was called against.
|
||||||
assert len(_branches(client)) == 2
|
assert len(_branches(client)) == 2
|
||||||
|
|
||||||
|
|
||||||
@@ -366,10 +370,10 @@ def test_switching_restores_the_script_and_world_state(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_cooldown_clock_travels_with_the_branch(client):
|
def test_the_cooldown_clock_travels_with_the_branch(client):
|
||||||
"""The world-state clock is a depth, and depths repeat across branches — so
|
"""The world-state clock is a depth, and depths repeat across branches,
|
||||||
it can only be right if each branch carries its own. It does, for free: the
|
so it can only be correct if each branch carries its own. It does,
|
||||||
clock lives inside `_meta.last_changed`, which is part of the world state a
|
without extra work: the clock lives inside `_meta.last_changed`, which
|
||||||
switch restores."""
|
is part of the world state a switch restores."""
|
||||||
ScriptedProvider.replies = [
|
ScriptedProvider.replies = [
|
||||||
"Drained.\n```state\n{\"player.mana\": -10}\n```",
|
"Drained.\n```state\n{\"player.mana\": -10}\n```",
|
||||||
"Untouched.",
|
"Untouched.",
|
||||||
@@ -393,9 +397,9 @@ def test_the_cooldown_clock_travels_with_the_branch(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_retry_does_not_advance_the_cooldown_clock(client):
|
def test_a_retry_does_not_advance_the_cooldown_clock(client):
|
||||||
"""SP5's one carried-over open item. A retry re-runs the *same* turn, so
|
"""SP5's one carried-over open item. A retry re-runs the same turn, so
|
||||||
the clock the cooldown rules read must not move — it used to be the reused
|
the clock the cooldown rules read must not move. The reused `index`
|
||||||
`index` that guaranteed this, and it is the reused depth now."""
|
used to guarantee this; the reused depth guarantees it now."""
|
||||||
ScriptedProvider.replies = [
|
ScriptedProvider.replies = [
|
||||||
"Drained.\n```state\n{\"player.mana\": -10}\n```",
|
"Drained.\n```state\n{\"player.mana\": -10}\n```",
|
||||||
"Drained again.\n```state\n{\"player.mana\": -10}\n```",
|
"Drained again.\n```state\n{\"player.mana\": -10}\n```",
|
||||||
@@ -404,18 +408,19 @@ def test_a_retry_does_not_advance_the_cooldown_clock(client):
|
|||||||
first = _state(client.adv_id)[1]["_meta"]["last_changed"]["player.mana"]
|
first = _state(client.adv_id)[1]["_meta"]["last_changed"]["player.mana"]
|
||||||
_retry(client)
|
_retry(client)
|
||||||
assert _state(client.adv_id)[1]["_meta"]["last_changed"]["player.mana"] == first
|
assert _state(client.adv_id)[1]["_meta"]["last_changed"]["player.mana"] == first
|
||||||
# ...and the second attempt's drain landed, rather than being rejected for
|
# The second attempt's drain must land, instead of being rejected for a
|
||||||
# a cooldown it should never have been measured against.
|
# cooldown it was never actually subject to.
|
||||||
assert _state(client.adv_id)[1]["player"]["mana"] == 40
|
assert _state(client.adv_id)[1]["player"]["mana"] == 40
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------- derived work
|
# --------------------------------------------------------- derived work
|
||||||
|
|
||||||
def test_a_memory_on_the_line_left_behind_is_out_of_range_on_the_fork(client):
|
def test_a_memory_on_the_line_left_behind_is_out_of_range_on_the_fork(client):
|
||||||
"""Nothing is moved or withdrawn when a branch forks. The memory hangs off
|
"""Nothing is moved or removed when a branch forks. The memory attaches
|
||||||
the coordinate the *parent's* attempt still occupies, and the lineage caps
|
to the coordinate the parent's attempt still occupies, and the lineage
|
||||||
the parent one depth short of it — so the fork simply cannot see it, and
|
caps the parent one depth short of it. The fork therefore cannot see
|
||||||
resummarizes that ground from the text it actually tells."""
|
this memory, and resummarizes that span from the text it actually
|
||||||
|
contains."""
|
||||||
from app import tree
|
from app import tree
|
||||||
|
|
||||||
discarded = _divergent_story(client)
|
discarded = _divergent_story(client)
|
||||||
@@ -441,8 +446,8 @@ def test_a_memory_on_the_line_left_behind_is_out_of_range_on_the_fork(client):
|
|||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
adventure = db.get(models.Adventure, client.adv_id)
|
adventure = db.get(models.Adventure, client.adv_id)
|
||||||
# Still there, untouched — it describes the parent's story, which is
|
# The memory is still there, untouched, because it describes the
|
||||||
# unchanged.
|
# parent's story, which is unchanged.
|
||||||
assert [m.text for m in db.query(models.Memory).all()] == ["Attempt two happened."]
|
assert [m.text for m in db.query(models.Memory).all()] == ["Attempt two happened."]
|
||||||
path = lineage.path_of(db, adventure)
|
path = lineage.path_of(db, adventure)
|
||||||
visible = db.query(models.Memory).filter(
|
visible = db.query(models.Memory).filter(
|
||||||
@@ -450,8 +455,8 @@ def test_a_memory_on_the_line_left_behind_is_out_of_range_on_the_fork(client):
|
|||||||
path.clause(models.Memory),
|
path.clause(models.Memory),
|
||||||
).all()
|
).all()
|
||||||
assert visible == [], "a sibling's memory reached this branch"
|
assert visible == [], "a sibling's memory reached this branch"
|
||||||
# ...and the mark reads as one depth short of it, so the block is due
|
# The cursor reads one depth short of the memory, so this branch
|
||||||
# again on this branch rather than silently claimed as read.
|
# treats the block as due again instead of silently marking it read.
|
||||||
assert cursors.MEMORY.depth(db, adventure) == 1
|
assert cursors.MEMORY.depth(db, adventure) == 1
|
||||||
finally:
|
finally:
|
||||||
db.close()
|
db.close()
|
||||||
@@ -460,20 +465,22 @@ def test_a_memory_on_the_line_left_behind_is_out_of_range_on_the_fork(client):
|
|||||||
# ------------------------------------------------------------------- undo
|
# ------------------------------------------------------------------- undo
|
||||||
|
|
||||||
def test_undo_stops_at_the_fork(client):
|
def test_undo_stops_at_the_fork(client):
|
||||||
"""Taking back a turn on a fork must never reach across into the line it
|
"""Undoing a turn on a fork must never reach into the branch it forked
|
||||||
was forked from — those turns are that branch's story too."""
|
from. Those turns belong to that branch's story too."""
|
||||||
discarded = _divergent_story(client)
|
discarded = _divergent_story(client)
|
||||||
_fork(client, discarded)
|
_fork(client, discarded)
|
||||||
rows_before = len(_rows(client.adv_id))
|
rows_before = len(_rows(client.adv_id))
|
||||||
|
|
||||||
r = client.post(f"/api/adventures/{client.adv_id}/undo")
|
r = client.post(f"/api/adventures/{client.adv_id}/undo")
|
||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
# The promoted attempt goes, and the player action in front of it stays:
|
# The promoted attempt is removed, and the player action before it
|
||||||
# it is on the parent, and the parent still tells it.
|
# stays, because that action belongs to the parent and the parent
|
||||||
|
# still has it.
|
||||||
assert len(_rows(client.adv_id)) == rows_before - 1
|
assert len(_rows(client.adv_id)) == rows_before - 1
|
||||||
assert _texts(client) == ["You enter a cave.", "> You look around."]
|
assert _texts(client) == ["You enter a cave.", "> You look around."]
|
||||||
|
|
||||||
# Nothing left of this branch's own: refuse rather than eat the parent's.
|
# Nothing is left of this branch's own turns, so undo must refuse
|
||||||
|
# instead of removing the parent's turns.
|
||||||
r = client.post(f"/api/adventures/{client.adv_id}/undo")
|
r = client.post(f"/api/adventures/{client.adv_id}/undo")
|
||||||
assert r.status_code == 400
|
assert r.status_code == 400
|
||||||
assert "forked from" in r.json()["detail"]
|
assert "forked from" in r.json()["detail"]
|
||||||
@@ -497,10 +504,11 @@ def test_the_branch_list_is_the_tree(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_fork_agrees_with_the_lineage_computed_by_hand(client):
|
def test_fork_agrees_with_the_lineage_computed_by_hand(client):
|
||||||
"""`test_branch_clause.make_branch` has computed a fork's lineage since SP2,
|
"""`test_branch_clause.make_branch` has computed a fork's lineage by
|
||||||
by hand, precisely so the fixture could not pass by agreeing with a bug in
|
hand since SP2, precisely so the fixture could not pass by repeating a
|
||||||
the code under test. SP5 is when that code exists — so check the two
|
bug in the code under test. SP5 introduces that code, so this test
|
||||||
against each other rather than letting them drift apart."""
|
checks the two against each other instead of letting them drift
|
||||||
|
apart."""
|
||||||
from tests.test_branch_clause import make_branch
|
from tests.test_branch_clause import make_branch
|
||||||
|
|
||||||
discarded = _divergent_story(client)
|
discarded = _divergent_story(client)
|
||||||
@@ -512,8 +520,8 @@ def test_fork_agrees_with_the_lineage_computed_by_hand(client):
|
|||||||
real = lineage.branch_of(db, adventure)
|
real = lineage.branch_of(db, adventure)
|
||||||
parent = db.get(models.Branch, real.parent_branch_id)
|
parent = db.get(models.Branch, real.parent_branch_id)
|
||||||
by_hand = make_branch(db, adventure, parent=parent, fork_depth=real.fork_depth)
|
by_hand = make_branch(db, adventure, parent=parent, fork_depth=real.fork_depth)
|
||||||
# Same shape, its own id: compare the ancestry, which is the part that
|
# The two branches share a shape but not an id, so compare only the
|
||||||
# is arithmetic rather than allocation.
|
# ancestry, which is the arithmetic part rather than the allocated id.
|
||||||
assert lineage.entries_of(real)[1:] == lineage.entries_of(by_hand)[1:]
|
assert lineage.entries_of(real)[1:] == lineage.entries_of(by_hand)[1:]
|
||||||
assert lineage.entries_of(real)[0] == (real.id, None)
|
assert lineage.entries_of(real)[0] == (real.id, None)
|
||||||
finally:
|
finally:
|
||||||
@@ -521,9 +529,9 @@ def test_fork_agrees_with_the_lineage_computed_by_hand(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_deep_fork_chain_reads_for_what_one_branch_costs(client):
|
def test_a_deep_fork_chain_reads_for_what_one_branch_costs(client):
|
||||||
"""Clause count is bounded by the window, not by fork count — the property
|
"""Clause count is bounded by the window, not by fork count. This is
|
||||||
the whole lineage cache exists for, now measured through real forks rather
|
the property the whole lineage cache exists for, now measured through
|
||||||
than hand-built rows."""
|
real forks instead of hand-built rows."""
|
||||||
from tools import dbmeter
|
from tools import dbmeter
|
||||||
|
|
||||||
ScriptedProvider.replies = ["First take.", "Second take.", "Onward."]
|
ScriptedProvider.replies = ["First take.", "Second take.", "Onward."]
|
||||||
@@ -555,9 +563,9 @@ def test_a_deep_fork_chain_reads_for_what_one_branch_costs(client):
|
|||||||
try:
|
try:
|
||||||
adventure = db.get(models.Adventure, client.adv_id)
|
adventure = db.get(models.Adventure, client.adv_id)
|
||||||
entries = lineage.entries_of(lineage.branch_of(db, adventure))
|
entries = lineage.entries_of(lineage.branch_of(db, adventure))
|
||||||
# The whole ancestry is there to be named...
|
# The whole ancestry is available to be named.
|
||||||
assert len(entries) == len(branches)
|
assert len(entries) == len(branches)
|
||||||
# ...and the windowed read names as few of them as the window needs.
|
# The windowed read names as few of them as the window needs.
|
||||||
path = lineage.path_of(db, adventure)
|
path = lineage.path_of(db, adventure)
|
||||||
assert path.prefix_covering(60) <= len(entries)
|
assert path.prefix_covering(60) <= len(entries)
|
||||||
finally:
|
finally:
|
||||||
@@ -565,8 +573,8 @@ def test_a_deep_fork_chain_reads_for_what_one_branch_costs(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_switching_to_a_branch_of_another_adventure_is_a_404(client):
|
def test_switching_to_a_branch_of_another_adventure_is_a_404(client):
|
||||||
"""A branch id names one adventure, so the two ids in the URL have to
|
"""A branch id names one adventure, so the two ids in the URL must
|
||||||
agree — otherwise a guessed number reads somebody else's story."""
|
agree. Otherwise a guessed number reads somebody else's story."""
|
||||||
other = client.post("/api/adventures", json={"title": "Elsewhere"}).json()["id"]
|
other = client.post("/api/adventures", json={"title": "Elsewhere"}).json()["id"]
|
||||||
ScriptedProvider.replies = ["Elsewhere."]
|
ScriptedProvider.replies = ["Elsewhere."]
|
||||||
r = client.post(f"/api/adventures/{other}/actions", json={"type": "do", "text": "wait"})
|
r = client.post(f"/api/adventures/{other}/actions", json={"type": "do", "text": "wait"})
|
||||||
|
|||||||
@@ -1,21 +1,21 @@
|
|||||||
"""Phase 14 SP7 — naming a branch, and throwing one away.
|
"""Phase 14 SP7: renaming a branch and deleting one.
|
||||||
|
|
||||||
SP5 gave the tree a fork and a switch. Neither of them ever removes anything,
|
SP5 gave the tree a fork and a switch. Neither operation removes anything,
|
||||||
and nothing in the design prunes a tree on its own, so an adventure that is
|
and nothing in the design prunes a tree automatically, so an adventure that
|
||||||
retried and forked enough grows without a ceiling. Delete is what stands
|
is retried and forked enough grows without a limit. Delete is what limits
|
||||||
between the tree and that, which is why it ships with the view that first makes
|
that growth. It ships with the same view that first makes a fork reachable,
|
||||||
a fork reachable rather than in some later subphase.
|
rather than in a later subphase.
|
||||||
|
|
||||||
Two rules carry most of this file:
|
Two rules carry most of this file:
|
||||||
|
|
||||||
* **A name is chosen, so it is stored; a label is derived, so it is not.** An
|
* A name is chosen, so the database stores it. A label is derived, so the
|
||||||
unnamed branch keeps NULL and the client draws it from its fork depth. A
|
database does not. An unnamed branch keeps NULL, and the client draws the
|
||||||
generated "branch 4" in the column would be a lie the moment branch 3 is
|
label from its fork depth. A generated label such as "branch 4" stored in
|
||||||
deleted.
|
the column would become wrong the moment branch 3 is deleted.
|
||||||
* **The delete may never take the ground under the reader.** Refusing the head
|
* Delete must never remove a branch the reader depends on. Refusing to
|
||||||
is the obvious half; refusing an *ancestor* of the head is the same mistake
|
delete the head is the obvious case. Refusing to delete an ancestor of the
|
||||||
wearing a disguise, and it is the one that would leave `head_branch_id`
|
head is the same rule applied one level up: deleting it would leave
|
||||||
pointing at a row the cascade removed.
|
`head_branch_id` pointing at a row the cascade removed.
|
||||||
|
|
||||||
python -m pytest tests/test_branch_management.py -v
|
python -m pytest tests/test_branch_management.py -v
|
||||||
"""
|
"""
|
||||||
@@ -136,7 +136,7 @@ def _texts(client) -> list[str]:
|
|||||||
|
|
||||||
|
|
||||||
def _discarded_on(adv_id, branch_id=None) -> int:
|
def _discarded_on(adv_id, branch_id=None) -> int:
|
||||||
"""An AI attempt nobody built on, optionally restricted to one branch."""
|
"""An AI attempt with no turn built on it, optionally restricted to one branch."""
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
q = db.query(models.Action).filter(
|
q = db.query(models.Action).filter(
|
||||||
@@ -152,7 +152,7 @@ def _discarded_on(adv_id, branch_id=None) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def _forked(client):
|
def _forked(client):
|
||||||
"""A story with one fork. Returns (root id, forked id); the fork is head.
|
"""A story with one fork. Returns (root id, forked id). The fork is head.
|
||||||
|
|
||||||
start · do · [attempt one | ATTEMPT TWO] · do · next turn
|
start · do · [attempt one | ATTEMPT TWO] · do · next turn
|
||||||
└── forked here
|
└── forked here
|
||||||
@@ -184,10 +184,10 @@ def _counts(adv_id, branch_ids):
|
|||||||
# ------------------------------------------------------------------ naming
|
# ------------------------------------------------------------------ naming
|
||||||
|
|
||||||
def test_a_branch_starts_unnamed(client):
|
def test_a_branch_starts_unnamed(client):
|
||||||
"""NULL, not a generated label — the client draws one from the fork depth.
|
"""NULL, not a generated label. The client draws a label from the fork depth.
|
||||||
|
|
||||||
A name written here would go stale the moment a branch before it is
|
A name stored here would go stale the moment a branch before it is
|
||||||
deleted and the ordinals shift under it.
|
deleted and the ordinals shift.
|
||||||
"""
|
"""
|
||||||
root, forked = _forked(client)
|
root, forked = _forked(client)
|
||||||
assert [b["name"] for b in _branches(client)] == [None, None]
|
assert [b["name"] for b in _branches(client)] == [None, None]
|
||||||
@@ -206,9 +206,8 @@ def test_a_name_is_stored_and_read_back(client):
|
|||||||
def test_a_blank_name_goes_back_to_unnamed(client):
|
def test_a_blank_name_goes_back_to_unnamed(client):
|
||||||
"""A name of spaces is not a name anyone chose.
|
"""A name of spaces is not a name anyone chose.
|
||||||
|
|
||||||
Storing one would give the client an empty label to draw where it would
|
Storing it would give the client an empty label instead of falling back
|
||||||
otherwise fall back to the fork depth — a branch that looks nameless and
|
to the fork depth. The branch would look nameless and appear broken.
|
||||||
reads as broken.
|
|
||||||
"""
|
"""
|
||||||
root, forked = _forked(client)
|
root, forked = _forked(client)
|
||||||
_rename(client, forked, "briefly named")
|
_rename(client, forked, "briefly named")
|
||||||
@@ -226,9 +225,10 @@ def test_a_name_longer_than_the_column_is_refused(client):
|
|||||||
def test_a_rename_hands_back_the_row_the_listing_would_give(client):
|
def test_a_rename_hands_back_the_row_the_listing_would_give(client):
|
||||||
"""Renaming a branch does not change how many turns are on it.
|
"""Renaming a branch does not change how many turns are on it.
|
||||||
|
|
||||||
`own_actions` was hard-coded to 0 in this response, which only stayed
|
`own_actions` was hard-coded to 0 in this response. That bug stayed
|
||||||
invisible because the panel throws the body away and refetches. Anything
|
invisible because the panel discards the response body and refetches.
|
||||||
that trusted the reply would draw a branch that had just lost its turns.
|
Anything that trusted the reply would show a branch that had just lost
|
||||||
|
its turns.
|
||||||
"""
|
"""
|
||||||
root, forked = _forked(client)
|
root, forked = _forked(client)
|
||||||
listed = {b["id"]: b for b in _branches(client)}
|
listed = {b["id"]: b for b in _branches(client)}
|
||||||
@@ -278,10 +278,11 @@ def test_the_branch_being_read_cannot_be_deleted(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_an_ancestor_of_the_branch_being_read_cannot_be_deleted(client):
|
def test_an_ancestor_of_the_branch_being_read_cannot_be_deleted(client):
|
||||||
"""The same mistake as deleting the head, wearing a disguise.
|
"""Deleting an ancestor of the head is the same mistake as deleting the head.
|
||||||
|
|
||||||
`parent_branch_id` cascades, so deleting a branch the head was forked from
|
`parent_branch_id` cascades, so deleting a branch the head was forked
|
||||||
would take the head with it and leave `head_branch_id` pointing at nothing.
|
from would delete the head too, and leave `head_branch_id` pointing at
|
||||||
|
nothing.
|
||||||
"""
|
"""
|
||||||
root, forked = _forked(client)
|
root, forked = _forked(client)
|
||||||
# A fork of the fork, so `forked` is an ancestor of the head rather than
|
# A fork of the fork, so `forked` is an ancestor of the head rather than
|
||||||
@@ -310,7 +311,8 @@ def test_deleting_a_branch_leaves_the_line_it_forked_from_untouched(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_deleting_a_branch_takes_its_nodes_and_its_descendants(client):
|
def test_deleting_a_branch_takes_its_nodes_and_its_descendants(client):
|
||||||
"""One statement, however deep the subtree — the cascade does the walking."""
|
"""One statement deletes the whole subtree, regardless of depth. The
|
||||||
|
cascade performs the traversal."""
|
||||||
root, forked = _forked(client)
|
root, forked = _forked(client)
|
||||||
_retry(client)
|
_retry(client)
|
||||||
_play(client, "press on")
|
_play(client, "press on")
|
||||||
@@ -333,10 +335,11 @@ def test_deleting_a_branch_takes_its_nodes_and_its_descendants(client):
|
|||||||
def test_deleting_a_branch_clears_a_cursor_that_stood_on_it(client):
|
def test_deleting_a_branch_clears_a_cursor_that_stood_on_it(client):
|
||||||
"""Harmless on Postgres, a real bug on SQLite.
|
"""Harmless on Postgres, a real bug on SQLite.
|
||||||
|
|
||||||
Postgres never reuses a branch id, so a stale anchor simply never resolves.
|
Postgres never reuses a branch id, so a stale anchor simply never
|
||||||
SQLite hands the freed id to the next fork, at which point the anchor
|
resolves. SQLite assigns the freed id to the next fork. The anchor then
|
||||||
resolves onto a branch it has never seen and reports a stretch of story as
|
resolves onto a branch it never saw, and reports a stretch of story as
|
||||||
already summarized — which loses it from the memories for good.
|
already summarized. That stretch is then permanently excluded from the
|
||||||
|
memories.
|
||||||
"""
|
"""
|
||||||
root, forked = _forked(client)
|
root, forked = _forked(client)
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
@@ -355,7 +358,7 @@ def test_deleting_a_branch_clears_a_cursor_that_stood_on_it(client):
|
|||||||
try:
|
try:
|
||||||
adventure = db.get(models.Adventure, client.adv_id)
|
adventure = db.get(models.Adventure, client.adv_id)
|
||||||
assert cursors.MEMORY.stored(adventure) == (None, cursors.NO_DEPTH)
|
assert cursors.MEMORY.stored(adventure) == (None, cursors.NO_DEPTH)
|
||||||
# The one standing on ground that survived is left exactly where it was.
|
# The cursor on a branch that still exists is left exactly where it was.
|
||||||
assert cursors.SUMMARY.stored(adventure) == (root, 1)
|
assert cursors.SUMMARY.stored(adventure) == (root, 1)
|
||||||
finally:
|
finally:
|
||||||
db.close()
|
db.close()
|
||||||
@@ -398,11 +401,11 @@ def test_a_hand_written_memory_is_anchored_where_it_was_written(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_drawer_shows_the_path_being_read_and_nothing_else(client):
|
def test_the_drawer_shows_the_path_being_read_and_nothing_else(client):
|
||||||
"""The bank you can see is the bank the model can see.
|
"""The memories a reader can see match the memories the model can retrieve.
|
||||||
|
|
||||||
An adventure-wide list would show memories from branches this story never
|
An adventure-wide list would include memories from branches this story
|
||||||
went down — which are never retrieved — and a reader cannot tell those from
|
never took. Those memories are never retrieved, and a reader could not
|
||||||
the ones actually in play.
|
distinguish them from the ones actually in play.
|
||||||
"""
|
"""
|
||||||
root, forked = _forked(client)
|
root, forked = _forked(client)
|
||||||
on_the_fork = _add_memory(client, "Took the other door.")
|
on_the_fork = _add_memory(client, "Took the other door.")
|
||||||
@@ -412,9 +415,10 @@ def test_the_drawer_shows_the_path_being_read_and_nothing_else(client):
|
|||||||
assert {m["id"] for m in _memories(client)} == {on_the_root}, \
|
assert {m["id"] for m in _memories(client)} == {on_the_root}, \
|
||||||
"the fork's memory is not on this story"
|
"the fork's memory is not on this story"
|
||||||
|
|
||||||
# Switching to the fork shows its own memory — and the root's, because a
|
# Switching to the fork shows its own memory and the root's. A fork
|
||||||
# fork borrows its ancestors up to the point it left them. The relationship
|
# borrows its ancestors up to the point where it diverged from them.
|
||||||
# is asymmetric on purpose; the parent never went down the fork.
|
# The relationship is asymmetric on purpose: the parent never took the
|
||||||
|
# fork's path.
|
||||||
_switch(client, forked)
|
_switch(client, forked)
|
||||||
listed = {m["id"] for m in _memories(client)}
|
listed = {m["id"] for m in _memories(client)}
|
||||||
assert on_the_fork in listed
|
assert on_the_fork in listed
|
||||||
@@ -422,8 +426,9 @@ def test_the_drawer_shows_the_path_being_read_and_nothing_else(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_drawer_and_retrieval_agree_on_what_is_visible(client):
|
def test_the_drawer_and_retrieval_agree_on_what_is_visible(client):
|
||||||
"""One predicate, so a memory can never be listed but unretrievable (or the
|
"""One predicate decides visibility, so a memory can never be listed but
|
||||||
reverse). Two spellings of "on this path" would eventually drift."""
|
unretrievable, or the reverse. Two separate definitions of "on this
|
||||||
|
path" would eventually diverge."""
|
||||||
root, forked = _forked(client)
|
root, forked = _forked(client)
|
||||||
_add_memory(client, "Took the other door.")
|
_add_memory(client, "Took the other door.")
|
||||||
_switch(client, root)
|
_switch(client, root)
|
||||||
@@ -447,10 +452,11 @@ def test_the_drawer_and_retrieval_agree_on_what_is_visible(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_deleting_a_branch_deletes_the_memories_written_on_it(client):
|
def test_deleting_a_branch_deletes_the_memories_written_on_it(client):
|
||||||
"""Not merely out of view — the row goes with the branch, through the
|
"""Deleting a branch removes its memories from the database, not just
|
||||||
cascade. That is what keeps "the drawer shows only your path" from
|
from view: the cascade deletes the row along with the branch. This
|
||||||
stranding anything: a memory you cannot see is on a branch you can still
|
keeps a hidden memory from becoming unreachable in a different way: a
|
||||||
switch to, and deleting that branch takes it for good."""
|
memory you cannot currently see is on a branch you can still switch to,
|
||||||
|
and deleting that branch deletes the memory permanently."""
|
||||||
root, forked = _forked(client)
|
root, forked = _forked(client)
|
||||||
doomed = _add_memory(client, "Took the other door.")
|
doomed = _add_memory(client, "Took the other door.")
|
||||||
_switch(client, root)
|
_switch(client, root)
|
||||||
@@ -473,10 +479,12 @@ def test_deleting_a_branch_deletes_the_memories_written_on_it(client):
|
|||||||
# ------------------------------------------------------------------- backup
|
# ------------------------------------------------------------------- backup
|
||||||
|
|
||||||
def test_a_bundle_carries_the_name_a_player_chose(client):
|
def test_a_bundle_carries_the_name_a_player_chose(client):
|
||||||
"""A name is a decision, so it travels — the rule the v2 format is built on.
|
"""A name is a decision, so the export includes it. This is the rule the
|
||||||
|
v2 format is built on.
|
||||||
|
|
||||||
`lineage` and the head depth stay out because they are computed from what
|
`lineage` and the head depth stay out of the export because the
|
||||||
the file already carries; a name is computed from nothing.
|
importer can compute them from what the file already carries. A name
|
||||||
|
is computed from nothing, so the export must carry it.
|
||||||
"""
|
"""
|
||||||
root, forked = _forked(client)
|
root, forked = _forked(client)
|
||||||
_rename(client, root, "the long way")
|
_rename(client, root, "the long way")
|
||||||
|
|||||||
@@ -1,24 +1,25 @@
|
|||||||
"""Phase 14 SP6 — the export bundle carries the tree.
|
"""Phase 14 SP6: the export bundle carries the tree.
|
||||||
|
|
||||||
A bundle is the only part of this phase a migration can never reach: the file
|
A bundle is the only part of this phase a migration can never reach: the
|
||||||
is already on somebody's disk. So there are two formats, and the two halves of
|
file already exists on somebody's disk. So there are two formats, and the
|
||||||
this file watch different things.
|
two halves of this file check different properties.
|
||||||
|
|
||||||
**v2 has to be lossless for a story that forked**, which v1 could not be — it
|
v2 must be lossless for a story that forked. v1 could not be, because it
|
||||||
had one list and there were two stories, so it interleaved them by `index` and
|
stored one list for two stories, interleaved by `index`, which read back
|
||||||
read as a mangled story. Losslessness here means the *tree*: every branch, the
|
as a mangled story. Losslessness here means the tree: every branch, the
|
||||||
fork point it left its parent at, which attempt at each turn is the story, and
|
fork point it left on its parent, which attempt at each turn is the
|
||||||
what each node left behind — because that last one is what a branch switch puts
|
story, and what each node left behind. That last item is what a branch
|
||||||
back, and a tree nobody can switch inside is not the tree that was exported.
|
switch restores, and a tree nobody can switch inside is not the tree
|
||||||
|
that was exported.
|
||||||
|
|
||||||
**v1 has to still import**, because a backup that stops importing is not a
|
v1 must still import, because a backup that stops importing is not a
|
||||||
backup.
|
backup.
|
||||||
|
|
||||||
Underneath both is the rule the module is built on: a bundle carries what was
|
Both formats follow one rule: a bundle carries what was chosen and never
|
||||||
*chosen* and never what is *derived*. The lineage, the head depth, the legacy
|
what is derived. The lineage, the head depth, the legacy `index`, and the
|
||||||
`index` and the variant ordinals are all rebuilt on the way in, so a
|
variant ordinals are all rebuilt on the way in, so a hand-edited file
|
||||||
hand-edited file cannot disagree with itself — and the tests that matter most
|
cannot disagree with itself. The tests that matter most here hand the
|
||||||
here are the ones that hand it a file which does.
|
importer a file that does disagree with itself.
|
||||||
|
|
||||||
python -m pytest tests/test_bundle_v2.py -v
|
python -m pytest tests/test_bundle_v2.py -v
|
||||||
"""
|
"""
|
||||||
@@ -44,8 +45,8 @@ from app.routers import adventures
|
|||||||
|
|
||||||
SCHEMA = {"player": {"hp": {"min": 0, "max": 100, "initial": 100}}}
|
SCHEMA = {"player": {"hp": {"min": 0, "max": 100, "initial": 100}}}
|
||||||
|
|
||||||
# Ten gold a turn, so the script scoreboard is a number that says how many turns
|
# Ten gold a turn, so the stored gold total tells how many turns the
|
||||||
# the story behind it has — which makes an after-snapshot visible from outside.
|
# story behind it played. This makes an after-snapshot visible from outside.
|
||||||
GOLD_SCRIPT = """
|
GOLD_SCRIPT = """
|
||||||
const modifier = (text) => {
|
const modifier = (text) => {
|
||||||
state.gold = (state.gold || 0) + 10;
|
state.gold = (state.gold || 0) + 10;
|
||||||
@@ -161,7 +162,7 @@ def _texts(client, adv_id) -> list[str]:
|
|||||||
|
|
||||||
|
|
||||||
def _every_branch_story(client, adv_id) -> list[list[str]]:
|
def _every_branch_story(client, adv_id) -> list[list[str]]:
|
||||||
"""What each branch tells, in branch order — the whole tree as text."""
|
"""What each branch tells, in branch order. This is the whole tree as text."""
|
||||||
stories = []
|
stories = []
|
||||||
for branch in _branches(client, adv_id):
|
for branch in _branches(client, adv_id):
|
||||||
_switch(client, adv_id, branch["id"])
|
_switch(client, adv_id, branch["id"])
|
||||||
@@ -213,13 +214,14 @@ def _script_state(adv_id) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
def _forked_story(client) -> int:
|
def _forked_story(client) -> int:
|
||||||
"""A story that went two ways, and stayed both.
|
"""A story that went two ways and stayed both.
|
||||||
|
|
||||||
root: start · do · [attempt two] · do · next turn
|
root: start > do > [attempt two] > do > next turn
|
||||||
fork: [ATTEMPT ONE] · do · elsewhere
|
fork: [ATTEMPT ONE] > do > elsewhere
|
||||||
|
|
||||||
The *discarded* attempt is the one that gets promoted, because a fork moves
|
The discarded attempt is the one this function promotes, because a
|
||||||
the take you are leaving for and leaves the line you came from untouched.
|
fork moves the attempt being left for and leaves the line it came
|
||||||
|
from untouched.
|
||||||
|
|
||||||
Returns the adventure id, with the head on the fork.
|
Returns the adventure id, with the head on the fork.
|
||||||
"""
|
"""
|
||||||
@@ -237,11 +239,11 @@ def _forked_story(client) -> int:
|
|||||||
# ------------------------------------------------------- the round trip (v2)
|
# ------------------------------------------------------- the round trip (v2)
|
||||||
|
|
||||||
def test_a_forked_story_survives_the_round_trip(client):
|
def test_a_forked_story_survives_the_round_trip(client):
|
||||||
"""The headline: both futures come back, and both are still readable.
|
"""The main claim: both futures come back, and both are still readable.
|
||||||
|
|
||||||
This is the thing v1 could not do. The check is not "the same rows" — the
|
This is the thing v1 could not do. The check is not "the same rows",
|
||||||
ids are new — but "the same stories", read the way a player reads them: by
|
since the ids are new, but "the same stories", read the way a player
|
||||||
switching to a branch and looking at what it says.
|
reads them: by switching to a branch and looking at what it says.
|
||||||
"""
|
"""
|
||||||
original = _forked_story(client)
|
original = _forked_story(client)
|
||||||
before = _every_branch_story(client, original)
|
before = _every_branch_story(client, original)
|
||||||
@@ -254,11 +256,11 @@ def test_a_forked_story_survives_the_round_trip(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_fork_point_comes_back_where_it_was_put(client):
|
def test_the_fork_point_comes_back_where_it_was_put(client):
|
||||||
"""`fork_depth` is stored, never inferred — including through a file.
|
"""`fork_depth` is stored, never inferred, including through a file.
|
||||||
|
|
||||||
Inferring it from where two branches' nodes first differ would be a guess
|
Inferring it from where two branches' nodes first differ would guess
|
||||||
about how the story was played, and a wrong one the moment an attempt
|
at how the story was played, and that guess fails as soon as an
|
||||||
happens to repeat its parent's text.
|
attempt happens to repeat its parent's text.
|
||||||
"""
|
"""
|
||||||
original = _forked_story(client)
|
original = _forked_story(client)
|
||||||
before = [(b["parent_branch_id"] is None, b["fork_depth"]) for b in _branches(client, original)]
|
before = [(b["parent_branch_id"] is None, b["fork_depth"]) for b in _branches(client, original)]
|
||||||
@@ -275,22 +277,23 @@ def test_the_head_comes_back_on_the_branch_it_was_left_on(client):
|
|||||||
|
|
||||||
copy = _imported(client, _export(client, original))
|
copy = _imported(client, _export(client, original))
|
||||||
assert [b["is_head"] for b in _branches(client, copy)] == head_before
|
assert [b["is_head"] for b in _branches(client, copy)] == head_before
|
||||||
# And the tip it sits at is derived from the nodes that arrived, not read
|
# The tip it sits at is derived from the nodes that arrived, not
|
||||||
# out of the file — the bundle never says how deep a branch goes.
|
# read from the file. The bundle never states how deep a branch goes.
|
||||||
assert _texts(client, copy) == _texts(client, original)
|
assert _texts(client, copy) == _texts(client, original)
|
||||||
|
|
||||||
|
|
||||||
def test_a_switch_in_the_copy_restores_what_that_branch_left_behind(client):
|
def test_a_switch_in_the_copy_restores_what_that_branch_left_behind(client):
|
||||||
"""The after-snapshots are why the bundle carries them.
|
"""This test justifies why the bundle carries after-snapshots.
|
||||||
|
|
||||||
The gold script adds ten a turn, so the scoreboard is a count of the story
|
The gold script adds ten a turn, so the stored gold total counts the
|
||||||
behind it. A bundle that carried the actions but not the outcomes would
|
turns behind it. A bundle that carried the actions but not the
|
||||||
import a tree that reads correctly and switches wrong.
|
outcomes would import a tree that reads correctly but switches to the
|
||||||
|
wrong state.
|
||||||
"""
|
"""
|
||||||
original = _forked_story(client)
|
original = _forked_story(client)
|
||||||
# One more turn on the fork, so the two tips are genuinely different
|
# Play one more turn on the fork, so the two tips end up at
|
||||||
# numbers: played turn for turn, the branches earn the same gold and a
|
# genuinely different totals. Turn for turn, both branches earn the
|
||||||
# switch that restored nothing at all would still look right.
|
# same gold, so a switch that restored nothing would still look right.
|
||||||
ScriptedProvider.replies = ["Further still."]
|
ScriptedProvider.replies = ["Further still."]
|
||||||
_play(client, original, "press on")
|
_play(client, original, "press on")
|
||||||
|
|
||||||
@@ -340,12 +343,12 @@ def test_a_memory_comes_back_on_the_node_it_hangs_off(client):
|
|||||||
# --------------------------------------------------- what is not in the file
|
# --------------------------------------------------- what is not in the file
|
||||||
|
|
||||||
def test_the_lineage_is_rebuilt_rather_than_carried(client):
|
def test_the_lineage_is_rebuilt_rather_than_carried(client):
|
||||||
"""A cache of `parent` + `fork_depth` is not a second thing to ship.
|
"""A cache of `parent` plus `fork_depth` is not a second thing to ship.
|
||||||
|
|
||||||
The file says where each branch forked; the ancestry that makes the fork
|
The file states where each branch forked. The ancestry that makes the
|
||||||
readable is computed from that on the way in, capped at the fork exactly as
|
fork readable is computed from that value on the way in, capped at
|
||||||
`tree.fork` caps it. Shipping the cache too would put two sources of truth
|
the fork exactly as `tree.fork` caps it. Shipping the cache too would
|
||||||
for one fact in a file anybody can hand-edit.
|
put two sources of truth for one fact in a file anybody can hand-edit.
|
||||||
"""
|
"""
|
||||||
original = _forked_story(client)
|
original = _forked_story(client)
|
||||||
exported = _export(client, original)
|
exported = _export(client, original)
|
||||||
@@ -354,18 +357,19 @@ def test_the_lineage_is_rebuilt_rather_than_carried(client):
|
|||||||
root, forked = _branch_rows(_imported(client, exported))
|
root, forked = _branch_rows(_imported(client, exported))
|
||||||
assert root.lineage == [[root.id, None]]
|
assert root.lineage == [[root.id, None]]
|
||||||
assert forked.lineage == [[forked.id, None], [root.id, forked.fork_depth]]
|
assert forked.lineage == [[forked.id, None], [root.id, forked.fork_depth]]
|
||||||
# Which is the arithmetic the reader depends on: the parent is capped one
|
# This is the arithmetic the reader depends on. The parent is capped
|
||||||
# depth short of the attempt that was promoted, so the fork cannot see it.
|
# one depth short of the attempt that was promoted, so the fork
|
||||||
|
# cannot see it.
|
||||||
assert lineage.entries_of(forked) == [(forked.id, None), (root.id, forked.fork_depth)]
|
assert lineage.entries_of(forked) == [(forked.id, None), (root.id, forked.fork_depth)]
|
||||||
|
|
||||||
|
|
||||||
def test_the_legacy_index_is_reissued_so_two_branches_never_share_one(client):
|
def test_the_legacy_index_is_reissued_so_two_branches_never_share_one(client):
|
||||||
"""`index` is a fact about the adventure, and `depth` is one about a path.
|
"""`index` describes the adventure. `depth` describes a path.
|
||||||
|
|
||||||
Two branches have a node at depth 3, so depth cannot be the number
|
Two branches can each have a node at depth 3, so depth cannot be the
|
||||||
`max_action_index` hands out next. The import allocates one per turn
|
number `max_action_index` hands out next. The import allocates one
|
||||||
instead: siblings share it, the way SP4 leaves them, and no two coordinates
|
index per turn instead. Siblings share an index, the way SP4 leaves
|
||||||
do.
|
them, and no two coordinates share one.
|
||||||
"""
|
"""
|
||||||
copy = _imported(client, _export(client, _forked_story(client)))
|
copy = _imported(client, _export(client, _forked_story(client)))
|
||||||
rows = _rows(copy)
|
rows = _rows(copy)
|
||||||
@@ -376,8 +380,9 @@ def test_the_legacy_index_is_reissued_so_two_branches_never_share_one(client):
|
|||||||
"one index per turn, whatever branch it is on"
|
"one index per turn, whatever branch it is on"
|
||||||
assert len(by_index) == len({(r.branch_id, r.depth) for r in rows})
|
assert len(by_index) == len({(r.branch_id, r.depth) for r in rows})
|
||||||
|
|
||||||
# The case that makes the rule necessary: the fork and the line it left
|
# This is the case that makes the rule necessary: the fork and the
|
||||||
# both hold a turn at depth 2, and they are not the same turn.
|
# branch it left both hold a turn at depth 2, and they are not the
|
||||||
|
# same turn.
|
||||||
at_depth_2 = [r for r in rows if r.depth == 2]
|
at_depth_2 = [r for r in rows if r.depth == 2]
|
||||||
assert len({r.branch_id for r in at_depth_2}) == 2
|
assert len({r.branch_id for r in at_depth_2}) == 2
|
||||||
assert len({r.index for r in at_depth_2}) == 2, "same depth, different turns"
|
assert len({r.index for r in at_depth_2}) == 2, "same depth, different turns"
|
||||||
@@ -386,8 +391,9 @@ def test_the_legacy_index_is_reissued_so_two_branches_never_share_one(client):
|
|||||||
# ------------------------------------------------------- a file that is wrong
|
# ------------------------------------------------------- a file that is wrong
|
||||||
|
|
||||||
def test_a_node_naming_a_branch_the_file_does_not_list_is_refused(client):
|
def test_a_node_naming_a_branch_the_file_does_not_list_is_refused(client):
|
||||||
"""Refused, not half-applied. A tree missing a branch is a story that
|
"""The import must refuse this file rather than half-apply it. A tree
|
||||||
silently stops, which is the failure this whole phase exists to end."""
|
missing a branch is a story that silently stops, which is the failure
|
||||||
|
this whole phase exists to prevent."""
|
||||||
payload = _export(client, _forked_story(client))
|
payload = _export(client, _forked_story(client))
|
||||||
payload["branches"] = payload["branches"][:1]
|
payload["branches"] = payload["branches"][:1]
|
||||||
before = _adventure_count()
|
before = _adventure_count()
|
||||||
@@ -399,8 +405,9 @@ def test_a_node_naming_a_branch_the_file_does_not_list_is_refused(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_branch_forking_from_one_listed_after_it_is_refused(client):
|
def test_a_branch_forking_from_one_listed_after_it_is_refused(client):
|
||||||
"""The ordering rule buys acyclicity for the price of a comparison — and a
|
"""The ordering rule guarantees no cycles at the cost of one
|
||||||
cycle would be an import that never returns rather than one that fails."""
|
comparison. Without it, a cycle would produce an import that never
|
||||||
|
returns instead of one that fails cleanly."""
|
||||||
payload = _export(client, _forked_story(client))
|
payload = _export(client, _forked_story(client))
|
||||||
payload["branches"] = [{"parent": 1, "forkDepth": 0}, {"parent": None, "forkDepth": None}]
|
payload["branches"] = [{"parent": 1, "forkDepth": 0}, {"parent": None, "forkDepth": None}]
|
||||||
before = _adventure_count()
|
before = _adventure_count()
|
||||||
@@ -439,8 +446,9 @@ def test_more_branches_than_the_cap_is_refused(client, monkeypatch):
|
|||||||
def test_a_turn_the_file_gives_no_live_attempt_still_tells_one(client):
|
def test_a_turn_the_file_gives_no_live_attempt_still_tells_one(client):
|
||||||
"""A coordinate with nothing live is a turn no read can see.
|
"""A coordinate with nothing live is a turn no read can see.
|
||||||
|
|
||||||
The file is allowed to be wrong about this — it is a text file — so the
|
The file is allowed to be wrong about this, since it is a text file
|
||||||
import picks the first attempt rather than importing a story with a hole.
|
someone can edit. The import picks the first attempt instead of
|
||||||
|
importing a story with a gap.
|
||||||
"""
|
"""
|
||||||
payload = _export(client, _forked_story(client))
|
payload = _export(client, _forked_story(client))
|
||||||
for node in payload["actions"]:
|
for node in payload["actions"]:
|
||||||
@@ -456,7 +464,8 @@ def test_a_turn_the_file_gives_no_live_attempt_still_tells_one(client):
|
|||||||
# ------------------------------------------------------------- the v1 reader
|
# ------------------------------------------------------------- the v1 reader
|
||||||
|
|
||||||
def test_a_v1_bundle_still_imports(client):
|
def test_a_v1_bundle_still_imports(client):
|
||||||
"""The reader stays after the writer goes: those files are already saved."""
|
"""The v1 reader must remain even after the v1 writer is gone, because
|
||||||
|
those files already exist and are saved."""
|
||||||
payload = {
|
payload = {
|
||||||
"format": bundle.LEGACY_FORMAT,
|
"format": bundle.LEGACY_FORMAT,
|
||||||
"title": "Old backup",
|
"title": "Old backup",
|
||||||
@@ -474,8 +483,8 @@ def test_a_v1_bundle_still_imports(client):
|
|||||||
copy = _imported(client, payload)
|
copy = _imported(client, payload)
|
||||||
|
|
||||||
assert _texts(client, copy) == [OPENING, "> You go north.", "Two."]
|
assert _texts(client, copy) == [OPENING, "> You go north.", "Two."]
|
||||||
# One branch, and the `variants` array split back into the sibling group it
|
# The import produces one branch, and the `variants` array splits
|
||||||
# always described.
|
# back into the sibling group it always described.
|
||||||
assert len(_branches(client, copy)) == 1
|
assert len(_branches(client, copy)) == 1
|
||||||
ai = [r for r in _rows(copy) if r.type == "ai"]
|
ai = [r for r in _rows(copy) if r.type == "ai"]
|
||||||
assert [(r.text, r.live) for r in ai] == [("One.", False), ("Two.", True)]
|
assert [(r.text, r.live) for r in ai] == [("One.", False), ("Two.", True)]
|
||||||
@@ -483,8 +492,9 @@ def test_a_v1_bundle_still_imports(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_v1_bundle_with_a_cursor_lands_it_on_a_node(client):
|
def test_a_v1_bundle_with_a_cursor_lands_it_on_a_node(client):
|
||||||
"""v1 counts covered actions; the tree anchors them. The translation needs
|
"""v1 counts covered actions. The tree anchors them to a node instead.
|
||||||
the nodes to exist, so it happens after they are written."""
|
The translation needs the nodes to exist first, so it runs after they
|
||||||
|
are written."""
|
||||||
payload = {
|
payload = {
|
||||||
"format": bundle.LEGACY_FORMAT, "title": "Old backup",
|
"format": bundle.LEGACY_FORMAT, "title": "Old backup",
|
||||||
"memoryCursor": 2, "summaryCursor": 2,
|
"memoryCursor": 2, "summaryCursor": 2,
|
||||||
@@ -507,7 +517,8 @@ def test_a_v1_bundle_with_a_cursor_lands_it_on_a_node(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_v2_bundle_brings_its_anchors_back(client):
|
def test_a_v2_bundle_brings_its_anchors_back(client):
|
||||||
"""The other direction: v2 carries the anchor and the count is read off it."""
|
"""The other direction: v2 carries the anchor directly, and the legacy
|
||||||
|
count is derived from it."""
|
||||||
original = _forked_story(client)
|
original = _forked_story(client)
|
||||||
tip = [a for a in _rows(original) if a.live][-1]
|
tip = [a for a in _rows(original) if a.live][-1]
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
@@ -535,13 +546,14 @@ def test_a_v2_bundle_brings_its_anchors_back(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_v1_memory_that_summarises_nothing_lands_on_the_root(client):
|
def test_a_v1_memory_that_summarises_nothing_lands_on_the_root(client):
|
||||||
"""The import has to answer the question migration 62 answered.
|
"""The import must answer the question migration 62 already answered.
|
||||||
|
|
||||||
A v1 file has no depths, and a memory the player typed has no `sourceEnd`
|
A v1 file has no depths, and a memory the player typed has no
|
||||||
to derive one from — so it used to come back with a NULL depth, which is the
|
`sourceEnd` to derive one from. It used to come back with a NULL
|
||||||
exact state SP7 removed from the schema. `Path._entry_clause` compares
|
depth, the exact state SP7 removed from the schema.
|
||||||
`depth <= max_depth` and a NULL fails it, so the memory would read fine
|
`Path._entry_clause` compares `depth <= max_depth`, and a NULL fails
|
||||||
until the imported adventure was forked and then vanish from the new branch.
|
that comparison. The memory would read fine until the imported
|
||||||
|
adventure forked, then vanish from the new branch.
|
||||||
"""
|
"""
|
||||||
payload = {
|
payload = {
|
||||||
"format": bundle.LEGACY_FORMAT, "title": "Old backup",
|
"format": bundle.LEGACY_FORMAT, "title": "Old backup",
|
||||||
@@ -572,12 +584,13 @@ def test_a_v1_memory_that_summarises_nothing_lands_on_the_root(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_action_cap_counts_the_rows_a_v1_file_expands_into(client, monkeypatch):
|
def test_the_action_cap_counts_the_rows_a_v1_file_expands_into(client, monkeypatch):
|
||||||
"""The cap has to count what gets written, not what the file lists.
|
"""The cap must count what gets written, not what the file lists.
|
||||||
|
|
||||||
A v1 turn carries its retries in a `variants` array, and SP4 made every
|
A v1 turn carries its retries in a `variants` array, and SP4 made
|
||||||
attempt a row — so one entry can become ten. Counting entries lets a file
|
every attempt a row, so one entry can expand into ten. Counting
|
||||||
inside the cap write a multiple of it, and the body limit is no help: the
|
entries instead of rows would let a file inside the cap write a
|
||||||
text is tiny, it is the row count that is the cost.
|
multiple of it. The body-size limit does not help here: the text is
|
||||||
|
tiny, and the row count is the actual cost.
|
||||||
"""
|
"""
|
||||||
monkeypatch.setattr(auth, "MULTI_USER", True)
|
monkeypatch.setattr(auth, "MULTI_USER", True)
|
||||||
monkeypatch.setattr(limits, "MAX_ACTIONS_PER_ADVENTURE", 6)
|
monkeypatch.setattr(limits, "MAX_ACTIONS_PER_ADVENTURE", 6)
|
||||||
|
|||||||
+23
-18
@@ -1,7 +1,8 @@
|
|||||||
"""HTTP tests for the AI Chat scratchpad (power users only).
|
"""HTTP tests for the AI Chat scratchpad (power users only).
|
||||||
|
|
||||||
Covers the access gate, the streamed reply, and the demo-key model pinning —
|
Covers the access gate, the streamed reply, and the demo-key model pinning.
|
||||||
the part that must not let a public visitor reach paid models through this page.
|
This pinning must not let a public visitor reach paid models through this
|
||||||
|
page.
|
||||||
|
|
||||||
python -m pytest tests/test_chat.py -v
|
python -m pytest tests/test_chat.py -v
|
||||||
"""
|
"""
|
||||||
@@ -59,13 +60,14 @@ def client(monkeypatch):
|
|||||||
|
|
||||||
monkeypatch.setattr(chat, "OpenAICompatibleProvider", FakeProvider)
|
monkeypatch.setattr(chat, "OpenAICompatibleProvider", FakeProvider)
|
||||||
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
|
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
|
||||||
# Multi-user mode is what makes the power-user gate meaningful (local mode
|
# Multi-user mode is what makes the power-user gate meaningful, because
|
||||||
# trusts everyone); the allowlist is set per-test.
|
# local mode trusts everyone. The allowlist is set per test.
|
||||||
monkeypatch.setattr(auth, "MULTI_USER", True)
|
monkeypatch.setattr(auth, "MULTI_USER", True)
|
||||||
monkeypatch.setattr(auth, "POWER_USERS", {"power@example.com"})
|
monkeypatch.setattr(auth, "POWER_USERS", {"power@example.com"})
|
||||||
# These tests deliberately do NOT stub resolve_provider_config: the point is
|
# These tests deliberately do not stub resolve_provider_config. The
|
||||||
# to exercise the real BYOK-vs-demo decision, since that is what keeps the
|
# point is to exercise the real BYOK-vs-demo decision, since that
|
||||||
# shared key off paid models. Each test picks a mode with _byok/_demo below.
|
# decision is what keeps the shared key off paid models. Each test
|
||||||
|
# picks a mode with _byok/_demo below.
|
||||||
|
|
||||||
def _current_user(db=Depends(get_db)):
|
def _current_user(db=Depends(get_db)):
|
||||||
return db.get(models.User, user_id)
|
return db.get(models.User, user_id)
|
||||||
@@ -152,8 +154,9 @@ def test_demo_key_pins_model_to_whitelist(client, monkeypatch):
|
|||||||
|
|
||||||
|
|
||||||
def test_demo_key_ignores_an_off_whitelist_settings_model(client, monkeypatch):
|
def test_demo_key_ignores_an_off_whitelist_settings_model(client, monkeypatch):
|
||||||
"""The override isn't the only untrusted input — Settings.model is user-set
|
"""The override is not the only untrusted input. `Settings.model` is
|
||||||
too, and it must be pinned the same way when there's no BYOK key."""
|
also user-set, and it must be pinned the same way when there is no
|
||||||
|
BYOK key."""
|
||||||
_demo(monkeypatch)
|
_demo(monkeypatch)
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
@@ -167,8 +170,8 @@ def test_demo_key_ignores_an_off_whitelist_settings_model(client, monkeypatch):
|
|||||||
|
|
||||||
|
|
||||||
def test_demo_key_endpoint_cannot_be_redirected(client, monkeypatch):
|
def test_demo_key_endpoint_cannot_be_redirected(client, monkeypatch):
|
||||||
"""A user-controlled endpoint_url would leak the key itself, which is worse
|
"""A user-controlled `endpoint_url` would leak the key itself, which is
|
||||||
than spending it — the demo branch pins the URL too."""
|
worse than spending it. The demo branch pins the URL too."""
|
||||||
_demo(monkeypatch)
|
_demo(monkeypatch)
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
@@ -183,8 +186,8 @@ def test_demo_key_endpoint_cannot_be_redirected(client, monkeypatch):
|
|||||||
|
|
||||||
def test_provider_config_refuses_server_funded_paid_model(monkeypatch):
|
def test_provider_config_refuses_server_funded_paid_model(monkeypatch):
|
||||||
"""The structural backstop: a hand-built config (a future code path that
|
"""The structural backstop: a hand-built config (a future code path that
|
||||||
forgets to go through resolve_provider_config) can't run a server-funded
|
forgets to go through resolve_provider_config) cannot run a
|
||||||
turn on an off-whitelist model."""
|
server-funded turn on an off-whitelist model."""
|
||||||
monkeypatch.setattr(auth, "DEMO_API_KEY", "demo-key")
|
monkeypatch.setattr(auth, "DEMO_API_KEY", "demo-key")
|
||||||
monkeypatch.setattr(auth, "DEMO_MODELS", ["free/allowed"])
|
monkeypatch.setattr(auth, "DEMO_MODELS", ["free/allowed"])
|
||||||
with pytest.raises(ValueError):
|
with pytest.raises(ValueError):
|
||||||
@@ -196,9 +199,10 @@ def test_provider_config_refuses_server_funded_paid_model(monkeypatch):
|
|||||||
|
|
||||||
def test_byok_user_may_reuse_the_demo_keys_value(client, monkeypatch):
|
def test_byok_user_may_reuse_the_demo_keys_value(client, monkeypatch):
|
||||||
"""Regression: the demo key is just an OpenRouter key, so a user can paste
|
"""Regression: the demo key is just an OpenRouter key, so a user can paste
|
||||||
that same value into their own Settings. That's BYOK — they're paying — and
|
that same value into their own Settings. That is still BYOK, because
|
||||||
it must not trip the guard. It used to raise on every resolution, which
|
the user is paying, and it must not trip the guard. It used to raise
|
||||||
500'd GET /auth/me and took the whole SPA down (no nav, no chat)."""
|
on every resolution, which returned a 500 from `GET /auth/me` and
|
||||||
|
broke the entire SPA (no nav, no chat)."""
|
||||||
monkeypatch.setattr(auth, "demo_enabled", lambda: True)
|
monkeypatch.setattr(auth, "demo_enabled", lambda: True)
|
||||||
monkeypatch.setattr(auth, "DEMO_API_KEY", "shared-key")
|
monkeypatch.setattr(auth, "DEMO_API_KEY", "shared-key")
|
||||||
monkeypatch.setattr(auth, "DEMO_ENDPOINT_URL", "http://demo")
|
monkeypatch.setattr(auth, "DEMO_ENDPOINT_URL", "http://demo")
|
||||||
@@ -221,14 +225,15 @@ def test_byok_user_may_reuse_the_demo_keys_value(client, monkeypatch):
|
|||||||
|
|
||||||
|
|
||||||
def test_resolve_provider_config_is_the_single_choke_point(monkeypatch):
|
def test_resolve_provider_config_is_the_single_choke_point(monkeypatch):
|
||||||
"""Turns, AI Chat and the connection test all resolve through this one
|
"""Turns, AI Chat, and the connection test all resolve through this one
|
||||||
function, so pinning it here pins every caller. No DB or HTTP needed."""
|
function, so pinning it here pins every caller. No DB or HTTP needed."""
|
||||||
monkeypatch.setattr(auth, "demo_enabled", lambda: True)
|
monkeypatch.setattr(auth, "demo_enabled", lambda: True)
|
||||||
monkeypatch.setattr(auth, "DEMO_API_KEY", "demo-key")
|
monkeypatch.setattr(auth, "DEMO_API_KEY", "demo-key")
|
||||||
monkeypatch.setattr(auth, "DEMO_ENDPOINT_URL", "http://demo")
|
monkeypatch.setattr(auth, "DEMO_ENDPOINT_URL", "http://demo")
|
||||||
monkeypatch.setattr(auth, "DEMO_MODELS", ["free/allowed"])
|
monkeypatch.setattr(auth, "DEMO_MODELS", ["free/allowed"])
|
||||||
|
|
||||||
# No key of their own: endpoint AND model are pinned, whatever they set.
|
# No key of their own: both endpoint and model are pinned, regardless
|
||||||
|
# of what they set.
|
||||||
no_key = models.Settings(endpoint_url="http://mine/v1", api_key="", model="expensive/paid")
|
no_key = models.Settings(endpoint_url="http://mine/v1", api_key="", model="expensive/paid")
|
||||||
assert auth.resolve_provider_config(no_key) == auth.ProviderConfig(
|
assert auth.resolve_provider_config(no_key) == auth.ProviderConfig(
|
||||||
"http://demo", "demo-key", "free/allowed", True)
|
"http://demo", "demo-key", "free/allowed", True)
|
||||||
|
|||||||
@@ -1,19 +1,19 @@
|
|||||||
"""Guards on how much the database is asked for.
|
"""Guards on how much the database is asked for.
|
||||||
|
|
||||||
context_snapshot holds the entire assembled prompt for a turn — 163 KB a row
|
`context_snapshot` holds the entire assembled prompt for a turn. That is 163 kB
|
||||||
averaged over production, 232 KB on the longest adventure, and 89% of the
|
per row averaged over production, 232 kB on the longest adventure, and 89% of the
|
||||||
database. It used to be pulled for every action on every adventure load and
|
database. It used to be fetched for every action on every adventure load and
|
||||||
every turn, to read two tiny things out of it. These tests fail if that
|
every turn, to read two small values out of it. These tests fail if that
|
||||||
regresses.
|
returns.
|
||||||
|
|
||||||
Two kinds of guard live here, and both are needed:
|
Two kinds of guard live here, and both are needed:
|
||||||
|
|
||||||
* **column guards** assert which columns a statement names. That is the shape
|
* Column guards assert which columns a statement names. That is the shape both
|
||||||
both of this project's egress blowouts took — one query quietly carrying a
|
of this project's egress regressions took: one query carrying a column nobody
|
||||||
column nobody read.
|
read.
|
||||||
* **byte ceilings** assert what a request actually costs. Every column guard
|
* Byte ceilings assert what a request costs. Every column guard would still pass
|
||||||
would still pass if a response grew tenfold within the columns it is allowed
|
if a response grew tenfold within the columns it is allowed to read, which is
|
||||||
to read, which is what a story that keeps getting longer does.
|
what a story that keeps getting longer does.
|
||||||
|
|
||||||
python -m pytest tests/test_egress.py -v
|
python -m pytest tests/test_egress.py -v
|
||||||
"""
|
"""
|
||||||
@@ -46,10 +46,10 @@ from tools.fakeprose import prose
|
|||||||
# column enormous, plus the small world_state slice the UI actually needs.
|
# column enormous, plus the small world_state slice the UI actually needs.
|
||||||
#
|
#
|
||||||
# Varied text, not `"x" * 20_000`. The column is stored compressed now
|
# Varied text, not `"x" * 20_000`. The column is stored compressed now
|
||||||
# (migration 43), and a repeated character compresses about a thousandfold —
|
# by migration 43, and a repeated character compresses about a thousandfold.
|
||||||
# which would make the byte ceilings below pass against a fixture that costs
|
# That would make the byte ceilings below pass against a fixture that costs
|
||||||
# nothing, testing nothing. Prose-shaped filler compresses like the prompts
|
# nothing, which tests nothing. Prose-shaped filler compresses like the prompts
|
||||||
# this stands in for.
|
# it stands in for.
|
||||||
_SNAPSHOT_RNG = random.Random(20_260_817)
|
_SNAPSHOT_RNG = random.Random(20_260_817)
|
||||||
BIG_SNAPSHOT = {
|
BIG_SNAPSHOT = {
|
||||||
"system": prose(_SNAPSHOT_RNG, 20_000),
|
"system": prose(_SNAPSHOT_RNG, 20_000),
|
||||||
@@ -149,8 +149,8 @@ def test_the_state_snapshots_are_not_fetched_in_bulk(client, sql_log):
|
|||||||
"""All four are rollback snapshots, only ever needed for the single node
|
"""All four are rollback snapshots, only ever needed for the single node
|
||||||
being undone, retried past or switched to.
|
being undone, retried past or switched to.
|
||||||
|
|
||||||
The `_after` pair is the live one since SP4 and the `_before` pair is dead
|
The `_after` pair is the live one since SP4, and the `_before` pair is
|
||||||
weight until SP8 drops it — a page load must pay for neither.
|
unused until SP8 drops it. A page load must pay for neither.
|
||||||
"""
|
"""
|
||||||
client.get(f"/api/adventures/{client.adv_id}")
|
client.get(f"/api/adventures/{client.adv_id}")
|
||||||
selects = action_selects(sql_log)
|
selects = action_selects(sql_log)
|
||||||
@@ -195,8 +195,11 @@ def test_variant_count_survives_variants_being_deferred(client):
|
|||||||
|
|
||||||
def test_counting_actions_does_not_name_the_deferred_columns(client, sql_log):
|
def test_counting_actions_does_not_name_the_deferred_columns(client, sql_log):
|
||||||
"""A count that wraps the entity select in a subquery names every column in
|
"""A count that wraps the entity select in a subquery names every column in
|
||||||
the emitted SQL — no bytes come back, but the database still reads them and
|
the emitted SQL.
|
||||||
the guard above cannot tell it apart from a real bulk fetch."""
|
|
||||||
|
No bytes come back, but the database still reads them, and the guard above
|
||||||
|
cannot distinguish that from a real bulk fetch.
|
||||||
|
"""
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
adventure = db.get(models.Adventure, client.adv_id)
|
adventure = db.get(models.Adventure, client.adv_id)
|
||||||
@@ -214,7 +217,8 @@ def test_counting_actions_does_not_name_the_deferred_columns(client, sql_log):
|
|||||||
|
|
||||||
|
|
||||||
def test_snapshot_is_still_reachable_on_demand(client):
|
def test_snapshot_is_still_reachable_on_demand(client):
|
||||||
"""Deferred means lazy, not gone — Insights still gets the full thing."""
|
"""Deferred means lazy rather than absent. Insights still gets the whole
|
||||||
|
snapshot."""
|
||||||
r = client.get(f"/api/adventures/{client.adv_id}")
|
r = client.get(f"/api/adventures/{client.adv_id}")
|
||||||
action_id = r.json()["actions"][0]["id"]
|
action_id = r.json()["actions"][0]["id"]
|
||||||
r = client.get(f"/api/adventures/{client.adv_id}/actions/{action_id}/context")
|
r = client.get(f"/api/adventures/{client.adv_id}/actions/{action_id}/context")
|
||||||
@@ -231,9 +235,9 @@ def as_json_snapshot_column(db) -> None:
|
|||||||
|
|
||||||
Migration 36 lifts world_delta out of the snapshot with SQL JSON
|
Migration 36 lifts world_delta out of the snapshot with SQL JSON
|
||||||
functions, so it can only run while the column still *is* JSON. In a real
|
functions, so it can only run while the column still *is* JSON. In a real
|
||||||
upgrade it always is — 36 runs seven migrations before 43 compresses the
|
upgrade it always is, because 36 runs seven migrations before 43 compresses
|
||||||
column into a BLOB — but `create_all` builds today's schema, so a test
|
the column into a BLOB. `create_all` builds today's schema, so a test that
|
||||||
calling that backfill has to rebuild the schema it was written against.
|
calls that backfill has to rebuild the schema it was written against.
|
||||||
"""
|
"""
|
||||||
db.execute(text("ALTER TABLE actions DROP COLUMN context_snapshot"))
|
db.execute(text("ALTER TABLE actions DROP COLUMN context_snapshot"))
|
||||||
db.execute(text("ALTER TABLE actions ADD COLUMN context_snapshot JSON"))
|
db.execute(text("ALTER TABLE actions ADD COLUMN context_snapshot JSON"))
|
||||||
@@ -269,9 +273,11 @@ def test_backfill_populates_world_delta_from_existing_snapshots(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_backfill_populates_variant_count_from_existing_variants(client):
|
def test_backfill_populates_variant_count_from_existing_variants(client):
|
||||||
"""Migration 37 counts the lists server-side — reading them into Python to
|
"""Migration 37 counts the lists on the server.
|
||||||
count them would mean pulling the column across the wire once to stop
|
|
||||||
pulling it across forever."""
|
Reading them into Python to count them would fetch the column over the wire
|
||||||
|
once in order to stop fetching it on every request.
|
||||||
|
"""
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
db.execute(text("UPDATE actions SET variant_count = 0"))
|
db.execute(text("UPDATE actions SET variant_count = 0"))
|
||||||
@@ -304,20 +310,20 @@ def test_backfill_leaves_actions_without_world_state_alone(client):
|
|||||||
|
|
||||||
# ---------------------------------------------------------------- byte ceilings
|
# ---------------------------------------------------------------- byte ceilings
|
||||||
#
|
#
|
||||||
# The tests above assert which *columns* a statement names, which is the shape
|
# The tests above assert which columns a statement names, which is the shape both
|
||||||
# both of this project's egress blowouts took. They would all still pass if a
|
# of this project's egress regressions took. They would all still pass if a
|
||||||
# response quietly grew tenfold within the columns it is allowed to read — and
|
# response grew tenfold within the columns it is allowed to read, and a story
|
||||||
# a story that keeps getting longer does exactly that. These put a number on it.
|
# that keeps getting longer does that. These tests put a number on it.
|
||||||
#
|
#
|
||||||
# Ceilings are per action rather than absolute, so they mean the same thing
|
# The ceilings are per action rather than absolute, so they mean the same thing
|
||||||
# whatever size the fixture is set to, and they are generous: the point is to
|
# whatever size the fixture is, and they are generous. They exist to catch a
|
||||||
# catch a tenfold regression, not to freeze today's byte count.
|
# tenfold regression rather than to freeze today's byte count.
|
||||||
|
|
||||||
ACTIONS_IN_FIXTURE = 12
|
ACTIONS_IN_FIXTURE = 12
|
||||||
|
|
||||||
# 3 kB an action against a real 994 B, measured on production 2026-08-17.
|
# 3 kB an action against a real 994 B, measured on production 2026-08-17.
|
||||||
# Anything that pulls a deferred column blows past this by two orders of
|
# Anything that pulls a deferred column blows past this by two orders of
|
||||||
# magnitude — see test_the_ceiling_discriminates below.
|
# magnitude. See `test_the_ceiling_discriminates` below.
|
||||||
PAGE_LOAD_BYTES_PER_ACTION = 3_000
|
PAGE_LOAD_BYTES_PER_ACTION = 3_000
|
||||||
|
|
||||||
|
|
||||||
@@ -325,8 +331,9 @@ PAGE_LOAD_BYTES_PER_ACTION = 3_000
|
|||||||
def meter():
|
def meter():
|
||||||
"""A byte meter on the shared engine, removed again afterwards.
|
"""A byte meter on the shared engine, removed again afterwards.
|
||||||
|
|
||||||
Requested *after* `client` in a test's arguments so that building the
|
A test requests this after `client` in its arguments, so that building the
|
||||||
fixture — a write path nobody plays — is not charged to any scope.
|
fixture, which is a write path no player takes, is not charged to any
|
||||||
|
scope.
|
||||||
"""
|
"""
|
||||||
m = dbmeter.Meter()
|
m = dbmeter.Meter()
|
||||||
m.attach(engine)
|
m.attach(engine)
|
||||||
@@ -364,8 +371,8 @@ def test_the_action_list_stays_under_its_byte_ceiling(client, meter):
|
|||||||
|
|
||||||
|
|
||||||
def test_reading_one_action_does_not_cost_the_whole_story(client, meter):
|
def test_reading_one_action_does_not_cost_the_whole_story(client, meter):
|
||||||
"""The snapshot is reachable on demand, and that request should pay for
|
"""The snapshot is reachable on demand, and that request pays for one row
|
||||||
one row's worth — not the adventure's."""
|
rather than for the whole adventure."""
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
action_id = db.query(models.Action.id).order_by(models.Action.id).first()[0]
|
action_id = db.query(models.Action.id).order_by(models.Action.id).first()[0]
|
||||||
@@ -459,10 +466,10 @@ def test_the_ceiling_discriminates(client, meter):
|
|||||||
budget. If this ever stops exceeding it, the fixture has gone too small for
|
budget. If this ever stops exceeding it, the fixture has gone too small for
|
||||||
the tests above to mean anything.
|
the tests above to mean anything.
|
||||||
|
|
||||||
The margin used to be a hundredfold and is now about six. That is not the
|
The margin used to be a hundredfold and is now about six. The guard has not
|
||||||
guard weakening — it is migration 43 compressing the column, and the
|
weakened. Migration 43 compresses the column, and the fixture text is
|
||||||
fixture text being prose-shaped so it compresses like a real prompt rather
|
prose-shaped, so it compresses like a real prompt rather than like a repeated
|
||||||
than like a repeated character.
|
character.
|
||||||
"""
|
"""
|
||||||
budget = ACTIONS_IN_FIXTURE * PAGE_LOAD_BYTES_PER_ACTION
|
budget = ACTIONS_IN_FIXTURE * PAGE_LOAD_BYTES_PER_ACTION
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
|
|||||||
@@ -1,9 +1,9 @@
|
|||||||
"""Migration 38: embeddings move from a JSON list to packed float32.
|
"""Migration 38: embeddings move from a JSON list to packed float32.
|
||||||
|
|
||||||
The conversion has to be exact, because nothing re-embeds — a memory whose
|
The conversion has to be exact, because nothing re-embeds. A memory whose
|
||||||
vector shifts is silently ranked wrong forever, with no error anywhere to say
|
vector shifts is silently ranked wrong forever, with no error anywhere to
|
||||||
so. So these tests check the numbers survive the round trip bit for bit, and
|
report it. These tests check that the numbers survive the round trip bit
|
||||||
that the migration reaches every row however many there are.
|
for bit, and that the migration reaches every row, however many there are.
|
||||||
|
|
||||||
python -m pytest tests/test_embedding_blob.py -v
|
python -m pytest tests/test_embedding_blob.py -v
|
||||||
"""
|
"""
|
||||||
@@ -29,8 +29,8 @@ from tests import schema_rewind
|
|||||||
|
|
||||||
|
|
||||||
def float32(value: float) -> float:
|
def float32(value: float) -> float:
|
||||||
"""`value` as the double nearest to its float32 truncation — what an
|
"""`value` as the double nearest to its float32 truncation. This is
|
||||||
embedding endpoint's JSON actually holds."""
|
what an embedding endpoint's JSON actually holds."""
|
||||||
return struct.unpack("<f", struct.pack("<f", value))[0]
|
return struct.unpack("<f", struct.pack("<f", value))[0]
|
||||||
|
|
||||||
|
|
||||||
@@ -71,7 +71,7 @@ def test_pack_round_trips_exactly():
|
|||||||
|
|
||||||
|
|
||||||
def test_packed_vector_is_four_bytes_per_dimension():
|
def test_packed_vector_is_four_bytes_per_dimension():
|
||||||
"""The whole point: 1536 dims is 6 KB here against ~31 KB as JSON."""
|
"""1536 dims takes 6 KB packed, against about 31 KB as JSON."""
|
||||||
vector = sample_vector(random.Random(2))
|
vector = sample_vector(random.Random(2))
|
||||||
blob = vectors.pack(vector)
|
blob = vectors.pack(vector)
|
||||||
assert len(blob) == 1536 * 4
|
assert len(blob) == 1536 * 4
|
||||||
@@ -84,9 +84,9 @@ def test_pack_handles_the_extremes():
|
|||||||
|
|
||||||
|
|
||||||
def test_unpack_returns_a_compact_array():
|
def test_unpack_returns_a_compact_array():
|
||||||
"""These are held in memory between turns, so the container matters: an
|
"""These vectors stay in memory between turns, so the container type
|
||||||
array("f") is the 4 bytes a component the column is, a list of Python
|
matters. An array("f") stores each component in 4 bytes, the same width
|
||||||
floats is eight times that."""
|
as the column. A list of Python floats uses eight times that."""
|
||||||
vector = sample_vector(random.Random(9))
|
vector = sample_vector(random.Random(9))
|
||||||
unpacked = vectors.unpack(vectors.pack(vector))
|
unpacked = vectors.unpack(vectors.pack(vector))
|
||||||
assert unpacked.typecode == "f"
|
assert unpacked.typecode == "f"
|
||||||
@@ -95,15 +95,16 @@ def test_unpack_returns_a_compact_array():
|
|||||||
|
|
||||||
|
|
||||||
def test_pack_rounds_a_value_float32_cannot_hold():
|
def test_pack_rounds_a_value_float32_cannot_hold():
|
||||||
"""The guarantee is exactness for vectors that came from an embedding
|
"""The exactness guarantee applies only to vectors that came from an
|
||||||
model, which computes in float32 — not for arbitrary doubles. Worth
|
embedding model, which computes in float32, not to arbitrary doubles.
|
||||||
pinning down, because it is the line the round-trip claim sits on."""
|
This test pins down that boundary, because it is where the round-trip
|
||||||
|
claim holds."""
|
||||||
assert vectors.unpack(vectors.pack([1e-38]))[0] != 1e-38
|
assert vectors.unpack(vectors.pack([1e-38]))[0] != 1e-38
|
||||||
assert vectors.unpack(vectors.pack([1e-38]))[0] == pytest.approx(1e-38)
|
assert vectors.unpack(vectors.pack([1e-38]))[0] == pytest.approx(1e-38)
|
||||||
|
|
||||||
|
|
||||||
def test_cosine_moved_but_still_reachable_from_memorybank():
|
def test_cosine_moved_but_still_reachable_from_memorybank():
|
||||||
"""Callers import it from memorybank; the maths lives in vectors."""
|
"""Callers import it from memorybank. The math lives in vectors."""
|
||||||
assert memorybank.cosine is vectors.cosine
|
assert memorybank.cosine is vectors.cosine
|
||||||
assert vectors.cosine([1.0, 0.0], [1.0, 0.0]) == pytest.approx(1.0)
|
assert vectors.cosine([1.0, 0.0], [1.0, 0.0]) == pytest.approx(1.0)
|
||||||
assert vectors.cosine([1.0, 0.0], [0.0, 1.0]) == pytest.approx(0.0)
|
assert vectors.cosine([1.0, 0.0], [0.0, 1.0]) == pytest.approx(0.0)
|
||||||
@@ -149,11 +150,11 @@ def test_set_vector_none_clears_both(db, adventure):
|
|||||||
def add_legacy_json_column(db) -> None:
|
def add_legacy_json_column(db) -> None:
|
||||||
"""Put `memories.embedding` back for the length of a test.
|
"""Put `memories.embedding` back for the length of a test.
|
||||||
|
|
||||||
Migration 42 dropped it and the model no longer declares it, so
|
Migration 42 dropped it, and the model no longer declares it, so
|
||||||
`create_all` does not produce it — but everything below is testing the
|
`create_all` does not produce it. Everything below tests the upgrade
|
||||||
upgrade *from* a database that still has it, which is the only state in
|
from a database that still has the column, which is the only state
|
||||||
which the backfill has any work to do. Re-adding it by hand is what keeps
|
where the backfill has any work to do. Re-adding it by hand keeps these
|
||||||
these tests honest about the schema they claim to be starting from.
|
tests honest about the schema they claim to start from.
|
||||||
"""
|
"""
|
||||||
db.execute(text("ALTER TABLE memories ADD COLUMN embedding JSON"))
|
db.execute(text("ALTER TABLE memories ADD COLUMN embedding JSON"))
|
||||||
db.commit()
|
db.commit()
|
||||||
@@ -193,8 +194,9 @@ def test_backfill_converts_every_existing_vector(db, adventure):
|
|||||||
|
|
||||||
|
|
||||||
def test_backfill_reaches_past_one_batch(db, adventure):
|
def test_backfill_reaches_past_one_batch(db, adventure):
|
||||||
"""It loops on id, and an off-by-one there would silently leave the tail
|
"""It loops on id, and an off-by-one there would silently leave the
|
||||||
of a big bank unconverted — which reads as "not embedded yet"."""
|
tail of a big bank unconverted. That failure reads as "not embedded
|
||||||
|
yet"."""
|
||||||
count = migrations.BACKFILL_BATCH * 2 + 3
|
count = migrations.BACKFILL_BATCH * 2 + 3
|
||||||
expected = seed_json_only(db, adventure, count=count, dims=4)
|
expected = seed_json_only(db, adventure, count=count, dims=4)
|
||||||
|
|
||||||
@@ -237,8 +239,8 @@ def test_backfill_is_idempotent(db, adventure):
|
|||||||
|
|
||||||
|
|
||||||
def test_backfill_skips_a_malformed_row_without_stopping(db, adventure):
|
def test_backfill_skips_a_malformed_row_without_stopping(db, adventure):
|
||||||
"""One bad row must not strand every row after it — the loop orders by id,
|
"""One bad row must not strand every row after it. The loop orders by
|
||||||
so an exception here would leave the rest of the bank unconverted."""
|
id, so an exception here would leave the rest of the bank unconverted."""
|
||||||
expected = seed_json_only(db, adventure, count=2)
|
expected = seed_json_only(db, adventure, count=2)
|
||||||
broken = models.Memory(adventure_id=adventure.id, text="broken")
|
broken = models.Memory(adventure_id=adventure.id, text="broken")
|
||||||
db.add(broken)
|
db.add(broken)
|
||||||
@@ -288,9 +290,9 @@ def test_bootstrap_adds_the_columns_and_backfills_them(db, adventure):
|
|||||||
# embedded must still read as not embedded afterwards.
|
# embedded must still read as not embedded afterwards.
|
||||||
assert by_id[unembedded_id] == (None, False)
|
assert by_id[unembedded_id] == (None, False)
|
||||||
|
|
||||||
# ...and migration 42, at the end of the same run, takes the JSON column
|
# Migration 42, at the end of the same run, removes the JSON column.
|
||||||
# away. Ordering matters: 38 reads it, 42 drops it, and an upgrade that
|
# Ordering matters: 38 reads it, 42 drops it, and an upgrade that ran
|
||||||
# ran them the other way round would arrive with an empty bank.
|
# them in the other order would arrive with an empty bank.
|
||||||
with engine.begin() as conn:
|
with engine.begin() as conn:
|
||||||
columns = {row[1] for row in conn.execute(text("PRAGMA table_info(memories)"))}
|
columns = {row[1] for row in conn.execute(text("PRAGMA table_info(memories)"))}
|
||||||
assert "embedding" not in columns
|
assert "embedding" not in columns
|
||||||
|
|||||||
@@ -1,21 +1,21 @@
|
|||||||
"""Switching embedding models must re-embed the bank.
|
"""Switching embedding models must re-embed the bank.
|
||||||
|
|
||||||
Vectors from two different models are not comparable — different space, often
|
Vectors from two different models are not comparable. They live in
|
||||||
different width — so changing the model has to throw the stored ones away and
|
different spaces and often have different widths. Changing the model must
|
||||||
let the post-turn pass rebuild them.
|
discard the stored vectors and let the post-turn pass rebuild them.
|
||||||
|
|
||||||
That worked while the vectors lived in `memories.embedding`: the settings
|
This worked while the vectors lived in `memories.embedding`. The settings
|
||||||
route nulled that column and the embed queue picked the rows up. Migration 38
|
route nulled that column, and the embed queue picked up the rows. Migration
|
||||||
moved the vectors to `embedding_blob` with an `embedded` flag beside them, and
|
38 moved the vectors to `embedding_blob` and added an `embedded` flag beside
|
||||||
the bulk clear kept nulling the old column alone. The blob survived, the flag
|
them, but the bulk clear kept nulling only the old column. The blob
|
||||||
stayed true, `_embed_pending` (which looks for `embedded IS FALSE`) never saw
|
survived, the flag stayed true, and `_embed_pending` (which filters on
|
||||||
the rows, and the bank went on ranking against the previous model's vectors
|
`embedded IS FALSE`) never saw the rows. The bank kept ranking against the
|
||||||
for good.
|
previous model's vectors.
|
||||||
|
|
||||||
Nothing reports this. `cosine` returns 0.0 on a width mismatch, so a
|
Nothing reports this failure. `cosine` returns 0.0 on a width mismatch, so a
|
||||||
different-width model scores every memory zero and retrieval quietly returns
|
different-width model scores every memory zero, and retrieval silently
|
||||||
whichever rows sort first; a same-width model scores plausible-looking
|
returns whichever rows sort first. A same-width model scores plausible
|
||||||
garbage.
|
garbage instead.
|
||||||
|
|
||||||
python -m pytest tests/test_embedding_model_switch.py -v
|
python -m pytest tests/test_embedding_model_switch.py -v
|
||||||
"""
|
"""
|
||||||
@@ -114,8 +114,9 @@ def test_changing_the_model_clears_every_vector(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_cleared_memories_are_queued_for_re_embedding(client):
|
def test_cleared_memories_are_queued_for_re_embedding(client):
|
||||||
"""The flag is not cosmetic: it is the only thing `_embed_pending` filters
|
"""The `embedded` flag is not cosmetic. It is the only condition
|
||||||
on, so this is the assertion that the bank actually recovers."""
|
`_embed_pending` filters on, so this test confirms the bank actually
|
||||||
|
recovers."""
|
||||||
client.put("/api/settings", json={"embedding_model": "model-b"})
|
client.put("/api/settings", json={"embedding_model": "model-b"})
|
||||||
|
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
@@ -155,7 +156,7 @@ def test_retrieval_uses_no_stale_vector_after_the_switch(client, monkeypatch):
|
|||||||
|
|
||||||
|
|
||||||
def test_an_unrelated_settings_change_keeps_the_vectors(client):
|
def test_an_unrelated_settings_change_keeps_the_vectors(client):
|
||||||
"""Only an embedding-model change may clear the bank — re-embedding costs
|
"""Only an embedding-model change may clear the bank. Re-embedding costs
|
||||||
an API call per memory."""
|
an API call per memory."""
|
||||||
r = client.put("/api/settings", json={"model": "some-other-chat-model"})
|
r = client.put("/api/settings", json={"model": "some-other-chat-model"})
|
||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
|
|||||||
@@ -1,7 +1,8 @@
|
|||||||
"""Guest retention policy — app/cleanup.py.
|
"""Guest retention policy: app/cleanup.py.
|
||||||
|
|
||||||
Covers the two things that matter: that idle guests and their whole data
|
These tests cover the two things that matter. Idle guests and their whole
|
||||||
graph actually go, and that nothing else ever does.
|
data graph must actually be deleted, and nothing else must ever be
|
||||||
|
deleted.
|
||||||
|
|
||||||
python -m pytest tests/test_guest_cleanup.py -v
|
python -m pytest tests/test_guest_cleanup.py -v
|
||||||
"""
|
"""
|
||||||
@@ -30,8 +31,8 @@ def db(tmp_path):
|
|||||||
|
|
||||||
@event.listens_for(engine, "connect")
|
@event.listens_for(engine, "connect")
|
||||||
def _fk(dbapi_connection, _record):
|
def _fk(dbapi_connection, _record):
|
||||||
# The whole policy leans on ON DELETE CASCADE; SQLite ignores every
|
# The whole policy relies on ON DELETE CASCADE. SQLite ignores every
|
||||||
# one of them unless this is set (same as database.py does).
|
# one of them unless this is set, the same as database.py does.
|
||||||
cur = dbapi_connection.cursor()
|
cur = dbapi_connection.cursor()
|
||||||
cur.execute("PRAGMA foreign_keys=ON")
|
cur.execute("PRAGMA foreign_keys=ON")
|
||||||
cur.close()
|
cur.close()
|
||||||
@@ -86,7 +87,7 @@ def test_keeps_guest_inside_the_window(db):
|
|||||||
|
|
||||||
|
|
||||||
def test_boundary_is_not_yet_stale(db):
|
def test_boundary_is_not_yet_stale(db):
|
||||||
# Exactly 5 days survives; the comparison is strict.
|
# Exactly 5 days survives. The comparison is strict.
|
||||||
user = make_user(db, days_idle=cleanup.RETENTION_DAYS)
|
user = make_user(db, days_idle=cleanup.RETENTION_DAYS)
|
||||||
assert sweep(db) == 0
|
assert sweep(db) == 0
|
||||||
assert alive(db, user.id)
|
assert alive(db, user.id)
|
||||||
@@ -112,8 +113,9 @@ def test_recent_visit_beats_an_old_created_at(db):
|
|||||||
# ---------- what must never go ----------
|
# ---------- what must never go ----------
|
||||||
|
|
||||||
def test_spares_registered_users(db):
|
def test_spares_registered_users(db):
|
||||||
"""Registering upgrades the guest row in place, so an idle account here is
|
"""Registering upgrades the guest row in place. An idle account here is
|
||||||
a real user with real data — the whole point of signing up."""
|
a real user with real data, which is what signing up is meant to
|
||||||
|
protect."""
|
||||||
user = make_user(db, days_idle=400, guest=False, email="a@b.com")
|
user = make_user(db, days_idle=400, guest=False, email="a@b.com")
|
||||||
assert sweep(db) == 0
|
assert sweep(db) == 0
|
||||||
assert alive(db, user.id)
|
assert alive(db, user.id)
|
||||||
@@ -127,7 +129,7 @@ def test_spares_the_local_mode_user(db):
|
|||||||
|
|
||||||
|
|
||||||
def test_spares_a_guest_flagged_row_that_has_an_email(db):
|
def test_spares_a_guest_flagged_row_that_has_an_email(db):
|
||||||
# Shouldn't exist, but both clauses are checked so it can't be collected.
|
# This row should not exist, but both clauses are checked so it cannot be collected.
|
||||||
user = make_user(db, days_idle=400, guest=True, email="odd@b.com")
|
user = make_user(db, days_idle=400, guest=True, email="odd@b.com")
|
||||||
assert sweep(db) == 0
|
assert sweep(db) == 0
|
||||||
assert alive(db, user.id)
|
assert alive(db, user.id)
|
||||||
@@ -164,10 +166,10 @@ def test_enabled_requires_multi_user(monkeypatch):
|
|||||||
# ---------- the cascade ----------
|
# ---------- the cascade ----------
|
||||||
|
|
||||||
def test_deletes_the_whole_data_graph(db):
|
def test_deletes_the_whole_data_graph(db):
|
||||||
"""One DELETE has to take the adventure, its actions and memories, the
|
"""One DELETE must remove the adventure, its actions and memories, the
|
||||||
story cards and the settings row with it — nothing is loaded into Python,
|
story cards, and the settings row. Nothing is loaded into Python, so if
|
||||||
so if the FK cascade isn't reaching, rows are silently orphaned (or the
|
the FK cascade does not reach a table, its rows are silently orphaned,
|
||||||
statement errors) rather than tidied."""
|
or the statement fails, instead of being removed."""
|
||||||
user = make_user(db, days_idle=30)
|
user = make_user(db, days_idle=30)
|
||||||
scenario = models.Scenario(user_id=user.id, title="S")
|
scenario = models.Scenario(user_id=user.id, title="S")
|
||||||
db.add(scenario)
|
db.add(scenario)
|
||||||
|
|||||||
@@ -1,17 +1,18 @@
|
|||||||
"""The context builder reads a window of the story, not all of it.
|
"""The context builder reads a window of the story, not all of it.
|
||||||
|
|
||||||
Walking `adventure.actions` every turn made a turn cost O(story length), so a
|
Walking `adventure.actions` every turn made the turn cost O(story length).
|
||||||
long adventure read hundreds of KB to use the tail of it — and the cost grew
|
A long adventure read hundreds of KB to use only the tail of it, and the
|
||||||
with every turn played. `app.context.history` serves tails, slices and counts
|
cost grew with every turn played. `app.context.history` serves tails,
|
||||||
from SQL instead.
|
slices, and counts from SQL instead.
|
||||||
|
|
||||||
Two things have to hold, and both are easy to break by accident:
|
Two things must hold, and both are easy to break by accident:
|
||||||
|
|
||||||
* the window must produce **exactly** the prompt the full story produced, or
|
* The window must produce exactly the prompt the full story produced.
|
||||||
this is a behaviour change wearing an optimization's clothes;
|
Otherwise, the change alters behavior even though it looks like a pure
|
||||||
* the helpers must agree with the old list arithmetic, because memorybank's
|
optimization.
|
||||||
cursors are *positions* in that list and a cursor off by one silently
|
* The helpers must agree with the old list arithmetic, because
|
||||||
summarizes the wrong actions.
|
memorybank's cursors are positions in that list. A cursor off by one
|
||||||
|
silently summarizes the wrong actions.
|
||||||
|
|
||||||
python -m pytest tests/test_history_window.py -v
|
python -m pytest tests/test_history_window.py -v
|
||||||
"""
|
"""
|
||||||
@@ -92,16 +93,16 @@ def story():
|
|||||||
|
|
||||||
|
|
||||||
def full_window(adventure, budget_tokens, token_counter, exclude_action_id=None):
|
def full_window(adventure, budget_tokens, token_counter, exclude_action_id=None):
|
||||||
"""Stand-in for window_covering that hands back the entire story, i.e. the
|
"""Stand-in for `window_covering` that returns the entire story. This is
|
||||||
behaviour this module replaced."""
|
the behavior this module replaced."""
|
||||||
return history.story_actions(adventure, exclude_action_id)
|
return history.story_actions(adventure, exclude_action_id)
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture()
|
@pytest.fixture()
|
||||||
def actions_loaded():
|
def actions_loaded():
|
||||||
"""Counts Action rows the ORM materializes, i.e. how much of the story was
|
"""Counts the `Action` rows the ORM materializes, which shows how much of
|
||||||
actually fetched. rowcount is meaningless for SELECT on SQLite, so count
|
the story was actually fetched. `rowcount` is meaningless for a SELECT
|
||||||
the objects the mapper builds instead."""
|
on SQLite, so this counts the objects the mapper builds instead."""
|
||||||
loaded = {"n": 0}
|
loaded = {"n": 0}
|
||||||
|
|
||||||
def on_load(target, context):
|
def on_load(target, context):
|
||||||
@@ -133,7 +134,7 @@ def test_window_builds_the_same_prompt_as_the_whole_story(story, budget, monkeyp
|
|||||||
|
|
||||||
|
|
||||||
def test_window_matches_on_the_retry_shape(story, monkeypatch):
|
def test_window_matches_on_the_retry_shape(story, monkeypatch):
|
||||||
"""Retry excludes the action being regenerated; the exclusion has to reach
|
"""Retry excludes the action being regenerated. The exclusion must reach
|
||||||
the window query, not just the in-memory filter."""
|
the window query, not just the in-memory filter."""
|
||||||
db, adventure, settings = story
|
db, adventure, settings = story
|
||||||
last = history.tail(adventure, 1)[0]
|
last = history.tail(adventure, 1)[0]
|
||||||
@@ -148,7 +149,8 @@ def test_window_matches_on_the_retry_shape(story, monkeypatch):
|
|||||||
|
|
||||||
|
|
||||||
def test_reported_total_is_the_whole_story_not_the_window(story):
|
def test_reported_total_is_the_whole_story_not_the_window(story):
|
||||||
"""Insights says "N of M actions included"; M must not become the window."""
|
"""Insights reports "N of M actions included." M must not become the
|
||||||
|
window size."""
|
||||||
db, adventure, settings = story
|
db, adventure, settings = story
|
||||||
settings.context_token_budget = 4096
|
settings.context_token_budget = 4096
|
||||||
report = builder.build_context(adventure, settings)[2]
|
report = builder.build_context(adventure, settings)[2]
|
||||||
@@ -160,7 +162,7 @@ def test_reported_total_is_the_whole_story_not_the_window(story):
|
|||||||
|
|
||||||
def test_building_context_reads_far_less_than_the_whole_story(story, actions_loaded):
|
def test_building_context_reads_far_less_than_the_whole_story(story, actions_loaded):
|
||||||
db, adventure, settings = story
|
db, adventure, settings = story
|
||||||
# Expire first: expiring afterwards would discard the unflushed change and
|
# Expire first. Expiring afterward would discard the unflushed change and
|
||||||
# silently put the budget back to its default.
|
# silently put the budget back to its default.
|
||||||
db.expire_all()
|
db.expire_all()
|
||||||
# Small enough that the budget, not the length of the story, decides.
|
# Small enough that the budget, not the length of the story, decides.
|
||||||
@@ -171,9 +173,10 @@ def test_building_context_reads_far_less_than_the_whole_story(story, actions_loa
|
|||||||
included = report["history"]["included"]
|
included = report["history"]["included"]
|
||||||
|
|
||||||
assert included < ACTION_COUNT, "fixture is too short to prove anything"
|
assert included < ACTION_COUNT, "fixture is too short to prove anything"
|
||||||
# The window aims a margin past the budget and re-asks if it fell short, so
|
# The window targets a margin past the budget and requests more if it
|
||||||
# it reads somewhat more than it includes. What matters is that the read is
|
# falls short, so it reads somewhat more than it includes. What matters
|
||||||
# a function of the token budget, not of how long the story has got.
|
# is that the read depends on the token budget, not on the length of
|
||||||
|
# the story.
|
||||||
assert actions_loaded["n"] < ACTION_COUNT // 2, (
|
assert actions_loaded["n"] < ACTION_COUNT // 2, (
|
||||||
f"read {actions_loaded['n']} action rows out of {ACTION_COUNT} to "
|
f"read {actions_loaded['n']} action rows out of {ACTION_COUNT} to "
|
||||||
f"include {included} — the window is not bounding the read"
|
f"include {included} — the window is not bounding the read"
|
||||||
@@ -213,13 +216,14 @@ def test_helpers_agree_with_the_full_list(story):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_depth_boundary_survives_a_middle_action_being_deleted(story):
|
def test_a_depth_boundary_survives_a_middle_action_being_deleted(story):
|
||||||
"""The case that has broken the cursors twice before, and the reason they
|
"""The case that broke the cursors twice before. This is the reason the
|
||||||
are depths now.
|
cursors are depths now.
|
||||||
|
|
||||||
A *position* answers "how much story is past this point?" by counting from
|
A position answers "how much story is past this point?" by counting
|
||||||
the start, so deleting anything in front of the mark changes which action
|
from the start. Deleting anything in front of the mark changes which
|
||||||
the mark names. A depth names the same node either way — the only thing
|
action the mark names. A depth names the same node either way. The
|
||||||
that changes is the count of what comes after, which is what did change.
|
only thing that changes is the count of what comes after, and that
|
||||||
|
count is the one thing that should change here.
|
||||||
"""
|
"""
|
||||||
db, adventure, settings = story
|
db, adventure, settings = story
|
||||||
actions = history.story_actions(adventure)
|
actions = history.story_actions(adventure)
|
||||||
@@ -236,9 +240,9 @@ def test_a_depth_boundary_survives_a_middle_action_being_deleted(story):
|
|||||||
assert history.count_after(adventure, mark) == before, "the mark moved"
|
assert history.count_after(adventure, mark) == before, "the mark moved"
|
||||||
assert [a.id for a in history.after(adventure, mark, 3)] == next_three
|
assert [a.id for a in history.after(adventure, mark, 3)] == next_three
|
||||||
|
|
||||||
# ...and deleting something *after* it is the one thing that does change
|
# Deleting something after the mark is the one change that does affect
|
||||||
# the count, because that is a fact about the story rather than about the
|
# the count, because that count reflects the story, not the coordinate
|
||||||
# coordinate system.
|
# system.
|
||||||
db.delete(history.after(adventure, mark, 1)[0])
|
db.delete(history.after(adventure, mark, 1)[0])
|
||||||
db.commit()
|
db.commit()
|
||||||
db.expire(adventure)
|
db.expire(adventure)
|
||||||
@@ -258,7 +262,7 @@ def test_blank_actions_are_excluded_the_same_way_in_sql_and_python(story):
|
|||||||
# SQL path (relationship not loaded)
|
# SQL path (relationship not loaded)
|
||||||
from_sql = history.count(adventure)
|
from_sql = history.count(adventure)
|
||||||
# Python path (relationship loaded)
|
# Python path (relationship loaded)
|
||||||
adventure.actions # noqa: B018 — force the collection into memory
|
adventure.actions # noqa: B018 - force the collection into memory
|
||||||
from_python = history.count(adventure)
|
from_python = history.count(adventure)
|
||||||
|
|
||||||
assert from_sql == from_python == ACTION_COUNT
|
assert from_sql == from_python == ACTION_COUNT
|
||||||
|
|||||||
@@ -1,17 +1,17 @@
|
|||||||
"""The turn prompt asks for a turn that fits inside `max_output_tokens`.
|
"""The turn prompt asks for a turn that fits inside `max_output_tokens`.
|
||||||
|
|
||||||
`max_output_tokens` is a hard wall the endpoint enforces mid-sentence. The state
|
`max_output_tokens` is a hard limit the endpoint enforces mid-sentence. The
|
||||||
block is emitted *after* the narration, so a long turn hits the wall partway
|
state block is emitted after the narration, so a long turn hits the limit
|
||||||
through the block and the deltas are lost — silently, since nothing reads
|
partway through the block, and the deltas are lost. Nothing reads
|
||||||
`finish_reason`. The prompt now carries a word budget derived from the cap so
|
`finish_reason`, so this loss happens silently. The prompt now carries a
|
||||||
the model lands just inside it.
|
word budget derived from the cap, so the model lands just inside it.
|
||||||
|
|
||||||
Two things are easy to break here:
|
Two things are easy to break here:
|
||||||
|
|
||||||
* the hint must be stated in **words**, not tokens — a model cannot count its
|
* The hint must be stated in words, not tokens. A model cannot count its
|
||||||
own tokens, and a hint it cannot follow is just wasted budget;
|
own tokens, and a hint it cannot follow is wasted budget.
|
||||||
* it must not displace `EMIT_REMINDER` from the last position, which is the
|
* The hint must not displace `EMIT_REMINDER` from the last position, which
|
||||||
whole mechanism keeping the state block emitted at all (see
|
is the whole mechanism that keeps the state block emitted at all (see
|
||||||
test_worldstate.py and the emit-reliability fix).
|
test_worldstate.py and the emit-reliability fix).
|
||||||
|
|
||||||
python -m pytest tests/test_length_hint.py -v
|
python -m pytest tests/test_length_hint.py -v
|
||||||
@@ -100,7 +100,7 @@ def asked_words(cap):
|
|||||||
|
|
||||||
def test_buffer_leaves_room_for_overshoot():
|
def test_buffer_leaves_room_for_overshoot():
|
||||||
"""The stated number must sit meaningfully under the real ceiling, or an
|
"""The stated number must sit meaningfully under the real ceiling, or an
|
||||||
on-target-but-slightly-long turn still hits the wall."""
|
on-target-but-slightly-long turn still hits the limit."""
|
||||||
for cap in (400, 800, 1500, 2400):
|
for cap in (400, 800, 1500, 2400):
|
||||||
asked = asked_words(cap)
|
asked = asked_words(cap)
|
||||||
ceiling = (cap - builder.LENGTH_HEADROOM) * builder.WORDS_PER_TOKEN
|
ceiling = (cap - builder.LENGTH_HEADROOM) * builder.WORDS_PER_TOKEN
|
||||||
@@ -109,9 +109,10 @@ def test_buffer_leaves_room_for_overshoot():
|
|||||||
|
|
||||||
|
|
||||||
def test_hint_is_phrased_as_a_ceiling_not_a_budget():
|
def test_hint_is_phrased_as_a_ceiling_not_a_budget():
|
||||||
"""Measured: budget phrasing ("keep this turn under about N words") reads as a
|
"""Measured: budget phrasing ("keep this turn under about N words")
|
||||||
target to fill and moved the mean turn from 174 to 246 words — toward the wall
|
reads as a target to fill. It moved the mean turn from 174 to 246
|
||||||
it exists to avoid. The limit framing must survive future prompt edits."""
|
words, toward the limit it exists to avoid. The limit framing must
|
||||||
|
survive future prompt edits."""
|
||||||
hint = builder.length_hint(800, has_ws=True)
|
hint = builder.length_hint(800, has_ws=True)
|
||||||
assert "must not exceed" in hint
|
assert "must not exceed" in hint
|
||||||
assert "under about" not in hint
|
assert "under about" not in hint
|
||||||
@@ -119,15 +120,15 @@ def test_hint_is_phrased_as_a_ceiling_not_a_budget():
|
|||||||
|
|
||||||
|
|
||||||
def test_hint_states_a_floor_as_well_as_a_ceiling():
|
def test_hint_states_a_floor_as_well_as_a_ceiling():
|
||||||
"""A ceiling alone is one-sided: a terse model has nothing to act on but the
|
"""A ceiling alone is one-sided: a terse model has nothing to act on but
|
||||||
"only as much as the moment needs" clause and collapses to two paragraphs.
|
the "only as much as the moment needs" clause and produces only two
|
||||||
The floor is what makes the same prompt land in the same place across models
|
paragraphs. The floor is what makes the same prompt produce a similar
|
||||||
that lean opposite ways."""
|
length across models with different tendencies."""
|
||||||
hint = builder.length_hint(800, has_ws=True)
|
hint = builder.length_hint(800, has_ws=True)
|
||||||
assert "506" in hint and "177" in hint
|
assert "506" in hint and "177" in hint
|
||||||
assert "should not stop short of" in hint
|
assert "should not stop short of" in hint
|
||||||
# Asymmetric on purpose: the wall is a wall, the floor is a floor, and neither
|
# Asymmetric on purpose: the ceiling is a hard limit and the floor is a
|
||||||
# is phrased as a number to hit.
|
# soft target, and neither is phrased as a specific number to reach.
|
||||||
assert hint.index("must not exceed") < hint.index("should not stop short of")
|
assert hint.index("must not exceed") < hint.index("should not stop short of")
|
||||||
|
|
||||||
|
|
||||||
@@ -139,9 +140,10 @@ def test_floor_stays_well_under_the_ceiling():
|
|||||||
|
|
||||||
|
|
||||||
def test_floor_is_dropped_when_the_cap_is_too_tight_for_one():
|
def test_floor_is_dropped_when_the_cap_is_too_tight_for_one():
|
||||||
"""At a tight cap a short turn is the correct turn, and the tight-cap wording
|
"""At a tight cap a short turn is the correct turn. The tight-cap
|
||||||
is the one measured to keep the state block alive (0/6 truncations at cap 250
|
wording is the one measured to keep the state block from being
|
||||||
against 2/6 unhinted) — so it is left exactly as it was."""
|
truncated (0/6 truncations at cap 250, against 2/6 unhinted), so it
|
||||||
|
is left exactly as it was."""
|
||||||
hint = builder.length_hint(250, has_ws=True)
|
hint = builder.length_hint(250, has_ws=True)
|
||||||
assert "should not stop short of" not in hint
|
assert "should not stop short of" not in hint
|
||||||
assert "much shorter" in hint
|
assert "much shorter" in hint
|
||||||
@@ -199,13 +201,15 @@ def test_emit_reminder_keeps_the_last_word(story):
|
|||||||
|
|
||||||
|
|
||||||
def test_prompt_stays_inside_the_budget_on_a_long_story(story):
|
def test_prompt_stays_inside_the_budget_on_a_long_story(story):
|
||||||
"""Regression guard: the hint is appended after history has already spent
|
"""Regression guard: the hint is appended after history has already
|
||||||
the budget, so it must be reserved up front like EMIT_REMINDER is.
|
spent the budget, so it must be reserved up front like `EMIT_REMINDER`
|
||||||
|
is.
|
||||||
|
|
||||||
Weak on purpose — the history loop stops *before* crossing its budget, so it
|
This check is weak on purpose. The history loop stops before crossing
|
||||||
leaves about one action of slack and the ~30-token hint hides inside it.
|
its budget, so it leaves about one action of slack, and the roughly
|
||||||
This catches a hint that grows large, not a missing reservation; the
|
30-token hint fits inside that slack. This catches a hint that grows
|
||||||
reservation itself is not observable from the outside."""
|
large, not a missing reservation. The reservation itself is not
|
||||||
|
observable from the outside."""
|
||||||
db, adventure, settings, _ = story
|
db, adventure, settings, _ = story
|
||||||
for i in range(4, 120):
|
for i in range(4, 120):
|
||||||
db.add(models.Action(
|
db.add(models.Action(
|
||||||
@@ -225,8 +229,9 @@ def test_prompt_stays_inside_the_budget_on_a_long_story(story):
|
|||||||
|
|
||||||
|
|
||||||
def test_hint_is_counted_in_the_reported_totals(story):
|
def test_hint_is_counted_in_the_reported_totals(story):
|
||||||
"""Insights reports what the turn actually costs; a section that reaches the
|
"""Insights reports what the turn actually costs. A section that
|
||||||
model but not the accounting makes that number a lie."""
|
reaches the model but not the accounting makes that reported cost
|
||||||
|
inaccurate."""
|
||||||
db, adventure, settings, _ = story
|
db, adventure, settings, _ = story
|
||||||
settings.max_output_tokens = 800
|
settings.max_output_tokens = 800
|
||||||
|
|
||||||
|
|||||||
@@ -1,20 +1,21 @@
|
|||||||
"""Phase 14 SP3 — memories hang off nodes, and the marks are nodes too.
|
"""Phase 14 SP3: memories attach to nodes, and the marks are nodes too.
|
||||||
|
|
||||||
Two claims, and neither of them fails loudly if it is wrong:
|
Two claims, and neither fails loudly if it is wrong:
|
||||||
|
|
||||||
* **A memory belongs to the path that produced it.** A memory made on branch B
|
* A memory belongs to the path that produced it. A memory made on branch B
|
||||||
must be invisible from A, and the memories of a shared ancestor must be
|
must be invisible from A, and the memories of a shared ancestor must be
|
||||||
visible from both — without anything being copied when a fork happens. The
|
visible from both, without anything being copied when a fork happens. The
|
||||||
failure mode is a prompt quietly carrying a summary of a story the player
|
failure mode is a prompt that quietly carries a summary of a story the
|
||||||
abandoned.
|
player abandoned.
|
||||||
* **Retrieval reads the *whole* lineage, and that stays affordable.** The story
|
* Retrieval reads the whole lineage, and that stays affordable. The story
|
||||||
is read through a window, but recall is long-range by definition and cannot
|
is read through a window, but recall is long-range by definition and
|
||||||
be — so the clause names every ancestor, and the bet is that memories are
|
cannot use one. So the clause names every ancestor. The bet is that
|
||||||
sparse enough (one per six actions) for that to be tens of small rows even
|
memories are sparse enough, one per six actions, for that to stay tens of
|
||||||
twenty forks deep. Measured below rather than asserted.
|
small rows even twenty forks deep. This file measures that bet below
|
||||||
|
rather than asserting it.
|
||||||
|
|
||||||
Nothing in the product forks yet, so the fork is built by hand, exactly as
|
Nothing in the product forks yet, so this file builds the fork by hand,
|
||||||
`test_branch_clause.py` builds it.
|
exactly as `test_branch_clause.py` builds it.
|
||||||
|
|
||||||
python -m pytest tests/test_memory_nodes.py -v
|
python -m pytest tests/test_memory_nodes.py -v
|
||||||
"""
|
"""
|
||||||
@@ -50,9 +51,10 @@ class StubEmbedder:
|
|||||||
# --------------------------------------------------------------- the fixture
|
# --------------------------------------------------------------- the fixture
|
||||||
|
|
||||||
def make_branch(db, adventure, parent=None, fork_depth=None):
|
def make_branch(db, adventure, parent=None, fork_depth=None):
|
||||||
"""A branch row whose lineage is its parent's, capped, plus itself — the
|
"""A branch row whose lineage is its parent's lineage, capped, plus
|
||||||
computation SP5 will do at fork time, written out so the fixture cannot
|
itself. This is the computation SP5 performs at fork time. The fixture
|
||||||
pass by agreeing with a bug in the code under test."""
|
reimplements it here so it cannot pass by agreeing with a bug in the
|
||||||
|
code under test."""
|
||||||
branch = models.Branch(
|
branch = models.Branch(
|
||||||
adventure_id=adventure.id,
|
adventure_id=adventure.id,
|
||||||
parent_branch_id=parent.id if parent else None,
|
parent_branch_id=parent.id if parent else None,
|
||||||
@@ -106,12 +108,13 @@ def add_memory(db, adventure, text, node, vector=(1.0, 0.0, 0.0), **kwargs):
|
|||||||
|
|
||||||
@pytest.fixture()
|
@pytest.fixture()
|
||||||
def forked():
|
def forked():
|
||||||
"""A0..A3, then B4 B5 off A3, then C6 C7 off B5 — with a memory hung off
|
"""A0..A3, then B4 B5 off A3, then C6 C7 off B5, with a memory attached
|
||||||
one node of each branch, and A playing on past the fork it was left at.
|
to one node of each branch. A keeps playing past the fork point where B
|
||||||
|
left it.
|
||||||
|
|
||||||
The head is C, so the story is A0 A1 A2 A3 B4 B5 C6 C7 and the memories in
|
The head is C, so the story is A0 A1 A2 A3 B4 B5 C6 C7, and the
|
||||||
play are A's and B's and C's — but not the one on A5, which is on a sibling
|
memories in play are A's, B's, and C's. The one on A5 is excluded: it
|
||||||
of B4 and belongs to a story nobody is reading.
|
is on a sibling of B4 and belongs to a story nobody is reading.
|
||||||
"""
|
"""
|
||||||
Base.metadata.create_all(bind=engine)
|
Base.metadata.create_all(bind=engine)
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
@@ -181,8 +184,9 @@ def retrieved(adventure, settings) -> set[str]:
|
|||||||
# ------------------------------------------------------------- the isolation
|
# ------------------------------------------------------------- the isolation
|
||||||
|
|
||||||
def test_a_memory_on_a_sibling_is_not_retrieved(forked):
|
def test_a_memory_on_a_sibling_is_not_retrieved(forked):
|
||||||
"""The whole point. A5 is a node of the story that was abandoned when B
|
"""The whole point of this file. A5 is a node of the story that was
|
||||||
forked, and the memory hanging off it must not reach a prompt on C."""
|
abandoned when B forked, and the memory attached to it must not reach a
|
||||||
|
prompt on C."""
|
||||||
db, adventure, settings, ids = forked
|
db, adventure, settings, ids = forked
|
||||||
assert retrieved(adventure, settings) == {
|
assert retrieved(adventure, settings) == {
|
||||||
"on the shared trunk", "on B", "on C"
|
"on the shared trunk", "on B", "on C"
|
||||||
@@ -196,30 +200,30 @@ def test_a_shared_ancestor_is_visible_from_both_branches(forked):
|
|||||||
switch_to(db, adventure, ids["a"], 5)
|
switch_to(db, adventure, ids["a"], 5)
|
||||||
from_a = retrieved(adventure, settings)
|
from_a = retrieved(adventure, settings)
|
||||||
assert "on the shared trunk" in from_a
|
assert "on the shared trunk" in from_a
|
||||||
# ...and from A, the branches taken off it are the ones out of reach.
|
# From A, the branches taken off it are the ones out of reach.
|
||||||
assert from_a == {"on the shared trunk", "on A's own continuation"}
|
assert from_a == {"on the shared trunk", "on A's own continuation"}
|
||||||
|
|
||||||
|
|
||||||
def test_the_lineage_is_read_whole_not_windowed(forked):
|
def test_the_lineage_is_read_whole_not_windowed(forked):
|
||||||
"""The story is read through a window; recall is not. The trunk memory is
|
"""The story is read through a window. Recall is not. The trunk memory
|
||||||
four nodes and two forks back, and is still a candidate."""
|
is four nodes and two forks back, and is still a candidate."""
|
||||||
db, adventure, settings, ids = forked
|
db, adventure, settings, ids = forked
|
||||||
path = lineage.path_of(db, adventure)
|
path = lineage.path_of(db, adventure)
|
||||||
assert len(path) == 3
|
assert len(path) == 3
|
||||||
# The window a *story* read would use here names one entry. Retrieval names
|
# The window a story read would use here names one entry. Retrieval
|
||||||
# all three, which is the difference this test exists to pin.
|
# names all three, which is the difference this test checks.
|
||||||
assert path.prefix_covering(2) == 1
|
assert path.prefix_covering(2) == 1
|
||||||
assert "on the shared trunk" in retrieved(adventure, settings)
|
assert "on the shared trunk" in retrieved(adventure, settings)
|
||||||
|
|
||||||
|
|
||||||
def test_a_hand_written_memory_is_anchored_where_it_was_typed(forked):
|
def test_a_hand_written_memory_is_anchored_where_it_was_typed(forked):
|
||||||
"""SP7: a typed memory takes the head, so it obeys the same rule as a
|
"""SP7: a typed memory takes the head, so it obeys the same rule as a
|
||||||
summarised one.
|
summarized one.
|
||||||
|
|
||||||
It used to carry no depth, which sounded like "belongs to the whole
|
It used to carry no depth, which sounded like "belongs to the whole
|
||||||
adventure" and behaved like "cannot be capped at a fork" — it followed the
|
adventure" and behaved like "cannot be capped at a fork." It followed
|
||||||
reader onto branches whose story it never described. Anchoring it makes the
|
the reader onto branches whose story it never described. Anchoring it
|
||||||
bank answer one question rather than two.
|
makes the bank answer one question instead of two.
|
||||||
"""
|
"""
|
||||||
db, adventure, settings, ids = forked
|
db, adventure, settings, ids = forked
|
||||||
switch_to(db, adventure, ids["a"], 5)
|
switch_to(db, adventure, ids["a"], 5)
|
||||||
@@ -228,14 +232,14 @@ def test_a_hand_written_memory_is_anchored_where_it_was_typed(forked):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_typed_memory_survives_a_fork_of_the_ground_it_was_typed_on(forked):
|
def test_a_typed_memory_survives_a_fork_of_the_ground_it_was_typed_on(forked):
|
||||||
"""The half of the old behaviour that was right, kept.
|
"""The half of the old behavior that was correct, kept.
|
||||||
|
|
||||||
Typed on the shared trunk it is still there after forking away — but
|
A memory typed on the shared trunk is still there after forking away,
|
||||||
because the fork's path goes through that node, not because the memory was
|
but only because the fork's path goes through that node, not because
|
||||||
exempt from being capped.
|
the memory is exempt from being capped.
|
||||||
"""
|
"""
|
||||||
db, adventure, settings, ids = forked
|
db, adventure, settings, ids = forked
|
||||||
switch_to(db, adventure, ids["a"], 3) # the trunk B, and so C, branch from
|
switch_to(db, adventure, ids["a"], 3) # the node B, and so C, forked from
|
||||||
add_memory(db, adventure, "typed on the trunk", None)
|
add_memory(db, adventure, "typed on the trunk", None)
|
||||||
|
|
||||||
switch_to(db, adventure, ids["c"], 7)
|
switch_to(db, adventure, ids["c"], 7)
|
||||||
@@ -243,12 +247,12 @@ def test_a_typed_memory_survives_a_fork_of_the_ground_it_was_typed_on(forked):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_typed_memory_does_not_follow_you_onto_a_path_it_is_not_on(forked):
|
def test_a_typed_memory_does_not_follow_you_onto_a_path_it_is_not_on(forked):
|
||||||
"""And the half that was wrong, fixed.
|
"""The other half of the old behavior, which was wrong, is now fixed.
|
||||||
|
|
||||||
A5 is A's own continuation past the point B left it, so it is a sibling of
|
A5 is A's own continuation past the point where B left it, so it is a
|
||||||
the story C tells — precisely where the `sibling` memory sits, and excluded
|
sibling of the story C tells. This is exactly where the `sibling`
|
||||||
for precisely the same reason. Typing rather than summarising buys no
|
memory sits, and it is excluded for the same reason. Typing a memory
|
||||||
exemption from the path.
|
instead of summarizing it grants no exemption from the path rule.
|
||||||
"""
|
"""
|
||||||
db, adventure, settings, ids = forked
|
db, adventure, settings, ids = forked
|
||||||
switch_to(db, adventure, ids["a"], 5)
|
switch_to(db, adventure, ids["a"], 5)
|
||||||
@@ -271,10 +275,10 @@ def test_a_mark_moves_to_the_node_the_memory_covers(forked):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_mark_from_a_sibling_reads_as_nothing_covered(forked):
|
def test_a_mark_from_a_sibling_reads_as_nothing_covered(forked):
|
||||||
"""A mark is a node, so moving to another story has to be answered rather
|
"""A mark is a node, so switching to another story must resolve the
|
||||||
than assumed. Ground this path never travelled is not covered ground, and
|
mark's meaning rather than assume it. A path segment this story never
|
||||||
the fallback for 'I don't know' has to be redoing the work, not skipping
|
took is not covered, and the fallback for "not covered" must be redoing
|
||||||
it."""
|
the work, not skipping it."""
|
||||||
db, adventure, settings, ids = forked
|
db, adventure, settings, ids = forked
|
||||||
cursors.MEMORY.anchor_at(adventure, ids["nodes"]["C7"])
|
cursors.MEMORY.anchor_at(adventure, ids["nodes"]["C7"])
|
||||||
db.commit()
|
db.commit()
|
||||||
@@ -301,9 +305,10 @@ def test_a_mark_never_moves_forward_on_a_rewind(forked):
|
|||||||
# ---------------------------------------------------- what the passes read
|
# ---------------------------------------------------- what the passes read
|
||||||
|
|
||||||
def test_the_summary_folds_in_only_the_path_it_is_on(forked, monkeypatch):
|
def test_the_summary_folds_in_only_the_path_it_is_on(forked, monkeypatch):
|
||||||
"""`_update_story_summary` gathers the memories past its mark. On C that is
|
"""`_update_story_summary` gathers the memories past its mark. On C
|
||||||
B's and C's — never the one on A's own continuation, whose depth would
|
that is B's and C's memories. It never includes the one on A's own
|
||||||
otherwise put it squarely inside the range."""
|
continuation, even though that memory's depth would otherwise put it
|
||||||
|
inside the range."""
|
||||||
db, adventure, settings, ids = forked
|
db, adventure, settings, ids = forked
|
||||||
monkeypatch.setattr(memorybank, "SUMMARY_INTERVAL", 1)
|
monkeypatch.setattr(memorybank, "SUMMARY_INTERVAL", 1)
|
||||||
|
|
||||||
@@ -360,7 +365,7 @@ def test_a_block_is_summarized_from_the_path_and_hung_off_its_last_node(
|
|||||||
assert ["B4", "B5", "C6", "C7"] == [line for line in second.split() if line[0] in "ABC"]
|
assert ["B4", "B5", "C6", "C7"] == [line for line in second.split() if line[0] in "ABC"]
|
||||||
made = db.query(models.Memory).filter_by(text="Memory 1.").one()
|
made = db.query(models.Memory).filter_by(text="Memory 1.").one()
|
||||||
assert (made.branch_id, made.depth) == (ids["a"], 3)
|
assert (made.branch_id, made.depth) == (ids["a"], 3)
|
||||||
# The mark ends up on the node the *second* block hangs off — the tip.
|
# The mark ends up on the node the second block attaches to, which is the tip.
|
||||||
assert cursors.MEMORY.stored(adventure) == (ids["c"], 7)
|
assert cursors.MEMORY.stored(adventure) == (ids["c"], 7)
|
||||||
|
|
||||||
|
|
||||||
@@ -368,8 +373,8 @@ def test_a_block_is_summarized_from_the_path_and_hung_off_its_last_node(
|
|||||||
|
|
||||||
@pytest.fixture()
|
@pytest.fixture()
|
||||||
def deeply_forked():
|
def deeply_forked():
|
||||||
"""A story forked twenty times, with a memory every six actions — the
|
"""A story forked twenty times, with a memory every six actions. This
|
||||||
density the post-turn pass actually produces."""
|
is the density the post-turn pass actually produces."""
|
||||||
Base.metadata.create_all(bind=engine)
|
Base.metadata.create_all(bind=engine)
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
user = models.User(is_guest=False, email="deepmem@example.com")
|
user = models.User(is_guest=False, email="deepmem@example.com")
|
||||||
@@ -423,10 +428,11 @@ def deeply_forked():
|
|||||||
|
|
||||||
|
|
||||||
def test_retrieving_from_a_deep_fork_costs_what_a_flat_story_costs(deeply_forked):
|
def test_retrieving_from_a_deep_fork_costs_what_a_flat_story_costs(deeply_forked):
|
||||||
"""The bet, in bytes. Retrieval names all twenty-two branches instead of
|
"""The bet from the module docstring, measured in bytes. Retrieval
|
||||||
one — but it is fetching an id and a flag per memory, and there are the
|
names all twenty-two branches instead of one, but it fetches only an id
|
||||||
same fourteen either way, so the clause is where the difference is and the
|
and a flag per memory, and both stories return the same fourteen
|
||||||
clause is not what crosses the wire."""
|
memories. The clause is where the difference shows up, and the clause
|
||||||
|
is not what crosses the wire."""
|
||||||
db, flat_story, forked_story = deeply_forked
|
db, flat_story, forked_story = deeply_forked
|
||||||
settings = db.query(models.Settings).one()
|
settings = db.query(models.Settings).one()
|
||||||
flat_id, forked_id = flat_story.id, forked_story.id
|
flat_id, forked_id = flat_story.id, forked_story.id
|
||||||
@@ -445,8 +451,8 @@ def test_retrieving_from_a_deep_fork_costs_what_a_flat_story_costs(deeply_forked
|
|||||||
finally:
|
finally:
|
||||||
meter.detach()
|
meter.detach()
|
||||||
|
|
||||||
# Measured 2026-08-18: 1,807 B against 1,823 B — the same fourteen rows,
|
# Measured 2026-08-18: 1,807 B against 1,823 B. Both figures cover the
|
||||||
# named through twenty-two branch terms instead of one.
|
# same fourteen rows, named through twenty-two branch terms instead of one.
|
||||||
assert flat_bytes > 0, "the meter saw nothing; it is measuring the wrong connection"
|
assert flat_bytes > 0, "the meter saw nothing; it is measuring the wrong connection"
|
||||||
assert forked_bytes < flat_bytes * 1.5, (
|
assert forked_bytes < flat_bytes * 1.5, (
|
||||||
f"retrieval on a 20-fork story cost {forked_bytes:,} B against the "
|
f"retrieval on a 20-fork story cost {forked_bytes:,} B against the "
|
||||||
@@ -457,17 +463,19 @@ def test_retrieving_from_a_deep_fork_costs_what_a_flat_story_costs(deeply_forked
|
|||||||
# ------------------------------------------------------- the opening node
|
# ------------------------------------------------------- the opening node
|
||||||
|
|
||||||
def test_a_typed_memory_on_the_opening_node_survives_that_node_going(forked):
|
def test_a_typed_memory_on_the_opening_node_survives_that_node_going(forked):
|
||||||
"""The one place a node and its memories part company.
|
"""The one exception where a node and its memories are not withdrawn
|
||||||
|
together.
|
||||||
|
|
||||||
A memory anchored to a node is withdrawn with the node, which is the rule
|
A memory anchored to a node is withdrawn with the node. This is the
|
||||||
and is deliberate: it described that turn, and the turn is leaving. But
|
rule, and it is deliberate: the memory described that turn, and the
|
||||||
migration 62 parked *every* memory written before memories had coordinates
|
turn is leaving. But migration 62 parked every memory written before
|
||||||
on depth 0 — the only landing spot visible from every branch — so the
|
memories had coordinates on depth 0, the only landing spot visible from
|
||||||
opening node carries a whole bank it never produced. Withdrawing it would
|
every branch. As a result, the opening node carries a whole bank of
|
||||||
retire all of that in one click, for every adventure predating the tree.
|
memories it never produced. Withdrawing it would delete all of those
|
||||||
|
memories at once, for every adventure that predates the tree.
|
||||||
|
|
||||||
A memory with no `source_start` covers no stretch of story, so nothing about
|
A memory with no `source_start` covers no stretch of story, so nothing
|
||||||
it can go stale. It stays.
|
about it can go stale. It stays.
|
||||||
"""
|
"""
|
||||||
db, adventure, settings, ids = forked
|
db, adventure, settings, ids = forked
|
||||||
typed = models.Memory(
|
typed = models.Memory(
|
||||||
@@ -489,9 +497,10 @@ def test_a_typed_memory_on_the_opening_node_survives_that_node_going(forked):
|
|||||||
def test_a_summary_of_the_opening_node_is_still_withdrawn(forked):
|
def test_a_summary_of_the_opening_node_is_still_withdrawn(forked):
|
||||||
"""The exception is about memories that describe nothing, not about depth 0.
|
"""The exception is about memories that describe nothing, not about depth 0.
|
||||||
|
|
||||||
A summary that genuinely ends on the opening node describes text that is
|
A summary that genuinely ends on the opening node describes text that
|
||||||
going, so it goes too — otherwise the root would collect exactly the
|
is being removed, so the summary is removed too. Otherwise the root
|
||||||
dangling rows `forget_node` replaced `prune_dangling_memories` to prevent.
|
would collect exactly the dangling rows that `forget_node` replaced
|
||||||
|
`prune_dangling_memories` to prevent.
|
||||||
"""
|
"""
|
||||||
db, adventure, settings, ids = forked
|
db, adventure, settings, ids = forked
|
||||||
derived = add_memory(db, adventure, "the opening, summarised", ids["nodes"]["A0"])
|
derived = add_memory(db, adventure, "the opening, summarised", ids["nodes"]["A0"])
|
||||||
|
|||||||
@@ -1,11 +1,12 @@
|
|||||||
"""Ranking the memory bank without reading the memory bank.
|
"""Ranking the memory bank without reading the memory bank.
|
||||||
|
|
||||||
Retrieval used to walk `adventure.memories`, which loaded every row *with its
|
Retrieval used to walk `adventure.memories`, which loaded every row with
|
||||||
vector* — 96% of everything a turn read. It now asks SQL which memories are in
|
its vector. That vector data was 96% of everything a turn read. Retrieval
|
||||||
play, holds their vectors in process, and fetches text for the five it picks.
|
now asks SQL which memories are in play, holds their vectors in process,
|
||||||
|
and fetches text for only the five it picks.
|
||||||
|
|
||||||
Three things have to stay true for that to be safe, and each is a separate
|
Three things have to stay true for that to be safe, and each is a separate
|
||||||
failure that no error message would ever report:
|
failure that no error message would report:
|
||||||
|
|
||||||
* the ranking picks the same memories it always did;
|
* the ranking picks the same memories it always did;
|
||||||
* nothing bulk-reads a vector column again;
|
* nothing bulk-reads a vector column again;
|
||||||
@@ -80,8 +81,8 @@ def adventure(db, settings):
|
|||||||
)
|
)
|
||||||
db.add(adv)
|
db.add(adv)
|
||||||
db.flush()
|
db.flush()
|
||||||
# Retrieval builds its query from the newest actions; with none, it returns
|
# Retrieval builds its query from the newest actions. With none, it
|
||||||
# before ranking anything.
|
# returns before ranking anything.
|
||||||
for i in range(2):
|
for i in range(2):
|
||||||
db.add(models.Action(
|
db.add(models.Action(
|
||||||
adventure_id=adv.id, index=i, type="ai", text=f"Something happened {i}."
|
adventure_id=adv.id, index=i, type="ai", text=f"Something happened {i}."
|
||||||
@@ -189,8 +190,8 @@ def test_update_stats_bumps_only_the_used(db, adventure, settings, bank):
|
|||||||
|
|
||||||
|
|
||||||
def test_dry_runs_do_not_bump_the_counters(db, adventure, settings, bank):
|
def test_dry_runs_do_not_bump_the_counters(db, adventure, settings, bank):
|
||||||
"""Insights assembles a context without spending a turn; it must not look
|
"""Insights assembles a context without spending a turn. It must not
|
||||||
like the memories were used."""
|
look like the memories were used."""
|
||||||
retrieve(adventure, settings, StubEmbedder(), update_stats=False)
|
retrieve(adventure, settings, StubEmbedder(), update_stats=False)
|
||||||
db.commit()
|
db.commit()
|
||||||
db.expire_all()
|
db.expire_all()
|
||||||
@@ -207,18 +208,19 @@ def memory_selects(statements):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_json_column_is_gone(db):
|
def test_the_json_column_is_gone(db):
|
||||||
"""`memories.embedding` held the vectors before migration 38 and nothing
|
"""`memories.embedding` held the vectors before migration 38, and
|
||||||
read it afterwards; migration 42 dropped it. Bringing it back would restore
|
nothing read it afterward. Migration 42 dropped it. Restoring it would
|
||||||
4 MB of dead weight and a second place vectors can be written from — which
|
bring back 4 MB of dead weight and a second place vectors can be
|
||||||
is how the model-switch bug happened (test_embedding_model_switch.py)."""
|
written from. That second place is how the model-switch bug happened
|
||||||
|
(test_embedding_model_switch.py)."""
|
||||||
columns = {c["name"] for c in sa_inspect(engine).get_columns("memories")}
|
columns = {c["name"] for c in sa_inspect(engine).get_columns("memories")}
|
||||||
assert "embedding" not in columns
|
assert "embedding" not in columns
|
||||||
assert {"embedding_blob", "embedded"} <= columns
|
assert {"embedding_blob", "embedded"} <= columns
|
||||||
|
|
||||||
|
|
||||||
def test_the_catalogue_query_carries_no_vectors(db, adventure, settings, bank, sql_log):
|
def test_the_catalogue_query_carries_no_vectors(db, adventure, settings, bank, sql_log):
|
||||||
"""The query that decides *which* memories are in play must stay tiny —
|
"""The query that decides which memories are in play must stay tiny.
|
||||||
this is the one that used to drag the whole bank across."""
|
This is the query that used to pull the whole bank across the wire."""
|
||||||
retrieve(adventure, settings, StubEmbedder())
|
retrieve(adventure, settings, StubEmbedder())
|
||||||
catalogue = [s for s in memory_selects(sql_log) if "memories.pinned" in s]
|
catalogue = [s for s in memory_selects(sql_log) if "memories.pinned" in s]
|
||||||
assert catalogue, "expected a catalogue query"
|
assert catalogue, "expected a catalogue query"
|
||||||
@@ -261,9 +263,10 @@ def test_only_top_k_texts_are_fetched(db, adventure, settings, bank, sql_log):
|
|||||||
# ---------------------------------------------------------------- staleness
|
# ---------------------------------------------------------------- staleness
|
||||||
|
|
||||||
def test_a_rewritten_vector_is_not_served_from_cache(db, adventure, settings, bank):
|
def test_a_rewritten_vector_is_not_served_from_cache(db, adventure, settings, bank):
|
||||||
"""The cache's one genuine hazard: a memory keeps its id while its vector
|
"""The cache's one genuine hazard: a memory keeps its id while its
|
||||||
changes, so an id-set check alone would go on serving the old one. Editing
|
vector changes, so an id-set check alone would continue serving the
|
||||||
a memory's text and re-embedding it does exactly that.
|
old vector. Editing a memory's text and re-embedding it does exactly
|
||||||
|
that.
|
||||||
"""
|
"""
|
||||||
settings.memory_top_k = 1
|
settings.memory_top_k = 1
|
||||||
db.commit()
|
db.commit()
|
||||||
@@ -325,9 +328,9 @@ def test_eviction_marks_the_least_recently_used(db, adventure, settings):
|
|||||||
|
|
||||||
|
|
||||||
def test_eviction_breaks_ties_on_use_count(db, adventure, settings):
|
def test_eviction_breaks_ties_on_use_count(db, adventure, settings):
|
||||||
"""Two memories last wanted at the same moment: the one the story has
|
"""Two memories last used at the same moment: the one the story has
|
||||||
leaned on less goes. Only a tiebreak — ranking on the count first is what
|
used less is the one that goes. This must be only a tiebreak. Ranking
|
||||||
used to freeze the bank (see below)."""
|
on the count first is what used to freeze the bank (see below)."""
|
||||||
settings.memory_bank_capacity = 1
|
settings.memory_bank_capacity = 1
|
||||||
db.commit()
|
db.commit()
|
||||||
now = models.utcnow()
|
now = models.utcnow()
|
||||||
@@ -344,12 +347,13 @@ def test_eviction_breaks_ties_on_use_count(db, adventure, settings):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_newborn_is_not_evicted_by_the_bank_it_joins(db, adventure, settings):
|
def test_a_newborn_is_not_evicted_by_the_bank_it_joins(db, adventure, settings):
|
||||||
"""The bank used to shut itself. Eviction ranked on use_count first, and a
|
"""The bank used to stop accepting new memories. Eviction ranked on
|
||||||
memory written this turn has never been used, so the moment every survivor
|
use_count first, and a memory written this turn has never been used.
|
||||||
had been retrieved even once the newborn was the lowest row in the bank and
|
Once every existing memory had been retrieved even once, the newborn
|
||||||
was retired in the same post-turn run that wrote it — before retrieval ever
|
became the lowest-ranked row in the bank. Eviction then removed it in
|
||||||
saw it. That state is absorbing: counts only go up, so no memory written
|
the same post-turn run that wrote it, before retrieval ever saw it.
|
||||||
after it could ever get in either."""
|
That state never recovers: counts only go up, so no memory written
|
||||||
|
after it could get in either."""
|
||||||
settings.memory_bank_capacity = 3
|
settings.memory_bank_capacity = 3
|
||||||
db.commit()
|
db.commit()
|
||||||
now = models.utcnow()
|
now = models.utcnow()
|
||||||
@@ -370,9 +374,9 @@ def test_a_newborn_is_not_evicted_by_the_bank_it_joins(db, adventure, settings):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_full_bank_still_turns_over(db, adventure, settings):
|
def test_a_full_bank_still_turns_over(db, adventure, settings):
|
||||||
"""The same failure seen over several turns: a bank at capacity has to keep
|
"""The same failure seen over several turns: a bank at capacity must
|
||||||
taking on what the story is doing now, or the adventure stops remembering
|
keep accepting new memories, or the adventure stops remembering
|
||||||
anything past the point it filled up."""
|
anything past the point where it filled up."""
|
||||||
settings.memory_bank_capacity = 3
|
settings.memory_bank_capacity = 3
|
||||||
now = models.utcnow()
|
now = models.utcnow()
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|||||||
@@ -1,22 +1,24 @@
|
|||||||
"""Memories must never describe narration that is no longer in the story, and
|
"""Memories must never describe narration that is no longer in the story, and
|
||||||
must never skip a stretch of it.
|
must never skip a stretch of it.
|
||||||
|
|
||||||
For six phases the answer was a **holdback**: summarization stopped one action
|
For six phases, the answer was a holdback. Summarization stopped one action
|
||||||
short of the newest, because only the last action was retryable and a retry
|
short of the newest, because only the last action was retryable, and a
|
||||||
rewrote `Action.text` under a mark that had already moved past it. SP4 ended
|
retry rewrote `Action.text` under a mark that had already moved past it.
|
||||||
that — a retry writes a sibling node and the coordinate's derived work is
|
SP4 ended that: a retry writes a sibling node, and the coordinate's derived
|
||||||
withdrawn as it does, which is the same repair undo and delete already made.
|
work is withdrawn as it happens, using the same repair that undo and delete
|
||||||
So the holdback is gone, and the first half of this file now asserts the
|
already made. The holdback is gone, so the first half of this file now
|
||||||
property that replaced it: a block forms as soon as there is a block, and
|
asserts the property that replaced it. A block forms as soon as there is a
|
||||||
changing what a coordinate says takes back what was derived from it.
|
block, and changing what a coordinate says takes back what was derived from
|
||||||
|
it.
|
||||||
|
|
||||||
Phase 14 SP3 changed what the mark *is*. It used to be a count of covered story
|
Phase 14 SP3 changed what the mark is. It used to be a count of covered
|
||||||
actions, and the second half of this file is the price of that: deleting an
|
story actions, and the second half of this file is the cost of that.
|
||||||
action from in front of a position slid a never-summarized action into the
|
Deleting an action from in front of a position slid a never-summarized
|
||||||
covered range, so every delete had to slide the cursors too. The mark is a node
|
action into the covered range, so every delete had to slide the cursors
|
||||||
now — `(branch_id, depth)` — and a node does not move when something in front
|
too. The mark is a node now, `(branch_id, depth)`, and a node does not move
|
||||||
of it is deleted, so those tests assert that nothing happens where they used to
|
when something in front of it is deleted. Those tests now assert that
|
||||||
assert that the right correction happened.
|
nothing happens, where they used to assert that the right correction
|
||||||
|
happened.
|
||||||
|
|
||||||
python -m pytest tests/test_memory_settling.py -v
|
python -m pytest tests/test_memory_settling.py -v
|
||||||
"""
|
"""
|
||||||
@@ -110,13 +112,14 @@ def run_memories(db, adventure, stub, monkeypatch):
|
|||||||
# --------------------------------------------------- no holdback, since SP4
|
# --------------------------------------------------- no holdback, since SP4
|
||||||
|
|
||||||
def test_a_block_forms_as_soon_as_the_story_holds_one(db, monkeypatch):
|
def test_a_block_forms_as_soon_as_the_story_holds_one(db, monkeypatch):
|
||||||
"""Covered to action 5 with 12 actions: block 6-11 ends on the *newest*
|
"""Covered to action 5 with 12 actions: block 6-11 ends on the newest
|
||||||
action, and is summarized now rather than a turn later.
|
action, and is summarized now rather than a turn later.
|
||||||
|
|
||||||
This is exactly the case the holdback existed to refuse. What makes it safe
|
This is exactly the case the holdback existed to refuse. What makes it
|
||||||
is no longer that the block stops short — it is that a retry of node 11
|
safe is no longer that the block stops short. It is that a retry of
|
||||||
would withdraw this memory on its way past (see
|
node 11 would withdraw this memory on its way past (see
|
||||||
`test_deleting_a_summarized_node_withdraws_its_memory`, the same repair).
|
`test_deleting_a_summarized_node_withdraws_its_memory`, the same
|
||||||
|
repair).
|
||||||
"""
|
"""
|
||||||
adventure = make_adventure(db, 12)
|
adventure = make_adventure(db, 12)
|
||||||
cover(db, adventure, 6)
|
cover(db, adventure, 6)
|
||||||
@@ -128,8 +131,8 @@ def test_a_block_forms_as_soon_as_the_story_holds_one(db, monkeypatch):
|
|||||||
assert "Action 11." in stub.excerpts[0]
|
assert "Action 11." in stub.excerpts[0]
|
||||||
memory = db.query(models.Memory).one()
|
memory = db.query(models.Memory).one()
|
||||||
assert (memory.source_start, memory.source_end) == (6, 11)
|
assert (memory.source_start, memory.source_end) == (6, 11)
|
||||||
# The mark and the memory name the same node — that is what keeps them from
|
# The mark and the memory name the same node. That is what keeps them
|
||||||
# drifting apart however gappy the depths underneath are.
|
# from drifting apart, however gappy the underlying depths are.
|
||||||
assert (memory.branch_id, memory.depth) == cursors.MEMORY.stored(adventure)
|
assert (memory.branch_id, memory.depth) == cursors.MEMORY.stored(adventure)
|
||||||
assert covered_depth(db, adventure) == 11
|
assert covered_depth(db, adventure) == 11
|
||||||
|
|
||||||
@@ -155,13 +158,13 @@ def test_the_first_memory_lands_at_memory_start(db, monkeypatch):
|
|||||||
|
|
||||||
|
|
||||||
def test_legacy_caught_up_adventure_is_not_rewound(db, monkeypatch):
|
def test_legacy_caught_up_adventure_is_not_rewound(db, monkeypatch):
|
||||||
"""An adventure summarized under the OLD rule carries a cursor equal to its
|
"""An adventure summarized under the old rule carries a cursor equal to
|
||||||
action count — one past the end of the story. That used to need a clamp on
|
its action count, one past the end of the story. That used to require a
|
||||||
every post-turn pass, and clamping it to the settled count re-covered an
|
clamp on every post-turn pass, and clamping it to the settled count
|
||||||
action.
|
re-covered an action.
|
||||||
|
|
||||||
A mark that names a node has no such edge: the newest action is the node,
|
A mark that names a node has no such edge. The newest action is the
|
||||||
and "everything after it" is empty until the story grows.
|
node, and "everything after it" is empty until the story grows.
|
||||||
"""
|
"""
|
||||||
adventure = make_adventure(db, 12)
|
adventure = make_adventure(db, 12)
|
||||||
db.add(models.Memory(adventure_id=adventure.id, text="A", source_start=0, source_end=5))
|
db.add(models.Memory(adventure_id=adventure.id, text="A", source_start=0, source_end=5))
|
||||||
@@ -238,9 +241,9 @@ def test_deleting_a_middle_action_leaves_the_mark_where_it_was(db):
|
|||||||
with node 5 gone.
|
with node 5 gone.
|
||||||
"""
|
"""
|
||||||
adventure = summarized_adventure(db)
|
adventure = summarized_adventure(db)
|
||||||
# Node 4 is inside memory A's block but is not the node it hangs off, so
|
# Node 4 is inside memory A's block but is not the node it hangs off,
|
||||||
# nothing is withdrawn — the same reading the old code had, where only a
|
# so nothing is withdrawn. The old code read it the same way: only a
|
||||||
# memory whose *end* had fallen off the story was pruned.
|
# memory whose end had fallen off the story was pruned.
|
||||||
victim = db.query(models.Action).filter_by(adventure_id=adventure.id, index=4).one()
|
victim = db.query(models.Action).filter_by(adventure_id=adventure.id, index=4).one()
|
||||||
|
|
||||||
assert memorybank.forget_node(db, adventure, victim) == 0
|
assert memorybank.forget_node(db, adventure, victim) == 0
|
||||||
@@ -267,12 +270,13 @@ def test_deleting_a_later_action_leaves_the_mark_alone(db):
|
|||||||
|
|
||||||
|
|
||||||
def test_deleting_a_summarized_node_withdraws_its_memory(db):
|
def test_deleting_a_summarized_node_withdraws_its_memory(db):
|
||||||
"""Discarding the memory isn't enough — the story it covered is still
|
"""Discarding the memory is not enough. The story it covered is still
|
||||||
behind the mark, so the mark has to come back to where that block began.
|
behind the mark, so the mark has to move back to where that block began.
|
||||||
|
|
||||||
Memory B ends on node 11, so deleting node 11 is what withdraws it. The old
|
Memory B ends on node 11, so deleting node 11 withdraws it. The old code
|
||||||
code found this by scanning for a memory whose covered range had fallen off
|
found this by scanning for a memory whose covered range had fallen off
|
||||||
the end of the story; the memory hangs off the node now, so it is a lookup.
|
the end of the story. Now the memory hangs off the node, so finding it
|
||||||
|
is a lookup.
|
||||||
"""
|
"""
|
||||||
adventure = summarized_adventure(db)
|
adventure = summarized_adventure(db)
|
||||||
victim = db.query(models.Action).filter_by(adventure_id=adventure.id, index=11).one()
|
victim = db.query(models.Action).filter_by(adventure_id=adventure.id, index=11).one()
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ def hosted(monkeypatch):
|
|||||||
|
|
||||||
|
|
||||||
def _resolves_to(monkeypatch, ip: str):
|
def _resolves_to(monkeypatch, ip: str):
|
||||||
"""Pin getaddrinfo so we test the address decision, not real DNS."""
|
"""Pin `getaddrinfo` so the test exercises the address decision, not real DNS."""
|
||||||
monkeypatch.setattr(
|
monkeypatch.setattr(
|
||||||
netguard.socket, "getaddrinfo",
|
netguard.socket, "getaddrinfo",
|
||||||
lambda *a, **k: [(2, 1, 6, "", (ip, 443))],
|
lambda *a, **k: [(2, 1, 6, "", (ip, 443))],
|
||||||
@@ -64,6 +64,6 @@ def test_unresolvable_host_is_blocked(hosted, monkeypatch):
|
|||||||
|
|
||||||
def test_noop_in_local_mode(monkeypatch):
|
def test_noop_in_local_mode(monkeypatch):
|
||||||
monkeypatch.setattr(auth, "MULTI_USER", False)
|
monkeypatch.setattr(auth, "MULTI_USER", False)
|
||||||
# Local installs legitimately reach localhost (Ollama) — never blocked.
|
# Local installs must reach localhost (Ollama). The guard never blocks local mode.
|
||||||
assert netguard.endpoint_block_reason("http://localhost:11434/v1") is None
|
assert netguard.endpoint_block_reason("http://localhost:11434/v1") is None
|
||||||
assert netguard.endpoint_block_reason("http://127.0.0.1:11434/v1") is None
|
assert netguard.endpoint_block_reason("http://127.0.0.1:11434/v1") is None
|
||||||
|
|||||||
@@ -1,25 +1,26 @@
|
|||||||
"""Prompt caching: the prompt has to start with the same bytes every turn.
|
"""Prompt caching: the prompt has to start with the same bytes every turn.
|
||||||
|
|
||||||
Every endpoint that caches prompts caches a *prefix* — it reuses the request up
|
Every endpoint that caches prompts caches a prefix. It reuses the request
|
||||||
to the first byte that differs from last time and no further. So the cost of a
|
up to the first byte that differs from last time, and no further. The cost
|
||||||
turn is decided by layout: one section that changes each turn, placed near the
|
of a turn is therefore decided by layout: one section that changes each
|
||||||
top, re-prices everything underneath it, and underneath it is the story
|
turn, placed near the top, re-prices everything underneath it, and
|
||||||
history, which is most of the prompt.
|
underneath it is the story history, which makes up most of the prompt.
|
||||||
|
|
||||||
Three things have to hold, and each is easy to undo by accident:
|
Three things must hold, and each is easy to undo by accident:
|
||||||
|
|
||||||
* the static block is byte-identical across turns — adding a section that moves
|
* The static block is byte-identical across turns. Adding a section that
|
||||||
(live stats, retrieved memories, a rewritten summary) to `system_sections` is
|
moves (live stats, retrieved memories, a rewritten summary) to
|
||||||
the mistake this file exists to catch;
|
`system_sections` is the mistake this file exists to catch.
|
||||||
* the sections that move sit *after* the history, but still *before* the tail
|
* The sections that move sit after the history, but still before the tail
|
||||||
that is last for its own reasons (front memory, the length hint, and
|
that is last for its own reasons: front memory, the length hint, and
|
||||||
EMIT_REMINDER, which is what keeps the state block emitted at all);
|
`EMIT_REMINDER`, which is what keeps the state block emitted at all.
|
||||||
* moving a section out of the system block does not drop it from the token
|
* Moving a section out of the system block does not drop it from the
|
||||||
budget — it is still in the prompt.
|
token budget. It is still in the prompt.
|
||||||
|
|
||||||
Plus the two request-level halves: preferring one OpenRouter upstream, since
|
This file also covers two request-level concerns: preferring one
|
||||||
each upstream holds its own cache, and reading back the usage the endpoint
|
OpenRouter upstream, since each upstream holds its own cache, and reading
|
||||||
reports so the hit rate is measurable rather than assumed.
|
back the usage the endpoint reports, so the hit rate is measurable rather
|
||||||
|
than assumed.
|
||||||
|
|
||||||
python -m pytest tests/test_prompt_caching.py -v
|
python -m pytest tests/test_prompt_caching.py -v
|
||||||
"""
|
"""
|
||||||
@@ -68,8 +69,8 @@ def test_fallbacks_stay_on():
|
|||||||
|
|
||||||
|
|
||||||
def test_non_openrouter_endpoints_get_no_provider_field():
|
def test_non_openrouter_endpoints_get_no_provider_field():
|
||||||
"""Ollama and friends reject fields they do not know — the same trap the
|
"""Ollama and other providers reject fields they do not know. This is
|
||||||
`reasoning` param is written around."""
|
the same problem the `reasoning` param works around."""
|
||||||
body = _routed("http://localhost:11434/v1", "deepseek/deepseek-v4-flash-0731")
|
body = _routed("http://localhost:11434/v1", "deepseek/deepseek-v4-flash-0731")
|
||||||
assert "provider" not in body
|
assert "provider" not in body
|
||||||
|
|
||||||
@@ -85,8 +86,9 @@ def test_unknown_vendors_are_left_alone():
|
|||||||
# ------------------------------------------------------ reading usage back
|
# ------------------------------------------------------ reading usage back
|
||||||
|
|
||||||
def test_usage_is_recorded_from_a_final_chunk():
|
def test_usage_is_recorded_from_a_final_chunk():
|
||||||
"""In a stream the usage block rides on a last chunk carrying no choices,
|
"""In a stream, the usage block arrives in a final chunk that carries
|
||||||
which is why it is read separately from the text extraction."""
|
no choices, which is why it is read separately from the text
|
||||||
|
extraction."""
|
||||||
provider = OpenAICompatibleProvider("https://openrouter.ai/api/v1", "k", "m")
|
provider = OpenAICompatibleProvider("https://openrouter.ai/api/v1", "k", "m")
|
||||||
assert provider.last_usage is None
|
assert provider.last_usage is None
|
||||||
provider._record_usage({"choices": [{"delta": {"content": "hi"}}]})
|
provider._record_usage({"choices": [{"delta": {"content": "hi"}}]})
|
||||||
@@ -109,8 +111,8 @@ def test_a_later_chunk_without_usage_does_not_erase_it():
|
|||||||
# ------------------------------------------------------------ prompt layout
|
# ------------------------------------------------------------ prompt layout
|
||||||
|
|
||||||
def _with_hp(world_state, hp):
|
def _with_hp(world_state, hp):
|
||||||
"""`world_state` is nested by group, and the JSON column only notices a
|
"""`world_state` is nested by group, and the JSON column only detects a
|
||||||
whole new object — so build one rather than mutating in place."""
|
whole new object. Build a new one instead of mutating in place."""
|
||||||
return {**world_state, "player": {**world_state["player"], "hp": hp}}
|
return {**world_state, "player": {**world_state["player"], "hp": hp}}
|
||||||
|
|
||||||
|
|
||||||
@@ -153,8 +155,9 @@ def story():
|
|||||||
|
|
||||||
|
|
||||||
def test_changing_a_stat_leaves_the_static_block_untouched(story):
|
def test_changing_a_stat_leaves_the_static_block_untouched(story):
|
||||||
"""The whole point. Live values used to sit third from the top, so a single
|
"""The whole point. Live values used to sit third from the top, so a
|
||||||
point of damage re-priced the instructions, the plot and the history."""
|
single point of damage re-priced the instructions, the plot, and the
|
||||||
|
history."""
|
||||||
db, adventure, settings = story
|
db, adventure, settings = story
|
||||||
before, _, _ = builder.build_context(adventure, settings)
|
before, _, _ = builder.build_context(adventure, settings)
|
||||||
adventure.world_state = _with_hp(adventure.world_state, 40)
|
adventure.world_state = _with_hp(adventure.world_state, 40)
|
||||||
@@ -169,8 +172,8 @@ def test_the_static_block_holds_the_things_that_do_not_move(story):
|
|||||||
system_text, story_text, _ = builder.build_context(adventure, settings)
|
system_text, story_text, _ = builder.build_context(adventure, settings)
|
||||||
for fixed in ("Write in second person.", "The hero hunts bandits."):
|
for fixed in ("Write in second person.", "The hero hunts bandits."):
|
||||||
assert fixed in system_text
|
assert fixed in system_text
|
||||||
# The stat *guide* is derived from the schema and so is fixed; the live
|
# The stat guide is derived from the schema, so it is fixed. The live
|
||||||
# values it describes are not, and belong to the story text.
|
# values it describes are not fixed, and belong to the story text.
|
||||||
assert "Stat guide" in system_text
|
assert "Stat guide" in system_text
|
||||||
for moves in ("The hero left the village.", "hp 100/100"):
|
for moves in ("The hero left the village.", "hp 100/100"):
|
||||||
assert moves not in system_text
|
assert moves not in system_text
|
||||||
@@ -186,8 +189,9 @@ def test_volatile_sections_sit_after_the_history(story):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_tail_stays_the_tail(story):
|
def test_the_tail_stays_the_tail(story):
|
||||||
"""front memory, the length hint and EMIT_REMINDER are last for reasons of
|
"""Front memory, the length hint, and `EMIT_REMINDER` are last for
|
||||||
their own, and the live sections must not have displaced them."""
|
reasons of their own, and the live sections must not have displaced
|
||||||
|
them."""
|
||||||
db, adventure, settings = story
|
db, adventure, settings = story
|
||||||
_, story_text, report = builder.build_context(adventure, settings)
|
_, story_text, report = builder.build_context(adventure, settings)
|
||||||
labels = [s["label"] for s in report["sections"]]
|
labels = [s["label"] for s in report["sections"]]
|
||||||
@@ -199,8 +203,8 @@ def test_the_tail_stays_the_tail(story):
|
|||||||
|
|
||||||
def test_live_sections_are_still_charged_to_the_budget(story):
|
def test_live_sections_are_still_charged_to_the_budget(story):
|
||||||
"""They moved out of `system_sections`, so it would be easy to stop
|
"""They moved out of `system_sections`, so it would be easy to stop
|
||||||
counting them in `reserved` — and then the history, which is budgeted with
|
counting them in `reserved`. If that happened, the history, which is
|
||||||
what is left over, would quietly overrun."""
|
budgeted with what is left over, would quietly overrun."""
|
||||||
db, adventure, settings = story
|
db, adventure, settings = story
|
||||||
for i in range(6, 90):
|
for i in range(6, 90):
|
||||||
db.add(models.Action(
|
db.add(models.Action(
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
"""Regression tests for the X-Forwarded-For rate-limit bypass and the
|
"""Regression tests for the X-Forwarded-For rate-limit bypass and the
|
||||||
per-account login throttle added to close it.
|
per-account login throttle added to close it.
|
||||||
|
|
||||||
Background: uvicorn's --forwarded-allow-ips "*" trusted the LEFTMOST
|
Background: uvicorn's `--forwarded-allow-ips "*"` trusted the leftmost
|
||||||
X-Forwarded-For entry, which the client controls, so rotating the header
|
`X-Forwarded-For` entry. The client controls that entry, so rotating the
|
||||||
handed out a fresh rate-limit bucket per request. client_ip now reads the
|
header issued a fresh rate-limit bucket on every request. `client_ip` now
|
||||||
hop the trusted edge appends (rightmost), and login has an email-keyed throttle
|
reads the hop the trusted edge appends, which is the rightmost one. Login
|
||||||
that no IP trick can dilute.
|
also has an email-keyed throttle that no IP trick can weaken.
|
||||||
|
|
||||||
python -m pytest tests/test_ratelimit_hardening.py -v
|
python -m pytest tests/test_ratelimit_hardening.py -v
|
||||||
"""
|
"""
|
||||||
@@ -35,15 +35,15 @@ class _Req:
|
|||||||
|
|
||||||
def test_client_ip_takes_appended_rightmost_hop(monkeypatch):
|
def test_client_ip_takes_appended_rightmost_hop(monkeypatch):
|
||||||
monkeypatch.setattr(limits, "TRUSTED_PROXY_HOPS", 1)
|
monkeypatch.setattr(limits, "TRUSTED_PROXY_HOPS", 1)
|
||||||
# Attacker prepends a fake IP; the edge appends the real one on the right.
|
# An attacker prepends a fake IP. The edge appends the real one on the right.
|
||||||
req = _Req("203.0.113.9, 198.51.100.77")
|
req = _Req("203.0.113.9, 198.51.100.77")
|
||||||
assert limits.client_ip(req) == "198.51.100.77"
|
assert limits.client_ip(req) == "198.51.100.77"
|
||||||
|
|
||||||
|
|
||||||
def test_client_ip_ignores_spoofed_leftmost(monkeypatch):
|
def test_client_ip_ignores_spoofed_leftmost(monkeypatch):
|
||||||
monkeypatch.setattr(limits, "TRUSTED_PROXY_HOPS", 1)
|
monkeypatch.setattr(limits, "TRUSTED_PROXY_HOPS", 1)
|
||||||
# Whatever the client stuffs to the left, the keyed IP stays the real hop —
|
# The keyed IP stays the real hop regardless of what the client adds on
|
||||||
# so rotating it no longer mints a new bucket.
|
# the left, so rotating that value no longer creates a new bucket.
|
||||||
a = limits.client_ip(_Req("1.1.1.1, 198.51.100.77"))
|
a = limits.client_ip(_Req("1.1.1.1, 198.51.100.77"))
|
||||||
b = limits.client_ip(_Req("2.2.2.2, 198.51.100.77"))
|
b = limits.client_ip(_Req("2.2.2.2, 198.51.100.77"))
|
||||||
c = limits.client_ip(_Req("evil, junk, 198.51.100.77"))
|
c = limits.client_ip(_Req("evil, junk, 198.51.100.77"))
|
||||||
@@ -74,11 +74,11 @@ def _multi_user(monkeypatch):
|
|||||||
|
|
||||||
def test_login_throttle_blocks_after_limit():
|
def test_login_throttle_blocks_after_limit():
|
||||||
email = "victim@example.com"
|
email = "victim@example.com"
|
||||||
# Up to the limit: allowed, each a recorded failure.
|
# Each attempt up to the limit is allowed and recorded as a failure.
|
||||||
for _ in range(limits.LOGIN_FAIL_LIMIT):
|
for _ in range(limits.LOGIN_FAIL_LIMIT):
|
||||||
limits.check_login_allowed(email) # does not raise
|
limits.check_login_allowed(email) # does not raise
|
||||||
limits.note_login_failure(email)
|
limits.note_login_failure(email)
|
||||||
# One more crosses the line.
|
# One more failure exceeds the limit.
|
||||||
with pytest.raises(limits.HTTPException) as exc:
|
with pytest.raises(limits.HTTPException) as exc:
|
||||||
limits.check_login_allowed(email)
|
limits.check_login_allowed(email)
|
||||||
assert exc.value.status_code == 429
|
assert exc.value.status_code == 429
|
||||||
@@ -89,7 +89,7 @@ def test_login_throttle_is_per_account():
|
|||||||
limits.note_login_failure("a@example.com")
|
limits.note_login_failure("a@example.com")
|
||||||
with pytest.raises(limits.HTTPException):
|
with pytest.raises(limits.HTTPException):
|
||||||
limits.check_login_allowed("a@example.com")
|
limits.check_login_allowed("a@example.com")
|
||||||
# A different account is unaffected — this is not an IP bucket.
|
# A different account is unaffected because the throttle keys on email, not IP.
|
||||||
limits.check_login_allowed("b@example.com") # must not raise
|
limits.check_login_allowed("b@example.com") # must not raise
|
||||||
|
|
||||||
|
|
||||||
@@ -98,7 +98,7 @@ def test_successful_login_clears_the_streak():
|
|||||||
for _ in range(limits.LOGIN_FAIL_LIMIT):
|
for _ in range(limits.LOGIN_FAIL_LIMIT):
|
||||||
limits.note_login_failure(email)
|
limits.note_login_failure(email)
|
||||||
limits.note_login_success(email)
|
limits.note_login_success(email)
|
||||||
limits.check_login_allowed(email) # streak wiped — must not raise
|
limits.check_login_allowed(email) # failure streak cleared, must not raise
|
||||||
|
|
||||||
|
|
||||||
def test_throttle_is_noop_in_local_mode(monkeypatch):
|
def test_throttle_is_noop_in_local_mode(monkeypatch):
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ def _body(reasoning_max_tokens, api_mode="chat", max_tokens=1000):
|
|||||||
|
|
||||||
|
|
||||||
def test_zero_sends_nothing():
|
def test_zero_sends_nothing():
|
||||||
"""Ollama and friends reject unknown fields — 0 must stay silent."""
|
"""Ollama and other providers reject unknown fields. Sending 0 must not add a `reasoning` field."""
|
||||||
assert "reasoning" not in _body(0)
|
assert "reasoning" not in _body(0)
|
||||||
|
|
||||||
|
|
||||||
@@ -36,7 +36,7 @@ def test_negative_turns_reasoning_off():
|
|||||||
|
|
||||||
|
|
||||||
def test_off_is_not_merely_excluded():
|
def test_off_is_not_merely_excluded():
|
||||||
"""`exclude: true` still thinks and still bills; we want it actually off."""
|
"""`exclude: true` still generates and bills for reasoning tokens. The off setting must omit the field entirely instead of relying on `exclude`."""
|
||||||
assert _body(-1)["reasoning"].get("exclude") is None
|
assert _body(-1)["reasoning"].get("exclude") is None
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -137,7 +137,7 @@ def test_retry_keeps_the_discarded_attempt(client):
|
|||||||
|
|
||||||
_retry(client)
|
_retry(client)
|
||||||
actions = _actions(client)
|
actions = _actions(client)
|
||||||
# One AI action still, not two — the retry replaced the live text in place.
|
# One AI action still, not two. The retry replaced the live text in place.
|
||||||
assert [a["type"] for a in actions] == ["start", "do", "ai"]
|
assert [a["type"] for a in actions] == ["start", "do", "ai"]
|
||||||
last = actions[-1]
|
last = actions[-1]
|
||||||
assert last["text"] == "Attempt two."
|
assert last["text"] == "Attempt two."
|
||||||
@@ -151,18 +151,18 @@ def test_retry_keeps_the_discarded_attempt(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_retry_context_excludes_the_attempt_being_replaced(client):
|
def test_retry_context_excludes_the_attempt_being_replaced(client):
|
||||||
"""The whole point of a retry is a fresh take on the *same* turn. The row
|
"""A retry produces a fresh take on the same turn. The row survives the
|
||||||
now survives the retry (it holds the variant history), so it is still in
|
retry because it holds the variant history, so it is still in
|
||||||
`adventure.actions` while the replacement context is assembled — it must be
|
`adventure.actions` while the replacement context is assembled. The
|
||||||
filtered out, or the model is asked to continue *past* the attempt it is
|
context builder must filter it out, or the model continues past the
|
||||||
supposed to be replacing and writes a sequel that blends both."""
|
attempt it is replacing and writes a sequel that blends both."""
|
||||||
ScriptedProvider.replies = ["Attempt one.", "Attempt two."]
|
ScriptedProvider.replies = ["Attempt one.", "Attempt two."]
|
||||||
_play(client)
|
_play(client)
|
||||||
_retry(client)
|
_retry(client)
|
||||||
|
|
||||||
retry_story = ScriptedProvider.prompts[-1][1]
|
retry_story = ScriptedProvider.prompts[-1][1]
|
||||||
assert "Attempt one." not in retry_story
|
assert "Attempt one." not in retry_story
|
||||||
# The turn's own player action must still be there — it's what to respond to.
|
# The turn's own player action must still be there. It is what the model responds to.
|
||||||
assert "look around" in retry_story
|
assert "look around" in retry_story
|
||||||
assert "You enter a cave." in retry_story
|
assert "You enter a cave." in retry_story
|
||||||
|
|
||||||
@@ -257,7 +257,7 @@ def test_cannot_switch_a_turn_the_story_moved_past(client):
|
|||||||
f"/api/adventures/{client.adv_id}/actions/{retried['id']}/variant", json={"index": 0})
|
f"/api/adventures/{client.adv_id}/actions/{retried['id']}/variant", json={"index": 0})
|
||||||
assert r.status_code == 400
|
assert r.status_code == 400
|
||||||
assert "latest message" in r.json()["detail"]
|
assert "latest message" in r.json()["detail"]
|
||||||
# Still readable, though — that's the whole point of keeping them.
|
# The variant is still readable. Keeping every attempt browsable is why it still exists.
|
||||||
variants = client.get(
|
variants = client.get(
|
||||||
f"/api/adventures/{client.adv_id}/actions/{retried['id']}/variants").json()
|
f"/api/adventures/{client.adv_id}/actions/{retried['id']}/variants").json()
|
||||||
assert [v["text"] for v in variants] == ["One.", "Two."]
|
assert [v["text"] for v in variants] == ["One.", "Two."]
|
||||||
@@ -333,7 +333,7 @@ def test_export_and_import_round_trips_variants(client):
|
|||||||
bundle = client.get(f"/api/adventures/{client.adv_id}/export").json()
|
bundle = client.get(f"/api/adventures/{client.adv_id}/export").json()
|
||||||
# SP6: the attempts are nodes in the bundle too, sharing one coordinate,
|
# SP6: the attempts are nodes in the bundle too, sharing one coordinate,
|
||||||
# and `live` says which of them the story tells. The `variants` array
|
# and `live` says which of them the story tells. The `variants` array
|
||||||
# survives only in the v1 *reader* — see the hand-edited bundle below.
|
# survives only in the v1 reader. See the hand-edited bundle below.
|
||||||
ai = [a for a in bundle["actions"] if a["type"] == "ai"]
|
ai = [a for a in bundle["actions"] if a["type"] == "ai"]
|
||||||
assert [(a["text"], a["live"]) for a in ai] == [("One.", False), ("Two.", True)]
|
assert [(a["text"], a["live"]) for a in ai] == [("One.", False), ("Two.", True)]
|
||||||
assert len({(a["branch"], a["depth"]) for a in ai}) == 1
|
assert len({(a["branch"], a["depth"]) for a in ai}) == 1
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
"""Scenario cover art + the Continue-card snippet.
|
"""Scenario cover art and the Continue-card snippet.
|
||||||
|
|
||||||
Unit tests for the data-URI handling in app/images.py, then HTTP tests that the
|
Unit tests for the data-URI handling in app/images.py. Then HTTP tests
|
||||||
list endpoints advertise a cacheable `image_url` (never the inline base64), that
|
confirm that the list endpoints advertise a cacheable `image_url` instead
|
||||||
the image route serves real bytes, and that an adventure's snippet comes from
|
of the inline base64, that the image route serves real bytes, and that an
|
||||||
the latest *narration* rather than the player's last line.
|
adventure's snippet comes from the latest narration rather than the
|
||||||
|
player's last line.
|
||||||
|
|
||||||
python -m pytest tests/test_scenario_art.py -v
|
python -m pytest tests/test_scenario_art.py -v
|
||||||
"""
|
"""
|
||||||
@@ -41,13 +42,14 @@ def test_decode_returns_bytes_and_content_type():
|
|||||||
|
|
||||||
|
|
||||||
def test_decode_tolerates_wrapped_base64():
|
def test_decode_tolerates_wrapped_base64():
|
||||||
"""A hand-pasted URI can carry newlines; b64decode(validate=True) won't."""
|
"""A hand-pasted URI can carry newlines. `b64decode(validate=True)` rejects them."""
|
||||||
wrapped = "data:image/png;base64," + "\n".join(
|
wrapped = "data:image/png;base64," + "\n".join(
|
||||||
base64.b64encode(PNG_BYTES).decode()[i:i + 24] for i in range(0, 100, 24)
|
base64.b64encode(PNG_BYTES).decode()[i:i + 24] for i in range(0, 100, 24)
|
||||||
)
|
)
|
||||||
# Only asserting it doesn't raise and doesn't silently return a partial
|
# This test only confirms that the call does not raise and does not
|
||||||
# decode of a truncated payload — the wrapped prefix here is not the whole
|
# silently return a partial decode of a truncated payload. The wrapped
|
||||||
# image, so a None result is also acceptable; what matters is no exception.
|
# prefix here is not the whole image, so a None result is also
|
||||||
|
# acceptable. What matters is that no exception occurs.
|
||||||
images.decode(wrapped)
|
images.decode(wrapped)
|
||||||
|
|
||||||
|
|
||||||
@@ -59,7 +61,7 @@ def test_decode_rejects_non_data_uris_and_garbage():
|
|||||||
|
|
||||||
|
|
||||||
def test_decode_rejects_svg():
|
def test_decode_rejects_svg():
|
||||||
"""SVG can carry script and these bytes are served from our own origin."""
|
"""SVG can carry script, and the app serves these bytes from its own origin."""
|
||||||
svg = "data:image/svg+xml;base64," + base64.b64encode(b"<svg/>").decode()
|
svg = "data:image/svg+xml;base64," + base64.b64encode(b"<svg/>").decode()
|
||||||
assert images.decode(svg) is None
|
assert images.decode(svg) is None
|
||||||
|
|
||||||
@@ -151,7 +153,7 @@ def test_scenario_list_advertises_a_url_and_hides_the_base64(client):
|
|||||||
|
|
||||||
pictured = rows["Pictured"]
|
pictured = rows["Pictured"]
|
||||||
assert pictured["image_url"].startswith(f"/api/scenarios/{client.ids['scenario']}/image?v=")
|
assert pictured["image_url"].startswith(f"/api/scenarios/{client.ids['scenario']}/image?v=")
|
||||||
# The whole point: a list response must never carry the inline image.
|
# A list response must never carry the inline image.
|
||||||
assert "image" not in pictured
|
assert "image" not in pictured
|
||||||
|
|
||||||
assert rows["Emoji"]["image_url"] == ""
|
assert rows["Emoji"]["image_url"] == ""
|
||||||
@@ -180,7 +182,7 @@ def test_single_scenario_still_returns_the_raw_uri_for_editing(client):
|
|||||||
def test_adventure_list_carries_snippet_and_inherited_art(client):
|
def test_adventure_list_carries_snippet_and_inherited_art(client):
|
||||||
row = next(r for r in client.get("/api/adventures").json()
|
row = next(r for r in client.get("/api/adventures").json()
|
||||||
if r["id"] == client.ids["adventure"])
|
if r["id"] == client.ids["adventure"])
|
||||||
# Latest narration, whitespace collapsed — not the player's "I draw my sword."
|
# Latest narration, whitespace collapsed. Not the player's line, "I draw my sword."
|
||||||
assert row["snippet"] == "Steel rings. The corridor answers."
|
assert row["snippet"] == "Steel rings. The corridor answers."
|
||||||
assert row["image_url"].startswith(f"/api/scenarios/{client.ids['scenario']}/image?v=")
|
assert row["image_url"].startswith(f"/api/scenarios/{client.ids['scenario']}/image?v=")
|
||||||
assert row["action_count"] == 3
|
assert row["action_count"] == 3
|
||||||
|
|||||||
@@ -75,12 +75,13 @@ def make_scenario(client, **kwargs):
|
|||||||
|
|
||||||
|
|
||||||
def edit_scenario(scenario_id, cards=None, **fields):
|
def edit_scenario(scenario_id, cards=None, **fields):
|
||||||
"""Author-side edit, straight to the DB (the scenario API is tested elsewhere).
|
"""Author-side edit, straight to the database. The scenario API is
|
||||||
|
tested elsewhere.
|
||||||
|
|
||||||
`cards` is the scenario's full card list afterwards. Cards are matched to
|
`cards` is the scenario's full card list afterwards. Cards are matched
|
||||||
existing rows by name and edited in place, exactly as ScenarioEditor does
|
to existing rows by name and edited in place, exactly as ScenarioEditor
|
||||||
(PATCH /story-cards/{id}) — card ids are stable across authoring, which is
|
does (PATCH /story-cards/{id}). Card ids stay stable across authoring,
|
||||||
what source_ref tracking relies on.
|
which is what source_ref tracking relies on.
|
||||||
"""
|
"""
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -1,18 +1,18 @@
|
|||||||
"""context_snapshot, stored compressed.
|
"""context_snapshot, stored compressed.
|
||||||
|
|
||||||
The column is 89% of the database and the free tier allows 512 MB. Reads were
|
The column makes up 89% of the database, and the free tier allows only
|
||||||
solved by deferring it; this is about the storage ceiling. Postgres already
|
512 MB. Deferring the column already solved the read cost, so this is
|
||||||
TOASTs it and only gets 1.7x, because pglz is tuned for fast decompression of
|
about the storage ceiling. Postgres already TOASTs the column and gets
|
||||||
data a query might filter on — and nothing ever filters on an assembled
|
only a 1.7x ratio, because pglz favors fast decompression for data a query
|
||||||
prompt.
|
might filter on. Nothing ever filters on an assembled prompt.
|
||||||
|
|
||||||
Three things have to hold, and only the first is obvious:
|
Three things must hold, and only the first is obvious:
|
||||||
|
|
||||||
* what goes in comes back out, exactly, including a snapshot written before
|
* What goes in comes back out exactly, including a snapshot written before
|
||||||
the conversion and one that is NULL;
|
the conversion and one that is NULL.
|
||||||
* the model still hands callers a dict, so no call site changes;
|
* The model still hands callers a dict, so no call site changes.
|
||||||
* migration 44 drops the original column, so the backfill is the one
|
* Migration 44 drops the original column, so the backfill is the one
|
||||||
destructive step in this file — it must convert every row or abort.
|
destructive step in this file. It must convert every row or abort.
|
||||||
|
|
||||||
python -m pytest tests/test_snapshot_compression.py -v
|
python -m pytest tests/test_snapshot_compression.py -v
|
||||||
"""
|
"""
|
||||||
@@ -147,8 +147,9 @@ def test_null_stays_null(db, adventure):
|
|||||||
|
|
||||||
|
|
||||||
def test_an_unreadable_snapshot_reads_as_none_rather_than_raising(db, adventure):
|
def test_an_unreadable_snapshot_reads_as_none_rather_than_raising(db, adventure):
|
||||||
"""One corrupt row must not 500 the turn that happens to load it. The
|
"""One corrupt row must not return a 500 error for the turn that loads
|
||||||
snapshot is a debugging view; the story is the thing that matters."""
|
it. The snapshot is a debugging view. The story is what actually
|
||||||
|
matters."""
|
||||||
action = models.Action(
|
action = models.Action(
|
||||||
adventure_id=adventure.id, index=0, type="ai", text="t",
|
adventure_id=adventure.id, index=0, type="ai", text="t",
|
||||||
context_snapshot={"a": "b"},
|
context_snapshot={"a": "b"},
|
||||||
@@ -236,9 +237,9 @@ def test_bootstrap_leaves_the_column_named_context_snapshot(db, adventure):
|
|||||||
def test_the_backfill_aborts_rather_than_dropping_unconvertible_data(
|
def test_the_backfill_aborts_rather_than_dropping_unconvertible_data(
|
||||||
db, adventure, monkeypatch
|
db, adventure, monkeypatch
|
||||||
):
|
):
|
||||||
"""Migration 44 destroys the original. If anything cannot be converted the
|
"""Migration 44 destroys the original column. If anything fails to
|
||||||
whole run has to roll back with the column still there — the alternative is
|
convert, the whole run must roll back with the column still there.
|
||||||
losing somebody's prompts to a bug in this file."""
|
Otherwise a bug in this file could lose someone's prompts."""
|
||||||
seed_pre_43(db, adventure)
|
seed_pre_43(db, adventure)
|
||||||
db.close()
|
db.close()
|
||||||
|
|
||||||
|
|||||||
@@ -1,20 +1,22 @@
|
|||||||
"""Tests for undo/retry rolling back the shared script_state scoreboard
|
"""Tests for undo and retry rolling back the shared `script_state`
|
||||||
(plan/11-state-revert-and-retry-fix.md).
|
(plan/11-state-revert-and-retry-fix.md).
|
||||||
|
|
||||||
Phase 14 SP4 turned the snapshots around. An action used to carry the state as
|
Phase 14 SP4 reversed the snapshots. An action used to carry the state as
|
||||||
it stood *before* it ran, and rolling back read the snapshot off the action
|
it stood before it ran, and rolling back read the snapshot off the action
|
||||||
being removed. It carries what it left *behind* now, and rolling back reads it
|
being removed. Now it carries the state it left behind, and rolling back
|
||||||
off the node in front — which is the same number arrived at from the other
|
reads that state off the node in front of it. This is the same value
|
||||||
side, and the only version a retry can use: attempts at one turn share a
|
reached from the other direction, and it is the only version a retry can
|
||||||
starting position and differ precisely in their outcome.
|
use, because attempts at one turn share a starting position and differ
|
||||||
|
only in their outcome.
|
||||||
|
|
||||||
Run from the backend dir: python -m pytest tests/test_state_revert.py -v
|
Run from the backend dir: python -m pytest tests/test_state_revert.py -v
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
import tempfile
|
import tempfile
|
||||||
|
|
||||||
# Point the app at a throwaway SQLite file BEFORE importing anything that binds
|
# Point the app at a throwaway SQLite file before importing anything that
|
||||||
# the engine at import time (app.database reads AIDND_DB_PATH on import).
|
# binds the engine at import time. `app.database` reads `AIDND_DB_PATH`
|
||||||
|
# on import.
|
||||||
_tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
|
_tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
|
||||||
_tmp.close()
|
_tmp.close()
|
||||||
os.environ["AIDND_DB_PATH"] = _tmp.name
|
os.environ["AIDND_DB_PATH"] = _tmp.name
|
||||||
@@ -77,9 +79,9 @@ def _forget_snapshots(db, adv):
|
|||||||
# ---------------------------------------------------------------- undo
|
# ---------------------------------------------------------------- undo
|
||||||
|
|
||||||
def test_undo_reverts_state_to_before_the_turn(db):
|
def test_undo_reverts_state_to_before_the_turn(db):
|
||||||
# A turn took the scoreboard from {gold:0} -> {gold:10}. The node in front
|
# A turn moved script_state from {gold:0} to {gold:10}. The node in
|
||||||
# of the turn is what says where it started; current state is the mutated
|
# front of the turn records where it started. The current state is
|
||||||
# one.
|
# the mutated one.
|
||||||
user, adv = _make_adventure(db, {"gold": 10})
|
user, adv = _make_adventure(db, {"gold": 10})
|
||||||
_add(db, adv, 0, "start", state_after={"gold": 0})
|
_add(db, adv, 0, "start", state_after={"gold": 0})
|
||||||
_add(db, adv, 1, "do", state_after={"gold": 0})
|
_add(db, adv, 1, "do", state_after={"gold": 0})
|
||||||
@@ -165,9 +167,9 @@ def test_undo_prunes_memory_covering_removed_actions(db):
|
|||||||
# -------------------------------------------------------- withdrawing a node
|
# -------------------------------------------------------- withdrawing a node
|
||||||
|
|
||||||
def test_forget_node_withdraws_only_what_that_node_produced(db):
|
def test_forget_node_withdraws_only_what_that_node_produced(db):
|
||||||
"""Phase 14 SP3: a memory hangs off the node its block ends on, so removing
|
"""Phase 14 SP3: a memory attaches to the node where its block ends, so
|
||||||
a node is a lookup rather than a scan for memories that have fallen off the
|
removing a node is a lookup rather than a scan for memories that
|
||||||
end of the story."""
|
reference actions the story no longer has."""
|
||||||
user, adv = _make_adventure(db, {})
|
user, adv = _make_adventure(db, {})
|
||||||
_add(db, adv, 0, "do")
|
_add(db, adv, 0, "do")
|
||||||
second = _add(db, adv, 1, "ai")
|
second = _add(db, adv, 1, "ai")
|
||||||
@@ -216,9 +218,9 @@ def test_restore_state_ignores_a_node_with_no_outcome(db):
|
|||||||
# ---------------------------------------------------------------- retry
|
# ---------------------------------------------------------------- retry
|
||||||
|
|
||||||
def test_retry_restores_the_state_the_turn_started_from(db, monkeypatch):
|
def test_retry_restores_the_state_the_turn_started_from(db, monkeypatch):
|
||||||
# Retry must roll the scoreboard back to what the node in front of the AI
|
# Retry must roll script_state back to what the node in front of the AI
|
||||||
# action left behind, so regeneration doesn't stack output mutations on top
|
# action left behind, so regeneration does not stack output mutations
|
||||||
# of the attempt being replaced.
|
# on top of the attempt being replaced.
|
||||||
user, adv = _make_adventure(db, {"gold": 20}) # 20 = double-applied bug value
|
user, adv = _make_adventure(db, {"gold": 20}) # 20 = double-applied bug value
|
||||||
_add(db, adv, 0, "start", state_after={"gold": 0})
|
_add(db, adv, 0, "start", state_after={"gold": 0})
|
||||||
_add(db, adv, 1, "do", state_after={"gold": 10})
|
_add(db, adv, 1, "do", state_after={"gold": 10})
|
||||||
|
|||||||
@@ -1,20 +1,21 @@
|
|||||||
"""Phase 14 SP0 — the regression contract for the story tree.
|
"""Phase 14 SP0: the regression contract for the story tree.
|
||||||
|
|
||||||
This file exists to answer one question, over and over, as the storage model is
|
This file exists to answer one question repeatedly, as the storage model is
|
||||||
replaced underneath the app: **do existing adventures still behave exactly as
|
replaced underneath the app: do existing adventures still behave exactly as
|
||||||
they did?**
|
they did?
|
||||||
|
|
||||||
So it drives the product the way a player does — over HTTP, asserting only on
|
It drives the product the way a player does, over HTTP, asserting only on
|
||||||
API responses — and never reaches into the ORM to check how something is
|
API responses. It never reaches into the ORM to check how something is
|
||||||
stored. Everything it asserts is true of the linear implementation today and
|
stored. Everything it asserts is true of the linear implementation today and
|
||||||
must stay true of the tree, because a linear story is a tree with one branch.
|
must stay true of the tree, because a linear story is a tree with one branch.
|
||||||
|
|
||||||
python -m pytest tests/test_story_tree_baseline.py -v
|
python -m pytest tests/test_story_tree_baseline.py -v
|
||||||
|
|
||||||
**It must pass UNMODIFIED through SP1 (schema), SP2 (branch clause) and SP3
|
This file must pass unmodified through SP1 (schema), SP2 (branch clause),
|
||||||
(memories on nodes).** If a change here looks necessary in one of those
|
and SP3 (memories on nodes). If a change here looks necessary in one of
|
||||||
subphases, the change is wrong, not the test. SP4 is the first subphase allowed
|
those subphases, the change is wrong, not the test. SP4 is the first
|
||||||
to move it, and only for the variant-count semantics called out in plan/14.
|
subphase allowed to move it, and only for the variant-count semantics
|
||||||
|
called out in plan/14.
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
import tempfile
|
import tempfile
|
||||||
@@ -73,9 +74,9 @@ class ScriptedProvider:
|
|||||||
|
|
||||||
|
|
||||||
def _make_world(monkeypatch, *, seeded_actions: int = 0):
|
def _make_world(monkeypatch, *, seeded_actions: int = 0):
|
||||||
"""A user + scenario + adventure, with `seeded_actions` extra story actions
|
"""Create a user, a scenario, and an adventure, with `seeded_actions`
|
||||||
written straight to the database (paging wants more turns than it is worth
|
extra story actions written straight to the database. Paging tests need
|
||||||
playing one at a time)."""
|
more turns than it is practical to play one at a time."""
|
||||||
Base.metadata.create_all(bind=engine)
|
Base.metadata.create_all(bind=engine)
|
||||||
setup = SessionLocal()
|
setup = SessionLocal()
|
||||||
user = models.User(is_guest=False, email="baseline@example.com")
|
user = models.User(is_guest=False, email="baseline@example.com")
|
||||||
@@ -181,8 +182,8 @@ def _state(adv_id):
|
|||||||
# ------------------------------------------------------- opening the story
|
# ------------------------------------------------------- opening the story
|
||||||
|
|
||||||
def test_adventure_opens_on_a_window_with_a_total(long_client):
|
def test_adventure_opens_on_a_window_with_a_total(long_client):
|
||||||
"""The payload is the newest window plus the whole story's length — that is
|
"""The payload includes the newest window and the whole story's length.
|
||||||
how the client knows there is more above."""
|
That is how the client knows more actions exist above it."""
|
||||||
payload = _open(long_client)
|
payload = _open(long_client)
|
||||||
seeded = adventures.ACTION_PAGE + 11 # + the start action
|
seeded = adventures.ACTION_PAGE + 11 # + the start action
|
||||||
assert payload["action_count"] == seeded
|
assert payload["action_count"] == seeded
|
||||||
@@ -247,7 +248,7 @@ def test_retry_replaces_the_text_and_keeps_the_attempt(client):
|
|||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
|
|
||||||
actions = _actions(client)
|
actions = _actions(client)
|
||||||
# Still one AI action for this turn — a retry is a new take, not a new turn.
|
# This turn still has one AI action. A retry creates a new take, not a new turn.
|
||||||
assert [a["type"] for a in actions] == ["start", "do", "ai"]
|
assert [a["type"] for a in actions] == ["start", "do", "ai"]
|
||||||
assert actions[-1]["text"] == "Attempt two."
|
assert actions[-1]["text"] == "Attempt two."
|
||||||
|
|
||||||
@@ -279,14 +280,14 @@ def test_switching_back_to_an_earlier_attempt_restores_it(client):
|
|||||||
json={"index": 0})
|
json={"index": 0})
|
||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
assert _texts(client)[-1] == "Attempt one."
|
assert _texts(client)[-1] == "Attempt one."
|
||||||
# And the script state that attempt produced comes back with it.
|
# The script state that attempt produced comes back with it.
|
||||||
script_state, _ = _state(client.adv_id)
|
script_state, _ = _state(client.adv_id)
|
||||||
assert script_state["gold"] == 10
|
assert script_state["gold"] == 10
|
||||||
|
|
||||||
|
|
||||||
def test_only_the_newest_turn_can_be_switched(client):
|
def test_only_the_newest_turn_can_be_switched(client):
|
||||||
"""An older turn's alternatives stay readable but not selectable — the
|
"""An older turn's alternatives stay readable but not selectable. The
|
||||||
story after it was written as a continuation of what is live."""
|
story after it continues from what is live."""
|
||||||
ScriptedProvider.replies = ["One.", "Again.", "Two."]
|
ScriptedProvider.replies = ["One.", "Again.", "Two."]
|
||||||
_play(client)
|
_play(client)
|
||||||
client.post(f"/api/adventures/{client.adv_id}/retry")
|
client.post(f"/api/adventures/{client.adv_id}/retry")
|
||||||
@@ -313,7 +314,7 @@ def test_undo_removes_the_whole_turn_and_rolls_state_back(client):
|
|||||||
# Both the AI action and the player action that prompted it are gone.
|
# Both the AI action and the player action that prompted it are gone.
|
||||||
assert [a["type"] for a in page["actions"]] == ["start", "do", "ai"]
|
assert [a["type"] for a in page["actions"]] == ["start", "do", "ai"]
|
||||||
assert page["total"] == 3
|
assert page["total"] == 3
|
||||||
# ...and the second turn's ten gold went with them.
|
# The second turn's ten gold is also gone.
|
||||||
script_state, _ = _state(client.adv_id)
|
script_state, _ = _state(client.adv_id)
|
||||||
assert script_state["gold"] == 10
|
assert script_state["gold"] == 10
|
||||||
|
|
||||||
@@ -338,9 +339,9 @@ def test_the_opening_cannot_be_undone(client):
|
|||||||
# ------------------------------------------------------------------ paging
|
# ------------------------------------------------------------------ paging
|
||||||
|
|
||||||
def test_paging_walks_back_to_the_start_without_gaps_or_repeats(long_client):
|
def test_paging_walks_back_to_the_start_without_gaps_or_repeats(long_client):
|
||||||
"""The property that matters: page all the way up and you have seen every
|
"""The property that matters: paging all the way up must show every
|
||||||
action exactly once, in order. This is the invariant a tree must preserve —
|
action exactly once, in order. A tree must preserve this invariant. The
|
||||||
the anchor is an action, never a position."""
|
anchor is always an action, never a position."""
|
||||||
payload = _open(long_client)
|
payload = _open(long_client)
|
||||||
total = payload["action_count"]
|
total = payload["action_count"]
|
||||||
seen = [a["id"] for a in payload["actions"]]
|
seen = [a["id"] for a in payload["actions"]]
|
||||||
@@ -376,7 +377,7 @@ def test_paging_past_the_start_reports_the_end(long_client):
|
|||||||
r = long_client.get(f"/api/adventures/{long_client.adv_id}/actions",
|
r = long_client.get(f"/api/adventures/{long_client.adv_id}/actions",
|
||||||
params={"limit": 5})
|
params={"limit": 5})
|
||||||
oldest = r.json()["actions"][0]["id"]
|
oldest = r.json()["actions"][0]["id"]
|
||||||
# Walk right off the front.
|
# Continue paging until it reaches the oldest action.
|
||||||
seen_all = False
|
seen_all = False
|
||||||
cursor = oldest
|
cursor = oldest
|
||||||
for _ in range(100):
|
for _ in range(100):
|
||||||
@@ -399,8 +400,9 @@ def test_editing_an_action_sticks(client):
|
|||||||
r = client.patch(f"/api/adventures/{client.adv_id}/actions/{action_id}",
|
r = client.patch(f"/api/adventures/{client.adv_id}/actions/{action_id}",
|
||||||
json={"text": "Edited."})
|
json={"text": "Edited."})
|
||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
# Re-read rather than trusting the response: an edit that only lives in the
|
# Re-read from the database instead of trusting the response. An edit
|
||||||
# response has silently reverted for anyone who reloads.
|
# that only appears in the response has already reverted for anyone who
|
||||||
|
# reloads.
|
||||||
assert _texts(client)[-1] == "Edited."
|
assert _texts(client)[-1] == "Edited."
|
||||||
|
|
||||||
|
|
||||||
@@ -450,13 +452,13 @@ def test_memories_are_created_listed_and_deleted(client):
|
|||||||
# ------------------------------------------------------------------ export
|
# ------------------------------------------------------------------ export
|
||||||
|
|
||||||
def test_export_carries_the_whole_story(client):
|
def test_export_carries_the_whole_story(client):
|
||||||
"""A bundle is a backup: every action, not the window.
|
"""A bundle is a backup: it contains every action, not just the window.
|
||||||
|
|
||||||
SP6 changed the format assertion below, as this note said it would. It also
|
SP6 changed the format assertion below, as this note said it would. It
|
||||||
changed one more line in this file than the note allowed for — the
|
also changed one more line than the note allowed for: the `variants`
|
||||||
`variants` array in `test_export_keeps_retry_attempts`, which is the same
|
array in `test_export_keeps_retry_attempts`. That change reflects the
|
||||||
fact seen from the other side: a bundle that has coordinates has no use for
|
same fact: a bundle that stores coordinates has no use for a repeating
|
||||||
a repeating group. Everything else here still passes unmodified.
|
group. Everything else here still passes unmodified.
|
||||||
"""
|
"""
|
||||||
ScriptedProvider.replies = ["One.", "Two."]
|
ScriptedProvider.replies = ["One.", "Two."]
|
||||||
_play(client, "go north")
|
_play(client, "go north")
|
||||||
|
|||||||
@@ -1,20 +1,22 @@
|
|||||||
"""Phase 14 SP9 — editing the take you are actually reading.
|
"""Phase 14 SP9: editing the take you are actually reading.
|
||||||
|
|
||||||
The pager can park a turn on take 2 of 4. The transcript row it sits in is
|
The pager can display a turn on take 2 of 4. The transcript row for that
|
||||||
still keyed by the *live* take, because that is the row the story tells and the
|
turn stays keyed by the live take, because that is the row the transcript
|
||||||
one the window carries; the take being read is only in the client's hand, by
|
reports and the one the window carries. The take currently displayed is
|
||||||
its own node id.
|
known only to the client, by its own node id.
|
||||||
|
|
||||||
So "edit this" has two ids to choose from, and the page shipped choosing the
|
So "edit this" has two ids to choose from, and the page shipped with the
|
||||||
wrong one: it opened the editor on the live take's text and saved over it, from
|
wrong one. It opened the editor on the live take's text and saved over it,
|
||||||
a row that was showing take 2. That is a client bug and it is fixed in
|
even from a row that was displaying take 2. This is a client bug, and it
|
||||||
Play.jsx, but the fix rests on something only the server can promise —
|
is fixed in Play.jsx. The fix depends on a guarantee only the server can
|
||||||
|
make:
|
||||||
|
|
||||||
a take is an ordinary row to the edit endpoint, addressed by its own id,
|
A take is an ordinary row to the edit endpoint, addressed by its own id,
|
||||||
whether or not it is the one the path runs through
|
whether or not it is the one the path runs through.
|
||||||
|
|
||||||
— and on the group listing telling the truth about it afterwards. Both are
|
The fix also depends on the group listing reporting this correctly
|
||||||
asserted here so the client's fix cannot be quietly undermined.
|
afterward. This file asserts both guarantees, so the client's fix cannot
|
||||||
|
be silently broken.
|
||||||
|
|
||||||
python -m pytest tests/test_take_edit.py -v
|
python -m pytest tests/test_take_edit.py -v
|
||||||
"""
|
"""
|
||||||
@@ -119,7 +121,7 @@ def _edit(client, action_id, text):
|
|||||||
|
|
||||||
|
|
||||||
def _live_ai(client):
|
def _live_ai(client):
|
||||||
"""The AI row the story currently tells, as the transcript reports it."""
|
"""The AI row the transcript currently reports as live."""
|
||||||
adv = client.get(f"/api/adventures/{client.adv_id}").json()
|
adv = client.get(f"/api/adventures/{client.adv_id}").json()
|
||||||
return [a for a in adv["actions"] if a["type"] == "ai"][-1]
|
return [a for a in adv["actions"] if a["type"] == "ai"][-1]
|
||||||
|
|
||||||
@@ -145,8 +147,9 @@ def test_the_pager_reads_four_distinct_takes(client):
|
|||||||
def test_editing_a_take_that_is_not_live_edits_that_take(client):
|
def test_editing_a_take_that_is_not_live_edits_that_take(client):
|
||||||
"""The bug, at the level the client's fix depends on.
|
"""The bug, at the level the client's fix depends on.
|
||||||
|
|
||||||
Saving against take 2's own id must land on take 2 — not be refused for
|
Saving against take 2's own id must land on take 2. The request must
|
||||||
being off the path, and not be redirected onto the live row.
|
not be refused for being off the path, and must not be redirected onto
|
||||||
|
the live row.
|
||||||
"""
|
"""
|
||||||
row, takes = _four_takes(client)
|
row, takes = _four_takes(client)
|
||||||
second = takes[1]
|
second = takes[1]
|
||||||
@@ -171,12 +174,13 @@ def test_editing_a_take_leaves_the_live_one_alone(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_take_edit_survives_paging_away_and_back(client):
|
def test_a_take_edit_survives_paging_away_and_back(client):
|
||||||
"""The listing is the pager's only source, so the edit has to be in it.
|
"""The listing is the pager's only source, so the edit must appear in it.
|
||||||
|
|
||||||
(The client caches this list per message; the fix drops that cache after an
|
The client caches this list per message, and the fix drops that cache
|
||||||
edit. If the server ever started answering from a copy of its own, stepping
|
after an edit. If the server ever started answering from a stale copy
|
||||||
away and back would show the words before the edit and nobody would see it
|
of its own, stepping away and back would show the text before the
|
||||||
here — hence the round trip.)
|
edit, and this test would not catch it. That is why this test makes
|
||||||
|
the round trip.
|
||||||
"""
|
"""
|
||||||
row, takes = _four_takes(client)
|
row, takes = _four_takes(client)
|
||||||
_edit(client, takes[1]["id"], "Take 2, rewritten.")
|
_edit(client, takes[1]["id"], "Take 2, rewritten.")
|
||||||
|
|||||||
@@ -1,18 +1,21 @@
|
|||||||
"""Phase 14 SP9 — a turn's takes are grouped by their parent, not by where they sit.
|
"""Phase 14 SP9: a turn's takes are grouped by their parent, not by where
|
||||||
|
they sit.
|
||||||
|
|
||||||
SP4 made every take a node at the same (branch, depth). That coordinate answers
|
SP4 made every take a node at the same (branch, depth). That coordinate
|
||||||
"which takes belong to this turn" right up until one of them is forked onto its
|
answers "which takes belong to this turn" until one of them forks onto its
|
||||||
own branch — at which point it *leaves* the coordinate and reads as the only
|
own branch. At that point it leaves the coordinate and reads as the only
|
||||||
take of its turn, with its siblings unreachable from the line it was taken on.
|
take of its turn, with its siblings unreachable from the line it was taken
|
||||||
|
on.
|
||||||
|
|
||||||
The parent does not move when a branch does, which is the whole of the fix. It
|
The parent does not move when a branch does, which is the whole fix. It
|
||||||
also gets the nesting right for free: takes under C1 and takes under C2 share a
|
also gets the nesting right without extra work: takes under C1 and takes
|
||||||
depth and, until one forks, a branch. Only the parent separates them.
|
under C2 share a depth and, until one forks, a branch. Only the parent
|
||||||
|
separates them.
|
||||||
|
|
||||||
And the branch itself is lazy now. Stepping between takes creates nothing —
|
The branch itself is lazy now. Stepping between takes creates nothing:
|
||||||
looking is free. The fork happens on the first thing *written* below a take the
|
looking is free. The fork happens on the first write below a take the
|
||||||
story moved past, which is the first moment the player has said which line they
|
story moved past, which is the first moment the player has said which line
|
||||||
mean.
|
they mean.
|
||||||
|
|
||||||
python -m pytest tests/test_take_parentage.py -v
|
python -m pytest tests/test_take_parentage.py -v
|
||||||
"""
|
"""
|
||||||
@@ -124,7 +127,8 @@ def _branch_count(adv_id) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def _ai_rows(adv_id) -> list[models.Action]:
|
def _ai_rows(adv_id) -> list[models.Action]:
|
||||||
"""Every AI node ever written, oldest first — live or not, any branch."""
|
"""Every AI node ever written, oldest first. Includes both live and
|
||||||
|
discarded nodes, on any branch."""
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
return (
|
return (
|
||||||
@@ -152,7 +156,7 @@ def _group_size(action_id: int) -> int:
|
|||||||
# ------------------------------------------------------------------ tests
|
# ------------------------------------------------------------------ tests
|
||||||
|
|
||||||
def test_retaken_turn_groups_all_its_takes(client):
|
def test_retaken_turn_groups_all_its_takes(client):
|
||||||
"""The baseline the rest of the file leans on: three takes, one turn."""
|
"""The baseline the rest of the file relies on: three takes, one turn."""
|
||||||
_play(client)
|
_play(client)
|
||||||
_retry(client)
|
_retry(client)
|
||||||
_retry(client)
|
_retry(client)
|
||||||
@@ -178,7 +182,7 @@ def test_a_forked_take_keeps_its_siblings(client):
|
|||||||
first_take = _ai_rows(client.adv_id)[0]
|
first_take = _ai_rows(client.adv_id)[0]
|
||||||
assert first_take.live is False
|
assert first_take.live is False
|
||||||
|
|
||||||
# Writing below it is what forks -- see the next test.
|
# Writing below it is what forks. See the next test.
|
||||||
_play(client, "go back and try this instead", after_id=first_take.id)
|
_play(client, "go back and try this instead", after_id=first_take.id)
|
||||||
|
|
||||||
assert _group_size(first_take.id) == 3, (
|
assert _group_size(first_take.id) == 3, (
|
||||||
@@ -219,9 +223,9 @@ def test_writing_below_a_passed_take_forks_exactly_once(client):
|
|||||||
def test_takes_under_one_parent_do_not_count_takes_under_its_sibling(client):
|
def test_takes_under_one_parent_do_not_count_takes_under_its_sibling(client):
|
||||||
"""The player's own example: 3/3 on one line, 2/2 on the other.
|
"""The player's own example: 3/3 on one line, 2/2 on the other.
|
||||||
|
|
||||||
C1 and C2 are takes of the same turn. What is played *below* each of them
|
C1 and C2 are takes of the same turn. What is played below each of them
|
||||||
is a different turn, and the two must not pool -- they share a depth, and
|
is a different turn, and the two turns must not merge. They share a
|
||||||
until the fork they share a branch too.
|
depth, and until the fork they share a branch too.
|
||||||
"""
|
"""
|
||||||
_play(client)
|
_play(client)
|
||||||
_retry(client) # two takes at this turn: C1, C2 (C2 live)
|
_retry(client) # two takes at this turn: C1, C2 (C2 live)
|
||||||
@@ -276,12 +280,12 @@ def _done_action(response) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
def test_the_streamed_action_carries_the_pager_too(client):
|
def test_the_streamed_action_carries_the_pager_too(client):
|
||||||
"""Found by driving it, not by testing it.
|
"""Found while exercising the app manually, not by a targeted test.
|
||||||
|
|
||||||
A retry's reply *is* the second take of its turn, so it arrives needing a
|
A retry's reply is the second take of its turn, so it arrives needing a
|
||||||
pager. The stream builds its own ActionOut and so missed the annotation:
|
pager. The stream builds its own ActionOut and missed the annotation.
|
||||||
the pager appeared only once the page was reloaded, which is exactly the
|
The pager appeared only once the page was reloaded, which is exactly
|
||||||
moment nobody reloads.
|
the moment nobody reloads.
|
||||||
"""
|
"""
|
||||||
_play(client)
|
_play(client)
|
||||||
r = client.post(f"/api/adventures/{client.adv_id}/retry")
|
r = client.post(f"/api/adventures/{client.adv_id}/retry")
|
||||||
@@ -365,11 +369,12 @@ def test_a_players_own_turn_can_be_played_again(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_retaken_player_turn_is_not_formatted_twice(client):
|
def test_a_retaken_player_turn_is_not_formatted_twice(client):
|
||||||
"""Found by driving it: "> You > You open the door."
|
"""Found while exercising the app manually: "> You > You open the door."
|
||||||
|
|
||||||
The editor is seeded from the stored text, which is already the formatted
|
The editor is seeded from the stored text, which is already in the
|
||||||
form — the same text plain edit puts in the box and writes back verbatim.
|
formatted form. Plain edit puts that same text in the box and writes
|
||||||
Running it through the formatter again doubles the prefix.
|
it back verbatim. Running it through the formatter again doubles the
|
||||||
|
prefix.
|
||||||
"""
|
"""
|
||||||
_play(client, "open the door")
|
_play(client, "open the door")
|
||||||
_play(client, "press on")
|
_play(client, "press on")
|
||||||
@@ -423,7 +428,7 @@ def test_an_ai_turn_at_the_tip_takes_no_branch(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_an_ai_turn_the_story_moved_past_takes_a_branch(client):
|
def test_an_ai_turn_the_story_moved_past_takes_a_branch(client):
|
||||||
"""Retry could never reach here at all — it only ever saw the newest turn."""
|
"""Retry could never reach this case: it only ever saw the newest turn."""
|
||||||
_play(client)
|
_play(client)
|
||||||
_play(client, "press on")
|
_play(client, "press on")
|
||||||
before = _branch_count(client.adv_id)
|
before = _branch_count(client.adv_id)
|
||||||
@@ -445,7 +450,7 @@ def test_the_old_line_still_has_its_continuation(client):
|
|||||||
first_ai = _ai_rows(client.adv_id)[0]
|
first_ai = _ai_rows(client.adv_id)[0]
|
||||||
_take(client, first_ai.id, "")
|
_take(client, first_ai.id, "")
|
||||||
|
|
||||||
# The new take is what this branch tells; "press on" belonged to the other.
|
# The new take is what this branch tells. "press on" belonged to the other.
|
||||||
blob = "\n".join(_path_texts(client))
|
blob = "\n".join(_path_texts(client))
|
||||||
assert "press on" not in blob
|
assert "press on" not in blob
|
||||||
kept = "\n".join(a.text for a in _user_rows(client.adv_id))
|
kept = "\n".join(a.text for a in _user_rows(client.adv_id))
|
||||||
@@ -471,10 +476,10 @@ def test_the_opening_has_no_other_take(client):
|
|||||||
def test_retaking_even_the_newest_player_turn_forks(client):
|
def test_retaking_even_the_newest_player_turn_forks(client):
|
||||||
"""A player turn is never the tip: the reply to it is.
|
"""A player turn is never the tip: the reply to it is.
|
||||||
|
|
||||||
Retaking the *last* thing the player typed still has a story to protect —
|
Retaking the last thing the player typed still puts existing story at
|
||||||
the AI answered it, and that answer was written for the old text. So the
|
risk: the AI answered it, and that answer was written for the old
|
||||||
branch is owed here too, and the guard against forking for nothing only
|
text. So this case must fork too. The guard against forking for
|
||||||
ever fires for a player action with no reply under it.
|
nothing only fires for a player action with no reply under it.
|
||||||
"""
|
"""
|
||||||
_play(client, "open the door")
|
_play(client, "open the door")
|
||||||
before = _branch_count(client.adv_id)
|
before = _branch_count(client.adv_id)
|
||||||
|
|||||||
@@ -1,15 +1,17 @@
|
|||||||
"""Phase 14 SP9 — what a take does to the shared state.
|
"""Phase 14 SP9: what a take does to the shared state.
|
||||||
|
|
||||||
A turn does not only write text. A script mutates `script_state`, the referee
|
A turn does not only write text. A script mutates `script_state`, and the
|
||||||
mutates `world_state`, and both are *shared* — they belong to the adventure, not
|
referee mutates `world_state`. Both are shared: they belong to the
|
||||||
to the node. So playing a turn again has to put them back to where they were
|
adventure, not to the node. Playing a turn again must put them back to
|
||||||
before that turn ran, or the new take stacks its mutations on top of the one it
|
where they were before that turn ran. Otherwise the new take stacks its
|
||||||
replaces and the numbers drift every time the player asks for another take.
|
mutations on top of the one it replaces, and the numbers drift every time
|
||||||
|
the player asks for another take.
|
||||||
|
|
||||||
`retry` has done this since SP4 (`attempts.roll_back_before`). These are the
|
`retry` has provided this guarantee since SP4 (`attempts.roll_back_before`).
|
||||||
same guarantee for the two roads SP9 opened: a take of a turn the story moved
|
These tests confirm the same guarantee for the two roads SP9 opened: a take
|
||||||
past, and a take of the player's own turn. Both create a branch, which is the
|
of a turn the story moved past, and a take of the player's own turn. Both
|
||||||
interesting part — the rollback has to survive leaving the line it was on.
|
create a branch, which matters because the rollback must survive leaving
|
||||||
|
the line it was on.
|
||||||
|
|
||||||
python -m pytest tests/test_take_state.py -v
|
python -m pytest tests/test_take_state.py -v
|
||||||
"""
|
"""
|
||||||
@@ -34,9 +36,9 @@ from app.routers import adventures
|
|||||||
|
|
||||||
SCHEMA = {"player": {"hp": {"min": 0, "max": 100, "initial": 100}}}
|
SCHEMA = {"player": {"hp": {"min": 0, "max": 100, "initial": 100}}}
|
||||||
|
|
||||||
# Ten gold a turn, every turn. A number that only ever goes up is the clearest
|
# Ten gold a turn, every turn. A number that only ever increases makes a
|
||||||
# possible witness to a rollback: if a take stacks instead of replacing, it says
|
# rollback failure obvious: if a take stacks instead of replacing, the gold
|
||||||
# so in one digit.
|
# total is off by exactly one turn's worth.
|
||||||
GOLD_SCRIPT = """
|
GOLD_SCRIPT = """
|
||||||
const modifier = (text) => {
|
const modifier = (text) => {
|
||||||
state.gold = (state.gold || 0) + 10;
|
state.gold = (state.gold || 0) + 10;
|
||||||
@@ -146,7 +148,7 @@ def _rows(adv_id, type_):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_script_runs_once_a_turn(client):
|
def test_the_script_runs_once_a_turn(client):
|
||||||
"""The premise the rest of the file rests on."""
|
"""This test establishes the baseline the rest of the file depends on."""
|
||||||
_play(client)
|
_play(client)
|
||||||
assert _gold(client.adv_id) == 10
|
assert _gold(client.adv_id) == 10
|
||||||
_play(client, "press on")
|
_play(client, "press on")
|
||||||
@@ -156,9 +158,9 @@ def test_the_script_runs_once_a_turn(client):
|
|||||||
def test_a_take_of_a_past_ai_turn_does_not_stack_its_script(client):
|
def test_a_take_of_a_past_ai_turn_does_not_stack_its_script(client):
|
||||||
"""Two turns played, then the first one taken again.
|
"""Two turns played, then the first one taken again.
|
||||||
|
|
||||||
The take leaves the path just before turn one, so the state it starts from
|
The take leaves the path just before turn one. The state it starts from
|
||||||
is the state turn one started from — nothing, not the twenty that two turns
|
is the state turn one started from: zero gold, not the twenty that two
|
||||||
had accumulated. Then its own run adds ten.
|
turns accumulated. The take's own run then adds ten.
|
||||||
"""
|
"""
|
||||||
_play(client)
|
_play(client)
|
||||||
_play(client, "press on")
|
_play(client, "press on")
|
||||||
@@ -182,11 +184,11 @@ def test_a_take_of_a_player_turn_does_not_stack_its_script(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_writing_below_a_passed_take_starts_from_that_take_s_state(client):
|
def test_writing_below_a_passed_take_starts_from_that_take_s_state(client):
|
||||||
"""The `after_id` road, which forks on the way to writing.
|
"""The `after_id` path, which forks while writing.
|
||||||
|
|
||||||
The take being written under produced the state its own turn left behind —
|
The take being written under produced a state of ten gold, from its own
|
||||||
ten — and the turn played on top of it adds the next ten. The twenty the
|
turn. The turn played on top of it adds another ten. The twenty gold
|
||||||
abandoned line reached has nothing to do with this branch.
|
that the abandoned line reached has no effect on this branch.
|
||||||
"""
|
"""
|
||||||
_play(client)
|
_play(client)
|
||||||
r = client.post(f"/api/adventures/{client.adv_id}/retry")
|
r = client.post(f"/api/adventures/{client.adv_id}/retry")
|
||||||
|
|||||||
@@ -1,16 +1,17 @@
|
|||||||
"""Phase 14 SP1 — every existing adventure becomes a tree with one branch.
|
"""Phase 14 SP1: every existing adventure becomes a tree with one branch.
|
||||||
|
|
||||||
The migration this file watches is the one that cannot be re-run: it reads
|
The migration this file tests cannot be re-run. It reads `index` and
|
||||||
`index` and writes `depth`, and from SP2 on the reads follow `depth`. If it
|
writes `depth`, and from SP2 on, the reads follow `depth`. If it mis-maps
|
||||||
mis-maps a row, that row does not error — it *disappears from the story*, which
|
a row, that row does not raise an error. It disappears from the story,
|
||||||
is why the assertions here are about every row rather than about a sample.
|
which is why the assertions here check every row instead of a sample.
|
||||||
|
|
||||||
The fixture is a genuine **schema 45** database, not a current one with an old
|
The fixture is a genuine schema-45 database, not a current one with an
|
||||||
stamp. `create_all` always builds the current schema, so the three tables the
|
old stamp. `create_all` always builds the current schema, so the three
|
||||||
tree touches are dropped and rebuilt from frozen pre-tree DDL below; the
|
tables the tree touches are dropped and rebuilt from the frozen pre-tree
|
||||||
migration then runs its real ALTERs against them, including the one that adds a
|
DDL below. The migration then runs its real ALTER statements against
|
||||||
foreign key. A pre-migration database built any other way (stamp rewound,
|
them, including the one that adds a foreign key. A pre-migration database
|
||||||
columns left in place) would quietly skip the DDL and test half the change.
|
built any other way, such as a rewound stamp with the columns left in
|
||||||
|
place, would silently skip the DDL and test only half the change.
|
||||||
|
|
||||||
python -m pytest tests/test_tree_migration.py -v
|
python -m pytest tests/test_tree_migration.py -v
|
||||||
"""
|
"""
|
||||||
@@ -34,10 +35,11 @@ from app.context import history
|
|||||||
from app.database import Base, SessionLocal, engine, get_db
|
from app.database import Base, SessionLocal, engine, get_db
|
||||||
from app.main import app
|
from app.main import app
|
||||||
|
|
||||||
# The three tables as they stood at schema 45, frozen. This is a snapshot of a
|
# The three tables as they stood at schema 45, frozen. This is a snapshot
|
||||||
# past schema and must NOT be updated to track models.py — the whole point is
|
# of a past schema. It must not be updated to track `models.py`, because
|
||||||
# that it lacks what SP1 adds. SQLite spelling only; the migration's Postgres
|
# the whole point is that it lacks what SP1 adds. This DDL uses SQLite
|
||||||
# half is exercised against a real server at deploy time (see plan/14).
|
# syntax only. The migration's Postgres half is exercised against a real
|
||||||
|
# server at deploy time (see plan/14).
|
||||||
PRE_TREE_DDL = (
|
PRE_TREE_DDL = (
|
||||||
"""
|
"""
|
||||||
CREATE TABLE adventures (
|
CREATE TABLE adventures (
|
||||||
@@ -96,15 +98,17 @@ PRE_TREE_DDL = (
|
|||||||
""",
|
""",
|
||||||
)
|
)
|
||||||
|
|
||||||
# The story of adventure "Gapped": index 3 is missing, because deleting a middle
|
# The story of adventure "Gapped": index 3 is missing, because deleting a
|
||||||
# action never renumbered the ones after it. The gap has to survive as a gap.
|
# middle action never renumbered the ones after it. The migration must
|
||||||
|
# preserve the gap.
|
||||||
GAPPED_INDEXES = (0, 1, 2, 4)
|
GAPPED_INDEXES = (0, 1, 2, 4)
|
||||||
STRAIGHT_INDEXES = (0, 1)
|
STRAIGHT_INDEXES = (0, 1)
|
||||||
# "Blank" holds an action whose text is nothing but whitespace. It is a row of
|
# "Blank" holds an action whose text is nothing but whitespace. It is a
|
||||||
# the adventure but not of the *story*, so a cursor counting covered actions
|
# row of the adventure but not of the story, so a cursor counting covered
|
||||||
# never counted it — and migration 56 has to skip it the same way, using a
|
# actions never counted it. Migration 56 must skip it the same way, using
|
||||||
# frozen copy of the story-text predicate. This is the one duplicated
|
# a frozen copy of the story-text predicate. This predicate is the one
|
||||||
# definition in the change, so it gets the one case that can tell.
|
# duplicated definition in the change, so this is the one test case that
|
||||||
|
# can catch it drifting.
|
||||||
BLANK_INDEXES = (0, 1, 2, 3)
|
BLANK_INDEXES = (0, 1, 2, 3)
|
||||||
BLANK_AT = 2
|
BLANK_AT = 2
|
||||||
|
|
||||||
@@ -113,15 +117,16 @@ BLANK_AT = 2
|
|||||||
def pre_tree():
|
def pre_tree():
|
||||||
"""A schema-45 database with three adventures in it, returned as the ids
|
"""A schema-45 database with three adventures in it, returned as the ids
|
||||||
(gapped, straight, empty) their stories were written under."""
|
(gapped, straight, empty) their stories were written under."""
|
||||||
# Every test in this module shares one temp file, and a setup that fails
|
# Every test in this module shares one temp file, and a setup that
|
||||||
# before its yield never reaches a teardown — so start from empty rather
|
# fails before its yield never reaches a teardown. Start from empty
|
||||||
# than from whatever the last one left.
|
# instead of from whatever the previous test left.
|
||||||
Base.metadata.drop_all(bind=engine)
|
Base.metadata.drop_all(bind=engine)
|
||||||
Base.metadata.create_all(bind=engine)
|
Base.metadata.create_all(bind=engine)
|
||||||
with engine.begin() as conn:
|
with engine.begin() as conn:
|
||||||
# `branches` and the six new columns never existed at 45. Dropping the
|
# `branches` and the six new columns never existed at schema 45.
|
||||||
# tables is the only way to lose the columns: SQLite refuses to drop a
|
# Dropping the tables is the only way to remove the columns.
|
||||||
# column a foreign key names, which is exactly the case for branch_id.
|
# SQLite refuses to drop a column that a foreign key references,
|
||||||
|
# and that is exactly the case for `branch_id`.
|
||||||
for table in ("actions", "memories", "branches", "adventures"):
|
for table in ("actions", "memories", "branches", "adventures"):
|
||||||
conn.execute(text(f"DROP TABLE IF EXISTS {table}"))
|
conn.execute(text(f"DROP TABLE IF EXISTS {table}"))
|
||||||
for ddl in PRE_TREE_DDL:
|
for ddl in PRE_TREE_DDL:
|
||||||
@@ -132,11 +137,12 @@ def pre_tree():
|
|||||||
))
|
))
|
||||||
|
|
||||||
# The cursors as schema 45 held them: counts of covered story actions.
|
# The cursors as schema 45 held them: counts of covered story actions.
|
||||||
# Gapped's story is 0,1,2,4 — so "3 covered" is the node at depth 2 and
|
# Gapped's story is 0, 1, 2, 4, so "3 covered" is the node at depth
|
||||||
# "4 covered" is the node at depth 4, which is the whole reason a count
|
# 2 and "4 covered" is the node at depth 4. This is the whole
|
||||||
# and a depth are not the same number. Straight is caught up past its
|
# reason a count and a depth are not the same number. Straight is
|
||||||
# own end (5 covered, 2 actions), which is a state the older rule left
|
# caught up past its own end (5 covered, 2 actions), a state the
|
||||||
# behind and the clamp used to paper over every post-turn pass.
|
# older rule left behind that a clamp used to mask on every
|
||||||
|
# post-turn pass.
|
||||||
cursors_at = {"Gapped": (3, 4), "Straight": (5, 0), "Empty": (0, 0),
|
cursors_at = {"Gapped": (3, 4), "Straight": (5, 0), "Empty": (0, 0),
|
||||||
"Blank": (3, 0)}
|
"Blank": (3, 0)}
|
||||||
ids = {}
|
ids = {}
|
||||||
@@ -276,18 +282,19 @@ def test_memories_attach_to_the_node_they_summarised(pre_tree):
|
|||||||
assert depth == source_end, "the memory hangs off the last action it covered"
|
assert depth == source_end, "the memory hangs off the last action it covered"
|
||||||
assert branch_id is not None
|
assert branch_id is not None
|
||||||
|
|
||||||
# A hand-written memory summarised no node, so SP7's migration 62 lands it
|
# A hand-written memory summarised no node, so SP7's migration 62
|
||||||
# at depth 0 of its branch. 0 is at or before every fork point, so it stays
|
# lands it at depth 0 of its branch. Depth 0 is at or before every
|
||||||
# visible from exactly the paths it was visible from before — anchoring
|
# fork point, so the memory stays visible from exactly the paths it
|
||||||
# takes nothing out of anybody's existing bank.
|
# was visible from before. Anchoring it this way removes nothing from
|
||||||
|
# anybody's existing memory bank.
|
||||||
manual = rows("SELECT depth, branch_id FROM memories WHERE source_end IS NULL")
|
manual = rows("SELECT depth, branch_id FROM memories WHERE source_end IS NULL")
|
||||||
assert manual and all(depth == 0 and branch is not None for depth, branch in manual)
|
assert manual and all(depth == 0 and branch is not None for depth, branch in manual)
|
||||||
|
|
||||||
|
|
||||||
def test_the_cursors_become_the_nodes_they_named(pre_tree):
|
def test_the_cursors_become_the_nodes_they_named(pre_tree):
|
||||||
"""SP3, migration 56. A count of covered actions and a depth are different
|
"""SP3, migration 56. A count of covered actions and a depth become
|
||||||
numbers the moment the story has a gap in it, which every adventure anyone
|
different numbers as soon as the story has a gap in it, and every
|
||||||
has ever deleted from does."""
|
adventure with a deleted action has one."""
|
||||||
migrations.bootstrap(engine)
|
migrations.bootstrap(engine)
|
||||||
|
|
||||||
def marks(title):
|
def marks(title):
|
||||||
@@ -298,9 +305,9 @@ def test_the_cursors_become_the_nodes_they_named(pre_tree):
|
|||||||
)
|
)
|
||||||
return row
|
return row
|
||||||
|
|
||||||
# Gapped's story is 0,1,2,4. "3 covered" is the *third* action, at depth 2 —
|
# Gapped's story is 0, 1, 2, 4. "3 covered" is the third action, at
|
||||||
# reading the count as a depth would have handed the summarizer node 3,
|
# depth 2. Reading the count as a depth would hand the summarizer node
|
||||||
# which does not exist, and quietly skipped node 4 forever.
|
# 3, which does not exist, and silently skip node 4 forever.
|
||||||
memory_depth, summary_depth, memory_branch, summary_branch = marks("Gapped")
|
memory_depth, summary_depth, memory_branch, summary_branch = marks("Gapped")
|
||||||
assert (memory_depth, summary_depth) == (2, 4)
|
assert (memory_depth, summary_depth) == (2, 4)
|
||||||
root = scalar(
|
root = scalar(
|
||||||
@@ -319,10 +326,11 @@ def test_the_cursors_become_the_nodes_they_named(pre_tree):
|
|||||||
# Nothing covered stays nothing covered, and names no branch.
|
# Nothing covered stays nothing covered, and names no branch.
|
||||||
assert marks("Empty") == (migrations.NO_DEPTH, migrations.NO_DEPTH, None, None)
|
assert marks("Empty") == (migrations.NO_DEPTH, migrations.NO_DEPTH, None, None)
|
||||||
|
|
||||||
# A whitespace-only action is a row but not a story action, so it was never
|
# A whitespace-only action is a row but not a story action, so it was
|
||||||
# counted — "3 covered" of 0,1,[blank],3 is the node at depth 3, not 2. The
|
# never counted. "3 covered" of 0, 1, [blank], 3 is the node at depth
|
||||||
# migration's copy of the story-text predicate is the only place that rule
|
# 3, not 2. The migration's copy of the story-text predicate is the
|
||||||
# is written twice, so this is the case that catches it drifting.
|
# only place that rule is written twice, so this is the test case
|
||||||
|
# that catches it drifting.
|
||||||
assert marks("Blank")[0] == 3
|
assert marks("Blank")[0] == 3
|
||||||
|
|
||||||
# The legacy columns are left exactly as they were: a rolled-back build
|
# The legacy columns are left exactly as they were: a rolled-back build
|
||||||
@@ -333,8 +341,9 @@ def test_the_cursors_become_the_nodes_they_named(pre_tree):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_branch_clause_index_exists(pre_tree):
|
def test_the_branch_clause_index_exists(pre_tree):
|
||||||
"""SP2's reads are only cheap if this exists — and `create_all` does not add
|
"""SP2's reads are only cheap if this index exists. `create_all` does
|
||||||
an index to a table it did not create, which is what migration 52 is for."""
|
not add an index to a table it did not create, which is what
|
||||||
|
migration 52 handles."""
|
||||||
migrations.bootstrap(engine)
|
migrations.bootstrap(engine)
|
||||||
|
|
||||||
assert scalar(
|
assert scalar(
|
||||||
@@ -354,10 +363,10 @@ def test_running_it_again_changes_nothing(pre_tree):
|
|||||||
rows("SELECT id, branch_id, depth FROM memories ORDER BY id"),
|
rows("SELECT id, branch_id, depth FROM memories ORDER BY id"),
|
||||||
)
|
)
|
||||||
|
|
||||||
# Twice through the deploy path, then the data pass on its own — the stamp
|
# Run the deploy path twice, then the data pass on its own. The stamp
|
||||||
# stops the first, the NULL guards stop the second, and a migration that
|
# stops the first run, the NULL guards stop the second, and a
|
||||||
# only survives because of the stamp is one bad rescue away from doubling
|
# migration that only survives because of the stamp is one bad rescue
|
||||||
# every branch.
|
# away from doubling every branch.
|
||||||
migrations.bootstrap(engine)
|
migrations.bootstrap(engine)
|
||||||
with engine.begin() as conn:
|
with engine.begin() as conn:
|
||||||
migrations._backfill_tree(conn)
|
migrations._backfill_tree(conn)
|
||||||
@@ -379,8 +388,8 @@ def test_running_it_again_changes_nothing(pre_tree):
|
|||||||
def client(monkeypatch):
|
def client(monkeypatch):
|
||||||
"""The app on a migrated database, so new rows go through the real writers.
|
"""The app on a migrated database, so new rows go through the real writers.
|
||||||
|
|
||||||
Everything the migration fixes is only half the job: no migration will ever
|
Fixing existing rows is only half the job. No migration ever revisits
|
||||||
visit a row written after it ran, and a row without a branch is a row no
|
a row written after it ran, and a row without a branch is a row no
|
||||||
read can see.
|
read can see.
|
||||||
"""
|
"""
|
||||||
Base.metadata.drop_all(bind=engine)
|
Base.metadata.drop_all(bind=engine)
|
||||||
@@ -442,11 +451,12 @@ def test_a_blank_adventure_has_a_branch_before_anything_is_played(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_hand_written_memory_is_anchored_at_the_head(client):
|
def test_a_hand_written_memory_is_anchored_at_the_head(client):
|
||||||
"""SP7: nothing carries a NULL depth any more.
|
"""SP7: nothing carries a NULL depth anymore.
|
||||||
|
|
||||||
On an adventure with no story yet the head is NO_DEPTH (-1), which reads as
|
On an adventure with no story yet, the head is `NO_DEPTH` (-1), which
|
||||||
"before the first node" and so is in range of every branch — right for a
|
reads as "before the first node" and so falls in range of every
|
||||||
note written before anything has happened.
|
branch. This is correct for a note written before anything has
|
||||||
|
happened.
|
||||||
"""
|
"""
|
||||||
adventure_id = client.post("/api/adventures", json={}).json()["id"]
|
adventure_id = client.post("/api/adventures", json={}).json()["id"]
|
||||||
|
|
||||||
@@ -466,9 +476,10 @@ def test_a_hand_written_memory_is_anchored_at_the_head(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_deleting_a_branch_takes_its_nodes_with_it(client):
|
def test_deleting_a_branch_takes_its_nodes_with_it(client):
|
||||||
"""`ON DELETE CASCADE` on both `branch_id` columns, so the database removes a
|
"""`ON DELETE CASCADE` on both `branch_id` columns means the database
|
||||||
branch's nodes rather than any code remembering to. SP7 ships delete-a-branch
|
removes a branch's nodes instead of relying on application code to
|
||||||
on top of exactly this, and nothing else has to load a branch to do it."""
|
remember to. SP7 builds delete-a-branch directly on top of this, and
|
||||||
|
no other code needs to load a branch to do it."""
|
||||||
adventure_id = client.post(
|
adventure_id = client.post(
|
||||||
"/api/adventures", json={"scenario_id": client.scenario_id}
|
"/api/adventures", json={"scenario_id": client.scenario_id}
|
||||||
).json()["id"]
|
).json()["id"]
|
||||||
@@ -509,8 +520,8 @@ def test_deleting_an_adventure_takes_its_branch_with_it(client):
|
|||||||
|
|
||||||
|
|
||||||
def test_deleting_the_newest_action_moves_the_head_back(client):
|
def test_deleting_the_newest_action_moves_the_head_back(client):
|
||||||
"""The head is a cache, and a cache that only ever moves forward is wrong
|
"""The head is a cache. A cache that only ever moves forward becomes
|
||||||
the first time someone undoes a turn."""
|
wrong the first time someone undoes a turn."""
|
||||||
adventure_id = client.post(
|
adventure_id = client.post(
|
||||||
"/api/adventures", json={"scenario_id": client.scenario_id}
|
"/api/adventures", json={"scenario_id": client.scenario_id}
|
||||||
).json()["id"]
|
).json()["id"]
|
||||||
@@ -540,10 +551,11 @@ def test_deleting_the_newest_action_moves_the_head_back(client):
|
|||||||
|
|
||||||
# ---------------------------------------- SP4: variants become sibling rows
|
# ---------------------------------------- SP4: variants become sibling rows
|
||||||
|
|
||||||
# One turn's retry history as schema 45 stored it: a JSON array on the AI row,
|
# One turn's retry history as schema 45 stored it: a JSON array on the
|
||||||
# with `variant_index` naming the entry `text` mirrors. The live one is
|
# AI row, with `variant_index` naming the entry `text` mirrors. The live
|
||||||
# deliberately not the last written — a migration that assumed it was would
|
# entry is deliberately not the last one written. A migration that
|
||||||
# look right on every fixture where the player never went back.
|
# assumed it was would still pass on every fixture where the player
|
||||||
|
# never paged back.
|
||||||
RETRY_VARIANTS = [
|
RETRY_VARIANTS = [
|
||||||
{"text": "Attempt one.", "reasoning": None,
|
{"text": "Attempt one.", "reasoning": None,
|
||||||
"script_state": {"gold": 10}, "created_at": "2026-01-01T00:00:00",
|
"script_state": {"gold": 10}, "created_at": "2026-01-01T00:00:00",
|
||||||
@@ -563,10 +575,10 @@ RETRY_VARIANTS = [
|
|||||||
]
|
]
|
||||||
LIVE_VARIANT = 1
|
LIVE_VARIANT = 1
|
||||||
|
|
||||||
# The whole turn's assembled prompt, stored once. The attempts differ only in
|
# The whole turn's assembled prompt, stored once. The attempts differ
|
||||||
# the three slices above, which is the arrangement SP4 has to preserve — giving
|
# only in the three slices above, and SP4 must preserve that arrangement.
|
||||||
# each sibling a copy of this would multiply the biggest column in the database
|
# Giving each sibling a copy of this prompt would multiply the largest
|
||||||
# by the retry count.
|
# column in the database by the retry count.
|
||||||
RETRY_SNAPSHOT = {
|
RETRY_SNAPSHOT = {
|
||||||
"sections": [{"label": "history", "text": "A long prompt.", "tokens": 4}],
|
"sections": [{"label": "history", "text": "A long prompt.", "tokens": 4}],
|
||||||
"prompt": {"system": "S", "story": "A long prompt."},
|
"prompt": {"system": "S", "story": "A long prompt."},
|
||||||
@@ -578,11 +590,13 @@ RETRY_SNAPSHOT = {
|
|||||||
|
|
||||||
@pytest.fixture()
|
@pytest.fixture()
|
||||||
def pre_split():
|
def pre_split():
|
||||||
"""A schema-45 adventure with one retried turn, plus a plain turn each side.
|
"""A schema-45 adventure with one retried turn, plus a plain turn on
|
||||||
|
each side.
|
||||||
|
|
||||||
Separate from `pre_tree` so SP1's assertions keep counting what they were
|
This fixture is separate from `pre_tree` so SP1's assertions keep
|
||||||
written to count. The story is: 0 start, 1 do, 2 ai (three attempts), 3 do,
|
counting what they were written to count. The story is: 0 start, 1
|
||||||
and the adventure's live state is the one attempt 1 produced.
|
do, 2 ai (three attempts), 3 do. The adventure's live state is the
|
||||||
|
one attempt 1 produced.
|
||||||
"""
|
"""
|
||||||
Base.metadata.drop_all(bind=engine)
|
Base.metadata.drop_all(bind=engine)
|
||||||
Base.metadata.create_all(bind=engine)
|
Base.metadata.create_all(bind=engine)
|
||||||
@@ -603,8 +617,9 @@ def pre_split():
|
|||||||
adventure_id = conn.execute(
|
adventure_id = conn.execute(
|
||||||
text("SELECT id FROM adventures WHERE title = 'Retried'")
|
text("SELECT id FROM adventures WHERE title = 'Retried'")
|
||||||
).scalar()
|
).scalar()
|
||||||
# `state_before` on each row: the scoreboard as that action found it.
|
# `state_before` on each row records the script state as that
|
||||||
# SP4 reads them one row along to build the `state_after` pair.
|
# action found it. SP4 reads each row's `state_before` from the
|
||||||
|
# next row to build the `state_after` pair.
|
||||||
for index, kind, before in (
|
for index, kind, before in (
|
||||||
(0, "start", None), (1, "do", {"gold": 0}),
|
(0, "start", None), (1, "do", {"gold": 0}),
|
||||||
(2, "ai", {"gold": 0}), (3, "do", {"gold": 20}),
|
(2, "ai", {"gold": 0}), (3, "do", {"gold": 20}),
|
||||||
@@ -653,7 +668,7 @@ def test_each_attempt_becomes_a_row_at_the_turns_coordinate(pre_split):
|
|||||||
'AND "index" = 2', a=pre_split,
|
'AND "index" = 2', a=pre_split,
|
||||||
)
|
)
|
||||||
assert len(coordinates) == 1
|
assert len(coordinates) == 1
|
||||||
# ...and the rest of the story is untouched, still one row per turn.
|
# The rest of the story is untouched, still one row per turn.
|
||||||
assert scalar("SELECT count(*) FROM actions WHERE adventure_id = :a", a=pre_split) == 6
|
assert scalar("SELECT count(*) FROM actions WHERE adventure_id = :a", a=pre_split) == 6
|
||||||
|
|
||||||
|
|
||||||
@@ -703,10 +718,11 @@ def test_each_attempt_keeps_the_outcome_it_produced(pre_split):
|
|||||||
]
|
]
|
||||||
assert parsed[0][2] == {"player": {"hp": 95}}
|
assert parsed[0][2] == {"player": {"hp": 95}}
|
||||||
assert parsed[1][2] == {"player": {"hp": 60}}
|
assert parsed[1][2] == {"player": {"hp": 60}}
|
||||||
# Attempt three recorded no world state — an adventure with no RPG layer,
|
# Attempt three recorded no world state: either an adventure with no
|
||||||
# or a take made before the column existed. It stays NULL rather than
|
# RPG layer, or a take made before the column existed. It stays NULL
|
||||||
# borrowing a neighbour's, and switching to it leaves the RPG layer alone:
|
# instead of borrowing a neighbor's, so switching to it leaves the RPG
|
||||||
# exactly what `apply_variant` did with an entry that had no world state.
|
# layer alone. This matches what `apply_variant` did with an entry
|
||||||
|
# that had no world state.
|
||||||
assert parsed[2][2] is None
|
assert parsed[2][2] is None
|
||||||
|
|
||||||
|
|
||||||
@@ -739,7 +755,8 @@ def test_the_split_survives_being_run_again(pre_split):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_migrated_story_reads_back_as_one_turn(pre_split):
|
def test_the_migrated_story_reads_back_as_one_turn(pre_split):
|
||||||
"""The point of all of it: the reads see a four-action story, not six."""
|
"""This is the point of the whole migration: reads see a four-action
|
||||||
|
story, not six."""
|
||||||
migrations.bootstrap(engine)
|
migrations.bootstrap(engine)
|
||||||
|
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
"""End-to-end HTTP tests for undo/retry state revert, driving real turns through
|
"""End-to-end HTTP tests for undo and retry state revert. These tests drive
|
||||||
the actual routes + scripting engine with only the LLM provider mocked.
|
real turns through the actual routes and scripting engine, with only the LLM
|
||||||
|
provider mocked.
|
||||||
|
|
||||||
A script's output hook adds 10 gold each turn; we assert the shared scoreboard
|
A script's output hook adds 10 gold each turn. These tests confirm that the
|
||||||
behaves correctly across play / undo / retry.
|
adventure's stored gold total stays correct across play, undo, and retry.
|
||||||
|
|
||||||
python -m pytest tests/test_turn_flow_integration.py -v
|
python -m pytest tests/test_turn_flow_integration.py -v
|
||||||
"""
|
"""
|
||||||
@@ -64,7 +65,7 @@ def client(monkeypatch):
|
|||||||
adv_id, user_id = adv.id, user.id
|
adv_id, user_id = adv.id, user.id
|
||||||
setup.close()
|
setup.close()
|
||||||
|
|
||||||
# Force a real (non-demo) turn that uses our fake provider.
|
# Force a real, non-demo turn that uses the fake provider.
|
||||||
monkeypatch.setattr(adventures, "OpenAICompatibleProvider", FakeProvider)
|
monkeypatch.setattr(adventures, "OpenAICompatibleProvider", FakeProvider)
|
||||||
monkeypatch.setattr(auth, "resolve_provider_config", lambda s: auth.ProviderConfig(
|
monkeypatch.setattr(auth, "resolve_provider_config", lambda s: auth.ProviderConfig(
|
||||||
"http://fake", "k", "test-model", False))
|
"http://fake", "k", "test-model", False))
|
||||||
@@ -107,7 +108,7 @@ def test_play_then_undo_reverts_gold(client):
|
|||||||
|
|
||||||
r = client.post(f"/api/adventures/{client.adv_id}/undo")
|
r = client.post(f"/api/adventures/{client.adv_id}/undo")
|
||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
assert _state(client.adv_id) == {} # scoreboard rolled back
|
assert _state(client.adv_id) == {} # gold reverted to zero
|
||||||
|
|
||||||
|
|
||||||
def test_two_turns_then_undo_reverts_only_last(client):
|
def test_two_turns_then_undo_reverts_only_last(client):
|
||||||
@@ -123,7 +124,8 @@ def test_retry_does_not_double_apply_gold(client):
|
|||||||
_play(client)
|
_play(client)
|
||||||
assert _state(client.adv_id) == {"gold": 10}
|
assert _state(client.adv_id) == {"gold": 10}
|
||||||
|
|
||||||
# Before the fix this produced 20 (output hook ran twice); now it stays 10.
|
# Before the fix, this produced 20 because the output hook ran twice.
|
||||||
|
# Now it stays 10.
|
||||||
r = client.post(f"/api/adventures/{client.adv_id}/retry")
|
r = client.post(f"/api/adventures/{client.adv_id}/retry")
|
||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
assert _state(client.adv_id) == {"gold": 10}
|
assert _state(client.adv_id) == {"gold": 10}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
"""Unit tests for the RPG world-state engine (Phase 12): delta extraction and
|
"""Unit tests for the RPG world-state engine (Phase 12): delta extraction and
|
||||||
the clamp/cooldown/milestone referee.
|
the code that enforces clamp, cooldown, and milestone rules.
|
||||||
|
|
||||||
python -m pytest tests/test_worldstate.py -v
|
python -m pytest tests/test_worldstate.py -v
|
||||||
"""
|
"""
|
||||||
@@ -67,7 +67,8 @@ def test_text_stat_noop_when_unchanged():
|
|||||||
def test_per_npc_distinct_stats():
|
def test_per_npc_distinct_stats():
|
||||||
ws, _ = w.apply_delta(fresh(), SCHEMA, {"npc.drake.ferocity": 20}, 1)
|
ws, _ = w.apply_delta(fresh(), SCHEMA, {"npc.drake.ferocity": 20}, 1)
|
||||||
assert ws["npc"]["drake"]["ferocity"] == 70
|
assert ws["npc"]["drake"]["ferocity"] == 70
|
||||||
# gwen has no "ferocity" stat, drake has no "trust" — cross paths are rejected.
|
# gwen has no `ferocity` stat, and drake has no `trust` stat. Cross paths
|
||||||
|
# are rejected.
|
||||||
ws, report = w.apply_delta(ws, SCHEMA, {"npc.gwen.ferocity": 5, "npc.bogus.trust": 5}, 2)
|
ws, report = w.apply_delta(ws, SCHEMA, {"npc.gwen.ferocity": 5, "npc.bogus.trust": 5}, 2)
|
||||||
reasons = {r["reason"] for r in report["rejected"]}
|
reasons = {r["reason"] for r in report["rejected"]}
|
||||||
assert reasons == {"unknown npc stat", "unknown npc"}
|
assert reasons == {"unknown npc stat", "unknown npc"}
|
||||||
@@ -134,7 +135,8 @@ def test_flags_toggle_both_ways():
|
|||||||
ws, report = w.apply_delta(ws, SCHEMA, {"flags.has_key": True}, 1)
|
ws, report = w.apply_delta(ws, SCHEMA, {"flags.has_key": True}, 1)
|
||||||
assert ws["flags"]["has_key"] is True
|
assert ws["flags"]["has_key"] is True
|
||||||
assert report["applied"]
|
assert report["applied"]
|
||||||
# flip back off — flags are two-way (unlike sticky milestones).
|
# Setting it back to false works, because flags are two-way, unlike
|
||||||
|
# sticky milestones.
|
||||||
ws, _ = w.apply_delta(ws, SCHEMA, {"flags.has_key": False}, 2)
|
ws, _ = w.apply_delta(ws, SCHEMA, {"flags.has_key": False}, 2)
|
||||||
assert ws["flags"]["has_key"] is False
|
assert ws["flags"]["has_key"] is False
|
||||||
# setting to the same value is a no-op.
|
# setting to the same value is a no-op.
|
||||||
@@ -150,8 +152,9 @@ def test_flag_rejects_non_bool_and_unknown():
|
|||||||
|
|
||||||
|
|
||||||
def test_override_sets_absolute_value_bypassing_cap():
|
def test_override_sets_absolute_value_bypassing_cap():
|
||||||
# A manual override isn't policed by max_delta_per_turn like a turn is —
|
# A manual override does not obey max_delta_per_turn the way a turn
|
||||||
# it sets the value directly (still clamped to min/max).
|
# does. It sets the value directly, though the value is still clamped
|
||||||
|
# to min/max.
|
||||||
ws, report = w.apply_override(fresh(), SCHEMA, {"player.hp": 10})
|
ws, report = w.apply_override(fresh(), SCHEMA, {"player.hp": 10})
|
||||||
assert ws["player"]["hp"] == 10
|
assert ws["player"]["hp"] == 10
|
||||||
assert report["applied"] == [{"path": "player.hp", "old": 100, "new": 10}]
|
assert report["applied"] == [{"path": "player.hp", "old": 100, "new": 10}]
|
||||||
@@ -202,9 +205,10 @@ def test_override_rejects_unknown_and_bad_type():
|
|||||||
|
|
||||||
def test_reference_includes_desc_and_bands_independently():
|
def test_reference_includes_desc_and_bands_independently():
|
||||||
guide = w.render_reference(SCHEMA)
|
guide = w.render_reference(SCHEMA)
|
||||||
# hp has both a description and a band ladder.
|
# hp has both a description and a set of value bands.
|
||||||
assert "very weak" in guide and "range 0–100" in guide
|
assert "very weak" in guide and "range 0–100" in guide
|
||||||
# day (a counter here has no desc/bands) contributes nothing; flags show desc.
|
# A counter like day has no desc or bands, so it contributes nothing
|
||||||
|
# here. Flags do show their desc.
|
||||||
assert "has_key (flag) — Holds the key." in guide
|
assert "has_key (flag) — Holds the key." in guide
|
||||||
# NPCs contribute their own description and per-NPC stat lines.
|
# NPCs contribute their own description and per-NPC stat lines.
|
||||||
assert "NPC Gwen (gwen) — A loyal ranger." in guide
|
assert "NPC Gwen (gwen) — A loyal ranger." in guide
|
||||||
@@ -232,7 +236,8 @@ def test_extract_no_block():
|
|||||||
|
|
||||||
|
|
||||||
def test_extract_prose_ending_in_brace_not_eaten():
|
def test_extract_prose_ending_in_brace_not_eaten():
|
||||||
# A bare object with no dotted keys is not a delta — leave the text alone.
|
# A bare object with no dotted keys is not a delta, so the text stays
|
||||||
|
# as is.
|
||||||
clean, delta = w.extract_delta('He said {this}')
|
clean, delta = w.extract_delta('He said {this}')
|
||||||
assert delta == {}
|
assert delta == {}
|
||||||
assert clean == "He said {this}"
|
assert clean == "He said {this}"
|
||||||
|
|||||||
@@ -36,8 +36,9 @@ SCHEMA = {
|
|||||||
"milestones": {"win": {"desc": "Win the fight"}},
|
"milestones": {"win": {"desc": "Win the fight"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
# The faked model narrates and appends a delta that exceeds the per-turn cap
|
# The faked model narrates and appends a delta that exceeds the per-turn
|
||||||
# (so we can see the engine clamp it), flips a flag, and completes a milestone.
|
# cap, so the test can confirm the engine clamps it. It also flips a flag
|
||||||
|
# and completes a milestone.
|
||||||
AI_REPLY = (
|
AI_REPLY = (
|
||||||
"The goblin's blade bites deep and Gwen nods at your resolve.\n\n"
|
"The goblin's blade bites deep and Gwen nods at your resolve.\n\n"
|
||||||
'```state\n{"player.hp": -80, "npc.gwen.trust": 15, "flags.alarm": true, "milestones.win": true}\n```'
|
'```state\n{"player.hp": -80, "npc.gwen.trust": 15, "flags.alarm": true, "milestones.win": true}\n```'
|
||||||
@@ -131,7 +132,7 @@ def test_turn_applies_clamped_delta_and_strips_block(client):
|
|||||||
# The state block is not shown to the player.
|
# The state block is not shown to the player.
|
||||||
assert "```state" not in _last_ai_text(client.adv_id)
|
assert "```state" not in _last_ai_text(client.adv_id)
|
||||||
assert "goblin's blade" in _last_ai_text(client.adv_id)
|
assert "goblin's blade" in _last_ai_text(client.adv_id)
|
||||||
# ...but the raw model reply (with the block) is kept for the Insights view.
|
# The raw model reply, including the block, is kept for the Insights view.
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
snap = db.get(models.Adventure, client.adv_id).actions[-1].context_snapshot
|
snap = db.get(models.Adventure, client.adv_id).actions[-1].context_snapshot
|
||||||
@@ -175,7 +176,7 @@ def test_override_world_state_endpoint(client):
|
|||||||
# persisted to the DB, not just the response.
|
# persisted to the DB, not just the response.
|
||||||
assert _world(client.adv_id)["player"]["hp"] == 5
|
assert _world(client.adv_id)["player"]["hp"] == 5
|
||||||
|
|
||||||
# bypasses max_delta_per_turn (30) — a direct correction, not a turn.
|
# This bypasses max_delta_per_turn (30) because it is a direct correction, not a turn.
|
||||||
r = client.put(f"/api/adventures/{client.adv_id}/world-state", json={"player.hp": 100})
|
r = client.put(f"/api/adventures/{client.adv_id}/world-state", json={"player.hp": 100})
|
||||||
assert r.json()["state"]["player"]["hp"] == 100
|
assert r.json()["state"]["player"]["hp"] == 100
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
"""Build a bootable adventure that has actually gone two ways (Phase 14, SP7).
|
"""Build a bootable adventure that has actually gone two ways (Phase 14, SP7).
|
||||||
|
|
||||||
`tools.stress_session --keep` answers the scroll question and nothing else: its
|
`tools.stress_session --keep` answers the scrolling question and nothing else.
|
||||||
world state and script state are both empty, so it cannot show whether a branch
|
Its world state and script state are both empty, so it cannot show whether a
|
||||||
switch puts the scoreboard back. This builds the small counterpart — a stat
|
branch switch restores them. This script builds the small counterpart: a stat
|
||||||
schema, a script that counts gold, two attempts at one turn that do visibly
|
schema, a script that counts gold, two attempts at one turn that do visibly
|
||||||
different damage, a fork, and a hand-written memory on each branch.
|
different damage, a fork, and a hand-written memory on each branch.
|
||||||
|
|
||||||
@@ -17,9 +17,10 @@ branches apart, and then four of them went on showing the branch just left.
|
|||||||
AIDND_DB_PATH=/tmp/branches.db .venv/Scripts/python.exe \\
|
AIDND_DB_PATH=/tmp/branches.db .venv/Scripts/python.exe \\
|
||||||
-m uvicorn app.main:app --port 8010
|
-m uvicorn app.main:app --port 8010
|
||||||
|
|
||||||
Then open http://127.0.0.1:8010/ — the SPA is served out of `frontend/dist`, so
|
Then open http://127.0.0.1:8010/. The SPA is served out of `frontend/dist`, so
|
||||||
run `npm run build` first if it is stale. Expect hp 60 on "The hard way down"
|
run `npm run build` first if that directory is stale. Expect hp 60 on "The hard
|
||||||
and hp 95 on the fork, and expect both to move the moment you switch.
|
way down" and hp 95 on the fork, and expect both to change as soon as you
|
||||||
|
switch.
|
||||||
|
|
||||||
No LLM is called: the provider is scripted and its replies carry their own
|
No LLM is called: the provider is scripted and its replies carry their own
|
||||||
fenced `state` blocks.
|
fenced `state` blocks.
|
||||||
@@ -119,9 +120,9 @@ def play(text):
|
|||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
|
|
||||||
|
|
||||||
# Turn one: a scratch. Retried into a beating. The story continues from the
|
# Turn one is a scratch, retried into a beating. The story continues from the
|
||||||
# beating, so the scratch is the attempt left behind — and the two differ by 35
|
# beating, so the scratch is the attempt left behind. The two differ by 35 hit
|
||||||
# hit points, which is the number a switch has to put back.
|
# points, which is the number a switch has to restore.
|
||||||
ScriptedProvider.replies = [
|
ScriptedProvider.replies = [
|
||||||
"You ease the door open and a nail catches your wrist.\n"
|
"You ease the door open and a nail catches your wrist.\n"
|
||||||
"```state\n{\"player.hp\": -5}\n```",
|
"```state\n{\"player.hp\": -5}\n```",
|
||||||
|
|||||||
+14
-12
@@ -4,7 +4,8 @@ Query *counts* are easy to see and have never been the problem here. Both
|
|||||||
egress blowouts this project has had were one query fetching a column nobody
|
egress blowouts this project has had were one query fetching a column nobody
|
||||||
read: `context_snapshot` on every action, then every memory's embedding on
|
read: `context_snapshot` on every action, then every memory's embedding on
|
||||||
every turn. Counting statements would have shown nothing wrong in either case,
|
every turn. Counting statements would have shown nothing wrong in either case,
|
||||||
and at development scale — ten rows, no embeddings — so would a stopwatch.
|
and at development scale, meaning ten rows and no embeddings, so would a
|
||||||
|
stopwatch.
|
||||||
|
|
||||||
So this meter measures bytes, and it measures them at the only place the truth
|
So this meter measures bytes, and it measures them at the only place the truth
|
||||||
is available: the DBAPI cursor, after the driver has decoded a row and before
|
is available: the DBAPI cursor, after the driver has decoded a row and before
|
||||||
@@ -20,9 +21,9 @@ Usage:
|
|||||||
print(meter.render())
|
print(meter.render())
|
||||||
|
|
||||||
`attach()` wraps the pool's connection factory, so it reuses the engine the app
|
`attach()` wraps the pool's connection factory, so it reuses the engine the app
|
||||||
already configured rather than rebuilding one beside it — nothing about
|
already configured rather than building a second one. Nothing about connect
|
||||||
connect args, pre-ping or the SQLite foreign-key pragma has to be repeated
|
args, pre-ping, or the SQLite foreign-key pragma is repeated here, and the
|
||||||
here, and drift between the metered engine and the real one is impossible.
|
metered engine cannot diverge from the real one.
|
||||||
Existing pooled connections are dropped first, so a connection opened before
|
Existing pooled connections are dropped first, so a connection opened before
|
||||||
attaching cannot quietly stay unmetered.
|
attaching cannot quietly stay unmetered.
|
||||||
|
|
||||||
@@ -68,8 +69,8 @@ def value_bytes(value) -> int:
|
|||||||
if isinstance(value, (int, float)):
|
if isinstance(value, (int, float)):
|
||||||
return 8
|
return 8
|
||||||
if isinstance(value, (dict, list)):
|
if isinstance(value, (dict, list)):
|
||||||
# A JSON column the driver already parsed (psycopg does; SQLite does
|
# A JSON column the driver already parsed. psycopg parses it and SQLite
|
||||||
# not). Separators match what a database emits — no spaces.
|
# does not. The separators match what a database emits, with no spaces.
|
||||||
return len(json.dumps(value, separators=(",", ":"), default=str).encode("utf-8"))
|
return len(json.dumps(value, separators=(",", ":"), default=str).encode("utf-8"))
|
||||||
return len(str(value).encode("utf-8"))
|
return len(str(value).encode("utf-8"))
|
||||||
|
|
||||||
@@ -82,8 +83,9 @@ def _table_of(statement: str) -> str:
|
|||||||
match = _TABLE_RE.search(statement)
|
match = _TABLE_RE.search(statement)
|
||||||
if match:
|
if match:
|
||||||
return match.group(1).lower()
|
return match.group(1).lower()
|
||||||
# No table to name — BEGIN, a PRAGMA, a savepoint. Group those under the
|
# The statement names no table, such as BEGIN, a PRAGMA, or a savepoint.
|
||||||
# keyword so they stay countable instead of collapsing into one "?" bucket.
|
# Group those under the keyword, so they stay countable rather than collapse
|
||||||
|
# into one "?" bucket.
|
||||||
head = statement.strip().split(None, 1)
|
head = statement.strip().split(None, 1)
|
||||||
return head[0].lower()[:20] if head else "(empty)"
|
return head[0].lower()[:20] if head else "(empty)"
|
||||||
|
|
||||||
@@ -236,10 +238,10 @@ class _MeteredConnection:
|
|||||||
class _MeteredCursor:
|
class _MeteredCursor:
|
||||||
"""Counts the bytes of every row handed back.
|
"""Counts the bytes of every row handed back.
|
||||||
|
|
||||||
The fetch methods are wrapped rather than `execute`, because what a
|
The wrapper covers the fetch methods rather than `execute`, because what a
|
||||||
statement *costs* is not knowable when it is sent — `SELECT * FROM
|
statement costs is not known when it is sent. `SELECT * FROM memories` and
|
||||||
memories` and `SELECT count(*) FROM memories` look alike going out and
|
`SELECT count(*) FROM memories` look alike going out and differ by three
|
||||||
differ by three megabytes coming back.
|
megabytes coming back.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, cursor, meter: Meter) -> None:
|
def __init__(self, cursor, meter: Meter) -> None:
|
||||||
|
|||||||
@@ -16,12 +16,12 @@ import json
|
|||||||
import random
|
import random
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
# `stress_session` FIRST, and it is not a style preference. Importing it is what
|
# Import `stress_session` first. This is not a style preference. Importing it is
|
||||||
# points `AIDND_DB_PATH` at a throwaway file, and `app.database` reads that at
|
# what points `AIDND_DB_PATH` at a throwaway file, and `app.database` reads that
|
||||||
# module scope — so an `app` import above this line silently runs the whole
|
# at module scope, so an `app` import above this line runs the whole fixture
|
||||||
# fixture against `backend/data.db` instead. It fails by *working*: the first
|
# against `backend/data.db` instead. The failure looks like success: the first
|
||||||
# run seeds a synthetic user and adventure into the local database and reports
|
# run seeds a synthetic user and adventure into the local database and reports
|
||||||
# perfectly good numbers, and only the second run trips over the unique email.
|
# plausible numbers, and only the second run fails on the unique email.
|
||||||
from tools import stress_session as stress # noqa: I001 (see above)
|
from tools import stress_session as stress # noqa: I001 (see above)
|
||||||
|
|
||||||
from app import bundle, models, tree
|
from app import bundle, models, tree
|
||||||
@@ -31,8 +31,9 @@ from app.database import SessionLocal
|
|||||||
def _v1_shape(v2: dict) -> dict:
|
def _v1_shape(v2: dict) -> dict:
|
||||||
"""The same story as v1 would have written it, for a like-for-like count.
|
"""The same story as v1 would have written it, for a like-for-like count.
|
||||||
|
|
||||||
One entry per turn, the siblings folded back into a `variants` array, no
|
There is one entry per turn, with the siblings collected back into a
|
||||||
coordinates and no outcomes — which is exactly what v1 could carry.
|
`variants` array, no coordinates, and no outcomes. That is what version 1
|
||||||
|
could carry.
|
||||||
"""
|
"""
|
||||||
turns: dict[tuple[int, int], list[dict]] = {}
|
turns: dict[tuple[int, int], list[dict]] = {}
|
||||||
order: list[tuple[int, int]] = []
|
order: list[tuple[int, int]] = []
|
||||||
@@ -89,11 +90,12 @@ def main(argv=None) -> int:
|
|||||||
try:
|
try:
|
||||||
adventure = db.get(models.Adventure, adv_id)
|
adventure = db.get(models.Adventure, adv_id)
|
||||||
if forks:
|
if forks:
|
||||||
# Twenty divergences off one line, each a little deeper — the shape
|
# Twenty divergences off one line, each a little deeper, which is
|
||||||
# SP5 measured the fork cost on. The nodes are chosen up front and
|
# the shape SP5 measured the fork cost on. The nodes are chosen up
|
||||||
# the session is flushed after every fork: `fork` moves a row onto
|
# front and the session is flushed after every fork. `fork` moves a
|
||||||
# its new branch, and with `autoflush=False` a query issued before
|
# row onto its new branch, and with `autoflush=False` a query issued
|
||||||
# that move is written still finds the node where it used to be.
|
# before that move is written still finds the node at its old
|
||||||
|
# location.
|
||||||
root = adventure.head_branch_id
|
root = adventure.head_branch_id
|
||||||
candidates = [
|
candidates = [
|
||||||
node.id for node in
|
node.id for node in
|
||||||
|
|||||||
@@ -5,12 +5,13 @@ with one stat, because a map only cares about the shape. A screenshot cares
|
|||||||
about everything else: the world-state rail wants a scenario with bands, flags,
|
about everything else: the world-state rail wants a scenario with bands, flags,
|
||||||
milestones and a named cast, and the story wants prose somebody would read.
|
milestones and a named cast, and the story wants prose somebody would read.
|
||||||
|
|
||||||
So this drives the seeded **Bandit Camp** demo — the same scenario the older
|
So this script drives the seeded Bandit Camp demo through eight turns of written
|
||||||
images were shot on, so the set stays one product — through eight turns of
|
prose and written deltas. That is the same scenario the older images were shot
|
||||||
written prose and written deltas, and forks three discarded takes onto branches
|
on, which keeps the set consistent. It then forks three discarded attempts onto
|
||||||
of their own — one of them off a branch, so the map has to nest. It leaves the reader on the first telling, on a
|
branches of their own, one of them off a branch, so the map has to nest. It
|
||||||
turn that has a second take, so one screen shows the world state, the take
|
leaves the reader on the first telling, on a turn that has a second attempt, so
|
||||||
pager and the branch rail at once.
|
one screen shows the world state, the attempt pager, and the branch rail at
|
||||||
|
once.
|
||||||
|
|
||||||
cd backend
|
cd backend
|
||||||
.venv/Scripts/python.exe -m tools.shots_fixture /tmp/shots.db
|
.venv/Scripts/python.exe -m tools.shots_fixture /tmp/shots.db
|
||||||
@@ -123,9 +124,9 @@ TURNS = [
|
|||||||
),
|
),
|
||||||
]
|
]
|
||||||
|
|
||||||
# The takes the story did not keep. Each is retried at the turn of the same
|
# The attempts the story did not keep. Each one is retried at the turn with the
|
||||||
# index below, and then forked onto a branch of its own — a discarded take is
|
# same index below, and then forked onto a branch of its own. A discarded attempt
|
||||||
# the only thing a fork can be made of.
|
# is the only thing a fork can be made from.
|
||||||
RETAKES = {
|
RETAKES = {
|
||||||
3: (
|
3: (
|
||||||
"He passes close enough that you can smell the tar on his coat, and the "
|
"He passes close enough that you can smell the tar on his coat, and the "
|
||||||
@@ -145,8 +146,11 @@ RETAKES = {
|
|||||||
|
|
||||||
|
|
||||||
class ScriptedProvider:
|
class ScriptedProvider:
|
||||||
"""Serves whatever the driver loaded, so a retry can differ from the take
|
"""Serves whatever the driver loaded, so a retry can differ from the attempt
|
||||||
it replaces — which is the whole thing being photographed."""
|
it replaces.
|
||||||
|
|
||||||
|
That difference is what the screenshots show.
|
||||||
|
"""
|
||||||
|
|
||||||
next_reply = ("", {})
|
next_reply = ("", {})
|
||||||
|
|
||||||
@@ -165,9 +169,9 @@ auth.resolve_provider_config = lambda s: auth.ProviderConfig(
|
|||||||
limits.rate_limit = lambda *a, **k: None
|
limits.rate_limit = lambda *a, **k: None
|
||||||
limits.check_row_cap = lambda *a, **k: None
|
limits.check_row_cap = lambda *a, **k: None
|
||||||
|
|
||||||
# `bootstrap()` runs at import and seeds the public demo scenarios, so the
|
# `bootstrap()` runs at import and seeds the public demo scenarios, so the Bandit
|
||||||
# Bandit Camp is already here — the point of shooting on it is that it is the
|
# Camp already exists. Shooting on it matters because it is the scenario a
|
||||||
# scenario a visitor actually meets.
|
# visitor meets.
|
||||||
client = TestClient(app)
|
client = TestClient(app)
|
||||||
|
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
@@ -206,9 +210,11 @@ def retake(i):
|
|||||||
|
|
||||||
|
|
||||||
def retry_last():
|
def retry_last():
|
||||||
"""Retry whatever is at the tip, with whatever reply is loaded, and hand
|
"""Retries whatever is at the tip with whatever reply is loaded.
|
||||||
back the take the story just walked away from — the only thing a fork can
|
|
||||||
be made of."""
|
Returns the attempt the story just left, which is the only thing a fork can
|
||||||
|
be made from.
|
||||||
|
"""
|
||||||
before = live_ai_ids()
|
before = live_ai_ids()
|
||||||
r = client.post(f"{base}/retry")
|
r = client.post(f"{base}/retry")
|
||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
@@ -264,9 +270,9 @@ name([b for b in branches() if b["is_head"]][0]["id"], "Loud, and early")
|
|||||||
|
|
||||||
# ---- and a line that left that one again, so the map has to nest ----
|
# ---- and a line that left that one again, so the map has to nest ----
|
||||||
#
|
#
|
||||||
# Forking the take at the tip only switches to it — the attempts there are
|
# Forking the attempt at the tip only switches to it, because the attempts there
|
||||||
# still leaves nobody has built on. So the story is moved one turn past it
|
# are still leaves that nothing was built on. The story is therefore moved one
|
||||||
# first, and only then is the take it walked away from worth a branch.
|
# turn past it first, and only then does the attempt it left deserve a branch.
|
||||||
ScriptedProvider.next_reply = (
|
ScriptedProvider.next_reply = (
|
||||||
"Nobody comes. The horn was the wrong horn, or the camp has been empty of "
|
"Nobody comes. The horn was the wrong horn, or the camp has been empty of "
|
||||||
"anyone who cares since before you got here.", {"npc.gwen.trust": -4})
|
"anyone who cares since before you got here.", {"npc.gwen.trust": -4})
|
||||||
@@ -298,8 +304,8 @@ r = client.post(f"{base}/actions", json={"type": "do", "text": "Say nothing and
|
|||||||
assert r.status_code == 200, r.text
|
assert r.status_code == 200, r.text
|
||||||
name([b for b in branches() if b["is_head"]][0]["id"], "Said nothing")
|
name([b for b in branches() if b["is_head"]][0]["id"], "Said nothing")
|
||||||
|
|
||||||
# Leave the reader on the first telling, standing on the turn that has two
|
# Leave the reader on the first telling, on the turn that has two attempts. That
|
||||||
# takes — the one screen that shows the rail, the pager and the branches.
|
# is the one screen showing the rail, the pager, and the branches.
|
||||||
client.post(f"{base}/branches/{root}/switch")
|
client.post(f"{base}/branches/{root}/switch")
|
||||||
|
|
||||||
with engine.begin() as conn:
|
with engine.begin() as conn:
|
||||||
|
|||||||
+222
-194
@@ -1,4 +1,4 @@
|
|||||||
"""Drive a production-sized adventure through the real routes and report what
|
"""Drives a production-sized adventure through the real routes and reports what
|
||||||
each one costs in database bytes.
|
each one costs in database bytes.
|
||||||
|
|
||||||
cd backend
|
cd backend
|
||||||
@@ -6,43 +6,42 @@ each one costs in database bytes.
|
|||||||
.venv/Scripts/python.exe -m tools.stress_session --actions 200 --memories 100
|
.venv/Scripts/python.exe -m tools.stress_session --actions 200 --memories 100
|
||||||
.venv/Scripts/python.exe -m tools.stress_session --no-embeddings
|
.venv/Scripts/python.exe -m tools.stress_session --no-embeddings
|
||||||
|
|
||||||
**The memory bank is ON by default, and that is the point.** The round-two
|
The memory bank is on by default, and that matters. The round-two stress
|
||||||
stress harness ran without an embedding model configured, and embedding
|
harness ran with no embedding model configured, and embedding providers are
|
||||||
providers are BYOK-only by construction, so `retrieve_memories` returned early
|
BYOK-only by construction, so `retrieve_memories` returned early every time.
|
||||||
every time — the whole exercise measured the turn loop with its heaviest read
|
That run measured the turn loop with its heaviest read disabled and reported
|
||||||
switched off, and reported 23 MB for a playthrough that actually costs an order
|
23 MB for a playthrough that costs an order of magnitude more.
|
||||||
of magnitude more. `--no-embeddings` reproduces that blindness deliberately, to
|
`--no-embeddings` reproduces that configuration on purpose, to show the gap. It
|
||||||
show the gap; it is never the default and it prints a warning.
|
is never the default, and it prints a warning.
|
||||||
|
|
||||||
Everything here is synthetic. The fixture is generated to production *shape* —
|
Everything here is synthetic. The fixture is generated to the shape of
|
||||||
1536-dimension embeddings, ~74 KB context snapshots, retry variants — and no
|
production data, with 1536-dimension embeddings, context snapshots of about
|
||||||
real adventure, user or backup is ever read.
|
74 kB, and retry variants. It never reads a real adventure, user, or backup.
|
||||||
|
|
||||||
Only the network is faked: the LLM and the embedding endpoint. Routing,
|
Only the network is faked, which means the LLM and the embedding endpoint.
|
||||||
sessions, the ORM, the scripting engine and the context builder are the real
|
Routing, sessions, the ORM, the scripting engine, and the context builder are
|
||||||
ones, because the bugs this exists to catch live in exactly the layer a mock
|
the real ones, because the bugs this harness exists to catch are in the layer a
|
||||||
would replace.
|
mock would replace.
|
||||||
|
|
||||||
It runs on a throwaway SQLite file by default. What is being measured is which
|
It runs on a throwaway SQLite file by default. The measurement is which columns
|
||||||
columns of which rows a code path asks for, and that is decided by the ORM,
|
of which rows a code path requests, and the ORM decides that identically on both
|
||||||
identically on both dialects. The dialects disagree on how a value is encoded
|
dialects. The dialects differ in how a value is encoded on the wire, especially
|
||||||
on the wire — JSON especially — so treat the absolute figures as
|
JSON, so treat the absolute figures as production-shaped rather than
|
||||||
production-shaped rather than production-exact, and compare before against
|
production-exact, and compare one run against another.
|
||||||
after.
|
|
||||||
|
|
||||||
To measure the encodings SQLite cannot reach — bytea for the packed vectors,
|
To measure the encodings SQLite cannot reach, which are bytea for the packed
|
||||||
and json columns psycopg parses before the meter sees them — set
|
vectors and json columns that psycopg parses before the meter sees them, set
|
||||||
AIDND_STRESS_DATABASE_URL to a **throwaway** Postgres database:
|
`AIDND_STRESS_DATABASE_URL` to a throwaway Postgres database:
|
||||||
|
|
||||||
AIDND_STRESS_DATABASE_URL=postgresql://…/stress_scratch \
|
AIDND_STRESS_DATABASE_URL=postgresql://…/stress_scratch \
|
||||||
.venv/Scripts/python.exe -m tools.stress_session
|
.venv/Scripts/python.exe -m tools.stress_session
|
||||||
|
|
||||||
The harness writes, so it refuses any target whose database name does not say
|
The harness writes, so it refuses any target whose database name does not
|
||||||
'stress' or 'scratch'. Never point it at a database holding real users.
|
contain 'stress' or 'scratch'. Never point it at a database holding real users.
|
||||||
|
|
||||||
Calibration. The fixture is sized from production, re-measured 2026-08-17
|
Calibration. The fixture is sized from production and was re-measured on
|
||||||
against the live Neon database (aggregates only — counts and octet_length
|
2026-08-17 against the live Neon database, reading aggregates only: counts and
|
||||||
sums, never row contents):
|
`octet_length` sums, never row contents.
|
||||||
|
|
||||||
per action, text 886 B -> --narration-bytes 1700, alternating
|
per action, text 886 B -> --narration-bytes 1700, alternating
|
||||||
with a one-line player input
|
with a one-line player input
|
||||||
@@ -51,14 +50,13 @@ sums, never row contents):
|
|||||||
memory bank, largest 100 memories, 6,144 B a vector
|
memory bank, largest 100 memories, 6,144 B a vector
|
||||||
|
|
||||||
The previous defaults were wrong in both directions at once and happened to
|
The previous defaults were wrong in both directions at once and happened to
|
||||||
land near the right total: actions were modelled at ~2.1 KB against a real
|
land near the right total. Actions were modeled at about 2.1 kB against a real
|
||||||
886 B, and stories at 200 actions against a real 607. Width was flattering,
|
886 B, and stories at 200 actions against a real 607. The width was too large
|
||||||
length was not, and length is what a page load pays for.
|
and the length was too small, and the length is what a page load pays for.
|
||||||
|
|
||||||
Filler text is generated word by word rather than repeated. A repeated
|
Filler text is generated word by word rather than repeated. A repeated sentence
|
||||||
sentence compresses about a hundredfold and prose three- or fourfold, so the
|
compresses about a hundredfold and prose three- or fourfold, so the old fixture
|
||||||
old fixture would have made any compression measurement on context_snapshot
|
would have made any compression measurement on `context_snapshot` meaningless.
|
||||||
meaningless.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
@@ -68,11 +66,12 @@ from pathlib import Path
|
|||||||
|
|
||||||
|
|
||||||
def _early_keep(argv: list[str]) -> str:
|
def _early_keep(argv: list[str]) -> str:
|
||||||
"""--keep, read before argparse exists.
|
"""Reads `--keep` before argparse exists.
|
||||||
|
|
||||||
Where the database lives has to be decided before app.database is
|
The database location has to be decided before `app.database` is imported,
|
||||||
imported, and that import is three lines below. argparse still declares
|
and that import is a few lines below. argparse still declares the flag, so
|
||||||
the flag, so --help documents it and a typo is still an error."""
|
`--help` documents it and a typo is still an error.
|
||||||
|
"""
|
||||||
for i, arg in enumerate(argv):
|
for i, arg in enumerate(argv):
|
||||||
if arg == "--keep" and i + 1 < len(argv):
|
if arg == "--keep" and i + 1 < len(argv):
|
||||||
return argv[i + 1]
|
return argv[i + 1]
|
||||||
@@ -83,17 +82,17 @@ def _early_keep(argv: list[str]) -> str:
|
|||||||
|
|
||||||
_keep = _early_keep(sys.argv[1:])
|
_keep = _early_keep(sys.argv[1:])
|
||||||
|
|
||||||
# Must precede the app import: database.py reads these at module scope.
|
# This has to run before the app import, because `database.py` reads these at
|
||||||
|
# module scope.
|
||||||
#
|
#
|
||||||
# Default is a throwaway SQLite file. AIDND_STRESS_DATABASE_URL points the
|
# The default is a throwaway SQLite file. `AIDND_STRESS_DATABASE_URL` points the
|
||||||
# harness at a real Postgres instead, which is the only way to reach the
|
# harness at a real Postgres instead, which is the only way to reach the
|
||||||
# encodings SQLite cannot exercise: bytea for the packed vectors, and json
|
# encodings SQLite cannot exercise: bytea for the packed vectors, and json
|
||||||
# columns that psycopg parses into Python before the meter ever sees them.
|
# columns that psycopg parses into Python before the meter sees them.
|
||||||
#
|
#
|
||||||
# The name guard is not paranoia. This harness *writes* — it builds a whole
|
# The name guard matters. This harness writes a whole synthetic adventure, so a
|
||||||
# synthetic adventure — so a URL that happened to point at the production
|
# URL that pointed at the production database would seed it with fake users and
|
||||||
# database would quietly seed it with fake users and fake play. The target
|
# fake play. The target has to name itself as disposable.
|
||||||
# must say it is disposable.
|
|
||||||
_stress_url = os.environ.get("AIDND_STRESS_DATABASE_URL", "").strip()
|
_stress_url = os.environ.get("AIDND_STRESS_DATABASE_URL", "").strip()
|
||||||
if _stress_url:
|
if _stress_url:
|
||||||
if _keep:
|
if _keep:
|
||||||
@@ -111,10 +110,10 @@ if _stress_url:
|
|||||||
os.environ["AIDND_DATABASE_URL"] = _stress_url
|
os.environ["AIDND_DATABASE_URL"] = _stress_url
|
||||||
os.environ.pop("DATABASE_URL", None)
|
os.environ.pop("DATABASE_URL", None)
|
||||||
elif _keep:
|
elif _keep:
|
||||||
# A fixture to boot the app against rather than a temp file the report
|
# A fixture to start the app against, rather than a temporary file the
|
||||||
# discards. Rebuilt from empty every run: build_fixture() assumes an empty
|
# report discards. Every run rebuilds it from empty, because
|
||||||
# database on the SQLite path, and a second run would otherwise stack a
|
# `build_fixture()` assumes an empty database on the SQLite path and a
|
||||||
# second adventure beside the first.
|
# second run would otherwise add a second adventure next to the first.
|
||||||
_keep_path = Path(_keep).resolve()
|
_keep_path = Path(_keep).resolve()
|
||||||
_keep_path.parent.mkdir(parents=True, exist_ok=True)
|
_keep_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
_keep_path.unlink(missing_ok=True)
|
_keep_path.unlink(missing_ok=True)
|
||||||
@@ -149,21 +148,22 @@ from .fakeprose import prose
|
|||||||
|
|
||||||
EMBEDDING_DIMS = 1536
|
EMBEDDING_DIMS = 1536
|
||||||
|
|
||||||
# ~232 KB, measured on production's largest adventure (2026-08-17). The old
|
# About 232 kB, measured on the largest adventure in production on 2026-08-17.
|
||||||
# figure here was 74 KB, taken from the comment in models.py; the real column
|
# The old figure here was 74 kB, taken from the comment in `models.py`. The real
|
||||||
# averages 163 KB a row across the whole table and 232 KB on the adventure that
|
# column averages 163 kB per row across the whole table and 232 kB on the
|
||||||
# matters, because the assembled prompt grows with the story behind it.
|
# adventure that matters, because the assembled prompt grows with the story
|
||||||
|
# behind it.
|
||||||
#
|
#
|
||||||
# Built from varied text rather than one sentence repeated. A repeated sentence
|
# The text is varied rather than one sentence repeated. A repeated sentence
|
||||||
# compresses about a hundredfold and real prose three- or fourfold, so a
|
# compresses about a hundredfold and real prose three- or fourfold, so a fixture
|
||||||
# fixture made of repeats would make any compression measurement meaningless —
|
# built from repeats would make any compression measurement meaningless, and
|
||||||
# and shrinking this column is the open question it exists to answer.
|
# shrinking this column is the open question the fixture exists to answer.
|
||||||
SNAPSHOT_SYSTEM = None # set by _build_text()
|
SNAPSHOT_SYSTEM = None # set by _build_text()
|
||||||
SNAPSHOT_STORY = None
|
SNAPSHOT_STORY = None
|
||||||
|
|
||||||
PLAYER_INPUT = "> You crouch and look more closely at the grit on the floor."
|
PLAYER_INPUT = "> You crouch and look more closely at the grit on the floor."
|
||||||
|
|
||||||
# All three are bound by _build_text() from the fixture arguments.
|
# `_build_text()` sets all three from the fixture arguments.
|
||||||
NARRATION = None
|
NARRATION = None
|
||||||
MEMORY_TEXT = (
|
MEMORY_TEXT = (
|
||||||
"You found a bandit camp above the ford and agreed to guide Gwen through "
|
"You found a bandit camp above the ford and agreed to guide Gwen through "
|
||||||
@@ -172,15 +172,15 @@ MEMORY_TEXT = (
|
|||||||
|
|
||||||
|
|
||||||
def _build_text(args, rng: random.Random) -> None:
|
def _build_text(args, rng: random.Random) -> None:
|
||||||
"""Size the three variable-length fixture strings from the arguments.
|
"""Sizes the three variable-length fixture strings from the arguments.
|
||||||
|
|
||||||
Separate from build_fixture so the sizes are decided once, before anything
|
This is separate from `build_fixture` so that the sizes are decided once,
|
||||||
is written, and so a shape's cost is a function of the flags rather than of
|
before anything is written, and so that a shape's cost depends on the flags
|
||||||
how many rows happened to be generated first.
|
rather than on how many rows were generated first.
|
||||||
"""
|
"""
|
||||||
global NARRATION, SNAPSHOT_SYSTEM, SNAPSHOT_STORY
|
global NARRATION, SNAPSHOT_SYSTEM, SNAPSHOT_STORY
|
||||||
NARRATION = prose(rng, args.narration_bytes)
|
NARRATION = prose(rng, args.narration_bytes)
|
||||||
# The assembled prompt is a system block and the story so far; the split
|
# The assembled prompt is a system block plus the story so far. The split
|
||||||
# is roughly one to five in production.
|
# is roughly one to five in production.
|
||||||
SNAPSHOT_SYSTEM = prose(rng, args.snapshot_bytes // 6)
|
SNAPSHOT_SYSTEM = prose(rng, args.snapshot_bytes // 6)
|
||||||
SNAPSHOT_STORY = prose(rng, args.snapshot_bytes - args.snapshot_bytes // 6)
|
SNAPSHOT_STORY = prose(rng, args.snapshot_bytes - args.snapshot_bytes // 6)
|
||||||
@@ -190,7 +190,11 @@ def _build_text(args, rng: random.Random) -> None:
|
|||||||
|
|
||||||
|
|
||||||
class FakeProvider:
|
class FakeProvider:
|
||||||
"""The LLM. Streams one fixed line; no network, no cost, no variance."""
|
"""Stands in for the LLM, streaming one fixed line.
|
||||||
|
|
||||||
|
It makes no network call, costs nothing, and returns the same text every
|
||||||
|
time.
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(self, *a, **k):
|
def __init__(self, *a, **k):
|
||||||
pass
|
pass
|
||||||
@@ -203,8 +207,11 @@ class FakeProvider:
|
|||||||
|
|
||||||
|
|
||||||
class FakeEmbeddings:
|
class FakeEmbeddings:
|
||||||
"""The embedding endpoint. Returns vectors of the real width, so what the
|
"""Stands in for the embedding endpoint.
|
||||||
turn writes back weighs what production weighs."""
|
|
||||||
|
It returns vectors of the production width, so what the turn writes back is
|
||||||
|
the size production writes back.
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(self, rng: random.Random):
|
def __init__(self, rng: random.Random):
|
||||||
self.rng = rng
|
self.rng = rng
|
||||||
@@ -218,32 +225,34 @@ class FakeEmbeddings:
|
|||||||
|
|
||||||
# -------------------------------------------------------------- rich fixture
|
# -------------------------------------------------------------- rich fixture
|
||||||
|
|
||||||
# The correctness fixture, as against the scale one.
|
# The correctness fixture, as opposed to the scale fixture.
|
||||||
#
|
#
|
||||||
# The default fixture is sized from production and exists to weigh bytes, so it
|
# The default fixture is sized from production and exists to measure bytes, so it
|
||||||
# leaves every column it does not weigh at its default. That makes it a poor
|
# leaves every column it does not measure at its default. That makes it a poor
|
||||||
# witness for anything *semantic* — and phase 14 replaces the storage model
|
# test of semantics, and phase 14 replaces the storage model underneath all of
|
||||||
# underneath all of it. Measured on a freshly built default fixture, the
|
# it. On a freshly built default fixture, the columns a story tree has to migrate
|
||||||
# columns a story tree has to migrate correctly look like this:
|
# correctly hold this:
|
||||||
#
|
#
|
||||||
# state_after / world_state_after the same value on all 600 rows (they
|
# state_after, world_state_after The same value on all 600 rows. They
|
||||||
# were state_before, and NULL, until SP4
|
# were state_before, and NULL, until SP4
|
||||||
# turned them round). A rollback over
|
# reversed them. A rollback over identical
|
||||||
# identical snapshots proves nothing.
|
# snapshots tests nothing.
|
||||||
# scenario_id / world_state absent. No RPG layer, so the cooldown
|
# scenario_id, world_state Absent. There is no RPG layer, so the
|
||||||
# clock SP5 must not advance never runs.
|
# cooldown clock SP5 must not advance
|
||||||
# adventure_scripts none. script_state rollback is exactly
|
# never runs.
|
||||||
# what a branch switch reuses.
|
# adventure_scripts None. A branch switch reuses the
|
||||||
# memory_cursor / summary_cursor both 0. SP3 replaces the cursors.
|
# script_state rollback.
|
||||||
# sibling attempts two of byte-identical text with the
|
# memory_cursor, summary_cursor Both 0. SP3 replaces the cursors.
|
||||||
# first always live — so "which attempt
|
# sibling attempts Two attempts of byte-identical text with
|
||||||
# is live?", the one question SP4 has to
|
# the first always live, so the one
|
||||||
# answer, has no observable answer.
|
# question SP4 has to answer, which
|
||||||
|
# attempt is live, has no observable
|
||||||
|
# answer.
|
||||||
#
|
#
|
||||||
# --rich fills in exactly those and changes nothing else, so the measuring
|
# `--rich` fills in those columns and changes nothing else, so the measuring
|
||||||
# fixture's numbers stay comparable run to run. It is a *correctness* fixture:
|
# fixture's numbers stay comparable from run to run. It is a correctness fixture,
|
||||||
# prefer it small (--rich --actions 30), because what it is for is variety per
|
# so keep it small, such as `--rich --actions 30`. It provides variety per row
|
||||||
# row, not rows.
|
# rather than many rows.
|
||||||
|
|
||||||
RICH_SUMMARY = (
|
RICH_SUMMARY = (
|
||||||
"You tracked the bandits to a camp above the ford, freed Gwen from the "
|
"You tracked the bandits to a camp above the ford, freed Gwen from the "
|
||||||
@@ -260,8 +269,8 @@ RICH_CARDS = [
|
|||||||
"Taken from the quartermaster. Opens something below the camp."),
|
"Taken from the quartermaster. Opens something below the camp."),
|
||||||
]
|
]
|
||||||
|
|
||||||
# Ten gold a turn, so a state snapshot that failed to roll back reads as a
|
# Ten gold a turn, so a state snapshot that failed to roll back shows a wrong
|
||||||
# wrong total rather than as nothing. Same shape the retry tests use.
|
# total rather than no value. The retry tests use the same shape.
|
||||||
RICH_SCRIPT = """
|
RICH_SCRIPT = """
|
||||||
const modifier = (text) => {
|
const modifier = (text) => {
|
||||||
state.gold = (state.gold || 0) + 10;
|
state.gold = (state.gold || 0) + 10;
|
||||||
@@ -272,28 +281,31 @@ modifier(text);
|
|||||||
|
|
||||||
|
|
||||||
def rich_stat_schema() -> dict:
|
def rich_stat_schema() -> dict:
|
||||||
"""The demo RPG schema, read from the seed data rather than invented here.
|
"""Returns the demo RPG schema, read from the seed data rather than invented.
|
||||||
|
|
||||||
Using the real one means the fixture exercises bands, cooldowns,
|
Using the real schema means the fixture exercises bands, cooldowns,
|
||||||
max_delta_per_turn and NPC stat blocks as they are actually shaped — an
|
`max_delta_per_turn`, and NPC stat blocks in their real shapes. An invented
|
||||||
invented schema would drift from the thing it is standing in for.
|
schema would diverge from the one it stands in for.
|
||||||
"""
|
"""
|
||||||
path = seed.SEED_DIR / "04-rpg-world-state.json"
|
path = seed.SEED_DIR / "04-rpg-world-state.json"
|
||||||
return json.loads(path.read_text(encoding="utf-8"))["stat_schema"]
|
return json.loads(path.read_text(encoding="utf-8"))["stat_schema"]
|
||||||
|
|
||||||
|
|
||||||
def rich_script_state(turn: int) -> dict:
|
def rich_script_state(turn: int) -> dict:
|
||||||
"""The scoreboard as of `turn`. Monotonic, so any snapshot identifies the
|
"""Returns the script state as of `turn`.
|
||||||
turn it was taken at — which is what makes a bad rollback visible."""
|
|
||||||
|
The values increase with the turn, so any snapshot identifies the turn it was
|
||||||
|
taken at, which is what makes an incorrect rollback visible.
|
||||||
|
"""
|
||||||
return {"gold": turn * 10, "turn": turn}
|
return {"gold": turn * 10, "turn": turn}
|
||||||
|
|
||||||
|
|
||||||
def rich_world_state(schema: dict, turn: int) -> dict:
|
def rich_world_state(schema: dict, turn: int) -> dict:
|
||||||
"""A live world state that has actually been played to `turn`.
|
"""Returns a live world state that has been played to `turn`.
|
||||||
|
|
||||||
`instantiate` gives the initial picture; a fixture whose every row holds
|
`instantiate` returns the initial state, and a fixture whose every row holds
|
||||||
that same picture cannot tell a restored snapshot from an unrestored one.
|
that same state cannot distinguish a restored snapshot from an unrestored
|
||||||
So hp declines, mana drains and a flag flips partway through.
|
one. Here hp declines, mana declines, and a flag changes partway through.
|
||||||
"""
|
"""
|
||||||
ws = worldstate.instantiate(schema)
|
ws = worldstate.instantiate(schema)
|
||||||
ws["player"]["hp"] = max(20, 100 - turn)
|
ws["player"]["hp"] = max(20, 100 - turn)
|
||||||
@@ -306,28 +318,28 @@ def rich_world_state(schema: dict, turn: int) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
def rich_attempts(rng: random.Random, index: int) -> tuple[list[str], int]:
|
def rich_attempts(rng: random.Random, index: int) -> tuple[list[str], int]:
|
||||||
"""Distinguishable retry attempts, and which one the story tells.
|
"""Returns distinguishable retry attempts, and which one the story tells.
|
||||||
|
|
||||||
Every attempt in the default fixture carries the same text with the live
|
Every attempt in the default fixture carries the same text, with the live one
|
||||||
one pinned at 0. That is the one thing SP4 has to get right and the one
|
fixed at index 0. That is the behavior SP4 has to get right and the behavior
|
||||||
thing that fixture cannot witness, so here the texts differ, the counts
|
that fixture cannot test, so here the texts differ, the counts differ, and
|
||||||
differ, and the live one is often not the last.
|
the live attempt is often not the last.
|
||||||
"""
|
"""
|
||||||
count = 3 if index % 12 == 1 else 2
|
count = 3 if index % 12 == 1 else 2
|
||||||
texts = [
|
texts = [
|
||||||
f"[attempt {n + 1} of {count} at turn {index}] {prose(rng, 240)}"
|
f"[attempt {n + 1} of {count} at turn {index}] {prose(rng, 240)}"
|
||||||
for n in range(count)
|
for n in range(count)
|
||||||
]
|
]
|
||||||
# Deliberately not always the newest: a player who retried twice and then
|
# The live attempt is not always the newest. A player who retried twice and
|
||||||
# went back to the first take is the case that breaks anything assuming the
|
# then went back to the first attempt is the case that breaks code assuming
|
||||||
# live attempt is the last one written.
|
# the live attempt is the last one written.
|
||||||
return texts, (0 if index % 18 == 1 else count - 1)
|
return texts, (0 if index % 18 == 1 else count - 1)
|
||||||
|
|
||||||
|
|
||||||
def add_rich_extras(db, args, rng: random.Random, user, adventure) -> None:
|
def add_rich_extras(db, args, rng: random.Random, user, adventure) -> None:
|
||||||
"""Everything --rich adds beside the actions themselves.
|
"""Adds everything `--rich` contributes apart from the actions themselves.
|
||||||
|
|
||||||
Called with the adventure already flushed, before the actions are written,
|
The caller flushes the adventure first and writes the actions afterwards,
|
||||||
because the actions need the schema to snapshot a world state from.
|
because the actions need the schema to snapshot a world state from.
|
||||||
"""
|
"""
|
||||||
schema = rich_stat_schema()
|
schema = rich_stat_schema()
|
||||||
@@ -343,11 +355,11 @@ def add_rich_extras(db, args, rng: random.Random, user, adventure) -> None:
|
|||||||
adventure.scenario_id = scenario.id
|
adventure.scenario_id = scenario.id
|
||||||
adventure.world_state = rich_world_state(schema, args.actions)
|
adventure.world_state = rich_world_state(schema, args.actions)
|
||||||
adventure.script_state = rich_script_state(args.actions)
|
adventure.script_state = rich_script_state(args.actions)
|
||||||
# The post-turn passes only do anything when summarization is on and the
|
# The post-turn passes do nothing unless summarization is on and the marks
|
||||||
# marks are somewhere other than the start. Written as positions here and
|
# are past the start. They are written here as positions and translated into
|
||||||
# translated into anchors once the actions exist (see build_fixture) — a
|
# anchors once the actions exist. See `build_fixture`. A database being
|
||||||
# position is what a database being migrated to SP3 still holds, so the
|
# migrated to SP3 still holds positions, so the fixture carries both forms
|
||||||
# fixture carries both and they have to say the same thing.
|
# and the two have to agree.
|
||||||
adventure.auto_summarize = True
|
adventure.auto_summarize = True
|
||||||
adventure.story_summary = RICH_SUMMARY
|
adventure.story_summary = RICH_SUMMARY
|
||||||
adventure.memory_cursor = max(0, args.actions - 8)
|
adventure.memory_cursor = max(0, args.actions - 8)
|
||||||
@@ -366,10 +378,12 @@ def add_rich_extras(db, args, rng: random.Random, user, adventure) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def add_second_adventure(db, rng: random.Random, user) -> int:
|
def add_second_adventure(db, rng: random.Random, user) -> int:
|
||||||
"""A short second adventure, so 'does this leak across adventures?' is a
|
"""Adds a short second adventure, so the fixture can detect cross-adventure
|
||||||
question the fixture can answer. A tree scopes every read by branch, and a
|
leaks.
|
||||||
branch clause that forgot its adventure would still look right on a
|
|
||||||
database holding exactly one."""
|
A tree scopes every read by branch, and a branch clause that omitted its
|
||||||
|
adventure would still look correct on a database holding one adventure.
|
||||||
|
"""
|
||||||
other = models.Adventure(
|
other = models.Adventure(
|
||||||
user_id=user.id, title="Stress (second)", script_state={},
|
user_id=user.id, title="Stress (second)", script_state={},
|
||||||
memory_bank_enabled=True, auto_summarize=False,
|
memory_bank_enabled=True, auto_summarize=False,
|
||||||
@@ -398,12 +412,12 @@ def add_second_adventure(db, rng: random.Random, user) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def _assert_live_variant_invariant(db, adventure_id: int) -> None:
|
def _assert_live_variant_invariant(db, adventure_id: int) -> None:
|
||||||
"""Exactly one attempt per turn is live, on every turn.
|
"""Checks that exactly one attempt per turn is live, on every turn.
|
||||||
|
|
||||||
Checked here rather than trusted, because a coordinate with two live
|
The check runs here rather than being assumed, because a coordinate with two
|
||||||
siblings tells its story twice and a coordinate with none drops a turn out
|
live siblings renders its turn twice and a coordinate with none omits the
|
||||||
of it — and both fail by *reading* wrong, never by raising. A fixture that
|
turn. Both failures produce wrong output rather than an exception, so a
|
||||||
quietly violated it would let wrong code look right.
|
fixture that broke the rule would let incorrect code look correct.
|
||||||
"""
|
"""
|
||||||
groups: dict[tuple, list[models.Action]] = {}
|
groups: dict[tuple, list[models.Action]] = {}
|
||||||
for action in (
|
for action in (
|
||||||
@@ -430,13 +444,15 @@ def _assert_live_variant_invariant(db, adventure_id: int) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def build_fixture(args, rng: random.Random) -> tuple[int, int]:
|
def build_fixture(args, rng: random.Random) -> tuple[int, int]:
|
||||||
"""A user, settings and one adventure at production scale. Returns
|
"""Builds a user, settings, and one adventure at production scale.
|
||||||
(adventure_id, user_id)."""
|
|
||||||
# A SQLite run gets a brand-new temp file every time, so the fixture can
|
Returns `(adventure_id, user_id)`.
|
||||||
# assume an empty database. A Postgres scratch target persists between
|
"""
|
||||||
# runs, and the second one would collide on the fixture user's unique
|
# A SQLite run gets a new temporary file every time, so the fixture can
|
||||||
# email — so empty it first. Only ever reached for a target whose name
|
# assume an empty database. A Postgres scratch target persists between runs,
|
||||||
# passed the 'stress'/'scratch' guard at the top of this module.
|
# and a second run would collide on the fixture user's unique email, so empty
|
||||||
|
# it first. This runs only for a target whose name passed the 'stress' or
|
||||||
|
# 'scratch' guard at the top of this module.
|
||||||
if _stress_url:
|
if _stress_url:
|
||||||
Base.metadata.drop_all(bind=engine)
|
Base.metadata.drop_all(bind=engine)
|
||||||
Base.metadata.create_all(bind=engine)
|
Base.metadata.create_all(bind=engine)
|
||||||
@@ -450,7 +466,7 @@ def build_fixture(args, rng: random.Random) -> tuple[int, int]:
|
|||||||
api_key=security.encrypt_secret("stress-key"),
|
api_key=security.encrypt_secret("stress-key"),
|
||||||
model="stress-model",
|
model="stress-model",
|
||||||
endpoint_url="https://fake.invalid/v1",
|
endpoint_url="https://fake.invalid/v1",
|
||||||
# The default this whole tool exists to stop anyone forgetting.
|
# The default this tool exists to keep anyone from forgetting.
|
||||||
embedding_model="" if args.no_embeddings else "openai/text-embedding-3-small",
|
embedding_model="" if args.no_embeddings else "openai/text-embedding-3-small",
|
||||||
memory_bank_capacity=args.capacity,
|
memory_bank_capacity=args.capacity,
|
||||||
))
|
))
|
||||||
@@ -459,8 +475,9 @@ def build_fixture(args, rng: random.Random) -> tuple[int, int]:
|
|||||||
title="Stress",
|
title="Stress",
|
||||||
script_state={},
|
script_state={},
|
||||||
memory_bank_enabled=True,
|
memory_bank_enabled=True,
|
||||||
# Off so a turn measures the turn. The post-turn pass is its own
|
# Off, so that a turn measures only the turn. The post-turn pass
|
||||||
# shape below; letting it fire mid-measurement would mix the two.
|
# is its own shape below, and running it during the measurement
|
||||||
|
# would combine the two.
|
||||||
auto_summarize=False,
|
auto_summarize=False,
|
||||||
)
|
)
|
||||||
db.add(adventure)
|
db.add(adventure)
|
||||||
@@ -472,7 +489,7 @@ def build_fixture(args, rng: random.Random) -> tuple[int, int]:
|
|||||||
is_ai = bool(i % 2)
|
is_ai = bool(i % 2)
|
||||||
retried = is_ai and i % 6 == 1
|
retried = is_ai and i % 6 == 1
|
||||||
if args.rich:
|
if args.rich:
|
||||||
# Distinguishable attempts, and a live one that is often not
|
# Distinguishable attempts, with a live one that is often not
|
||||||
# the last written.
|
# the last written.
|
||||||
texts, live = rich_attempts(rng, i) if retried else ([], 0)
|
texts, live = rich_attempts(rng, i) if retried else ([], 0)
|
||||||
if not retried:
|
if not retried:
|
||||||
@@ -490,10 +507,10 @@ def build_fixture(args, rng: random.Random) -> tuple[int, int]:
|
|||||||
type="ai" if is_ai else "do",
|
type="ai" if is_ai else "do",
|
||||||
text=body,
|
text=body,
|
||||||
# The turn's assembled prompt is stored once, on the
|
# The turn's assembled prompt is stored once, on the
|
||||||
# attempt the story tells; a superseded sibling keeps only
|
# attempt the story tells. A superseded sibling keeps only
|
||||||
# its own slices. That is the invariant `app/attempts.py`
|
# its own slices. `app/attempts.py` maintains that
|
||||||
# maintains, and a fixture that ignored it would multiply
|
# invariant, and a fixture that ignored it would multiply
|
||||||
# the biggest column in the database by the retry count.
|
# the largest column in the database by the retry count.
|
||||||
context_snapshot=(
|
context_snapshot=(
|
||||||
{"system": SNAPSHOT_SYSTEM, "story": SNAPSHOT_STORY}
|
{"system": SNAPSHOT_SYSTEM, "story": SNAPSHOT_STORY}
|
||||||
if n == live else
|
if n == live else
|
||||||
@@ -506,20 +523,21 @@ def build_fixture(args, rng: random.Random) -> tuple[int, int]:
|
|||||||
live=(n == live),
|
live=(n == live),
|
||||||
variant_index=n,
|
variant_index=n,
|
||||||
variant_count=len(texts) if len(texts) > 1 else 0,
|
variant_count=len(texts) if len(texts) > 1 else 0,
|
||||||
# Monotonic under --rich, so a bad rollback reads as a
|
# Under `--rich` the value increases with the turn, so an
|
||||||
# wrong number rather than as nothing. Left to
|
# incorrect rollback shows a wrong number rather than no
|
||||||
# `tree.stamp_outcome` otherwise, which writes the
|
# value. Otherwise `tree.stamp_outcome` writes the
|
||||||
# adventure's (unchanging) state — true, and no witness.
|
# adventure's state, which does not change and therefore
|
||||||
|
# tests nothing.
|
||||||
state_after=rich_script_state(i) if args.rich else None,
|
state_after=rich_script_state(i) if args.rich else None,
|
||||||
world_state_after=(
|
world_state_after=(
|
||||||
rich_world_state(schema, i) if args.rich else None
|
rich_world_state(schema, i) if args.rich else None
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
# A fresh database is built by create_all and stamped LATEST,
|
# `create_all` builds a fresh database and stamps it LATEST,
|
||||||
# so no migration ever runs against it and the tree backfill
|
# so no migration runs against it and the tree backfill never
|
||||||
# never sees it. The fixture has to stamp its own nodes, or it
|
# sees it. The fixture has to stamp its own nodes, or it would
|
||||||
# would be the one database in the project whose actions have
|
# be the one database in the project whose actions have no
|
||||||
# no branch.
|
# branch.
|
||||||
tree.place_action(db, adventure, action)
|
tree.place_action(db, adventure, action)
|
||||||
db.add(action)
|
db.add(action)
|
||||||
|
|
||||||
@@ -529,14 +547,15 @@ def build_fixture(args, rng: random.Random) -> tuple[int, int]:
|
|||||||
text=f"{MEMORY_TEXT} ({i})",
|
text=f"{MEMORY_TEXT} ({i})",
|
||||||
source_start=i * memorybank.MEMORY_INTERVAL,
|
source_start=i * memorybank.MEMORY_INTERVAL,
|
||||||
source_end=i * memorybank.MEMORY_INTERVAL + memorybank.MEMORY_INTERVAL - 1,
|
source_end=i * memorybank.MEMORY_INTERVAL + memorybank.MEMORY_INTERVAL - 1,
|
||||||
# A bank in play is not uniformly active: some memories are
|
# A bank in play is not uniformly active. Some memories are
|
||||||
# pinned (always retrieved) and some evicted but kept for the
|
# pinned, which means they are always retrieved, and some are
|
||||||
# UI. Both states have to survive being re-attached to nodes.
|
# evicted but kept for the UI. Both states have to survive being
|
||||||
|
# re-attached to nodes.
|
||||||
pinned=bool(args.rich and i % 9 == 0),
|
pinned=bool(args.rich and i % 9 == 0),
|
||||||
forgotten=bool(args.rich and i % 11 == 5),
|
forgotten=bool(args.rich and i % 11 == 5),
|
||||||
)
|
)
|
||||||
# Through the same door the app uses, so the fixture cannot end up
|
# Use the same call the app uses, so the fixture cannot store
|
||||||
# storing vectors in a shape production never produces.
|
# vectors in a shape production never produces.
|
||||||
memorybank.set_vector(
|
memorybank.set_vector(
|
||||||
memory, [rng.uniform(-1.0, 1.0) for _ in range(EMBEDDING_DIMS)]
|
memory, [rng.uniform(-1.0, 1.0) for _ in range(EMBEDDING_DIMS)]
|
||||||
)
|
)
|
||||||
@@ -544,11 +563,11 @@ def build_fixture(args, rng: random.Random) -> tuple[int, int]:
|
|||||||
db.add(memory)
|
db.add(memory)
|
||||||
|
|
||||||
if args.rich:
|
if args.rich:
|
||||||
# The memory/summary marks as nodes, translated from the positions
|
# Translate the memory mark and the summary mark from the
|
||||||
# `add_rich_layers` set now that there are actions to point at —
|
# positions `add_rich_layers` set into nodes, now that actions exist
|
||||||
# through the same call the v1 importer uses. A fixture stamped
|
# to point at. This uses the same call the v1 importer uses. A
|
||||||
# LATEST never meets a migration, so if this is skipped it is the
|
# fixture stamped LATEST never runs a migration, so skipping this
|
||||||
# one database whose marks are only positions.
|
# would leave the one database whose marks are only positions.
|
||||||
db.flush()
|
db.flush()
|
||||||
cursors.anchor_at_position(adventure, cursors.MEMORY, adventure.memory_cursor)
|
cursors.anchor_at_position(adventure, cursors.MEMORY, adventure.memory_cursor)
|
||||||
cursors.anchor_at_position(adventure, cursors.SUMMARY, adventure.summary_cursor)
|
cursors.anchor_at_position(adventure, cursors.SUMMARY, adventure.summary_cursor)
|
||||||
@@ -571,8 +590,8 @@ def install_fakes(user_id: int, rng: random.Random) -> None:
|
|||||||
)
|
)
|
||||||
limits.rate_limit = lambda *a, **k: None
|
limits.rate_limit = lambda *a, **k: None
|
||||||
limits.check_row_cap = lambda *a, **k: None
|
limits.check_row_cap = lambda *a, **k: None
|
||||||
# Fire-and-forget post-turn work would land inside whichever scope happened
|
# Post-turn work scheduled in the background would be counted inside
|
||||||
# to be open. It is measured on purpose, as its own shape.
|
# whatever scope was open. It is measured deliberately, as its own shape.
|
||||||
memorybank.schedule_post_turn = lambda adventure: None
|
memorybank.schedule_post_turn = lambda adventure: None
|
||||||
|
|
||||||
def _current_user(db=Depends(get_db)):
|
def _current_user(db=Depends(get_db)):
|
||||||
@@ -585,21 +604,23 @@ def install_fakes(user_id: int, rng: random.Random) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def shape_list(client, meter, adv_id):
|
def shape_list(client, meter, adv_id):
|
||||||
"""The adventures index — every adventure's latest narration."""
|
"""Measures the adventures index, which returns each adventure's latest
|
||||||
|
narration."""
|
||||||
with meter.scope("GET /adventures (index)"):
|
with meter.scope("GET /adventures (index)"):
|
||||||
r = client.get("/api/adventures")
|
r = client.get("/api/adventures")
|
||||||
_check(r)
|
_check(r)
|
||||||
|
|
||||||
|
|
||||||
def shape_load(client, meter, adv_id):
|
def shape_load(client, meter, adv_id):
|
||||||
"""Opening a finished adventure: the whole story, in one response."""
|
"""Measures opening a finished adventure, which returns the whole story in
|
||||||
|
one response."""
|
||||||
with meter.scope(f"GET /adventures/{{id}} (page load)"):
|
with meter.scope(f"GET /adventures/{{id}} (page load)"):
|
||||||
r = client.get(f"/api/adventures/{adv_id}")
|
r = client.get(f"/api/adventures/{adv_id}")
|
||||||
_check(r)
|
_check(r)
|
||||||
|
|
||||||
|
|
||||||
def shape_turn(client, meter, adv_id):
|
def shape_turn(client, meter, adv_id):
|
||||||
"""One played turn, memory retrieval included."""
|
"""Measures one played turn, including memory retrieval."""
|
||||||
with meter.scope("POST /adventures/{id}/actions (one turn)"):
|
with meter.scope("POST /adventures/{id}/actions (one turn)"):
|
||||||
r = client.post(
|
r = client.post(
|
||||||
f"/api/adventures/{adv_id}/actions",
|
f"/api/adventures/{adv_id}/actions",
|
||||||
@@ -609,21 +630,24 @@ def shape_turn(client, meter, adv_id):
|
|||||||
|
|
||||||
|
|
||||||
def shape_insights(client, meter, adv_id):
|
def shape_insights(client, meter, adv_id):
|
||||||
"""The Insights dry run — assembles a context without spending a turn."""
|
"""Measures the Insights dry run, which assembles a context without playing
|
||||||
|
a turn."""
|
||||||
with meter.scope("GET /adventures/{id}/context (insights)"):
|
with meter.scope("GET /adventures/{id}/context (insights)"):
|
||||||
r = client.get(f"/api/adventures/{adv_id}/context")
|
r = client.get(f"/api/adventures/{adv_id}/context")
|
||||||
_check(r)
|
_check(r)
|
||||||
|
|
||||||
|
|
||||||
def shape_memories(client, meter, adv_id):
|
def shape_memories(client, meter, adv_id):
|
||||||
"""The Memories drawer — every memory, and none of their vectors."""
|
"""Measures the Memories drawer, which returns every memory and no
|
||||||
|
vectors."""
|
||||||
with meter.scope("GET /adventures/{id}/memories (drawer)"):
|
with meter.scope("GET /adventures/{id}/memories (drawer)"):
|
||||||
r = client.get(f"/api/adventures/{adv_id}/memories")
|
r = client.get(f"/api/adventures/{adv_id}/memories")
|
||||||
_check(r)
|
_check(r)
|
||||||
|
|
||||||
|
|
||||||
def shape_post_turn(client, meter, adv_id):
|
def shape_post_turn(client, meter, adv_id):
|
||||||
"""Summarization, embedding and eviction, after the turn is saved."""
|
"""Measures summarization, embedding, and eviction after the turn is
|
||||||
|
saved."""
|
||||||
with meter.scope("run_post_turn (background)"):
|
with meter.scope("run_post_turn (background)"):
|
||||||
asyncio.run(memorybank.run_post_turn(adv_id))
|
asyncio.run(memorybank.run_post_turn(adv_id))
|
||||||
|
|
||||||
@@ -647,21 +671,24 @@ def _check(response) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def make_bootable() -> None:
|
def make_bootable() -> None:
|
||||||
"""Two edits that turn a measurement fixture into a database the app will
|
"""Applies the two edits that make a measurement fixture servable by the app.
|
||||||
actually serve. Both exist because build_fixture() builds a database for
|
|
||||||
the meter, not for a browser."""
|
Both edits are needed because `build_fixture()` builds a database for the
|
||||||
|
meter rather than for a browser.
|
||||||
|
"""
|
||||||
from app.migrations import LATEST_VERSION
|
from app.migrations import LATEST_VERSION
|
||||||
|
|
||||||
with engine.begin() as conn:
|
with engine.begin() as conn:
|
||||||
# create_all() builds the current schema but leaves the stamp at its
|
# `create_all()` builds the current schema but leaves the stamp at its
|
||||||
# default, and bootstrap() reads a stamped-but-not-fresh database as
|
# default, and `bootstrap()` reads an unstamped existing database as
|
||||||
# ancient — it would replay all of the migrations against a schema
|
# very old. It would replay every migration against a schema that
|
||||||
# that already has every column, and fail on the first one.
|
# already has every column, and fail on the first one.
|
||||||
conn.execute(text(f"PRAGMA user_version = {LATEST_VERSION}"))
|
conn.execute(text(f"PRAGMA user_version = {LATEST_VERSION}"))
|
||||||
# In local mode (AIDND_MULTI_USER unset) get_current_user() looks for
|
# In local mode, which is when `AIDND_MULTI_USER` is unset,
|
||||||
# the row with email IS NULL and is_guest false. The fixture's user is
|
# `get_current_user()` looks for the row with `email IS NULL` and
|
||||||
# a registered one, so without this nothing owns the adventure and the
|
# `is_guest` false. The fixture's user is a registered one, so without
|
||||||
# app opens on an empty library.
|
# this update nothing owns the adventure and the app opens on an empty
|
||||||
|
# library.
|
||||||
conn.execute(text("UPDATE users SET email = NULL, is_guest = 0"))
|
conn.execute(text("UPDATE users SET email = NULL, is_guest = 0"))
|
||||||
|
|
||||||
|
|
||||||
@@ -675,9 +702,9 @@ def print_keep_notes(path: str, actions: int) -> None:
|
|||||||
print(f" .venv/Scripts/python.exe -m uvicorn app.main:app --port {port}")
|
print(f" .venv/Scripts/python.exe -m uvicorn app.main:app --port {port}")
|
||||||
print(f" cd frontend && AIDND_API_PORT={port} npm run dev")
|
print(f" cd frontend && AIDND_API_PORT={port} npm run dev")
|
||||||
print()
|
print()
|
||||||
# 8000 is the vite proxy's default and another local app squats it, which
|
# 8000 is the vite proxy's default, and another local app already listens
|
||||||
# shadows this API with its own SPA catch-all and looks like an empty
|
# there. That app shadows this API with its own SPA catch-all route, which
|
||||||
# database rather than a proxy problem.
|
# looks like an empty database rather than a proxy problem.
|
||||||
print(f" Port {port} rather than 8000 on purpose; AIDND_API_PORT points vite at it.")
|
print(f" Port {port} rather than 8000 on purpose; AIDND_API_PORT points vite at it.")
|
||||||
|
|
||||||
|
|
||||||
@@ -688,8 +715,8 @@ def parse_args(argv=None):
|
|||||||
p = argparse.ArgumentParser(
|
p = argparse.ArgumentParser(
|
||||||
prog="tools.stress_session", description=__doc__.splitlines()[0]
|
prog="tools.stress_session", description=__doc__.splitlines()[0]
|
||||||
)
|
)
|
||||||
# 607 is production's longest adventure as of 2026-08-17, and length is
|
# 607 is the longest adventure in production as of 2026-08-17. Length is
|
||||||
# the dimension the old default (200) got wrong: real actions are lighter
|
# the dimension the old default of 200 got wrong. Real actions are smaller
|
||||||
# than this fixture used to make them, but real stories run three times
|
# than this fixture used to make them, but real stories run three times
|
||||||
# longer, and length is what a page load pays for.
|
# longer, and length is what a page load pays for.
|
||||||
p.add_argument("--actions", type=int, default=600,
|
p.add_argument("--actions", type=int, default=600,
|
||||||
@@ -697,14 +724,14 @@ def parse_args(argv=None):
|
|||||||
"production's longest adventure is 607)")
|
"production's longest adventure is 607)")
|
||||||
p.add_argument("--memories", type=int, default=100,
|
p.add_argument("--memories", type=int, default=100,
|
||||||
help="memories, all embedded (default: 100)")
|
help="memories, all embedded (default: 100)")
|
||||||
# Deliberately not the app's default (80): a measuring instrument should
|
# This is not the app's default of 80. A measuring instrument holds the
|
||||||
# hold the fixture at the size asked for rather than evict it mid-run.
|
# fixture at the requested size rather than evict from it during a run.
|
||||||
p.add_argument("--capacity", type=int, default=200,
|
p.add_argument("--capacity", type=int, default=200,
|
||||||
help="Settings.memory_bank_capacity; lower it below "
|
help="Settings.memory_bank_capacity; lower it below "
|
||||||
"--memories to exercise eviction (default: 200)")
|
"--memories to exercise eviction (default: 200)")
|
||||||
# Production's longest adventure carries 886 B of text per action averaged
|
# The longest adventure in production carries 886 B of text per action,
|
||||||
# over both kinds. AI actions alternate with a one-line player input, so
|
# averaged over both kinds. AI actions alternate with a one-line player
|
||||||
# the AI half has to be about twice that.
|
# input, so the AI half has to be about twice that.
|
||||||
p.add_argument("--narration-bytes", type=int, default=1700,
|
p.add_argument("--narration-bytes", type=int, default=1700,
|
||||||
help="length of an AI action's text; alternating with a "
|
help="length of an AI action's text; alternating with a "
|
||||||
"one-line player input this averages ~890 B/action, "
|
"one-line player input this averages ~890 B/action, "
|
||||||
@@ -730,9 +757,10 @@ def parse_args(argv=None):
|
|||||||
"one — prefer it small (--rich --actions 30). Changes "
|
"one — prefer it small (--rich --actions 30). Changes "
|
||||||
"what the shapes cost, so do not compare a --rich run "
|
"what the shapes cost, so do not compare a --rich run "
|
||||||
"against a plain one")
|
"against a plain one")
|
||||||
# Read at import time by _early_keep as well — the database location has
|
# `_early_keep` also reads this flag at import time, because the database
|
||||||
# to be settled before app.database loads. Declared here so it appears in
|
# location has to be decided before `app.database` loads. It is declared
|
||||||
# --help and an unknown spelling is still rejected.
|
# here so that it appears in `--help` and an unknown flag is still
|
||||||
|
# rejected.
|
||||||
p.add_argument("--keep", metavar="PATH", default="",
|
p.add_argument("--keep", metavar="PATH", default="",
|
||||||
help="write the fixture to PATH and leave it bootable, so "
|
help="write the fixture to PATH and leave it bootable, so "
|
||||||
"the app can serve it in a browser (default: a temp "
|
"the app can serve it in a browser (default: a temp "
|
||||||
@@ -756,8 +784,8 @@ def main(argv=None) -> int:
|
|||||||
install_fakes(user_id, rng)
|
install_fakes(user_id, rng)
|
||||||
|
|
||||||
meter = Meter()
|
meter = Meter()
|
||||||
# After the fixture: building it is a write path nobody plays, and its
|
# Attach after the fixture is built. Building it is a write path no player
|
||||||
# bytes would drown everything the shapes report.
|
# takes, and its bytes would hide everything the shapes report.
|
||||||
meter.attach(engine)
|
meter.attach(engine)
|
||||||
|
|
||||||
print(f"fixture: {args.actions} actions × {args.narration_bytes} B "
|
print(f"fixture: {args.actions} actions × {args.narration_bytes} B "
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
"""A tree with a shape worth drawing — for driving the branch map by hand.
|
"""A tree with a shape worth drawing, for driving the branch map by hand.
|
||||||
|
|
||||||
`tools.branch_fixture` builds two branches of equal length, which is the case
|
`tools.branch_fixture` builds two branches of equal length, which is the case
|
||||||
the panel-refresh bug needed and the smallest tree that proves a switch. A map
|
the panel-refresh bug needed and the smallest tree that proves a switch. A map
|
||||||
|
|||||||
Reference in New Issue
Block a user