Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fork AI-DnD at d72f7c1 as the production base

Browse Source 
Establishes the production fork lineage decided in ADR 009. Upstream
AI-DnD is merged with --allow-unrelated-histories so this repository
carries the real upstream history alongside the planning package, and
future upstream commits can still be fetched and cherry-picked against
matching paths.

Upstream:  https://github.com/parththakkar106/AI-DnD
Commit:    d72f7c1bda
           "Stop paying twice for a block a retry can still throw away"
           Mon Aug 31 16:14:24 2026 +0000
           tip of upstream/main at the time of the fork
License:   MIT (c) 2026 Parth Thakkar — retained verbatim in LICENSE

No upstream file is modified by this commit; it is an import only.
Milestone M1 hardening lands on top of it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017foPNqFjAJa2Ngebf5mEfL
This commit is contained in:
JesseMarkowitz
2026-09-02 02:40:28 -04:00
co-authored by Claude Opus 5
parent 717670afe0 d72f7c1bda
commit 7f182a86e9
236 changed files with 53490 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.dockerignore
+19
View File
@@ -0,0 +1,19 @@
.git
.claude
plan
*.md
LICENSE
start.ps1
start.sh
# Secrets & local data — never in the image
*.env
backend/openrouter_key.env
*.db
# Build/runtime artifacts rebuilt inside the image
backend/.venv
**/__pycache__
**/*.pyc
frontend/node_modules
frontend/dist
.gitattributes
+5
View File
@@ -0,0 +1,5 @@
# Shell scripts must stay LF or the shebang/exec breaks on Linux/macOS,
# including when a Windows checkout is COPY'd into a Docker build.
*.sh text eol=lf
Dockerfile text eol=lf
*.ps1 text eol=crlf
.github/workflows/ci.yml
+74
View File
@@ -0,0 +1,74 @@
name: CI
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
backend:
name: Backend tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12" # matches the Dockerfile
cache: pip
cache-dependency-path: |
backend/requirements.txt
backend/requirements-dev.txt
- name: Install dependencies
working-directory: backend
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt -r requirements-dev.txt
- name: Run tests
working-directory: backend
run: python -m pytest tests/ -q
frontend:
name: Frontend lint + build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "24" # matches the Dockerfile
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci
- name: Lint
working-directory: frontend
run: npm run lint
- name: Build
working-directory: frontend
run: npm run build
docker:
name: Docker image builds
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- name: Build image
uses: docker/build-push-action@v6
with:
context: .
push: false
cache-from: type=gha
cache-to: type=gha,mode=max
.gitignore
+21
View File
@@ -0,0 +1,21 @@
# Secrets
*.env
backend/openrouter_key.env
# Python
backend/.venv/
__pycache__/
*.pyc
# Database
*.db
# Node
node_modules/
frontend/dist/
# Misc
.claude/
# Phase 8: auto-generated session/encryption secret (lives next to the DB)
secret.key
Dockerfile
+46
View File
@@ -0,0 +1,46 @@
# Stage 1 — build the React SPA. Node 24: package-lock.json is written by
# npm 11, which older bundled npms (node 22's 10.x) refuse as out-of-sync.
FROM node:24-alpine AS frontend-build
WORKDIR /build
COPY frontend/package.json frontend/package-lock.json ./
RUN npm ci
COPY frontend/ ./
RUN npm run build
# Stage 2 — build Python wheels (quickjs compiles from source if no wheel
# matches, so keep the toolchain out of the final image)
FROM python:3.12-slim AS python-build
RUN apt-get update && apt-get install -y --no-install-recommends gcc make \
&& rm -rf /var/lib/apt/lists/*
COPY backend/requirements.txt /tmp/requirements.txt
RUN pip wheel --no-cache-dir -r /tmp/requirements.txt -w /wheels
# Stage 3 — runtime
FROM python:3.12-slim
WORKDIR /app
COPY --from=python-build /wheels /wheels
RUN pip install --no-cache-dir /wheels/* && rm -rf /wheels
# Layout mirrors the repo: main.py finds the SPA at ../../frontend/dist
# relative to backend/app/main.py.
COPY backend/app /app/backend/app
COPY --from=frontend-build /build/dist /app/frontend/dist
# Database lives on a volume; parent dir is created by the app if missing.
ENV AIDND_DB_PATH=/data/data.db
VOLUME /data
EXPOSE 8000
WORKDIR /app/backend
# --proxy-headers lets uvicorn fix up the request scheme (https) behind the
# platform's edge. We deliberately do NOT pass --forwarded-allow-ips "*": that
# made uvicorn trust the LEFTMOST X-Forwarded-For value, which the client fully
# controls, so anyone could rotate the header to dodge the per-IP rate limits.
# The client IP used for rate limiting is derived in limits._client_ip from the
# hop the edge appends (rightmost), which a client cannot spoof past; tune with
# AIDND_TRUSTED_PROXY_HOPS if the platform adds more proxy hops.
# Single worker on purpose: the turn lock, rate limiter, and debug log are
# in-process state.
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", \
"--proxy-headers"]
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Parth Thakkar
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+292
View File
@@ -0,0 +1,292 @@
# AI D&D
[![CI](https://github.com/parththakkar106/AI-DnD/actions/workflows/ci.yml/badge.svg)](https://github.com/parththakkar106/AI-DnD/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
An AI Dungeon-style interactive storytelling app that runs entirely on your own machine, with
your own AI model. Create scenarios, play open-ended adventures where an LLM narrates the
world, and extend the engine with **JavaScript scripts compatible with real AI Dungeon
scripting**.
> ### ▶️ Try it live: **[parththakkar106.github.io/AI-DnD](https://parththakkar106.github.io/AI-DnD/)**
> The project page loads instantly and launches the hosted demo in one tap. Play a scenario as
> a guest: no sign-up and no API key needed. The demo runs on a free tier that sleeps, so the
> first load after it's been idle takes about 30 to 60 seconds to wake up.
>
> For the internals, read the **[design notes](https://parththakkar106.github.io/AI-DnD/guide.html)**.
> They walk through the context budgeting, the world-state referee, and the memory bank, and
> state the reasoning behind each one ([Markdown version](docs/GUIDE.md)).
Built with FastAPI and SQLAlchemy on the backend and React (Vite) on the frontend, running on
SQLite locally and Postgres in the cloud. It works with **any OpenAI-compatible endpoint**:
Ollama and LM Studio locally, or OpenRouter, OpenAI, Groq, or vLLM in the cloud. Endpoint, key,
and model are all runtime settings, and OpenRouter's free-tier models make the whole experience
cost nothing.
![The play screen, with the world-state rail open](docs/images/play-world-state.jpg)
*The play screen. The left rail shows live world state. The AI proposes changes each turn, and
a Python engine decides what actually sticks. The chip under the narration reports what
changed. The `‹ 2/2 ›` under a turn steps between the takes it has. Writing below a take that
isn't the live one starts a new branch.*
## Features
- **The full play loop.** Do / Say / Story / Continue actions, streamed AI responses (SSE),
retry, undo, and edit. Reasoning models are supported: "thinking" streams into a collapsible
💭 panel with its own token budget.
- **A branching story tree.** The story is a tree, not a list. Any turn can hold more than one
**take**, and `‹ 2/4 ›` steps between them. Stepping is free: the story below simply empties,
and the server is told nothing. Writing below a take that isn't the live one is what makes a
branch. Branches borrow their ancestors' turns instead of copying them, so a fork costs about
100 bytes, and a 20-fork story loads within 1% of the same story flat. Switching restores that
line's world state, script state, and cooldown clocks. A branch panel switches, renames, and
deletes; **⌗ See the tree** draws every line against the story's own clock
(`backend/app/tree.py`, `backend/app/context/lineage.py`).
- **An RPG world-state engine.** A scenario can declare stats, flags, milestones, and a named
cast; the adventure carries their live values. The AI proposes deltas, and a Python engine
referees them: it clamps values to range, enforces per-turn caps and cooldowns, keeps counters
monotonic and milestones sticky, then strips the machine-readable block out of the prose
(`backend/app/worldstate/engine.py`). Word-labeled bands (`40–60: minor damage`) make the
model reliable at it. No dice and no scripting are required.
- **AI Dungeon-compatible context engine.** Memory, author's note, and story cards (world
info) are triggered by keywords in recent story text, then assembled under a token budget
(`backend/app/context/builder.py`).
- **Insights: total prompt transparency.** Every turn stores the exact prompt sent to the
model. Open 🔍 on any AI action to see each context component, its token cost, and why it was
included.
- **JavaScript scripting, AI Dungeon-compatible.** `onInput` / `onModelContext` / `onOutput`
modifiers share `state` and a `worldEntries` API, and run in an embedded quickjs sandbox
(`backend/app/scripting/`). Real AI Dungeon scripts import and run as is. An in-app CodeMirror
editor is included.
- **Auto-summarization and Memory Bank.** The modern AI Dungeon memory system: AI-generated
memories every few actions, a running story summary, and embedding-based retrieval that
pulls old-but-relevant facts back into context, with similarity scores visible in Insights
(`backend/app/memorybank.py`).
- **Undo and retry that actually roll back state.** Undo and retry roll back the world state
and script state to a per-node snapshot, not just the text, and prune the memories that
covered the removed turns. Nothing a retry replaces is discarded: the old attempt stays as
another take of that turn, one keystroke and one click from becoming a branch of its own.
- **Import and export.** AI Dungeon-compatible formats for scripts and scenarios; JSON for
everything else. An adventure exports as `ai-dnd-adventure-v2`, which carries the whole tree:
every branch, every take, and the fork points, since those were chosen rather than computed.
Files saved in the old single-line format still import.
- **Optional accounts for hosted deployments.** By default the app is single-user with zero
auth friction. Set `AIDND_MULTI_USER=1` and visitors play instantly as guests (signed
session cookie), can register (email and password) at any point to keep their data, and each
user gets isolated data plus their own encrypted-at-rest API key. A server-funded **shared
demo key** with a daily turn cap lets people try it without bringing a key
(`backend/app/auth.py`). Each new guest is also given a copy of a short pre-played
adventure, so the first screen shows real turns and their world-state changes without
spending a demo turn (`backend/app/starter.py`).
## Screenshots
| | |
|---|---|
| ![Insights panel](docs/images/insights.jpg) | ![Scenario editor](docs/images/scenario-editor-npcs.jpg) |
| **Insights**: the exact prompt for the next turn, broken into components with token counts and the trigger word that pulled each story card in. | **Authoring**: stats with ranges, per-turn caps, cooldowns, and word-labeled bands; NPCs the AI addresses by id. |
| ![Script editor](docs/images/script-editor.jpg) | ![Home](docs/images/home.jpg) |
| **Scripting**: the three AI Dungeon hooks with shared persistent `state`, run in a quickjs sandbox. | **Home**: continue a story in progress or start from a scenario. |
| ![The branch map](docs/images/branch-map.jpg) | ![The branches panel](docs/images/branches-panel.jpg) |
| **The tree**: one lane per line, from the moment it left its parent to the moment it ends. The horizontal axis is the story's own clock, so a short branch reads as short. | **Branches**: every line the story has taken, and the three things you can do to one. A line the one you're reading was forked from can't be deleted, and says so. |
## Quick start
### Docker (any OS)
```sh
docker compose up --build
```
Open http://localhost:8000. Your data persists in a named volume across restarts.
### Windows
```powershell
cd backend; python -m venv .venv; .\.venv\Scripts\pip.exe install -r requirements.txt; cd ..
cd frontend; npm install; cd ..
.\start.ps1
```
Open http://localhost:5173 (dev servers; API docs at http://localhost:8000/docs).
### macOS / Linux
```sh
./start.sh # creates the venv and installs dependencies on first run
```
Open http://localhost:5173.
## Connect a model
Open **Settings** in the app and point it at any OpenAI-compatible endpoint:
| Provider | Endpoint URL | Notes |
|---|---|---|
| Ollama (local) | `http://localhost:11434/v1` | free, private; also serves embedding models for the Memory Bank (e.g. `nomic-embed-text`) |
| LM Studio (local) | `http://localhost:1234/v1` | free, private |
| OpenRouter | `https://openrouter.ai/api/v1` | `:free` models cost nothing (no embeddings on the free tier) |
| OpenAI / Groq / vLLM / … | provider's `/v1` URL | anything speaking `/v1/chat/completions` |
| Claude Code CLI (local) | `http://127.0.0.1:8787/v1` | your Claude subscription instead of an API key; see [Playing against Claude locally](#playing-against-claude-locally) |
Model name, API key, generation parameters, and (optionally) summary and embedding models for
the Memory Bank are all configured there too. No config files and no rebuild are needed.
### Playing against Claude locally
`backend/tools/claude_shim.py` serves an OpenAI-compatible endpoint backed by the
`claude` command line tool, so you can play the demos against a real model without an
API key. Each request spawns one `claude --print` process, which suits the turn engine:
the app assembles the whole prompt every turn and expects a stateless endpoint.
```sh
cd backend
.venv/Scripts/python.exe tools/claude_shim.py # listens on 127.0.0.1:8787
```
In Settings, choose the OpenAI-compatible provider, set the base URL to
`http://127.0.0.1:8787/v1`, put any non-empty string in the API key field, and pick
`sonnet`. The shim ignores the key and authenticates as you, through the CLI. Set the
reasoning budget to `0` or `-1`: a positive budget sends a `reasoning.max_tokens` field
that Claude 5 models reject.
Embeddings are not served. Leave the embedding model blank, or point the Memory Bank at
a real endpoint.
Run it against a local backend only. The endpoint has no authentication, and anything
reaching it spends your Claude quota. `app/netguard.py` blocks localhost endpoints when
`AIDND_MULTI_USER` is set, so a deployed instance cannot be pointed at it.
## How a turn works
```
player input
→ onInput script modifier
→ assemble context: [narrator prompt] + [world state + stat guide] + [AI instructions]
+ [plot essentials] + [story summary] + [retrieved memories]
+ [triggered story cards] + [history along this branch, token-budgeted]
+ [author's note] + [player action]
→ onModelContext script modifier
→ snapshot context (Insights)
→ provider adapter → AI (streamed)
→ extract + referee the world-state delta block, strip it from the prose
→ onOutput script modifier
→ store & render
```
## Architecture
```
frontend/ React + Vite SPA ──HTTP/SSE──► backend/ FastAPI
├─ routers/ auth, scenarios, adventures, story cards, scripts, chat, settings, analytics, debug
├─ models.py SQLAlchemy: User, Scenario, Adventure, Branch, Action, StoryCard, Script, Settings, Memory
├─ migrations.py hand-rolled, versioned via PRAGMA user_version (64 and counting)
├─ auth.py guest/registered users, sessions, shared demo key
├─ security.py password hashing, cookie signing, API-key encryption
├─ tree.py forking, promotion, and where a node is placed
├─ attempts.py the takes of one turn, grouped by parent
├─ context/ prompt assembly under a token budget + lineage/history windowing
├─ worldstate/ the stat engine: clamps, cooldowns, bands, milestones
├─ scripting/ quickjs sandbox + AI Dungeon API surface
├─ memorybank.py auto-summarization + embedding retrieval
├─ analytics.py buffered visit counters + the owner's dashboard query
├─ bundle.py the export/import formats, v2 (tree) and a v1 reader
├─ providers/ OpenAI-compatible adapter, streaming
└─ data.db SQLite (path overridable via AIDND_DB_PATH)
```
In production the backend serves the built SPA from one port (see `Dockerfile`). In
development, Vite proxies `/api` to FastAPI.
## Tests
549 backend tests: unit tests plus full HTTP integration through the real quickjs scripting
engine, with the LLM provider mocked. CI runs them on every push, alongside the frontend
lint/build and a Docker image build.
```sh
cd backend && pip install -r requirements.txt -r requirements-dev.txt
python -m pytest tests/
```
## Notes on performance
Two of the tests exist because of bugs that were measured rather than guessed at. They are the
most interesting engineering in the repo.
- **Database egress, cut about 189x.** Every adventure load pulled `Action.context_snapshot`,
the entire assembled prompt at about 74 KB per turn, to read two small fields off it. Moving
those fields into their own columns and marking the heavy ones `deferred` took one adventure
load from 38.5 MB to 0.20 MB. The backfill runs server-side with dialect-specific SQL, so the
old data never crosses the wire. `tests/test_egress.py` hooks into SQLAlchemy's cursor events
and fails if a bulk load ever names those columns again.
- **Turn cost, made flat.** Assembling a turn walked the whole story, so cost scaled with story
length: 839 KB of reads at turn 200. `backend/app/context/history.py` now serves tails and
slices from SQL and measures what it fetched. The same turn now costs 129 KB and stops
growing at around turn 50.
- **Branching that costs nothing to read.** A branch stores no turns. It stores where it left
its parent, and borrows everything above that. A 40-turn story forked twenty times loads in
31,652 bytes against 31,433 bytes for the same story flat: a 1.007x ratio, or about 103 bytes
per branch. Reads stay cheap because the lineage is windowed the same way the history is, so
the number of SQL clauses is bounded by the context window rather than by the number of
forks.
## Visit analytics
The hosted demo keeps its own analytics: an owner-only dashboard at `/analytics` shows
traffic, which shared scenarios get played, turns and demo-key spend, errors, and a funnel
from *visited* to *played a turn* to *signed up*. It is visible only to the emails listed in
`AIDND_ANALYTICS_EMAILS`, and the route returns 404 for everyone else.
This is built into the app rather than added with a third-party script, for reasons specific
to this project: the CSP allows only `script-src 'self'`, ad blockers block the popular
trackers, and none of those trackers can see the measurement that matters here, a turn. Counts
are aggregated in memory and flushed as UPSERTs, so a visit is a write and never a read, and
every dashboard query is a `GROUP BY` that returns tens of rows regardless of traffic volume.
That matters: see the egress note above for what reading rows per request costs on this stack.
## Deploy (Render)
The repo ships a [`render.yaml`](render.yaml) blueprint: one Docker web service that serves
the SPA and API same-origin, backed by external [Neon](https://neon.tech) Postgres. The free
Render tier has no persistent disk, so the database lives off-box.
1. Create a **Neon** project and copy its pooled connection string.
2. In Render, choose **New → Blueprint** and point it at this repo. Render reads
`render.yaml`.
3. Fill in the secrets it prompts for (`sync: false` vars): `AIDND_DATABASE_URL` (the Neon
string); `AIDND_DEMO_API_KEY` and `AIDND_DEMO_MODELS` to offer a no-signup demo; and
`AIDND_ANALYTICS_EMAILS` (your own account's email) to see the Visitors dashboard.
`AIDND_SECRET_KEY` is generated automatically and stays stable across deploys.
4. Deploy. Pushes to `main` auto-deploy after this. The health check is `/api/health`.
On the free tier the service sleeps after about 15 minutes idle, and the first request after
that takes about 30 to 60 seconds to wake it. Point any keep-warm pinger at `/api/health`,
which deliberately doesn't touch the database: waking the database around the clock costs far
more than the cold start saves.
If you put another proxy or CDN in front of Render, set `AIDND_TRUSTED_PROXY_HOPS` to the
number of proxies in the chain. It defaults to 1. The rate limiter reads the client IP that
many entries from the right of `X-Forwarded-For`, because the trusted edge appends the real
one last. Leave it at 1 behind two proxies and the limiter reads an entry the caller
supplied, so anyone can rotate the header for a fresh rate-limit bucket per request.
## Repo notes
- `plan/` holds the phased implementation plan this project was built from, kept as a build
log. All fourteen phases are complete. The later files (11, 12, 14) also serve as design
notes for the state-revert, world-state, and story-tree work.
[`plan/STATUS.md`](plan/STATUS.md) is the running thread: what shipped, what was measured,
and what is owed next.
- [`docs/GUIDE.md`](docs/GUIDE.md) holds design notes: how each subsystem works and why it was
built that way, with the measurements behind the decisions. It is also rendered as a
[reading page](https://parththakkar106.github.io/AI-DnD/guide.html).
- `backend/.env.example` lists the few environment variables the backend reads.
- [`docs/self-review.md`](docs/self-review.md) records a full-codebase self-review pass and
what came out of it. All correctness findings are resolved.
## License
[MIT](LICENSE)
backend/.env.example
+110
View File
@@ -0,0 +1,110 @@
# Environment variables read by the backend.
#
# NOTE: the app reads real environment variables — it does NOT auto-load this
# file. Set them in your shell, in docker-compose.yml, or in your host's
# dashboard. This file is documentation (and a template for deploy configs).
# Absolute path for the SQLite database file. Parent directory is created if
# missing. Default when unset: backend/data.db
# Docker compose sets this to /data/data.db (a named volume).
AIDND_DB_PATH=
# ---------------------------------------------------------------------------
# Phase 9 — production hardening
# ---------------------------------------------------------------------------
# Switch from SQLite to a server database (hosted deploys use Neon Postgres).
# Any SQLAlchemy URL; postgres:// and postgresql:// schemes are rewritten to
# the psycopg3 driver automatically. The platform-conventional DATABASE_URL
# is honored too (AIDND_DATABASE_URL wins if both are set). Unset = SQLite.
AIDND_DATABASE_URL=
# Comma-separated list of allowed CORS origins. Only needed when the frontend
# is served from a different origin than the API; the production build is
# served same-origin by FastAPI, so hosted deploys can leave this unset.
# Default: http://localhost:5173,http://127.0.0.1:5173 (the Vite dev server).
AIDND_CORS_ORIGINS=
# How many proxy hops the rate limiter trusts in `X-Forwarded-For`. It reads
# the entry that many places from the right, because the trusted edge appends
# the real client IP last. Set this to the number of proxies in front of the
# app. Default: 1, which is correct for a single edge such as Render.
#
# Get it wrong in either direction and the rate limits weaken. Too low reads an
# entry the caller supplied, so anyone can rotate the header for a fresh
# rate-limit bucket per request and walk past the auth and guest limits. Too
# high reads past the real client. Only multi-user mode rate-limits at all, so
# local installs can ignore this.
AIDND_TRUSTED_PROXY_HOPS=
# ---------------------------------------------------------------------------
# Phase 8 — optional accounts & multi-user (all optional; defaults keep the
# app in frictionless single-user "local mode")
# ---------------------------------------------------------------------------
# "1"/"true" turns on multi-user mode: guest sessions via signed cookies,
# register/login UI, per-user data. Leave unset for local installs.
AIDND_MULTI_USER=
# Secret for signing session cookies and encrypting stored API keys at rest.
# If unset in local mode, one is auto-generated into `secret.key` next to the
# database (fine for local/docker-volume runs). REQUIRED when
# AIDND_MULTI_USER is on — the app refuses to start without it, because a
# regenerated secret on an ephemeral hosted filesystem would log out every
# user on each deploy. Generate one:
# python -c "import secrets; print(secrets.token_urlsafe(48))"
AIDND_SECRET_KEY=
# Session cookie Secure flag (HTTPS-only). Defaults to on when
# AIDND_MULTI_USER is on, off otherwise — set 0/1 only to override (e.g. 0
# when testing multi-user mode over plain http on a LAN address).
AIDND_COOKIE_SECURE=
# --- Shared demo key (BYOK fallback; only active when AIDND_MULTI_USER=1) ---
# Users with no API key of their own get this server-funded endpoint with a
# model whitelist and a per-day turn cap. Unset = no demo, users must bring
# their own key. Memory bank/auto-summarization are disabled on demo turns.
AIDND_DEMO_API_KEY=
# Default endpoint if unset: https://openrouter.ai/api/v1
AIDND_DEMO_ENDPOINT_URL=
# Comma-separated model whitelist. Default: google/gemma-4-26b-a4b-it:free
AIDND_DEMO_MODELS=
# Successful AI turns per user per day on the demo key. Default: 20
AIDND_DEMO_TURNS_PER_DAY=
# Comma-separated emails of "power users" (trusted testers) who bypass the daily
# demo cap entirely — unmetered turns on the shared demo key — and get the AI Chat
# page (a plain scratchpad for talking to a model, hidden from everyone else).
# Registered accounts only (guests have no email). Matched case-insensitively.
# Local (single-user) installs are always treated as power users.
AIDND_POWER_USERS=
# --- Visit analytics ---
# Comma-separated emails allowed to see the Visitors dashboard (/analytics) and
# its nav link. Deliberately separate from AIDND_POWER_USERS: a trusted tester
# gets unmetered turns, which is no reason to hand them the traffic numbers.
# Unset = nobody sees it in a hosted deploy. Local installs always can, and are
# the only mode where the viewer's own visits are still counted (excluding them
# would leave the page permanently empty on the machine it's developed on).
# Collection itself is always on; only the dashboard is gated.
AIDND_ANALYTICS_EMAILS=
# Days to keep the one-row-per-visitor-per-day table that makes the funnel
# count people rather than clicks. The daily counters are aggregate and kept
# forever. Default: 400. Set 0 to keep visitor-days forever.
AIDND_ANALYTICS_RETENTION_DAYS=
# --- Guest retention (only active when AIDND_MULTI_USER=1) ---
# Every first visit mints a guest account, so a public demo collects one row
# per visitor. A guest with no activity for this many days is deleted along
# with its scenarios, adventures and actions. Registered accounts are never
# touched. Default: 5. Set 0 to keep guests forever.
AIDND_GUEST_RETENTION_DAYS=
# How often a running process re-checks. The sweep also runs once at startup,
# which is what actually fires on hosts that sleep. Default: 6
AIDND_CLEANUP_INTERVAL_HOURS=
# The AI endpoint/API key/model are NOT env vars — they are configured at
# runtime in the app's Settings page and stored (encrypted) in the database.
#
# Rate limits, request size limits, and per-user row caps are hardcoded with
# generous values (see backend/app/limits.py) and active only in multi-user
# mode — local installs are never throttled.
backend/app/__init__.py
View File
backend/app/accesslog.py
+168
View File
@@ -0,0 +1,168 @@
"""The access log: who arrived, when, and from where.
The deliberate opposite of analytics.py. That module counts and stores nothing
that points at a person; this one records addresses, email addresses and
devices, because an access log that cannot identify the access is not an access
log. The two live in separate modules and separate tables on purpose, so that
the anonymity of the counters is a property of the code rather than a convention
someone has to remember.
Owner-only, and never shown to the people it records.
Four kinds of row:
- `session` A browser that has a session made a request. For a guest, this
is their first visit.
- `login` An existing account signed in.
- `register` A guest upgraded to an account.
- `login_failed` A password attempt that did not match, with the address tried.
Session rows are the only ones that need thinning. `/auth/me` runs on every page
load, and one row per load would be noise rather than a log. A row is written
when the day or the address changes for that user. That is the granularity a log
is read at, such as seen on the 3rd from 1.2.3.4, and it still records someone
moving networks during a day.
"""
import logging
import threading
from sqlalchemy import desc, or_, select
from sqlalchemy.orm import Session
from . import analytics, models
logger = logging.getLogger(__name__)
SESSION = "session"
LOGIN = "login"
REGISTER = "register"
LOGIN_FAILED = "login_failed"
MAX_UA = 200
# user id -> (day, ip) of the last session row written for them. Process-local
# like the rate limiter's windows, and for the same reason: this is a single
# process, and the worst case after a restart is one redundant row per user.
_last_session: dict[int, tuple[str, str]] = {}
_guard = threading.Lock()
_MAX_TRACKED = 10_000
def _client_ip(request) -> str:
# This import is deferred. `limits` imports `auth`, which the routers that
# call this function import, so a module-level import here would create a
# cycle. The spoof resistance lives in `limits` and must not be
# reimplemented. A second, looser answer to which address belongs to the
# client is how one of them ends up trusting a header it should not.
from . import limits
return limits.client_ip(request)
def describe(user: models.User) -> str:
"""Returns how a user is named in the log.
A guest has no email, and their id is the only handle anyone has for them.
The third case is a local install's implicit single user, who also has no
email but is the operator rather than a visitor. Naming that user "Guest #1"
would be wrong in the one row they are certain to read.
"""
if user.email:
return user.email
return f"Guest #{user.id}" if user.is_guest else f"Local user #{user.id}"
def _country(request) -> str:
"""Returns the edge's country header, or "" when there is none.
The blank differs from the counters' "(unknown)" label. A table column reads
better as a dash than as a word, and an empty string is the correct value for
a country that is not known.
"""
country = analytics.country_of(request.headers)
return "" if country == analytics.UNKNOWN else country
def record(
db: Session,
kind: str,
request,
*,
user: models.User | None = None,
who: str | None = None,
) -> None:
"""Writes one row.
This function never raises. The log observes sign-in rather than guarding it,
and a logging failure must not lock anyone out.
"""
try:
event = models.AccessEvent(
kind=kind,
user_id=user.id if user is not None else None,
who=(who if who is not None else describe(user) if user else "")[:320],
is_guest=bool(user.is_guest) if user is not None else False,
ip=_client_ip(request)[:45],
country=_country(request),
device=analytics.device_of(request.headers.get("user-agent", "")),
user_agent=(request.headers.get("user-agent") or "")[:MAX_UA],
)
db.add(event)
db.commit()
except Exception: # pragma: no cover - defensive
db.rollback()
logger.exception("Access log write failed; continuing.")
def note_session(db: Session, user: models.User, request) -> None:
"""Records that a session made a request, at most one row per day per address."""
try:
today = analytics._today()
ip = _client_ip(request)
with _guard:
if _last_session.get(user.id) == (today, ip):
return
_last_session[user.id] = (today, ip)
if len(_last_session) > _MAX_TRACKED:
# Nothing here needs to persist. Clearing the map costs at most
# one extra row per active user.
_last_session.clear()
_last_session[user.id] = (today, ip)
except Exception: # pragma: no cover - defensive
logger.exception("Access log session check failed; continuing.")
return
record(db, SESSION, request, user=user)
def recent(
db: Session,
*,
limit: int = 50,
before_id: int | None = None,
kind: str | None = None,
query: str | None = None,
) -> dict:
"""Returns a page of the log, newest first.
The page is anchored on a row id rather than an offset, as the story pager
is. Rows keep arriving while the log is read, and an offset would shift the
page under whoever is reading it.
"""
statement = select(models.AccessEvent).order_by(desc(models.AccessEvent.id))
if before_id is not None:
statement = statement.where(models.AccessEvent.id < before_id)
if kind:
statement = statement.where(models.AccessEvent.kind == kind)
if query:
like = f"%{query.strip()}%"
statement = statement.where(or_(
models.AccessEvent.who.ilike(like),
models.AccessEvent.ip.ilike(like),
models.AccessEvent.country.ilike(like),
))
# Requesting one extra row reports whether more rows exist, without a
# second COUNT over the whole table.
rows = list(db.scalars(statement.limit(limit + 1)))
has_more = len(rows) > limit
return {"events": rows[:limit], "has_more": has_more}
backend/app/analytics.py
+583
View File
@@ -0,0 +1,583 @@
"""Visit analytics for the hosted demo.
This is a small self-hosted counter that answers whether anyone visited and
whether they played. It is built into the app rather than added with a
third-party script, because the CSP in `main.py` allows scripts from 'self'
only, ad blockers block the popular trackers, and none of those trackers can see
what is worth knowing here: turns taken, demo-key spend, and which seeded
scenario people pick.
Three rules shape the design:
1. It stores nothing personal. It records no IP addresses, no user agents, no
user ids, and no title of anything a player wrote. A visitor appears only as
an HMAC of their user id, which is one-way and salted with the app's secret
key, so these tables cannot be joined back to an account even by someone
holding the database. Story content never reaches this module. What one
specific person did is unanswerable by design, and only totals are
available.
2. Egress is the budget. Neon bills for bytes leaving the database, and this
project has already paid for forgetting that once. Counts are therefore
aggregated in memory and flushed as UPSERTs, so a visit is a write and never
a read, and every dashboard query is a GROUP BY that returns tens of rows
rather than per-visit rows. A month of traffic costs a few kilobytes to read
back.
3. The numbers come from the server, not from the browser. The client reports
one thing, which is the page that was viewed. Everything with meaning, such
as a turn happening or an account being created, is recorded by the code that
performs it, where a stranger cannot fake it and an extension cannot block
it.
Storage is two tables, both bounded. `analytics_daily` holds one counter row per
day, metric, and label, which is a few dozen rows a day.
`analytics_visitor_days` holds one row per visitor per day carrying the funnel
flags, which is what makes the funnel count people rather than clicks. It is the
only table that grows with traffic, and cleanup ages it out.
"""
import hmac
import logging
import os
import re
import threading
from datetime import timedelta
from hashlib import sha256
from urllib.parse import urlsplit
from sqlalchemy import case, func, or_, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy.orm import Session
from . import models, security
logger = logging.getLogger(__name__)
# ---------- Metrics ----------
# `metric` is the family, and `label` is the bucket within it. One generic
# counter table is better than a column per measurement, because adding a new
# question later costs nothing rather than a migration.
M_PAGE = "pageview"
M_EVENT = "event"
M_REFERRER = "referrer"
M_DEVICE = "device"
M_COUNTRY = "country"
M_SCENARIO = "scenario" # Which seeded or public scenario was played.
M_ERROR = "error" # "<status> <route>" for a 4xx or 5xx on /api.
EV_SCENARIO_OPEN = "scenario_opened"
EV_ADVENTURE = "adventure_created"
EV_IMPORT = "adventure_imported"
EV_TURN = "turn"
EV_DEMO_TURN = "demo_turn" # A turn billed to the shared demo key.
EV_TURN_ERROR = "turn_error"
EV_SIGNUP = "signup"
EV_LOGIN = "login"
# Events that are also funnel steps. Recording one sets a flag on the visitor's
# row for the day, so the funnel counts distinct visitor-days rather than repeat
# clicks. This name-to-column map is the whole definition of the funnel, and the
# dashboard reads it back in this order.
FUNNEL_FLAGS = {
EV_SCENARIO_OPEN: "opened",
EV_ADVENTURE: "created",
EV_TURN: "played",
EV_SIGNUP: "signed_up",
}
OTHER = "(other)"
NONE_LABEL = "(direct)"
UNKNOWN = "(unknown)"
# ---------- Bounds ----------
# These bounds exist so that a hostile visitor can add rows to these tables no
# faster than an honest one. The only label a client can influence is the
# referrer, and together these caps mean the worst it can do is fill one day's
# referrer list and then be folded into "(other)".
MAX_LABEL_LEN = 80
MAX_LABELS_PER_METRIC = 200 # Distinct labels per metric per day, then OTHER.
MAX_PENDING = 4000 # Buffered entries before an inline flush.
FLUSH_INTERVAL_SECONDS = 60
# How long the per-visitor-day rows are kept. The daily counters are small and
# are kept indefinitely. These rows are the ones that scale with traffic. A
# visitor whose last visit ages out counts as new again, which is an acceptable
# trade at this horizon and keeps the table from being a permanent record of
# anyone.
RETENTION_DAYS = int(os.environ.get("AIDND_ANALYTICS_RETENTION_DAYS", "400") or 400)
_HOST_OK = re.compile(r"^[a-z0-9.-]+$")
_COUNTRY_OK = re.compile(r"^[A-Z]{2}$")
_NUMERIC_SEGMENT = re.compile(r"^\d+$")
# SPA routes, in the form the dashboard shows them. Any other path a client
# reports becomes OTHER, so the page list cannot be filled with junk and cannot
# record which adventure someone is reading.
KNOWN_ROUTES = {
"/", "/adventures", "/scenarios", "/scenarios/:id", "/play/:id",
"/scripts", "/scripts/:id", "/settings", "/chat", "/analytics",
}
# ---------- In-process buffer ----------
# The deployment is a single process, which is the same assumption `limits.py`
# makes, so a plain dict under a lock is the whole design. Losing up to a minute
# of counts to a hard restart is acceptable for traffic numbers, and the flusher
# also runs on shutdown. On Render's free tier the service is idle when it
# sleeps, so the buffer it sleeps on is empty.
_counts: dict[tuple[str, str, str], int] = {}
_visits: dict[tuple[str, str], set[str]] = {} # (day, visitor) -> flags.
_labels_seen: dict[tuple[str, str], set[str]] = {} # (day, metric) -> labels.
_guard = threading.Lock()
def _today() -> str:
return models.utcnow().date().isoformat()
def record(metric: str, label: str = "", *, n: int = 1) -> None:
"""Adds `n` to one counter.
This function never raises. Analytics must not fail a request that it is
only observing.
"""
try:
day = _today()
label = (label or "").strip()[:MAX_LABEL_LEN]
with _guard:
seen = _labels_seen.setdefault((day, metric), set())
if label not in seen:
if len(seen) >= MAX_LABELS_PER_METRIC:
label = OTHER
else:
seen.add(label)
key = (day, metric, label)
_counts[key] = _counts.get(key, 0) + n
pending = len(_counts) + len(_visits)
except Exception: # pragma: no cover - defensive
logger.exception("Analytics counter failed; continuing.")
return
if pending >= MAX_PENDING:
flush()
def visitor_id(user: models.User) -> str:
"""Returns a stable, one-way handle for one visitor.
The handle is an HMAC of the user id under the app's secret key. It is
stable, so a returning visitor can be distinguished from a new one. It is
one-way, so nothing in the analytics tables points back at an account. It is
keyed, so a client cannot compute one and claim to be someone else. One
consequence follows: rotating `AIDND_SECRET_KEY` makes every returning
visitor look new.
"""
digest = hmac.new(security.SECRET_KEY, f"visitor:{user.id}".encode(), sha256)
return digest.hexdigest()[:32]
def record_visit(user: models.User | None, *, flag: str | None = None) -> None:
"""Records that this visitor was here today, and optionally sets one funnel
flag.
Without a user the call does nothing. A page loaded before a session exists
still counts as a pageview, but not as a person.
"""
if user is None:
return
try:
with _guard:
flags = _visits.setdefault((_today(), visitor_id(user)), set())
if flag:
flags.add(flag)
except Exception: # pragma: no cover - defensive
logger.exception("Analytics visit failed; continuing.")
def record_event(name: str, user: models.User | None = None) -> None:
"""Records one event, and credits the visitor's day if it is a funnel step.
This is the whole interface the call sites use.
"""
record(M_EVENT, name)
record_visit(user, flag=FUNNEL_FLAGS.get(name))
# ---------- Normalizing what the browser reports ----------
def normalize_route(path: str) -> str:
"""Reduces a client-reported path to one of `KNOWN_ROUTES`.
Numeric segments become ":id". That bounds the label count, and it keeps
which adventure someone opened out of the statistics.
"""
path = (path or "/").split("?")[0].split("#")[0]
if not path.startswith("/"):
path = "/" + path
if len(path) > 1:
path = path.rstrip("/")
parts = [":id" if _NUMERIC_SEGMENT.match(p) else p for p in path.split("/")]
route = "/".join(parts) or "/"
return route if route in KNOWN_ROUTES else OTHER
def normalize_referrer(referrer: str, own_host: str = "") -> str:
"""Returns the sending site as a bare host.
This app's own host means an internal navigation, which is not a referral.
In that case the function returns "", which tells the caller to skip it.
"""
if not referrer:
return NONE_LABEL
host = (urlsplit(referrer).hostname or "").lower().lstrip(".")
if not host or not _HOST_OK.match(host) or len(host) > MAX_LABEL_LEN:
return OTHER
if host == (own_host or "").lower() or host in ("localhost", "127.0.0.1"):
return ""
return host[4:] if host.startswith("www.") else host
def api_route_label(scope: dict, status: int) -> str:
"""Returns an error bucket such as "500 /api/adventures/{adventure_id}".
The label uses the route template, never the request path. That keeps one
bucket per endpoint rather than one per adventure id. It also bounds the
table: an unmatched path is chosen entirely by the caller, so labeling by it
would let anyone create rows by requesting arbitrary paths.
"""
template = getattr(scope.get("route"), "path", None)
return f"{status} {template}" if template else f"{status} (unmatched)"
def device_of(user_agent: str) -> str:
"""Returns "mobile", "tablet", or "desktop", and nothing more specific.
The user-agent string itself is never stored, because it is a fingerprint
and the useful answer is one word.
"""
ua = (user_agent or "").lower()
if not ua:
return UNKNOWN
if any(bot in ua for bot in ("bot", "crawler", "spider", "headless", "preview")):
return "bot"
if "ipad" in ua or "tablet" in ua or ("android" in ua and "mobile" not in ua):
return "tablet"
if any(m in ua for m in ("mobi", "iphone", "ipod", "android", "phone")):
return "mobile"
return "desktop"
# Geo headers an edge network may add. Render fronts services with a CDN that
# can set `cf-ipcountry`, and the others cost nothing to check. A value is
# trusted only if it looks like an ISO code, because a client can send any
# header, so the worst case is a wrong country rather than an unbounded label.
_GEO_HEADERS = ("cf-ipcountry", "x-vercel-ip-country", "x-geo-country", "x-country-code")
def country_of(headers) -> str:
for name in _GEO_HEADERS:
value = (headers.get(name) or "").strip().upper()
if _COUNTRY_OK.match(value) and value != "XX":
return value
return UNKNOWN
# ---------- Flushing ----------
def _insert(db: Session):
return sqlite_insert if db.get_bind().dialect.name == "sqlite" else pg_insert
def _drain() -> tuple[dict, dict]:
with _guard:
counts, visits = _counts.copy(), _visits.copy()
_counts.clear()
_visits.clear()
# The label sets bound cardinality within one day, so drop the
# previous day's rather than grow a map that never shrinks.
today = _today()
for key in [k for k in _labels_seen if k[0] != today]:
del _labels_seen[key]
return counts, visits
def _restore(counts: dict, visits: dict) -> None:
"""Returns a failed flush's work to the buffer, so the next flush retries it."""
with _guard:
for key, n in counts.items():
_counts[key] = _counts.get(key, 0) + n
for key, flags in visits.items():
_visits.setdefault(key, set()).update(flags)
def flush(db: Session | None = None) -> None:
"""Writes the buffer out. This is safe to call from anywhere and never raises."""
counts, visits = _drain()
if not counts and not visits:
return
own_session = db is None
if own_session:
from .database import SessionLocal
db = SessionLocal()
try:
_write_counts(db, counts)
_write_visits(db, visits)
db.commit()
except Exception:
db.rollback()
_restore(counts, visits)
logger.exception("Analytics flush failed; counts held for the next one.")
finally:
if own_session:
db.close()
def _write_counts(db: Session, counts: dict) -> None:
if not counts:
return
table = models.AnalyticsDaily.__table__
rows = [
{"day": day, "metric": metric, "label": label, "hits": hits}
for (day, metric, label), hits in counts.items()
]
stmt = _insert(db)(table).values(rows)
db.execute(stmt.on_conflict_do_update(
index_elements=["day", "metric", "label"],
set_={"hits": table.c.hits + stmt.excluded.hits},
))
def _write_visits(db: Session, visits: dict) -> None:
if not visits:
return
table = models.AnalyticsVisitorDay.__table__
ids = {visitor for _, visitor in visits}
# One indexed lookup decides new against returning for the whole batch. It
# is the only read this module makes outside the dashboard, and it returns
# short hashes for the visitors active right now, so the batch bounds it.
known = set(db.scalars(
select(models.AnalyticsVisitorDay.visitor)
.where(models.AnalyticsVisitorDay.visitor.in_(ids))
.distinct()
))
rows = [
{
"day": day,
"visitor": visitor,
"is_new": visitor not in known,
**{column: column in flags for column in FUNNEL_FLAGS.values()},
}
for (day, visitor), flags in visits.items()
]
stmt = _insert(db)(table).values(rows)
db.execute(stmt.on_conflict_do_update(
index_elements=["day", "visitor"],
# Flags only turn on, and `is_new` is absent on purpose. The first
# write of a visitor's first day is what decided it.
set_={
column: or_(table.c[column], stmt.excluded[column])
for column in FUNNEL_FLAGS.values()
},
))
def purge_old_visitor_days(db: Session) -> int:
"""Deletes visitor-day rows past the retention horizon.
The cleanup sweeper calls this. The daily counters are never purged, because
they are aggregates, they are small, and this project keeps its history.
"""
if RETENTION_DAYS <= 0:
return 0
cutoff = (models.utcnow().date() - timedelta(days=RETENTION_DAYS)).isoformat()
removed = db.query(models.AnalyticsVisitorDay).filter(
models.AnalyticsVisitorDay.day < cutoff
).delete(synchronize_session=False)
db.commit()
return removed or 0
# ---------- Reading it back ----------
# Every query below is an aggregate. The database does the counting and returns
# tens of rows, however much traffic is behind them. No query here can return a
# row that belongs to one visitor.
TOP_N = 12
def _top(rows: list[dict], limit: int = TOP_N) -> list[dict]:
return rows[:limit]
def summary(db: Session, days: int = 30) -> dict:
"""Returns everything the dashboard shows for the last `days` days, including
today.
The function flushes first, so the numbers include the last minute.
"""
flush(db)
today = models.utcnow().date()
since = (today - timedelta(days=days - 1)).isoformat()
daily = models.AnalyticsDaily
visitor = models.AnalyticsVisitorDay
# 1. Every counter in the window, reduced to (metric, label) totals. The
# page, referrer, country, device, scenario, and error tables all come
# from this one pass rather than from a query each.
by_metric: dict[str, list[dict]] = {}
for metric, label, hits in db.execute(
select(daily.metric, daily.label, func.sum(daily.hits))
.where(daily.day >= since)
.group_by(daily.metric, daily.label)
):
by_metric.setdefault(metric, []).append({"label": label, "hits": int(hits)})
for rows in by_metric.values():
rows.sort(key=lambda row: -row["hits"])
events = {row["label"]: row["hits"] for row in by_metric.get(M_EVENT, [])}
# 2. The two per-day series the dashboard draws.
pageviews_by_day = {
day: int(hits)
for day, hits in db.execute(
select(daily.day, func.sum(daily.hits))
.where(daily.day >= since, daily.metric == M_PAGE)
.group_by(daily.day)
)
}
turns_by_day = {
day: int(hits)
for day, hits in db.execute(
select(daily.day, func.sum(daily.hits))
.where(daily.day >= since, daily.metric == M_EVENT, daily.label == EV_TURN)
.group_by(daily.day)
)
}
# 3. People, per day. There is one row per visitor per day, so COUNT(*) is
# already the day's unique visitors and no DISTINCT is needed.
visitors_by_day: dict[str, dict] = {}
for day, total, fresh in db.execute(
select(
visitor.day,
func.count(),
func.sum(case((visitor.is_new, 1), else_=0)),
)
.where(visitor.day >= since)
.group_by(visitor.day)
):
visitors_by_day[day] = {"visitors": int(total), "new": int(fresh or 0)}
# 4. The funnel over the whole window, counting each person once.
# COUNT(DISTINCT CASE WHEN flag THEN visitor END) ignores the NULLs the
# CASE leaves for everyone who did not reach that step.
unique, unique_new, *reached = db.execute(
select(
func.count(func.distinct(visitor.visitor)),
func.count(func.distinct(case((visitor.is_new, visitor.visitor)))),
*[
func.count(func.distinct(case((visitor.__table__.c[column], visitor.visitor))))
for column in FUNNEL_FLAGS.values()
],
).where(visitor.day >= since)
).one()
series = []
for offset in range(days):
day = (today - timedelta(days=days - 1 - offset)).isoformat()
counted = visitors_by_day.get(day, {})
series.append({
"day": day,
"visitors": counted.get("visitors", 0),
"new": counted.get("new", 0),
"pageviews": pageviews_by_day.get(day, 0),
"turns": turns_by_day.get(day, 0),
})
visits = sum(row["visitors"] for row in series)
pageviews = sum(pageviews_by_day.values())
turns = events.get(EV_TURN, 0)
errors = by_metric.get(M_ERROR, [])
return {
"days": days,
"since": since,
"until": today.isoformat(),
"generated_at": models.utcnow().isoformat(),
"totals": {
# `visitors` counts each person once for the window. `visits`
# counts them once per day they returned, which is the closest
# measure to "sessions" that does not track sessions.
"visitors": int(unique),
"new_visitors": int(unique_new),
"visits": visits,
"pageviews": pageviews,
"turns": turns,
"demo_turns": events.get(EV_DEMO_TURN, 0),
"adventures": events.get(EV_ADVENTURE, 0),
"signups": events.get(EV_SIGNUP, 0),
"logins": events.get(EV_LOGIN, 0),
"turn_errors": events.get(EV_TURN_ERROR, 0),
"errors": sum(row["hits"] for row in errors),
"turns_per_visit": round(turns / visits, 1) if visits else 0,
"pages_per_visit": round(pageviews / visits, 1) if visits else 0,
},
"series": series,
# Step 0 is everyone who arrived, so the drop-off between it and
# "Opened a scenario" appears as a step like any other.
"funnel": [{"step": "Visited", "count": int(unique)}] + [
{"step": step, "count": int(count)}
for step, count in zip(
["Opened a scenario", "Started an adventure", "Played a turn", "Signed up"],
reached,
)
],
"pages": _top(by_metric.get(M_PAGE, [])),
"referrers": _top(by_metric.get(M_REFERRER, [])),
"countries": _top(by_metric.get(M_COUNTRY, [])),
"devices": by_metric.get(M_DEVICE, []),
"scenarios": _top(by_metric.get(M_SCENARIO, [])),
"errors": _top(errors),
"events": by_metric.get(M_EVENT, []),
}
# ---------- Background flusher ----------
# This matches the start and stop pair in `cleanup`, so the lifespan in
# `main.py` reads the same way for both. The interval bounds how much a hard
# restart can lose.
async def _flush_loop() -> None:
import asyncio
from starlette.concurrency import run_in_threadpool
while True:
await asyncio.sleep(FLUSH_INTERVAL_SECONDS)
# This is blocking database work, so keep it off the event loop, which
# is also serving SSE turn streams.
await run_in_threadpool(flush)
def start_flusher():
import asyncio
return asyncio.create_task(_flush_loop())
async def stop_flusher(task) -> None:
"""Cancels the loop and writes out whatever it was holding.
A deploy is the one restart that is both frequent and predictable, so it
should not be what loses a minute of counts.
"""
import asyncio
from starlette.concurrency import run_in_threadpool
if task is not None:
task.cancel()
try:
await task
except asyncio.CancelledError:
pass
await run_in_threadpool(flush)
backend/app/attempts.py
+261
View File
@@ -0,0 +1,261 @@
"""Phase 14, SP4: the attempts at one turn.
A retry used to rewrite the AI action in place and append the discarded attempt
to a JSON list on the same row. Seven separate bugs came from that arrangement.
The row's `text` duplicated one entry of a repeating group, a second column
duplicated its length, and every reader that touched the story during a retry
had to be told to ignore the row.
Now an attempt is a node. A retry writes a sibling at the same `(branch_id,
depth)` and marks it live. The previous attempt stays as it was written, at the
same coordinate, with `live = False`. Nothing is duplicated, so nothing can
diverge.
Two invariants hold the arrangement together, and this module is the only place
that maintains either one:
* Exactly one sibling in a group is live. `lineage.Path.clause` selects on it, so
the other attempts are invisible to every read of the story, and none of those
reads has to know that attempts exist.
* The assembled prompt is stored once per turn, on the live sibling. A
`context_snapshot` is about 163 kB of prompt that every attempt at a turn
shares, plus a few hundred bytes that differ, listed in `ATTEMPT_KEYS`. Giving
each sibling its own copy would make a retry a permanent multiplier on the
largest column in the database, which is what the JSON list was invented to
avoid. The prompt therefore moves with the live flag, and a superseded sibling
keeps only its own slices.
Ordering inside a group comes from `id`, not from `created_at`. Two attempts made
in the same second still have to page in the order they were made, and `id`
increases with every insert. SP8 dropped `variant_index`, an explicit ordinal
that carried the same order, once a run of the suite confirmed that the two
agreed in every group.
"""
import copy
from sqlalchemy.orm import Session, undefer
from . import models
from .context import lineage
# The slices of a context snapshot that belong to one attempt rather than to the
# turn. They are the world-state delta the attempt proposed and what the engine
# did with it, the script report, the model's literal reply, and the endpoint's
# token accounting. Each attempt is its own API call, and a retry is the call
# most likely to read the prompt back out of cache. Everything else in a snapshot
# is the prompt, which is assembled once per turn.
ATTEMPT_KEYS = ("world_state", "script", "raw_output", "usage")
# ------------------------------------------------------------------ reading
def group(db: Session, action: models.Action) -> list[models.Action]:
"""Returns every attempt at `action`'s turn, oldest first.
The query keys on the parent rather than on the coordinate (SP9). The two
agree until an attempt is forked onto its own branch. That attempt keeps its
parent but leaves the `(branch, depth)` its siblings are still at, so a
coordinate would report it as the only attempt at its turn, showing `1/1`
where the player should see `1/3`.
The parent also nests groups correctly without extra work. Attempts under C1
and attempts under C2 share a depth, and until one of them forks they share a
branch. Only the parent separates them, which is what makes a pager under C2
read `2/2` rather than count C1's three as well.
There are two fallbacks, and both mean the row predates the key being asked
about. A node with no branch is a pre-tree row that no path contains, and a
node with no parent is a pre-SP9 row the backfill could not place. Under the
rule each was written with, both are the only attempt at their turn.
"""
if action.branch_id is None or action.depth is None:
return [action]
if action.parent_id is None:
# This row is pre-SP9, and the coordinate is the key those rows were
# written under. A root node also reaches this branch and is genuinely
# alone, because nothing is an attempt at the opening of a story.
return (
db.query(models.Action)
.filter(
models.Action.adventure_id == action.adventure_id,
models.Action.branch_id == action.branch_id,
models.Action.depth == action.depth,
models.Action.parent_id.is_(None),
)
.order_by(models.Action.id)
.all()
)
return (
db.query(models.Action)
.filter(
models.Action.adventure_id == action.adventure_id,
models.Action.parent_id == action.parent_id,
)
.order_by(models.Action.id)
.all()
)
def on_branch(rows: list[models.Action], node: models.Action) -> list[models.Action]:
"""Returns the attempts in `rows` that are on `node`'s own branch.
`group` reports which attempts belong to this turn, and since SP9 that spans
branches. An attempt forked onto its own line is still an attempt at the same
turn, which is the reason for keying on the parent.
Deletion is the one caller that must not follow a group across branches. An
attempt on another branch is reachable through that branch and belongs to the
story someone is telling there. Removing it because a turn was undone here
would delete a line nobody asked about. The same parent and the same branch
together are the coordinate, which is what every attempt at this turn meant
before a fork could move one out of it.
"""
return [row for row in rows if row.branch_id == node.branch_id]
def live_in(rows: list[models.Action]) -> models.Action | None:
for row in rows:
if row.live:
return row
return None
def preceding(
db: Session, adventure: models.Adventure, node: models.Action
) -> models.Action | None:
"""Returns the node the story tells immediately before `node`.
This reads "before this turn" as a fact about the path rather than as a
snapshot taken from inside the turn, which is what makes the after-snapshots
sufficient on their own. The query undefers both of them, because the only
reason to fetch this row is to restore what it left behind.
"""
if node.depth is None:
return None
return (
db.query(models.Action)
.filter(
models.Action.adventure_id == adventure.id,
lineage.path_of(db, adventure).clause(models.Action),
models.Action.depth < node.depth,
)
.options(
undefer(models.Action.state_after),
undefer(models.Action.world_state_after),
)
.order_by(models.Action.depth.desc(), models.Action.id.desc())
.first()
)
# ------------------------------------------------------------------ writing
def restore_state(adventure: models.Adventure, node: models.Action | None) -> None:
"""Restores the script state and world state that `node` left behind.
A NULL snapshot means leave the live state as it is, never reset it. Rows
written before SP4 that the migration could not derive an outcome for carry
NULLs, and overwriting a running adventure's state with an empty dict would
be worse than doing nothing.
"""
if node is None:
return
if isinstance(node.state_after, dict):
adventure.script_state = copy.deepcopy(node.state_after)
if isinstance(node.world_state_after, dict):
adventure.world_state = copy.deepcopy(node.world_state_after)
def snapshot_outcome(adventure: models.Adventure, node: models.Action) -> None:
"""Records on `node` the state of the adventure now that the node has played."""
state = adventure.script_state if isinstance(adventure.script_state, dict) else {}
world = adventure.world_state if isinstance(adventure.world_state, dict) else {}
node.state_after = copy.deepcopy(state)
node.world_state_after = copy.deepcopy(world)
def roll_back_before(
db: Session, adventure: models.Adventure, node: models.Action
) -> None:
"""Rewinds the shared state to what it was before `node` was played."""
restore_state(adventure, preceding(db, adventure, node))
def add_attempt(
db: Session,
adventure: models.Adventure,
previous: models.Action,
replacement: models.Action,
) -> None:
"""Places `replacement` next to `previous` as the newer attempt at that turn.
The placement is done here rather than through `tree.place_action`, which
moves the head. A sibling is not a new turn. It is another attempt at the
turn the head is already on.
"""
replacement.branch_id = previous.branch_id
replacement.depth = previous.depth
# Copy the parent rather than resolve it from the path. An attempt belongs
# to the turn it is an attempt at, and that is what `group` keys on.
# Resolving it here would ask which node is live one depth back. That is the
# same node right now, and it stops being the same node once the story forks
# away from this turn.
replacement.parent_id = previous.parent_id
replacement.live = True
# The replacement takes its place at the end of the group, because `group`
# orders by `id` and this row has no id yet. Switching a three-attempt turn
# back to attempt 1 and retrying therefore still pages 1, 2, 3, 4, which is
# the order the attempts were made in.
previous.live = False
# The replacement was assembled with a fresh snapshot, so the prompt for
# this turn is now the one it carries. The superseded attempt keeps only the
# slices that were its own.
keep_own_slices(previous)
def make_live(
db: Session, adventure: models.Adventure, node: models.Action
) -> list[models.Action]:
"""Makes `node` the attempt the story tells, and restores its outcome.
Returns the group, so that a caller reporting on it does not read it twice.
"""
rows = group(db, node)
previous = live_in(rows)
if previous is not None and previous is not node:
hand_over_the_prompt(previous, node)
for row in rows:
row.live = row is node
restore_state(adventure, node)
return rows
# ------------------------------------------------- the prompt, stored once
def keep_own_slices(node: models.Action) -> None:
"""Reduces `node`'s snapshot to the slices that are only its own."""
snapshot = node.context_snapshot
if not isinstance(snapshot, dict):
return
node.context_snapshot = {
key: snapshot[key] for key in ATTEMPT_KEYS if key in snapshot
} or None
def hand_over_the_prompt(giver: models.Action, taker: models.Action) -> None:
"""Moves the turn's assembled prompt from one attempt to another.
The caller runs this when the live flag moves, so that the row in the story
is always the row the Insights viewer can explain. Nothing is copied. The
prompt exists once before and once after, on whichever sibling is being read.
"""
held = giver.context_snapshot if isinstance(giver.context_snapshot, dict) else {}
shared = {k: v for k, v in held.items() if k not in ATTEMPT_KEYS}
if not shared:
return
keep_own_slices(giver)
own = taker.context_snapshot if isinstance(taker.context_snapshot, dict) else {}
taker.context_snapshot = shared | {
k: v for k, v in own.items() if k in ATTEMPT_KEYS
}
backend/app/auth.py
+263
View File
@@ -0,0 +1,263 @@
"""Phase 8: user resolution, sessions, and the shared demo key.
The `AIDND_MULTI_USER` environment variable selects one of two modes:
* Local mode, the default. Every request resolves to one automatically created
local user. There are no cookies and no login UI, so a clone or a
docker-compose run behaves like the single-user app from before Phase 8.
* Multi-user mode, used for hosted deployments. Requests carry a signed session
cookie. `GET /api/auth/me` creates a guest user on the first visit, and
registering upgrades that guest in place so their data survives. A request
without a valid session gets a 401, and the frontend re-establishes the
session through `/me`.
The shared demo key, which is the fallback when a user brings no key of their
own, is also configured here. A user whose settings hold no API key is routed to
a server-funded endpoint with a model allowlist and a per-day turn cap.
"""
import os
from dataclasses import dataclass
from datetime import timezone
from fastapi import Depends, HTTPException, Request
from sqlalchemy.orm import Session
from . import models, security
from .database import get_db
def _env_flag(name: str) -> bool:
return os.environ.get(name, "").strip().lower() in ("1", "true", "yes", "on")
MULTI_USER = _env_flag("AIDND_MULTI_USER")
SESSION_COOKIE = "aidnd_session"
# Secure cookies are on by default in multi-user mode, because a hosted
# deployment serves HTTPS and browsers also accept Secure on http://localhost.
# `AIDND_COOKIE_SECURE` overrides the default with 0 or 1. Use 0 when testing
# multi-user mode over plain HTTP on a LAN address.
_cookie_secure_env = os.environ.get("AIDND_COOKIE_SECURE", "").strip().lower()
COOKIE_SECURE = (
_cookie_secure_env in ("1", "true", "yes", "on")
if _cookie_secure_env
else MULTI_USER
)
COOKIE_MAX_AGE = 60 * 60 * 24 * 365
# ---------- Shared demo key (BYOK fallback) ----------
DEMO_API_KEY = os.environ.get("AIDND_DEMO_API_KEY", "").strip()
DEMO_ENDPOINT_URL = (
os.environ.get("AIDND_DEMO_ENDPOINT_URL", "").strip()
or "https://openrouter.ai/api/v1"
)
DEMO_MODELS = [
m.strip()
for m in os.environ.get("AIDND_DEMO_MODELS", "").split(",")
if m.strip()
] or ["google/gemma-4-26b-a4b-it:free"]
DEMO_TURNS_PER_DAY = int(os.environ.get("AIDND_DEMO_TURNS_PER_DAY", "20") or 20)
# Trusted testers, listed by email, who bypass the daily demo cap and take
# unmetered turns on the shared demo key. The list is comma-separated, and the
# match ignores case.
POWER_USERS = {
e.strip().lower()
for e in os.environ.get("AIDND_POWER_USERS", "").split(",")
if e.strip()
}
# Who can see the visit analytics. This is a separate list from `POWER_USERS` on
# purpose. A trusted tester gets unmetered turns and the AI Chat page, which is
# not a reason to give them the site's traffic numbers. An empty list, which is
# the default, means nobody sees the dashboard in a hosted deployment.
ANALYTICS_EMAILS = {
e.strip().lower()
for e in os.environ.get("AIDND_ANALYTICS_EMAILS", "").split(",")
if e.strip()
}
DEMO_CAP_MESSAGE = (
f"You've used all {DEMO_TURNS_PER_DAY} free demo turns for today. "
"Add your own API key in Settings to keep playing (it resets tomorrow)."
)
def demo_enabled() -> bool:
# The demo key is a hosted-deployment feature. A local install talks to
# whatever endpoint Settings points at, even with no API key, such as
# Ollama.
return MULTI_USER and bool(DEMO_API_KEY)
@dataclass
class ProviderConfig:
"""What the turn engine connects with, after the decision between a
user-supplied key and the demo key.
Build one of these with `resolve_provider_config()`.
"""
endpoint_url: str
api_key: str
model: str
using_demo: bool
def __post_init__(self) -> None:
# A second guard around server-funded turns. `resolve_provider_config()`
# already pins the model, and this makes the pin a property of the config
# object too, so a later caller cannot construct an unpinned one. This
# raise is unreachable by design. Reaching it means a new code path
# bypassed the pinning, which is worth failing on rather than billing
# for.
#
# The test is `using_demo`, not `api_key == DEMO_API_KEY`. Keying on the
# key value looks stricter and is wrong. The demo key is an ordinary
# OpenRouter key, so a user can legitimately paste that same key into
# their own Settings. Every resolution then raised, which returned a 500
# even from `GET /auth/me` and took the whole SPA down. `using_demo` is
# what means the server is paying, and only the demo branch below sets
# it.
if self.using_demo and self.model not in DEMO_MODELS:
raise ValueError(
f"Refusing to use the shared demo key with non-whitelisted model {self.model!r}"
)
def resolve_provider_config(
settings: models.Settings, *, model_override: str | None = None
) -> ProviderConfig:
"""Returns the user's own key when they have one, and the shared demo key
otherwise.
The demo branch is the security-relevant one, and it is the only place the
allowlist rule lives. Every caller has to come through this function rather
than build a `ProviderConfig` itself. On the demo key:
* The model is pinned to `DEMO_MODELS`, so a caller-supplied override from
the AI Chat page, or a hand-edited Settings row, cannot point a
server-funded key at a paid model. An unrecognized model falls back to
`DEMO_MODELS[0]`.
* The endpoint is pinned to `DEMO_ENDPOINT_URL`, so the key cannot be
redirected to a URL the user controls and captured there.
`model_override` is a per-request preference and never a grant. It is used
verbatim with the user's own key, and on the demo key only when the model is
on the allowlist.
"""
key = settings.api_key_plain
requested = (model_override or "").strip() or settings.model
if key or not demo_enabled():
return ProviderConfig(settings.endpoint_url, key, requested, False)
model = requested if requested in DEMO_MODELS else DEMO_MODELS[0]
return ProviderConfig(DEMO_ENDPOINT_URL, DEMO_API_KEY, model, True)
def _today() -> str:
return models.utcnow().date().isoformat()
def is_power_user(user: models.User) -> bool:
"""Returns whether this user is a trusted tester.
A trusted tester gets unmetered demo turns, plus tooling that is not part of
the game, such as the AI Chat scratchpad. A local install is always trusted,
because it runs on the operator's own machine with their own API key. The
provider debug log is local-only for the same reason.
"""
if not MULTI_USER:
return True
return bool(user.email) and user.email.lower() in POWER_USERS
def is_owner(user: models.User) -> bool:
"""Returns whether this user may see the visit analytics.
A local install always may, because it runs on the operator's own machine and
shows their own visits. The provider debug log follows the same reasoning. A
hosted deployment checks `AIDND_ANALYTICS_EMAILS`.
"""
if not MULTI_USER:
return True
return bool(user.email) and user.email.lower() in ANALYTICS_EMAILS
def demo_turns_left(user: models.User) -> int:
# A power user is never capped, so report the full cap and let the banner
# read "N of N" rather than count down.
if is_power_user(user):
return DEMO_TURNS_PER_DAY
used = user.demo_turns_used if user.demo_turns_date == _today() else 0
return max(0, DEMO_TURNS_PER_DAY - used)
def count_demo_turn(user: models.User) -> None:
"""Records one demo turn. The caller's commit stores it."""
if is_power_user(user):
return # A power user's turns do not count against the cap.
today = _today()
if user.demo_turns_date != today:
user.demo_turns_date = today
user.demo_turns_used = 0
user.demo_turns_used += 1
# ---------- User resolution ----------
def local_user(db: Session) -> models.User:
"""Returns the single implicit user used in local mode.
A migration gives this user ownership of data written before Phase 8. On a
fresh database the user is created on first use.
"""
user = (
db.query(models.User)
.filter(models.User.email.is_(None), models.User.is_guest.is_(False))
.order_by(models.User.id)
.first()
)
if user is None:
user = models.User(is_guest=False)
db.add(user)
db.commit()
return user
def _touch(user: models.User, db: Session) -> None:
now = models.utcnow()
last = user.last_seen_at
if last is not None and last.tzinfo is None:
# SQLite returns DateTime columns without a timezone. They were stored
# as UTC.
last = last.replace(tzinfo=timezone.utc)
if last is None or (now - last).total_seconds() > 3600:
user.last_seen_at = now
db.commit()
def resolve_session_user(request: Request, db: Session) -> models.User | None:
token = request.cookies.get(SESSION_COOKIE)
if not token:
return None
user_id = security.verify_session(token)
if user_id is None:
return None
return db.get(models.User, user_id)
def get_current_user(request: Request, db: Session = Depends(get_db)) -> models.User:
"""The dependency every router uses to resolve the current user.
In multi-user mode a 401 means the frontend has to establish a session again
through `GET /api/auth/me`.
"""
if not MULTI_USER:
user = local_user(db)
else:
user = resolve_session_user(request, db)
if user is None:
raise HTTPException(401, "No session. Call GET /api/auth/me first.")
_touch(user, db)
return user
backend/app/bundle.py
+735
View File
@@ -0,0 +1,735 @@
"""Phase 14, SP6: the export bundle, as a tree.
A bundle is the one place a story leaves the database, and the only part of the
tree no migration can reach. A file downloaded today has to still import into a
build shipped next year. The format is therefore versioned, both versions are
defined here, and nothing else in the app knows either of them.
Version 1 is a flat list of turns, each with an optional `variants` array, which
is the repeating group SP4 unpacked into rows. Nothing writes that shape now.
The reader stays, because bundles already on people's disks still use it, and a
backup that stops importing is not a backup.
Version 2 carries the tree. It holds three things version 1 could not, and each
one is required:
* The branches, because a forked adventure is two stories and a flat list holds
one. A version 1 export interleaved them by turn number, which read as a
garbled story rather than as lost data.
* `live`, because a coordinate can hold several attempts at one turn and exactly
one of them is the story.
* Both after-snapshots, because they are what a branch switch and an undo
restore. A bundle carrying the actions but not the outcomes would import a
tree nobody could switch inside.
## The rule about what a bundle carries
A bundle carries what was chosen, never what is derived. The head branch, the
fork points, the live flags, and the anchors are decisions somebody made, so
they are in the file. `lineage` and the head depth are computed from those, and
the import recomputes them:
* `lineage` is a cache of `parent` plus `fork_depth`. Shipping it as well would
put a second source of truth for one fact into a file anyone can hand-edit,
and the two could then disagree without any read reporting it.
* The head depth is the tip of the head branch, which is a fact about the nodes
that arrived with it.
Every hand-editable coordinate is therefore checked before a row is written, in
`plan`, rather than repaired afterwards. An import that fails partway leaves an
adventure holding half a tree, and a tree missing a branch is a story that stops
without reporting anything.
"""
from datetime import datetime
from fastapi import HTTPException
from sqlalchemy import insert, update
from sqlalchemy.orm import Session, undefer
from . import attempts, models, schemas
from .context import cursors, lineage
FORMAT = "ai-dnd-adventure-v2"
LEGACY_FORMAT = "ai-dnd-adventure-v1"
# `actions.type` is VARCHAR(20), and a raw-dict import bypasses the schemas.
TYPE_MAX = 20
# ---------------------------------------------------------------- exporting
def export(db: Session, adventure: models.Adventure) -> dict:
"""Returns the whole adventure as a version 2 bundle.
The export is not scoped to a path. A backup holds the entire tree, not the
branch its owner is currently reading. Both after-snapshots are undeferred in
the one query, because they are per-node columns nothing else reads in bulk
and requesting them a row at a time would cost one query per turn.
The bundle carries no context snapshots. It has never carried the assembled
prompts, and it still does not. They run about 163 kB per turn, they explain
a generation rather than form part of the story, and the Insights viewer they
feed reads the adventure they came from.
"""
branches = (
db.query(models.Branch)
.filter(models.Branch.adventure_id == adventure.id)
.order_by(models.Branch.id)
.all()
)
# Branch ids are local to the file and are positions in this list, because
# the database ids they hold here are already in use on the importing
# side.
local = {branch.id: i for i, branch in enumerate(branches)}
nodes = (
db.query(models.Action)
.filter(models.Action.adventure_id == adventure.id)
.options(
undefer(models.Action.state_after),
undefer(models.Action.world_state_after),
)
.order_by(
models.Action.branch_id, models.Action.depth, models.Action.id,
)
.all()
)
return {
"format": FORMAT,
"title": adventure.title,
"memory": adventure.memory,
"authorsNote": adventure.authors_note,
"aiInstructions": adventure.ai_instructions,
"storySummary": adventure.story_summary,
# Phase 18. A bundle written before personas existed has no key here,
# and the import below reads it with `.get`, so it lands with an empty
# persona — which is the same as having none. No FORMAT bump needed.
"persona": {
"name": adventure.persona_name,
"pronouns": adventure.persona_pronouns,
"desc": adventure.persona_desc,
},
"scriptState": adventure.script_state,
"worldState": adventure.world_state,
"autoSummarize": adventure.auto_summarize,
"memoryBankEnabled": adventure.memory_bank_enabled,
# Write a root entry even for an adventure whose branch row was never
# created. A story with no branch is a pre-tree story, and the tree it
# belongs to is the root. `_local` places its nodes there.
"branches": [_exported_branch(b, local) for b in branches] or [_ROOT],
"headBranch": local.get(adventure.head_branch_id, 0),
"memoryCursor": _exported_anchor(adventure, cursors.MEMORY, local),
"summaryCursor": _exported_anchor(adventure, cursors.SUMMARY, local),
"memories": [_exported_memory(m, local) for m in adventure.memories],
"storyCards": [
{"type": c.type, "name": c.name, "keys": c.keys,
"entry": c.entry, "notes": c.notes}
for c in adventure.story_cards
],
"scripts": [
{
"position": s.position, "enabled": s.enabled,
"name": s.name, "description": s.description,
"library": s.library_js, "input": s.input_js,
"context": s.context_js, "output": s.output_js,
}
for s in adventure.scripts
],
"actions": [_exported_node(a, local) for a in nodes],
}
_ROOT = {"parent": None, "forkDepth": None}
def _local(branch_id: int | None, local: dict[int, int]) -> int:
return local.get(branch_id, 0) if branch_id is not None else 0
def _exported_branch(branch: models.Branch, local: dict[int, int]) -> dict:
parent = (
local.get(branch.parent_branch_id)
if branch.parent_branch_id is not None else None
)
out = (
dict(_ROOT) if parent is None
else {"parent": parent, "forkDepth": branch.fork_depth}
)
# A player chose the name, so it goes into the file. That is the same rule
# that puts the fork points in the file and leaves `lineage` out. An unnamed
# branch omits the key rather than carry a null, which keeps the file for an
# unnamed tree byte-identical to the one SP6 wrote.
if branch.name:
out["name"] = branch.name
return out
def _exported_node(action: models.Action, local: dict[int, int]) -> dict:
node = {
"branch": _local(action.branch_id, local),
"depth": action.depth,
"live": bool(action.live),
"type": action.type,
"text": action.text,
"createdAt": action.created_at.isoformat() if action.created_at else None,
}
if action.reasoning:
node["reasoning"] = action.reasoning
# `{}` and an absent key mean different things. `{}` means the node left an
# empty state behind, and an absent key means the state is unknown and the
# live state stays as it is. An empty snapshot is therefore written rather
# than omitted. It costs about eighteen bytes per row, and it decides
# whether an undo clears a score or leaves it in place.
if action.state_after is not None:
node["stateAfter"] = action.state_after
if action.world_state_after is not None:
node["worldStateAfter"] = action.world_state_after
if action.world_delta:
node["worldDelta"] = action.world_delta
return node
def _exported_memory(memory: models.Memory, local: dict[int, int]) -> dict:
return {
"text": memory.text, "pinned": memory.pinned, "forgotten": memory.forgotten,
"sourceStart": memory.source_start, "sourceEnd": memory.source_end,
"useCount": memory.use_count,
# The node this memory is attached to. A hand-written memory summarizes
# no node, so it has a branch and no depth, and it keeps that shape
# here.
"branch": _local(memory.branch_id, local),
"depth": memory.depth,
}
def _imported_persona(persona) -> dict:
"""Reads a bundle's `persona` block into `Adventure` keyword arguments.
A raw-dict import bypasses the schemas, so the strings are truncated to the
widths the columns declare, in the same way the rest of `_import` does. A
bundle written before Phase 18 has no block at all, and an empty persona is
the same as having none.
"""
if not isinstance(persona, dict):
return {}
return {
"persona_name": str(persona.get("name") or "")[:schemas.PERSONA_NAME_MAX],
"persona_pronouns":
str(persona.get("pronouns") or "")[:schemas.PERSONA_PRONOUNS_MAX],
"persona_desc": str(persona.get("desc") or ""),
}
def _exported_anchor(
adventure: models.Adventure, cursor: cursors.Cursor, local: dict[int, int]
) -> dict:
branch_id, depth = cursor.stored(adventure)
return {
"branch": local.get(branch_id) if branch_id is not None else None,
"depth": depth,
}
# ---------------------------------------------------------------- importing
def check_format(bundle: dict) -> str:
"""Returns the bundle's version, or raises a 400."""
fmt = bundle.get("format")
if fmt in (FORMAT, LEGACY_FORMAT):
return fmt
raise HTTPException(
400,
f"Not an adventure export file (expected format {FORMAT} or {LEGACY_FORMAT}).",
)
def plan(bundle: dict, version: str) -> dict:
"""Returns the bundle's tree, checked and normalized, before a row is written.
The function has no side effects. It opens no session, needs no adventure,
and creates nothing. It catches everything a hand-edited file can get wrong
about the shape of a tree, because the alternative is an import that fails
partway and leaves an adventure holding a story with a gap in it.
Both versions produce the same shape, so `write` never learns that there are
two formats. A version 1 bundle is a linear story, which is a tree with one
branch, and its `variants` array is a sibling group written the old way.
"""
branches = (
_planned_branches(bundle) if version == FORMAT else [dict(_ROOT)]
)
nodes = (
_planned_nodes(bundle, len(branches)) if version == FORMAT
else _planned_v1_nodes(bundle)
)
return {
"branches": branches,
"nodes": nodes,
"memories": _planned_memories(bundle, len(branches)),
"head": _as_index(bundle.get("headBranch"), len(branches), default=0),
# Version 2 records where the derived work reached. Version 1 counted
# it, and a count cannot become a node until the nodes exist. See
# `settle`.
"anchors": _planned_anchors(bundle, len(branches)) if version == FORMAT else None,
"positions": None if version == FORMAT else {
"memory": _as_int(bundle.get("memoryCursor"), 0),
"summary": _as_int(bundle.get("summaryCursor"), 0),
},
}
def _planned_branches(bundle: dict) -> list[dict]:
raw = bundle.get("branches")
entries = [b for b in raw if isinstance(b, dict)] if isinstance(raw, list) else []
if not entries:
return [dict(_ROOT)]
specs: list[dict] = []
for i, entry in enumerate(entries):
parent = entry.get("parent")
name = _planned_branch_name(entry, i)
if parent is None:
specs.append(dict(_ROOT, **({"name": name} if name else {})))
continue
# A branch may fork only from a branch listed before it. The export
# writes them that way, because branches are numbered in creation order
# and a parent always exists first. Requiring it here guarantees the
# graph is acyclic for the cost of one comparison. A lineage is computed
# by walking to the parent, so a cycle would be an import that never
# returns.
if not _is_int(parent) or not 0 <= parent < i:
raise HTTPException(
400,
f"Branch {i} forks from branch {parent!r}, which is not one of "
f"the {i} branches listed before it.",
)
fork_depth = entry.get("forkDepth")
if not _is_int(fork_depth):
raise HTTPException(
400,
f"Branch {i} forks from branch {parent} but does not say at "
f"what depth.",
)
specs.append({
"parent": parent, "forkDepth": fork_depth,
**({"name": name} if name else {}),
})
return specs
def _planned_branch_name(entry: dict, i: int) -> str | None:
"""Returns the name a branch entry carries, or `None` if nobody named it.
The check runs before the row is created rather than being left to the
column, for the reason the planner exists. A 400 from a function with no side
effects is better than a half-written adventure and a database error three
branches in.
"""
raw = entry.get("name")
if raw is None:
return None
if not isinstance(raw, str):
raise HTTPException(400, f"Branch {i} has a name that is not text.")
name = raw.strip()
if len(name) > schemas.BRANCH_NAME_MAX:
raise HTTPException(
400,
f"Branch {i}'s name is longer than {schemas.BRANCH_NAME_MAX} "
f"characters.",
)
return name or None
def _planned_nodes(bundle: dict, branches: int) -> list[dict]:
raw = bundle.get("actions")
nodes: list[dict] = []
for entry in raw if isinstance(raw, list) else []:
if not isinstance(entry, dict) or not str(entry.get("text") or ""):
continue
branch = entry.get("branch", 0)
if not _is_int(branch) or not 0 <= branch < branches:
raise HTTPException(
400,
f"An action names branch {branch!r}, but the file lists "
f"{branches}.",
)
depth = entry.get("depth")
if not _is_int(depth) or depth < 0:
raise HTTPException(
400, f"An action on branch {branch} has no depth to sit at."
)
nodes.append({
"branch": branch,
"depth": depth,
"live": bool(entry.get("live", True)),
"type": str(entry.get("type") or "story")[:TYPE_MAX],
"text": str(entry.get("text") or ""),
"reasoning": _as_text(entry.get("reasoning")),
"stateAfter": _as_dict(entry.get("stateAfter")),
"worldStateAfter": _as_dict(entry.get("worldStateAfter")),
"worldDelta": _as_dict(entry.get("worldDelta")),
"createdAt": _as_time(entry.get("createdAt")),
})
return nodes
def _planned_v1_nodes(bundle: dict) -> list[dict]:
"""Returns a version 1 bundle's turns as nodes, one node per attempt.
The `variants` array is the repeating group SP4 unpacked, so reading one
performs the same split that migration 60 does. Every attempt becomes a row
at the turn's coordinate, and `variantIndex` selects the live one. The index
is clamped, because a hand-edited bundle can name an attempt its own list
does not contain, and a turn with no live node is a turn no read can see.
The depth is the bundle's `index`. A version 1 bundle has one branch, where
the two numbers agree.
"""
raw = bundle.get("actions")
nodes: list[dict] = []
for i, entry in enumerate(raw if isinstance(raw, list) else []):
if not isinstance(entry, dict) or not str(entry.get("text") or ""):
continue
depth = _as_int(entry.get("index"), i)
kind = str(entry.get("type") or "story")[:TYPE_MAX]
variants = [v for v in (entry.get("variants") or []) if isinstance(v, dict)]
if not variants:
variants = [{"text": entry["text"], "reasoning": entry.get("reasoning")}]
live = min(max(_as_int(entry.get("variantIndex"), 0), 0), len(variants) - 1)
for n, variant in enumerate(variants):
text = str(variant.get("text") or "")
if not text:
continue
nodes.append({
"branch": 0,
"depth": max(depth, 0),
"live": n == live,
"type": kind,
"text": text,
"reasoning": _as_text(variant.get("reasoning")),
"stateAfter": None,
"worldStateAfter": None,
"worldDelta": None,
"createdAt": _as_time(variant.get("createdAt")),
})
return nodes
def _planned_memories(bundle: dict, branches: int) -> list[dict]:
raw = bundle.get("memories")
out: list[dict] = []
for entry in raw if isinstance(raw, list) else []:
if not isinstance(entry, dict) or not str(entry.get("text") or "").strip():
continue
out.append({
"text": str(entry["text"]),
"pinned": bool(entry.get("pinned", False)),
"forgotten": bool(entry.get("forgotten", False)),
"sourceStart": entry.get("sourceStart"),
"sourceEnd": entry.get("sourceEnd"),
"useCount": _as_int(entry.get("useCount"), 0),
# A value that is out of range, rather than absent, means the file
# disagrees with itself. The root is the safe reading, because a
# memory on a branch nothing can see never reaches a prompt
# again.
"branch": _as_index(entry.get("branch"), branches, default=0),
"depth": entry.get("depth") if _is_int(entry.get("depth")) else None,
})
return out
def _planned_anchors(bundle: dict, branches: int) -> dict:
anchors = {}
for name in ("memory", "summary"):
raw = bundle.get(f"{name}Cursor")
raw = raw if isinstance(raw, dict) else {}
branch = raw.get("branch")
anchors[name] = (
_as_index(branch, branches) if branch is not None else None,
_as_int(raw.get("depth"), lineage.NO_DEPTH),
)
return anchors
# ------------------------------------------------------------------ writing
def write(db: Session, adventure: models.Adventure, story: dict) -> None:
"""Writes a planned tree onto a newly created adventure.
The order is fixed. Branches come first, because a node needs a branch id.
The nodes come next, because the head and the anchors name a node.
"""
ids = _write_branches(db, adventure, story["branches"])
_write_nodes(db, adventure, story["nodes"], ids)
_write_memories(db, adventure, story["memories"], ids)
_point_the_head(adventure, story, ids)
_write_anchors(adventure, story, ids)
def _write_branches(
db: Session, adventure: models.Adventure, specs: list[dict]
) -> list[int]:
"""Writes one row per branch, computing the lineage rather than reading it.
The rows are inserted through Core and the lineage is written second, for the
reason `tree.root_branch` gives: the lineage names the row's own id. The
parent's cached ancestry is capped at this fork, which is the arithmetic
`tree.fork` performs. A fork made now and a fork made a year ago and then
exported have to produce the same rows.
"""
ids: list[int] = []
lineages: list[list[list]] = []
for spec in specs:
parent = spec.get("parent")
fork_depth = spec.get("forkDepth")
new_id = db.execute(
insert(models.Branch).values(
adventure_id=adventure.id,
parent_branch_id=ids[parent] if parent is not None else None,
fork_depth=fork_depth if parent is not None else None,
lineage=[],
name=spec.get("name"),
created_at=models.utcnow(),
)
).inserted_primary_key[0]
entries = [[new_id, None]]
if parent is not None:
entries += [
[branch_id, fork_depth if cap is None else min(cap, fork_depth)]
for branch_id, cap in lineages[parent]
]
db.execute(
update(models.Branch)
.where(models.Branch.id == new_id)
.values(lineage=entries)
)
ids.append(new_id)
lineages.append(entries)
return ids
def _write_nodes(
db: Session, adventure: models.Adventure, specs: list[dict], ids: list[int]
) -> None:
"""Writes the nodes, grouped into the turns they are attempts at.
One value is decided here rather than read from the file. Exactly one
attempt in each group is made live, because a file can name none or several,
and a turn with no live node disappears from the story.
"""
groups: dict[tuple[int, int], list[models.Action]] = {}
for spec in specs:
key = (spec["branch"], spec["depth"])
action = models.Action(
adventure_id=adventure.id,
branch_id=ids[spec["branch"]],
depth=spec["depth"],
type=spec["type"],
text=spec["text"],
reasoning=spec["reasoning"],
live=spec["live"],
state_after=spec["stateAfter"],
world_state_after=spec["worldStateAfter"],
world_delta=spec["worldDelta"],
)
if spec["createdAt"] is not None:
action.created_at = spec["createdAt"]
db.add(action)
groups.setdefault(key, []).append(action)
for rows in groups.values():
live = next((row for row in rows if row.live), rows[0])
for row in rows:
row.live = row is live
def _write_memories(
db: Session, adventure: models.Adventure, specs: list[dict], ids: list[int]
) -> None:
for spec in specs:
memory = models.Memory(
adventure_id=adventure.id,
text=spec["text"],
pinned=spec["pinned"],
forgotten=spec["forgotten"],
source_start=spec["sourceStart"],
source_end=spec["sourceEnd"],
use_count=spec["useCount"],
branch_id=ids[spec["branch"]],
depth=spec["depth"],
)
# A version 1 memory has no depth of its own. `source_end` is the index
# of the last action it summarizes, which on one branch is that node's
# depth.
if memory.depth is None and memory.source_end is not None:
memory.depth = memory.source_end
# A version 1 memory that summarizes nothing, which means one the player
# typed, has no depth to derive. Leaving it NULL here would recreate on
# import the state migration 62 exists to end. `Path._entry_clause`
# compares `depth <= max_depth`, which a NULL fails, so the memory would
# disappear from every branch as soon as the imported adventure was
# forked. The root is the answer the migration gives, for the same
# reason: 0 is at or before every fork point, so the memory is visible
# from every path this adventure can grow.
if memory.depth is None:
memory.depth = lineage.ROOT_DEPTH
db.add(memory)
def _point_the_head(
adventure: models.Adventure, story: dict, ids: list[int]
) -> None:
"""Sets which branch the story is played on, and how deep it goes.
The branch comes from the file and the depth does not. The tip of a branch is
whatever arrived on it, and a branch with no nodes of its own sits at its
fork point, which is the last node its story contains. That node is borrowed
but it is still the tip. This is the rule `tree.refresh_head` applies, run
here before a flush.
"""
head = story["head"]
adventure.head_branch_id = ids[head]
depths = [n["depth"] for n in story["nodes"] if n["branch"] == head]
if depths:
adventure.head_depth = max(depths)
return
fork_depth = story["branches"][head].get("forkDepth")
adventure.head_depth = fork_depth if fork_depth is not None else lineage.NO_DEPTH
def _write_anchors(
adventure: models.Adventure, story: dict, ids: list[int]
) -> None:
"""Records how far the memories and the summary have read. Version 2 only.
A version 1 bundle stores a count instead, and a count cannot be resolved to
a node until the nodes are in the database. `settle` does that part
afterwards.
"""
anchors = story["anchors"]
if anchors is None:
return
for cursor in cursors.ALL:
branch, depth = anchors[cursor.name]
cursor.anchor(adventure, ids[branch] if branch is not None else None, depth)
def settle(adventure: models.Adventure, story: dict) -> None:
"""Resolves a version 1 bundle's counts into anchors, once the nodes exist.
A version 1 file records how far the memories and the summary have read as
a count, so the anchor is found by counting that far along the story. A
version 2 file carries the anchor itself, which `_write_anchors` has
already stored, so there is nothing left to do.
The caller runs this after the flush, because counting needs the actions to
be queryable.
"""
positions = story["positions"]
if positions is None:
return
for cursor in cursors.ALL:
cursors.anchor_at_position(adventure, cursor, positions[cursor.name])
def materialize(
db: Session, payload: dict, story: dict, user_id: int | None
) -> models.Adventure:
"""Writes a planned bundle into a new adventure owned by `user_id`.
Call `check_format` and `plan` first, and apply any rate or size limits
before calling this. The split exists because those checks belong to the
import endpoint alone: the guest starter writes a file the server ships, so
it has nothing to rate-limit and no untrusted list to cap.
The caller commits. Two flushes happen here, because the anchors and the
legacy counts describe one boundary in two coordinate systems, and aligning
them needs the actions to be queryable.
"""
# A raw-dict import bypasses the schemas, so truncate strings bound for
# VARCHAR columns. Postgres enforces the widths. See `schemas.py`.
adventure = models.Adventure(
user_id=user_id,
title=str(payload.get("title") or "Imported Adventure")[:schemas.NAME_MAX],
memory=str(payload.get("memory") or ""),
authors_note=str(payload.get("authorsNote") or ""),
ai_instructions=str(payload.get("aiInstructions") or ""),
story_summary=str(payload.get("storySummary") or ""),
script_state=payload.get("scriptState") or {},
world_state=payload.get("worldState") or {},
auto_summarize=bool(payload.get("autoSummarize", False)),
memory_bank_enabled=bool(payload.get("memoryBankEnabled", False)),
**_imported_persona(payload.get("persona")),
)
db.add(adventure)
db.flush()
for card in payload.get("storyCards") or []:
if isinstance(card, dict):
db.add(models.StoryCard(
adventure_id=adventure.id,
type=str(card.get("type") or "")[:schemas.CARD_TYPE_MAX],
name=str(card.get("name") or "")[:schemas.NAME_MAX],
keys=str(card.get("keys") or ""),
entry=str(card.get("entry") or ""),
notes=str(card.get("notes") or ""),
))
for i, item in enumerate(payload.get("scripts") or []):
if isinstance(item, dict):
db.add(models.AdventureScript(
adventure_id=adventure.id,
position=int(item.get("position", i)),
enabled=bool(item.get("enabled", True)),
name=str(item.get("name") or "Imported Script")[:schemas.NAME_MAX],
description=str(item.get("description") or ""),
library_js=str(item.get("library") or ""),
input_js=str(item.get("input") or ""),
context_js=str(item.get("context") or ""),
output_js=str(item.get("output") or ""),
))
write(db, adventure, story)
db.flush()
db.expire(adventure, ["actions"])
settle(adventure, story)
return adventure
# ------------------------------------------------------------------ reading
# Small coercions. A raw-dict import bypasses the schemas, so every value from a
# bundle is whatever the JSON held.
def _is_int(value) -> bool:
"""Returns whether `value` is an integer. Python counts `True` as an `int`,
and a coordinate does not."""
return isinstance(value, int) and not isinstance(value, bool)
def _as_int(value, default: int) -> int:
return value if _is_int(value) else default
def _as_index(value, count: int, default: int | None = None) -> int | None:
"""Returns a local branch number, or `default` when the file names a branch
that is not present.
An out-of-range value means the file disagrees with itself, and it is not a
value any read can be given.
"""
return value if _is_int(value) and 0 <= value < count else default
def _as_text(value) -> str | None:
return str(value) if value else None
def _as_dict(value) -> dict | None:
return value if isinstance(value, dict) else None
def _as_time(value) -> datetime | None:
if not isinstance(value, str):
return None
try:
return datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return None
backend/app/cleanup.py
+178
View File
@@ -0,0 +1,178 @@
"""Retention policy for throwaway guest accounts.
In multi-user mode every first visit creates a `users` row through
`GET /api/auth/me`, so a public demo accumulates one account per visitor. Most
of those visitors never return, and each one leaves behind whatever scenarios,
adventures, actions, and memories they generated. This module deletes guests
that have been inactive for `AIDND_GUEST_RETENTION_DAYS`, which defaults to 5,
along with everything they made.
Why this is safe to run unattended:
- Only rows with `is_guest` AND `email IS NULL` are ever touched, and both
clauses are checked rather than either alone. Registering upgrades the row
in place (is_guest -> False), so a guest who signs up keeps everything;
local mode's implicit single user is also is_guest=False.
- Idle time is `COALESCE(last_seen_at, created_at)`. `auth._touch` writes
`last_seen_at` at most once an hour, and a guest created by `/auth/me` has
NULL there until its second request, so `created_at` is the correct floor for
a new visitor. Without the coalesce, those rows look arbitrarily old.
- Nothing a guest owns is reachable by anyone else. `is_public` is an
output-only field, as `schemas.ScenarioBase` shows, so the only shared
scenarios are the seeded ones, which have a NULL `user_id` and are outside
this filter. Deleting a guest cannot remove content from another user.
The sweep uses one Core DELETE rather than an ORM cascade. `db.delete(user)`
would SELECT every adventure, action, memory, and story card into Python only to
delete them, which on Neon is the egress pattern that has already cost this
project once. Every foreign key from `users` downward is ON DELETE CASCADE, from
users to scenarios, adventures, scripts, and settings, and from those to actions,
memories, and cards, so the database deletes the whole graph in one statement and
returns a row count.
The scan gets no index. The sweep runs a few times a day against a table holding
at most a few thousand rows, which does not justify a migration and the schema
surface it adds.
"""
import asyncio
import logging
import os
from datetime import datetime, timedelta
from sqlalchemy import delete, func
from sqlalchemy.orm import Session
from starlette.concurrency import run_in_threadpool
from . import analytics, auth, models
from .database import SessionLocal
logger = logging.getLogger(__name__)
def _int_env(name: str, default: int) -> int:
try:
return int(os.environ.get(name, "").strip() or default)
except ValueError:
logger.warning("%s is not an integer; using %d.", name, default)
return default
# Days of inactivity before a guest account is deleted. A value of 0 or less
# disables the policy, for a deployment that keeps everything.
RETENTION_DAYS = _int_env("AIDND_GUEST_RETENTION_DAYS", 5)
# How often a long-lived process re-checks. Hours, not minutes: nothing here is
# time-critical, and on Render's free tier the service sleeps and cold-starts
# often enough that the startup sweep does most of the work by itself.
SWEEP_INTERVAL_SECONDS = _int_env("AIDND_CLEANUP_INTERVAL_HOURS", 6) * 3600
def enabled() -> bool:
"""Guests only exist in multi-user mode, so local runs skip the sweep
rather than pointing a DELETE at a database that has nothing to collect."""
return auth.MULTI_USER and RETENTION_DAYS > 0
def anything_to_sweep() -> bool:
"""Whether the periodic task is worth starting at all. The two jobs it runs
are independent: a deployment can keep every guest forever and still want
its analytics rows aged out, and vice versa."""
return enabled() or analytics.RETENTION_DAYS > 0
def delete_stale_guests(db: Session, *, now: datetime | None = None) -> int:
"""Delete guests idle for RETENTION_DAYS or more. Returns the row count.
The caller owns error handling; `sweep` is the safe wrapper.
"""
if RETENTION_DAYS <= 0:
return 0
# Stored timestamps are UTC without a timezone on both backends. SQLite
# drops the timezone, and the Postgres columns are TIMESTAMP WITHOUT TIME
# ZONE with the session pinned to UTC in `database.py`. Match that, so the
# comparison does not depend on how a dialect renders a value that carries a
# timezone.
reference = now or models.utcnow()
cutoff = reference.replace(tzinfo=None) - timedelta(days=RETENTION_DAYS)
stmt = (
delete(models.User)
.where(
models.User.is_guest.is_(True),
models.User.email.is_(None),
func.coalesce(models.User.last_seen_at, models.User.created_at) < cutoff,
)
# Without this option, the "auto" strategy cannot evaluate coalesce in
# Python and falls back to fetching every matching primary key first.
# That is a second round trip for no benefit, because this session holds
# no User objects to synchronize.
.execution_options(synchronize_session=False)
)
removed = db.execute(stmt).rowcount or 0
db.commit()
return removed
def sweep() -> int:
"""One pass, with its own session. Never raises: a failed cleanup must not
be able to take the app down (same rule as seeding). Returns the guest
count, which is the number worth logging about."""
if not anything_to_sweep():
return 0
db = SessionLocal()
try:
# Ages out the per-visitor analytics rows, on its own terms: it is not
# about guests, and it must still happen on a deployment that has
# chosen to keep every account it ever minted.
aged = analytics.purge_old_visitor_days(db)
if aged:
logger.info("Aged out %d analytics visitor-day row(s).", aged)
removed = delete_stale_guests(db) if enabled() else 0
if removed:
logger.info(
"Cleaned up %d guest account(s) idle for %d+ days.",
removed,
RETENTION_DAYS,
)
return removed
except Exception:
db.rollback()
logger.exception("Guest cleanup failed; continuing without it.")
return 0
finally:
db.close()
async def _sweep_loop() -> None:
while True:
# Blocking DB work: keep it off the event loop, which is also serving
# SSE turn streams.
await run_in_threadpool(sweep)
await asyncio.sleep(SWEEP_INTERVAL_SECONDS)
def start_sweeper() -> asyncio.Task | None:
"""Kick off the periodic sweep; None when there is nothing to sweep."""
if not enabled():
logger.info("Guest cleanup disabled (multi_user=%s, retention_days=%d).",
auth.MULTI_USER, RETENTION_DAYS)
else:
logger.info(
"Guest cleanup on: deleting guests idle %d+ days, every %d hour(s).",
RETENTION_DAYS,
SWEEP_INTERVAL_SECONDS // 3600,
)
if not anything_to_sweep():
return None
return asyncio.create_task(_sweep_loop())
async def stop_sweeper(task: asyncio.Task | None) -> None:
if task is None:
return
task.cancel()
try:
await task
except asyncio.CancelledError:
pass
backend/app/compression.py
+70
View File
@@ -0,0 +1,70 @@
"""Storing a JSON column compressed.
`actions.context_snapshot` holds the entire assembled prompt for a turn. It is
89% of the database, which is 150.8 MB of JSON across 944 actions in production
and 232 kB per row on the longest adventure, and the free tier this deploys to
allows 512 MB. Reads are not the problem. The column is deferred, so a page load
never touches it and exactly one endpoint fetches one row of it at a time.
Storage is the problem, and storage has a hard limit.
Postgres already compresses the column. TOAST brings 150.8 MB down to about
89 MB, a factor of 1.7. Postgres chooses pglz for decompression speed on data a
query might filter on, and no query filters on this column. A prompt is written
once and read whole, occasionally, by one screen. zlib at the application layer
reaches three to four times on the same text, and the cost is one decompression
on a request that already makes an LLM call.
Doing it as a TypeDecorator rather than a second column keeps every call site
writing `action.context_snapshot = {...}` and reading a dict back, and keeps
`deferred=True`, `undefer()` and `load_only()` naming the same attribute they
named before. The storage format changes; nothing else does.
Level 6 is zlib's default and the knee of the curve here: 9 spends noticeably
more CPU on prompt text for about a percent more space.
"""
from __future__ import annotations
import json
import zlib
from sqlalchemy import LargeBinary
from sqlalchemy.types import TypeDecorator
LEVEL = 6
def pack(value) -> bytes:
"""A JSON-able value as compressed UTF-8."""
raw = json.dumps(value, separators=(",", ":"), default=str).encode("utf-8")
return zlib.compress(raw, LEVEL)
def unpack(blob: bytes) -> object:
"""The value `pack` was given."""
return json.loads(zlib.decompress(bytes(blob)).decode("utf-8"))
class CompressedJSON(TypeDecorator):
"""A JSON column stored as zlib-compressed UTF-8 in a BLOB/BYTEA.
`cache_ok = True`: the type carries no per-instance configuration, so
SQLAlchemy may reuse a compiled statement across instances of it.
"""
impl = LargeBinary
cache_ok = True
def process_bind_param(self, value, dialect):
return None if value is None else pack(value)
def process_result_value(self, value, dialect):
# Tolerate a row the backfill has not reached yet, or one written
# before the conversion: a snapshot that cannot be read back is worth
# less than the screen that shows it, and never worth a 500 on the
# turn that happens to load it.
if value is None:
return None
try:
return unpack(value)
except (zlib.error, UnicodeDecodeError, ValueError):
return None
backend/app/context/__init__.py
+19
View File
@@ -0,0 +1,19 @@
from . import history
from .builder import (
build_context,
count_tokens,
match_cards,
render_persona,
truncate_to_last_tokens,
)
from .history import story_actions
__all__ = [
"build_context",
"count_tokens",
"history",
"match_cards",
"render_persona",
"story_actions",
"truncate_to_last_tokens",
]
backend/app/context/builder.py
+473
View File
@@ -0,0 +1,473 @@
"""Context assembly per AI Dungeon's memory system
(help.aidungeon.com/faq/the-memory-system):
[AI Instructions] always included
[Player Character] always included when the adventure has a persona
[Plot Essentials] always included (classic "Memory")
[Story Summary] always included (manual in Phase 3, auto in Phase 6)
[Used Memories] top-K memory-bank retrievals (Phase 6, when enabled)
[Triggered Story Cards] "World Lore: <entry>", conditional; first dropped when over budget
[Story history] newest actions that fit the remaining token budget
[Author's Note] injected AUTHORS_NOTE_DEPTH actions before the end of history
[Latest player action] (+ script frontMemory right after it, Phase 4)
The list above comes from AI Dungeon's design. The order does not. This module
emits every fixed section first and every changing section after the history,
because prompt caching bills on a shared prefix. A section that changes near the
top of the prompt re-prices everything below it. See the comments on the static
block and the live sections in `build_context`.
"""
import functools
from dataclasses import dataclass
import tiktoken
from .. import models, worldstate
from . import history
AUTHORS_NOTE_DEPTH = 3 # actions from the end of history
CARD_BUDGET_SHARE = 0.4 # max share of non-reserved budget that story cards may take
NPC_WINDOW = 6 # actions of story searched for NPC trigger words ("in scene")
SEPARATOR = "\n\n"
# Output-length guidance. The endpoint enforces `max_output_tokens` as a hard
# limit, and it truncates the reply mid-sentence when the model reaches it. The
# state block is emitted last, so truncation removes it. Asking the model to
# finish inside the limit prevents the truncation.
LENGTH_HEADROOM = 50 # Tokens reserved from the cap for the state block.
# Models cannot count their own tokens, but they do follow a word budget, so the
# hint states a number of words. English prose averages 0.75 words per token.
WORDS_PER_TOKEN = 0.75
# Models regularly exceed a word budget, and the cap it protects is a hard
# limit. Aiming 10% below the real ceiling leaves room for that overshoot, so it
# does not consume the state block.
LENGTH_BUFFER = 0.90
MIN_LENGTH_HINT_WORDS = 40 # Below this, the hint adds nothing useful.
# A ceiling on its own gives one-sided guidance, and models respond to it
# differently. A verbose model treats it as a limit. A terse model has only the
# instruction to write as much as the moment needs, and it produces two
# paragraphs. Adding a floor turns the guidance into a range, so the same prompt
# produces a similar length from either model. The floor is a share of the
# ceiling so that it can never approach the ceiling.
LENGTH_FLOOR_SHARE = 0.35
# Below this word count, a floor means nothing, because a short turn is the
# correct turn at a tight cap. The wording used at a tight cap is also the
# wording that was measured to preserve the state block, so it is unchanged.
MIN_LENGTH_FLOOR_WORDS = 60
# The floor prevents a collapse to two paragraphs. It does not ask for an essay.
# At a 2400-token cap, the share alone would request a minimum of 555 words. A
# reader who wants longer turns can ask for them in the author's note.
MAX_LENGTH_FLOOR_WORDS = 300
@functools.lru_cache(maxsize=1)
def _encoding() -> tiktoken.Encoding:
return tiktoken.get_encoding("cl100k_base")
def count_tokens(text: str) -> int:
return len(_encoding().encode(text))
def truncate_to_last_tokens(text: str, budget: int) -> str:
tokens = _encoding().encode(text)
if len(tokens) <= budget:
return text
return _encoding().decode(tokens[-budget:])
@dataclass
class Section:
label: str
text: str
@property
def tokens(self) -> int:
return count_tokens(self.text)
def length_hint(max_output_tokens: int, *, has_ws: bool) -> str:
"""Ask for a turn that fits inside the output cap, stated as a word budget.
Returns an empty string when the cap is too small to state usefully. The
model can exceed the hint, so the hint earns its tokens only when there is
enough room for that overshoot to stay inside the cap.
"""
words = int((max_output_tokens - LENGTH_HEADROOM) * WORDS_PER_TOKEN * LENGTH_BUFFER)
if words < MIN_LENGTH_HINT_WORDS:
return ""
tail = (
" Finish the narration and append the state block well inside the limit."
if has_ws
else " Bring the turn to a close well inside the limit rather than "
"stopping mid-sentence."
)
# State the number as a ceiling, never as a budget. In measurements, the
# wording "keep this turn under about N words" read to the model as a target
# to fill. It raised the average from 174 words to 246 across five runs, and
# every hinted run was longer than every unhinted run. The hint therefore
# pushed turns toward the limit it exists to avoid. Naming the number as a
# limit, and adding that a typical turn is much shorter, held the average at
# 170 while still preserving the state block at tight caps.
floor = min(int(words * LENGTH_FLOOR_SHARE), MAX_LENGTH_FLOOR_WORDS)
if floor < MIN_LENGTH_FLOOR_WORDS:
return (
f"[Hard limit: this turn must not exceed {words} words. Write only as "
f"much as the moment needs — a typical turn is much shorter.{tail}]"
)
# Both numbers are bounds, and the wording is deliberately asymmetric. The
# ceiling uses "must not exceed", because the endpoint enforces it. The floor
# uses "should not stop short of". Neither reads as a target, which the
# measurement above shows is what matters. The clause that asks the model to
# prefer the lower end does the job the earlier wording did, which was to
# keep a verbose model away from the ceiling. It now has a number beneath it,
# so a terse model reading the same clause stops at the floor rather than at
# forty words.
return (
f"[Hard limit: this turn must not exceed {words} words, and it should not "
f"stop short of about {floor}. Prefer the lower end of that range unless "
f"the scene genuinely needs more.{tail}]"
)
def render_persona(adventure: models.Adventure) -> str:
"""Returns the Player Character section, or "" when there is no persona.
The three fields are independent. A name alone is enough, a description
alone is enough, and the wording holds together for either. Pronouns are
stated because the summarizer in `memorybank` writes about the protagonist
in the third person, and a model that has to infer a pronoun from a name
will sometimes infer wrongly and then repeat that error in every memory it
writes.
"""
name = adventure.persona_name.strip()
pronouns = adventure.persona_pronouns.strip()
desc = adventure.persona_desc.strip()
if not (name or desc):
return ""
head = f"You are {name}" if name else ""
if head and pronouns:
head += f" ({pronouns})"
# Joined with a space, not `SEPARATOR`: this is one short paragraph about
# one character, and a blank line inside it reads as two unrelated notes.
body = " ".join(part for part in (f"{head}." if head else "", desc) if part)
return f"Player character:\n{body}"
def _script_memory(adventure: models.Adventure) -> dict:
"""Script-provided memory overrides (populated by Phase 4 scripting)."""
state = adventure.script_state if isinstance(adventure.script_state, dict) else {}
memory = state.get("memory")
return memory if isinstance(memory, dict) else {}
def _history_text(action: models.Action) -> str:
"""Returns an AI turn as the model should see it in replayed history.
The result is the narration with its state block appended again,
reconstructed from the stored delta. The app strips that block before
storing and displaying the turn. Without this function, every past AI turn
would appear to have emitted no state, and the model would copy that pattern
and stop emitting state itself. Player turns and turns with no block pass
through unchanged.
The block replays the changes the engine ACCEPTED, not the ones the model
sent. Replaying what was sent showed the model a refused change standing as
though it had been applied, while the live values in the same prompt
disagreed with it. Nothing marked which of the two was true, so the model
read its own refused change as correct and sent it again.
This function reads `world_delta` rather than `context_snapshot`. It runs
for every action in the replayed history, and `context_snapshot` is deferred
so that a turn never loads the prompt archive from the database.
"""
text = action.text
wd = action.world_delta if isinstance(action.world_delta, dict) else None
if wd:
block = worldstate.render_delta_block(worldstate.applied_delta(wd))
if block:
text = f"{text}\n{block}"
return text
def _visible_npcs(actions: list[models.Action], stat_schema: dict) -> dict[str, str]:
"""Returns the NPCs whose trigger words appear in the recent story.
These are the NPCs in scene, and the prompt includes stats for them only.
The result maps an NPC id to its display name.
`actions` holds only the most recent actions. See `NPC_WINDOW`.
"""
recent = SEPARATOR.join(a.text for a in actions).lower()
visible: dict[str, str] = {}
for npc_key, ndef in (stat_schema.get("npcs") or {}).items():
if not isinstance(ndef, dict):
continue
if any(trigger in recent for trigger in worldstate.npc_triggers(ndef, npc_key)):
visible[npc_key] = worldstate.npc_name(ndef, npc_key)
return visible
def match_cards(cards: list[models.StoryCard], window_text: str) -> list[dict]:
"""Returns one record per matched story card, naming the keyword that matched.
Matching follows AI Dungeon's rules. It ignores case, respects spaces, and
matches partial words, so "boat" matches "boats".
Public since Phase 18b: `memorybank.cast_brief` runs the same rule over the
block it is about to summarize, so that the summarizer is told who the
characters in that stretch of story are. One rule, one implementation.
"""
haystack = window_text.lower()
matched = []
for card in cards:
for key in (k.strip().lower() for k in card.keys.split(",")):
if key and key in haystack:
matched.append(
{"id": card.id, "name": card.name, "keyword": key, "entry": card.entry}
)
break
return matched
def build_context(
adventure: models.Adventure,
settings: models.Settings,
memory_bank: dict | None = None,
exclude_action_id: int | None = None,
) -> tuple[str, str, dict]:
"""Returns (system_text, story_text, context_report). `memory_bank` is the
result of memorybank.retrieve_memories (None when the bank is off);
`exclude_action_id` omits one action from the story (see history.py)."""
script_mem = _script_memory(adventure)
# ----- The static block, which is identical on every turn -----
# This ordering exists to reduce cost. Prompt caching matches a prefix. The
# endpoint reuses the prompt up to the first byte that differs from the
# previous request, and no further. A section that changes near the top
# therefore re-prices everything below it, and what sits below it is the
# story history, which is most of the prompt. Sections that change from turn
# to turn go after the history, among the live sections. Placing them there
# also gives them the most recency, which is why `EMIT_REMINDER` goes last.
system_sections: list[Section] = [Section("narrator", settings.narrator_prompt.strip())]
# RPG world state (Phase 12): the instructions for reporting changes. The
# live values go into a live section below. The guide derived from the
# schema and the emit rule do not change while the scenario is unchanged.
stat_schema = adventure.scenario.stat_schema if adventure.scenario else None
has_ws = worldstate.has_schema(stat_schema)
persona_name = adventure.persona_name.strip()
if has_ws:
guide = worldstate.render_reference(stat_schema, persona_name)
if guide:
system_sections.append(Section("world_state_guide", guide))
system_sections.append(Section("world_state_rule", worldstate.EMIT_RULE))
if isinstance(script_mem.get("context"), str) and script_mem["context"].strip():
system_sections.append(Section("script_context", script_mem["context"].strip()))
if adventure.ai_instructions.strip():
system_sections.append(Section("ai_instructions", adventure.ai_instructions.strip()))
# Phase 18. This sits in the static block because only the user can edit it,
# so it never changes mid-story and stays inside the cached prefix. It is
# emitted whether or not the adventure has an RPG layer: an adventure with
# no stats still has a protagonist, and that is the case the persona was
# added for.
persona_text = render_persona(adventure)
if persona_text:
system_sections.append(Section("persona", persona_text))
if adventure.memory.strip():
system_sections.append(
Section("plot_essentials", f"Plot essentials:\n{adventure.memory.strip()}")
)
# ----- Live sections, which hold everything that changes -----
# This code builds them here and places them after the history further down.
# They are ordered from least to most volatile, so a turn that changes only
# the fastest-moving section leaves the others cached. The summary is
# rewritten every few turns. Lore changes with the scene. The retrieved
# memories change on most turns, and the stat values change on nearly every
# turn. `world_lore` is added below, because the history window determines
# which cards trigger and that window is not known yet.
summary_section = (
Section("story_summary", f"Story summary:\n{adventure.story_summary.strip()}")
if adventure.story_summary.strip()
else None
)
memories_section = None
if memory_bank and memory_bank.get("used"):
lines_text = "\n".join(f"- {m['text']}" for m in memory_bank["used"])
memories_section = Section("used_memories", f"Memories:\n{lines_text}")
world_state_section = None
refusal_note = ""
if has_ws:
# One read serves both the in-scene NPCs and the refusal note below.
recent = history.tail(adventure, NPC_WINDOW, exclude_action_id)
block = worldstate.render_state_section(
adventure.world_state, stat_schema, _visible_npcs(recent, stat_schema),
persona_name,
)
if block:
world_state_section = Section("world_state", block)
# Corrections for the previous AI turn only. A refusal the model has
# already had one chance to fix is stale, and repeating it every turn
# would price a correction into the whole rest of the adventure.
last_ai = next((a for a in reversed(recent) if a.type == "ai"), None)
if last_ai is not None:
refusal_note = worldstate.render_refusals(last_ai.world_delta)
authors_note_text = adventure.authors_note.strip()
if isinstance(script_mem.get("authorsNote"), str) and script_mem["authorsNote"].strip():
authors_note_text = script_mem["authorsNote"].strip()
authors_note = f"[Author's note: {authors_note_text}]" if authors_note_text else ""
front_memory = ""
if isinstance(script_mem.get("frontMemory"), str):
front_memory = script_mem["frontMemory"].strip()
length_note = length_hint(settings.max_output_tokens, has_ws=has_ws)
# The live sections sit below the history, but they are still part of the
# prompt, so they still count against the budget. `world_lore` is the
# exception, because the code below budgets it out of `available`.
reserved = (
sum(s.tokens for s in system_sections)
+ sum(
s.tokens
for s in (summary_section, memories_section, world_state_section)
if s is not None
)
+ count_tokens(authors_note)
+ count_tokens(front_memory)
+ count_tokens(length_note)
+ (count_tokens(worldstate.EMIT_REMINDER) if has_ws else 0)
+ count_tokens(refusal_note)
)
available = max(256, settings.context_token_budget - reserved)
# Only the newest actions can reach the prompt, because the code below
# either truncates the text to `available` tokens or stops at the budget.
# Fetch a window that is provably larger than that and no larger. Otherwise
# a long adventure reads its whole history on every turn and uses only the
# end of it.
actions = history.window_covering(
adventure, available, count_tokens, exclude_action_id
)
# ----- Story cards: triggered by recent story text (the window history could fill) -----
trigger_window = truncate_to_last_tokens(SEPARATOR.join(a.text for a in actions), available)
triggered = match_cards(adventure.story_cards, trigger_window)
card_budget = int(available * CARD_BUDGET_SHARE)
card_records = []
lore_lines: list[str] = []
used = 0
for match in triggered:
line = f"World Lore: {match['entry'].strip()}"
tokens = count_tokens(line)
included = used + tokens <= card_budget
if included:
lore_lines.append(line)
used += tokens
card_records.append(
{"id": match["id"], "name": match["name"], "keyword": match["keyword"],
"included": included}
)
lore_section = (
Section("world_lore", "\n".join(lore_lines)) if lore_lines else None
)
# ----- Story history: newest first until the remaining budget is spent -----
history_budget = available - used
included_actions: list[models.Action] = []
spent = 0
oldest_truncated = False
for action in reversed(actions):
# Budget against the text as it appears in the prompt, which includes
# the state block when this adventure tracks world state.
rendered = _history_text(action) if has_ws else action.text
tokens = count_tokens(rendered) + count_tokens(SEPARATOR)
if spent + tokens > history_budget:
if not included_actions:
# Even the newest action alone is over budget: hard-truncate it.
included_actions.append(
models.Action(
adventure_id=action.adventure_id,
type=action.type,
text=truncate_to_last_tokens(action.text, history_budget),
)
)
oldest_truncated = True
break
included_actions.append(action)
spent += tokens
included_actions.reverse()
# ----- Assemble the story text, with the author's note near the end -----
# Append each AI turn's state block again. The app strips it before storage,
# and the recent history has to show the model the pattern to follow.
texts = [_history_text(a) if has_ws else a.text for a in included_actions]
note_sections: list[Section] = []
if authors_note:
pos = max(0, len(texts) - AUTHORS_NOTE_DEPTH)
before, after = texts[:pos], texts[pos:]
if before:
note_sections.append(Section("history", SEPARATOR.join(before)))
note_sections.append(Section("authors_note", authors_note))
note_sections.append(Section("recent_history", SEPARATOR.join(after)))
else:
note_sections.append(Section("history", SEPARATOR.join(texts)))
# The live sections, ordered from least to most volatile. See the comment
# where they are built. They go below the history so that the history stays
# cached, and above the final sections so that those stay last.
for live in (summary_section, lore_section, memories_section, world_state_section):
if live is not None:
note_sections.append(live)
if front_memory:
note_sections.append(Section("front_memory", front_memory))
# Place the length hint just above the emit reminder, which keeps the last
# position. The length budget applies to the narration, and the reminder
# applies to the block that follows it, so this is also the order in which
# the model acts.
note_sections.append(Section("length_hint", length_note))
if has_ws:
# A correction for the previous turn sits directly above the reminder
# to emit a block, which is the instruction it modifies.
if refusal_note:
note_sections.append(Section("world_state_refusals", refusal_note))
# The emit rule sits in the system block, far from where the model
# generates text, so repeat it last where it has the most effect.
note_sections.append(Section("world_state_reminder", worldstate.EMIT_REMINDER))
story_sections = [s for s in note_sections if s.text]
system_text = SEPARATOR.join(s.text for s in system_sections if s.text)
story_text = SEPARATOR.join(s.text for s in story_sections)
all_sections = [s for s in system_sections if s.text] + story_sections
report = {
"sections": [
{"label": s.label, "text": s.text, "tokens": s.tokens} for s in all_sections
],
"prompt": {"system": system_text, "story": story_text},
"tokens": {
"total": count_tokens(system_text) + count_tokens(story_text),
"budget": settings.context_token_budget,
},
"cards": card_records,
"memories": memory_bank,
"history": {
"included": len(included_actions),
# The count covers the whole story rather than the window fetched
# above. Insights reports how many of the total actions it
# included, so this number must be the real total.
"total": history.count(adventure, exclude_action_id),
"oldest_truncated": oldest_truncated,
},
"settings": {
"model": settings.model,
"api_mode": settings.api_mode,
"temperature": settings.temperature,
"max_output_tokens": settings.max_output_tokens,
},
}
return system_text, story_text, report
backend/app/context/cursors.py
+172
View File
@@ -0,0 +1,172 @@
"""Phase 14: tracks how far along a story the derived work has reached.
Two things are built from the story and stored beside it: the memories and the
story summary. Both need to record where they stopped.
That mark used to be a count, such as "the first 12 story actions are covered".
A count is a position in a list, and this list changes. If you delete an action
in front of the mark, every later action moves down one slot, so the mark now
covers an action it never read. The rules in `memorybank` for sliding cursors,
rewinding them, and converting between positions and `Action.index` all existed
to correct for that, and each rule was a chance to introduce a silent error.
A cursor here is an anchor instead. It stores `(branch_id, depth)`, naming the
node up to and including which the work is done. Deleting an action does not
move it, because a depth is a coordinate along a path rather than a position in
a list. The question "what is not covered yet" becomes
`history.count_after(anchor)`, which asks about the story rather than about a
list index, and it stays correct no matter what is deleted in front of it.
The branch half of the anchor is what makes it survive forking. A depth alone is
ambiguous once two branches both hold a node at depth 41. The anchor names the
branch, and `Path.depth_on` reads it back as a depth on whichever story is being
played. That read caps the depth at the fork, or reports nothing covered if the
anchor sits on a branch this path does not contain. Until forking ships there is
one branch, so this always returns the stored depth. That is the point: the
coordinate system is correct before anything depends on it.
`NO_DEPTH`, which is -1, means nothing is covered. A new adventure therefore
needs no special case, because every node is deeper than -1.
"""
from sqlalchemy.orm import Session
from .. import models
from . import history, lineage
NO_DEPTH = lineage.NO_DEPTH
class Cursor:
"""One anchor on the adventure row, for either the memory bank or the summary.
The anchor is a pair of columns rather than a foreign key to the node. The
node can be deleted, which is what undo does, and the boundary still means
something afterwards. A foreign key would repeatedly fail to resolve.
"""
def __init__(self, name: str):
self.name = name
self.branch_field = f"{name}_cursor_branch_id"
self.depth_field = f"{name}_cursor_depth"
# ------------------------------------------------------------- reading
def stored(self, adventure: models.Adventure) -> tuple[int | None, int]:
"""Returns the anchor as written, without resolving it against a path."""
depth = getattr(adventure, self.depth_field)
return getattr(adventure, self.branch_field), (
NO_DEPTH if depth is None else depth
)
def depth(self, db: Session, adventure: models.Adventure) -> int:
"""Returns the anchor as a depth on the story being played now."""
branch_id, depth = self.stored(adventure)
return lineage.path_of(db, adventure).depth_on(branch_id, depth)
# ------------------------------------------------------------- writing
def anchor(
self, adventure: models.Adventure, branch_id: int | None, depth: int
) -> None:
"""Sets the anchor to a coordinate supplied directly.
This is the plain setter beneath `anchor_at`. Only an import supplies a
coordinate with no node to read it from. A v2 bundle carries the anchor
itself, as described in `app/bundle.py`, and the node it named lives in
a different database.
"""
setattr(adventure, self.branch_field, branch_id)
setattr(adventure, self.depth_field, max(depth, NO_DEPTH))
def anchor_at(self, adventure: models.Adventure, node: models.Action) -> None:
"""Marks the work done up to and including `node`.
This uses the node's own branch rather than the adventure's head. A block
of six actions can end before the fork that created the current branch,
and the coverage belongs where those actions are.
"""
self.anchor(adventure, node.branch_id, lineage.NO_DEPTH
if node.depth is None else node.depth)
def clear(self, adventure: models.Adventure) -> None:
"""Clears the anchor, so that nothing counts as covered.
Call this when the branch the anchor referred to is deleted. On Postgres
a stale branch id would never resolve. On SQLite the next fork can reuse
an id that was just freed, and a stale anchor would then resolve onto a
branch it never saw and report that stretch of story as summarized.
Clearing the anchor costs one re-summarize, which is the safe direction
to be wrong in.
"""
self.anchor(adventure, None, NO_DEPTH)
def rewind_to(
self, adventure: models.Adventure, branch_id: int | None, depth: int
) -> None:
"""Moves the anchor back to `depth` if it is past that depth.
The anchor never moves forward here. Moving backward is the only
direction that is safe without knowing what else changed. Covering
ground twice costs one summarizer call. Skipping ground removes a
stretch of story from the memories permanently.
"""
_, current = self.stored(adventure)
if current <= depth:
return
setattr(adventure, self.branch_field, branch_id)
setattr(adventure, self.depth_field, max(depth, NO_DEPTH))
MEMORY = Cursor("memory")
SUMMARY = Cursor("summary")
ALL = (MEMORY, SUMMARY)
def rewind_all(
adventure: models.Adventure, branch_id: int | None, depth: int
) -> None:
"""Returns a stretch of story to both the memory pass and the summary pass.
The two move together because they cover the same actions from different
directions. The summary folds in the memories, so withdrawing a memory
without rewinding the summary would leave the summary claiming to have read
something that no longer exists.
"""
for cursor in ALL:
cursor.rewind_to(adventure, branch_id, depth)
def anchor_at_position(
adventure: models.Adventure, cursor: Cursor, position: int
) -> None:
"""Sets `cursor` from a count of covered story actions.
A v1 bundle stores its mark as a count, and so does a database written
before the anchors existed.
The action at `position` in depth order is the node that carries the same
meaning, and it keeps that meaning after something in front of it is
deleted. A position past the end of the story is not an invalid value. An
adventure that was fully caught up under the old rule can hold one, and it
means the same thing as the tip, so this function anchors at the tip.
`migrations._backfill_cursor_anchors` implements this rule in SQL for every
adventure at once, without loading any of them. The two must agree.
"""
if position <= 0:
return
covered = history.slice_(adventure, position - 1, 1) or history.tail(adventure, 1)
if covered:
cursor.anchor_at(adventure, covered[0])
def position_of(adventure: models.Adventure, depth: int) -> int:
"""Returns how many story actions lie at or before `depth`.
This reads an anchor back as a count. The v1 export bundle stores cursors as
counts, and builds that have never used depths read v1 bundles. This
function is the only remaining code that speaks that coordinate system. The
v2 format introduced in SP6 replaces it.
"""
return max(history.count(adventure) - history.count_after(adventure, depth), 0)
backend/app/context/history.py
+458
View File
@@ -0,0 +1,458 @@
"""Reads part of a story without loading all of it.
`story_actions()` used to walk `adventure.actions`, which loads every row of the
adventure. Every caller then discarded nearly all of those rows. The context
builder joins the story and immediately trims it to the token budget. The
in-scene NPC check reads the last 6 actions. Memory retrieval reads the last 4.
The post-turn cursor clamp needs only a count. A turn on a 200-action adventure
read about 840 KB in order to use about 70 KB, and the cost grew with every
turn.
This module serves those shapes from SQL directly, as a tail, a slice, or a
count. A read is therefore bounded by the context budget rather than by the
length of the story.
Three rules hold the module together:
- There is one definition of a story action. That definition decides both what
a reader sees and what the summarizer receives, so the SQL and the Python must
agree exactly. `_STORY_TEXT` and `is_story_text()` express the same rule
twice. Keep them in step.
- No caller loads the same rows twice. If `adventure.actions` is already in
memory, every helper here slices that collection instead of running a query.
The scripting pipeline hands the whole history to user scripts, as AI Dungeon
does, so a scripted adventure costs no more than it did before.
- Every read applies the branch clause, as of Phase 14. `adventure.actions`
holds the actions of every branch rather than the story being played, so the
in-memory path filters the collection down to the path before slicing it, in
the same way the SQL does. Skipping that filter would build a prompt from two
different stories without reporting an error, which is why `lineage.Path`
owns both forms of the rule.
Reads order by `depth` rather than `index`. Since SP4, two different rows can
hold the same value for both columns, because the attempts at one turn share
them. Only `depth` together with the `live` test in the branch clause
identifies the row the story uses.
SP3 added the reads that count from a node rather than from the start:
`count_after`, `after`, and `newest`. The memory bank used to ask for positions
12 through 18 of the story, and the answer to that question changes when an
action in front of those positions is deleted. It now asks for the six actions
after depth 41, which is the question that forking requires in any case.
"""
from sqlalchemy import func, inspect as sa_inspect
from sqlalchemy.orm import Session, defer, object_session
from .. import models
from . import lineage
# How many of the newest actions to read before checking whether they cover the
# token budget. If they do not, the next size comes from the average action
# length just measured rather than from doubling the previous size. Doubling
# overshoots, which means reading hundreds of actions in order to use sixty.
WINDOW_START = 32
WINDOW_MARGIN = 0.15 # Aim this far past the budget, so a second round is rare.
WINDOW_STEP = 8 # Read at least this many more actions in each round.
# `depth` is the ordering key, and `id` breaks ties. Without `id`, the database
# would choose the order. Two rows can share a depth: a pre-tree row, which has
# a NULL depth and is invisible to reads, or a pair of sibling attempts.
_OLDEST_FIRST = (models.Action.depth, models.Action.id)
_NEWEST_FIRST = (models.Action.depth.desc(), models.Action.id.desc())
def _sql_stripped(column):
"""`column` with leading/trailing whitespace removed, portably.
SQLite and Postgres both accept `trim()` with a single argument, but that
form removes spaces only. Python's `str.strip()` also removes newlines and
tabs. Without this helper, an action containing only a newline would count
as story text in SQL but not in Python. Both dialects spell `replace()` and
`trim()` the same way, so this function converts the other whitespace to
spaces first. It does not handle form feed or vertical tab, because nothing
produces them.
"""
folded = column
for char in ("\n", "\r", "\t"):
folded = func.replace(folded, char, " ")
return func.trim(folded)
_STORY_TEXT = _sql_stripped(models.Action.text) != ""
def is_story_text(text: str) -> bool:
"""Returns whether `text` counts as story text.
This is the Python form of `_STORY_TEXT`. Keep the two in step.
"""
return bool(text.strip())
def _loaded_actions(adventure: models.Adventure) -> list[models.Action] | None:
"""The adventure's actions if they are already in memory, else None.
Slicing a collection that is already loaded costs nothing, and running a
query beside it would fetch the same rows a second time.
"""
state = sa_inspect(adventure)
if state.detached or "actions" in state.unloaded:
return None
return list(adventure.actions)
def _from_memory(
adventure: models.Adventure, exclude_action_id: int | None
) -> list[models.Action] | None:
"""The story, from the already-loaded collection, or None to go to SQL.
The collection holds the adventure's actions, which means the actions of
every branch. Filtering it down to the path here applies the same rule that
the SQL applies. Without that filter, the context builder would receive a
prompt built from siblings of the story being played.
Resolving the path requires a session to read the branch row from. If there
is no session, this function returns None so that the caller falls back to
SQL rather than guessing.
"""
loaded = _loaded_actions(adventure)
if loaded is None:
return None
db = _session(adventure)
if db is None:
return None
path = lineage.path_of(db, adventure)
rows = [
a for a in loaded
if path.contains(a)
and is_story_text(a.text)
and (exclude_action_id is None or a.id != exclude_action_id)
]
# `Adventure.actions` is ordered by `index`; a path is ordered by depth.
rows.sort(key=path.sort_key)
return rows
def _filters(
adventure: models.Adventure,
path: lineage.Path,
exclude_action_id: int | None,
entries: int | None = None,
) -> list:
# `adventure_id` is redundant beside the branch clause, because branch ids
# are unique and a branch already identifies one adventure. The filter
# remains because it costs little, it catches a node written onto another
# adventure's branch, and it makes the query easier to read.
conditions = [
models.Action.adventure_id == adventure.id,
path.clause(models.Action, count=entries),
_STORY_TEXT,
]
if exclude_action_id is not None:
conditions.append(models.Action.id != exclude_action_id)
return conditions
def _query(
db: Session,
adventure: models.Adventure,
path: lineage.Path,
exclude_action_id: int | None,
entries: int | None = None,
):
# Reasoning traces are never read from replayed history and can be larger
# than the narration itself on a reasoning model.
return (
db.query(models.Action)
.filter(*_filters(adventure, path, exclude_action_id, entries))
.options(defer(models.Action.reasoning))
)
def _count_query(
db: Session,
adventure: models.Adventure,
path: lineage.Path,
exclude_action_id: int | None,
entries: int | None = None,
):
"""A real `SELECT count(...)`.
This function deliberately avoids `_query(...).count()`. That form wraps the
entity select in a subquery, so the emitted SQL names every column,
including the deferred columns that this design keeps off the wire. Neither
form returns those bytes to the client, but the database still reads them,
and an egress guard that inspects the SQL cannot tell the two forms apart.
"""
return db.query(func.count(models.Action.id)).filter(
*_filters(adventure, path, exclude_action_id, entries)
)
def _session(adventure: models.Adventure) -> Session | None:
return object_session(adventure)
def _path(db: Session, adventure: models.Adventure) -> lineage.Path:
return lineage.path_of(db, adventure)
# ------------------------------------------------------------------ the API
def story_actions(
adventure: models.Adventure, exclude_action_id: int | None = None
) -> list[models.Action]:
"""Every story action, oldest first.
Call this function when you need the whole story. User scripts receive it,
which matches AI Dungeon's scripting API. Use `tail`, `slice_`, or `count`
when you need only part of the story.
`exclude_action_id` removes one action from the result. Retry uses it. The
attempt being replaced is still the live node of its turn, because it stays
live until a replacement exists, but it must not appear in the context that
is assembled to replace it.
"""
in_memory = _from_memory(adventure, exclude_action_id)
if in_memory is not None:
return in_memory
db = _session(adventure)
if db is None:
return []
return (
_query(db, adventure, _path(db, adventure), exclude_action_id)
.order_by(*_OLDEST_FIRST)
.all()
)
def count(adventure: models.Adventure, exclude_action_id: int | None = None) -> int:
"""How many story actions there are, without fetching any of them."""
in_memory = _from_memory(adventure, exclude_action_id)
if in_memory is not None:
return len(in_memory)
db = _session(adventure)
if db is None:
return 0
return (
_count_query(db, adventure, _path(db, adventure), exclude_action_id).scalar()
or 0
)
def tail_range(
adventure: models.Adventure,
skip: int,
limit: int,
exclude_action_id: int | None = None,
) -> list[models.Action]:
"""`limit` story actions ending `skip` actions before the end, oldest first.
Passing `skip=0` returns the newest slice. Passing `skip=32` and `limit=16`
returns the 16 actions immediately older than the newest 32. A growing
window therefore fetches only the actions it does not already hold.
This read is the reason the lineage window exists. The path's ranges do not
overlap and they descend, so the newest N nodes come from the newest few
lineage entries and the query never has to name the rest of the ancestry. A
story that has forked 200 times reads its tail with as few clauses as one
that has never forked.
`prefix_covering` estimates how many entries that takes, using depth
arithmetic alone. The estimate falls short only when an action was deleted
from the middle of the story. In that case this function widens the read to
the whole lineage, at the cost of one more query.
"""
if limit <= 0 or skip < 0:
return []
in_memory = _from_memory(adventure, exclude_action_id)
if in_memory is not None:
stop = len(in_memory) - skip
return in_memory[max(stop - limit, 0):stop] if stop > 0 else []
db = _session(adventure)
if db is None:
return []
path = _path(db, adventure)
entries = path.prefix_covering(skip + limit)
while True:
rows = (
_query(db, adventure, path, exclude_action_id, entries)
.order_by(*_NEWEST_FIRST)
.offset(skip)
.limit(limit)
.all()
)
if len(rows) >= limit or entries >= len(path):
break
entries = len(path) # short: widen once, to everything, and re-ask
rows.reverse()
return rows
def tail(
adventure: models.Adventure, limit: int, exclude_action_id: int | None = None
) -> list[models.Action]:
"""The newest `limit` story actions, returned oldest first."""
return tail_range(adventure, 0, limit, exclude_action_id)
def slice_(
adventure: models.Adventure,
start: int,
length: int,
exclude_action_id: int | None = None,
) -> list[models.Action]:
"""Story actions at positions [start, start + length), oldest first.
Positions index into the same filtered, depth-ordered list that the memory
cursors count in, which is why the SQL filter must match the Python filter
exactly.
This function counts from the oldest end, so it names the whole lineage. No
prefix of the ancestry contains the first ten actions of the story.
"""
if length <= 0 or start < 0:
return []
in_memory = _from_memory(adventure, exclude_action_id)
if in_memory is not None:
return in_memory[start:start + length]
db = _session(adventure)
if db is None:
return []
return (
_query(db, adventure, _path(db, adventure), exclude_action_id)
.order_by(*_OLDEST_FIRST)
.offset(start)
.limit(length)
.all()
)
def depth_of(action: models.Action) -> int:
"""`action.depth`, with the no-depth case spelled once.
A row with no depth predates the tree. No path contains such a row, so it
appears only in a collection that is already loaded. It sorts before the
story rather than after it.
"""
return action.depth if action.depth is not None else lineage.NO_DEPTH
def count_after(
adventure: models.Adventure, depth: int, exclude_action_id: int | None = None
) -> int:
"""How many story actions lie past `depth` on the path.
This replaces the older calculation, which compared the length of the story
with the position of the cursor. Deleting an action in front of the boundary
makes this number smaller, which is correct. It does not move the boundary
to a different action, which is the error that positions produced.
`covering_after` reports which lineage entries can hold a node deeper than
the boundary, so a cursor near the tip names one branch however many forks
lie below it.
"""
in_memory = _from_memory(adventure, exclude_action_id)
if in_memory is not None:
return sum(1 for a in in_memory if depth_of(a) > depth)
db = _session(adventure)
if db is None:
return 0
path = _path(db, adventure)
return (
_count_query(
db, adventure, path, exclude_action_id, path.covering_after(depth)
)
.filter(models.Action.depth > depth)
.scalar()
or 0
)
def after(
adventure: models.Adventure,
depth: int,
limit: int,
exclude_action_id: int | None = None,
) -> list[models.Action]:
"""The oldest `limit` story actions past `depth`, oldest first.
This returns the next block that the summarizer has not read. It asks a
question about the story rather than using an offset into a list whose
entries move.
"""
if limit <= 0:
return []
in_memory = _from_memory(adventure, exclude_action_id)
if in_memory is not None:
return [a for a in in_memory if depth_of(a) > depth][:limit]
db = _session(adventure)
if db is None:
return []
path = _path(db, adventure)
return (
_query(db, adventure, path, exclude_action_id, path.covering_after(depth))
.filter(models.Action.depth > depth)
.order_by(*_OLDEST_FIRST)
.limit(limit)
.all()
)
def newest(adventure: models.Adventure) -> models.Action | None:
"""The newest story action, or None on an empty story.
This returns a row rather than a count and an offset, because it names the
node an anchor moves to when derived work reaches the end of the story.
It used to return the second newest action. The memory bank held one action
back because retry rewrote a row, so a memory that covered the newest action
could describe narration the player had already replaced. Since SP4, a retry
writes a sibling row instead, and the derived work at a coordinate is
withdrawn when the story at that coordinate changes. There is nothing left
to hold back.
"""
rows = tail(adventure, 1)
return rows[0] if rows else None
def window_covering(
adventure: models.Adventure,
budget_tokens: int,
token_counter,
exclude_action_id: int | None = None,
) -> list[models.Action]:
"""Returns the newest story actions whose combined text exceeds `budget_tokens`.
The result always holds at least as much text as the context builder can
include, and never less.
This function counts tokens rather than estimating a characters-per-token
ratio, so the prompt matches what loading the whole story would produce. It
budgets against the raw text, which is never longer than the rendered
history text, so any error causes it to fetch slightly more than needed.
Each round fetches only the actions it does not already hold, so no row is
read twice however many rounds the loop takes.
"""
actions: list[models.Action] = []
tokens = 0
size = WINDOW_START
while True:
older = tail_range(
adventure, len(actions), size - len(actions), exclude_action_id
)
if not older:
return actions # The result already holds the whole story.
actions = older + actions
tokens += sum(token_counter(a.text) for a in older)
if len(actions) < size:
return actions # That was the whole story.
if tokens > budget_tokens:
return actions
# The window is still short. Estimate how many actions the budget needs
# at the average length just measured, then read that many.
average = tokens / len(actions)
projected = int(budget_tokens / average * (1 + WINDOW_MARGIN)) + WINDOW_STEP
size = max(projected, size + WINDOW_STEP)
backend/app/context/lineage.py
+278
View File
@@ -0,0 +1,278 @@
"""Phase 14: decides which nodes make up one story.
`tree.py` decides where a node is written. This module decides which nodes a
read can see, and it is the only place that makes that decision.
A branch owns the nodes played on it and inherits everything before its fork
point from its ancestors. "The story on branch C" is therefore not a value you
can filter a column on. It is an OR of ranges::
(branch_id = C) -- C's own nodes, through to the tip
OR (branch_id = B AND depth <= 5)
OR (branch_id = A AND depth <= 3)
The `branches.lineage` column records exactly that list, newest first. The fork
computes it once, so no read walks parent pointers to rebuild it.
The shape of the list gives two properties that the windowed reads depend on:
- The ranges do not overlap, and they descend. A branch's own nodes always sit
deeper than its fork point, and each lineage entry is capped at the fork depth
of the branch below it. Ordering the whole clause by `depth` descending
therefore reads entry 0's nodes, then entry 1's, then entry 2's. A tail read
can use the newest few entries and stop.
- The number of clauses depends on the size of the context window, not on how
many times the story has forked. A story with 200 forks whose newest branch
runs 40 turns reads with a single clause, because the window is covered before
the second entry is reached. `prefix_covering` implements this, and it is why
`history.window_covering` can keep its current shape.
Everything in this module reads. Nothing here creates a branch or writes a row.
An adventure with no branch has no story, and the write side repairs that. See
`tree.place_action` and the flush listener in `models.py`.
"""
from sqlalchemy import and_, false, or_
from sqlalchemy.orm import Session
from .. import models
# The depth of an adventure that has no actions. This mirrors `tree.NO_DEPTH`.
# It is duplicated here so that a read never has to import the write half.
NO_DEPTH = -1
# The opening node of an adventure. Depth 0 exists only on the root branch,
# because a fork starts its own nodes after the depth it forked at. This
# constant therefore names one node per adventure rather than one per branch.
#
# Migration 62 also placed every memory written before memories had coordinates
# at depth 0. That is why both places that can retire a memory must handle this
# depth: `memorybank.forget_node`, and a v1 import that has no depth to read.
ROOT_DEPTH = 0
def entries_of(branch: models.Branch) -> list[tuple[int, int | None]]:
"""Returns `branch.lineage` as (branch_id, max_depth) pairs, newest first.
An empty lineage means this branch alone, through to its tip. That is not an
error. It is what a root branch's lineage means, and it is also what a
branch row holds between the moment it is inserted and the moment its own id
is written into the column. The fallback is therefore correct rather than a
guess.
"""
raw = branch.lineage if isinstance(branch.lineage, list) else []
entries: list[tuple[int, int | None]] = []
for item in raw:
# JSON round-trips lists, but a hand-written row might hold tuples.
if not isinstance(item, (list, tuple)) or not item:
continue
branch_id = item[0]
max_depth = item[1] if len(item) > 1 else None
if not isinstance(branch_id, int):
continue
entries.append((branch_id, max_depth if isinstance(max_depth, int) else None))
return entries or [(branch.id, None)]
class Path:
"""One story, expressed as a SQL clause and as a Python predicate.
The object holds the lineage entries newest first, plus the depth of the
tip. The tip is used only to estimate how much story each entry covers.
"""
def __init__(self, entries: list[tuple[int, int | None]], tip: int | None = None):
self.entries = entries
self.tip = tip
def __bool__(self) -> bool:
return bool(self.entries)
def __len__(self) -> int:
return len(self.entries)
# ---------------------------------------------------------------- SQL
def clause(
self,
model=models.Action,
count: int | None = None,
):
"""Returns the branch clause over `model`, which is `Action` or `Memory`.
`count` limits the clause to the newest `count` lineage entries, which
produces a windowed read. Pass `None` for the whole lineage. Any caller
that counts from the oldest end, such as a slice or a total, must pass
`None`.
Every row this clause selects has a depth. Memories were once an
exception, because a hand-written memory had a branch but no depth and
needed an escape clause here to avoid being capped at a fork. SP7
anchors those memories at the head instead. See `tree.place_memory`. A
memory is now on a path rather than exempt from one, and a row with no
depth is a pre-tree leftover that no read should return.
Actions must also be live, as of SP4. One coordinate can hold several
attempts at a turn, and the story uses one of them. The other attempts
sit at the same branch and depth, and this clause excludes them once, so
that no read of the story has to account for retries. Only
`app/attempts.py` looks past this filter.
An empty path returns `false` rather than no filter at all. An adventure
whose nodes carry no branch has no story, and the correct way to show
that is an empty page rather than every branch at once.
"""
entries = self.entries if count is None else self.entries[:count]
if not entries:
return false()
on_path = or_(*[self._entry_clause(model, b, d) for b, d in entries])
if model is models.Action:
return and_(on_path, models.Action.live.is_(True))
return on_path
@staticmethod
def _entry_clause(model, branch_id: int, max_depth: int | None):
if max_depth is None:
return model.branch_id == branch_id
return and_(model.branch_id == branch_id, model.depth <= max_depth)
# ------------------------------------------------------------- Python
def contains(self, node) -> bool:
"""Returns whether `node` is on this path.
This is the Python equivalent of `clause()`. Keep the two in step.
Callers use it where the rows are already in memory. The scripting
pipeline hands user scripts the whole history, so a loaded collection
can be reduced to the path without a second query.
The method checks `live` first, and only on rows that define the
attribute, because memories have no siblings.
"""
if getattr(node, "live", True) is False:
return False
for branch_id, max_depth in self.entries:
if node.branch_id != branch_id:
continue
if max_depth is None:
return True
if node.depth is not None and node.depth <= max_depth:
return True
return False
def sort_key(self, node) -> tuple[int, int]:
"""Returns a sort key that orders nodes from oldest to newest.
`depth` is the ordering key. `id` breaks ties, which a pre-tree row with
a NULL depth or a pair of siblings can produce.
"""
return (node.depth if node.depth is not None else NO_DEPTH, node.id or 0)
# ------------------------------------------------------------ windowing
def prefix_covering(self, rows: int) -> int:
"""Returns how many lineage entries hold the newest `rows` nodes.
The result is an estimate from depth arithmetic rather than a query.
Entry *i* covers the depths between its own cap and the cap of the entry
below it, and a path holds at most one node per depth. The estimate is
therefore never too large. It is too small only when the story has gaps,
which happens after an action is deleted from the middle. In that case
the caller widens the read to the whole lineage, which costs one extra
query.
"""
total = len(self.entries)
if rows <= 0 or total == 0:
return total
covered = 0
for i, (_, max_depth) in enumerate(self.entries):
top = self.tip if max_depth is None else max_depth
below = self.entries[i + 1][1] if i + 1 < total else NO_DEPTH
if top is None or below is None:
# Either no tip was recorded, or a hand-written row is missing a
# cap. There is nothing to estimate from, so return every entry.
# Guessing low would hide the older half of the story.
return total
covered += max(top - below, 0)
if covered >= rows:
return i + 1
return total
def covering_after(self, depth: int) -> int:
"""Returns how many lineage entries can hold a node deeper than `depth`.
This is the counterpart to `prefix_covering`, and it is exact rather than
an estimate. Entry *i* holds nothing deeper than its own cap, and the
caps descend, so the first entry capped at or below `depth` ends the
search. That entry and every older one fall behind the boundary. As a
result, reading the story after the cursor touches one branch on any
story whose cursor sits on its newest branch, however many times the
story has forked.
"""
for i, (_, max_depth) in enumerate(self.entries):
if max_depth is not None and max_depth <= depth:
return i
return len(self.entries)
def depth_on(self, branch_id: int | None, depth: int) -> int:
"""Reads a stored `(branch_id, depth)` anchor as a depth on this path.
An anchor records how far along a story some derived work reached, such
as which actions the memories cover or what the summary folded in. The
anchor names a node, so moving to a different path needs an explicit
answer. There are two cases:
- The anchor's branch is on this path. The depth stands, capped at the
fork where this path leaves that branch, because nothing past the fork
belongs to this story.
- The anchor's branch is not on this path. The work was done on a branch
this story does not contain, so nothing here counts as covered.
The second case cannot occur while an adventure has one branch, because
the anchor is always set from a node on it. It exists because the safe
answer to an unknown anchor is to redo the work rather than skip it.
"""
if depth <= NO_DEPTH:
return NO_DEPTH
if branch_id is None:
# The anchor predates the tree, or someone set it by hand. There is
# only one story, so the depth is a position in it.
return depth
for entry_branch, max_depth in self.entries:
if entry_branch == branch_id:
return depth if max_depth is None else min(depth, max_depth)
return NO_DEPTH
def branch_of(db: Session, adventure: models.Adventure) -> models.Branch | None:
"""Returns the branch this adventure is read at, or None if it has none.
This function is deliberately not `tree.head_branch`, which creates a branch.
A GET request must not write. An adventure with no branch row also has no
nodes that carry a branch, so both answers agree that there is no story.
"""
if adventure.head_branch_id is not None:
branch = db.get(models.Branch, adventure.head_branch_id)
if branch is not None:
return branch
# The head points at a branch that no longer exists. Fall through to the
# root, which is the same recovery that `tree.head_branch` performs on
# the write side.
return (
db.query(models.Branch)
.filter(
models.Branch.adventure_id == adventure.id,
models.Branch.parent_branch_id.is_(None),
)
.order_by(models.Branch.id)
.first()
)
def path_of(db: Session, adventure: models.Adventure) -> Path:
"""Returns the story that the adventure's head currently sits on."""
branch = branch_of(db, adventure)
if branch is None:
return Path([], adventure.head_depth)
return Path(entries_of(branch), adventure.head_depth)
backend/app/database.py
+74
View File
@@ -0,0 +1,74 @@
import os
from pathlib import Path
from sqlalchemy import create_engine, event
from sqlalchemy.orm import DeclarativeBase, sessionmaker
# AIDND_DB_PATH lets deployments (Docker volume, hosted disk) relocate the
# SQLite database; default stays backend/data.db for local runs. The parent
# directory also hosts the auto-generated secret.key (see security.py), so
# DB_PATH stays defined even when Postgres is in use.
_env_db_path = os.environ.get("AIDND_DB_PATH")
DB_PATH = (
Path(_env_db_path).resolve()
if _env_db_path
else Path(__file__).resolve().parent.parent / "data.db"
)
# `AIDND_DATABASE_URL`, or the conventional `DATABASE_URL`, switches the app to
# a server database. Any SQLAlchemy URL works, and hosted deploys use Postgres,
# which Phase 9 settled on Neon for. If neither variable is set, the app uses
# SQLite.
DATABASE_URL = (
os.environ.get("AIDND_DATABASE_URL", "").strip()
or os.environ.get("DATABASE_URL", "").strip()
)
def _normalize_url(url: str) -> str:
"""Map the postgres:// / postgresql:// schemes hosts hand out to the
psycopg3 driver installed in requirements.txt."""
for prefix in ("postgres://", "postgresql://"):
if url.startswith(prefix):
return "postgresql+psycopg://" + url[len(prefix):]
return url
if DATABASE_URL:
engine = create_engine(
_normalize_url(DATABASE_URL),
# Serverless Postgres (Neon) suspends idle databases; pre-ping
# replaces silently-dead pooled connections instead of erroring.
pool_pre_ping=True,
# Store/read naive UTC like SQLite does, regardless of server default.
connect_args={"options": "-c timezone=UTC"},
)
else:
DB_PATH.parent.mkdir(parents=True, exist_ok=True)
engine = create_engine(
f"sqlite:///{DB_PATH}",
connect_args={"check_same_thread": False},
)
@event.listens_for(engine, "connect")
def _enable_sqlite_foreign_keys(dbapi_connection, _record):
# SQLite ships with foreign keys OFF per connection; without this every
# ondelete=CASCADE/SET NULL in models.py is silently ignored.
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA foreign_keys=ON")
cursor.close()
SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False)
class Base(DeclarativeBase):
pass
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
backend/app/debuglog.py
+71
View File
@@ -0,0 +1,71 @@
"""In-memory ring buffer of recent provider requests/responses for the debug page.
API keys never enter the log: only the request body (which carries no
credentials) and response text are recorded, both truncated.
"""
import itertools
from collections import deque
from datetime import datetime, timezone
MAX_ENTRIES = 30
MAX_TEXT = 6000
_entries: deque[dict] = deque(maxlen=MAX_ENTRIES)
_ids = itertools.count(1)
def _clip(text: str) -> str:
if len(text) <= MAX_TEXT:
return text
return text[:MAX_TEXT] + f"\n… [{len(text) - MAX_TEXT} more chars truncated]"
def _clip_obj(obj):
if isinstance(obj, str):
return _clip(obj)
if isinstance(obj, dict):
return {k: _clip_obj(v) for k, v in obj.items()}
if isinstance(obj, list):
return [_clip_obj(v) for v in obj]
return obj
def start_entry(url: str, model: str, body: dict) -> dict:
entry = {
"id": next(_ids),
"time": datetime.now(timezone.utc).isoformat(),
"url": url,
"model": model,
"request": _clip_obj(body),
"status": "pending",
"response": "",
"usage": None,
"error": None,
}
_entries.appendleft(entry)
return entry
def finish_entry(
entry: dict,
*,
response: str = "",
error: str | None = None,
usage: dict | None = None,
) -> None:
"""Finishes a log entry.
`usage` is the endpoint's own token accounting, when it reported any. On
OpenRouter it carries `prompt_tokens_details.cached_tokens`, which is the
only direct measure of whether the prompt prefix is being cached, and it is
worth seeing next to the request that produced it.
"""
entry["response"] = _clip(response)
entry["usage"] = usage
entry["error"] = error
entry["status"] = "error" if error else "ok"
def recent() -> list[dict]:
return list(_entries)
backend/app/images.py
+67
View File
@@ -0,0 +1,67 @@
"""Scenario cover art: inline data URIs in, cacheable URLs out.
A scenario's `image` column holds either an `https://` URL or a base64
`data:image/…` URI (the editor downscales uploads before storing one). Sending
those data URIs inside list responses would balloon them, so lists advertise a
`image_url` pointing at `GET /api/scenarios/{id}/image` instead, and the bytes
are fetched once and cached by the browser.
"""
import base64
import binascii
import re
# Only raster formats a browser renders in an <img>. SVG is deliberately absent:
# it can carry script, and these bytes are served from our own origin.
DATA_URI_RE = re.compile(
r"^data:(image/(?:png|jpeg|webp|gif|avif));base64,([A-Za-z0-9+/=\s]+)$",
re.IGNORECASE,
)
def public_url(scenario_id: int, image: str, version: object) -> str:
"""The URL a client should load for this scenario's art ("" if none).
`version` is any object with a stable repr, and is normally the row's
`updated_at`. It becomes a cache-buster, so the image response can be marked
immutable and still refresh as soon as the author replaces the picture.
"""
if not image:
return ""
if DATA_URI_RE.match(image):
stamp = int(version.timestamp()) if hasattr(version, "timestamp") else 0
return f"/api/scenarios/{scenario_id}/image?v={stamp}"
# Anything else must be an absolute https URL. http:// is rejected rather
# than passed through: the deployed app is https, so a browser would block
# it as mixed content and the author would just see a broken image.
return image if image.startswith("https://") else ""
def sanitize(value: object, max_length: int) -> str:
"""Coerce an untrusted `image` value from an import bundle to a safe one.
A value that is not a supported data URI or an https URL, or that is too
large to store, becomes "". A hostile or merely unfamiliar bundle therefore
cannot pass in a `javascript:` URI or exceed the column cap.
"""
if not isinstance(value, str) or not value or len(value) > max_length:
return ""
if DATA_URI_RE.match(value):
return value if decode(value) is not None else ""
return value if value.startswith("https://") else ""
def decode(image: str) -> tuple[bytes, str] | None:
"""`(bytes, content_type)` for a stored data URI, or None if it isn't one."""
match = DATA_URI_RE.match(image or "")
if not match:
return None
try:
# validate=True rejects anything outside the base64 alphabet, so strip
# the newlines a hand-pasted or line-wrapped URI may carry first.
payload = re.sub(r"\s+", "", match.group(2))
return base64.b64decode(payload, validate=True), match.group(1).lower()
except (binascii.Error, ValueError):
# The base64 is truncated or hand-edited. Treat it as no image rather
# than return a 500.
return None
backend/app/limits.py
+345
View File
@@ -0,0 +1,345 @@
"""Phase 9: abuse guards for hosted, multi-user deployments.
Rate limits and row caps do nothing in local mode, because a single local player
should never be throttled by their own app. The values are hardcoded on purpose.
They are generous enough that a legitimate player never notices them, and tight
enough that a hostile visitor cannot exhaust the demo key, saturate the CPU, or
fill the database.
"""
import json
import os
import threading
import time
from collections import defaultdict, deque
from fastapi import HTTPException, Request
from sqlalchemy import func
from sqlalchemy.orm import Session
from . import auth, models
# ---------- Rate limiting ----------
# Fixed windows per scope and caller. The windows live in memory, which is
# enough for the single-process deployment this app targets. The worst case
# after a restart is a brief extra allowance.
# Maps a scope to (max requests, window seconds).
RATE_LIMITS: dict[str, tuple[int, int]] = {
"turn": (10, 60), # AI turn generation. The demo key also has a daily cap.
"chat": (30, 60), # The AI Chat scratchpad, for power users.
"script-test": (30, 60), # Sandboxed, but each run costs up to 2s of CPU.
"connection-test": (10, 60), # Outbound HTTP to a user-supplied URL.
"import": (30, 60), # Large writes.
"auth": (10, 300), # Register and login attempts, per IP.
"guest": (30, 300), # New guest users, per IP. Each one is a database row.
# Pageview beacons. The limit is generous, because a real reader clicking
# around a SPA sends a handful a minute, and it is low enough that nobody
# can inflate the traffic numbers faster than by reloading the page.
"analytics": (120, 60),
}
_windows: dict[tuple[str, str], deque] = defaultdict(deque)
_windows_guard = threading.Lock()
# How many proxy hops sit between the app and the real client. On Render, and on
# most platforms, that is one, because the platform's edge appends the connecting
# IP to the right of `X-Forwarded-For`. A client can prepend any value on the
# left, but it cannot push a value past the edge's own append, so the trustworthy
# client IP is the entry that many places from the right rather than uvicorn's
# leftmost choice. Trusting the leftmost entry let anyone rotate
# `X-Forwarded-For` to get a fresh rate-limit bucket per request and bypass the
# auth and guest limits. If the deployment adds more hops, set
# `AIDND_TRUSTED_PROXY_HOPS`.
TRUSTED_PROXY_HOPS = max(1, int(os.environ.get("AIDND_TRUSTED_PROXY_HOPS", "1") or 1))
def client_ip(request: Request) -> str:
"""Returns the real client IP, resisting a spoofed `X-Forwarded-For`.
The function reads the hop the trusted edge appended, which is the rightmost
entry minus any extra trusted hops. If no forwarded header is present, which
happens locally, in development, and on a direct connection, it falls back to
the socket peer.
The function is public because the access log needs the same answer. Two
functions that each decide which address belongs to the caller is how one of
them ends up trusting a header it should not.
"""
forwarded = request.headers.get("x-forwarded-for")
if forwarded:
parts = [p.strip() for p in forwarded.split(",") if p.strip()]
if parts:
return parts[-min(TRUSTED_PROXY_HOPS, len(parts))]
return request.client.host if request.client else "unknown"
def rate_limit(scope: str, request: Request, user: models.User | None = None) -> None:
"""Raises a 429 when the caller exceeds the scope's window.
The window is keyed per user when a user is known, because an account
survives an IP change, and per IP otherwise.
"""
if not auth.MULTI_USER:
return
limit, window_seconds = RATE_LIMITS[scope]
key = (scope, f"u{user.id}" if user else f"ip{client_ip(request)}")
now = time.time()
with _windows_guard:
window = _windows[key]
while window and window[0] < now - window_seconds:
window.popleft()
if len(window) >= limit:
raise HTTPException(
429, "You're doing that too fast — wait a minute and try again."
)
window.append(now)
if len(_windows) > 10_000:
_prune(now)
# ---------- Per-account login throttle ----------
# This is defense in depth next to the per-IP `auth` limit. A botnet dilutes
# that limit, because many real source IPs each get their own bucket, so it
# cannot by itself stop a distributed guessing run against one account. This cap
# keys on the target email rather than on the caller, so guessing one account's
# password stays expensive however many addresses the guesses come from.
#
# Only failures count, and a correct password clears the record. The window
# slides over a short period rather than locking the account, so a user who
# mistypes a few times recovers within minutes. The trade-off is that an
# attacker can keep a known account throttled, which is an inconvenience and is
# preferable to letting the account be brute-forced.
LOGIN_FAIL_LIMIT = 8 # Failed attempts per account.
LOGIN_FAIL_WINDOW = 900 # The window in seconds, which is 15 minutes.
_login_fails: dict[str, deque] = defaultdict(deque)
_login_guard = threading.Lock()
def check_login_allowed(email: str) -> None:
"""Raises a 429 when an account has too many recent failed logins.
Call this before verifying the password, so that a guess never reaches the
hash.
"""
if not auth.MULTI_USER:
return
now = time.time()
with _login_guard:
window = _login_fails[email]
while window and window[0] < now - LOGIN_FAIL_WINDOW:
window.popleft()
if len(window) >= LOGIN_FAIL_LIMIT:
raise HTTPException(
429,
"Too many failed sign-in attempts for this account — "
"wait a few minutes and try again.",
)
def note_login_failure(email: str) -> None:
"""Records one failed attempt against `email`."""
if not auth.MULTI_USER:
return
now = time.time()
with _login_guard:
_login_fails[email].append(now)
if len(_login_fails) > 10_000: # Bound the map against a flood of unique emails.
stale = [
key for key, window in _login_fails.items()
if not window or window[-1] < now - LOGIN_FAIL_WINDOW
]
for key in stale:
del _login_fails[key]
def note_login_success(email: str) -> None:
"""Clears the account's failure record after a correct password."""
with _login_guard:
_login_fails.pop(email, None)
def _prune(now: float) -> None:
"""Drops callers whose whole window has expired, so the per-IP dict stays bounded.
Call this with the guard held.
"""
longest = max(seconds for _, seconds in RATE_LIMITS.values())
stale = [key for key, window in _windows.items()
if not window or window[-1] < now - longest]
for key in stale:
del _windows[key]
# ---------- Per-user row caps ----------
MAX_ADVENTURES_PER_USER = 100
MAX_SCENARIOS_PER_USER = 200
MAX_SCRIPTS_PER_USER = 200
MAX_STORY_CARDS_PER_OWNER = 200 # Per scenario or per adventure.
MAX_MEMORIES_PER_ADVENTURE = 1000
MAX_ACTIONS_PER_ADVENTURE = 5000
# Phase 14, SP6. A tree holds one branch per divergence somebody built a story
# on, so a tree with more branches than the story has turns came from a file
# rather than from play. The cap applies to imports only. Forking is a POST that
# adds one row and has no cap of its own, and the cap that matters there is
# `MAX_ACTIONS_PER_ADVENTURE` above.
MAX_BRANCHES_PER_ADVENTURE = 1000
def check_row_cap(
kind: str,
db: Session,
user: models.User,
*,
adventure: models.Adventure | None = None,
scenario_id: int | None = None,
adventure_id: int | None = None,
) -> None:
"""Raises a 409 when creating one more row of `kind` would exceed its cap.
The caller has already checked ownership of the scenario or adventure passed
in.
"""
if not auth.MULTI_USER:
return
if kind == "adventures":
count = _count(db, models.Adventure, models.Adventure.user_id == user.id)
cap, subject, hint = (
MAX_ADVENTURES_PER_USER, "adventures",
"delete one you no longer play to make room",
)
elif kind == "scenarios":
count = _count(db, models.Scenario, models.Scenario.user_id == user.id)
cap, subject, hint = (
MAX_SCENARIOS_PER_USER, "scenarios", "delete one to make room"
)
elif kind == "scripts":
count = _count(db, models.Script, models.Script.user_id == user.id)
cap, subject, hint = (
MAX_SCRIPTS_PER_USER, "scripts", "delete one to make room"
)
elif kind == "story_cards":
owner_filter = (
models.StoryCard.scenario_id == scenario_id
if scenario_id is not None
else models.StoryCard.adventure_id == adventure_id
)
count = _count(db, models.StoryCard, owner_filter)
cap, subject, hint = (
MAX_STORY_CARDS_PER_OWNER, "story cards here", "delete one to make room"
)
elif kind == "memories":
count = _count(db, models.Memory, models.Memory.adventure_id == adventure.id)
cap, subject, hint = (
MAX_MEMORIES_PER_ADVENTURE, "memories in this adventure",
"delete some to make room",
)
elif kind == "actions":
# Count every action in the adventure, which is the whole tree rather
# than the path being played. That number is what costs storage, and
# nothing is pruned automatically, so it is the right one to cap. It does
# mean a heavily branched adventure reaches the cap while its story is
# shorter than the cap, which is why the message counts "actions in this
# adventure" rather than turns.
count = _count(db, models.Action, models.Action.adventure_id == adventure.id)
cap, subject, hint = (
MAX_ACTIONS_PER_ADVENTURE, "actions in this adventure",
"export it and continue in a new adventure",
)
else: # pragma: no cover. This is a programming error, not user input.
raise ValueError(f"Unknown row cap kind: {kind}")
if count >= cap:
raise HTTPException(409, f"You've reached the limit of {cap} {subject} — {hint}.")
def _count(db: Session, model, condition) -> int:
return db.query(func.count(model.id)).filter(condition).scalar() or 0
_BUNDLE_LIST_CAPS = {
"story_cards": MAX_STORY_CARDS_PER_OWNER,
"memories": MAX_MEMORIES_PER_ADVENTURE,
"actions": MAX_ACTIONS_PER_ADVENTURE,
"branches": MAX_BRANCHES_PER_ADVENTURE,
}
def check_bundle_lists(**lists) -> None:
"""Raises a 409 when an import bundle's lists exceed the caps live creation uses.
The keyword arguments are `story_cards`, `memories`, `actions`, and
`branches`.
"""
if not auth.MULTI_USER:
return
for name, value in lists.items():
cap = _BUNDLE_LIST_CAPS[name]
if isinstance(value, list) and len(value) > cap:
noun = name.replace("_", " ")
raise HTTPException(
409, f"This file contains {len(value)} {noun} — the limit is {cap}."
)
# ---------- Request body size ----------
# The limit is generous enough for the largest legitimate payload, which is an
# adventure export holding thousands of actions. It applies in every mode, and no
# honest request approaches it.
MAX_BODY_BYTES = 2 * 1024 * 1024
MAX_IMPORT_BODY_BYTES = 20 * 1024 * 1024
class BodySizeLimitMiddleware:
"""Rejects oversized request bodies by their declared `Content-Length`.
This is pure ASGI rather than `BaseHTTPMiddleware`, so SSE responses stream
through unchanged. A chunked upload with no length is refused, because every
real client of this API sends `Content-Length`, including browser fetch and
curl with a file.
"""
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope["type"] == "http" and scope.get("method") in ("POST", "PUT", "PATCH"):
headers = {k.decode("latin-1").lower(): v.decode("latin-1")
for k, v in scope.get("headers", [])}
limit = (
MAX_IMPORT_BODY_BYTES
if scope.get("path", "").endswith("/import")
else MAX_BODY_BYTES
)
length = headers.get("content-length")
problem = None
if length is None:
if "chunked" in headers.get("transfer-encoding", "").lower():
problem = (411, "Content-Length is required.")
else:
try:
if int(length) > limit:
problem = (
413,
f"Request too large (limit {limit // (1024 * 1024)} MB).",
)
except ValueError:
problem = (400, "Invalid Content-Length.")
if problem:
await _send_json_error(send, *problem)
return
await self.app(scope, receive, send)
async def _send_json_error(send, status: int, detail: str) -> None:
body = json.dumps({"detail": detail}).encode()
await send({
"type": "http.response.start",
"status": status,
"headers": [(b"content-type", b"application/json"),
(b"content-length", str(len(body)).encode())],
})
await send({"type": "http.response.body", "body": body})
backend/app/main.py
+176
View File
@@ -0,0 +1,176 @@
import os
from contextlib import asynccontextmanager
from pathlib import Path
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from starlette.exceptions import HTTPException as StarletteHTTPException
from . import analytics, cleanup
from .auth import MULTI_USER
from .database import engine
from .limits import BodySizeLimitMiddleware
from .migrations import bootstrap
from .routers import (
adventures, analytics as analytics_router, auth, chat, debug, scenarios, scripts,
settings, story_cards,
)
from .seed import seed_public_scenarios
bootstrap(engine)
seed_public_scenarios(engine)
# Production serves the SPA same-origin, so CORS only matters for the Vite dev
# server; AIDND_CORS_ORIGINS overrides for any other cross-origin setup.
CORS_ORIGINS = [
o.strip()
for o in os.environ.get("AIDND_CORS_ORIGINS", "").split(",")
if o.strip()
] or ["http://localhost:5173", "http://127.0.0.1:5173"]
@asynccontextmanager
async def lifespan(_app: FastAPI):
# Sweeps once on boot, then on an interval. Booting is the reliable
# trigger on Render's free tier, where the service sleeps after ~15
# minutes and a long-running timer rarely gets to fire.
sweeper = cleanup.start_sweeper()
# Visit counters are buffered in memory and written in batches; this is
# what turns them into rows, and stop_flusher writes out the last batch so
# a deploy doesn't drop it.
flusher = analytics.start_flusher()
try:
yield
finally:
await cleanup.stop_sweeper(sweeper)
await analytics.stop_flusher(flusher)
# The interactive API docs stay local-only: in multi-user mode they just hand
# strangers a map of the API surface.
app = FastAPI(
title="AI D&D",
docs_url=None if MULTI_USER else "/docs",
redoc_url=None,
openapi_url=None if MULTI_USER else "/openapi.json",
lifespan=lifespan,
)
app.add_middleware(
CORSMiddleware,
allow_origins=CORS_ORIGINS,
allow_methods=["*"],
allow_headers=["*"],
)
app.add_middleware(BodySizeLimitMiddleware)
class SecurityHeadersMiddleware:
"""Standard hardening headers on every response. Pure ASGI (wraps `send`)
so SSE streams pass through unbuffered. The CSP allows exactly what the
SPA uses: same-origin everything, inline styles (React), Google Fonts."""
_HEADERS = [
(b"x-content-type-options", b"nosniff"),
(b"referrer-policy", b"same-origin"),
(b"x-frame-options", b"DENY"),
(
b"content-security-policy",
b"default-src 'self'; "
b"script-src 'self'; "
b"style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; "
b"font-src https://fonts.gstatic.com; "
b"img-src 'self' data:; "
b"connect-src 'self'; "
b"frame-ancestors 'none'",
),
]
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope["type"] != "http":
return await self.app(scope, receive, send)
async def send_with_headers(message):
if message["type"] == "http.response.start":
message.setdefault("headers", [])
message["headers"] = list(message["headers"]) + self._HEADERS
await send(message)
await self.app(scope, receive, send_with_headers)
class ApiErrorMiddleware:
"""Counts failed API responses for the analytics dashboard.
This is middleware rather than an exception handler, because it observes
what the client received. A 429 from a rate limiter, a 404 from routing, and
a 500 from a handler that never returned all reach it the same way. It is
pure ASGI for the same reason as the headers above: an SSE turn must not be
buffered on its way out. It watches `/api` only, because a 404 on the SPA
mount is a page load rather than a fault.
"""
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope["type"] != "http" or not scope.get("path", "").startswith("/api"):
return await self.app(scope, receive, send)
async def send_counting(message):
if message["type"] == "http.response.start" and message["status"] >= 400:
# The router has already put the matched route on the scope by
# the time a response starts, so the label can name the
# endpoint rather than the caller's path.
analytics.record(
analytics.M_ERROR,
analytics.api_route_label(scope, message["status"]),
)
await send(message)
await self.app(scope, receive, send_counting)
app.add_middleware(ApiErrorMiddleware)
app.add_middleware(SecurityHeadersMiddleware)
app.include_router(auth.router)
app.include_router(scenarios.router)
app.include_router(adventures.router)
app.include_router(story_cards.router)
app.include_router(scripts.router)
app.include_router(settings.router)
app.include_router(chat.router)
app.include_router(debug.router)
app.include_router(analytics_router.router)
@app.get("/api/health")
def health():
return {"ok": True}
# In production, serve the built frontend (frontend/dist) as static files.
class SPAStaticFiles(StaticFiles):
"""Serve index.html for unknown paths so client-side routes (/play/3)
survive a page reload. API routes are matched before this mount."""
async def get_response(self, path, scope):
try:
response = await super().get_response(path, scope)
except StarletteHTTPException as exc:
if exc.status_code != 404:
raise
return await super().get_response("index.html", scope)
if response.status_code == 404:
return await super().get_response("index.html", scope)
return response
frontend_dist = Path(__file__).resolve().parent.parent.parent / "frontend" / "dist"
if frontend_dist.is_dir():
app.mount("/", SPAStaticFiles(directory=frontend_dist, html=True), name="frontend")
backend/app/memorybank.py
+828
View File
@@ -0,0 +1,828 @@
"""Phase 6: automatic summarization and the embedding memory bank.
This follows AI Dungeon's memory system. See
help.aidungeon.com/faq/the-memory-system.
After each turn, `run_post_turn` runs as a fire-and-forget task with its own
database session. It does three things:
- Every `MEMORY_INTERVAL` actions, starting once the adventure reaches
`MEMORY_START` actions, it summarizes each uncovered block of actions into a
short memory. A block waits until `SETTLE_SLACK` actions sit past it, so a
memory never ends on the action a retry could replace.
- Every `SUMMARY_INTERVAL` actions, it rewrites the story summary to include the
new memories. The rewrite always starts from the text the user edited and
never discards it.
- It embeds new memories through an OpenAI-compatible `/v1/embeddings` endpoint,
then evicts the bank down to its capacity. Evicted memories are marked as
forgotten and kept so that the UI can still show them.
When the app generates a turn, `retrieve_memories` embeds the recent story text
and ranks the bank by cosine similarity. The highest-ranked memories become the
Memories section of the context.
Every AI call in this module is best-effort. A failure is logged to the debug
page and retried on a later turn, because the cursors advance only after a call
succeeds.
"""
import asyncio
from array import array
from collections import OrderedDict
from sqlalchemy import func, select, update
from sqlalchemy.orm import Session, defer, object_session
from . import models, tree, vectors
from .context import (
cursors,
history,
lineage,
match_cards,
story_actions,
truncate_to_last_tokens,
)
from .database import SessionLocal
from .providers import OpenAICompatibleProvider, ProviderError
from .vectors import cosine # re-exported: the ranking lives here, the maths there
MEMORY_INTERVAL = 6 # actions per memory
MEMORY_START = 12 # first memory once the adventure reaches this many actions
SUMMARY_INTERVAL = 15 # actions between Story Summary updates
MAX_MEMORIES_PER_RUN = 5 # cap catch-up work (e.g. imported adventures) per turn
MAX_EMBED_BATCH = 32
RETRIEVAL_WINDOW_TOKENS = 600 # recent story text used as the similarity query
RETRIEVAL_WINDOW_ACTIONS = 4 # ...taken from this many of the newest actions
SUMMARY_MAX_WORDS = 250
MEMORY_EXCERPT_TOKENS = 2000 # of the block, when a block is longer than this
# How much story has to sit past a block before that block is summarized.
#
# This is not the pre-SP4 holdback returning. That rule made the newest action
# invisible to the summarizer everywhere, and it existed because a retry
# rewrote an action's text in place, so a memory covering the tip could end up
# describing narration the player had already replaced. Sibling attempts and
# `forget_node` settled that, and correctness does not rest on the number
# below: a delete or an undo anywhere in the story is still repaired by
# withdrawing what the coordinate produced.
#
# This is a cost rule, and it is only about the tip. Retry and take-switching
# both refuse anything but the newest action (`takes.retry_action`,
# `takes.switch_take`), and both withdraw the derived work at the coordinate
# they change. So a memory whose block ends on the tip is the one memory a
# player can still throw away, and every retry of that turn pays for it twice:
# once to write it, once to write it again. A block closes every
# MEMORY_INTERVAL actions and a normal turn writes two of them, so that is one
# turn in three.
#
# One action of slack moves the block end out of reach of both endpoints, and
# it buys nothing back in context quality: the block that just closed is still
# in the history window in full, so a memory of it tells the model what it can
# already read. Memories earn their place once the raw text has scrolled out,
# which is never the turn the block closed.
SETTLE_SLACK = 1
# ---- The cast brief (Phase 18b) ----
# How many characters the brief names, and how much of each description it
# carries. The cast is authored content rather than generated, so it is small in
# practice; these are ceilings against a scenario with a very large cast, not a
# budget anyone is expected to reach.
MAX_CAST_MEMBERS = 8
CAST_ENTRY_CHARS = 240
SETTING_TOKENS = 300 # of `adventure.memory`, the plot essentials
# A ceiling on one memory, in words. Measured, not guessed: with only
# "1-2 plain sentences" to go on, a real model wrote 34 words for one block and
# 105 for the next, and a 105-word memory is a paragraph. Five of those are
# injected per turn at the default `memory_top_k`, so the bank's cost is set
# here. A stated number also holds the length steady between memories, which is
# the same consistency the framing rule buys for the wording.
MEMORY_MAX_WORDS = 50
# The framing rule is the larger half of this prompt, and it is worth the
# tokens. Without it the model chooses a person per call, so one bank ends up
# holding "You entered the crypt", "The player entered the crypt" and "He
# entered the crypt" for the same kind of event.
#
# The reason is stated, not just the rule. A memory is retrieved in isolation
# months of story later, and a model told why bare pronouns fail complies far
# more consistently than one handed a bare instruction.
#
# The protagonist's name lives in the user message, in the Cast, rather than
# here. That keeps this prompt constant across every adventure and every call.
MEMORY_SYSTEM_PROMPT = (
"You compress interactive-fiction story excerpts into memories. Respond with "
"1-2 plain sentences in past tense stating the concrete facts and events "
f"(names, places, items, promises, injuries), in at most {MEMORY_MAX_WORDS} "
"words. Keep the details a later scene could turn on — a name, a promise, an "
"injury, where something is — and drop the ones it could not.\n\n"
"Write in the third person. The excerpt is written in the second person: "
'"you" is the protagonist, who is named in the Cast. Refer to the '
'protagonist by that name, never as "you". If the Cast gives no name for '
'them, call them "the player". Name the other characters too rather than '
'writing "he", "she" or "they" on their own — this memory will be read on '
"its own, much later, with nothing around it to say who a pronoun meant.\n\n"
"No preamble, no commentary."
)
SUMMARY_SYSTEM_PROMPT = (
"You maintain the running summary of an interactive-fiction story. Respond "
"with only the updated summary: a single plain-prose overview of the plot "
f"so far, at most {SUMMARY_MAX_WORDS} words. Preserve important established "
"facts; compress older events harder than recent ones.\n\n"
"Write in the third person, and name the characters. Refer to the "
'protagonist by the name given in the Cast, or as "the player" if the Cast '
'gives no name. Never address them as "you".'
)
# Adventures with a post-turn task currently running (single-process app).
_running: set[int] = set()
# Strong references to tasks that are still running. The event loop holds only
# weak references, so without this set a fire-and-forget task can be garbage
# collected before it finishes.
_tasks: set[asyncio.Task] = set()
# Both factories below use the user's own key by construction. They read the
# endpoint and key from `Settings` and never from `auth.DEMO_*`, so
# summarization and embedding cannot spend the shared demo key. Their call sites
# are also skipped when `using_demo` is true.
#
# Do not change these to accept a `ProviderConfig`. `summary_model` and
# `embedding_model` are free-form user input and are not on the demo allowlist.
def summary_provider(settings: models.Settings) -> OpenAICompatibleProvider:
return OpenAICompatibleProvider(
settings.endpoint_url,
settings.api_key_plain,
settings.summary_model or settings.model,
settings.api_mode,
settings.reasoning_max_tokens,
)
def embedding_provider(settings: models.Settings) -> OpenAICompatibleProvider:
return OpenAICompatibleProvider(
settings.endpoint_url, settings.api_key_plain, settings.embedding_model
)
def set_vector(memory: models.Memory, vector: list[float] | None) -> None:
"""Store (or clear) a memory's embedding.
This function updates both columns that describe the vector. The ranking
reads `embedding_blob`, and everything else reads the `embedded` flag.
Routing every write through one function keeps the two in step, and it makes
this the only place a stored vector changes. That is why the cache below can
be invalidated here and nowhere else.
One caller cannot use this function: the bulk clear in
`routers/settings.py` that runs when the embedding model changes. It sets
the same two columns directly. See the comment there.
"""
memory.embedding_blob = None if vector is None else vectors.pack(vector)
memory.embedded = vector is not None
cached = _vector_cache.get(memory.adventure_id)
if cached is not None:
cached.pop(memory.id, None)
# ---------- The vector cache ----------
# Maps an adventure id to a dict of memory id to vector, with the
# most recently used adventure last.
#
# Turns for one adventure arrive one after another, and the bank changes little
# between them. Reading every vector on each turn fetches the same 600 KB
# repeatedly. This cache stores vectors as `array("f")`, which uses 4 bytes per
# component and matches the 6 KB the column holds. A list of Python floats would
# use eight times as much.
#
# Two rules keep the cache correct. Any code that changes a vector calls
# `set_vector`, which removes that entry. Any code that removes a memory from
# play removes it from the catalogue query below, and the next read discards
# entries that the catalogue no longer lists. Eviction, deletion, pruning, and
# an edit that clears the vector all work this way. No code path has to remember
# to invalidate the cache, which is the error this design avoids.
#
# The cache lives in the process, so it assumes one worker, which is what the
# deploy runs. With two workers, each keeps its own copy. Both stay correct
# about eviction and deletion, but a vector rewritten by one worker can remain
# stale in the other until that memory leaves the catalogue.
_vector_cache: OrderedDict[int, dict[int, array]] = OrderedDict()
VECTOR_CACHE_ADVENTURES = 8 # ~600 KB each at a 100-memory bank
def forget_cached_vectors(adventure_id: int) -> None:
"""Drops an adventure's cached vectors.
Call this only when the adventure itself is deleted. Every other case
corrects itself, as described in the comment above.
"""
_vector_cache.pop(adventure_id, None)
def _vectors_for(db: Session, adventure_id: int, ids: list[int]) -> dict[int, array]:
"""The vectors for `ids`, reading only the ones not already held."""
cached = _vector_cache.get(adventure_id)
if cached is None:
cached = _vector_cache[adventure_id] = {}
_vector_cache.move_to_end(adventure_id)
while len(_vector_cache) > VECTOR_CACHE_ADVENTURES:
_vector_cache.popitem(last=False)
wanted = set(ids)
for gone in set(cached) - wanted:
del cached[gone]
missing = [memory_id for memory_id in ids if memory_id not in cached]
if missing:
rows = db.execute(
select(models.Memory.id, models.Memory.embedding_blob)
.where(models.Memory.id.in_(missing))
).all()
for memory_id, blob in rows:
if blob:
cached[memory_id] = vectors.unpack(blob)
return cached
def forget_node(db: Session, adventure: models.Adventure, action: models.Action) -> int:
"""Withdraw what a node produced, because the node is being removed.
Call this before deleting `action`, which undo and the delete-action
endpoint both do. A memory attaches to the node its block ends on, so
finding the memories that describe a node is a lookup on `(branch_id,
depth)`. The earlier `prune_dangling_memories` instead scanned for rows
whose covered range no longer existed, so it could only detect the problem
after it occurred.
Deleting the memory is half the work. The stretch of story it covered still
sits behind the cursors. Without a rewind, those actions count as
summarized while nothing describes them, and nothing reports the problem for
the rest of the adventure. `source_start` records where that stretch began,
so the anchor moves to the node before it. That depth is valid whether or
not a node still occupies it.
The opening node is the one exception, because migration 62 placed the whole
pre-coordinate bank on it. See the comment on `lineage.ROOT_DEPTH`.
Returns the number of memories withdrawn.
"""
if action.branch_id is None or action.depth is None:
return 0 # A pre-tree row. No path contains it, so nothing refers to it.
doomed = (
db.query(models.Memory)
.filter(
models.Memory.adventure_id == adventure.id,
models.Memory.branch_id == action.branch_id,
models.Memory.depth == action.depth,
)
.all()
)
if action.depth == lineage.ROOT_DEPTH:
# Special case for the opening node, and only for memories that
# describe no stretch of story. Migration 62 placed every memory written
# before memories had coordinates at depth 0. That choice preserved
# every memory, but it also placed them all on one node, so withdrawing
# that node would delete a player's entire bank in one action.
#
# A memory with no `source_start` was either typed by the player or
# migrated. It describes no actions, so no deletion can invalidate it,
# and it stays. A memory that genuinely summarizes a block ending here is
# still withdrawn, because the text it describes is being deleted.
doomed = [m for m in doomed if m.source_start is not None]
if not doomed:
return 0
starts = [m.source_start for m in doomed if m.source_start is not None]
for memory in doomed:
db.delete(memory)
if starts:
cursors.rewind_all(adventure, action.branch_id, min(starts) - 1)
return len(doomed)
def source_block(db: Session, memory: models.Memory) -> list[models.Action]:
"""The actions a memory was written from, oldest first.
The inverse of what `_create_due_memories` recorded. `source_start` and
`source_end` are depths, and `branch_id` says which path they are depths
on — that branch's own lineage, not the adventure's current path. A memory
written before a fork must still read back from the branch it was written
on, whichever branch the adventure has since moved to.
Returns `[]` for a memory that describes no stretch of story. Those are
hand-written, or migrated from before memories had coordinates (see
`lineage.ROOT_DEPTH`), and there is no block to read.
The range may come back shorter than `MEMORY_INTERVAL`. An action inside it
can have been deleted since, and a memory whose block is now partial still
describes the actions that remain.
"""
if memory.source_start is None or memory.source_end is None:
return []
if memory.branch_id is None:
return [] # A pre-tree row: no path contains it.
branch = db.get(models.Branch, memory.branch_id)
if branch is None:
return []
path = lineage.Path(lineage.entries_of(branch))
rows = (
db.query(models.Action)
.filter(
models.Action.adventure_id == memory.adventure_id,
# This clause excludes the sibling attempts at a retried turn, so
# the block holds the one text the story used.
path.clause(models.Action),
models.Action.depth >= memory.source_start,
models.Action.depth <= memory.source_end,
)
# `id` breaks a tie on `depth`, as everywhere else that orders actions.
.order_by(models.Action.depth, models.Action.id)
# Reasoning traces are never part of an excerpt and can outweigh the
# narration on a reasoning model.
.options(defer(models.Action.reasoning))
.all()
)
return [a for a in rows if history.is_story_text(a.text)]
# ---------- The cast brief ----------
def _cast_line(name: str, entry: str, *, protagonist: bool = False) -> str:
"""One roster line: who they are, and nothing about where they stand now."""
who = f"{name} — the protagonist" if protagonist else f"{name} —"
entry = " ".join(entry.split()) # collapse newlines: this is a one-line roster
if len(entry) > CAST_ENTRY_CHARS:
entry = entry[:CAST_ENTRY_CHARS].rsplit(" ", 1)[0] + "…"
if protagonist:
return f"- {who}." if not entry else f"- {who}. {entry}"
return f"- {name} — {entry}" if entry else f"- {name}"
def cast_brief(adventure: models.Adventure, text: str) -> str:
"""Returns who appears in `text`, and what the story is about.
This is the context the summarizer never had. It was handed six actions of
second-person prose and nothing else, so the only honest memory it could
write for `You push the door open. She grabs your arm.` was "You entered a
room and she stopped you." — which, retrieved forty turns later, names
nobody.
Three rules hold this together.
**Fixed descriptions only, never live values.** It is tempting to add
`Gwen: trust 40 (wary)`. That would make the same event summarized at two
different times come out framed differently, which is the fault this whole
change exists to remove.
**The cast comes from the story cards, not from `stat_schema`.** Every NPC a
scenario defines is already turned into a story card at adventure creation
(`scenario_text.scenario_card_specs`), deduplicated against the hand-written
ones by name. Reading the cards therefore covers the schema NPCs, the
author's own cards, and an adventure with no RPG layer at all, through one
path instead of three.
**Keyword matching alone is not enough here, which is why the roster is
topped up.** The turn prompt includes a card only when its trigger words
appear, and that is right for lore: a card nobody mentioned is not relevant
to the next sentence. It is wrong for this brief. The block that most needs
a cast is exactly the one written in bare pronouns — "she grabs your arm"
matches no keyword, and the summarizer is then left guessing at precisely
the moment it was given this brief to stop guessing. So matched cards come
first, and any remaining slots are filled with the other **character**
cards. Places and items are not topped up: an unmentioned tavern is not
who "she" was.
Walking `adventure.story_cards` is a relationship load, which this module
otherwise avoids. It is affordable here for two reasons the memory bank's
own reads were not: a card is five short text columns with no vector, and
this runs once per `MEMORY_INTERVAL` actions in the background task rather
than on every turn. `build_context` already walks the same collection.
"""
lines: list[str] = []
name = adventure.persona_name.strip()
pronouns = adventure.persona_pronouns.strip()
if name or adventure.persona_desc.strip():
who = f"{name} ({pronouns})" if name and pronouns else (name or "The player")
lines.append(_cast_line(who, adventure.persona_desc.strip(), protagonist=True))
seen = {name.lower()} if name else set()
def add(card_name: str, entry: str) -> bool:
"""Adds one roster line. Returns False once the roster is full."""
card_name = (card_name or "").strip()
if card_name and card_name.lower() not in seen:
seen.add(card_name.lower())
lines.append(_cast_line(card_name, (entry or "").strip()))
return len(lines) < MAX_CAST_MEMBERS
room = True
for card in match_cards(adventure.story_cards, text):
room = add(card["name"], card["entry"])
if not room:
break
if room:
for card in adventure.story_cards:
if (card.type or "").strip().lower() != "character":
continue
if not add(card.name, card.entry):
break
parts = []
if lines:
parts.append("Cast:\n" + "\n".join(lines))
setting = adventure.memory.strip()
if setting:
parts.append("Setting:\n" + truncate_to_last_tokens(setting, SETTING_TOKENS))
return "\n\n".join(parts)
# ---------- Retrieval (runs inside the turn, before build_context) ----------
async def retrieve_memories(
adventure: models.Adventure,
settings: models.Settings,
*,
update_stats: bool,
exclude_action_id: int | None = None,
) -> dict | None:
"""Returns the memories to inject, or None when the bank is off.
The result is a dict of the form
`{"used": [{id, text, similarity, pinned}], "error": str | None}`. It is
None when the memory bank is disabled for this adventure.
Set `update_stats` to True to increment the use counters. Only real turns
should do this, not the dry runs that Insights performs.
`exclude_action_id` removes the action being retried from the similarity
query, so that a discarded attempt cannot influence which memories are
returned.
"""
if not adventure.memory_bank_enabled:
return None
if not settings.embedding_model.strip():
return {"used": [], "error": "No embedding model configured in Settings."}
db = object_session(adventure)
if db is None:
return {"used": [], "error": None}
# Select which memories are in play, and nothing else about them. This code
# used to walk `adventure.memories`, which loaded every row of the bank,
# including its vector. That cost about 31 KB per memory and about 3 MB per
# turn, which was 96% of everything a turn read. An id and a flag come to
# about eight bytes per row.
#
# The branch clause uses the whole lineage here rather than the window the
# story is read through. Retrieval exists to recall events from far back in
# the story, such as what happened forty turns ago. The full lineage stays
# affordable because memories are sparse, at roughly one per six actions, so
# even a heavily forked story returns only tens of small rows.
catalogue = db.execute(
select(models.Memory.id, models.Memory.pinned).where(
models.Memory.adventure_id == adventure.id,
lineage.path_of(db, adventure).clause(models.Memory),
models.Memory.forgotten.is_(False),
models.Memory.embedded.is_(True),
)
).all()
if not catalogue:
return {"used": [], "error": None}
recent = history.tail(adventure, RETRIEVAL_WINDOW_ACTIONS, exclude_action_id)
query = truncate_to_last_tokens(
"\n\n".join(a.text for a in recent), RETRIEVAL_WINDOW_TOKENS
)
if not query.strip():
return {"used": [], "error": None}
try:
[query_vec] = await embedding_provider(settings).embed([query])
except ProviderError as exc:
return {"used": [], "error": str(exc)}
held = _vectors_for(db, adventure.id, [memory_id for memory_id, _ in catalogue])
scored = sorted(
(
(cosine(query_vec, held[memory_id]), memory_id, pinned)
for memory_id, pinned in catalogue
if memory_id in held
),
key=lambda row: row[0],
reverse=True,
)
# Pinned memories are always used, and they count toward `top_k`, so the
# injected set stays within the budget unless the pinned memories alone
# exceed it.
top_k = max(1, settings.memory_top_k)
used = [row for row in scored if row[2]]
remaining = max(0, top_k - len(used))
used += [row for row in scored if not row[2]][:remaining]
used.sort(key=lambda row: row[0], reverse=True)
if not used:
return {"used": [], "error": None}
# Fetch the text only now, and only for the `top_k` rows that were chosen.
used_ids = [memory_id for _, memory_id, _ in used]
texts = dict(
db.execute(
select(models.Memory.id, models.Memory.text)
.where(models.Memory.id.in_(used_ids))
).all()
)
if update_stats:
# Pass `synchronize_session=False` because nothing in this request
# reads the counters back. Matching the UPDATE against loaded objects
# would require loading those objects, which is the cost this code path
# exists to avoid.
db.execute(
update(models.Memory)
.where(models.Memory.id.in_(used_ids))
.values(use_count=models.Memory.use_count + 1, last_used_at=models.utcnow())
.execution_options(synchronize_session=False)
)
return {
"used": [
{"id": memory_id, "text": texts.get(memory_id, ""),
"similarity": round(score, 4), "pinned": pinned}
for score, memory_id, pinned in used
],
"error": None,
}
# ---------- Post-turn background work ----------
def schedule_post_turn(adventure: models.Adventure) -> None:
"""Fire-and-forget summarization/embedding work after a turn is saved."""
if not (adventure.auto_summarize or adventure.memory_bank_enabled):
return
if adventure.id in _running:
return
task = asyncio.get_running_loop().create_task(run_post_turn(adventure.id))
_tasks.add(task)
task.add_done_callback(_tasks.discard)
async def run_post_turn(adventure_id: int) -> None:
if adventure_id in _running:
return
_running.add(adventure_id)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, adventure_id)
if adventure is None:
return
# Settings are per-user (Phase 8): use the adventure owner's row.
settings = (
db.query(models.Settings)
.filter(models.Settings.user_id == adventure.user_id)
.first()
)
if settings is None:
return
# This code no longer clamps the cursors. Undo can leave the story
# shorter than the mark. When the mark was a position, a value past the
# end of the list stalled the pass until the story grew back, so every
# post-turn run clamped it. That clamp introduced its own error, because
# clamping to the settled count rewound a caught-up adventure by one
# step and covered an action twice.
#
# An anchor past the tip is not an invalid value. `settled_after`
# reports that there is nothing to do, and once the story grows past the
# anchor the pass resumes where it stopped.
if adventure.auto_summarize:
await _create_due_memories(adventure, settings, db)
await _update_story_summary(adventure, settings, db)
if adventure.memory_bank_enabled and settings.embedding_model.strip():
await _embed_pending(adventure, settings, db)
_evict_over_capacity(adventure, settings, db)
finally:
db.close()
_running.discard(adventure_id)
async def summarize_block(
adventure: models.Adventure,
provider: OpenAICompatibleProvider,
block: list[models.Action],
) -> str:
"""Writes one memory from one block of story.
Both callers come through here, which is the point of the function. The
pass below writes a memory as the story reaches it; `tools/rewrite_memories`
rewrites one an older prompt produced. Assembling the prompt in two places
would mean a rewritten memory was written by a prompt that never shipped,
and nothing would report the difference.
Raises `ProviderError`, which each caller handles its own way: the pass
below leaves the cursor alone and retries next turn, and the tool leaves the
old text in place and moves on.
"""
raw = "\n\n".join(a.text for a in block)
excerpt = truncate_to_last_tokens(raw, MEMORY_EXCERPT_TOKENS)
# Match the cast against the untruncated block. The excerpt is what the
# model reads, but a character named in the part that was trimmed is still
# one the memory may have to name.
brief = cast_brief(adventure, raw)
prompt = f"Story excerpt:\n\n{excerpt}\n\nMemory:"
return await provider.complete(
MEMORY_SYSTEM_PROMPT, f"{brief}\n\n{prompt}" if brief else prompt
)
async def _create_due_memories(
adventure: models.Adventure, settings: models.Settings, db: Session
) -> None:
provider = summary_provider(settings)
for _ in range(MAX_MEMORIES_PER_RUN):
# Re-read the anchor on every pass. Committing a memory does not change
# the story, but this loop is the only code that moves the anchor, so
# both numbers must be current.
anchor = cursors.MEMORY.depth(db, adventure)
if history.count_after(adventure, anchor) < MEMORY_INTERVAL + SETTLE_SLACK:
return # No settled block of story sits past the mark. The block
# itself is still MEMORY_INTERVAL actions; the slack asks
# for story past its end. See `SETTLE_SLACK`.
if history.count(adventure) < MEMORY_START:
return # The adventure is too short to have started summarizing.
# The order of those two checks is deliberate. The usual answer is that
# no memory is due, and the first check settles that without measuring
# the length of the whole story.
block = history.after(adventure, anchor, MEMORY_INTERVAL)
if len(block) < MEMORY_INTERVAL:
return
try:
text = await summarize_block(adventure, provider, block)
except ProviderError:
return # Logged on the debug page. The cursor is unchanged, so the
# next turn retries this block.
if not text:
return
memory = models.Memory(
adventure_id=adventure.id,
text=text,
source_start=block[0].depth,
source_end=block[-1].depth,
)
# Attach the memory to the node it summarizes, so that a fork inherits
# the memories of the path it forked from and no others. Then move the
# mark to that same node. Both values record how far this pass has
# reached, and taking them from one row keeps them in step even when the
# depths have gaps.
tree.attach_memory(memory, block[-1])
db.add(memory)
cursors.MEMORY.anchor_at(adventure, block[-1])
db.commit()
async def _update_story_summary(
adventure: models.Adventure, settings: models.Settings, db: Session
) -> None:
anchor = cursors.SUMMARY.depth(db, adventure)
uncovered = history.count_after(adventure, anchor)
if uncovered < SUMMARY_INTERVAL:
return
# Where the summary stands once this run succeeds. Read this before the AI
# call rather than after it. The mark records the end of the story as this
# pass saw it, and a turn that arrives during the call must not be counted
# as read.
caught_up = history.newest(adventure)
if caught_up is None:
return
# Include the memories for the stretch that the summary has not read, which
# means every memory attached to a node past the anchor. The marks and the
# memories are now depths on one path, so no coordinate conversion remains.
# If memory creation has fallen behind, for example because the last attempt
# failed, this falls back to the raw story text.
new_events = db.execute(
select(models.Memory.text)
.where(
models.Memory.adventure_id == adventure.id,
lineage.path_of(db, adventure).clause(models.Memory),
models.Memory.depth > anchor,
)
.order_by(models.Memory.depth)
).scalars().all()
if new_events:
events_text = "\n".join(f"- {t}" for t in new_events)
else:
block = history.after(adventure, anchor, uncovered)
events_text = truncate_to_last_tokens("\n\n".join(a.text for a in block), 2000)
current = adventure.story_summary.strip()
# The summary is built from the memories, so it inherits their framing for
# free once they are named and third-person. It still gets the brief of its
# own, because the fallback above hands it raw second-person story text
# whenever memory creation has fallen behind.
brief = cast_brief(adventure, f"{current}\n\n{events_text}")
user_prompt = (
f"Current story summary:\n{current or '(none yet)'}\n\n"
f"New events since the last update:\n{events_text}\n\n"
"Updated summary:"
)
if brief:
user_prompt = f"{brief}\n\n{user_prompt}"
try:
text = await summary_provider(settings).complete(
SUMMARY_SYSTEM_PROMPT, user_prompt, max_tokens=600
)
except ProviderError:
return
if not text:
return
adventure.story_summary = text
cursors.SUMMARY.anchor_at(adventure, caught_up)
db.commit()
async def _embed_pending(
adventure: models.Adventure, settings: models.Settings, db: Session
) -> None:
# Use a query rather than walking `adventure.memories`. That walk ran on
# every turn and loaded the whole bank's vectors in order to find the few
# rows with none.
#
# Neither this query nor the eviction below applies a branch clause, and
# that is deliberate. Whether a row is embedded is a fact about the row, not
# about the path being played. Skipping a sibling branch's memories would
# only postpone the work until someone switched branches and needed them
# ranked. Capacity works the same way. The bank belongs to the adventure,
# and the memories of a branch nobody is reading are the right ones to evict
# first.
pending = (
db.query(models.Memory)
.filter(
models.Memory.adventure_id == adventure.id,
models.Memory.embedded.is_(False),
models.Memory.forgotten.is_(False),
)
.order_by(models.Memory.id)
.limit(MAX_EMBED_BATCH)
.all()
)
if not pending:
return
try:
new = await embedding_provider(settings).embed([m.text for m in pending])
except ProviderError:
return
for memory, vector in zip(pending, new):
set_vector(memory, vector)
db.commit()
def _evict_over_capacity(
adventure: models.Adventure, settings: models.Settings, db: Session
) -> None:
# The database performs both the count and the ranking, and returns neither
# the rows nor the vectors. Counting by walking `adventure.memories` fetched
# every vector in the bank on every turn, whether or not the bank was over
# capacity.
in_this_bank = (models.Memory.adventure_id == adventure.id,
models.Memory.forgotten.is_(False))
active = db.execute(
select(func.count(models.Memory.id)).where(*in_this_bank)
).scalar() or 0
overflow = active - max(1, settings.memory_bank_capacity)
if overflow <= 0:
return
# Evict the least recently used memory first, and use the use count only to
# break ties.
#
# Ordering by use count first froze the bank. A memory written on this turn
# has never been used, so once every other memory had been retrieved at
# least once, the new memory held the lowest count in the bank. The same
# post-turn run that wrote it then evicted it, one pass after embedding it.
# Use counts only increase, so the bank never recovered. An adventure kept
# whatever memories it held when the bank first filled, and every later
# memory was summarized, marked as forgotten, and never ranked.
#
# Ordering by recency avoids that. A new memory carries the newest
# timestamp, so it is the last row to be evicted rather than the first, and
# it remains until other memories are used. Demoting the use count costs
# little, because retrieving a useful memory also makes it recent. The two
# orderings differ only for memories that were used once and have not been
# retrieved since, which are the rows a full bank should evict.
doomed = db.execute(
select(models.Memory.id)
.where(*in_this_bank, models.Memory.pinned.is_(False))
.order_by(
func.coalesce(models.Memory.last_used_at, models.Memory.created_at),
models.Memory.use_count,
)
.limit(overflow)
).scalars().all()
if not doomed:
return # Every active memory is pinned, so the pins override capacity.
db.execute(
update(models.Memory)
.where(models.Memory.id.in_(doomed))
.values(forgotten=True)
.execution_options(synchronize_session=False)
)
db.commit()
# The bulk UPDATE bypassed the loaded objects, so code that still holds the
# collection would otherwise see the evicted memories as active.
db.expire(adventure, ["memories"])
backend/app/migrations.py
+1128
View File
File diff suppressed because it is too large Load Diff
backend/app/models.py
+720
View File
@@ -0,0 +1,720 @@
from datetime import datetime, timezone
from sqlalchemy import (
JSON, Boolean, Column, DateTime, Float, ForeignKey, Index, Integer, LargeBinary,
String, Table, Text, UniqueConstraint, event,
)
from sqlalchemy.orm import Mapped, Session, mapped_column, relationship
from .compression import CompressedJSON
from .database import Base
def utcnow() -> datetime:
return datetime.now(timezone.utc)
class User(Base):
"""Phase 8: optional accounts.
Three kinds of row share this table:
- The local user has a NULL email and `is_guest` set to False. Single-user
mode creates this row automatically. It owns everything that a database
from before Phase 8 contained.
- Guests have a NULL email and `is_guest` set to True. Multi-user mode
creates one on a visitor's first visit and identifies it only by the
session cookie.
- Registered users have an email. Registration upgrades a guest row in
place, so the guest's data survives without being reassigned.
"""
__tablename__ = "users"
id: Mapped[int] = mapped_column(primary_key=True)
email: Mapped[str | None] = mapped_column(String(320), unique=True, nullable=True)
password_hash: Mapped[str | None] = mapped_column(String(300), nullable=True)
is_guest: Mapped[bool] = mapped_column(Boolean, default=True)
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
last_seen_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
# Shared demo key usage (resets when the UTC date changes).
demo_turns_used: Mapped[int] = mapped_column(Integer, default=0)
demo_turns_date: Mapped[str] = mapped_column(String(10), default="")
scenario_scripts = Table(
"scenario_scripts",
Base.metadata,
Column("scenario_id", ForeignKey("scenarios.id", ondelete="CASCADE"), primary_key=True),
Column("script_id", ForeignKey("scripts.id", ondelete="CASCADE"), primary_key=True),
)
class Scenario(Base):
__tablename__ = "scenarios"
id: Mapped[int] = mapped_column(primary_key=True)
# NULL owner + is_public = seeded demo content, readable by everyone.
user_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), nullable=True
)
is_public: Mapped[bool] = mapped_column(Boolean, default=False)
title: Mapped[str] = mapped_column(String(200), default="Untitled Scenario")
description: Mapped[str] = mapped_column(Text, default="")
prompt: Mapped[str] = mapped_column(Text, default="")
# Plot components (AI Dungeon terminology; `memory` == Plot Essentials)
memory: Mapped[str] = mapped_column(Text, default="")
authors_note: Mapped[str] = mapped_column(Text, default="")
ai_instructions: Mapped[str] = mapped_column(Text, default="")
tags: Mapped[str] = mapped_column(String(500), default="")
# Cover art. The value is either an "https://" URL or an inline
# "data:image/...;base64,..." URI. The editor downscales uploads before
# storing them. An empty value tells the UI to fall back to an emoji sigil
# or to generated art. The image is stored in the row rather than on disk,
# because Render's free tier provides no persistent volume. Storing it here
# also keeps export bundles self-contained.
image: Mapped[str] = mapped_column(Text, default="")
# A single emoji or glyph, used when `image` is empty. This is a separate
# column because the value is a character rather than a location, so
# nothing needs to fetch or cache it.
icon: Mapped[str] = mapped_column(String(16), default="")
# Phase 12: the RPG world-state template. It holds stat definitions, which
# include bands and rules, and milestones. A NULL or empty value means the
# scenario has no RPG layer.
stat_schema: Mapped[dict | None] = mapped_column(JSON, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
updated_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow, onupdate=utcnow)
story_cards: Mapped[list["StoryCard"]] = relationship(
back_populates="scenario", cascade="all, delete-orphan"
)
adventures: Mapped[list["Adventure"]] = relationship(back_populates="scenario")
scripts: Mapped[list["Script"]] = relationship(secondary=scenario_scripts)
class Adventure(Base):
__tablename__ = "adventures"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), nullable=True
)
scenario_id: Mapped[int | None] = mapped_column(
ForeignKey("scenarios.id", ondelete="SET NULL"), nullable=True
)
title: Mapped[str] = mapped_column(String(200), default="Untitled Adventure")
memory: Mapped[str] = mapped_column(Text, default="")
authors_note: Mapped[str] = mapped_column(Text, default="")
ai_instructions: Mapped[str] = mapped_column(Text, default="")
story_summary: Mapped[str] = mapped_column(Text, default="")
# Phase 18: who the player is playing as. The AI never writes these — they
# are user-only, which is what lets them sit in the cached system block
# rather than below the history with the values that change. An empty
# `persona_name` means the adventure has no persona, and every read below
# falls back to the wording used before this existed.
#
# These are adventure columns rather than part of the scenario's
# `stat_schema`, for two reasons. An adventure with no RPG layer still has a
# protagonist, and `worldstate.schema._initials` treats every dict inside a
# stat section as a stat definition, so a persona placed there would be
# instantiated, rendered and given an `initial` value as though it were one.
persona_name: Mapped[str] = mapped_column(String(80), default="")
persona_pronouns: Mapped[str] = mapped_column(String(40), default="")
persona_desc: Mapped[str] = mapped_column(Text, default="")
script_state: Mapped[dict] = mapped_column(JSON, default=dict)
# Phase 12: live RPG world state (world/player/npc stats + milestones),
# instantiated from the scenario's stat_schema. Empty when there's no RPG layer.
world_state: Mapped[dict] = mapped_column(JSON, default=dict)
# The ${Placeholder} answers collected when this adventure was started, kept
# so "Update from scenario" can re-fill freshly copied scenario text with the
# same values. NULL for adventures created before this column existed.
placeholders: Mapped[dict | None] = mapped_column(JSON, nullable=True)
# Phase 6: opt-in per adventure (extra AI calls)
auto_summarize: Mapped[bool] = mapped_column(Boolean, default=False)
memory_bank_enabled: Mapped[bool] = mapped_column(Boolean, default=False)
# Phase 14, SP3: how far the memory pass and the summary pass have read,
# each as a coordinate. Each pair
# holds the branch and depth of the last action that pass covered. A
# position moves when an action in front of it is deleted, so the mark
# silently starts covering an action it never read. A depth is a coordinate
# along a path, so deleting an action does not move it. NO_DEPTH, which is
# -1, means that nothing is covered yet, so the first block needs no special
# case. These are plain integers rather than foreign keys, for the reason
# given on `head_branch_id` below. See `context/cursors.py`.
memory_cursor_branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
memory_cursor_depth: Mapped[int] = mapped_column(Integer, default=-1)
summary_cursor_branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
summary_cursor_depth: Mapped[int] = mapped_column(Integer, default=-1)
# Phase 14: where the story is being played. `head_branch_id` names the
# branch, and `head_depth` gives the depth of its newest node.
#
# `head_branch_id` is deliberately not a ForeignKey. `branches.adventure_id`
# already points from branches to adventures, so a constraint in this
# direction would make the two tables a cycle that `create_all` cannot
# order. The usual fix is `use_alter`, which needs an ALTER statement that
# SQLite does not provide. The column caches a pointer, and
# `tree.head_branch` treats a head that names a missing branch as a bug to
# recover from rather than a state to preserve.
head_branch_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
# The depth of the tip, so the next node is always head_depth + 1.
# NO_DEPTH (-1) for an adventure with no actions yet.
head_depth: Mapped[int] = mapped_column(Integer, default=-1)
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
updated_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow, onupdate=utcnow)
scenario: Mapped[Scenario | None] = relationship(back_populates="adventures")
story_cards: Mapped[list["StoryCard"]] = relationship(
back_populates="adventure", cascade="all, delete-orphan"
)
# Every action in the adventure, across all branches. This collection is
# the tree, not the story being played. Ordering it by depth does not make
# it a story, because a path is a selection out of the tree. Code that shows
# a story to a reader goes through `context.history`, which applies the
# branch clause. This relationship exists for ownership and for the
# delete-orphan cascade.
actions: Mapped[list["Action"]] = relationship(
back_populates="adventure",
cascade="all, delete-orphan",
order_by="Action.id",
)
scripts: Mapped[list["AdventureScript"]] = relationship(
back_populates="adventure",
cascade="all, delete-orphan",
order_by="AdventureScript.position",
)
memories: Mapped[list["Memory"]] = relationship(
back_populates="adventure",
cascade="all, delete-orphan",
order_by="Memory.id",
)
class Branch(Base):
"""Phase 14: one path through an adventure's story tree.
A branch does not own a copy of the story. It holds the nodes played on it,
and it inherits everything before its fork point from its ancestors. Reading
branch C means reading C's nodes, then B's nodes up to the depth where C
forked, then A's nodes up to the depth where B forked. The `lineage` column
records that list, so a read becomes one OR clause per entry instead of a
walk up parent pointers.
Until forking ships, each adventure has one root branch and every node
belongs to it. This is not a partly migrated state. A linear story is a tree
with one branch, which is why writing these columns changes nothing that a
reader can observe.
This class defines no ORM relationships, by design. `actions.branch_id` and
`memories.branch_id` both use ON DELETE CASCADE, so the database removes a
deleted branch's nodes. A relationship would make SQLAlchemy load those rows
first, and loading every action of a branch is what the windowed reads exist
to avoid.
"""
__tablename__ = "branches"
id: Mapped[int] = mapped_column(primary_key=True)
adventure_id: Mapped[int] = mapped_column(ForeignKey("adventures.id", ondelete="CASCADE"))
# NULL on a root branch.
parent_branch_id: Mapped[int | None] = mapped_column(
ForeignKey("branches.id", ondelete="CASCADE"), nullable=True
)
# The depth at which this branch left its parent. The fork records this
# value, and no code infers it later. Deriving it from the first depth where
# two branches' nodes differ would produce a wrong answer whenever an
# attempt repeats its parent's text.
fork_depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
# The ancestry, newest first, as [[branch_id, max_depth], ...]. A NULL
# `max_depth` means the entry extends to the tip of that branch. Any other
# value is the fork depth of the branch below it, inclusive. The fork
# computes this list once from the parent's lineage plus one entry, so no
# read has to reconstruct it.
lineage: Mapped[list] = mapped_column(JSON, default=list)
# The name the player gave this line of the story, or NULL if no one named
# it. The column stores NULL rather than a generated name such as
# "branch 4", because it records what the player chose rather than what the
# app derived. A stored default would also become wrong as soon as an
# earlier branch is deleted and the ordinals shift. The UI labels an unnamed
# branch by its fork depth, which deleting a branch does not change.
name: Mapped[str | None] = mapped_column(String(80), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
class Memory(Base):
"""Phase 6: an auto-summarized (or hand-written) fact about the adventure.
The vector lives in `embedding_blob` as packed float32 (see vectors.py).
NULL until embedded, which also marks it for backfill when an embedding
model becomes available.
Cosine ranking runs in Python, so the vectors travel over the wire. Measure
that cost in bytes rather than in rows. A few hundred vectors stored as JSON
come to about 10 MB, fetched again on every turn. Size any new column by the
bytes it adds, not by the number of rows.
"""
__tablename__ = "memories"
id: Mapped[int] = mapped_column(primary_key=True)
adventure_id: Mapped[int] = mapped_column(ForeignKey("adventures.id", ondelete="CASCADE"))
text: Mapped[str] = mapped_column(Text, default="")
# The vector, stored as little-endian float32. This column is deferred
# because it is wider than the rest of the row combined and only one code
# path reads it. Code that loads memories in bulk, such as the Memories
# drawer, eviction, and the embed queue, must select the columns it needs
# instead of loading whole entities.
embedding_blob: Mapped[bytes | None] = mapped_column(
LargeBinary, nullable=True, deferred=True
)
# The stretch of story this memory summarizes, given as depths on
# `branch_id`. Both are NULL for a hand-written memory, which summarizes no
# actions. `source_end` is the depth of the node the memory
# attaches to, and `depth` below mirrors it. `source_start` is where the
# stretch begins, which is where the summarizer resumes if the memory is
# withdrawn.
source_start: Mapped[int | None] = mapped_column(Integer, nullable=True)
source_end: Mapped[int | None] = mapped_column(Integer, nullable=True)
# Phase 14: the node that produced this memory, meaning the last action the
# memory summarizes. Derived data attaches to the node it came from, which
# is what makes forking cheap. Memories on a shared ancestor are shared
# automatically, and a memory that covers part of branch B is not visible
# from any path that does not go through B.
#
# Every memory has a coordinate, including a hand-written one, which takes
# the head as of the moment it was written (SP7). A NULL depth used to mean
# that the memory belonged to the adventure rather than to a path. A fork
# cannot cap a NULL, so such a memory followed the reader onto branches
# whose story it did not describe.
branch_id: Mapped[int | None] = mapped_column(
ForeignKey("branches.id", ondelete="CASCADE"), nullable=True
)
depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
# Whether `embedding_blob` is set. `memorybank.set_vector` keeps this column
# current. Readers need only the yes-or-no answer, and fetching six
# kilobytes of vector to get it is too expensive.
embedded: Mapped[bool] = mapped_column(Boolean, default=False)
pinned: Mapped[bool] = mapped_column(Boolean, default=False)
forgotten: Mapped[bool] = mapped_column(Boolean, default=False) # evicted, kept for UI
use_count: Mapped[int] = mapped_column(Integer, default=0)
last_used_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
adventure: Mapped[Adventure] = relationship(back_populates="memories")
class StoryCard(Base):
"""Owned by either a scenario or an adventure (exactly one set)."""
__tablename__ = "story_cards"
id: Mapped[int] = mapped_column(primary_key=True)
scenario_id: Mapped[int | None] = mapped_column(
ForeignKey("scenarios.id", ondelete="CASCADE"), nullable=True
)
adventure_id: Mapped[int | None] = mapped_column(
ForeignKey("adventures.id", ondelete="CASCADE"), nullable=True
)
type: Mapped[str] = mapped_column(String(100), default="")
name: Mapped[str] = mapped_column(String(200), default="")
keys: Mapped[str] = mapped_column(Text, default="") # comma-separated triggers
entry: Mapped[str] = mapped_column(Text, default="")
notes: Mapped[str] = mapped_column(Text, default="")
# Set on adventure copies only. It records which piece of the scenario the
# card came from, as either "card:<scenario_card_id>" or "npc:<npc_key>".
# The "Update from scenario" action refreshes or removes exactly these
# cards. A NULL value means the player wrote the card, so the update leaves
# it alone, or that the copy predates this column, in which case the update
# matches it by name and then sets this value.
source_ref: Mapped[str | None] = mapped_column(String(64), nullable=True)
scenario: Mapped[Scenario | None] = relationship(back_populates="story_cards")
adventure: Mapped[Adventure | None] = relationship(back_populates="story_cards")
def _change_label(parts: list[str]) -> str:
"""Names a world-state path for the inline turn summary.
`npc.gwen.trust` becomes "gwen trust". Every other shape uses its last
segment, so `player.hp` becomes "hp".
"""
if len(parts) == 3 and parts[0] == "npc":
return f"{parts[1]} {parts[2]}"
return parts[-1] if parts else ""
class Action(Base):
__tablename__ = "actions"
# Phase 14: every story read selects one branch up to one depth, then
# another branch up to another depth, and so on. The pair (branch_id, depth)
# is the index those clauses need.
__table_args__ = (Index("ix_actions_branch_depth", "branch_id", "depth"),)
id: Mapped[int] = mapped_column(primary_key=True)
adventure_id: Mapped[int] = mapped_column(ForeignKey("adventures.id", ondelete="CASCADE"))
# Phase 14: the node's place in the tree. `depth` is a position along one
# path rather than a global turn number. Node A4 and node B4 are
# alternatives, not duplicates.
#
# Both columns are nullable because ALTER TABLE cannot add a NOT NULL column
# without a default, and no default makes sense for a branch. The migration
# fills these columns for existing rows, and `tree.place_action` fills them
# for new rows. From SP2 onward, a NULL `branch_id` marks a row that no read
# can see.
branch_id: Mapped[int | None] = mapped_column(
ForeignKey("branches.id", ondelete="CASCADE"), nullable=True
)
depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
# Phase 14, SP9: the node that this node was played after. It names the take
# that was live when this row was written, not whatever sits at depth - 1
# now.
#
# This column answers one question: which takes belong to the same turn. A
# coordinate cannot answer it. A take that is forked onto its own branch
# leaves the coordinate that its siblings still occupy, so the pager would
# show it as 1/1 next to their 1/3. Forking a branch does not change a
# node's parent.
#
# Code reads this column only to group takes, using one indexed lookup
# rather than a walk. Paths still resolve through `lineage`, which is why
# adding this column required no change to any read of the story.
#
# The value is NULL on a root node, and on pre-SP9 rows that the migration
# could not place. For those rows, `attempts.group` falls back to the
# coordinate, which is how they were written.
parent_id: Mapped[int | None] = mapped_column(
ForeignKey("actions.id", ondelete="SET NULL"), nullable=True, index=True
)
# Phase 14, SP4: whether this node is the one the story uses at its
# coordinate. Retry no longer rewrites a row. It writes a sibling at the
# same branch and depth, so one coordinate can hold several attempts while
# exactly one of them is on the path. `lineage.Path.clause` is the only
# place that reads this column, for the same reason it is the only place
# that knows about branches. If a discarded attempt reaches a read, the page
# renders the same turn twice.
#
# A node with no siblings is live, so the default is True and every pre-SP4
# row is already correct. The migration does not need to visit them.
live: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
type: Mapped[str] = mapped_column(String(20)) # start|do|say|story|continue|ai
text: Mapped[str] = mapped_column(Text, default="")
# Reasoning-model "thinking" that preceded the text (AI actions only).
reasoning: Mapped[str | None] = mapped_column(Text, nullable=True)
# The full assembled prompt for this turn, used by the Insights viewer.
# This is the largest column in the database. It averages 163 KB per row in
# production and 232 KB on the longest adventure, and it accounts for 89% of
# everything stored. Only one endpoint reads it, one action at a time.
#
# The column is expensive in two ways, so it has two protections.
# `deferred=True` protects reads, because SQLAlchemy loads the column only
# when code touches the attribute. A page load therefore costs nothing. Code
# that reads actions in bulk must not touch this attribute, which is why
# `world_delta` below exists. `CompressedJSON` protects storage, because
# this column determines when the free tier's 512 MB limit is reached. The
# attribute behaves like a plain dict in both cases. See compression.py.
context_snapshot: Mapped[dict | None] = mapped_column(
CompressedJSON, nullable=True, deferred=True
)
# The small slice of the snapshot that IS needed in bulk: this turn's RPG
# state changes, for the inline chips under an AI message (world_changes)
# and for re-attaching the emit block when replaying history to the model.
# Mirrors the active variant, same as text/reasoning/context_snapshot.
world_delta: Mapped[dict | None] = mapped_column(JSON, nullable=True)
# Phase 14, SP4: the shared script state and the RPG world state as they
# stood after this node was played. These columns record the node's outcome
# rather than its starting position.
#
# Two operations need this outcome, and neither can use a snapshot taken
# before the turn. Switching between siblings must restore the state that
# the chosen attempt produced, and the attempts differ in exactly that.
# Rolling back to before a turn means restoring the state that the preceding
# node left behind, which is one lookup along the path.
#
# The value is NULL on pre-SP4 rows for which the migration could not derive
# one. Every caller tolerates that. A missing snapshot means that the caller
# leaves the live state unchanged. It never means reset the state.
#
# These columns are deferred, because code reads them only for the single
# node being switched to, undone, or retried past.
state_after: Mapped[dict | None] = mapped_column(
JSON, nullable=True, deferred=True
)
world_state_after: Mapped[dict | None] = mapped_column(
JSON, nullable=True, deferred=True
)
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
adventure: Mapped[Adventure] = relationship(back_populates="actions")
@property
def world_changes(self) -> list[dict]:
"""Compact per-turn RPG state changes (Phase 12), for the inline summary
under an AI message. Labels are path-based (no schema needed):
`npc.gwen.trust` -> "gwen trust".
The summary reports refused changes as well as accepted ones. A stat the
engine clamped carries `clamped`, and a stat it refused outright becomes
a `rejected` entry carrying the reason. Reporting only the accepted
changes made a clamp indistinguishable from a change that never
happened: a value the model pushed past its ceiling came back as a
delta of 0 and rendered as an ordinary chip, so a refused update read on
screen as an applied one.
Reads `world_delta`, never `context_snapshot`. This runs for every
action in a list response, and touching the deferred snapshot here would
drag the entire prompt archive out of the database."""
wd = self.world_delta if isinstance(self.world_delta, dict) else None
if wd is None:
return []
clamped_paths = {
str(e.get("path", "")) for e in (wd.get("clamped") or []) if isinstance(e, dict)
}
out: list[dict] = []
for entry in wd.get("applied") or []:
path = str(entry.get("path", ""))
parts = path.split(".")
section, name = parts[0], parts[-1]
if section == "flags":
out.append({"kind": "flag", "label": name, "on": bool(entry.get("new"))})
elif section == "milestones":
out.append({"kind": "milestone", "label": name})
else:
old, new = entry.get("old"), entry.get("new")
delta = new - old if isinstance(old, (int, float)) and isinstance(new, (int, float)) else None
chip = {
"kind": "stat",
"label": _change_label(parts),
"delta": delta,
"value": new,
"clamped": path in clamped_paths,
}
# Carried only when the engine wrote one. It is empty for every
# accepted change, and a key per chip per action is paid on
# every page load.
if entry.get("fix"):
chip["fix"] = str(entry["fix"])
out.append(chip)
for entry in wd.get("rejected") or []:
if not isinstance(entry, dict):
continue
parts = str(entry.get("path", "")).split(".")
chip = {
"kind": "rejected",
"label": _change_label(parts),
"reason": str(entry.get("reason", "")),
}
if entry.get("fix"):
chip["fix"] = str(entry["fix"])
out.append(chip)
return out
class Script(Base):
__tablename__ = "scripts"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), nullable=True
)
name: Mapped[str] = mapped_column(String(200), default="Untitled Script")
description: Mapped[str] = mapped_column(Text, default="")
library_js: Mapped[str] = mapped_column(Text, default="")
input_js: Mapped[str] = mapped_column(Text, default="")
context_js: Mapped[str] = mapped_column(Text, default="")
output_js: Mapped[str] = mapped_column(Text, default="")
created_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow)
updated_at: Mapped[datetime] = mapped_column(DateTime, default=utcnow, onupdate=utcnow)
class AdventureScript(Base):
"""A script copied into an adventure at creation, so library edits don't
change running adventures unless the player explicitly re-syncs it from
`source_script_id`. `state` lives on Adventure.script_state (one shared
state per adventure, as in AI Dungeon)."""
__tablename__ = "adventure_scripts"
id: Mapped[int] = mapped_column(primary_key=True)
adventure_id: Mapped[int] = mapped_column(ForeignKey("adventures.id", ondelete="CASCADE"))
# The library Script that this copy was made from, which lets the player
# re-sync it on demand. The value is NULL for legacy copies that predate
# this column, and for demo-derived copies whose source the player does not
# own. Those copies fall back to matching by name, or cannot be synced.
source_script_id: Mapped[int | None] = mapped_column(
ForeignKey("scripts.id", ondelete="SET NULL"), nullable=True
)
position: Mapped[int] = mapped_column(Integer, default=0)
enabled: Mapped[bool] = mapped_column(Boolean, default=True)
name: Mapped[str] = mapped_column(String(200), default="Untitled Script")
description: Mapped[str] = mapped_column(Text, default="")
library_js: Mapped[str] = mapped_column(Text, default="")
input_js: Mapped[str] = mapped_column(Text, default="")
context_js: Mapped[str] = mapped_column(Text, default="")
output_js: Mapped[str] = mapped_column(Text, default="")
adventure: Mapped[Adventure] = relationship(back_populates="scripts")
class Settings(Base):
__tablename__ = "settings"
id: Mapped[int] = mapped_column(primary_key=True)
# Phase 8: one row per user (pre-Phase-8 DBs had a single id=1 row, which
# the migration assigns to the local user).
user_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), nullable=True, unique=True
)
endpoint_url: Mapped[str] = mapped_column(String(500), default="http://localhost:11434/v1")
# Encrypted at rest with Fernet, which produces a value that starts with
# "enc:". See security.py. To read the key, use `api_key_plain`.
api_key: Mapped[str] = mapped_column(String(500), default="")
model: Mapped[str] = mapped_column(String(200), default="")
api_mode: Mapped[str] = mapped_column(String(20), default="chat") # chat|completion
temperature: Mapped[float] = mapped_column(Float, default=0.8)
# 800 leaves room for a full scene; 400 tended to truncate mid-paragraph
# and left reasoning models with nothing after their thinking.
max_output_tokens: Mapped[int] = mapped_column(Integer, default=800)
# Separate thinking budget for reasoning models (OpenRouter-style
# `reasoning: {max_tokens}`); 0 = param not sent, -1 = reasoning explicitly
# off (`reasoning: {effort: none}`). Added on top of
# max_output_tokens so story output keeps its full budget.
reasoning_max_tokens: Mapped[int] = mapped_column(Integer, default=0)
context_token_budget: Mapped[int] = mapped_column(Integer, default=16384)
narrator_prompt: Mapped[str] = mapped_column(
Text,
default=(
"You are a masterful storyteller continuing an interactive adventure. "
"Continue the story naturally in second person, staying consistent with "
"everything established so far. Write vivid prose. Never speak for the "
"player or break character. Do not conclude the story; always leave room "
"for the player's next action."
),
)
# Phase 6: auto-summarization + memory bank
summary_model: Mapped[str] = mapped_column(String(200), default="") # "" = main model
embedding_model: Mapped[str] = mapped_column(String(200), default="") # "" = bank disabled
# This was 200. It was lowered mainly to improve retrieval quality. Ranking
# 200 memories to choose 5 selects from a large amount of noise, and the
# oldest memories describe a part of the story that the player has left
# behind. Cheaper reads are a secondary benefit rather than the reason.
memory_bank_capacity: Mapped[int] = mapped_column(Integer, default=80)
memory_top_k: Mapped[int] = mapped_column(Integer, default=5)
@property
def has_api_key(self) -> bool:
return bool(self.api_key)
@property
def api_key_plain(self) -> str:
from . import security # local import: models is imported before security
return security.decrypt_secret(self.api_key)
# ---------- Visit analytics (see analytics.py) ----------
# Two intentionally simple tables. Neither can hold text that a player wrote,
# and neither can be joined back to a `users` row, because the visitor column
# holds an HMAC and has no foreign key. When guest cleanup deletes an account,
# the history that account contributed remains intact and anonymous.
class AnalyticsDaily(Base):
"""One counter: how many times `label` happened within `metric` on `day`.
The table stores a generic triple of metric, label, and hits rather than one
column per statistic. Measuring something new therefore costs a constant
rather than a migration. The only writer is an UPSERT that runs from a
buffer. See `analytics.flush`.
"""
__tablename__ = "analytics_daily"
id: Mapped[int] = mapped_column(primary_key=True)
day: Mapped[str] = mapped_column(String(10), index=True) # YYYY-MM-DD, UTC
metric: Mapped[str] = mapped_column(String(32))
label: Mapped[str] = mapped_column(String(80), default="")
hits: Mapped[int] = mapped_column(Integer, default=0)
# The upsert target: one row per bucket per day, created or incremented.
__table_args__ = (
UniqueConstraint("day", "metric", "label", name="uq_analytics_daily_bucket"),
)
class AnalyticsVisitorDay(Base):
"""One visitor, one day, and which funnel steps they reached on it.
This table exists so that the funnel counts people rather than clicks. A
player who starts six adventures counts as one person who started an
adventure. `is_new` is set when the visitor has no earlier row, which is why
the visitor column also has an index of its own.
"""
__tablename__ = "analytics_visitor_days"
id: Mapped[int] = mapped_column(primary_key=True)
day: Mapped[str] = mapped_column(String(10))
# HMAC of the user id under the app secret; not reversible, not a key.
visitor: Mapped[str] = mapped_column(String(32))
is_new: Mapped[bool] = mapped_column(Boolean, default=False)
opened: Mapped[bool] = mapped_column(Boolean, default=False)
created: Mapped[bool] = mapped_column(Boolean, default=False)
played: Mapped[bool] = mapped_column(Boolean, default=False)
signed_up: Mapped[bool] = mapped_column(Boolean, default=False)
__table_args__ = (
UniqueConstraint("day", "visitor", name="uq_analytics_visitor_day"),
Index("ix_analytics_visitor", "visitor"),
)
class AccessEvent(Base):
"""One sign-in, registration, failed attempt, or session first-seen.
This table is the counterpart to the two above, and it is kept separate from
them on purpose. It identifies people by design, recording address, email,
and device. Keeping it in its own table and its own module means that the
structure enforces the anonymity of the counters rather than a convention.
`user_id` is a plain integer with no foreign key. An access log that
disappeared when the account did would not serve its purpose, and guest
cleanup deletes accounts on a schedule. `who` and `is_guest` are snapshots
for the same reason, so a row still reads correctly after the account is
gone.
"""
__tablename__ = "access_events"
id: Mapped[int] = mapped_column(primary_key=True)
at: Mapped[datetime] = mapped_column(DateTime, default=utcnow, index=True)
# session | login | register | login_failed
kind: Mapped[str] = mapped_column(String(16))
user_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
# The email for a registered account, or a label such as "Guest #12"
# otherwise. For a failed sign-in, this holds the address that was tried,
# which is the reason the row exists.
who: Mapped[str] = mapped_column(String(320), default="")
is_guest: Mapped[bool] = mapped_column(Boolean, default=False)
ip: Mapped[str] = mapped_column(String(45), default="") # 45 = max IPv6
country: Mapped[str] = mapped_column(String(16), default="")
device: Mapped[str] = mapped_column(String(16), default="")
user_agent: Mapped[str] = mapped_column(String(200), default="")
# Phase 14: the fallback under `tree.place_action`.
#
# Since SP2, reads filter on `branch_id` and `depth`. A node written without
# them is invisible to every page, every context build, and every memory pass.
# The failure is silent, because nothing raises an error. Every current writer
# places its nodes explicitly, but relying on that would also mean relying on
# every fixture, script, and test written from now on. The session therefore
# enforces the rule as rows travel to the database.
#
# This listener is registered here rather than in tree.py so that importing the
# models is enough to enable it. The invariant belongs to the rows, not to the
# module that usually writes them. The import sits inside the callback because
# tree.py imports this module.
@event.listens_for(Session, "before_flush")
def _place_new_nodes_on_the_tree(session, flush_context, instances):
from . import tree
tree.place_new_nodes(session)
backend/app/netguard.py
+52
View File
@@ -0,0 +1,52 @@
"""SSRF guard for the one place the server makes an outbound request to a
user-supplied address: the BYOK `endpoint_url` (connection test + turns/chat).
Without this guard, a hosted user could point `endpoint_url` at an internal
service or at the cloud metadata endpoint, 169.254.169.254, and have the server
fetch it. The connection test even returns part of the response. The guard
therefore refuses any URL that resolves to a non-public address.
The guard does nothing in local mode. A local install talking to
http://localhost:11434, which is Ollama, is the intended case. The guard applies
only to a hosted, multi-user deployment, where the endpoint comes from an
untrusted visitor.
"""
import ipaddress
import socket
from urllib.parse import urlparse
from . import auth
def endpoint_block_reason(url: str) -> str | None:
"""A human-readable reason this URL must NOT be fetched server-side, or None
if it's allowed. Resolves the host and rejects it if any resulting address
is non-public (private, loopback, link-local/metadata, reserved, …).
Checking at request time (not just on save) is deliberate: it resists a DNS
record that flips to a private IP after the value was stored.
"""
if not auth.MULTI_USER:
return None
parsed = urlparse(url)
if parsed.scheme not in ("http", "https"):
return "the endpoint URL must start with http:// or https://"
host = parsed.hostname
if not host:
return "the endpoint URL has no host"
port = parsed.port or (443 if parsed.scheme == "https" else 80)
try:
infos = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
except socket.gaierror:
return "the endpoint host could not be resolved"
for info in infos:
try:
ip = ipaddress.ip_address(info[4][0])
except ValueError:
return "the endpoint host resolved to an unrecognized address"
# is_global is the strict allowlist: private/loopback/link-local/CGNAT
# all report False, so this one check covers the metadata IP too.
if not ip.is_global or ip.is_multicast or ip.is_reserved:
return "the endpoint URL resolves to a non-public address"
return None
backend/app/providers/__init__.py
+4
View File
@@ -0,0 +1,4 @@
from .base import PromptParts, Provider, ProviderError
from .openai_compatible import OpenAICompatibleProvider
__all__ = ["PromptParts", "Provider", "ProviderError", "OpenAICompatibleProvider"]
backend/app/providers/base.py
+34
View File
@@ -0,0 +1,34 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import AsyncIterator
@dataclass
class PromptParts:
"""Assembled context, provider-agnostic. Providers map this to their wire format."""
system: str # narrator prompt + AI instructions + memory
story: str # the story text so far (already token-budgeted)
class ProviderError(Exception):
"""User-presentable provider failure (connection refused, bad key, model not found…)."""
class Provider(ABC):
# The endpoint's own token accounting for the most recent call, when it
# reported any. It notably carries `prompt_tokens_details.cached_tokens`,
# which is the only direct measure of whether the prompt prefix is being
# cached. A caller reads it after the call it made, and one provider is built
# per request, so nothing races.
last_usage: dict | None = None
@abstractmethod
def generate(
self,
parts: PromptParts,
*,
temperature: float,
max_tokens: int,
) -> AsyncIterator[tuple[str, str]]:
"""Yield ("text" | "reasoning", chunk) pairs. Raises ProviderError on failure."""
backend/app/providers/openai_compatible.py
+423
View File
@@ -0,0 +1,423 @@
import json
from typing import AsyncIterator
import httpx
from .. import debuglog, netguard
from .base import PromptParts, Provider, ProviderError
# Appended after the story text in chat mode, so a chat-tuned model continues
# the prose rather than replying conversationally.
CHAT_CONTINUE_HINT = "\n\n[Continue the story directly. Output only story text.]"
# OpenRouter serves one model from whichever upstream is available, and every
# upstream holds its own prompt cache, so a request routed somewhere new starts
# with a cold cache however stable the prompt is. Naming a preferred upstream
# makes routing deterministic, which is what allows a cache hit at all.
#
# `allow_fallbacks` stays at its default of true on purpose, because this is a
# preference rather than a restriction. If the named upstream is down, the
# request still goes elsewhere and only misses the cache, which is the behavior
# without this setting.
#
# This is a list rather than a value derived from the model slug. The vendor half
# of a slug is usually the provider slug, such as "deepseek/..." mapping to
# "deepseek", which was verified against /api/v1/providers, but not reliably.
# Google's models are served by "google-ai-studio" and "google-vertex", and there
# is no "google". Look a vendor up on the model's Providers tab before adding it
# here. A slug that does not exist is a routing preference that, at best, does
# nothing.
_OPENROUTER_HOST = "openrouter.ai"
_PREFERRED_UPSTREAM = {"deepseek": "deepseek"}
# Completion endpoints have no roles, so a chat has to be flattened into one
# labeled transcript that ends on "Assistant:" for the model to continue.
_ROLE_LABELS = {"system": "System", "user": "User", "assistant": "Assistant"}
def flatten_messages(messages: list[dict]) -> str:
turns = "\n\n".join(
f"{_ROLE_LABELS.get(m['role'], m['role'])}: {m['content']}" for m in messages
)
return f"{turns}\n\nAssistant:"
class OpenAICompatibleProvider(Provider):
"""Adapter for any /v1-style endpoint.
This covers Ollama, LM Studio, OpenAI, OpenRouter, vLLM, and Groq, among
others.
"""
def __init__(
self,
endpoint_url: str,
api_key: str,
model: str,
api_mode: str = "chat",
reasoning_max_tokens: int = 0,
):
self.base_url = endpoint_url.rstrip("/")
self.api_key = api_key
self.model = model
self.api_mode = api_mode # Either "chat" or "completion".
# The thinking budget for reasoning models, on top of `max_tokens`. A
# value of 0 means the `reasoning` parameter is not sent, because an
# endpoint that does not know the field may reject it. A negative value
# asks the endpoint to turn reasoning off.
self.reasoning_max_tokens = reasoning_max_tokens
# The token accounting from the last call, when the endpoint reported
# any. It holds the prompt and completion counts, plus, on OpenRouter,
# `prompt_tokens_details.cached_tokens`, which is the number of prompt
# tokens read from cache rather than billed in full. Every request method
# writes it, so a caller reads it after the call it made. One provider is
# built per request.
self.last_usage: dict | None = None
def _headers(self) -> dict:
headers = {"Content-Type": "application/json"}
if self.api_key:
headers["Authorization"] = f"Bearer {self.api_key}"
return headers
def _apply_reasoning_budget(self, body: dict) -> None:
"""Gives reasoning models their own thinking budget, in the OpenRouter style.
The method raises `max_tokens`, so the output keeps its full budget.
A negative budget does the opposite. It sends `effort: "none"` to turn
reasoning off on a model that reasons by default, such as DeepSeek V4
Flash. That differs from `exclude: true`, which still reasons and still
bills for it while hiding the trace. Zero still means send nothing, so an
endpoint that rejects unknown fields, such as Ollama, keeps working.
"""
if self.api_mode != "chat":
return
if self.reasoning_max_tokens < 0:
body["reasoning"] = {"effort": "none"}
elif self.reasoning_max_tokens > 0:
body["reasoning"] = {"max_tokens": self.reasoning_max_tokens}
body["max_tokens"] += self.reasoning_max_tokens
def _apply_provider_routing(self, body: dict) -> None:
"""Prefers one upstream on OpenRouter, so the prompt cache stays warm.
The method does nothing anywhere else. `provider` is an OpenRouter
extension, and Ollama and similar servers reject fields they do not know.
The `reasoning` parameter above is written around the same constraint.
"""
if _OPENROUTER_HOST not in self.base_url:
return
upstream = _PREFERRED_UPSTREAM.get(self.model.split("/", 1)[0].lower())
if upstream:
body["provider"] = {"order": [upstream]}
def _record_usage(self, payload: dict) -> None:
"""Records the endpoint's own token accounting, if it reported any.
OpenRouter now always reports usage, and `usage: {include: true}` and
`stream_options` are deprecated and do nothing. In a stream the usage
arrives on a final chunk that carries no choices, which is why this is
read separately from the text extraction.
"""
usage = payload.get("usage")
if isinstance(usage, dict) and usage:
self.last_usage = usage
def _request(self, parts: PromptParts, temperature: float, max_tokens: int) -> tuple[str, dict]:
if self.api_mode == "completion":
url = f"{self.base_url}/completions"
body = {
"model": self.model,
"prompt": f"{parts.system}\n\n{parts.story}",
"temperature": temperature,
"max_tokens": max_tokens,
"stream": True,
}
else:
url = f"{self.base_url}/chat/completions"
body = {
"model": self.model,
"messages": [
{"role": "system", "content": parts.system},
{"role": "user", "content": parts.story + CHAT_CONTINUE_HINT},
],
"temperature": temperature,
"max_tokens": max_tokens,
"stream": True,
}
self._apply_reasoning_budget(body)
self._apply_provider_routing(body)
return url, body
@staticmethod
def _extract_chunk(payload: dict) -> str:
choices = payload.get("choices") or []
if not choices:
return ""
choice = choices[0]
# A chat stream uses `delta.content`, and a completion stream uses
# `text`. The non-stream fallbacks are `message.content` and `text`.
delta = choice.get("delta") or {}
return (
delta.get("content")
or choice.get("text")
or (choice.get("message") or {}).get("content")
or ""
)
@staticmethod
def _extract_reasoning(payload: dict) -> str:
"""Returns a reasoning model's thinking text.
OpenRouter normalizes it to `reasoning`, and DeepSeek-style servers use
`reasoning_content`.
"""
choices = payload.get("choices") or []
if not choices:
return ""
choice = choices[0]
delta = choice.get("delta") or {}
message = choice.get("message") or {}
return (
delta.get("reasoning")
or delta.get("reasoning_content")
or message.get("reasoning")
or message.get("reasoning_content")
or ""
)
async def generate(
self, parts: PromptParts, *, temperature: float, max_tokens: int
) -> AsyncIterator[tuple[str, str]]:
"""Yields `("text", chunk)` and `("reasoning", chunk)` pairs."""
if not self.model:
raise ProviderError("No model configured — set one in Settings.")
url, body = self._request(parts, temperature, max_tokens)
async for event in self._stream(url, body):
yield event
async def chat(
self, messages: list[dict], *, temperature: float, max_tokens: int
) -> AsyncIterator[tuple[str, str]]:
"""Runs a plain multi-turn chat, with no story framing and no context
assembly.
The method sends `[{"role", "content"}, ...]` straight to the endpoint.
The AI Chat scratchpad uses it, and the turn engine uses `generate()`.
"""
if not self.model:
raise ProviderError("No model configured — set one in Settings.")
if self.api_mode == "completion":
url = f"{self.base_url}/completions"
body = {
"model": self.model,
"prompt": flatten_messages(messages),
"temperature": temperature,
"max_tokens": max_tokens,
"stream": True,
}
else:
url = f"{self.base_url}/chat/completions"
body = {
"model": self.model,
"messages": messages,
"temperature": temperature,
"max_tokens": max_tokens,
"stream": True,
}
self._apply_reasoning_budget(body)
self._apply_provider_routing(body)
async for event in self._stream(url, body):
yield event
async def _stream(self, url: str, body: dict) -> AsyncIterator[tuple[str, str]]:
"""Runs the shared SSE request for `generate()` and `chat()`.
The method POSTs a streaming request, yields `("text", chunk)` and
`("reasoning", chunk)` pairs, and logs the exchange.
"""
# SSRF guard for hosted mode. A user-supplied `endpoint_url` must not
# point at an internal or metadata address. This does nothing for a
# local install.
reason = netguard.endpoint_block_reason(url)
if reason:
raise ProviderError(f"This endpoint can't be used — {reason}.")
log = debuglog.start_entry(url, self.model, body)
received: list[str] = []
try:
async with httpx.AsyncClient(timeout=httpx.Timeout(120, connect=10)) as client:
async with client.stream("POST", url, json=body, headers=self._headers()) as resp:
if resp.status_code != 200:
detail = (await resp.aread()).decode(errors="replace")[:500]
raise ProviderError(self._friendly_http_error(resp.status_code, detail))
# Some servers ignore `stream=true` and return one plain
# JSON body, so buffer the non-SSE lines to fall back to.
saw_sse = False
raw_lines: list[str] = []
async for line in resp.aiter_lines():
if not line.startswith("data:"):
if not saw_sse:
raw_lines.append(line)
continue
saw_sse = True
data = line[5:].strip()
if data == "[DONE]":
debuglog.finish_entry(
log, response="".join(received), usage=self.last_usage
)
return
try:
payload = json.loads(data)
except ValueError:
continue
self._record_usage(payload)
reasoning = self._extract_reasoning(payload)
if reasoning:
yield "reasoning", reasoning
chunk = self._extract_chunk(payload)
if chunk:
received.append(chunk)
yield "text", chunk
if not saw_sse:
body_text = "\n".join(raw_lines).strip()
try:
payload = json.loads(body_text)
except ValueError:
raise ProviderError(
"AI endpoint returned neither an SSE stream nor JSON: "
+ body_text[:200]
)
self._record_usage(payload)
reasoning = self._extract_reasoning(payload)
if reasoning:
yield "reasoning", reasoning
chunk = self._extract_chunk(payload)
if chunk:
received.append(chunk)
yield "text", chunk
if not received:
raise ProviderError(
"AI endpoint returned a response with no text: "
+ body_text[:200]
)
debuglog.finish_entry(log, response="".join(received), usage=self.last_usage)
except httpx.ConnectError as exc:
error = f"Could not connect to {self.base_url} — is the AI server running?"
debuglog.finish_entry(log, response="".join(received), error=error)
raise ProviderError(error) from exc
except httpx.TimeoutException as exc:
debuglog.finish_entry(log, response="".join(received), error="Timed out")
raise ProviderError("The AI endpoint timed out.") from exc
except httpx.HTTPError as exc:
debuglog.finish_entry(log, response="".join(received), error=str(exc))
raise ProviderError(f"Request to AI endpoint failed: {exc}") from exc
except (ProviderError, GeneratorExit, BaseException) as exc:
status = "cancelled" if isinstance(exc, GeneratorExit) else str(exc)
debuglog.finish_entry(log, response="".join(received), error=status)
raise
async def complete(
self, system: str, user: str, *, temperature: float = 0.3, max_tokens: int = 400
) -> str:
"""Runs a single non-streaming completion, for background calls such as
summarization.
Unlike `generate()`, this adds no story-continuation framing.
"""
if not self.model:
raise ProviderError("No model configured — set one in Settings.")
if self.api_mode == "completion":
url = f"{self.base_url}/completions"
body = {
"model": self.model,
"prompt": f"{system}\n\n{user}",
"temperature": temperature,
"max_tokens": max_tokens,
"stream": False,
}
else:
url = f"{self.base_url}/chat/completions"
body = {
"model": self.model,
"messages": [
{"role": "system", "content": system},
{"role": "user", "content": user},
],
"temperature": temperature,
"max_tokens": max_tokens,
"stream": False,
}
self._apply_reasoning_budget(body)
self._apply_provider_routing(body)
log = debuglog.start_entry(url, self.model, body)
try:
async with httpx.AsyncClient(timeout=httpx.Timeout(120, connect=10)) as client:
resp = await client.post(url, json=body, headers=self._headers())
except httpx.HTTPError as exc:
debuglog.finish_entry(log, error=str(exc))
raise ProviderError(f"Request to AI endpoint failed: {exc}") from exc
if resp.status_code != 200:
error = self._friendly_http_error(resp.status_code, resp.text[:500])
debuglog.finish_entry(log, error=error)
raise ProviderError(error)
try:
payload = resp.json()
except ValueError as exc:
debuglog.finish_entry(log, error="Invalid JSON response")
raise ProviderError("AI endpoint returned invalid JSON.") from exc
self._record_usage(payload)
text = self._extract_chunk(payload)
debuglog.finish_entry(log, response=text, usage=self.last_usage)
return text.strip()
async def embed(self, texts: list[str]) -> list[list[float]]:
"""POSTs to /v1/embeddings. Here `self.model` is the embedding model."""
if not self.model:
raise ProviderError("No embedding model configured — set one in Settings.")
url = f"{self.base_url}/embeddings"
body = {"model": self.model, "input": texts}
log = debuglog.start_entry(url, self.model, body)
try:
async with httpx.AsyncClient(timeout=httpx.Timeout(60, connect=10)) as client:
resp = await client.post(url, json=body, headers=self._headers())
except httpx.HTTPError as exc:
debuglog.finish_entry(log, error=str(exc))
raise ProviderError(f"Embedding request failed: {exc}") from exc
if resp.status_code != 200:
error = self._friendly_http_error(resp.status_code, resp.text[:500])
debuglog.finish_entry(log, error=error)
raise ProviderError(error)
try:
data = resp.json().get("data", [])
vectors = [item["embedding"] for item in sorted(data, key=lambda d: d.get("index", 0))]
except (ValueError, KeyError, TypeError) as exc:
debuglog.finish_entry(log, error="Malformed embeddings response")
raise ProviderError("AI endpoint returned malformed embeddings.") from exc
if len(vectors) != len(texts):
debuglog.finish_entry(log, error="Embedding count mismatch")
raise ProviderError("AI endpoint returned the wrong number of embeddings.")
debuglog.finish_entry(log, response=f"{len(vectors)} vectors × {len(vectors[0]) if vectors else 0} dims")
return vectors
def _friendly_http_error(self, status: int, detail: str) -> str:
if status == 401:
return "Authentication failed — check your API key in Settings."
if status == 404:
return (
f"Endpoint or model not found (HTTP 404). Check the endpoint URL and that "
f"model '{self.model}' exists. {detail}"
)
if status == 429:
# OpenRouter's shared free tier has a per-day cap. Distinguish it
# from a short-term burst limit, so the message tells the reader what
# to do.
if "free-models-per-day" in detail:
return (
"The free demo has hit its daily request limit (resets at "
"00:00 UTC). Please try again later."
)
return "The AI is getting too many requests right now — wait a moment and try again."
return f"AI endpoint returned HTTP {status}: {detail}"
backend/app/routers/__init__.py
View File
backend/app/routers/adventures/__init__.py
+56
View File
@@ -0,0 +1,56 @@
"""The adventure endpoints, split across one module per group of routes.
Importing this package registers every route, because each endpoint module
decorates its handlers with the shared `router` from `deps`. The import order
below is the order FastAPI matches paths in. No two routes here shadow each
other, so the order is for reading rather than for correctness.
Read the modules in this order to follow a turn from end to end:
deps the router and the ownership check every endpoint runs
paging reading a window of actions and numbering its attempts
nodes moving around the story tree
turns playing a turn, and the lock that allows only one at a time
takes retries and the attempts that collect at one coordinate
branches where a story splits
What this package re-exports, and what it deliberately does not:
Pure helpers and handlers are re-exported below, so `adventures.ACTION_PAGE` and
`adventures.undo_turn` keep working. The names a test replaces are not, and you
must reach those as `adventures.turns.<name>`. Rebinding a re-exported alias
changes only the alias, so patching `adventures.generate_turn` would leave every
caller reading the original. Leaving those names off raises `AttributeError`
instead, which is the failure you want.
"""
from .deps import router
# Imported for the side effect of registering routes. The names are unused here.
from . import ( # noqa: F401
crud,
turns,
takes,
branches,
bundle_io,
scripts,
refresh,
insights,
memories,
actions,
)
from ... import limits # noqa: F401 `adventures.limits` is patched by tests.
from .crud import SNIPPET_MAX, _snippet
from .paging import ACTION_PAGE
from .takes import retry_action, undo_turn
from .turns import world_delta_of
__all__ = [
"ACTION_PAGE",
"SNIPPET_MAX",
"_snippet",
"limits",
"retry_action",
"router",
"undo_turn",
"world_delta_of",
]
backend/app/routers/adventures/actions.py
+105
View File
@@ -0,0 +1,105 @@
"""Reading, editing, and deleting individual actions.
`list_actions` pages through the current branch. The edit and delete endpoints
change one node, and deleting one removes the whole attempt group at its
coordinate through `nodes.delete_turn`.
"""
from fastapi import Depends, HTTPException
from sqlalchemy.orm import Session
from ... import attempts, models, schemas, tree
from ...database import get_db
from . import turns
from .deps import CurrentUser, current_adventure, router
from .nodes import db_tip, delete_turn
from .paging import ACTION_PAGE, action_window, annotate_takes
@router.get("/{adventure_id}/actions", response_model=schemas.ActionPage)
def list_actions(
before_id: int | None = None,
limit: int = ACTION_PAGE,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Returns a page of the story, working backwards from the newest action.
`before_id` is the oldest action the caller already holds, so scrolling up
asks for what comes before it. Omit `before_id` for the newest window. See
`action_window` for why this anchors on a row rather than an offset.
"""
limit = max(1, min(limit, ACTION_PAGE * 4))
actions, total, has_more = action_window(
db, adventure, before_id=before_id, limit=limit
)
return schemas.ActionPage(
actions=[
schemas.ActionOut.model_validate(a)
for a in annotate_takes(db, adventure.id, actions)
],
total=total,
has_more=has_more,
)
@router.patch("/{adventure_id}/actions/{action_id}", response_model=schemas.ActionOut)
def update_action(
adventure_id: int,
action_id: int,
payload: schemas.ActionUpdate,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
action = db.get(models.Action, action_id)
if action is None or action.adventure_id != adventure_id:
raise HTTPException(404, "Action not found")
# One row holds one text. Nothing mirrors it now, so nothing else has to be
# updated. The edit used to have to be written into the live variant entry
# as well, or paging away and back reverted it.
action.text = payload.text
db.commit()
return action
@router.delete("/{adventure_id}/actions/{action_id}", status_code=204)
def delete_action(
adventure_id: int,
action_id: int,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
action = db.get(models.Action, action_id)
if action is None or action.adventure_id != adventure_id:
raise HTTPException(404, "Action not found")
# The lock is held for the same reason undo holds it: this endpoint puts
# the shared state back, and a turn that is still generating is about to
# write it.
turns.acquire_turn_lock(adventure_id)
try:
# This works like undo. The turn is deleted with all of its attempts,
# and whatever it produced is withdrawn. The marks are depths, and a
# depth does not move when an action before it is deleted.
delete_turn(db, adventure, action)
db.flush()
db.expire(adventure, ["actions"])
# Deleting the newest action moves the tip. Deleting an action in the
# middle leaves a gap in the depths, which is intended. See
# `_backfill_tree`.
tree.refresh_head(db, adventure)
# The script state and the world state belong to the adventure, not to
# the node, so deleting the node does not take back what it did to
# them. Put them back to what the story now ends with, which is the
# same restore a branch switch does.
#
# The world state carries the cooldown clock in `_meta.last_changed`,
# and that clock is a depth. Leaving it set marked the deleted turn's
# changes as having happened at the depth the next turn is played at,
# so the referee refused them as changed too recently — on a turn the
# story no longer contains. Deleting a middle action restores the tip's
# own outcome, which is the state the adventure is already in.
attempts.restore_state(adventure, db_tip(db, adventure))
db.commit()
finally:
turns._active_turns.discard(adventure_id)
backend/app/routers/adventures/branches.py
+259
View File
@@ -0,0 +1,259 @@
"""The branch endpoints: list, rename, delete, and switch.
Attempts accumulate at the tip as siblings, which costs nothing. An attempt
becomes a branch only when the player continues the story from it and leaves the
line that moved past it. That is the same event as playing a turn past the
attempt. Creating the branch then rather than on the next turn means a branch
exists only for a divergence someone built on, and the line being left is not
modified.
"""
from fastapi import Depends, HTTPException
from sqlalchemy import func
from sqlalchemy.orm import Session
from ... import attempts, models, schemas, tree
from ...context import cursors
from ...context import lineage
from ...database import get_db
from . import turns
from .deps import CurrentUser, current_adventure, router
from .nodes import db_tip
from .paging import current_window
@router.get("/{adventure_id}/branches", response_model=list[schemas.BranchOut])
def list_branches(
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Returns every branch of the adventure and where each one leaves its parent.
A tree view is drawn from this shape. `fork_depth` gives the depth where the
line splits off, and `depth` gives the depth where it currently ends. The
whole picture costs one query over `branches` plus one grouped query over
`actions`, never one query per branch, so a view of a hundred forks does not
cost a hundred round trips.
"""
branches = (
db.query(models.Branch)
.filter(models.Branch.adventure_id == adventure.id)
.order_by(models.Branch.id)
.all()
)
owned = {
branch_id: (count, tip)
for branch_id, count, tip in db.query(
models.Action.branch_id,
func.count(models.Action.id),
func.max(models.Action.depth),
)
.filter(
models.Action.adventure_id == adventure.id,
models.Action.live.is_(True),
)
.group_by(models.Action.branch_id)
.all()
}
out = []
for branch in branches:
count, tip = owned.get(branch.id, (0, None))
out.append(schemas.BranchOut(
id=branch.id,
parent_branch_id=branch.parent_branch_id,
fork_depth=branch.fork_depth,
# A branch with no nodes of its own sits at its fork point. That
# node is the last one its story contains. The node is borrowed, but
# it is still the tip. This matches `tree.refresh_head`.
depth=tip if tip is not None else (
branch.fork_depth if branch.fork_depth is not None else tree.NO_DEPTH
),
own_actions=count,
is_head=(branch.id == adventure.head_branch_id),
name=branch.name,
created_at=branch.created_at,
))
return out
def get_branch_or_404(
adventure: models.Adventure, branch_id: int, db: Session
) -> models.Branch:
"""Returns one branch of this adventure.
If the branch belongs to another adventure, the 404 does not confirm that the
branch exists.
"""
branch = db.get(models.Branch, branch_id)
if branch is None or branch.adventure_id != adventure.id:
raise HTTPException(404, "Branch not found")
return branch
@router.patch(
"/{adventure_id}/branches/{branch_id}", response_model=schemas.BranchOut
)
def rename_branch(
branch_id: int,
payload: schemas.BranchRename,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Names a branch, or clears the name to leave it unnamed.
A blank string means the same thing as `null`. A name of only spaces is not a
name anyone chose, and storing one gives the client an empty label to draw
instead of the fork depth.
"""
branch = get_branch_or_404(adventure, branch_id, db)
name = (payload.name or "").strip()
branch.name = name or None
adventure.updated_at = models.utcnow()
db.commit()
db.refresh(branch)
# Read both numbers in one pass, and count them the way `list_branches`
# counts them, as live rows on this branch. A renamed branch is the same
# branch, so this response has to match the row the panel would fetch.
tip, own = (
db.query(func.max(models.Action.depth), func.count(models.Action.id))
.filter(
models.Action.adventure_id == adventure.id,
models.Action.branch_id == branch.id,
models.Action.live.is_(True),
)
.one()
)
return schemas.BranchOut(
id=branch.id,
parent_branch_id=branch.parent_branch_id,
fork_depth=branch.fork_depth,
depth=tip if tip is not None else (
branch.fork_depth if branch.fork_depth is not None else tree.NO_DEPTH
),
own_actions=own,
is_head=(branch.id == adventure.head_branch_id),
name=branch.name,
created_at=branch.created_at,
)
@router.delete("/{adventure_id}/branches/{branch_id}", status_code=204)
def delete_branch(
adventure_id: int,
branch_id: int,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Deletes a branch and everything forked from it.
Nothing prunes the tree automatically, so this endpoint is what keeps a
heavily retried adventure from growing without bound. That is why it ships
with the view that first lets anyone create a fork rather than after it.
Two kinds of branch cannot be deleted. The root cannot, because it holds the
turns every other branch borrows, so deleting it deletes the whole story. The
branch currently being read cannot, and neither can any branch it was forked
from, because the cascade would remove the head under the player and leave
`head_branch_id` dangling. Switch branches first.
Nodes and memories are deleted by `ON DELETE CASCADE`, and descendants by the
cascade on `branches.parent_branch_id`, so the delete is a single statement
however deep the subtree is.
"""
branch = get_branch_or_404(adventure, branch_id, db)
if branch.parent_branch_id is None:
raise HTTPException(
400, "This is the story's first branch — deleting it would delete "
"the adventure. Delete the adventure itself instead.",
)
head = db.get(models.Branch, adventure.head_branch_id)
# The head's lineage lists itself and every branch it borrows from, so one
# membership test covers both the branch being read and any branch forked
# from it.
if head is not None and branch.id in {
entry_id for entry_id, _ in lineage.entries_of(head)
}:
raise HTTPException(
400, "You are reading this branch, or one forked from it. Switch to "
"another branch first.",
)
turns.acquire_turn_lock(adventure_id)
try:
# Collect the subtree before the delete, because afterwards there is no
# way to ask which branches were removed. A cursor left pointing at a
# deleted branch is harmless on Postgres, which never reuses ids, but it
# is a bug on SQLite, where the next fork can receive the id that was
# just freed. A stale anchor then resolves onto a branch it never saw.
doomed = _branch_subtree(db, adventure, branch)
for cursor in cursors.ALL:
stored_branch, _ = cursor.stored(adventure)
if stored_branch in doomed:
cursor.clear(adventure)
# The deleted branch's memories are deleted with it, and their cached
# vectors drop out of the catalogue on the next read, so no
# invalidation call is needed. See the note on the `memorybank` cache.
db.delete(branch)
adventure.updated_at = models.utcnow()
db.commit()
finally:
turns._active_turns.discard(adventure_id)
def _branch_subtree(
db: Session, adventure: models.Adventure, root: models.Branch
) -> set[int]:
"""Returns `root` and every branch descended from it, following parent pointers.
The walk runs over the adventure's own branch rows rather than one query per
level. An adventure has few branches, so the walk costs one round trip, and a
recursive CTE would have to be written twice for the two dialects this
codebase supports.
"""
children: dict[int | None, list[int]] = {}
for bid, parent in db.query(models.Branch.id, models.Branch.parent_branch_id).filter(
models.Branch.adventure_id == adventure.id
):
children.setdefault(parent, []).append(bid)
found: set[int] = set()
stack = [root.id]
while stack:
current = stack.pop()
if current in found:
continue
found.add(current)
stack.extend(children.get(current, ()))
return found
@router.post(
"/{adventure_id}/branches/{branch_id}/switch", response_model=schemas.ActionPage
)
def switch_branch(
adventure_id: int,
branch_id: int,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Reads and plays a different branch of the story.
No row is copied and no row is rewritten. The head pointer moves, and the
shared script state and world state are restored to what that branch's tip
left behind. The restore is what makes a switch safe. Both states are stored
per adventure, so a branch that did not restore them would be played with
another branch's numbers, including the world-state cooldown clock inside the
snapshot.
"""
branch = db.get(models.Branch, branch_id)
if branch is None or branch.adventure_id != adventure.id:
raise HTTPException(404, "Branch not found")
turns.acquire_turn_lock(adventure_id)
try:
adventure.head_branch_id = branch.id
tree.refresh_head(db, adventure)
attempts.restore_state(adventure, db_tip(db, adventure))
adventure.updated_at = models.utcnow()
db.commit()
db.refresh(adventure)
return current_window(db, adventure)
finally:
turns._active_turns.discard(adventure_id)
backend/app/routers/adventures/bundle_io.py
+70
View File
@@ -0,0 +1,70 @@
"""Exporting an adventure to a bundle, and importing one back.
`app/bundle.py` owns the format and the version handling. These two endpoints
only check ownership and hand the work over.
"""
from fastapi import Body, Depends, Request
from sqlalchemy.orm import Session
from ... import analytics, bundle, limits, models, schemas
from ...database import get_db
from .deps import CurrentUser, current_adventure, router
@router.get("/{adventure_id}/export")
def export_adventure(
db: Session = Depends(get_db),
adv: models.Adventure = Depends(current_adventure),
):
"""Returns a full backup: plot components, story cards, scripts, state, and tree.
`app/bundle.py` owns the format, in both of its versions. A backup outlives
the schema, so no call site decides anything about its shape.
"""
return bundle.export(db, adv)
@router.post("/import", response_model=schemas.AdventureOut, status_code=201)
def import_adventure(
request: Request,
payload: dict = Body(...),
db: Session = Depends(get_db),
user: models.User = CurrentUser,
):
version = bundle.check_format(payload)
limits.rate_limit("import", request, user)
limits.check_row_cap("adventures", db, user)
limits.check_bundle_lists(
story_cards=payload.get("storyCards"),
memories=payload.get("memories"),
actions=payload.get("actions"),
branches=payload.get("branches"),
)
# Check the tree before the adventure row exists, so that an inconsistent
# file returns a 400 rather than leaving a half-imported adventure with a
# gap in its story.
story = bundle.plan(payload, version)
# Count again, this time over what is written. The check above reads the
# file's own lists, and in a v1 file one turn is one entry that carries its
# retries in a `variants` array. `plan()` expands that into one row per
# attempt, because SP4 made every attempt a node. A file of 5,000 turns with
# ten attempts each therefore passes a 5,000-action cap and writes 50,000
# rows, well inside the 20 MB body limit. `plan()` has no side effects and
# the adventure does not exist yet, so this check costs only the planning.
limits.check_bundle_lists(
actions=story["nodes"],
memories=story["memories"],
branches=story["branches"],
)
adventure = bundle.materialize(db, payload, story, user.id)
db.commit()
db.refresh(adventure)
# This is not a funnel step. A returning player imports a bundle, so it
# says nothing about how far a first-time visitor got. It is counted anyway,
# because it is the clearest evidence that anyone uses the export format.
analytics.record_event(analytics.EV_IMPORT, user)
return adventure
backend/app/routers/adventures/crud.py
+340
View File
@@ -0,0 +1,340 @@
"""Listing, creating, reading, renaming, and deleting adventures.
The world-state and script-state readers are here too, because they report an
adventure's stored state rather than play a turn.
"""
from fastapi import Body, Depends, HTTPException
from sqlalchemy import func
from sqlalchemy.orm import Session
from sqlalchemy.orm.attributes import set_committed_value
from ... import analytics, attempts, images, limits, memorybank, models, schemas, tree, worldstate
from ...database import get_db
from .deps import CurrentUser, current_adventure, router
from .paging import action_window, annotate_takes
from .scenario_text import fill_placeholders, scenario_card_specs
# How many characters of the last narration a Continue card shows. The limit is
# long enough to re-establish the scene and short enough to keep the card
# small.
SNIPPET_MAX = 220
def _snippet(text: str) -> str:
"""Condenses stored action text into a single line for a card."""
# `turns._generate_turn` strips the world-state block before storing AI text,
# so this function only has to normalize whitespace.
collapsed = " ".join((text or "").split())
if len(collapsed) <= SNIPPET_MAX:
return collapsed
# Cut at a word boundary rather than mid-word. CSS adds the ellipsis.
cut = collapsed[:SNIPPET_MAX].rsplit(" ", 1)[0]
return f"{cut}…"
# Action types that read as narration. `start` is the scenario's opening prompt,
# which is the only text a newly created adventure has. Without `start`, a new
# story's card would show no text at all. `do` and `say` are excluded because
# the card quotes the story's voice rather than the player's.
NARRATION_TYPES = ("ai", "story", "start")
def _latest_narration(db: Session, head_branches: dict[int, int | None]) -> dict[int, str]:
"""Maps each adventure id to the text of its most recent narrated action.
This runs one window-function query rather than one lookup per adventure, so
the list endpoint makes a fixed number of round trips.
The query is scoped by head branch rather than by the full lineage, and this
is the only place in the codebase that does so. A lineage clause per
adventure would add a hundred OR terms to the index screen's query to select
one row each. The two scopes differ only for a branch with no nodes of its
own, and playing a turn onto a branch is what creates it, so that state does
not occur. An adventure with no branch has no story to quote.
"""
branch_ids = [b for b in head_branches.values() if b is not None]
if not branch_ids:
return {}
ranked = (
db.query(
models.Action.adventure_id.label("adventure_id"),
models.Action.text.label("text"),
func.row_number()
.over(
partition_by=models.Action.adventure_id,
order_by=(models.Action.depth.desc(), models.Action.id.desc()),
)
.label("rank"),
)
.filter(
models.Action.adventure_id.in_(list(head_branches)),
models.Action.branch_id.in_(branch_ids),
models.Action.type.in_(NARRATION_TYPES),
# Sibling attempts share a depth, and the newest has the highest
# id. Without this filter the snippet quotes the attempt written
# last rather than the one the story tells. After you switch back to
# an earlier attempt, the index screen would quote the discarded one
# and disagree with the story on screen.
models.Action.live.is_(True),
)
.subquery()
)
rows = db.query(ranked.c.adventure_id, ranked.c.text).filter(ranked.c.rank == 1).all()
return {adventure_id: text for adventure_id, text in rows}
@router.get("", response_model=list[schemas.AdventureListItem])
def list_adventures(db: Session = Depends(get_db), user: models.User = CurrentUser):
# Select named columns rather than the whole Adventure entity. The entity is
# sixteen columns wide and includes `script_state`, `world_state`,
# `placeholders`, `story_summary`, `memory`, `authors_note`, and
# `ai_instructions`. That is about 15 kB per row in production, fetched once
# per adventure on every index load, and this screen uses none of it. Naming
# the columns also means a wide column added to Adventure later has to opt
# in to being listed here.
rows = (
db.query(
models.Adventure.id,
models.Adventure.scenario_id,
models.Adventure.title,
models.Adventure.updated_at,
models.Adventure.head_branch_id,
func.count(models.Action.id),
models.Scenario.title,
models.Scenario.image,
models.Scenario.icon,
models.Scenario.updated_at,
)
.outerjoin(models.Action)
.outerjoin(models.Scenario, models.Adventure.scenario_id == models.Scenario.id)
.filter(models.Adventure.user_id == user.id)
# Group by both primary keys. Postgres requires every selected column
# to be grouped or aggregated. The Adventure columns are covered by its
# own grouped primary key, but the Scenario columns come from a joined
# table and have to be listed as well. SQLite accepts the shorter form,
# and Postgres rejects it.
.group_by(
models.Adventure.id,
models.Scenario.id,
models.Scenario.title,
models.Scenario.image,
models.Scenario.icon,
models.Scenario.updated_at,
)
.order_by(models.Adventure.updated_at.desc())
.all()
)
narration = _latest_narration(db, {row[0]: row[4] for row in rows})
return [
schemas.AdventureListItem(
id=adv_id,
scenario_id=scenario_id,
scenario_title=scenario_title,
title=title,
updated_at=updated_at,
action_count=count,
snippet=_snippet(narration.get(adv_id, "")),
# The art belongs to the scenario, so the cache-busting stamp uses
# the scenario's `updated_at`, not the adventure's.
image_url=images.public_url(scenario_id, image or "", scenario_updated),
icon=icon or "",
)
# `count` counts every action in the adventure, not only the ones on
# the path. With one branch the two numbers are equal. After forking
# ships, the index screen overstates a story that has sibling branches.
# The fix belongs to SP5, which is where a fork can first exist.
for (adv_id, scenario_id, title, updated_at, _head_branch_id, count,
scenario_title, image, icon, scenario_updated) in rows
]
@router.post("", response_model=schemas.AdventureOut, status_code=201)
def create_adventure(
payload: schemas.AdventureCreate,
db: Session = Depends(get_db),
user: models.User = CurrentUser,
):
limits.check_row_cap("adventures", db, user)
scenario = None
if payload.scenario_id is not None:
scenario = db.get(models.Scenario, payload.scenario_id)
# A scenario is playable if the user owns it or if it is public.
if scenario is None or (scenario.user_id != user.id and not scenario.is_public):
raise HTTPException(404, "Scenario not found")
values = payload.placeholders
adventure = models.Adventure(
user_id=user.id,
scenario_id=scenario.id if scenario else None,
title=payload.title or (scenario.title if scenario else "Untitled Adventure"),
memory=fill_placeholders(scenario.memory, values) if scenario else "",
authors_note=fill_placeholders(scenario.authors_note, values) if scenario else "",
ai_instructions=fill_placeholders(scenario.ai_instructions, values) if scenario else "",
# Phase 12: seed the live RPG state from the scenario's template.
world_state=worldstate.instantiate(scenario.stat_schema) if scenario else {},
# Stored so that a later "Update from scenario" fills the copied text
# with the same answers instead of inserting literal `${...}` tokens.
placeholders=dict(values),
# Phase 18: the persona the player named before starting. It belongs to
# the adventure rather than the scenario, so nothing is copied here and
# "Update from scenario" never touches it.
persona_name=payload.persona_name.strip(),
persona_pronouns=payload.persona_pronouns.strip(),
persona_desc=payload.persona_desc.strip(),
)
db.add(adventure)
db.flush()
# Give every adventure a story tree as soon as it exists, before anything
# is played onto it. Otherwise the tree code has to tolerate a NULL head
# everywhere, which buys nothing.
tree.head_branch(db, adventure)
if scenario:
for ref, spec in scenario_card_specs(scenario, values).items():
db.add(models.StoryCard(adventure_id=adventure.id, source_ref=ref, **spec))
for position, script in enumerate(scenario.scripts):
db.add(
models.AdventureScript(
adventure_id=adventure.id,
source_script_id=script.id,
position=position,
name=script.name,
description=script.description,
library_js=script.library_js,
input_js=script.input_js,
context_js=script.context_js,
output_js=script.output_js,
)
)
if scenario.prompt.strip():
opening = models.Action(
adventure_id=adventure.id,
type="start",
text=fill_placeholders(scenario.prompt, values),
)
# Record the starting state on the opening node, so undoing or
# retrying the first turn has a state to roll back to.
attempts.snapshot_outcome(adventure, opening)
tree.place_action(db, adventure, opening)
db.add(opening)
db.commit()
db.refresh(adventure)
analytics.record_event(analytics.EV_ADVENTURE, user)
# Track which shared scenarios players pick. This is the only content this
# module records, and it records only public scenarios. A player's own
# scenario titles stay private.
if scenario is not None and scenario.is_public:
analytics.record(analytics.M_SCENARIO, scenario.title)
return adventure
@router.get("/{adventure_id}", response_model=schemas.AdventureOut)
def get_adventure(
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Returns the adventure and the newest window of its story.
`actions` holds the last `ACTION_PAGE` actions, not all of them.
`action_count` reports the real total, so the reader knows that more actions
exist above. `GET /{id}/actions` serves the older pages as the reader
scrolls up.
"""
actions, total, _ = action_window(db, adventure)
# Annotate before handing over the window. This path serializes through the
# relationship rather than building `ActionOut` itself, so the pager numbers
# have to be on the rows before Pydantic reads them.
annotate_takes(db, adventure.id, actions)
# Attach the window as if the relationship had loaded it.
# `set_committed_value` is the only safe way to do this. Assigning
# `adventure.actions = [...]` marks the collection dirty, and the
# relationship cascades delete-orphan, so the next flush deletes every
# action outside the window. `set_committed_value` records the rows as the
# already-loaded, unmodified value, so serialization triggers no lazy load
# and leaves nothing pending.
set_committed_value(adventure, "actions", actions)
out = schemas.AdventureOut.model_validate(adventure)
out.action_count = total
return out
@router.get("/{adventure_id}/script-state")
def get_script_state(
adventure: models.Adventure = Depends(current_adventure),
):
"""Returns the scripting `state` object.
The object holds every variable that scripts read and write through
`state.x`, persisted after each hook. It stays `{}` until a script sets a
variable.
"""
state = adventure.script_state if isinstance(adventure.script_state, dict) else {}
return {"state": state}
@router.get("/{adventure_id}/world-state")
def get_world_state(
adventure: models.Adventure = Depends(current_adventure),
):
"""Returns the live RPG world state and the scenario's `stat_schema`.
The play view uses both to render the character sheet and the milestones.
`schema` is null when the adventure has no RPG layer.
"""
schema = adventure.scenario.stat_schema if adventure.scenario else None
state = adventure.world_state if isinstance(adventure.world_state, dict) else {}
return {
"state": state,
"schema": schema if worldstate.has_schema(schema) else None,
}
@router.put("/{adventure_id}/world-state")
def override_world_state(
overrides: dict = Body(...),
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Edits the live RPG values directly, as a manual correction rather than a turn.
`overrides` maps paths such as `player.hp`, `npc.gwen.trust`, `flags.x`, and
`milestones.y` to their new absolute values. The endpoint rejects unknown
paths and wrong types one at a time, and applies the rest.
"""
schema = adventure.scenario.stat_schema if adventure.scenario else None
if not worldstate.has_schema(schema):
raise HTTPException(400, "This adventure has no RPG world-state layer")
state = adventure.world_state if isinstance(adventure.world_state, dict) else {}
new_state, report = worldstate.apply_override(state, schema, overrides)
adventure.world_state = new_state
db.commit()
return {"state": new_state, "report": report}
@router.patch("/{adventure_id}", response_model=schemas.AdventureOut)
def update_adventure(
payload: schemas.AdventureUpdate,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
for field, value in payload.model_dump(exclude_unset=True).items():
setattr(adventure, field, value)
db.commit()
return adventure
@router.delete("/{adventure_id}", status_code=204)
def delete_adventure(
adventure_id: int,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
db.delete(adventure)
db.commit()
# No later request reads this adventure's vectors, so drop them now. The
# cache would otherwise hold them until the process restarted.
memorybank.forget_cached_vectors(adventure_id)
backend/app/routers/adventures/deps.py
+42
View File
@@ -0,0 +1,42 @@
"""The router every endpoint module registers on, and the dependencies they share.
This module imports nothing else in the package. Keeping it at the bottom of the
import graph is what lets each endpoint module import the router without
importing its siblings.
"""
from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy.orm import Session
from ... import auth, models
from ...database import get_db
router = APIRouter(prefix="/api/adventures", tags=["adventures"])
CurrentUser = Depends(auth.get_current_user)
def get_adventure_or_404(
adventure_id: int, db: Session, user: models.User
) -> models.Adventure:
adventure = db.get(models.Adventure, adventure_id)
if adventure is None or adventure.user_id != user.id:
raise HTTPException(404, "Adventure not found")
return adventure
def current_adventure(
adventure_id: int,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
) -> models.Adventure:
"""Resolves the `{adventure_id}` in the path to the caller's own adventure.
Declare this in a handler's signature. The ownership check then reads in the
signature, where you look for it, and it runs before the handler body.
FastAPI caches a dependency for the length of one request, so the handler's
own `db` is the same session this one loaded the adventure from.
"""
return get_adventure_or_404(adventure_id, db, user)
backend/app/routers/adventures/insights.py
+50
View File
@@ -0,0 +1,50 @@
"""Read-only views of the context an adventure would send, or did send.
The dry run assembles a prompt without calling the model. The per-action endpoint
returns the prompt a turn was actually generated from. Neither writes anything.
"""
from fastapi import Depends, HTTPException
from sqlalchemy.orm import Session
from ... import auth, memorybank, models
from ...context import build_context
from ...database import get_db
from ..settings import get_settings
from .deps import CurrentUser, current_adventure, router
@router.get("/{adventure_id}/context")
async def dry_run_context(
db: Session = Depends(get_db),
user: models.User = CurrentUser,
adventure: models.Adventure = Depends(current_adventure),
):
"""Returns what the app would send to the AI if the player continued now."""
settings = get_settings(db, user)
if auth.resolve_provider_config(settings).using_demo:
memories = (
{"used": [], "error": "Memory bank is unavailable on the shared demo key."}
if adventure.memory_bank_enabled
else None
)
else:
memories = await memorybank.retrieve_memories(adventure, settings, update_stats=False)
_, _, report = build_context(adventure, settings, memories)
return report
@router.get("/{adventure_id}/actions/{action_id}/context")
def action_context(
adventure_id: int,
action_id: int,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
action = db.get(models.Action, action_id)
if action is None or action.adventure_id != adventure_id:
raise HTTPException(404, "Action not found")
if action.context_snapshot is None:
raise HTTPException(404, "No context snapshot for this action")
return action.context_snapshot
backend/app/routers/adventures/memories.py
+116
View File
@@ -0,0 +1,116 @@
"""CRUD for the memory bank entries attached to one adventure.
`app/memorybank.py` owns embedding and retrieval. These endpoints only edit the
rows.
"""
from fastapi import Depends, HTTPException
from sqlalchemy.orm import Session, load_only
from ... import limits, memorybank, models, schemas, tree
from ...context import lineage
from ...database import get_db
from .deps import CurrentUser, current_adventure, router
# The columns `schemas.MemoryOut` renders. `embedded` is a real column and
# belongs here. The vector it describes does not.
MEMORY_LIST_COLUMNS = (
models.Memory.adventure_id,
models.Memory.text,
models.Memory.pinned,
models.Memory.forgotten,
models.Memory.embedded,
models.Memory.use_count,
models.Memory.last_used_at,
models.Memory.source_start,
models.Memory.source_end,
models.Memory.created_at,
)
@router.get("/{adventure_id}/memories", response_model=list[schemas.MemoryOut])
def list_memories(
adventure_id: int,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
# Name the columns in a query rather than walk `adventure.memories`.
# Retrieval used to walk the relationship, which is why a turn cost
# megabytes: a relationship load returns whole entities, so it reads
# whatever the model carries. `embedding_blob` is deferred and would stay
# out today, so this rule is about the next wide column rather than that
# one.
#
# The drawer shows the same bank the model reads. The filter uses the same
# clause retrieval uses, so the drawer answers one question rather than two.
# An adventure-wide list would show memories from branches this story never
# went down, which are never retrieved, and a reader cannot tell those apart
# from the ones in play. No memory becomes unreachable, because a memory
# belongs to a branch: switching to that branch shows it, and deleting the
# branch deletes its memories.
return (
db.query(models.Memory)
.options(load_only(*MEMORY_LIST_COLUMNS))
.filter(
models.Memory.adventure_id == adventure_id,
lineage.path_of(db, adventure).clause(models.Memory),
)
.order_by(models.Memory.id)
.all()
)
@router.post("/{adventure_id}/memories", response_model=schemas.MemoryOut, status_code=201)
def create_memory(
payload: schemas.MemoryCreate,
db: Session = Depends(get_db),
user: models.User = CurrentUser,
adventure: models.Adventure = Depends(current_adventure),
):
"""Adds a memory manually. The next post-turn pass embeds it."""
limits.check_row_cap("memories", db, user, adventure=adventure)
if not payload.text.strip():
raise HTTPException(400, "Memory text cannot be empty")
memory = models.Memory(adventure_id=adventure.id, text=payload.text.strip())
# No node produced this memory, so it gets a branch but no depth.
tree.place_memory(db, adventure, memory)
db.add(memory)
db.commit()
db.refresh(memory)
return memory
@router.patch("/{adventure_id}/memories/{memory_id}", response_model=schemas.MemoryOut)
def update_memory(
adventure_id: int,
memory_id: int,
payload: schemas.MemoryUpdate,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
memory = db.get(models.Memory, memory_id)
if memory is None or memory.adventure_id != adventure_id:
raise HTTPException(404, "Memory not found")
fields = {k: v for k, v in payload.model_dump(exclude_unset=True).items() if v is not None}
if "text" in fields and fields["text"].strip() != memory.text:
memorybank.set_vector(memory, None) # Re-embed on the next post-turn pass.
for field, value in fields.items():
setattr(memory, field, value)
db.commit()
return memory
@router.delete("/{adventure_id}/memories/{memory_id}", status_code=204)
def delete_memory(
adventure_id: int,
memory_id: int,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
memory = db.get(models.Memory, memory_id)
if memory is None or memory.adventure_id != adventure_id:
raise HTTPException(404, "Memory not found")
db.delete(memory)
db.commit()
backend/app/routers/adventures/nodes.py
+139
View File
@@ -0,0 +1,139 @@
"""Moving around the story tree: what is newest, what comes next, what to remove.
These functions answer questions about action nodes without knowing which
endpoint asked. They do not touch the turn lock and they do not stream, so any
module in the package can import them.
"""
from fastapi import HTTPException
from sqlalchemy.orm import Session, undefer
from ... import attempts, memorybank, models, tree
from ...context import cursors
from ...context import lineage
def next_depth(adventure: models.Adventure) -> int:
"""Returns the depth for the next node played onto this story, one past the tip.
A depth is a position along one story, and the branch is what makes it
unambiguous. Two branches each hold a node at depth 4, and they are
alternatives rather than duplicates.
"""
return adventure.head_depth + 1
def last_action(adventure: models.Adventure, db: Session) -> models.Action | None:
"""Returns the newest action of any kind on the story being played, or `None`.
This runs a query rather than reading `adventure.actions[-1]`, which loads
the entire story to read one row. That collection also holds every branch's
actions, so it sometimes returns a row from the wrong branch.
"""
return (
db.query(models.Action)
.filter(
models.Action.adventure_id == adventure.id,
lineage.path_of(db, adventure).clause(models.Action),
)
.order_by(models.Action.depth.desc(), models.Action.id.desc())
.first()
)
def _move_to_after(
db: Session, adventure: models.Adventure, after_id: int | None
) -> None:
"""Moves the story to `after_id` before the turn is played.
This is where a branch is created (SP9). Reading an attempt that the story
moved past changes nothing on the server. Writing below one is the first time
the player states which line they mean, and that is when the fork happens.
An attempt already on the path needs no move, because the story is already
there.
"""
if after_id is None:
return
node = db.get(models.Action, after_id)
if node is None or node.adventure_id != adventure.id:
raise HTTPException(404, "Action not found")
if node.live and lineage.path_of(db, adventure).contains(node):
return
if not node.live and len(attempts.group(db, node)) < 2:
# The pager cannot reach this node, so no legitimate action put the
# player here.
raise HTTPException(400, "That take is not one of this turn's.")
stand_on(db, adventure, node)
db.commit()
db.refresh(adventure)
def delete_turn(
db: Session, adventure: models.Adventure, node: models.Action
) -> None:
"""Removes a turn, including every attempt at it and not only the one on screen.
A discarded attempt is a leaf at the same coordinate, and the only way to
reach it is through that coordinate. Leaving it behind when the turn is
deleted orphans a row that no read can reach. Whatever the turn produced is
withdrawn once, because a memory is attached to the coordinate rather than to
one attempt.
"""
memorybank.forget_node(db, adventure, node)
# Scoped to this node's branch (SP9). Groups span branches now, and an
# attempt forked onto its own line belongs to another branch's story. See
# `attempts.on_branch`.
for attempt in attempts.on_branch(attempts.group(db, node), node):
db.delete(attempt)
def db_tip(db: Session, adventure: models.Adventure) -> models.Action | None:
"""Returns the newest node of the story as it stands, with its outcome loaded."""
return (
db.query(models.Action)
.filter(
models.Action.adventure_id == adventure.id,
lineage.path_of(db, adventure).clause(models.Action),
)
.options(
undefer(models.Action.state_after),
undefer(models.Action.world_state_after),
)
.order_by(models.Action.depth.desc(), models.Action.id.desc())
.first()
)
def stand_on(
db: Session, adventure: models.Adventure, action: models.Action
) -> None:
"""Makes `action` the attempt the story tells, forking only if that is needed.
There are two cases, and the caller does not have to know which one applies.
While the turn is still the tip, its attempts are leaves that nothing was
built on, so this is a switch and no branch is created. Once the story has
moved past the turn, the line being left keeps every turn it has, so the
attempt needs a branch of its own.
The fork endpoint calls this function, and so does a turn played below an
attempt the story moved past. Both are the same operation, once as a request
and once as a step on the way to writing (SP9).
"""
newest = last_action(adventure, db)
at_the_tip = (
newest is not None
and newest.branch_id == action.branch_id
and newest.depth == action.depth
)
if at_the_tip:
# The story at this coordinate is about to change, so withdraw whatever
# was derived from it. A retry does the same thing. A fork needs none of
# this, because it leaves the coordinate and its memory where they are.
# See `tree.fork`.
memorybank.forget_node(db, adventure, action)
cursors.rewind_all(adventure, action.branch_id, (action.depth or 0) - 1)
attempts.make_live(db, adventure, action)
else:
tree.fork(db, adventure, action)
attempts.restore_state(adventure, action)
backend/app/routers/adventures/paging.py
+168
View File
@@ -0,0 +1,168 @@
"""Reading a window of actions, and numbering the attempts inside it.
Three callers build an action window: the adventure GET, the action list, and
every endpoint that returns a page after changing the story. They read the same
columns and apply the same numbering, so both live here.
"""
from sqlalchemy import func
from sqlalchemy.orm import Session, load_only
from ... import models, schemas
from ...context import lineage
# The columns `schemas.ActionOut` renders, listed explicitly.
#
# `deferred=True` in `models.py` keeps the four heavy columns out of bulk reads,
# but each new column then has to opt in to staying narrow. Both egress
# regressions this project has had came from a column that did not opt in. This
# tuple inverts the default: a new column costs nothing until you add it here.
#
# `world_delta` is listed because `ActionOut.world_changes` is computed from it.
# Omitting it saves no bytes. It converts one bulk read into one lazy load per
# row.
ACTION_LIST_COLUMNS = (
models.Action.adventure_id,
models.Action.type,
models.Action.text,
models.Action.reasoning,
models.Action.world_delta,
# SP9: the pager's key. If `parent_id` were deferred, every row on the page
# would cost a lazy load, which is the cost `load_only` is here to prevent.
# `branch_id` is listed for the same reason. The pager reads it to tell a
# local step from a branch switch.
models.Action.parent_id,
models.Action.branch_id,
models.Action.created_at,
)
# How many actions an adventure opens with, and how many arrive per scroll.
#
# Opening a finished adventure once fetched the whole story in one response.
# That reached 589.5 kB for the longest story in production, and it grew as
# stories grew. 60 actions is a few screens of reading. The common case of
# opening a story, reading the end, and taking a turn never pages, and the worst
# case is bounded by the window size rather than by the length of the story.
ACTION_PAGE = 60
def action_window(
db: Session,
adventure: models.Adventure,
before_id: int | None = None,
limit: int = ACTION_PAGE,
) -> tuple[list[models.Action], int, bool]:
"""Returns the `limit` actions immediately older than `before_id`, oldest first.
The return value is `(actions, total, has_more)`. If `before_id` is `None`,
the newest window is returned.
The query is scoped to the head branch's lineage, which is the story being
played, rather than to the adventure. A sibling branch's turns therefore
never appear in the transcript. `total` counts the same path, because it is
what tells the reader that more actions exist above.
The window is anchored on an action, never on a count or on arithmetic over
depth, for two reasons:
* Appends. Counting back from the newest action shifts every older position
when a turn lands. A reader who scrolls up while a turn is generating gets
a window that is one row off, which re-sends one action and skips another.
An anchor is stable, because "older than this action" means the same thing
before and after the story grows.
* The story tree. Depth is dense today, and branching ends that. Comparing
depths to order a path still works, but treating them as positions does
not.
`has_more` comes from requesting one row past the window rather than from a
second count, so it costs one row instead of a scan.
"""
path = lineage.path_of(db, adventure)
on_path = (
models.Action.adventure_id == adventure.id,
path.clause(models.Action),
)
total = db.query(func.count(models.Action.id)).filter(*on_path).scalar()
if limit <= 0:
return [], total, total > 0
query = db.query(models.Action).options(load_only(*ACTION_LIST_COLUMNS)).filter(*on_path)
if before_id is not None:
anchor = (
db.query(models.Action.depth)
.filter(models.Action.id == before_id, *on_path)
.scalar()
)
if anchor is None:
# The anchor was deleted while the reader scrolled, by an undo or
# by an edited turn, or it belongs to a branch this story is not on.
# No row can be older than a row that is not present, so report the
# end of the story rather than guess and return a duplicate page.
return [], total, False
query = query.filter(models.Action.depth < anchor)
rows = (
query.order_by(models.Action.depth.desc(), models.Action.id.desc())
.limit(limit + 1)
.all()
)
has_more = len(rows) > limit
rows = rows[:limit]
rows.reverse()
return rows, total, has_more
def annotate_takes(
db: Session, adventure_id: int, actions: list[models.Action]
) -> list[models.Action]:
"""Sets the `2/4` pager numbers on every action on a page (SP9).
This runs one query for the whole page rather than one per row. The pager
needs the shape of each turn's attempt group, and calling `attempts.group`
per action costs one query per message on screen.
This function reads the siblings rather than counting them. A group holds
only a few attempts, the page is bounded, and a count still needs a second
query for the ordinal. It fetches only the id and the parent, so it stays
cheap even when the text is large.
"""
parents = {a.parent_id for a in actions if a.parent_id is not None}
if parents:
rows = (
db.query(models.Action.id, models.Action.parent_id)
.filter(
models.Action.adventure_id == adventure_id,
models.Action.parent_id.in_(parents),
)
.order_by(models.Action.id)
.all()
)
else:
rows = []
siblings: dict[int, list[int]] = {}
for row_id, parent_id in rows:
siblings.setdefault(parent_id, []).append(row_id)
for action in actions:
ids = siblings.get(action.parent_id) if action.parent_id else None
if not ids:
# A root node, or a pre-SP9 row that the backfill could not place.
# It has one attempt, which is how it was written.
action.take_count, action.take_index = 1, 0
continue
action.take_count = len(ids)
action.take_index = ids.index(action.id) if action.id in ids else 0
return actions
def current_window(db: Session, adventure: models.Adventure) -> schemas.ActionPage:
actions, total, has_more = action_window(db, adventure)
return schemas.ActionPage(
actions=[
schemas.ActionOut.model_validate(a)
for a in annotate_takes(db, adventure.id, actions)
],
total=total,
has_more=has_more,
)
backend/app/routers/adventures/refresh.py
+195
View File
@@ -0,0 +1,195 @@
"""Copying a scenario's current content over an adventure's copy.
An adventure copies the scenario's plot text and story cards when it is created,
so that later authoring does not change a story in progress. The per-script "Sync
from library" in `scripts` works the same way. This module is the explicit
opt-out. The preview endpoint reports what would change, and the write endpoint
applies it.
"""
from fastapi import Body, Depends, HTTPException
from sqlalchemy.orm import Session
from ... import models, schemas, worldstate
from ...database import get_db
from . import turns
from .deps import CurrentUser, current_adventure, router
from .scenario_text import (
CARD_FIELDS, SCENARIO_TEXT_FIELDS, fill_placeholders, scenario_card_specs,
scenario_placeholder_names,
)
def resolve_source_scenario(
adventure: models.Adventure, db: Session, user: models.User
) -> models.Scenario | None:
"""Returns the scenario an adventure can refresh from.
The result is the scenario the adventure was started from, if that scenario
still exists and the user can still read it, which means the user owns it or
it is public. The result is `None` after the scenario is deleted, which sets
`scenario_id` to NULL, or after it stops being shared.
"""
if adventure.scenario_id is None:
return None
scenario = db.get(models.Scenario, adventure.scenario_id)
if scenario is None or (scenario.user_id != user.id and not scenario.is_public):
return None
return scenario
def _scenario_cards(adventure: models.Adventure) -> dict[str, models.StoryCard]:
"""Returns the adventure's scenario-derived cards, keyed by `source_ref`.
An adventure created before `source_ref` existed has none, so it falls back
to matching the scenario's cards by name. The fallback runs only when the
adventure has no tagged cards at all. Otherwise a player-authored card that
shares a scenario card's name would be adopted and overwritten.
"""
return {c.source_ref: c for c in adventure.story_cards if c.source_ref}
def _match_legacy(
adventure: models.Adventure, specs: dict[str, dict]
) -> dict[str, models.StoryCard]:
by_name: dict[str, models.StoryCard] = {}
for card in adventure.story_cards:
by_name.setdefault((card.name or "").strip().lower(), card)
matched: dict[str, models.StoryCard] = {}
for ref, spec in specs.items():
card = by_name.get((spec["name"] or "").strip().lower())
if card is not None:
matched[ref] = card
return matched
def plan_refresh(
adventure: models.Adventure, scenario: models.Scenario, values: dict[str, str]
) -> tuple[dict, dict, dict]:
"""Computes what a refresh would change, without modifying anything.
The return value is `(plan, specs, matched)`. `plan` is the summary the UI
shows, `specs` holds the scenario's card specs by ref, and `matched` holds
the existing adventure card for each ref that already has one.
"""
fields = {
field: {"old": getattr(adventure, field), "new": fill_placeholders(
getattr(scenario, field), values)}
for field in SCENARIO_TEXT_FIELDS
}
changed_fields = {f: v for f, v in fields.items() if v["old"] != v["new"]}
specs = scenario_card_specs(scenario, values)
tagged = _scenario_cards(adventure)
matched = tagged or _match_legacy(adventure, specs)
added, updated = [], []
for ref, spec in specs.items():
card = matched.get(ref)
if card is None:
added.append(spec["name"])
elif any(getattr(card, f) != spec[f] for f in CARD_FIELDS):
updated.append(card.name or spec["name"])
# Only a card the scenario produced is removable. A player-authored card
# has no `source_ref` and is never modified.
removed = [c.name for ref, c in tagged.items() if ref not in specs]
_, world = worldstate.reconcile(adventure.world_state, scenario.stat_schema)
plan = {
"scenario_id": scenario.id,
"scenario_title": scenario.title,
"fields": changed_fields,
"cards": {"added": added, "updated": updated, "removed": removed},
"world_state": world,
}
plan["has_changes"] = bool(
changed_fields or added or updated or removed
or world["added"] or world["removed"]
)
return plan, specs, matched
@router.get("/{adventure_id}/refresh", response_model=schemas.RefreshPlan)
def preview_refresh(
db: Session = Depends(get_db),
user: models.User = CurrentUser,
adventure: models.Adventure = Depends(current_adventure),
):
"""Returns what "Update from scenario" would change, for the confirm dialog."""
scenario = resolve_source_scenario(adventure, db, user)
if scenario is None:
raise HTTPException(404, "No scenario to update from")
stored = adventure.placeholders if isinstance(adventure.placeholders, dict) else {}
plan, _, _ = plan_refresh(adventure, scenario, stored)
# An adventure started before placeholder answers were stored has none, and
# an author can add a new `${...}` later. In both cases the player is asked
# for the missing names, and the answers are saved for next time.
plan["placeholders_needed"] = [
n for n in scenario_placeholder_names(scenario) if n not in stored
]
return plan
@router.post("/{adventure_id}/refresh", response_model=schemas.AdventureOut)
def refresh_from_scenario(
adventure_id: int,
payload: schemas.AdventureRefresh = Body(default=schemas.AdventureRefresh()),
db: Session = Depends(get_db),
user: models.User = CurrentUser,
adventure: models.Adventure = Depends(current_adventure),
):
"""Copies the scenario's current plot text, story cards, and stat schema over
this adventure's copy.
The refresh overwrites the plot fields and every scenario-derived card, adds
what the scenario gained, and removes what it dropped.
The refresh leaves these unchanged: the opening `start` action, because the
story is built on it and it is already part of the memories and the summary;
the adventure's own title; its story summary; its player-authored story
cards; and, through `worldstate.reconcile`, the live value of every stat the
schema still defines.
"""
scenario = resolve_source_scenario(adventure, db, user)
if scenario is None:
raise HTTPException(404, "No scenario to update from")
values = {**(adventure.placeholders if isinstance(adventure.placeholders, dict) else {}),
**payload.placeholders}
# A refresh rewrites the same state a turn is part-way through changing, so
# it takes the turn slot rather than run at the same time as the
# generator.
turns.acquire_turn_lock(adventure_id)
try:
_, specs, matched = plan_refresh(adventure, scenario, values)
for field in SCENARIO_TEXT_FIELDS:
setattr(adventure, field, fill_placeholders(getattr(scenario, field), values))
for ref, spec in specs.items():
card = matched.get(ref)
if card is None:
db.add(models.StoryCard(adventure_id=adventure.id, source_ref=ref, **spec))
continue
for field in CARD_FIELDS:
setattr(card, field, spec[field])
# Store the ref, so that a name-matched legacy card syncs by id
# next time.
card.source_ref = ref
for ref, card in _scenario_cards(adventure).items():
if ref not in specs:
db.delete(card)
adventure.world_state, _ = worldstate.reconcile(
adventure.world_state, scenario.stat_schema
)
adventure.placeholders = values
db.commit()
finally:
turns._active_turns.discard(adventure_id)
db.refresh(adventure)
return adventure
backend/app/routers/adventures/scenario_text.py
+106
View File
@@ -0,0 +1,106 @@
"""Copying a scenario's text and story cards onto an adventure.
An adventure holds its own copy of the scenario's plot text and cards, so that
later authoring does not change a story in progress. Two callers make that copy:
`crud.create_adventure` on the way in, and `refresh` when the player asks for the
scenario's current content.
"""
import re
from ... import models, worldstate
PLACEHOLDER_RE = re.compile(r"\$\{([^}]+)\}")
def fill_placeholders(text: str, values: dict[str, str]) -> str:
"""Replaces `${Name}` with the player-provided value.
Unknown names are left unchanged.
"""
if not text or not values:
return text
return PLACEHOLDER_RE.sub(
lambda m: values.get(m.group(1).strip(), m.group(0)), text
)
# Adventure fields that start as a copy of the scenario's text, so "Update from
# scenario" can copy them again. `title` is excluded because it is the
# adventure's own name, which players rename. `story_summary` is excluded
# because it is play output rather than scenario content.
SCENARIO_TEXT_FIELDS = ("memory", "authors_note", "ai_instructions")
# Story-card fields that are copied from the scenario and compared to detect
# drift.
CARD_FIELDS = ("type", "name", "keys", "entry", "notes")
def scenario_card_specs(scenario: models.Scenario, values: dict[str, str]) -> dict[str, dict]:
"""Returns every story card a scenario implies, keyed by a stable `source_ref`.
The result holds the scenario's own cards, keyed `card:<id>`, plus one card
per NPC defined in its `stat_schema`, keyed `npc:<key>`. Placeholders are
already filled in.
Adventure creation and refresh both call this function, so the two cannot
diverge.
"""
specs: dict[str, dict] = {}
existing_names = {(c.name or "").strip().lower() for c in scenario.story_cards}
for card in scenario.story_cards:
specs[f"card:{card.id}"] = {
"type": card.type,
"name": card.name,
"keys": fill_placeholders(card.keys, values),
"entry": fill_placeholders(card.entry, values),
"notes": card.notes,
}
# Phase 12: each defined NPC gets a story card, so its description works as
# lore and can trigger in a scene. If a card with that name already exists,
# skip the NPC.
for npc_key, ndef in (scenario.stat_schema or {}).get("npcs", {}).items():
if not isinstance(ndef, dict):
continue
name = worldstate.npc_name(ndef, npc_key)
if name.strip().lower() in existing_names:
continue
specs[f"npc:{npc_key}"] = {
"type": "character",
"name": name,
"keys": fill_placeholders(str(ndef.get("keys") or name), values),
"entry": fill_placeholders(str(ndef.get("desc") or ""), values),
"notes": "",
}
return specs
def _placeholder_names(*texts: str) -> list[str]:
"""Returns the unique `${Placeholder}` names across the given texts.
The order is first appearance first. This matches the frontend's
`extractPlaceholders`.
"""
names: list[str] = []
for text in texts:
for match in PLACEHOLDER_RE.finditer(text or ""):
name = match.group(1).strip()
if name and name not in names:
names.append(name)
return names
def scenario_placeholder_names(scenario: models.Scenario) -> list[str]:
"""Returns every placeholder the scenario's refreshable content asks for.
The opening prompt is excluded, because a refresh never rewrites it.
"""
texts = [scenario.memory, scenario.authors_note, scenario.ai_instructions]
for card in scenario.story_cards:
texts += [card.keys, card.entry]
for ndef in (scenario.stat_schema or {}).get("npcs", {}).values():
if isinstance(ndef, dict):
texts += [str(ndef.get("keys") or ""), str(ndef.get("desc") or "")]
return _placeholder_names(*texts)
backend/app/routers/adventures/scripts.py
+116
View File
@@ -0,0 +1,116 @@
"""The per-adventure copies of library scripts.
An adventure snapshots a library `Script` when it starts, so editing the library
does not change a story in progress. These endpoints report whether a snapshot
has fallen behind its library original, and copy the original over on request.
"""
from fastapi import Depends, HTTPException
from sqlalchemy.orm import Session
from ... import models, schemas
from ...database import get_db
from .deps import CurrentUser, current_adventure, router
# Fields that are copied from a library Script into its adventure-script
# snapshot, and compared to decide whether a copy is out of date.
SYNC_FIELDS = ("name", "description", "library_js", "input_js", "context_js", "output_js")
def resolve_library_script(
adv_script: models.AdventureScript, db: Session, user: models.User
) -> models.Script | None:
"""Returns the library Script an adventure script can re-sync from.
The result is the script this copy was made from. For a legacy copy with no
link, it is one of the player's own scripts with the same name. Only the
player's own scripts are considered, so a copy derived from a demo scenario
has nothing to sync to.
"""
if adv_script.source_script_id is not None:
script = db.get(models.Script, adv_script.source_script_id)
if script is not None and script.user_id == user.id:
return script
return (
db.query(models.Script)
.filter(models.Script.user_id == user.id, models.Script.name == adv_script.name)
.order_by(models.Script.updated_at.desc())
.first()
)
def _mark_out_of_date(
adv_script: models.AdventureScript, db: Session, user: models.User
) -> models.AdventureScript:
"""Attaches a transient `out_of_date` flag, which `AdventureScriptOut` reads.
The flag is `True` or `False` when a syncable library version exists, and
`None` when none exists.
"""
library = resolve_library_script(adv_script, db, user)
adv_script.out_of_date = (
None if library is None
else any(getattr(adv_script, f) != getattr(library, f) for f in SYNC_FIELDS)
)
return adv_script
@router.get("/{adventure_id}/scripts", response_model=list[schemas.AdventureScriptOut])
def list_adventure_scripts(
db: Session = Depends(get_db),
user: models.User = CurrentUser,
adventure: models.Adventure = Depends(current_adventure),
):
return [_mark_out_of_date(s, db, user) for s in adventure.scripts]
@router.post(
"/{adventure_id}/scripts/{adv_script_id}/sync",
response_model=schemas.AdventureScriptOut,
)
def sync_adventure_script(
adventure_id: int,
adv_script_id: int,
db: Session = Depends(get_db),
user: models.User = CurrentUser,
adventure: models.Adventure = Depends(current_adventure),
):
"""Overwrites this copy's code with the latest from its library script.
`enabled`, `position`, and the adventure's shared `script_state` are kept.
"""
script = db.get(models.AdventureScript, adv_script_id)
if script is None or script.adventure_id != adventure_id:
raise HTTPException(404, "Script not found")
library = resolve_library_script(script, db, user)
if library is None:
raise HTTPException(404, "No library script to sync from")
for field in SYNC_FIELDS:
setattr(script, field, getattr(library, field))
# Store the link, so that a name-matched legacy copy syncs by id next
# time.
script.source_script_id = library.id
db.commit()
db.refresh(script)
return _mark_out_of_date(script, db, user)
@router.patch(
"/{adventure_id}/scripts/{adv_script_id}", response_model=schemas.AdventureScriptOut
)
def update_adventure_script(
adventure_id: int,
adv_script_id: int,
payload: schemas.AdventureScriptUpdate,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
script = db.get(models.AdventureScript, adv_script_id)
if script is None or script.adventure_id != adventure_id:
raise HTTPException(404, "Script not found")
for field, value in payload.model_dump(exclude_unset=True).items():
setattr(script, field, value)
db.commit()
return script
backend/app/routers/adventures/takes.py
+388
View File
@@ -0,0 +1,388 @@
"""Retries, takes, and the attempts that pile up at one coordinate.
Retry first deleted the AI action and generated a replacement. A later version
kept the row and appended each attempt to a JSON list on it. Now every attempt is
its own node on the same branch at the same depth, and exactly one of them is
`live`. `app/attempts.py` owns the group and its invariants. The endpoints here
only query it.
"""
from fastapi import Depends, HTTPException, Request
from fastapi.responses import StreamingResponse
from sqlalchemy.orm import Session
from ... import attempts, limits, memorybank, models, schemas, tree
from ...context import cursors
from ...context import lineage
from ...database import get_db
from ...scripting import ScriptPipeline
from ...sse import SSE_HEADERS
from . import turns
from .deps import CurrentUser, current_adventure, router
from .nodes import delete_turn, last_action, stand_on
from .paging import action_window, annotate_takes, current_window
@router.post("/{adventure_id}/retry")
def retry_action(
adventure_id: int,
request: Request,
db: Session = Depends(get_db),
user: models.User = CurrentUser,
adventure: models.Adventure = Depends(current_adventure),
):
"""Regenerates the last AI action and keeps the discarded attempt.
The attempt on screen stays as it was written. The shared script state and
world state roll back to what the node before it left behind, and the new
attempt is stored as a sibling at the same coordinate. No text the AI wrote
is rewritten or deleted.
"""
limits.rate_limit("turn", request, user)
turns.check_demo_cap(db, user)
turns.acquire_turn_lock(adventure_id)
last_ai = None
try:
newest = last_action(adventure, db)
if newest is not None and newest.type == "ai":
last_ai = newest
# Roll the state back to before this AI turn's hooks ran, so that
# regenerating starts from a clean state rather than applying output
# mutations on top of the attempt being replaced. If the preceding
# node has no snapshot, which happens for a pre-SP4 row that the
# migration could not derive one for, this call does nothing and
# leaves the state as it is.
attempts.roll_back_before(db, adventure, last_ai)
db.commit()
db.refresh(adventure)
except BaseException:
turns._active_turns.discard(adventure_id)
raise
return StreamingResponse(
turns.with_turn_lock(
adventure_id,
turns.generate_turn(
adventure, db, ScriptPipeline(adventure, db), user, retry_of=last_ai
),
),
media_type="text/event-stream",
headers=SSE_HEADERS,
)
@router.get(
"/{adventure_id}/actions/{action_id}/variants",
response_model=list[schemas.VariantOut],
)
def list_variants(
adventure_id: int,
action_id: int,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Returns every attempt made for one AI turn.
The client fetches these on demand, because the adventure payload carries
only the counts. That keeps old narration out of every page load.
You can address the turn by any of its attempts, not only the live one.
Switching changes which row the story tells, and a client that holds an id it
received a moment ago still has to be able to ask about the same turn.
"""
action = db.get(models.Action, action_id)
if action is None or action.adventure_id != adventure_id:
raise HTTPException(404, "Action not found")
rows = attempts.group(db, action)
if len(rows) < 2:
return [] # Never retried, so the turn has one attempt.
return [
schemas.VariantOut(
id=row.id,
index=i,
text=row.text,
reasoning=row.reasoning,
branch_id=row.branch_id,
created_at=row.created_at.isoformat() if row.created_at else None,
active=row.live,
)
for i, row in enumerate(rows)
]
@router.post(
"/{adventure_id}/actions/{action_id}/variant", response_model=schemas.ActionOut
)
def select_variant(
adventure_id: int,
action_id: int,
payload: schemas.VariantSelect,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Makes an earlier attempt live again and restores the state it produced.
The restored state covers both the script state and the world state.
Only the last action can be switched. The turns after an older action were
written to continue the text that is currently active, so replacing that text
would leave the story contradicting itself. The attempts of earlier turns
stay readable through `list_variants`.
"""
action = db.get(models.Action, action_id)
if action is None or action.adventure_id != adventure_id:
raise HTTPException(404, "Action not found")
rows = attempts.group(db, action)
if not 0 <= payload.index < len(rows) or len(rows) < 2:
raise HTTPException(400, "No such attempt for this action")
newest = last_action(adventure, db)
if newest is None or newest.depth != action.depth or newest.branch_id != action.branch_id:
raise HTTPException(
400,
"Only the latest message can be switched — the story has already "
"continued from this one.",
)
turns.acquire_turn_lock(adventure_id)
try:
chosen = rows[payload.index]
if not chosen.live:
# The story at this coordinate is about to change, so withdraw
# anything derived from the previous text. A retry does the same
# thing for the same reason.
memorybank.forget_node(db, adventure, chosen)
cursors.rewind_all(adventure, chosen.branch_id, (chosen.depth or 0) - 1)
attempts.make_live(db, adventure, chosen)
adventure.updated_at = models.utcnow()
db.commit()
db.refresh(chosen)
# Return the row that is now in the story, which is a different row
# from the one the request addressed. An attempt is a node, so choosing
# one moves the story onto it rather than rewriting a row.
return chosen
finally:
turns._active_turns.discard(adventure_id)
@router.post(
"/{adventure_id}/actions/{action_id}/fork", response_model=schemas.ActionPage
)
def fork_from_attempt(
adventure_id: int,
action_id: int,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Continues the story from this attempt, forking a branch if one is needed.
There are three cases, and the first two do not fork:
* The attempt is already the one the story tells, so there is nothing to do.
* Its turn is the tip, so the attempts are still leaves that nothing was
built on. The endpoint switches, as `/variant` does, and creates no branch.
* The story has moved past its turn, so the endpoint forks. The attempt gets
a branch of its own, and the line it leaves keeps every turn it has.
"""
action = db.get(models.Action, action_id)
if action is None or action.adventure_id != adventure_id:
raise HTTPException(404, "Action not found")
# Check this before checking the shape of the turn, because a fork leaves
# the promoted attempt alone on its branch. A client that repeats the call,
# after a double click or a retried request, has to get the same answer
# rather than an error saying the turn it just forked has nothing to fork
# to.
if action.live:
# A live node already holds what its coordinate says, so there is no
# attempt here to promote. On the path being read this call does
# nothing, and it has to stay that way, so that a repeated call after a
# double click or a retried request gets the same answer. Off the path
# the node belongs to another line's story, and moving there is a branch
# switch.
#
# The membership test covers the whole lineage, not `head_branch_id`. A
# head borrows its ancestors' turns, so a live node on an ancestor is
# already being read. Forking it would move the live row off the parent
# and promote a sibling in its place, which rewrites the story on a
# branch nobody asked about and on this one, which borrows that depth.
if lineage.path_of(db, adventure).contains(action):
return current_window(db, adventure)
raise HTTPException(
400,
"That take is already the story on another branch. Switch to that "
"branch to read it.",
)
if len(attempts.group(db, action)) < 2:
raise HTTPException(
400, "This turn has only one take, so there is nothing to fork to."
)
turns.acquire_turn_lock(adventure_id)
try:
stand_on(db, adventure, action)
adventure.updated_at = models.utcnow()
db.commit()
db.refresh(adventure)
return current_window(db, adventure)
finally:
turns._active_turns.discard(adventure_id)
@router.post("/{adventure_id}/actions/{action_id}/takes")
def add_take(
adventure_id: int,
action_id: int,
payload: schemas.TakeCreate,
request: Request,
db: Session = Depends(get_db),
user: models.User = CurrentUser,
adventure: models.Adventure = Depends(current_adventure),
):
"""Plays a turn again, whoever wrote it.
This endpoint replaces two earlier operations. `retry` gave an AI turn
another attempt, but only for the newest turn, and a player's own message had
no attempts at all, so changing text you had typed meant overwriting it and
losing the story it led to. Here an AI turn regenerates, a player turn takes
the text you supply, and neither depends on where in the story it sits.
The tip is the only case that needs no branch, and only for an AI turn,
because nothing was played after it and its attempts are still leaves. A
player turn is never at the tip, since the reply to it is, so a player turn
that has been answered always takes a branch.
A branch is needed here for the same reason `fork` needs one. The turn being
replayed already has a story after it, and that story was written as a
continuation of the old text. `branch_at` leaves the path just before this
turn, so the new attempt is written at the same depth under the same parent,
and the line it leaves is unchanged. No node below is copied.
"""
limits.rate_limit("turn", request, user)
limits.check_row_cap("actions", db, user, adventure=adventure)
turns.check_demo_cap(db, user)
action = db.get(models.Action, action_id)
if action is None or action.adventure_id != adventure_id:
raise HTTPException(404, "Action not found")
if action.type not in ("do", "say", "story", "continue", "ai"):
# The opening is not a turn anyone played, so it has no second attempt.
# Editing the scenario is what changes it.
raise HTTPException(400, "The opening of a story has no other take.")
if action.depth is None or not lineage.path_of(db, adventure).contains(action):
raise HTTPException(400, "That turn is not on the story you are reading.")
turns.acquire_turn_lock(adventure_id)
retry_of = None
try:
newest = last_action(adventure, db)
at_the_tip = newest is not None and newest.id == action.id
if at_the_tip and action.type == "ai":
# Nothing was played after it, so its attempts are still leaves and
# a branch would serve no purpose. This is the `retry` path.
retry_of = action
attempts.roll_back_before(db, adventure, action)
else:
# The turn has a story after it, written as a continuation of the
# text that is there now. The new attempt leaves the path just
# before the turn, so that story keeps the attempt it was written
# for.
tree.branch_at(db, adventure, action.depth - 1)
attempts.roll_back_before(db, adventure, action)
adventure.updated_at = models.utcnow()
db.commit()
db.refresh(adventure)
except BaseException:
turns._active_turns.discard(adventure_id)
raise
if action.type == "ai":
# There is no player action to write. The action this turn answers is
# already on the path, borrowed from the line being left.
stream = turns.generate_turn(
adventure, db, ScriptPipeline(adventure, db), user, retry_of=retry_of
)
else:
stream = turns.run_player_turn(
adventure,
db,
schemas.ActionCreate(type=action.type, text=payload.text),
user,
# The client seeded its editor from the stored text, which already
# carries the "> You ..." conventions.
preformatted=True,
)
return StreamingResponse(
turns.with_turn_lock(adventure_id, stream),
media_type="text/event-stream",
headers=SSE_HEADERS,
)
@router.post("/{adventure_id}/undo", response_model=schemas.ActionPage)
def undo_turn(
adventure_id: int,
db: Session = Depends(get_db),
adventure: models.Adventure = Depends(current_adventure),
):
"""Deletes the last turn: the trailing AI action and its player action, if any.
The endpoint also rolls the shared `script_state` back to before that turn
ran, and it prunes any memory that summarized the removed actions. The turn
lock prevents an undo while a turn is still generating.
"""
turns.acquire_turn_lock(adventure_id)
try:
# Only the last turn is removed, so fetch the two actions it can
# consist of rather than the whole story.
newest = (
db.query(models.Action)
.filter(
models.Action.adventure_id == adventure.id,
lineage.path_of(db, adventure).clause(models.Action),
)
.order_by(models.Action.depth.desc(), models.Action.id.desc())
.limit(2)
.all()
)
if not newest or newest[0].type == "start":
raise HTTPException(400, "Nothing to undo")
last = newest[0]
before_that = newest[1] if len(newest) > 1 else None
# Undo only what this branch owns. Everything before the fork is
# borrowed from an ancestor and is part of that ancestor's story too, so
# an undo here must never delete a turn out of another branch. The test
# reads the row's own branch rather than the fork depth, because the
# branch is what decides the case.
if last.branch_id != adventure.head_branch_id:
raise HTTPException(
400, "Nothing to undo on this branch — the turns before it "
"belong to the branch it was forked from.",
)
first_removed = last
if (last.type == "ai" and before_that is not None
and before_that.type in ("do", "say", "story")
and before_that.branch_id == adventure.head_branch_id):
first_removed = before_that
# The state the story returns to once the turn is gone, which is what
# the node before the earliest removed one left behind. Read it before
# the deletes, while those rows are still in the story.
restore_to = attempts.preceding(db, adventure, first_removed)
delete_turn(db, adventure, last)
if first_removed is not last:
delete_turn(db, adventure, first_removed)
attempts.restore_state(adventure, restore_to)
db.flush() # Apply the deletes before anything reads the story back.
db.expire(adventure, ["actions"])
# The tip moves back with the deleted rows.
tree.refresh_head(db, adventure)
db.commit()
db.refresh(adventure)
# Return the newest window rather than the whole story. The client
# replaces its transcript with this response, and the transcript is a
# window. Returning everything would defeat the paging on the action a
# player is most likely to repeat several times in a row.
actions, total, has_more = action_window(db, adventure)
return schemas.ActionPage(
actions=[
schemas.ActionOut.model_validate(a)
for a in annotate_takes(db, adventure.id, actions)
],
total=total,
has_more=has_more,
)
finally:
turns._active_turns.discard(adventure_id)
backend/app/routers/adventures/turns.py
+421
View File
@@ -0,0 +1,421 @@
"""Playing a turn: the model call, the SSE stream, and the one-turn-at-a-time lock.
Everything a test needs to intercept lives here, and other modules reach it as
`turns.<name>` rather than importing it by value. That matters twice. The turn
lock guards one set only while one module owns it. And a test that replaces
`OpenAICompatibleProvider`, `generate_turn`, or `check_demo_cap` patches this
module, which every caller reads through.
"""
import threading
from fastapi import Depends, HTTPException, Request
from fastapi.responses import StreamingResponse
from sqlalchemy.orm import Session
from ... import (
analytics, attempts, auth, limits, memorybank, models, schemas, tree, worldstate,
)
from ...context import build_context, cursors
from ...database import get_db
from ...providers import OpenAICompatibleProvider, PromptParts, ProviderError
from ...scripting import ScriptPipeline
from ...sse import SSE_HEADERS, sse, turn_error
from ..settings import get_settings
from .deps import CurrentUser, current_adventure, router
from .nodes import _move_to_after, next_depth
from .paging import annotate_takes
def world_delta_of(snapshot: dict | None) -> dict | None:
"""Returns the bulk-read slice of a context snapshot, for `Action.world_delta`.
`context_snapshot` is deferred because it holds the whole assembled prompt.
The parts that every action needs get their own small column instead: the
world-change chips, the emit block replayed into history, and the refusal
note fed back to the model. Update this function wherever a snapshot is
written.
Carry all three report lists, not just `applied`. `Action.world_changes`
marks a chip from `clamped` and builds its refusal chips from `rejected`,
and `worldstate.refusals` reads both. Storing `applied` alone left every
consumer unable to tell a refused change from one that worked, which is the
distinction this column exists to carry. The two extra lists are subsets of
one turn's block, so they cost a few hundred bytes per action at most.
"""
ws = (snapshot or {}).get("world_state")
if not isinstance(ws, dict):
return None
report = ws.get("report") or {}
return {
"delta": ws.get("delta") or {},
"applied": report.get("applied") or [],
"clamped": report.get("clamped") or [],
"rejected": report.get("rejected") or [],
}
# One turn at a time per adventure. The set lives in memory, which is enough for
# a single-process local app. Sync endpoints run in a threadpool, so the
# check-and-add needs a lock. The check also has to run during the request
# rather than when the SSE generator first runs. Otherwise two rapid requests
# both pass the check and generate concurrently.
_active_turns: set[int] = set()
_active_turns_guard = threading.Lock()
def acquire_turn_lock(adventure_id: int):
"""Claims the adventure's turn slot atomically. `with_turn_lock` releases it."""
with _active_turns_guard:
if adventure_id in _active_turns:
raise HTTPException(409, "A turn is already generating for this adventure.")
_active_turns.add(adventure_id)
async def with_turn_lock(adventure_id: int, gen):
"""Wraps an SSE generator so that it releases the `acquire_turn_lock` lock."""
try:
async for event in gen:
yield event
finally:
_active_turns.discard(adventure_id)
def format_player_input(action_type: str, text: str) -> str:
"""Formats player input the way AI Dungeon does."""
text = text.strip()
if action_type == "say":
text = text.strip('"')
if text and text[-1] not in ".!?…":
text += "."
return f'> You say "{text}"'
if action_type == "do":
if text.lower().startswith("you "):
text = text[4:]
if text and text[-1] not in ".!?…":
text += "."
return f"> You {text}"
return text # The "story" type is appended as raw text.
def action_json(action: models.Action, db: Session | None = None) -> dict:
"""Serializes one action for the wire.
Passing `db` fills in the pager numbers, and a turn that was just played must
pass it. The attempt that turn created is often the second one at its
coordinate, so the message needs a pager that the client cannot infer from a
count of one. Without `db`, a retry showed no pager until the page reloaded.
The adventure GET has the same requirement and builds its window a third way.
"""
if db is not None:
annotate_takes(db, action.adventure_id, [action])
return schemas.ActionOut.model_validate(action).model_dump(mode="json")
async def generate_turn(
adventure: models.Adventure,
db: Session,
pipeline: ScriptPipeline,
user: models.User,
retry_of: models.Action | None = None,
):
"""Streams the AI continuation as SSE, then stores the result.
The continuation passes through the `context` and `output` script hooks.
If `retry_of` is set, the result is stored as a sibling of that AI action, at
the same turn and the same coordinate, and the discarded attempt stays where
it was written. Before calling, the caller must roll the adventure back to
the state before that turn. See `retry_action`. If this generator ends
without saving, the rollback is undone, so the state cannot diverge from the
text on screen.
"""
saved = False
try:
async for event in _generate_turn(adventure, db, pipeline, user, retry_of):
if event is _SAVED:
saved = True
continue
yield event
finally:
if retry_of is not None and not saved:
# The turn failed with a provider error, an empty reply, a script
# stop, or a disconnected client. No sibling was written, so the
# attempt on screen is still the live one. Restore the state it
# produced.
attempts.restore_state(adventure, retry_of)
db.commit()
# `_generate_turn` yields this sentinel once the action is committed. It tells
# the wrapper above to leave the rollback in place rather than reverse it.
_SAVED = object()
async def _generate_turn(
adventure: models.Adventure,
db: Session,
pipeline: ScriptPipeline,
user: models.User,
retry_of: models.Action | None = None,
):
settings = get_settings(db, user)
cfg = auth.resolve_provider_config(settings)
# On a retry, the attempt being replaced is still the live node of its turn,
# because it stays live until a replacement exists. Filter it out of the
# context. Otherwise the model reads the attempt it is replacing as
# established story and writes a sequel to it.
replacing_id = retry_of.id if retry_of is not None else None
if cfg.using_demo:
# The server-funded key makes no embedding or summarization calls, so
# memory retrieval is skipped. If the bank is on, return a note.
memories = (
{"used": [], "error": "Memory bank is unavailable on the shared demo key — add your own API key in Settings."}
if adventure.memory_bank_enabled
else None
)
else:
memories = await memorybank.retrieve_memories(
adventure, settings, update_stats=True, exclude_action_id=replacing_id
)
system_text, story_text, snapshot = build_context(
adventure, settings, memories, exclude_action_id=replacing_id
)
# onModelContext: scripts read, and can rewrite, the whole assembled
# context.
combined = f"{system_text}\n\n{story_text}" if system_text else story_text
modified, stop = pipeline.run("context", combined)
if stop:
yield sse({"type": "stopped", "script": pipeline.report()})
return
context_changed = modified != combined
parts = (
PromptParts(system="", story=modified)
if context_changed
else PromptParts(system=system_text, story=story_text)
)
snapshot["script"] = pipeline.report() | {
"context_changed": context_changed,
"context_before": combined if context_changed else None,
"context_after": modified if context_changed else None,
}
provider = OpenAICompatibleProvider(
cfg.endpoint_url, cfg.api_key, cfg.model, settings.api_mode,
settings.reasoning_max_tokens,
)
chunks: list[str] = []
reasoning_chunks: list[str] = []
try:
async for kind, chunk in provider.generate(
parts, temperature=settings.temperature, max_tokens=settings.max_output_tokens
):
if kind == "reasoning":
reasoning_chunks.append(chunk)
yield sse({"type": "reasoning", "text": chunk})
else:
chunks.append(chunk)
yield sse({"type": "chunk", "text": chunk})
except ProviderError as exc:
yield turn_error(str(exc))
return
text = "".join(chunks).strip()
# The model's literal reply, kept for the Insights "Raw AI output" view. It
# still contains the world-state block, which the code below strips.
raw_output = text
if not text:
# The model streamed reasoning but no story text, so it spent its whole
# budget on reasoning. Report that rather than "empty response".
if reasoning_chunks:
detail = (
"The model used its entire token budget on reasoning and returned no "
'story text. Raise "Max output tokens" in Settings, set a "Reasoning '
'max tokens" cap, or switch to a non-reasoning model.'
)
else:
detail = "The AI returned an empty response."
yield turn_error(detail)
return
# onOutput
text, _ = pipeline.run("output", text)
if not text.strip():
yield turn_error("A script's output modifier returned empty text.")
return
snapshot["script"] = snapshot["script"] | pipeline.report()
# RPG world state (Phase 12): read the AI's state delta out of the reply,
# apply it through the engine, and strip the block from the displayed text.
#
# A retry re-runs the same turn, so it is played at that turn's depth. The
# cooldown rules run on a position in the story, and a second attempt at turn
# 12 is still turn 12. This was `retry_of.index`, which held the same number
# until SP4. Depth stays correct once a branch has its own numbering.
ai_depth = retry_of.depth if retry_of is not None else next_depth(adventure)
stat_schema = adventure.scenario.stat_schema if adventure.scenario else None
if worldstate.has_schema(stat_schema):
text, delta = worldstate.extract_delta(text)
if not text.strip():
yield turn_error("The AI returned only a state update and no story text.")
return
new_world_state, ws_report = worldstate.apply_delta(
adventure.world_state, stat_schema, delta, ai_depth
)
adventure.world_state = new_world_state
snapshot["world_state"] = {"delta": delta, "report": ws_report, "state": new_world_state}
snapshot["raw_output"] = raw_output
# The cost the endpoint reports for the call, including how much of the
# prompt came from cache rather than being billed in full. This is recorded
# per attempt, next to the prompt it priced.
snapshot["usage"] = provider.last_usage
reasoning = "".join(reasoning_chunks).strip() or None
ai_action = models.Action(
adventure_id=adventure.id,
depth=ai_depth,
type="ai",
text=text,
reasoning=reasoning,
context_snapshot=snapshot,
world_delta=world_delta_of(snapshot),
)
attempts.snapshot_outcome(adventure, ai_action)
if retry_of is not None:
attempts.add_attempt(db, adventure, retry_of, ai_action)
db.add(ai_action)
# The text at this coordinate changed, so anything derived from it no
# longer describes the story. Withdraw the memory attached to the node
# and return that stretch to both passes. Before SP4 this code was
# unreachable, because the summarizer held the newest action back until
# a turn landed on top of it. `memorybank.SETTLE_SLACK` keeps a memory
# off the tip again, for cost rather than for correctness, so this is
# now the rare case: undo or delete can carry a summarized node back to
# the tip, and then a retry of it lands here. See `memorybank`.
memorybank.forget_node(db, adventure, retry_of)
cursors.rewind_all(adventure, retry_of.branch_id, ai_depth - 1)
# Flush so the new attempt has an id. The session does not autoflush,
# and attempts page in id order, so a read taken before this point puts
# the newest attempt nowhere.
db.flush()
else:
tree.place_action(db, adventure, ai_action)
db.add(ai_action)
adventure.updated_at = models.utcnow()
if cfg.using_demo:
# Successful demo turns count against the daily cap, which the endpoint
# checks before the turn starts. A failed provider call above returns
# before this line.
auth.count_demo_turn(user)
db.commit()
# Count the turn here, after every path on which it could still have failed,
# so the number means "stories advanced" rather than "requests attempted".
# The demo tally counts those same turns as spend on the server-funded key.
analytics.record_event(analytics.EV_TURN, user)
if cfg.using_demo:
analytics.record(analytics.M_EVENT, analytics.EV_DEMO_TURN)
db.refresh(ai_action)
yield _SAVED
yield sse({"type": "done", "action": action_json(ai_action, db), "script": pipeline.report()})
# Phase 6: schedule summarization and embedding without waiting for them.
# The task opens its own database session. It is skipped on the demo key,
# because background AI calls are unmetered spend on the server-funded
# key.
if not cfg.using_demo:
memorybank.schedule_post_turn(adventure)
def check_demo_cap(db: Session, user: models.User) -> None:
"""Checks the demo cap before a turn starts.
Checking first avoids storing a capped player's input and then leaving it
without a reply.
"""
settings = get_settings(db, user)
if auth.resolve_provider_config(settings).using_demo and auth.demo_turns_left(user) <= 0:
raise HTTPException(429, auth.DEMO_CAP_MESSAGE)
async def run_player_turn(
adventure: models.Adventure,
db: Session,
payload: schemas.ActionCreate,
user: models.User,
preformatted: bool = False,
):
"""Plays a player's turn: their action, then the reply to it.
`preformatted` means the text already carries the `> You ...` conventions and
is written as-is. That applies when the player retakes a turn they already
played (SP9). The editor is seeded with the stored text, which is already
formatted, and a plain edit puts that same text in the box and writes it back
verbatim. Formatting it a second time produces `> You > You ...`.
"""
pipeline = ScriptPipeline(adventure, db)
# An empty do, say, or story action behaves as a continue.
if payload.type != "continue" and payload.text.strip():
# onInput reads the formatted text, as in AI Dungeon: "> You ...".
formatted = (
payload.text.strip() if preformatted
else format_player_input(payload.type, payload.text)
)
modified, stop = pipeline.run("input", formatted)
if not modified.strip():
yield turn_error("A script's input modifier returned empty text.",
script=pipeline.report())
return
player_action = models.Action(
adventure_id=adventure.id,
depth=next_depth(adventure),
type=payload.type,
text=modified,
)
# The state after the input hook has run. The node leaves this state
# behind. The AI turn after it starts here, and a retry of that turn
# rolls back to here.
attempts.snapshot_outcome(adventure, player_action)
tree.place_action(db, adventure, player_action)
db.add(player_action)
db.commit()
db.refresh(player_action)
# The new action was added through its foreign key, so the loaded
# `adventure.actions` collection is stale. Without this expire,
# `build_context` for the AI action does not see the player action
# that was just saved.
db.expire(adventure, ["actions"])
yield sse({"type": "player", "action": action_json(player_action, db)})
if stop:
# If onInput returns `{ stop: true }`, skip the AI call.
yield sse({"type": "stopped", "script": pipeline.report()})
return
async for event in generate_turn(adventure, db, pipeline, user):
yield event
@router.post("/{adventure_id}/actions")
def create_action(
adventure_id: int,
payload: schemas.ActionCreate,
request: Request,
db: Session = Depends(get_db),
user: models.User = CurrentUser,
adventure: models.Adventure = Depends(current_adventure),
):
limits.rate_limit("turn", request, user)
limits.check_row_cap("actions", db, user, adventure=adventure)
check_demo_cap(db, user)
acquire_turn_lock(adventure_id)
try:
_move_to_after(db, adventure, payload.after_id)
except BaseException:
_active_turns.discard(adventure_id)
raise
return StreamingResponse(
with_turn_lock(adventure_id, run_player_turn(adventure, db, payload, user)),
media_type="text/event-stream",
headers=SSE_HEADERS,
)
backend/app/routers/analytics.py
+140
View File
@@ -0,0 +1,140 @@
"""Visit analytics: one endpoint the browser writes to, one the owner reads.
The split matters. `/collect` is public and accepts one fact, which page was
viewed, because anything a stranger can POST is a number a stranger can invent.
Everything the dashboard relies on, meaning turns, adventures, sign-ups, demo
spend, and errors, is recorded on the server by the code that performs it, so
those counts are as trustworthy as the app itself.
The two reading endpoints are owner-only and 404 for everyone else, the same
way the AI Chat router does: a feature nobody else can use is better off not
appearing to exist. `/summary` serves the anonymous counters (analytics.py,
which stores nothing that points at a person) and `/access` serves the access
log (accesslog.py, which identifies people on purpose).
"""
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response
from pydantic import BaseModel, Field
from sqlalchemy.orm import Session
from .. import accesslog, analytics, auth, limits, models
from ..database import get_db
router = APIRouter(prefix="/api/analytics", tags=["analytics"])
def owner(
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
) -> models.User:
"""Gates the reading half. It returns 404 rather than 403. See the module
docstring."""
if not auth.is_owner(user):
raise HTTPException(404, "Not found")
return user
Owner = Depends(owner)
class Pageview(BaseModel):
"""What the SPA reports on a page load or a route change.
`first` marks a real page load rather than a client-side navigation. The
facts that describe a visit rather than a view, which are where it came from,
on what kind of device, and from which country, are recorded only on a page
load, so a visitor who clicks through five pages is still one referral.
"""
path: str = Field("", max_length=300)
referrer: str = Field("", max_length=500)
first: bool = False
@router.post("/collect", status_code=204)
def collect(
payload: Pageview,
request: Request,
db: Session = Depends(get_db),
) -> Response:
"""Record one pageview. Always 204, even when nothing was counted: the
browser has no business knowing whether it was."""
limits.rate_limit("analytics", request)
# Resolved by hand rather than through get_current_user: a pageview that
# arrives before /auth/me has minted a session should still be counted as a
# view, not turned into a 401 the SPA has to handle.
user = (
auth.resolve_session_user(request, db)
if auth.MULTI_USER
else auth.local_user(db)
)
# The operator's own clicks are not traffic. This applies only in
# multi-user mode. Locally every user is the owner, and excluding them would
# leave the dashboard empty on the machine the app is developed on.
if auth.MULTI_USER and user is not None and auth.is_owner(user):
return Response(status_code=204)
analytics.record(analytics.M_PAGE, analytics.normalize_route(payload.path))
analytics.record_visit(user)
if payload.first:
referrer = analytics.normalize_referrer(
payload.referrer, request.url.hostname or ""
)
if referrer: # "" means same-origin, which is not a referral
analytics.record(analytics.M_REFERRER, referrer)
analytics.record(
analytics.M_DEVICE,
analytics.device_of(request.headers.get("user-agent", "")),
)
analytics.record(analytics.M_COUNTRY, analytics.country_of(request.headers))
return Response(status_code=204)
@router.get("/summary")
def summary(
days: int = Query(30, ge=1, le=365),
db: Session = Depends(get_db),
_user: models.User = Owner,
) -> dict:
"""Returns the whole dashboard in one aggregate response.
The response is a few kilobytes however much traffic is behind it.
"""
return analytics.summary(db, days)
@router.get("/access")
def access_log(
limit: int = Query(50, ge=1, le=200),
before_id: int | None = Query(None),
kind: str | None = Query(None),
q: str | None = Query(None, max_length=120),
db: Session = Depends(get_db),
_user: models.User = Owner,
) -> dict:
"""A page of the access log, newest first.
Unlike `/summary`, this returns rows about people, which is what it is for.
It is therefore behind the same owner gate, it is paged rather than returned
in full, and the people it describes have no endpoint that reaches it.
"""
page = accesslog.recent(
db, limit=limit, before_id=before_id, kind=kind, query=q
)
return {
"events": [
{
"id": event.id,
"at": event.at.isoformat(),
"kind": event.kind,
"who": event.who,
"is_guest": event.is_guest,
"ip": event.ip,
"country": event.country,
"device": event.device,
"user_agent": event.user_agent,
}
for event in page["events"]
],
"has_more": page["has_more"],
}
backend/app/routers/auth.py
+158
View File
@@ -0,0 +1,158 @@
import re
from fastapi import APIRouter, Depends, HTTPException, Request, Response
from sqlalchemy.orm import Session
from .. import (accesslog, analytics, auth, cleanup, limits, models, schemas,
security, starter)
from ..database import get_db
from .settings import get_settings
router = APIRouter(prefix="/api/auth", tags=["auth"])
EMAIL_RE = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$")
def _set_session_cookie(response: Response, user_id: int) -> None:
response.set_cookie(
auth.SESSION_COOKIE,
security.sign_session(user_id),
max_age=auth.COOKIE_MAX_AGE,
httponly=True,
samesite="lax",
secure=auth.COOKIE_SECURE,
path="/",
)
def me_payload(user: models.User, db: Session) -> dict:
settings = get_settings(db, user)
cfg = auth.resolve_provider_config(settings)
return {
"multi_user": auth.MULTI_USER,
"id": user.id,
"email": user.email,
"is_guest": user.is_guest,
# Trusted testers: unmetered demo turns, plus the AI Chat scratchpad.
"power_user": auth.is_power_user(user),
# Separate allowlist: shows the visit-analytics page and its nav link.
"analytics": auth.is_owner(user),
# How long an idle guest is kept before cleanup deletes it (None when
# the policy is off). Served rather than hardcoded in the UI so the
# number a guest is shown is the number actually enforced.
"guest_retention_days": cleanup.RETENTION_DAYS if cleanup.enabled() else None,
"demo": {
"enabled": auth.demo_enabled(),
"using_demo": cfg.using_demo,
"model": cfg.model if cfg.using_demo else None,
"turns_per_day": auth.DEMO_TURNS_PER_DAY,
"turns_left": auth.demo_turns_left(user) if auth.demo_enabled() else None,
"models": auth.DEMO_MODELS if auth.demo_enabled() else [],
},
}
@router.get("/me")
def me(request: Request, response: Response, db: Session = Depends(get_db)):
"""Returns the current user.
In multi-user mode this also establishes the session. If the cookie is
missing or invalid, the endpoint creates a guest user and sets a cookie. The
frontend calls it on load and after any 401.
"""
if not auth.MULTI_USER:
user = auth.local_user(db)
else:
user = auth.resolve_session_user(request, db)
if user is None:
# Each new guest is a database row, so cap how fast one IP can
# create them.
limits.rate_limit("guest", request)
user = models.User(is_guest=True)
db.add(user)
db.commit()
# The guest is committed first, so a failure while copying the
# starter adventure still leaves them with an account.
starter.give(db, user)
db.commit()
_set_session_cookie(response, user.id)
# This endpoint is the SPA's bootstrap call, so it is where a session first
# shows itself; accesslog thins the rows down to one per day per address.
accesslog.note_session(db, user, request)
return me_payload(user, db)
@router.post("/register")
def register(
payload: schemas.AuthCredentials,
request: Request,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
"""Upgrades the current guest in place.
The `user_id` does not change, so every adventure, scenario, script, and
setting they created as a guest is kept.
"""
if not auth.MULTI_USER:
raise HTTPException(400, "Accounts are disabled in local mode.")
limits.rate_limit("auth", request)
email = payload.email.strip().lower()
if not EMAIL_RE.match(email):
raise HTTPException(422, "Enter a valid email address.")
if len(payload.password) < 8:
raise HTTPException(422, "Password must be at least 8 characters.")
if not user.is_guest:
raise HTTPException(400, "This session is already registered.")
if db.query(models.User).filter(models.User.email == email).first():
raise HTTPException(409, "An account with this email already exists — log in instead.")
user.email = email
user.password_hash = security.hash_password(payload.password)
user.is_guest = False
db.commit()
analytics.record_event(analytics.EV_SIGNUP, user)
accesslog.record(db, accesslog.REGISTER, request, user=user)
return me_payload(user, db)
@router.post("/login")
def login(
payload: schemas.AuthCredentials,
request: Request,
response: Response,
db: Session = Depends(get_db),
):
"""Point this browser's session at an existing account. Any current guest
session is simply abandoned (its data stays under the guest user)."""
if not auth.MULTI_USER:
raise HTTPException(400, "Accounts are disabled in local mode.")
limits.rate_limit("auth", request)
email = payload.email.strip().lower()
# Per-account throttle: stops distributed guessing against one email even
# when the per-IP limit above is diluted across many source addresses.
limits.check_login_allowed(email)
user = db.query(models.User).filter(models.User.email == email).first()
if (
user is None
or not user.password_hash
or not security.verify_password(payload.password, user.password_hash)
):
limits.note_login_failure(email)
# Logged with the address that was tried, not the account that owns it:
# a guessing run against an address that has no account is exactly the
# thing worth being able to see.
accesslog.record(db, accesslog.LOGIN_FAILED, request, who=email)
raise HTTPException(401, "Incorrect email or password.")
limits.note_login_success(email)
_set_session_cookie(response, user.id)
analytics.record_event(analytics.EV_LOGIN, user)
accesslog.record(db, accesslog.LOGIN, request, user=user)
return me_payload(user, db)
@router.post("/logout")
def logout(response: Response):
if not auth.MULTI_USER:
raise HTTPException(400, "Accounts are disabled in local mode.")
response.delete_cookie(auth.SESSION_COOKIE, path="/")
return {"ok": True}
backend/app/routers/chat.py
+169
View File
@@ -0,0 +1,169 @@
"""AI Chat: a plain scratchpad for talking to a model directly.
Power users reach it, which means the `AIDND_POWER_USERS` email allowlist. It is
deliberately thin. It adds no story context, no scripts, and no world state, and
it persists nothing. The conversation lives in the browser and is posted in full
on each turn. It exists for testing models, prompts, and endpoints without
starting an adventure.
Model choice is free when the user brought their own API key. On the shared demo
key the model stays pinned to the `AIDND_DEMO_MODELS` allowlist, exactly as it is
for turns. The server funds that key, so this page must not let it reach paid
models.
"""
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi.responses import StreamingResponse
from sqlalchemy.orm import Session
from .. import auth, limits, models, schemas
from ..database import get_db
from ..providers import OpenAICompatibleProvider, ProviderError
from ..sse import SSE_HEADERS, sse
from .settings import get_settings, list_endpoint_models
router = APIRouter(prefix="/api/chat", tags=["chat"])
def power_user(
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
) -> models.User:
"""Gate for the whole router. 404 rather than 403 so the feature simply
doesn't appear to exist for everyone else."""
if not auth.is_power_user(user):
raise HTTPException(404, "Not found")
return user
PowerUser = Depends(power_user)
def _resolve_model(
settings: models.Settings, requested: str | None
) -> tuple[auth.ProviderConfig, str | None]:
"""Returns the provider config for this chat, plus a note when the requested
model was not used.
The pinning rule lives in `resolve_provider_config`. This function only
reports the substitution that call made, so one place decides what the demo
key may talk to.
"""
cfg = auth.resolve_provider_config(settings, model_override=requested)
wanted = (requested or "").strip()
if wanted and wanted != cfg.model:
return cfg, (
f"'{wanted}' isn't available on the shared demo key — using "
f"{cfg.model}. Add your own API key in Settings to use any model."
)
return cfg, None
@router.get("/config")
async def chat_config(
db: Session = Depends(get_db),
user: models.User = PowerUser,
):
"""Returns what this page can talk to.
The response holds the resolved endpoint and model, whether model choice is
pinned to the demo allowlist, and the endpoint's model listing. The listing
is best effort, and an unreachable endpoint returns an empty list.
"""
settings = get_settings(db, user)
cfg = auth.resolve_provider_config(settings)
listing = await list_endpoint_models(cfg)
return {
"endpoint_url": cfg.endpoint_url,
"model": cfg.model,
"using_demo": cfg.using_demo,
"api_mode": settings.api_mode,
"temperature": settings.temperature,
"max_tokens": settings.max_output_tokens,
# On the demo key the whitelist IS the list of choices; otherwise it's
# whatever the endpoint advertises (suggestions, not a restriction).
"models": auth.DEMO_MODELS if cfg.using_demo else listing.get("models", []),
"models_error": None if listing.get("ok") else listing.get("detail"),
}
async def run_chat(cfg: auth.ProviderConfig, settings: models.Settings, payload: schemas.ChatRequest,
note: str | None, db: Session, user: models.User):
"""Streams the reply as SSE, using the turn stream's event shape.
The generator emits `reasoning` and `chunk` events while generating and then
a `done` event, so the frontend reuses the same code.
"""
if note:
yield sse({"type": "note", "detail": note})
provider = OpenAICompatibleProvider(
cfg.endpoint_url, cfg.api_key, cfg.model, settings.api_mode,
settings.reasoning_max_tokens,
)
messages = [m.model_dump() for m in payload.messages]
chunks: list[str] = []
reasoning_chunks: list[str] = []
try:
async for kind, chunk in provider.chat(
messages,
temperature=payload.temperature if payload.temperature is not None else settings.temperature,
max_tokens=payload.max_tokens or settings.max_output_tokens,
):
if kind == "reasoning":
reasoning_chunks.append(chunk)
yield sse({"type": "reasoning", "text": chunk})
else:
chunks.append(chunk)
yield sse({"type": "chunk", "text": chunk})
except ProviderError as exc:
yield sse({"type": "error", "detail": str(exc)})
return
text = "".join(chunks).strip()
if not text:
detail = (
"The model used its entire token budget on reasoning and returned no "
"reply — raise max tokens, cap the reasoning budget in Settings, or "
"use a non-reasoning model."
if reasoning_chunks
else "The AI returned an empty response."
)
yield sse({"type": "error", "detail": detail})
return
if cfg.using_demo:
# Unmetered for power users (count_demo_turn is a no-op for them), but
# keep the call so the accounting stays right if the gate ever widens.
auth.count_demo_turn(user)
db.commit()
yield sse({
"type": "done",
"text": text,
"reasoning": "".join(reasoning_chunks).strip() or None,
"model": cfg.model,
})
@router.post("/stream")
def chat_stream(
payload: schemas.ChatRequest,
request: Request,
db: Session = Depends(get_db),
user: models.User = PowerUser,
):
total = sum(len(m.content) for m in payload.messages)
if total > schemas.CHAT_TOTAL_MAX:
raise HTTPException(
413, f"This conversation is too long to send ({total:,} characters) — "
"clear it or start a new one."
)
limits.rate_limit("chat", request, user)
settings = get_settings(db, user)
cfg, note = _resolve_model(settings, payload.model)
if not cfg.model:
raise HTTPException(400, "No model configured — set one in Settings or pick one here.")
return StreamingResponse(
run_chat(cfg, settings, payload, note, db, user),
media_type="text/event-stream",
headers=SSE_HEADERS,
)
backend/app/routers/debug.py
+19
View File
@@ -0,0 +1,19 @@
from fastapi import APIRouter, HTTPException
from .. import auth, debuglog
router = APIRouter(prefix="/api/debug", tags=["debug"])
@router.get("/requests")
def recent_requests():
"""Most-recent-first log of provider requests/responses (no API keys).
The log is a single process-wide ring buffer with no per-user attribution,
so in multi-user mode, which is how a hosted deployment runs, it would expose
other players' prompts. It is disabled there and available on a local
install.
"""
if auth.MULTI_USER:
raise HTTPException(403, "The debug log is only available on local installs.")
return debuglog.recent()
backend/app/routers/scenarios.py
+271
View File
@@ -0,0 +1,271 @@
from fastapi import APIRouter, Body, Depends, HTTPException, Request
from fastapi.responses import Response
from sqlalchemy import or_
from sqlalchemy.orm import Session
from .. import analytics, auth, images, limits, models, schemas
from ..database import get_db
router = APIRouter(prefix="/api/scenarios", tags=["scenarios"])
def get_scenario_or_404(
scenario_id: int, db: Session, user: models.User, *, edit: bool = False
) -> models.Scenario:
"""Visible = owned or public; editable = owned only."""
scenario = db.get(models.Scenario, scenario_id)
if scenario is None or (scenario.user_id != user.id and not scenario.is_public):
raise HTTPException(404, "Scenario not found")
if edit and scenario.user_id != user.id:
raise HTTPException(403, "This is a shared demo scenario — it can't be edited. Start an adventure from it, or duplicate it.")
return scenario
@router.get("", response_model=list[schemas.ScenarioListItem])
def list_scenarios(
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
return (
db.query(models.Scenario)
.filter(or_(models.Scenario.user_id == user.id, models.Scenario.is_public))
.order_by(models.Scenario.updated_at.desc())
.all()
)
@router.post("", response_model=schemas.ScenarioOut, status_code=201)
def create_scenario(
payload: schemas.ScenarioCreate,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
limits.check_row_cap("scenarios", db, user)
scenario = models.Scenario(**payload.model_dump(), user_id=user.id)
db.add(scenario)
db.commit()
return scenario
@router.get("/{scenario_id}", response_model=schemas.ScenarioOut)
def get_scenario(
scenario_id: int,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
scenario = get_scenario_or_404(scenario_id, db, user)
# A funnel step, recorded for shared scenarios only. Opening one is the
# first sign that a visitor is interested, and someone editing their own
# scenario is already past this point. Their titles are theirs rather than a
# statistic.
if scenario.is_public:
analytics.record_event(analytics.EV_SCENARIO_OPEN, user)
return scenario
@router.get("/{scenario_id}/image")
def get_scenario_image(
scenario_id: int,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
"""Serve an uploaded cover image as real bytes.
Lists point here instead of inlining the data URI. The response is marked
immutable and the URL carries a `?v=<updated_at>` stamp, so browsers cache
it indefinitely but pick up a new picture the moment the author saves one.
"""
scenario = get_scenario_or_404(scenario_id, db, user)
decoded = images.decode(scenario.image)
if decoded is None:
raise HTTPException(404, "This scenario has no uploaded image")
data, content_type = decoded
return Response(
content=data,
media_type=content_type,
headers={"Cache-Control": "private, max-age=31536000, immutable"},
)
@router.patch("/{scenario_id}", response_model=schemas.ScenarioOut)
def update_scenario(
scenario_id: int,
payload: schemas.ScenarioUpdate,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
scenario = get_scenario_or_404(scenario_id, db, user, edit=True)
data = payload.model_dump(exclude_unset=True)
script_ids = data.pop("script_ids", None)
for field, value in data.items():
setattr(scenario, field, value)
if script_ids is not None:
scripts = (
db.query(models.Script)
.filter(models.Script.id.in_(script_ids), models.Script.user_id == user.id)
.all()
)
if len(scripts) != len(set(script_ids)):
raise HTTPException(404, "One or more scripts not found")
scenario.scripts = sorted(scripts, key=lambda s: script_ids.index(s.id))
db.commit()
return scenario
@router.delete("/{scenario_id}", status_code=204)
def delete_scenario(
scenario_id: int,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
scenario = get_scenario_or_404(scenario_id, db, user, edit=True)
db.delete(scenario)
db.commit()
# ---------- Import / Export ----------
@router.get("/{scenario_id}/export")
def export_scenario(
scenario_id: int,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
s = get_scenario_or_404(scenario_id, db, user)
return {
"format": "ai-dnd-scenario-v1",
"title": s.title,
"description": s.description,
"prompt": s.prompt,
"memory": s.memory,
"authorsNote": s.authors_note,
"aiInstructions": s.ai_instructions,
"tags": s.tags,
"image": s.image,
"icon": s.icon,
"statSchema": s.stat_schema,
"storyCards": [
{"type": c.type, "name": c.name, "keys": c.keys, "entry": c.entry, "notes": c.notes}
for c in s.story_cards
],
"scripts": [
{
"name": sc.name, "description": sc.description, "library": sc.library_js,
"input": sc.input_js, "context": sc.context_js, "output": sc.output_js,
}
for sc in s.scripts
],
}
# Key aliases seen in AI Dungeon scenario exports, mapped best-effort.
_SCENARIO_KEYS = {
"title": "title",
"description": "description",
"prompt": "prompt",
"memory": "memory",
"authorsNote": "authors_note",
"authors_note": "authors_note",
"authorsNoteText": "authors_note",
"aiInstructions": "ai_instructions",
"ai_instructions": "ai_instructions",
"instructions": "ai_instructions",
}
_IGNORED_KEYS = {"format", "storyCards", "worldInfo", "worldInformation", "scripts", "tags",
"statSchema", "stat_schema", "image", "icon",
"createdAt", "updatedAt", "id", "publicId", "nsfw", "type", "options"}
@router.post("/import", status_code=201)
def import_scenario(
request: Request,
bundle: dict = Body(...),
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
"""Accepts our export format and AI Dungeon scenario exports best-effort;
reports any keys it didn't understand."""
limits.rate_limit("import", request, user)
limits.check_row_cap("scenarios", db, user)
fields: dict = {}
unmapped: list[str] = []
for key, value in bundle.items():
if key in _SCENARIO_KEYS and isinstance(value, str):
fields[_SCENARIO_KEYS[key]] = value
elif key not in _IGNORED_KEYS:
unmapped.append(key)
tags = bundle.get("tags")
if isinstance(tags, list):
fields["tags"] = ", ".join(str(t) for t in tags)
elif isinstance(tags, str):
fields["tags"] = tags
schema = bundle.get("statSchema") or bundle.get("stat_schema")
if isinstance(schema, dict):
fields["stat_schema"] = schema
# An AI Dungeon bundle also carries an `image`, so this reads it. The value
# is untrusted input, so it goes through `sanitize()` rather than a direct
# assignment.
image = images.sanitize(bundle.get("image"), schemas.IMAGE_MAX)
if image:
fields["image"] = image
icon = bundle.get("icon")
if isinstance(icon, str) and icon:
fields["icon"] = icon[:schemas.ICON_MAX]
scenario = models.Scenario(**fields, user_id=user.id)
if not scenario.title:
scenario.title = "Imported Scenario"
# A raw-dict import bypasses the schemas, so truncate to the VARCHAR widths.
# Postgres enforces them. See `schemas.py`. Column defaults have not been
# applied yet, because that happens at flush, so a bundle with no `tags` key
# leaves the attribute None, which is why the code says `or ""`.
scenario.title = scenario.title[:schemas.NAME_MAX]
scenario.tags = (scenario.tags or "")[:schemas.TAGS_MAX]
db.add(scenario)
db.flush()
# AI Dungeon exports have used all three names for the same list.
cards = (
bundle.get("storyCards")
or bundle.get("worldInfo")
or bundle.get("worldInformation")
or []
)
limits.check_bundle_lists(story_cards=cards)
for card in cards:
if not isinstance(card, dict):
continue
db.add(
models.StoryCard(
scenario_id=scenario.id,
type=str(card.get("type") or "")[:schemas.CARD_TYPE_MAX],
name=str(card.get("name") or card.get("title") or "")[:schemas.NAME_MAX],
keys=str(card.get("keys") or ""),
# AI Dungeon world info uses "value"; story cards use "entry".
entry=str(card.get("entry") or card.get("value") or ""),
notes=str(card.get("notes") or card.get("description") or ""),
)
)
for item in bundle.get("scripts") or []:
if not isinstance(item, dict):
continue
script = models.Script(
user_id=user.id,
name=str(item.get("name") or "Imported Script")[:schemas.NAME_MAX],
description=str(item.get("description") or ""),
library_js=str(item.get("library") or item.get("sharedLibrary") or ""),
input_js=str(item.get("input") or item.get("onInput") or ""),
context_js=str(item.get("context") or item.get("onModelContext") or ""),
output_js=str(item.get("output") or item.get("onOutput") or ""),
)
db.add(script)
db.flush()
scenario.scripts.append(script)
db.commit()
out = schemas.ScenarioOut.model_validate(scenario).model_dump(mode="json")
return {"scenario": out, "unmapped_keys": unmapped}
backend/app/routers/scripts.py
+159
View File
@@ -0,0 +1,159 @@
from fastapi import APIRouter, Body, Depends, HTTPException, Request
from sqlalchemy.orm import Session
from .. import auth, limits, models, schemas
from ..database import get_db
from ..scripting import run_hook
router = APIRouter(prefix="/api/scripts", tags=["scripts"])
HOOK_FIELDS = {"input": "input_js", "context": "context_js", "output": "output_js"}
def get_script_or_404(script_id: int, db: Session, user: models.User) -> models.Script:
script = db.get(models.Script, script_id)
if script is None or script.user_id != user.id:
raise HTTPException(404, "Script not found")
return script
@router.get("", response_model=list[schemas.ScriptOut])
def list_scripts(
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
return (
db.query(models.Script)
.filter(models.Script.user_id == user.id)
.order_by(models.Script.updated_at.desc())
.all()
)
@router.post("", response_model=schemas.ScriptOut, status_code=201)
def create_script(
payload: schemas.ScriptCreate,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
limits.check_row_cap("scripts", db, user)
script = models.Script(**payload.model_dump(), user_id=user.id)
db.add(script)
db.commit()
return script
@router.get("/{script_id}", response_model=schemas.ScriptOut)
def get_script(
script_id: int,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
return get_script_or_404(script_id, db, user)
@router.patch("/{script_id}", response_model=schemas.ScriptOut)
def update_script(
script_id: int,
payload: schemas.ScriptUpdate,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
script = get_script_or_404(script_id, db, user)
for field, value in payload.model_dump(exclude_unset=True).items():
setattr(script, field, value)
db.commit()
return script
@router.delete("/{script_id}", status_code=204)
def delete_script(
script_id: int,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
db.delete(get_script_or_404(script_id, db, user))
db.commit()
@router.post("/{script_id}/test")
def test_script(
script_id: int,
payload: schemas.ScriptTestRequest,
request: Request,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
"""Runs one hook against sample text, making no AI call and storing nothing."""
script = get_script_or_404(script_id, db, user)
limits.rate_limit("script-test", request, user)
result = run_hook(
script.library_js,
getattr(script, HOOK_FIELDS[payload.hook]),
payload.text,
payload.state,
history=[],
story_cards=[],
info={"actionCount": 0, "characterNames": [], "memoryLength": 0, "maxChars": 0},
)
return {
"text": result.text,
"stop": result.stop,
"state": result.state,
"storyCards": result.story_cards,
"logs": result.logs,
"error": result.error,
}
# ---------- Import / Export ----------
@router.get("/{script_id}/export")
def export_script(
script_id: int,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
"""JSON bundle matching how AI Dungeon scripts circulate."""
script = get_script_or_404(script_id, db, user)
return {
"name": script.name,
"description": script.description,
"library": script.library_js,
"input": script.input_js,
"context": script.context_js,
"output": script.output_js,
}
@router.post("/import", response_model=schemas.ScriptOut, status_code=201)
def import_script(
request: Request,
bundle: dict = Body(...),
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
"""Accepts our export bundle; tolerates *_js key names too."""
limits.rate_limit("import", request, user)
limits.check_row_cap("scripts", db, user)
def pick(*keys: str) -> str:
for key in keys:
value = bundle.get(key)
if isinstance(value, str):
return value
return ""
script = models.Script(
user_id=user.id,
# A raw-dict import bypasses the schemas, so truncate to the VARCHAR
# width.
name=(pick("name") or "Imported Script")[:schemas.NAME_MAX],
description=pick("description"),
library_js=pick("library", "library_js", "sharedLibrary"),
input_js=pick("input", "input_js", "onInput"),
context_js=pick("context", "context_js", "onModelContext"),
output_js=pick("output", "output_js", "onOutput"),
)
db.add(script)
db.commit()
return script
backend/app/routers/settings.py
+129
View File
@@ -0,0 +1,129 @@
import httpx
from fastapi import APIRouter, Depends, Request
from sqlalchemy.orm import Session
from starlette.concurrency import run_in_threadpool
from .. import auth, limits, models, netguard, schemas, security
from ..database import get_db
router = APIRouter(prefix="/api/settings", tags=["settings"])
def get_settings(db: Session, user: models.User) -> models.Settings:
"""Returns the user's settings row, creating it on first access.
Phase 8 made settings per user rather than global. They cover the endpoint,
the key, the models, and the memory configuration.
"""
settings = (
db.query(models.Settings).filter(models.Settings.user_id == user.id).first()
)
if settings is None:
settings = models.Settings(user_id=user.id)
db.add(settings)
db.commit()
return settings
@router.get("", response_model=schemas.SettingsOut)
def read_settings(
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
return get_settings(db, user)
@router.put("", response_model=schemas.SettingsOut)
def update_settings(
payload: schemas.SettingsUpdate,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
settings = get_settings(db, user)
fields = payload.model_dump(exclude_unset=True)
# Write-only API key: absent = unchanged, "" = cleared, else encrypted.
if "api_key" in fields:
fields["api_key"] = security.encrypt_secret(fields["api_key"].strip())
embedding_model_changed = (
"embedding_model" in fields
and fields["embedding_model"] != settings.embedding_model
)
for field, value in fields.items():
setattr(settings, field, value)
if embedding_model_changed:
# Vectors from the old model have a different dimensionality/space;
# clear them so the post-turn task re-embeds with the new model.
# This covers only this user's adventures, because settings are per
# user now.
#
# Both columns, and the flag. This is the one place that clears vectors
# in bulk rather than through memorybank.set_vector, and when the
# vectors moved to embedding_blob it kept nulling the old JSON column
# alone. The blob survived, `embedded` stayed true, and
# `_embed_pending`, which selects rows where `embedded IS FALSE`, never
# found the rows. The bank kept ranking against the previous model's
# vectors.
owned = (
db.query(models.Adventure.id)
.filter(models.Adventure.user_id == user.id)
.scalar_subquery()
)
db.query(models.Memory).filter(models.Memory.adventure_id.in_(owned)).update(
{"embedding_blob": None, "embedded": False}, synchronize_session=False
)
# No cache invalidation needed, and deliberately none added: clearing
# `embedded` drops these rows out of the catalogue query, so retrieval
# stops asking for them, and by the time _embed_pending puts one back
# it has gone through set_vector, which evicts that entry. The rule
# holds: anything that removes a memory from play corrects itself.
db.commit()
return settings
async def list_endpoint_models(cfg: auth.ProviderConfig) -> dict:
"""Fetches the endpoint's /models listing.
The call also serves as a connectivity check, so a failure returns
`{"ok": False, "detail": ...}` rather than raising.
"""
# SSRF guard. Never probe a non-public address the user supplied.
reason = await run_in_threadpool(netguard.endpoint_block_reason, cfg.endpoint_url)
if reason:
return {"ok": False, "detail": f"Can't reach that endpoint — {reason}."}
url = cfg.endpoint_url.rstrip("/") + "/models"
headers = {}
if cfg.api_key:
headers["Authorization"] = f"Bearer {cfg.api_key}"
try:
async with httpx.AsyncClient(timeout=10) as client:
resp = await client.get(url, headers=headers)
except httpx.HTTPError as exc:
return {"ok": False, "detail": f"Connection failed: {exc}"}
if resp.status_code != 200:
return {"ok": False, "detail": f"HTTP {resp.status_code}: {resp.text[:300]}"}
models_available: list[str] = []
try:
data = resp.json()
models_available = [m.get("id", "?") for m in data.get("data", [])]
except (ValueError, AttributeError, TypeError):
pass # The body is not JSON or has an unexpected shape. The endpoint
# is still reachable.
return {"ok": True, "models": models_available}
@router.post("/test")
async def test_connection(
request: Request,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
"""Runs a cheap connectivity check against whatever the turn engine would use.
That includes the shared demo endpoint, when the user has no key of their
own.
"""
limits.rate_limit("connection-test", request, user)
settings = get_settings(db, user)
return await list_endpoint_models(auth.resolve_provider_config(settings))
backend/app/routers/story_cards.py
+187
View File
@@ -0,0 +1,187 @@
from fastapi import APIRouter, Body, Depends, HTTPException, Request
from sqlalchemy.orm import Session
from .. import auth, limits, models, schemas
from ..database import get_db
router = APIRouter(prefix="/api/story-cards", tags=["story-cards"])
# The AI Dungeon world-info and story-card array format. Its field names differ
# from this app's columns: `value` maps to `entry`, `title` maps to `name`, and
# `description` maps to `notes`. The extra `useForCharacterCreation` flag has no
# equivalent here. It is ignored on import and written as false on export, so a
# round trip through AI Dungeon stays valid.
def _visible_owner(scenario_id, adventure_id, db, user):
"""Resolves the scenario or adventure a caller may read cards from.
Public demo scenarios are included. The function raises a 404 or a 422 when
it cannot resolve one.
"""
if (scenario_id is None) == (adventure_id is None):
raise HTTPException(422, "Provide exactly one of scenario_id or adventure_id")
if scenario_id is not None:
scenario = db.get(models.Scenario, scenario_id)
if scenario is None or (scenario.user_id != user.id and not scenario.is_public):
raise HTTPException(404, "Owner not found")
return scenario
adventure = db.get(models.Adventure, adventure_id)
if adventure is None or adventure.user_id != user.id:
raise HTTPException(404, "Owner not found")
return adventure
def _card_editable_or_404(card: models.StoryCard | None, user: models.User) -> models.StoryCard:
"""Cards inherit their scope from the owning scenario/adventure. Public
(demo) scenarios are visible to everyone but editable by no one."""
if card is not None:
owner = card.scenario if card.scenario_id is not None else card.adventure
if owner is not None and owner.user_id == user.id:
return card
raise HTTPException(404, "Story card not found")
@router.get("", response_model=list[schemas.StoryCardOut])
def list_story_cards(
scenario_id: int | None = None,
adventure_id: int | None = None,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
if (scenario_id is None) == (adventure_id is None):
raise HTTPException(422, "Provide exactly one of scenario_id or adventure_id")
if scenario_id is not None:
scenario = db.get(models.Scenario, scenario_id)
if scenario is None or (scenario.user_id != user.id and not scenario.is_public):
raise HTTPException(404, "Owner not found")
return sorted(scenario.story_cards, key=lambda c: c.id)
adventure = db.get(models.Adventure, adventure_id)
if adventure is None or adventure.user_id != user.id:
raise HTTPException(404, "Owner not found")
return sorted(adventure.story_cards, key=lambda c: c.id)
@router.post("", response_model=schemas.StoryCardOut, status_code=201)
def create_story_card(
payload: schemas.StoryCardCreate,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
if (payload.scenario_id is None) == (payload.adventure_id is None):
raise HTTPException(422, "Provide exactly one of scenario_id or adventure_id")
owner_model = models.Scenario if payload.scenario_id else models.Adventure
owner_id = payload.scenario_id or payload.adventure_id
owner = db.get(owner_model, owner_id)
if owner is None or owner.user_id != user.id:
raise HTTPException(404, "Owner not found")
limits.check_row_cap(
"story_cards", db, user,
scenario_id=payload.scenario_id, adventure_id=payload.adventure_id,
)
card = models.StoryCard(**payload.model_dump())
db.add(card)
db.commit()
return card
@router.patch("/{card_id}", response_model=schemas.StoryCardOut)
def update_story_card(
card_id: int,
payload: schemas.StoryCardUpdate,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
card = _card_editable_or_404(db.get(models.StoryCard, card_id), user)
for field, value in payload.model_dump(exclude_unset=True).items():
setattr(card, field, value)
db.commit()
return card
@router.delete("/{card_id}", status_code=204)
def delete_story_card(
card_id: int,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
card = _card_editable_or_404(db.get(models.StoryCard, card_id), user)
db.delete(card)
db.commit()
# ---------- Bulk import / export (AI Dungeon world-info format) ----------
@router.get("/export")
def export_story_cards(
scenario_id: int | None = None,
adventure_id: int | None = None,
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
owner = _visible_owner(scenario_id, adventure_id, db, user)
return [
{
"keys": c.keys,
"value": c.entry,
"type": c.type,
"title": c.name,
"description": c.notes,
"useForCharacterCreation": False,
}
for c in sorted(owner.story_cards, key=lambda c: c.id)
]
@router.post("/import", response_model=list[schemas.StoryCardOut], status_code=201)
def import_story_cards(
request: Request,
payload: dict = Body(...),
db: Session = Depends(get_db),
user: models.User = Depends(auth.get_current_user),
):
"""Append a list of story cards (AI Dungeon world-info format, or our own
export) to a scenario/adventure the caller owns."""
scenario_id = payload.get("scenario_id")
adventure_id = payload.get("adventure_id")
if (scenario_id is None) == (adventure_id is None):
raise HTTPException(422, "Provide exactly one of scenario_id or adventure_id")
owner_model = models.Scenario if scenario_id is not None else models.Adventure
owner = db.get(owner_model, scenario_id if scenario_id is not None else adventure_id)
if owner is None or owner.user_id != user.id:
raise HTTPException(404, "Owner not found")
# Accept a bare array or {"cards": [...]} / {"storyCards": [...]}, so an
# AI Dungeon world-info file dropped in as-is still works.
cards_in = payload.get("cards") or payload.get("storyCards") or payload.get("worldInfo")
if not isinstance(cards_in, list):
raise HTTPException(422, 'Expected a "cards" array of story cards.')
cards_in = [c for c in cards_in if isinstance(c, dict)]
limits.rate_limit("import", request, user)
limits.check_bundle_lists(story_cards=cards_in)
existing = len(owner.story_cards)
if auth.MULTI_USER and existing + len(cards_in) > limits.MAX_STORY_CARDS_PER_OWNER:
raise HTTPException(
409,
f"Importing {len(cards_in)} would exceed the limit of "
f"{limits.MAX_STORY_CARDS_PER_OWNER} story cards here "
f"({existing} already present) — remove some first.",
)
created = []
for card in cards_in:
obj = models.StoryCard(
scenario_id=scenario_id,
adventure_id=adventure_id,
type=str(card.get("type") or "")[:schemas.CARD_TYPE_MAX],
name=str(card.get("title") or card.get("name") or "")[:schemas.NAME_MAX],
keys=str(card.get("keys") or "")[:schemas.PROSE_MAX],
entry=str(card.get("value") or card.get("entry") or "")[:schemas.PROSE_MAX],
notes=str(card.get("description") or card.get("notes") or "")[:schemas.PROSE_MAX],
)
db.add(obj)
created.append(obj)
db.commit()
return created
backend/app/schemas.py
+510
View File
@@ -0,0 +1,510 @@
from datetime import datetime
from typing import Annotated, Literal
from pydantic import BaseModel, ConfigDict, Field, computed_field
from . import images
# Length caps (Phase 9). The VARCHAR caps are a correctness requirement rather
# than only an abuse limit. Postgres enforces column lengths and SQLite never
# did, so a longer value has to be a 422 here rather than a 500 at INSERT. The
# text-column caps are generous abuse ceilings that a legitimate player does not
# reach.
NAME_MAX = 200 # Titles and names. VARCHAR(200).
TAGS_MAX = 500 # VARCHAR(500).
CARD_TYPE_MAX = 100 # VARCHAR(100).
PROSE_MAX = 50_000 # Memory, author's note, prompts, entries, and notes.
SCRIPT_MAX = 200_000 # One JavaScript source file.
ACTION_MAX = 20_000 # One player action.
MEMORY_TEXT_MAX = 5_000
# A scenario cover image, stored inline as a base64 data URI. A 400x300 WebP at
# the quality the editor encodes runs about 20 to 40 kB. A cap of 400 kB leaves
# room for a client that downscales less aggressively, and it stops anyone from
# storing a multi-megabyte PNG in a row that every list request reads.
IMAGE_MAX = 400_000
ICON_MAX = 16 # One emoji or glyph. VARCHAR(16).
BRANCH_NAME_MAX = 80 # What a player called one line of the story. VARCHAR(80).
PERSONA_NAME_MAX = 80 # The protagonist's name. VARCHAR(80).
PERSONA_PRONOUNS_MAX = 40 # "they/them" and the like. VARCHAR(40).
Name = Annotated[str, Field(max_length=NAME_MAX)]
Tags = Annotated[str, Field(max_length=TAGS_MAX)]
CardType = Annotated[str, Field(max_length=CARD_TYPE_MAX)]
Prose = Annotated[str, Field(max_length=PROSE_MAX)]
ScriptSource = Annotated[str, Field(max_length=SCRIPT_MAX)]
ActionText = Annotated[str, Field(max_length=ACTION_MAX)]
Image = Annotated[str, Field(max_length=IMAGE_MAX)]
Icon = Annotated[str, Field(max_length=ICON_MAX)]
PersonaName = Annotated[str, Field(max_length=PERSONA_NAME_MAX)]
PersonaPronouns = Annotated[str, Field(max_length=PERSONA_PRONOUNS_MAX)]
class ORMModel(BaseModel):
model_config = ConfigDict(from_attributes=True)
# ---------- Story cards ----------
class StoryCardBase(BaseModel):
type: CardType = ""
name: Name = ""
keys: Prose = ""
entry: Prose = ""
notes: Prose = ""
class StoryCardCreate(StoryCardBase):
scenario_id: int | None = None
adventure_id: int | None = None
class StoryCardUpdate(BaseModel):
type: CardType | None = None
name: Name | None = None
keys: Prose | None = None
entry: Prose | None = None
notes: Prose | None = None
class StoryCardOut(ORMModel, StoryCardBase):
id: int
scenario_id: int | None
adventure_id: int | None
# ---------- Scenarios ----------
class ScenarioBase(BaseModel):
title: Name = "Untitled Scenario"
description: Prose = ""
prompt: Prose = ""
memory: Prose = ""
authors_note: Prose = ""
ai_instructions: Prose = ""
tags: Tags = ""
# Cover art, either an https URL or a base64 data URI. See `app/images.py`.
image: Image = ""
# The emoji or glyph shown when `image` is empty.
icon: Icon = ""
# Phase 12: the RPG world-state template, holding stat definitions, bands,
# rules, and milestones. `None` means the scenario has no RPG layer.
stat_schema: dict | None = None
class ScenarioCreate(ScenarioBase):
pass
class ScenarioUpdate(BaseModel):
title: Name | None = None
description: Prose | None = None
prompt: Prose | None = None
memory: Prose | None = None
authors_note: Prose | None = None
ai_instructions: Prose | None = None
tags: Tags | None = None
image: Image | None = None
icon: Icon | None = None
stat_schema: dict | None = None
script_ids: list[int] | None = None
class ScenarioOut(ORMModel, ScenarioBase):
id: int
is_public: bool = False # Shared demo content, read-only for everyone.
created_at: datetime
updated_at: datetime
story_cards: list[StoryCardOut] = []
scripts: list["ScriptOut"] = []
class ScenarioListItem(ORMModel):
id: int
title: str
description: str
tags: str
is_public: bool = False
updated_at: datetime
# Read from the row so that `image_url` can be derived, and excluded from
# the response, because a list of base64 data URIs would be megabytes of
# JSON.
image: str = Field("", exclude=True)
icon: str = ""
@computed_field
@property
def image_url(self) -> str:
return images.public_url(self.id, self.image, self.updated_at)
# ---------- Adventures ----------
class AdventureCreate(BaseModel):
scenario_id: int | None = None
title: Name | None = None
# The `${Placeholder}` values collected from the player at the start, which
# is the AI Dungeon behavior.
placeholders: dict[str, str] = {}
# Phase 18: who the player is playing as, collected by the same modal. These
# are independent of `placeholders`: a scenario that asks for `${Name}` is
# asking its own question, and nothing here fills it in.
persona_name: PersonaName = ""
persona_pronouns: PersonaPronouns = ""
persona_desc: Prose = ""
class AdventureUpdate(BaseModel):
title: Name | None = None
memory: Prose | None = None
authors_note: Prose | None = None
ai_instructions: Prose | None = None
story_summary: Prose | None = None
auto_summarize: bool | None = None
memory_bank_enabled: bool | None = None
persona_name: PersonaName | None = None
persona_pronouns: PersonaPronouns | None = None
persona_desc: Prose | None = None
class AdventureRefresh(BaseModel):
"""The body for "Update from scenario".
`placeholders` supplies answers the adventure has no stored value for. See
`AdventureCreate.placeholders`. The answers are merged over the stored ones
and saved.
"""
placeholders: dict[str, str] = {}
class RefreshPlan(BaseModel):
"""What a refresh would change. The confirm dialog is built from this."""
scenario_id: int
scenario_title: str
has_changes: bool
# Maps a field name to `{"old": ..., "new": ...}`, for differing fields
# only.
fields: dict[str, dict] = {}
# Maps "added", "updated", or "removed" to a list of card names.
cards: dict[str, list[str]] = {}
# Maps "added" or "removed" to a list of stat paths. Live values are
# otherwise kept.
world_state: dict[str, list[str]] = {}
# The `${Placeholder}` names the scenario asks for that the adventure has
# no stored answer to. The client collects these and sends them back.
placeholders_needed: list[str] = []
class ActionOut(ORMModel):
id: int
adventure_id: int
type: str
text: str
reasoning: str | None = None
# Phase 12: the compact RPG state changes for this turn, read from the
# model property.
world_changes: list[dict] = []
# SP9: the pager, such as `2/4`. It reports how many attempts this turn has
# and which one is on screen. It is keyed on the parent, so it counts the
# attempts of this turn rather than every node that shares a depth, and it
# keeps counting them after one has been forked onto its own branch.
#
# A turn nobody has retaken reads 1/1, which is most turns, and the client
# draws no pager for a count of one. The attempts themselves come from
# `GET /actions/{id}/variants`, so this payload stays small.
take_count: int = 1
take_index: int = 0
# Which line this node is on, so the pager can distinguish the two kinds of
# step without asking the server. An attempt on this branch is a leaf with
# nothing below it, so showing it is a local change. An attempt on another
# branch has a story of its own, so moving to it is a branch switch.
branch_id: int | None = None
created_at: datetime
class VariantOut(BaseModel):
# Since SP4 every attempt is its own node, so each one has an id, and the
# client needs that id. A fork is addressed by the attempt being promoted,
# not by its position in a group that renumbers whenever an attempt is
# added.
id: int
index: int
text: str
reasoning: str | None = None
# See `ActionOut.branch_id`. It decides whether choosing this attempt is a
# local step or a branch switch.
branch_id: int | None = None
created_at: str | None = None
active: bool = False
class VariantSelect(BaseModel):
index: int = Field(ge=0)
class BranchOut(ORMModel):
"""One line through the story tree (Phase 14, SP5).
This carries enough to draw the tree and nothing more. `fork_depth` is where
this line leaves its parent, and `depth` is where it currently ends, so a
fork is two numbers rather than a walk. `own_actions` counts the turns played
on this branch itself. The rest of its story is borrowed from its ancestors,
which is why the number is smaller than a reader expects.
"""
id: int
parent_branch_id: int | None = None
fork_depth: int | None = None
depth: int
own_actions: int = 0
is_head: bool = False
# NULL for a branch nobody has named. The client labels those from the fork
# depth rather than the server inventing a name. See the column comment.
name: str | None = None
created_at: datetime
class BranchRename(BaseModel):
"""A name a player chose, or `null` to make the branch unnamed again."""
name: Annotated[str, Field(max_length=BRANCH_NAME_MAX)] | None = None
class ActionUpdate(BaseModel):
text: ActionText
class ActionCreate(BaseModel):
type: Literal["do", "say", "story", "continue"]
text: ActionText = ""
# The node this action is played after (SP9). Omitting it means the tip,
# which is what every ordinary turn uses.
#
# Naming an attempt the story moved past is what creates a branch. Stepping
# between attempts costs nothing and creates nothing, and the fork happens
# on the first text written below one. That is the first moment the player
# states which line they mean. Before it, they were reading.
after_id: int | None = None
class TakeCreate(BaseModel):
"""Another attempt at a turn (SP9).
`text` is what the player says instead, and it applies only when the turn was
the player's. An AI turn's other attempt is generated, so the field is
ignored there rather than rejected. The client makes the same request for
both, and the node type decides what happens.
"""
text: ActionText = ""
class AdventureOut(ORMModel):
id: int
scenario_id: int | None
title: str
memory: str
authors_note: str
ai_instructions: str
story_summary: str
auto_summarize: bool
memory_bank_enabled: bool
persona_name: str
persona_pronouns: str
persona_desc: str
created_at: datetime
updated_at: datetime
story_cards: list[StoryCardOut] = []
# The newest window of the story, not all of it. Older pages arrive from
# `GET /{id}/actions` as the reader scrolls up. `action_count` is the whole
# story's length, which is how the client knows more actions exist above.
actions: list[ActionOut] = []
action_count: int = 0
class ActionPage(BaseModel):
"""A slice of the story, counted back from the newest action."""
actions: list[ActionOut] = []
total: int = 0
# Whether anything older than this slice exists. The server computes it, so
# the client never has to do arithmetic on positions to find the end.
has_more: bool = False
# ---------- Memory bank (Phase 6) ----------
class MemoryOut(ORMModel):
id: int
adventure_id: int
text: str
pinned: bool
forgotten: bool
embedded: bool # model property: embedding vector present
use_count: int
last_used_at: datetime | None
source_start: int | None
source_end: int | None
created_at: datetime
class MemoryCreate(BaseModel):
text: Annotated[str, Field(max_length=MEMORY_TEXT_MAX)]
class MemoryUpdate(BaseModel):
text: Annotated[str, Field(max_length=MEMORY_TEXT_MAX)] | None = None
pinned: bool | None = None
forgotten: bool | None = None
class AdventureListItem(ORMModel):
id: int
scenario_id: int | None
scenario_title: str | None = None
title: str
updated_at: datetime
action_count: int = 0
# The end of the most recent narration, so a Continue card can show the
# story rather than only a turn count.
snippet: str = ""
# Cover art inherited from the parent scenario. See `app/images.py`.
image_url: str = ""
icon: str = ""
# ---------- Scripts ----------
class ScriptBase(BaseModel):
name: Name = "Untitled Script"
description: Prose = ""
library_js: ScriptSource = ""
input_js: ScriptSource = ""
context_js: ScriptSource = ""
output_js: ScriptSource = ""
class ScriptCreate(ScriptBase):
pass
class ScriptUpdate(BaseModel):
name: Name | None = None
description: Prose | None = None
library_js: ScriptSource | None = None
input_js: ScriptSource | None = None
context_js: ScriptSource | None = None
output_js: ScriptSource | None = None
class ScriptOut(ORMModel, ScriptBase):
id: int
created_at: datetime
updated_at: datetime
class ScriptTestRequest(BaseModel):
hook: Literal["input", "context", "output"]
text: Prose = ""
state: dict = {}
class AdventureScriptOut(ORMModel):
id: int
adventure_id: int
position: int
enabled: bool
name: str
description: str
library_js: str
input_js: str
context_js: str
output_js: str
# The router sets this field, which is not stored. It is `True` when a
# syncable library version exists whose code differs from this copy, and
# `None` when there is nothing to sync from.
out_of_date: bool | None = None
class AdventureScriptUpdate(BaseModel):
enabled: bool | None = None
library_js: ScriptSource | None = None
input_js: ScriptSource | None = None
context_js: ScriptSource | None = None
output_js: ScriptSource | None = None
# ---------- Auth (Phase 8) ----------
class AuthCredentials(BaseModel):
email: Annotated[str, Field(max_length=320)] # VARCHAR(320).
# The upper bound keeps the scrypt cost constant. Without it, hashing a
# megabyte password would give an attacker free CPU time.
password: Annotated[str, Field(max_length=128)]
# ---------- Settings ----------
class SettingsOut(ORMModel):
endpoint_url: str
# The key itself is never returned. It is encrypted at rest and
# write-only.
has_api_key: bool
model: str
api_mode: str
temperature: float
max_output_tokens: int
reasoning_max_tokens: int
context_token_budget: int
narrator_prompt: str
summary_model: str
embedding_model: str
memory_bank_capacity: int
memory_top_k: int
ScenarioOut.model_rebuild()
# ---------- AI Chat (power users) ----------
# A scratchpad for talking to a model directly, with no story framing. The
# server persists nothing, so these caps are per-request abuse limits only.
CHAT_MESSAGE_MAX = 100_000 # One message.
CHAT_TOTAL_MAX = 400_000 # The whole conversation sent per request.
CHAT_MESSAGES_MAX = 200 # Turns per request.
class ChatMessage(BaseModel):
role: Literal["system", "user", "assistant"]
content: Annotated[str, Field(max_length=CHAT_MESSAGE_MAX)]
class ChatRequest(BaseModel):
messages: Annotated[list[ChatMessage], Field(min_length=1, max_length=CHAT_MESSAGES_MAX)]
# If this field is empty or omitted, the user's configured model is used.
model: Name | None = None
temperature: Annotated[float, Field(ge=0, le=5)] | None = None
max_tokens: Annotated[int, Field(ge=1, le=100_000)] | None = None
class SettingsUpdate(BaseModel):
endpoint_url: Annotated[str, Field(max_length=500)] | None = None # VARCHAR(500).
# Encryption expands the stored value by about four thirds into the same
# VARCHAR(500), so 256 plaintext characters is the largest safe input. The
# stored form is "enc:" plus Fernet plus base64.
api_key: Annotated[str, Field(max_length=256)] | None = None
model: Name | None = None
api_mode: Annotated[str, Field(max_length=20)] | None = None
temperature: Annotated[float, Field(ge=0, le=5)] | None = None
max_output_tokens: Annotated[int, Field(ge=1, le=100_000)] | None = None
# A value of -1 turns reasoning off explicitly, which sends
# `reasoning: {effort: none}`. A value of 0 sends nothing.
reasoning_max_tokens: Annotated[int, Field(ge=-1, le=100_000)] | None = None
context_token_budget: Annotated[int, Field(ge=256, le=200_000)] | None = None
narrator_prompt: Prose | None = None
summary_model: Name | None = None
embedding_model: Name | None = None
memory_bank_capacity: Annotated[int, Field(ge=1, le=1000)] | None = None
memory_top_k: Annotated[int, Field(ge=1, le=50)] | None = None
backend/app/scripting/__init__.py
+4
View File
@@ -0,0 +1,4 @@
from .engine import HookResult, run_hook
from .pipeline import ScriptPipeline
__all__ = ["HookResult", "ScriptPipeline", "run_hook"]
backend/app/scripting/engine.py
+146
View File
@@ -0,0 +1,146 @@
"""AI Dungeon-compatible script execution in an embedded QuickJS sandbox.
Each hook run is fully isolated (fresh Context), capped at 16 MB memory and
2 seconds CPU, with no filesystem/network/process access (QuickJS has none by
default). Scripts follow the AI Dungeon contract: define a `modifier(text)`
and call it as the last line; its return value `{ text, stop }` is the result.
"""
import json
from dataclasses import dataclass, field
import quickjs
MEMORY_LIMIT = 16 * 1024 * 1024
TIME_LIMIT_SECONDS = 2
HISTORY_WINDOW = 100 # recent actions exposed as `history`
# Globals per the official docs: text, state, history, storyCards, info,
# log/console.log, story card functions, plus legacy worldInfo aliases.
PRELUDE = """
"use strict";
var __logs = [];
var state = __DATA__.state;
var text = __DATA__.text;
var history = __DATA__.history;
var storyCards = __DATA__.storyCards;
var info = __DATA__.info;
function log(msg) {
__logs.push(typeof msg === "string" ? msg : JSON.stringify(msg));
}
var console = { log: log };
// Returns the new card's index, or false if a card with those keys exists —
// matching real AI Dungeon. Note index 0 is falsy; that quirk is upstream's.
function addStoryCard(keys, entry, type) {
for (var i = 0; i < storyCards.length; i++) {
if (storyCards[i].keys === keys) return false;
}
storyCards.push({ id: null, keys: keys || "", entry: entry || "", type: type || "" });
return storyCards.length - 1;
}
function updateStoryCard(index, keys, entry, type) {
var card = storyCards[index];
if (!card) throw new Error("Story card not found");
card.keys = keys;
card.entry = entry;
card.type = type;
}
function removeStoryCard(index) {
if (!storyCards[index]) throw new Error("Story card not found");
storyCards.splice(index, 1);
}
// Legacy aliases used by older AI Dungeon scripts.
var worldInfo = storyCards;
var worldEntries = storyCards;
function addWorldEntry(keys, entry) { return addStoryCard(keys, entry, ""); }
function updateWorldEntry(index, keys, entry) {
var card = storyCards[index];
if (!card) throw new Error("World entry not found");
card.keys = keys;
card.entry = entry;
}
function removeWorldEntry(index) { return removeStoryCard(index); }
"""
COLLECT = """
JSON.stringify({
result: (typeof __result === "undefined" || __result === null) ? null : __result,
state: state,
storyCards: storyCards,
logs: __logs
})
"""
@dataclass
class HookResult:
text: str
stop: bool = False
state: dict = field(default_factory=dict)
story_cards: list = field(default_factory=list)
logs: list = field(default_factory=list)
error: str | None = None
def run_hook(
library_js: str,
hook_js: str,
text: str,
state: dict,
history: list[dict],
story_cards: list[dict],
info: dict,
) -> HookResult:
"""Run one modifier hook. This function never raises. Failures return as
`.error` with text, state, and cards unchanged, so a bad script cannot
break a turn."""
unchanged = HookResult(text=text, state=state, story_cards=story_cards)
source = f"{library_js}\n;\n{hook_js}" if library_js.strip() else hook_js
if not source.strip():
return unchanged
data = {
"state": state,
"text": text,
"history": history[-HISTORY_WINDOW:],
"storyCards": story_cards,
"info": info,
}
try:
ctx = quickjs.Context()
ctx.set_memory_limit(MEMORY_LIMIT)
ctx.set_time_limit(TIME_LIMIT_SECONDS)
ctx.eval(f"var __DATA__ = {json.dumps(data)};")
ctx.eval(PRELUDE)
ctx.eval(f"var __SRC__ = {json.dumps(source)};")
# Indirect eval keeps the script in global scope, so `modifier(text)` as the
# script's final expression statement becomes the completion value.
ctx.eval("var __result = (0, eval)(__SRC__);")
collected = json.loads(ctx.eval(COLLECT))
except quickjs.JSException as exc:
unchanged.error = f"Script error: {exc}"
return unchanged
except Exception as exc: # memory limit, invalid JSON state, engine faults
unchanged.error = f"Script execution failed: {exc}"
return unchanged
result = collected.get("result")
new_text, stop = text, False
if isinstance(result, dict):
if isinstance(result.get("text"), str):
new_text = result["text"]
stop = bool(result.get("stop"))
elif isinstance(result, str):
new_text = result
new_state = collected.get("state")
return HookResult(
text=new_text,
stop=stop,
state=new_state if isinstance(new_state, dict) else {},
story_cards=collected.get("storyCards") or [],
logs=collected.get("logs") or [],
)
backend/app/scripting/pipeline.py
+112
View File
@@ -0,0 +1,112 @@
"""Runs an adventure's enabled scripts through a turn's hook points, applying
state and story-card mutations back to the database after each hook."""
from sqlalchemy.orm import Session
from .. import models
from ..context import history as context_history
from .engine import run_hook
MAX_STORY_CARDS = 5000 # AI Dungeon's per-adventure sanity cap
class ScriptPipeline:
def __init__(self, adventure: models.Adventure, db: Session):
self.adventure = adventure
self.db = db
self.logs: list[str] = []
self.errors: list[str] = []
@property
def message(self) -> str | None:
state = self.adventure.script_state
msg = state.get("message") if isinstance(state, dict) else None
return msg if isinstance(msg, str) and msg.strip() else None
def _history(self) -> list[dict]:
# Read the path rather than `adventure.actions`. That collection holds
# every branch's actions, and this is the documented history API a user
# script reads. Giving a script the siblings of the turn it is running on
# would be the same bug as building a prompt from them, and visible to
# the user.
#
# `story_actions` also drops rows with blank text, which
# `adventure.actions` kept, so this array is shorter than it was for an
# adventure that has any such rows. `info.actionCount` counts the same
# way. That is intended. A row with no text is this app's bookkeeping, it
# has no counterpart in the AI Dungeon history a ported script was
# written against, and the prompt has never included one. A script keyed
# on every N actions lands on different turns than it did before phase
# 14, and no reading of this is compatible with both.
return [
{"text": a.text, "rawText": a.text, "type": a.type}
for a in context_history.story_actions(self.adventure)
]
def _cards(self) -> list[dict]:
return [
{"id": c.id, "keys": c.keys, "entry": c.entry, "type": c.type}
for c in self.adventure.story_cards
]
def _info(self) -> dict:
return {
"actionCount": context_history.count(self.adventure),
"characterNames": [],
"memoryLength": len(self.adventure.memory),
"maxChars": 0,
}
def _apply_cards(self, returned: list) -> None:
existing = {c.id: c for c in self.adventure.story_cards}
seen_ids = set()
added = 0
for item in returned:
if not isinstance(item, dict):
continue
card_id = item.get("id")
keys = str(item.get("keys") or "")
entry = str(item.get("entry") or "")
card_type = str(item.get("type") or "")
if card_id in existing:
seen_ids.add(card_id)
card = existing[card_id]
card.keys, card.entry, card.type = keys, entry, card_type
elif len(existing) + added < MAX_STORY_CARDS:
self.db.add(
models.StoryCard(
adventure_id=self.adventure.id,
keys=keys, entry=entry, type=card_type,
)
)
added += 1
for card_id, card in existing.items():
if card_id not in seen_ids:
self.db.delete(card)
def run(self, hook: str, text: str) -> tuple[str, bool]:
"""Chain `hook` across all enabled scripts. Returns (text, stop)."""
state = self.adventure.script_state if isinstance(self.adventure.script_state, dict) else {}
for script in self.adventure.scripts:
hook_js = getattr(script, f"{hook}_js")
if not script.enabled or not hook_js.strip():
continue
result = run_hook(
script.library_js, hook_js, text, state,
self._history(), self._cards(), self._info(),
)
if result.error:
self.errors.append(f"{script.name} ({hook}): {result.error}")
continue # a broken script never breaks the turn
self.logs.extend(f"[{script.name}/{hook}] {line}" for line in result.logs)
self._apply_cards(result.story_cards)
state = result.state
self.adventure.script_state = state
self.db.commit()
text = result.text
if result.stop:
return text, True
return text, False
def report(self) -> dict:
return {"logs": self.logs, "errors": self.errors, "message": self.message}
backend/app/security.py
+131
View File
@@ -0,0 +1,131 @@
"""Phase 8: secrets and crypto primitives for optional accounts.
Everything derives from one server-side secret:
* Session cookies are HMAC-signed with it.
* Stored LLM API keys are Fernet-encrypted with a key derived from it.
The secret comes from `AIDND_SECRET_KEY`, or it is generated once into
`secret.key` next to the database, so a local install and a Docker volume work
with no configuration. Losing that file logs everyone out and makes the stored
API keys unreadable, and users then re-enter them. A multi-user deployment has
to set the environment variable, because a hosted filesystem is ephemeral and a
`secret.key` regenerated on every deploy would log out every user each time.
Passwords use `hashlib.scrypt`, which is in the standard library and backed by
OpenSSL, so this needs no separate hashing dependency.
"""
import base64
import hashlib
import hmac
import os
import secrets
from cryptography.fernet import Fernet, InvalidToken
from .database import DB_PATH
_SECRET_FILE = DB_PATH.parent / "secret.key"
def _load_secret() -> bytes:
env = os.environ.get("AIDND_SECRET_KEY", "").strip()
if env:
return env.encode()
# Same flag parse as auth.MULTI_USER (auth imports this module, so it
# can't be imported from there).
if os.environ.get("AIDND_MULTI_USER", "").strip().lower() in ("1", "true", "yes", "on"):
raise RuntimeError(
"AIDND_SECRET_KEY must be set when AIDND_MULTI_USER is on: an "
"auto-generated secret.key on an ephemeral hosted filesystem would "
"rotate on every deploy, logging out every user and orphaning "
"their stored API keys. Generate one with: "
"python -c \"import secrets; print(secrets.token_urlsafe(48))\""
)
if _SECRET_FILE.exists():
return _SECRET_FILE.read_bytes().strip()
secret = secrets.token_urlsafe(48).encode()
_SECRET_FILE.write_bytes(secret)
return secret
SECRET_KEY = _load_secret()
_fernet = Fernet(base64.urlsafe_b64encode(hashlib.sha256(SECRET_KEY).digest()))
# ---------- Password hashing (scrypt) ----------
_SCRYPT_N, _SCRYPT_R, _SCRYPT_P = 2**14, 8, 1
def hash_password(password: str) -> str:
salt = secrets.token_bytes(16)
key = hashlib.scrypt(
password.encode(), salt=salt, n=_SCRYPT_N, r=_SCRYPT_R, p=_SCRYPT_P
)
return f"scrypt${_SCRYPT_N}${_SCRYPT_R}${_SCRYPT_P}${salt.hex()}${key.hex()}"
def verify_password(password: str, stored: str) -> bool:
try:
scheme, n, r, p, salt_hex, key_hex = stored.split("$")
if scheme != "scrypt":
return False
key = hashlib.scrypt(
password.encode(), salt=bytes.fromhex(salt_hex),
n=int(n), r=int(r), p=int(p),
)
return hmac.compare_digest(key, bytes.fromhex(key_hex))
except (ValueError, AttributeError):
return False
# ---------- Session tokens ----------
# The token is "v1.<user_id>.<hmac>". It does not expire, because a long-lived
# guest session is what this is for.
def sign_session(user_id: int) -> str:
payload = f"v1.{user_id}"
sig = hmac.new(SECRET_KEY, payload.encode(), hashlib.sha256).hexdigest()
return f"{payload}.{sig}"
def verify_session(token: str) -> int | None:
try:
version, user_id, sig = token.split(".")
if version != "v1":
return None
payload = f"{version}.{user_id}"
expected = hmac.new(SECRET_KEY, payload.encode(), hashlib.sha256).hexdigest()
if not hmac.compare_digest(sig, expected):
return None
return int(user_id)
except (ValueError, AttributeError):
return None
# ---------- API-key encryption at rest ----------
# Stored values carry an "enc:" prefix so plaintext keys from pre-Phase-8
# databases can be recognized and migrated.
ENC_PREFIX = "enc:"
def encrypt_secret(plain: str) -> str:
if not plain:
return ""
return ENC_PREFIX + _fernet.encrypt(plain.encode()).decode()
def decrypt_secret(stored: str) -> str:
"""Returns the plaintext key. Tolerates legacy plaintext values (returned
as-is) and undecryptable tokens (secret rotated → treated as unset)."""
if not stored:
return ""
if not stored.startswith(ENC_PREFIX):
return stored
try:
return _fernet.decrypt(stored[len(ENC_PREFIX):].encode()).decode()
except (InvalidToken, ValueError):
return ""
backend/app/seed.py
+249
View File
@@ -0,0 +1,249 @@
"""Seed public demo scenarios on startup.
Every JSON file in ``seed_data/`` describes one demo scenario in the same
model-native shape the export endpoint produces. Seeded scenarios have a NULL
owner and ``is_public=True``, so every visitor (including guests) sees them and
can start an adventure from them, while nobody can edit them. Starting an
adventure copies the scenario's story cards and scripts into the adventure, so
the seeded scripts run for guests too.
Seed files are the source of truth for demo content: a scenario is inserted if
missing, reconciled in place when a seed file's content changes, and deleted
when no file claims its title any more, so an edit ships on the next deploy.
Rename a seed by changing its `title` and listing the old one under
`previous_titles`, which moves the rename onto the existing row. When a seed already matches, nothing is written, so
this stays cheap to run on every boot. An adventure already started from a demo
keeps its own copied cards and scripts and is unchanged. Only a new adventure
picks up the updated content.
"""
import json
import logging
from pathlib import Path
from sqlalchemy.engine import Engine
from . import models
from .database import SessionLocal
logger = logging.getLogger(__name__)
SEED_DIR = Path(__file__).resolve().parent / "seed_data"
# `title` is in this list so that a rename found through `previous_titles` is
# both detected by `_matches` and written by `_apply_scalars`.
_SCALARS = ("title", "description", "prompt", "memory", "authors_note", "ai_instructions",
"tags", "image", "icon")
_CARD_FIELDS = ("type", "name", "keys", "entry", "notes")
_SCRIPT_FIELDS = ("name", "library_js", "input_js", "context_js", "output_js")
def seed_public_scenarios(engine: Engine) -> None:
if not SEED_DIR.is_dir():
return
files = sorted(SEED_DIR.glob("*.json"))
if not files:
return
db = SessionLocal()
try:
changed = 0
# Every title the files claim, including the ones they used to use. The
# sweep below deletes the seeded rows this set does not name.
claimed: set[str] = set()
complete = True
for path in files:
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
logger.warning("Skipping seed file %s: %s", path.name, exc)
complete = False
continue
title = (data.get("title") or "").strip()
if not title:
continue
claimed.add(title)
claimed.update(str(old) for old in data.get("previous_titles") or [])
existing = find_seeded(db, title) or _find_renamed(db, data)
if existing is None:
_insert_scenario(db, data)
changed += 1
elif not _matches(existing, data):
_update_scenario(db, existing, data)
changed += 1
changed += _sweep_unclaimed(db, claimed) if complete else 0
if changed:
db.commit()
logger.info("Seeded/updated %d public demo scenario(s).", changed)
except Exception:
db.rollback()
# A seed failure must never take the app down; log and carry on.
logger.exception("Seeding public scenarios failed; continuing without them.")
finally:
db.close()
def _sweep_unclaimed(db, claimed: set[str]) -> int:
"""Deletes seeded scenarios no seed file claims any more, and returns how
many went.
A rename used to strand the row it left behind. `previous_titles` stops new
ones appearing, and this removes the ones already out there, which was
otherwise hand-work on every deployment. Only rows with a NULL owner and
`is_public` are considered, and a player's own scenario is neither, so
nothing anybody created can be reached from here.
An adventure started from a deleted demo survives. `adventures.scenario_id`
is `ON DELETE SET NULL`, so the story, its cards, and its scripts are its
own copies and stay; the adventure loses the cover art it inherited.
The caller skips this when a seed file failed to parse. A file that cannot
be read claims no title, and deleting on that basis would treat a syntax
error as an instruction to remove live content. An empty seed directory
never reaches here at all, for the same reason.
"""
stale = (
db.query(models.Scenario)
.filter(
models.Scenario.user_id.is_(None),
models.Scenario.is_public.is_(True),
models.Scenario.title.notin_(claimed) if claimed else True,
)
.all()
)
for scenario in stale:
logger.info("Removing seeded scenario %r; no seed file claims it.", scenario.title)
# The scripts are joined through a secondary table, so nothing cascades
# to them. They have a NULL owner and no other reader.
for script in list(scenario.scripts):
db.delete(script)
scenario.scripts = []
db.delete(scenario)
return len(stale)
def _card_tuple(source, get) -> tuple:
return tuple(get(source, f) for f in _CARD_FIELDS)
def _script_tuple(source, get) -> tuple:
return tuple(get(source, f) for f in _SCRIPT_FIELDS)
def find_seeded(db, title: str) -> models.Scenario | None:
"""Returns the seeded scenario with this exact title, if there is one."""
return (
db.query(models.Scenario)
.filter(
models.Scenario.title == title,
models.Scenario.user_id.is_(None),
models.Scenario.is_public.is_(True),
)
.first()
)
def _find_renamed(db, data: dict) -> models.Scenario | None:
"""Returns the row a renamed seed file used to own, so the rename lands on it.
A seed is matched by title, so renaming one inserts a second scenario and
strands the first. The stranded row stays public forever and has to be
deleted by hand on every deployment. List the old title under
`previous_titles` in the seed file and the rename updates the existing row
instead, which also keeps the adventures already started from it pointing at
a scenario that still exists.
Drop a `previous_titles` entry once every deployment has booted past it.
"""
for old in data.get("previous_titles") or []:
found = find_seeded(db, str(old))
if found is not None:
return found
return None
def _matches(scenario: models.Scenario, data: dict) -> bool:
"""True when the DB scenario already equals the seed file, so we can skip
the write and avoid churning rows on every boot."""
if any(getattr(scenario, f) != data.get(f, "") for f in _SCALARS):
return False
if (scenario.stat_schema or None) != (data.get("stat_schema") or None):
return False
have_cards = sorted(_card_tuple(c, lambda o, f: getattr(o, f)) for c in scenario.story_cards)
want_cards = sorted(
_card_tuple(c, lambda o, f: o.get(f, ""))
for c in (data.get("story_cards") or []) if isinstance(c, dict)
)
if have_cards != want_cards:
return False
have_scripts = sorted(_script_tuple(s, lambda o, f: getattr(o, f)) for s in scenario.scripts)
want_scripts = sorted(
_script_tuple(s, lambda o, f: (o.get(f, "") or ("Script" if f == "name" else "")))
for s in (data.get("scripts") or []) if isinstance(s, dict)
)
return have_scripts == want_scripts
def _insert_scenario(db, data: dict) -> None:
scenario = models.Scenario(user_id=None, is_public=True, title=data.get("title", ""))
_apply_scalars(scenario, data)
db.add(scenario)
db.flush()
_populate_children(db, scenario, data)
def _update_scenario(db, scenario: models.Scenario, data: dict) -> None:
_apply_scalars(scenario, data)
# Replace the child content in full. Demo content is owned by the server and
# cheap to rebuild, and replacing it this way keeps the scenario row, and the
# adventure foreign keys that point at it, intact.
for card in list(scenario.story_cards):
db.delete(card)
for script in list(scenario.scripts):
db.delete(script)
scenario.scripts = []
db.flush()
_populate_children(db, scenario, data)
def _apply_scalars(scenario: models.Scenario, data: dict) -> None:
for field in _SCALARS:
setattr(scenario, field, data.get(field, ""))
# Phase 12: RPG world-state template (a JSON dict, not a scalar string).
scenario.stat_schema = data.get("stat_schema") or None
def _populate_children(db, scenario: models.Scenario, data: dict) -> None:
for card in data.get("story_cards") or []:
if not isinstance(card, dict):
continue
db.add(
models.StoryCard(
scenario_id=scenario.id,
type=card.get("type", ""),
name=card.get("name", ""),
keys=card.get("keys", ""),
entry=card.get("entry", ""),
notes=card.get("notes", ""),
)
)
for item in data.get("scripts") or []:
if not isinstance(item, dict):
continue
script = models.Script(
user_id=None,
name=item.get("name", "Script"),
description=item.get("description", ""),
library_js=item.get("library_js", ""),
input_js=item.get("input_js", ""),
context_js=item.get("context_js", ""),
output_js=item.get("output_js", ""),
)
db.add(script)
db.flush()
scenario.scripts.append(script)
backend/app/seed_data/01-sunken-crypt-of-vharos.json
+67
View File
@@ -0,0 +1,67 @@
{
"title": "[Demo] The Sunken Crypt of Vharos",
"description": "A short dungeon-crawl demo scenario with story cards and one of each script hook, for testing the app end to end.",
"prompt": "Rain hammers the moors as you descend the moss-slick steps beneath the ruined chapel. Your torch gutters in the stale air. Below, the Sunken Crypt of Vharos waits — its iron door ajar, as if someone (or something) expected you. Mira's warning rings in your ears: bring back the Ember Amulet before nightfall, or the village of Hollowmere burns.\n\nYou stand before the iron door, water pooling around your boots.",
"memory": "The player is an adventurer hired by Mira, blacksmith of Hollowmere, to retrieve the Ember Amulet from the Sunken Crypt of Vharos before nightfall. The crypt is flooded, dark, and haunted. Tone: classic D&D dungeon crawl, dangerous but fair.",
"authors_note": "Keep scenes tense and grounded; offer clear choices; consequences matter.",
"ai_instructions": "Write in second person, present tense. End each response at a moment where the player can act.",
"tags": "demo, dungeon, fantasy, short",
"story_cards": [
{
"type": "character",
"name": "Mira the Blacksmith",
"keys": "Mira, blacksmith",
"entry": "Mira is Hollowmere's blacksmith: broad-shouldered, gray-braided, practical. She hired the player and paid half up front. She knows more about the crypt than she has admitted.",
"notes": "Secretly a descendant of Vharos."
},
{
"type": "location",
"name": "The Sunken Crypt",
"keys": "crypt, tomb, Vharos",
"entry": "A flooded burial complex beneath a ruined chapel. Knee-deep black water, collapsed pillars, and phosphorescent moss. Three chambers: the Drowned Hall, the Ossuary, and the sealed Reliquary where the Ember Amulet rests.",
"notes": ""
},
{
"type": "item",
"name": "The Ember Amulet",
"keys": "amulet, ember",
"entry": "A fist-sized garnet on a bronze chain that glows like a coal. It keeps Hollowmere's protective hearth-ward burning. Touching it bare-handed brands the flesh but does no lasting harm.",
"notes": ""
}
],
"scripts": [
{
"name": "[Demo] Dice Roller",
"description": "Input hook + shared library. Type '!roll 2d6' or '!roll d20' in a Do/Say/Story action and the command is replaced with the rolled result.",
"library_js": "// Shared library: available to every hook of this script.\nfunction rollDice(count, sides) {\n var total = 0, rolls = [];\n for (var i = 0; i < count; i++) {\n var r = Math.floor(Math.random() * sides) + 1;\n rolls.push(r);\n total += r;\n }\n return { total: total, rolls: rolls };\n}\n",
"input_js": "const modifier = (text) => {\n // Replace every \"!roll NdS\" (N optional) with the roll result.\n var out = text.replace(/!roll\\s+(\\d*)d(\\d+)/gi, function (m, n, s) {\n var count = parseInt(n || \"1\", 10);\n var sides = parseInt(s, 10);\n var res = rollDice(count, sides);\n log(\"Rolled \" + count + \"d\" + sides + \": [\" + res.rolls.join(\", \") + \"] = \" + res.total);\n return \"(rolled \" + count + \"d\" + sides + \": \" + res.total + \")\";\n });\n return { text: out };\n};\nmodifier(text);\n",
"context_js": "",
"output_js": ""
},
{
"name": "[Demo] Turn & HP Tracker",
"description": "Demonstrates persistent state. Counts turns; '!hp -3' or '!hp +5' in input adjusts HP (starts at 20). Current stats appear in state.message.",
"library_js": "",
"input_js": "const modifier = (text) => {\n if (state.hp === undefined) state.hp = 20;\n state.turns = (state.turns || 0) + 1;\n\n var out = text.replace(/!hp\\s*([+-]\\d+)/gi, function (m, delta) {\n state.hp += parseInt(delta, 10);\n return \"\";\n });\n\n state.message = \"Turn \" + state.turns + \" | HP: \" + state.hp + \"/20\";\n log(state.message);\n\n if (state.hp <= 0) {\n // stop:true ends the turn before the AI is called.\n return { text: out + \"\\n\\nYou have fallen. (HP reached 0 — turn stopped by script.)\", stop: true };\n }\n return { text: out };\n};\nmodifier(text);\n",
"context_js": "",
"output_js": ""
},
{
"name": "[Demo] Context Inspector",
"description": "Context hook: logs the size of the assembled context each turn and appends a style directive. Check the logs/context in the action's context snapshot.",
"library_js": "",
"input_js": "",
"context_js": "const modifier = (text) => {\n log(\"Context size: \" + text.length + \" chars, actions so far: \" + info.actionCount\n + \", story cards: \" + storyCards.length);\n // Anything returned here replaces what is sent to the model.\n return { text: text + \"\\n[Style: keep the response under three paragraphs.]\" };\n};\nmodifier(text);\n",
"output_js": ""
},
{
"name": "[Demo] Output Polish + Card Discovery",
"description": "Output hook: trims a trailing incomplete sentence from the AI reply, and auto-creates a story card the first time the ghost Vharos is mentioned.",
"library_js": "",
"input_js": "",
"context_js": "",
"output_js": "const modifier = (text) => {\n var out = text;\n\n // Drop a trailing sentence fragment (no ending punctuation).\n var m = out.match(/^([\\s\\S]*[.!?\"'\\u2026])[^.!?\"'\\u2026]*$/);\n if (m && m[1].length > 40) {\n if (m[1].length < out.length) log(\"Trimmed incomplete final sentence.\");\n out = m[1];\n }\n\n // Demonstrate script-created story cards.\n if (/vharos/i.test(out)) {\n var added = addStoryCard(\n \"Vharos, ghost, spirit\",\n \"Vharos was the crypt's architect, now a restless ghost bound to the amulet he was buried with. He speaks in echoes and cannot lie.\",\n \"character\"\n );\n if (added !== false) log(\"Vharos mentioned — story card created.\");\n }\n\n return { text: out };\n};\nmodifier(text);\n"
}
],
"icon": "🗝"
}
backend/app/seed_data/02-signal-from-the-derelict.json
+34
View File
@@ -0,0 +1,34 @@
{
"title": "[Demo] Signal from the Derelict",
"description": "A tense sci-fi mystery aboard a drifting research station. Showcases the memory and story-card context engine with no scripts required.",
"prompt": "The airlock cycles with a hiss, and the cold of the Meridian station folds around your suit. Six days ago it stopped answering hails; three hours ago your ship's antenna caught a single repeating pulse from inside — a heartbeat rhythm, too regular to be an accident. Emergency lighting stutters red down the corridor ahead. Your suit reads breathable air, faint traces of ozone, and no life signs.\n\nYou float at the threshold, the pulse ticking softly in your headset.",
"memory": "The player is a salvage contractor who boarded the derelict research station Meridian after it went silent six days ago. A repeating signal is coming from somewhere inside. The crew of nine is missing. Tone: grounded, tense sci-fi mystery — quiet dread over action.",
"authors_note": "Keep the atmosphere claustrophobic and quiet; reward careful investigation; make the station feel physical and specific.",
"ai_instructions": "Write in second person, present tense. Favor sensory detail and restraint. End each response where the player can act.",
"tags": "demo, sci-fi, mystery, short",
"story_cards": [
{
"type": "location",
"name": "Meridian Station",
"keys": "Meridian, station, ship",
"entry": "A mid-sized orbital research station in a decaying orbit around a gas giant. Ring layout: docking spur, crew habitat, the labs, and the sealed reactor core. Power is failing; artificial gravity is intermittent. It smells faintly of ozone and antiseptic.",
"notes": "Orbit decays fully in roughly 40 hours — a soft clock."
},
{
"type": "character",
"name": "Dr. Reyes",
"keys": "Reyes, doctor, chief scientist",
"entry": "Dr. Ana Reyes was Meridian's chief scientist: meticulous, secretive, protective of her work. Her personal logs are locked. The last public entry, six days ago, reads only: 'It answered back.'",
"notes": "Whereabouts unknown. Her lab is sealed from the inside."
},
{
"type": "item",
"name": "The Signal",
"keys": "signal, pulse, heartbeat",
"entry": "A repeating low-frequency pulse broadcasting from deep in the station, roughly one beat per second. It is not a standard distress code. The closer you get to the labs, the stronger it becomes.",
"notes": ""
}
],
"scripts": [],
"icon": "🛰"
}
backend/app/seed_data/03-hp-system-demo.json
+21
View File
@@ -0,0 +1,21 @@
{
"title": "[Demo] The Goblin Ambush (HP script)",
"description": "A tiny combat scene that shows script-driven state: the AI narrates and decides damage, a JavaScript script tracks your HP. Open the State panel on the left to watch it change.",
"prompt": "The forest path narrows between two moss-choked boulders. You are halfway through when the bushes erupt — three goblins, blades drawn, teeth bared. The lead one lunges at you with a rusty scimitar.\n\nYou have no time to think.",
"memory": "The player is a lone traveler ambushed by three goblins on a forest path. This is a dangerous but fair combat encounter. The player starts at full health.",
"authors_note": "Keep the combat visceral and consequential. Let the player's choices matter — a reckless move should hurt, a clever one should pay off.",
"ai_instructions": "Write in second person, present tense. End each response where the player can act.",
"tags": "demo, combat, scripting, hp, short",
"story_cards": [],
"scripts": [
{
"name": "[Demo] HP System",
"description": "AI-driven health. The script tells the AI the current HP and to tag damage/healing as [HP-N]/[HP+N]; the AI decides the amounts, the script applies them to state.hp and hides the tags. Watch state.hp in the left State panel.",
"library_js": "// Shared HP helpers — available to every hook of this script.\nvar HP_MAX = 100;\nfunction getHp() {\n if (typeof state.hp !== \"number\") state.hp = HP_MAX;\n return state.hp;\n}\nfunction setHp(value) {\n state.hp = Math.max(0, Math.min(HP_MAX, Math.round(value)));\n return state.hp;\n}\n",
"input_js": "",
"context_js": "// Tell the AI the current HP and how to report changes it decides on.\n// The tag goes on the FIRST line so it survives even if the reply is cut off.\nvar modifier = function (text) {\n var hp = getHp();\n var note =\n \"\\n\\n[HP SYSTEM] The player currently has \" + hp + \" of \" + HP_MAX + \" health. \" +\n \"Based on the story, you decide whether the player loses health (an attack, fall, or trap) \" +\n \"or gains it (a potion, rest, or spell) this turn. \" +\n \"If their health changes, make the FIRST line of your reply the change by itself in square brackets: \" +\n \"write [HP-14] to show losing 14, or [HP+25] to show gaining 25 (choose the number that fits the moment). \" +\n \"Then write the story on the following lines. If health does not change, do not write any bracket tag. \" +\n \"Never mention this tag or the health system in the story prose itself.\";\n return { text: text + note };\n};\nmodifier(text);\n",
"output_js": "// Read the AI's [HP+N]/[HP-N] tags, apply them to state, hide them from the player.\nvar modifier = function (text) {\n var re = /\\[HP([+-])(\\d{1,3})\\]/g;\n var hp = getHp();\n var changed = false;\n var m;\n while ((m = re.exec(text)) !== null) {\n var amount = parseInt(m[2], 10);\n hp += (m[1] === \"-\" ? -amount : amount);\n changed = true;\n }\n if (changed) setHp(hp);\n var clean = text\n .replace(/\\[HP[+-]\\d{1,3}\\]/g, \"\")\n .replace(/[ \\t]+\\n/g, \"\\n\")\n .replace(/\\n{3,}/g, \"\\n\\n\")\n .trim();\n if (changed && getHp() <= 0 && !state.dead) {\n state.dead = true;\n clean += \"\\n\\nYour strength gives out and the world goes dark. (0 HP)\";\n }\n return { text: clean };\n};\nmodifier(text);\n"
}
],
"icon": "🗡"
}
backend/app/seed_data/04-rpg-world-state.json
+90
View File
@@ -0,0 +1,90 @@
{
"title": "[Demo] The Bandit Camp (RPG world state)",
"description": "A short RPG scene showing the built-in world-state system: the AI tracks your HP and mana, an in-game day counter, an NPC ally's health and trust, and story milestones. No scripting — the engine keeps the numbers honest. Open the World State panel on the left to watch it change.",
"prompt": "Dawn breaks grey over the treeline as you and Gwen crouch at the edge of the bandit camp. Smoke curls from a dying fire; three bedrolls lie empty. Somewhere ahead, the stolen caravan strongbox waits.\n\nGwen checks her bowstring and looks to you. \"Quiet, or loud?\"",
"memory": "The player and Gwen, a loyal ranger ally, are raiding a bandit camp to recover a stolen strongbox. The player is a capable adventurer. This is a dangerous but winnable encounter.",
"authors_note": "Keep it tense and consequential. Reckless moves should cost health; clever ones should pay off. Gwen reacts to how the player treats her.",
"ai_instructions": "Write in second person, present tense. End each reply where the player can act. Let the world state guide the fiction — if the player is badly hurt, show it. Reflect what happens in the numbers each turn: change health when someone is hurt or healed, shift Gwen's trust based on how the player treats her, move the leader's aggression as the situation escalates or calms, and mark milestones as they are reached.",
"tags": "demo, rpg, world-state, combat, short",
"icon": "🏹",
"stat_schema": {
"world": {
"day": { "type": "counter", "min": 1, "initial": 1, "desc": "Which in-game day it is; only ever counts up." }
},
"player": {
"hp": {
"desc": "The player's physical health. At 0 they fall.",
"min": 0, "max": 100, "initial": 100, "max_delta_per_turn": 35,
"bands": [[0, 1, "dead"], [1, 25, "gravely wounded"], [25, 50, "hurt"],
[50, 80, "scratched"], [80, 100, "healthy"], [100, 101, "full health"]]
},
"mana": {
"desc": "Magical energy for spells; spent casting, restored by resting.",
"min": 0, "max": 50, "initial": 30, "max_delta_per_turn": 25,
"bands": [[0, 10, "drained"], [10, 30, "steady"], [30, 51, "brimming"]]
},
"outfit": {
"type": "text", "initial": "worn leather armor, travel-stained",
"desc": "What the player is currently wearing/carrying visibly; update it when gear is gained, lost, or changed."
}
},
"flags": {
"alarm_raised": { "desc": "True once the bandits know they're under attack; stealth is blown.", "initial": false },
"player_hidden": { "desc": "True while the player is out of sight in cover.", "initial": true }
},
"npcs": {
"gwen": {
"name": "Gwen",
"keys": "Gwen, ranger, her",
"desc": "A loyal ranger and the player's ally. Quick with a bow, dry-humoured, fiercely protective. Her trust rises when the player fights smart and watches her back, and falls when they are reckless with her life.",
"stats": {
"health": {
"desc": "Gwen's physical health.",
"min": 0, "max": 100, "initial": 100, "max_delta_per_turn": 35,
"bands": [[0, 1, "dead"], [1, 25, "gravely wounded"], [25, 50, "hurt"],
[50, 80, "scratched"], [80, 101, "healthy"]]
},
"trust": {
"desc": "How much Gwen trusts the player; rises with smart, loyal play and falls with recklessness.",
"min": -100, "max": 100, "initial": 20, "max_delta_per_turn": 20,
"bands": [[-100, -30, "hostile"], [-30, 30, "wary"], [30, 70, "friendly"],
[70, 101, "devoted"]]
}
}
},
"bandit_leader": {
"name": "Bandit Leader",
"keys": "leader, chief, boss, scarred",
"desc": "The scarred leader of the bandit camp, guarding the strongbox. Fights harder the more cornered he becomes.",
"stats": {
"health": {
"desc": "The leader's physical health.",
"min": 0, "max": 120, "initial": 120, "max_delta_per_turn": 40,
"bands": [[0, 1, "dead"], [1, 30, "near death"], [30, 70, "bloodied"],
[70, 121, "unhurt"]]
},
"aggression": {
"desc": "How aggressively the leader fights; climbs as the fight turns against him.",
"min": 0, "max": 100, "initial": 40, "max_delta_per_turn": 25,
"bands": [[0, 30, "cautious"], [30, 70, "fierce"], [70, 101, "berserk"]]
}
}
}
},
"milestones": {
"camp_cleared": { "desc": "Clear the bandit camp of enemies" },
"strongbox_found": { "desc": "Recover the stolen strongbox" },
"gwen_survives": { "desc": "Escape with Gwen still alive" }
}
},
"story_cards": [
{
"type": "location",
"name": "Bandit Camp",
"keys": "camp, bandits, strongbox",
"entry": "A rough camp of bandits in a forest clearing, holding a stolen caravan strongbox.",
"notes": ""
}
],
"scripts": []
}
backend/app/seed_data/05-league-championship.json
+170
View File
@@ -0,0 +1,170 @@
{
"title": "[Demo] Pokemon League Championship: Round One",
"previous_titles": ["[Demo] League Championship: Round One"],
"description": "A full Pokémon battle run on the built-in world-state system. Every member of your team is tracked as its own entity with HP and a status condition, a text field names who is currently out, and your potion supply runs down as you use it. The opponent's active Pokémon is tracked the same way. No scripting — the engine keeps the numbers honest. Open the World State panel on the left to watch the battle play out.",
"prompt": "The floodlights come up over the championship field and the referee raises both flags. Round One of eight. Beat the trainer in front of you or go home.\n\nAcross the packed rock terrain, Trainer Milo tosses his first Poké Ball. Graveler lands hard enough to crack the stone under it, four arms flexing, and settles into a low stance.\n\n\"Rock-types all the way down,\" Milo calls over the noise. \"Hope you brought something that hits harder than it looks.\"\n\nYour Pidgeotto is already in the air, wings locked into a slow circle above the field. It is fast, but Rock-type moves will tear it out of the sky, and Graveler is built to take a hit. Three Hyper Potions sit in your belt pouch.\n\nGraveler grinds one fist into its palm. The referee drops the flag.\n\nWhat's your move?",
"memory": "The player is a Trainer in Round One of an eight-round League Championship. The Round One opponent is Milo, a Rock-type specialist with three Pokémon: Graveler (Rock/Ground), Onix (Rock/Ground), and Kabutops (Rock/Water). The player's team is Pidgeotto (Normal/Flying), Ivysaur (Grass/Poison), Wartortle (Water), Machoke (Fighting), and Pikachu (Electric). Pidgeotto is out first. The player carries Hyper Potions, which restore HP and can be swapped for a status heal when needed. A battle is lost when every Pokémon on one side has fainted.",
"authors_note": "Play it like a real battle. Type matchups decide damage, status conditions stick around and cost the afflicted Pokémon turns, and switching costs the player a turn while the opponent attacks freely. Milo is competent: he switches to punish bad matchups and targets whatever is weakest.",
"ai_instructions": "Write in second person, present tense. Narrate the battle move by move, naming the active Pokémon on each side and the move each one uses. End every reply at a decision point where the player can choose a move, switch Pokémon, or use an item.\n\nReflect type advantage and disadvantage in the damage you narrate and in the HP numbers: a super-effective hit should take roughly double a neutral hit, a resisted hit roughly half. Ground and Rock moves hit Pikachu and Pidgeotto hard. Water, Grass, and Fighting moves are strong against Milo's Rock-types.\n\nEvery turn, update the state. Every number you send is a change to add to the current value, never the value you want it to end up at. The world state lines show you totals; the block you write carries the difference. A Pokémon at 96 HP that takes 30 damage is `-30`, not `66`.\n- Add 1 to `world.turn` in every reply, including this one. It counts the turns this battle has lasted, so it moves whether or not anything else does.\n- Send a negative number for HP lost and a positive number for HP restored on both sides (`npc.<name>.hp` for the player's own Pokémon, e.g. npc.pidgeotto.hp; `npc.milo.active_hp` for Milo's). Every Pokémon starts at its full HP, so a positive number on an undamaged one changes nothing.\n- Set a Pokémon's `status` stat (`npc.<name>.status`, or `npc.milo.active_status`) to the condition it just picked up: poisoned, burned, paralyzed, frozen, asleep, or none when it is cured or wakes up. Status matters. Poison and burn shave HP at the end of each turn, paralysis and freeze cost turns, sleep stops a Pokémon acting until it wakes.\n- When the player switches, set `player.active_pokemon` to the incoming Pokémon's name in full (not a delta).\n- When the player uses a potion, send `-1` for `player.potions` and a positive number for the healed Pokémon's HP. The count falls when a potion is spent. It never rises. Potions cannot be used once the count reaches 0.\n- When a Pokémon's HP hits 0 it faints, its status becomes `fainted`, and its trainer must send out a replacement (the player picks the next `player.active_pokemon`). A fainted Pokémon cannot be healed or switched back in.\n- Whenever Milo sends out a new Pokémon, set `npc.milo.active_pokemon` and `npc.milo.active_status` to their new values in full, and send `npc.milo.active_hp` as the positive change that carries it from where it stands to the newcomer's full HP. From a fainted Pokémon at 0 that is the newcomer's full HP: Graveler 98, Onix 90, Kabutops 88.\n\nWhen one of Milo's Pokémon faints, do both of these in the same reply: send `1` for `npc.milo.pokemon_fainted`, which adds one to the count rather than setting it, and set `npc.milo.active_pokemon` with a fresh `npc.milo.active_hp` and `npc.milo.active_status` for the one he sends out next. `npc.milo.pokemon_fainted` counts up from 0 and never goes down.\n\nMark each milestone as it happens, using the id shown beside the goal in the world state. Do not decide the whole battle in one reply; give the player a turn between every exchange.",
"tags": "demo, pokemon, battle, tournament, world-state, combat",
"icon": "◓",
"image": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAMAAAADACAYAAABS3GwHAAAMc0lEQVR42u2du44byRWG+w0m712bNtbrsQ0DTKyYMPQAhHIDjA0IYOZACRMHyggnCreTyTYYKNyoM2cCA4cK6FSJGSqkeWa7Z6nRcKZO3bou3wF+QIA4zWb3f6rOvZoGQRAEQRAEQcJL27azExYPMOPJIDmTej4QeTOgH7A74WiJ3dl1xuvKd8x54shURL86I/qtI8FdsRvuYVSMK94QEsJcWZ3QnbCfkOym2A/3usKsQmxJvzxhmwnhTRRCfsuSN4s8R3pZOQ8FkP4SDsNvRBmQO9IvKiD9c8qwgAn1ObHrQswbn2bSGie6/DBlB9mfRUeYtTwzp4fYavSYRxDfCX9pvz0u21/f4e/tb47/bL8zgnx2/Du5BoqAJEv86/abO6IKcX9ovz/+1F4f/9P+8fi/b/7sFXJNubZ8h3yXfKd8N4qAjMSfxSC+EO8f7W+PP7a/D0J0G8WQe5F7knuLpAgk2BKL6mxDmjEj4acmuylGhQhsPm2JGk1P/lWIGP5f218d/9X+LokV3scOIb9FflOgXMIKJhZg7owrfQmkf0oZAu0MmEURyb/2uer/rZ1lZd74NJPkt3veDdYwNINVX6Inpa/22l3BY0SJ3SAA+Zc+Vn15yRJC/G/7p+qJ/xDyTOTZeFKEAwV3/si/hfjZKsIWBruFN3cQP3tF2BEu1ZN/7mrySDkBNr4fH0GepQeTiAI7RWzfKYYvJQOQ1y/kmXrIJZAzCGXvj+YOZA0LD2YRfsEF8ncu9TmYO3HNIse6ow7Gf+ns3to+TFb9aXcDByW4rd45don0SDr/3+0fIOLEkHfgUFpRb4TIhfySwie0mVbI1KGsoj4lcCG/VDZCujQh7wYlCER+iTzUWLSWY5GdZZSoDiWwcXjlgWLv5+UXWCrBLaHORxJb2Pt5+gWWibOuVPJvIT9KUGWyzKa8AfJXrwSrUsg/tylkgzxlwbKgbp47+a+0VZ2yWkCYMmGxExyyjgxpw52YPZhDj4VHq3B6IT9KUIxTPPTwquL8kL8uJbDIEyxzIf9MY/eT5CJZpvAHZjkogGp0CeUNdZdNaEeupE7+NYVtIHAB3boI00fKZyEAEChLqdM0hTSmjzRQ4PSCc6dY2VTTp0Z+VakDTi94zCnOslRCm+2lhxd46jFOI0usSXjJJAFeNHgKymkT2xQcX+N4P6NLgMnIFWV+YJaF44vpAwKZQv1U5F9Q4QkSqRxdJL36M6sT2MwiTXYX0Kz+NLeASE00i+RWfxxfENEh7pNb/XF8QWSHeJHU6k+5A4jcO9CHJv+c1R8kvgvMQypAx+oPEt8FulDkv2L1B5nsAlchFGDN6g8y2QXWIRRgb/Llcuo4LwyEgHDLUAH2k4U+ifuDkHmBSUKips4vbY4gofbJzqcCHJjwADKbJHGIOuRKejp5QSAGFP3Dy2jmD87v0xEMqW6UUJ48J+l6egzyf/IZ+SyRNC/OcBfN/MH5/brJW16U5Wz8+z4KuQZDBKyd4UMU84eGl1/wQ/u9E+mfesZybZ6xumFm6aIAWya8mU84szwYTn1gIM9bNVFuGzz5VbP5I/a6w0np1pDvrLnTTmEG7W3JPyP687Rjq3DGgkHuoVaHWbHwzGwUYEX057KDG8LOd/EPanSUFQvQKlj4s7bkl8Mh0MF9g9qUQJEU64LZ/5AfJZgSQfwA09r/mkYd+ib/y5cvj69evbqD/BslCD5K8cp79Wct9r8r+a+vr4+vX78+vn///vjx48fj58+fH4X8n3xGPit/gxJ49QMWGgXYYP//Eu2xdXhldb+5ublI+OcgfyvXsHWMa4gOKfyAjUYBbon/q1eYe7x48eJuJbcl/kPIteSaNiFS8gH3uPV6wLVss4zn+xpivnz69Mkb+UfINeXa2vupIVlmaJ7uNAqAA6xLtNzBxdzRmEXajDGO8M/wOvun9MkP2tMLY5DfVglKrx1STIyYe4sAlV6ZqIn6xCS/jRKUbq4KF71FgkwjQCXblooHemeXxyb/CI1PUPKCpfDVNt4UoOQIkGnYUyIzIRxejWNsGh0quWdDEQkyUoC+5hIIzXGdPkOdLiFSjqc1LonovShAyZEF07i/JKimJv8I02RZyXkBw4hd7y0HcKmxO3eYOr9TOL6uDjHvzSAXkGqlY1JVl9fXyZB/hEvtUE1AATxgysiPj4gQCoACOCEF59fFGUYBUAAnPFXSPBXknng3jgpg2ghfO1Ij/wjejWODvGYMeq2Q7q1UFcBnZ1nBWKAADkgp/m+bD0ABUAAUAAVAATCBUACcYJxgnGDCoIRBCYOiACTCUAAUgFIIFIAHRTFc1QqwMyHAONavNJgSKctyaN7bzktDjLThpRoJccWbN2+KbYiR31bqezNsDe29tUSW+iA/fPhQbEuk/LZS35vPlshNrqHA2AmlnJriU07gRQwB+5sK0fd9sQ9UM3Mnl7EoKfksviFc9KkAi9ofqDaqkvxgrASjVhM9i4W30Yhv374t+qG+e/eumNGI8ltKflfCRW+jEU1zASlXRUaOLCQ9HLfkiJ02CuZ3PHrh26rStkx2PHrJvprSXN15PyCj5EiQNi+Q4gEZJcf9LSJAqgMyNrkWhYVYeW3r7Kc8IknuecoQbYJ5ENURSQtWmC+TY06H1sU+JO/0tyUnvSx3aNUheVc4wn6VINoxqRWRX9kGan5Mquag7FoedAglCFGlWhP5FSUQuoOyBwXo8APyUYIaya+w/zsbBVjhB1xWgpQa0OVeaiO/0v5f2SjAjETL09EhmxCpb8g91BDtcUxUzhobMfUDasgHPJUss4nRu0K+s4Ykl4f4/76xldMfb6k1Ma8diuEbyHfwvFW1WlsXBVjWXmtuk7wK4R/INUuvwA00BGzZuMjpAgfMIDtHWexzF2WQv5Vr1OjgejJ/Do2rmIZDa4wGaRxmsdelbFee06XGbvk/+Yx8tlbH1nP0p/OhAEuiQSDT6M+y8SGmZlBtSTGQdPLr0PgSUzNotVrxkkBQCMeimT82Y9NxhkEiA4AXjU8xTYrhDIMEnN9941tOF12bJmqIYIAQkTRFonEdQgGuTLef0idGgKQnP+hr/307w+wCYMLVv2tCienMIHYBMOHqP29CiunwXHYBMMHq3zehRRMSZRcAkVf/RRNDNLsAeQHgEvdPavW32QVSPE8L5AHlJLxFE1NMd4FaxvOBScdS9k1s0ewCNMyAgA0v8Vd/m10AhxgEcnz7ZioxnRyBQwwCOb72Ex88KsFWMyyWlww8jTp0a3j3XCN0wBQCkU2fQ7Can1BT5Go4rhPYDxFQDg1YNSmJxiGe+nhRkF65g3K4WN+kJoNDbGwK0T4JLNocR9Nn1qQopk0zTJQDFhPewjW7TGUKMUmCCQ9K8vdN6qI1hWqcaw+szllI1/SxHaZF7wA1/gosm5xEkyCr6VRDyG916ua2yVFMDtxGCSC/twOuE1QAVZaYylEqPJPN9sZopKeJhuaWqA3uqZZKYA5Vb/akV+oQ2ylGCaom/7YpUUwHa6EEVZO/a0qW0w+85RBoklwXcNuULkNkaGejBJRN5FHeYEn+XfYRn9BKQAFdcYVt9ZHfhxJI+Sx+QVr2vrKkuW7y+1ACaaDAL0jD3lc2s0D+R5Tg1vYsXXqMs+nh/crhrZ78riHS82kTjFyJO7pEOb2hrlBnzGTZeZSI3SDOqm8Z5Sk7yTVl2cTDxBmzSMPM6rRMbJVb3hC4gO7g8rCl+AqzyI+5Y1nI9rCqcw6zI0WIHppFhEztQpsezB0iPVP6BSjCpMTH3veoBEtXkwhFiEr8Q3Y9vBkowUw7cuUpRZBTx/ERfrbx5Vl4Iv5xeEczGBtOEdY+doPzsooai+zkNzuUL1xa9dcwNLPd4Ly0ovRdYVztHUoXWPUTzBkcPL/Mu3i3VDaWoAzyG+S3eIjhX1r1ie0nEC7dBni5X+wMOZlJcq+BVvovIjyENws3iy7VHY0KkcIOIfcwEt6xPgdzpxBFWMRQhPOIkhBPQog3Nzd3JQMhFEOuKdeW75Dvku/0GLkxJf4ChqEIKvNJiCqQcgIhrgnks+PfBTZjID6KACB+HYowd+k5qAgdhWvlR40kmbaH7PfYD8+EqE6F5lEXIpeQAQ7Db8fMQe4L7kpXhpH0FKohzyrDthAzaT/8FkiPWCnDbCi56DJRiP1wrysSVkgoJ1p8h80w2mM3Idl3wz1shnvCiUUmDbOOirEZ4ui9o4Lszq6zOSM6YUoke7Nq8QCYKwiCIAiCIEgM+T+tNIWP5O3yJwAAAABJRU5ErkJggg==",
"stat_schema": {
"world": {
"round": { "type": "counter", "min": 1, "max": 8, "initial": 1, "desc": "Which round of the eight-round championship this is; only ever counts up." },
"turn": { "type": "counter", "min": 0, "initial": 0, "max_delta_per_turn": 1, "desc": "How many turns the current battle has lasted; only ever counts up." }
},
"player": {
"potions": {
"desc": "Hyper Potions left in the player's bag. Each one restores HP or cures a status condition, and is used up when it is used. Send -1 when the player uses one. Never send the number remaining.",
"min": 0, "max": 5, "initial": 3, "max_delta_per_turn": 1,
"bands": [[0, 1, "out of potions"], [1, 2, "one left"], [2, 4, "a few left"], [4, 6, "well stocked"]]
},
"active_pokemon": {
"type": "text", "initial": "Pidgeotto",
"desc": "Which of the player's Pokémon is currently on the field: Pidgeotto, Ivysaur, Wartortle, Machoke, or Pikachu. Replace this the moment the player switches."
}
},
"flags": {
"sandstorm_active": { "desc": "True while a sandstorm is whipping across the field, chipping HP off any Pokémon that is not Rock, Ground, or Steel each turn.", "initial": false }
},
"npcs": {
"pidgeotto": {
"name": "Pidgeotto",
"keys": "Pidgeotto",
"desc": "The player's own Pokémon. Normal/Flying. Fast, but Rock-type moves hit it for four times damage.",
"stats": {
"hp": {
"desc": "Pidgeotto's HP. Send the change, not the new total: a negative number for damage taken, a positive number for HP restored.",
"min": 0, "max": 84, "initial": 84, "max_delta_per_turn": 60,
"bands": [[0, 1, "fainted"], [1, 20, "critical"], [20, 45, "hurt"], [45, 70, "scuffed"], [70, 85, "healthy"]]
},
"status": {
"type": "text", "initial": "none",
"desc": "Status condition: none, poisoned, burned, paralyzed, frozen, asleep, or fainted."
}
}
},
"ivysaur": {
"name": "Ivysaur",
"keys": "Ivysaur",
"desc": "The player's own Pokémon. Grass/Poison. Super effective on Milo's Rock/Ground types; can inflict poison or sleep.",
"stats": {
"hp": {
"desc": "Ivysaur's HP. Send the change, not the new total: a negative number for damage taken, a positive number for HP restored.",
"min": 0, "max": 96, "initial": 96, "max_delta_per_turn": 60,
"bands": [[0, 1, "fainted"], [1, 24, "critical"], [24, 50, "hurt"], [50, 78, "scuffed"], [78, 97, "healthy"]]
},
"status": {
"type": "text", "initial": "none",
"desc": "Status condition: none, poisoned, burned, paralyzed, frozen, asleep, or fainted."
}
}
},
"wartortle": {
"name": "Wartortle",
"keys": "Wartortle",
"desc": "The player's own Pokémon. Water. Bulky, and super effective on the whole of Milo's team.",
"stats": {
"hp": {
"desc": "Wartortle's HP. Send the change, not the new total: a negative number for damage taken, a positive number for HP restored.",
"min": 0, "max": 92, "initial": 92, "max_delta_per_turn": 60,
"bands": [[0, 1, "fainted"], [1, 23, "critical"], [23, 48, "hurt"], [48, 75, "scuffed"], [75, 93, "healthy"]]
},
"status": {
"type": "text", "initial": "none",
"desc": "Status condition: none, poisoned, burned, paralyzed, frozen, asleep, or fainted."
}
}
},
"machoke": {
"name": "Machoke",
"keys": "Machoke",
"desc": "The player's own Pokémon. Fighting. Hits Rock-types hardest of the team but is slow and takes Ground moves badly.",
"stats": {
"hp": {
"desc": "Machoke's HP. Send the change, not the new total: a negative number for damage taken, a positive number for HP restored.",
"min": 0, "max": 104, "initial": 104, "max_delta_per_turn": 65,
"bands": [[0, 1, "fainted"], [1, 26, "critical"], [26, 54, "hurt"], [54, 84, "scuffed"], [84, 105, "healthy"]]
},
"status": {
"type": "text", "initial": "none",
"desc": "Status condition: none, poisoned, burned, paralyzed, frozen, asleep, or fainted."
}
}
},
"pikachu": {
"name": "Pikachu",
"keys": "Pikachu",
"desc": "The player's own Pokémon. Electric. Frail, and its Electric moves do nothing at all to Ground-types.",
"stats": {
"hp": {
"desc": "Pikachu's HP. Send the change, not the new total: a negative number for damage taken, a positive number for HP restored.",
"min": 0, "max": 76, "initial": 76, "max_delta_per_turn": 60,
"bands": [[0, 1, "fainted"], [1, 19, "critical"], [19, 40, "hurt"], [40, 62, "scuffed"], [62, 77, "healthy"]]
},
"status": {
"type": "text", "initial": "none",
"desc": "Status condition: none, poisoned, burned, paralyzed, frozen, asleep, or fainted."
}
}
},
"milo": {
"name": "Trainer Milo",
"keys": "Milo, opponent, Graveler, Onix, Kabutops, rock",
"desc": "The Round One opponent, a Rock-type specialist with three Pokémon: Graveler, Onix, and Kabutops. He leads with Graveler, switches to punish a bad matchup, and aims for whichever of the player's Pokémon is closest to fainting.",
"stats": {
"active_pokemon": {
"type": "text", "initial": "Graveler",
"desc": "Which Pokémon Milo currently has on the field: Graveler, Onix, or Kabutops. Replace this the moment he sends out a different one."
},
"active_hp": {
"desc": "The HP of Milo's active Pokémon. Send the change, not the new total: a negative number for damage taken, a positive number for HP restored. When Milo sends out a different Pokémon, send the positive change that carries this from where it stands to the newcomer's full HP: Graveler 98, Onix 90, Kabutops 88.",
"min": 0, "max": 98, "initial": 98, "max_delta_per_turn": 98,
"bands": [[0, 1, "fainted"], [1, 25, "critical"], [25, 50, "hurt"], [50, 78, "scuffed"], [78, 99, "healthy"]]
},
"active_status": {
"type": "text", "initial": "none",
"desc": "The status condition on Milo's active Pokémon: none, poisoned, burned, paralyzed, frozen, asleep, or fainted."
},
"pokemon_fainted": {
"desc": "How many of Milo's three Pokémon have fainted. Add 1 each time one goes down. Milo loses the round when this reaches 3.",
"type": "counter",
"min": 0, "max": 3, "initial": 0, "max_delta_per_turn": 1,
"bands": [[0, 1, "full team"], [1, 2, "two left"], [2, 3, "last Pokémon"], [3, 4, "defeated"]]
}
}
}
},
"milestones": {
"graveler_defeated": { "desc": "Knock out Milo's lead Graveler" },
"status_landed": { "desc": "Inflict a status condition on one of Milo's Pokémon" },
"type_advantage_used": { "desc": "Land a super-effective hit by switching in the right Pokémon" },
"round_one_won": { "desc": "Knock out all three of Milo's Pokémon and advance to Round Two" }
}
},
"story_cards": [
{
"type": "location",
"name": "Championship Stadium",
"keys": "stadium, arena, battlefield, field, terrain",
"entry": "The League Championship's main battlefield. The floor is broken rock terrain, which gives Rock- and Ground-type moves solid footing and gives grounded Pokémon cover to duck behind. Eight rounds stand between any Trainer and the Champion's title, and this is Round One.",
"notes": "Rocky terrain. Rock and Ground moves get a slight edge. Boulders can be used as cover or thrown."
},
{
"type": "character",
"name": "Trainer Milo",
"keys": "Milo, opponent, rock trainer",
"entry": "Milo is the Round One opponent, a Rock-type specialist who won his qualifiers on raw defense. His team is Graveler (Rock/Ground), Onix (Rock/Ground), and Kabutops (Rock/Water). He leads with Graveler, likes Sandstorm and Rock Slide, and switches out the moment a matchup turns against him.",
"notes": "Graveler and Onix are weak to Water, Grass, Fighting, Ground, Ice, and Steel, and are immune to Electric. Kabutops is weak to Grass, Fighting, Ground, and Electric. All three resist Normal and Flying."
},
{
"type": "item",
"name": "Hyper Potion",
"keys": "potion, hyper potion, heal, item, bag",
"entry": "A pressurized spray that closes wounds fast. One dose restores a large chunk of a Pokémon's HP, or clears a single status condition instead. Using one costs your turn, and the opponent attacks freely while you do it. You start the round with three and cannot restock mid-battle.",
"notes": "Decrement player.potions by 1 per use. A potion cannot revive a fainted Pokémon."
}
],
"scripts": []
}
backend/app/sse.py
+30
View File
@@ -0,0 +1,30 @@
"""Server-sent events: the wire format, the headers, and the error frame.
Two routers stream: the turn engine in `routers/adventures/turns.py` and the
chat scratchpad in `routers/chat.py`. Both send JSON objects as SSE `data:`
frames, so the format lives here rather than in either one.
"""
import json
from . import analytics
# `no-cache` stops an intermediary from caching the stream. `X-Accel-Buffering`
# makes nginx-style reverse proxies, which hosted deploys use, flush each event
# immediately rather than buffer it.
SSE_HEADERS = {"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}
def sse(obj: dict) -> str:
"""Returns one SSE frame carrying `obj` as JSON."""
return f"data: {json.dumps(obj)}\n\n"
def turn_error(detail: str, **extra) -> str:
"""Returns an SSE error for a turn that could not be produced, and counts it.
A failed turn is still an HTTP 200 response, so the middleware's status-code
tally cannot see it. This metric exists so that a demo whose model refuses
every request does not report as healthy.
"""
analytics.record(analytics.M_EVENT, analytics.EV_TURN_ERROR)
return sse({"type": "error", "detail": detail, **extra})
backend/app/starter.py
+92
View File
@@ -0,0 +1,92 @@
"""Gives every new guest a short adventure that is already played.
A guest arriving on an empty account has nothing to look at, and the demo turns
are limited, so learning what the app does used to cost one of them. This copies
a small pre-played adventure into the new account instead. It opens on a story
with real turns in it, and the turn summaries show what the world-state engine
records: the changes it applied, and the changes it refused.
The file in `starter_data/` is an ordinary export bundle, produced by
`GET /api/adventures/{id}/export` and trimmed to its first two exchanges. To
replace it, play a new adventure, export it, and overwrite the file. Nothing
else here knows what the story contains.
The copy is the guest's own from the first moment: they can edit it, branch it,
delete it, or export it, and nothing links it back to the file. A returning
guest is not given a second one, because this runs only where a guest row is
created.
"""
import json
import logging
from pathlib import Path
from sqlalchemy.orm import Session
from . import bundle, models, seed
logger = logging.getLogger(__name__)
STARTER_FILE = Path(__file__).resolve().parent / "starter_data" / "pokemon-league.json"
def _load() -> dict | None:
"""Returns the starter bundle, or None when the file is missing or invalid.
The file ships with the server, so a failure here is a packaging mistake
rather than bad input. It is still tolerated: a guest with no starter
adventure can play, and a guest who cannot be created cannot.
"""
try:
return json.loads(STARTER_FILE.read_text(encoding="utf-8"))
except (OSError, ValueError):
logger.exception("Starter adventure is missing or unreadable; skipping it.")
return None
def give(db: Session, user: models.User) -> models.Adventure | None:
"""Copies the starter adventure into `user`, and returns it.
The caller commits. This writes rows and does not commit them, so a guest
and their first adventure land in one transaction: an account is never left
half-populated.
Failures are logged and swallowed. The starter is a convenience, and losing
it must not cost the visitor their session.
"""
payload = _load()
if payload is None:
return None
try:
# A savepoint, so a failure halfway through writing the tree discards
# only the starter's rows. Without it the caller's next commit would
# flush whatever part of the adventure the session still held.
with db.begin_nested():
story = bundle.plan(payload, bundle.check_format(payload))
adventure = bundle.materialize(db, payload, story, user.id)
_link_scenario(db, adventure, payload)
return adventure
except Exception:
logger.exception("Could not give user %s the starter adventure.", user.id)
return None
def _link_scenario(db: Session, adventure: models.Adventure, payload: dict) -> None:
"""Points the copy at the demo scenario it came from, when that seed exists.
A bundle does not carry a scenario id, because the id is local to one
database and an export outlives it. The link matters anyway: an adventure
has no cover art of its own, and inherits the scenario's. Without it the
starter shows a monogram tile while the demo it came from shows its own
artwork.
The name is read from `scenarioTitle` in the file. If no seeded scenario
answers to it, the adventure keeps a NULL `scenario_id`, which is the same
state an imported bundle is in and costs only the artwork.
"""
title = str(payload.get("scenarioTitle") or "").strip()
if not title:
return
scenario = seed.find_seeded(db, title)
if scenario is not None:
adventure.scenario_id = scenario.id
backend/app/starter_data/pokemon-league.json
+516
View File
@@ -0,0 +1,516 @@
{
"title": "Pokemon League Championship: Round One",
"scenarioTitle": "[Demo] Pokemon League Championship: Round One",
"format": "ai-dnd-adventure-v2",
"memory": "The player is a Trainer in Round One of an eight-round League Championship. The Round One opponent is Milo, a Rock-type specialist with three Pokémon: Graveler (Rock/Ground), Onix (Rock/Ground), and Kabutops (Rock/Water). The player's team is Pidgeotto (Normal/Flying), Ivysaur (Grass/Poison), Wartortle (Water), Machoke (Fighting), and Pikachu (Electric). Pidgeotto is out first. The player carries Hyper Potions, which restore HP and can be swapped for a status heal when needed. A battle is lost when every Pokémon on one side has fainted.",
"authorsNote": "Play it like a real battle. Type matchups decide damage, status conditions stick around and cost the afflicted Pokémon turns, and switching costs the player a turn while the opponent attacks freely. Milo is competent: he switches to punish bad matchups and targets whatever is weakest.",
"aiInstructions": "Write in second person, present tense. Narrate the battle move by move, naming the active Pokémon on each side and the move each one uses. End every reply at a decision point where the player can choose a move, switch Pokémon, or use an item.\n\nReflect type advantage and disadvantage in the damage you narrate and in the HP numbers: a super-effective hit should take roughly double a neutral hit, a resisted hit roughly half. Ground and Rock moves hit Pikachu and Pidgeotto hard. Water, Grass, and Fighting moves are strong against Milo's Rock-types.\n\nEvery turn, update the state. Every number you send is a change to add to the current value, never the value you want it to end up at. The world state lines show you totals; the block you write carries the difference. A Pokémon at 96 HP that takes 30 damage is `-30`, not `66`.\n- Add 1 to `world.turn` in every reply, including this one. It counts the turns this battle has lasted, so it moves whether or not anything else does.\n- Send a negative number for HP lost and a positive number for HP restored on both sides (`npc.<name>.hp` for the player's own Pokémon, e.g. npc.pidgeotto.hp; `npc.milo.active_hp` for Milo's). Every Pokémon starts at its full HP, so a positive number on an undamaged one changes nothing.\n- Set a Pokémon's `status` stat (`npc.<name>.status`, or `npc.milo.active_status`) to the condition it just picked up: poisoned, burned, paralyzed, frozen, asleep, or none when it is cured or wakes up. Status matters. Poison and burn shave HP at the end of each turn, paralysis and freeze cost turns, sleep stops a Pokémon acting until it wakes.\n- When the player switches, set `player.active_pokemon` to the incoming Pokémon's name in full (not a delta).\n- When the player uses a potion, send `-1` for `player.potions` and a positive number for the healed Pokémon's HP. The count falls when a potion is spent. It never rises. Potions cannot be used once the count reaches 0.\n- When a Pokémon's HP hits 0 it faints, its status becomes `fainted`, and its trainer must send out a replacement (the player picks the next `player.active_pokemon`). A fainted Pokémon cannot be healed or switched back in.\n- Whenever Milo sends out a new Pokémon, set `npc.milo.active_pokemon` and `npc.milo.active_status` to their new values in full, and send `npc.milo.active_hp` as the positive change that carries it from where it stands to the newcomer's full HP. From a fainted Pokémon at 0 that is the newcomer's full HP: Graveler 98, Onix 90, Kabutops 88.\n\nWhen one of Milo's Pokémon faints, do both of these in the same reply: send `1` for `npc.milo.pokemon_fainted`, which adds one to the count rather than setting it, and set `npc.milo.active_pokemon` with a fresh `npc.milo.active_hp` and `npc.milo.active_status` for the one he sends out next. `npc.milo.pokemon_fainted` counts up from 0 and never goes down.\n\nMark each milestone as it happens, using the id shown beside the goal in the world state. Do not decide the whole battle in one reply; give the player a turn between every exchange.",
"storySummary": "",
"scriptState": {},
"worldState": {
"world": {
"round": 1,
"turn": 2
},
"player": {
"potions": 3,
"active_pokemon": "Wartortle"
},
"npc": {
"pidgeotto": {
"hp": 84,
"status": "none"
},
"ivysaur": {
"hp": 96,
"status": "none"
},
"wartortle": {
"hp": 84,
"status": "none"
},
"machoke": {
"hp": 104,
"status": "none"
},
"pikachu": {
"hp": 76,
"status": "none"
},
"milo": {
"active_pokemon": "Onix",
"active_hp": 90,
"active_status": "none",
"pokemon_fainted": 1
}
},
"milestones": {
"graveler_defeated": {
"reached": true,
"at": 4
}
},
"flags": {
"sandstorm_active": false
},
"_meta": {
"last_changed": {
"world.turn": 4,
"player.active_pokemon": 2,
"npc.wartortle.hp": 2,
"npc.milo.active_hp": 4,
"npc.milo.pokemon_fainted": 4,
"npc.milo.active_pokemon": 4
}
}
},
"autoSummarize": false,
"memoryBankEnabled": false,
"branches": [
{
"parent": null,
"forkDepth": null
}
],
"headBranch": 0,
"memoryCursor": {
"branch": null,
"depth": -1
},
"summaryCursor": {
"branch": null,
"depth": -1
},
"memories": [],
"storyCards": [
{
"type": "location",
"name": "Championship Stadium",
"keys": "stadium, arena, battlefield, field, terrain",
"entry": "The League Championship's main battlefield. The floor is broken rock terrain, which gives Rock- and Ground-type moves solid footing and gives grounded Pokémon cover to duck behind. Eight rounds stand between any Trainer and the Champion's title, and this is Round One.",
"notes": "Rocky terrain. Rock and Ground moves get a slight edge. Boulders can be used as cover or thrown."
},
{
"type": "character",
"name": "Trainer Milo",
"keys": "Milo, opponent, rock trainer",
"entry": "Milo is the Round One opponent, a Rock-type specialist who won his qualifiers on raw defense. His team is Graveler (Rock/Ground), Onix (Rock/Ground), and Kabutops (Rock/Water). He leads with Graveler, likes Sandstorm and Rock Slide, and switches out the moment a matchup turns against him.",
"notes": "Graveler and Onix are weak to Water, Grass, Fighting, Ground, Ice, and Steel, and are immune to Electric. Kabutops is weak to Grass, Fighting, Ground, and Electric. All three resist Normal and Flying."
},
{
"type": "item",
"name": "Hyper Potion",
"keys": "potion, hyper potion, heal, item, bag",
"entry": "A pressurized spray that closes wounds fast. One dose restores a large chunk of a Pokémon's HP, or clears a single status condition instead. Using one costs your turn, and the opponent attacks freely while you do it. You start the round with three and cannot restock mid-battle.",
"notes": "Decrement player.potions by 1 per use. A potion cannot revive a fainted Pokémon."
},
{
"type": "character",
"name": "Pidgeotto",
"keys": "Pidgeotto",
"entry": "The player's own Pokémon. Normal/Flying. Fast, but Rock-type moves hit it for four times damage.",
"notes": ""
},
{
"type": "character",
"name": "Ivysaur",
"keys": "Ivysaur",
"entry": "The player's own Pokémon. Grass/Poison. Super effective on Milo's Rock/Ground types; can inflict poison or sleep.",
"notes": ""
},
{
"type": "character",
"name": "Wartortle",
"keys": "Wartortle",
"entry": "The player's own Pokémon. Water. Bulky, and super effective on the whole of Milo's team.",
"notes": ""
},
{
"type": "character",
"name": "Machoke",
"keys": "Machoke",
"entry": "The player's own Pokémon. Fighting. Hits Rock-types hardest of the team but is slow and takes Ground moves badly.",
"notes": ""
},
{
"type": "character",
"name": "Pikachu",
"keys": "Pikachu",
"entry": "The player's own Pokémon. Electric. Frail, and its Electric moves do nothing at all to Ground-types.",
"notes": ""
}
],
"scripts": [],
"actions": [
{
"branch": 0,
"depth": 0,
"live": true,
"type": "start",
"text": "The floodlights come up over the championship field and the referee raises both flags. Round One of eight. Beat the trainer in front of you or go home.\n\nAcross the packed rock terrain, Trainer Milo tosses his first Poké Ball. Graveler lands hard enough to crack the stone under it, four arms flexing, and settles into a low stance.\n\n\"Rock-types all the way down,\" Milo calls over the noise. \"Hope you brought something that hits harder than it looks.\"\n\nYour Pidgeotto is already in the air, wings locked into a slow circle above the field. It is fast, but Rock-type moves will tear it out of the sky, and Graveler is built to take a hit. Three Hyper Potions sit in your belt pouch.\n\nGraveler grinds one fist into its palm. The referee drops the flag.\n\nWhat's your move?",
"createdAt": "2026-08-28T12:24:29.635984",
"stateAfter": {},
"worldStateAfter": {
"world": {
"round": 1,
"turn": 0
},
"player": {
"potions": 3,
"active_pokemon": "Pidgeotto"
},
"npc": {
"pidgeotto": {
"hp": 84,
"status": "none"
},
"ivysaur": {
"hp": 96,
"status": "none"
},
"wartortle": {
"hp": 92,
"status": "none"
},
"machoke": {
"hp": 104,
"status": "none"
},
"pikachu": {
"hp": 76,
"status": "none"
},
"milo": {
"active_pokemon": "Graveler",
"active_hp": 98,
"active_status": "none",
"pokemon_fainted": 0
}
},
"milestones": {},
"flags": {
"sandstorm_active": false
},
"_meta": {
"last_changed": {}
}
}
},
{
"branch": 0,
"depth": 1,
"live": true,
"type": "do",
"text": "> You Recall Pidgeotto and send out Wartortle, then hit Graveler with a full-power Water Gun.",
"createdAt": "2026-08-28T12:24:48.236580",
"stateAfter": {},
"worldStateAfter": {
"world": {
"round": 1,
"turn": 0
},
"player": {
"potions": 3,
"active_pokemon": "Pidgeotto"
},
"npc": {
"pidgeotto": {
"hp": 84,
"status": "none"
},
"ivysaur": {
"hp": 96,
"status": "none"
},
"wartortle": {
"hp": 92,
"status": "none"
},
"machoke": {
"hp": 104,
"status": "none"
},
"pikachu": {
"hp": 76,
"status": "none"
},
"milo": {
"active_pokemon": "Graveler",
"active_hp": 98,
"active_status": "none",
"pokemon_fainted": 0
}
},
"milestones": {},
"flags": {
"sandstorm_active": false
},
"_meta": {
"last_changed": {}
}
}
},
{
"branch": 0,
"depth": 2,
"live": true,
"type": "ai",
"text": "You call Pidgeotto back with a flash of red light, and it banks gratefully out of Graveler's range before the crack of a Rock Throw can find it. Wartortle drops onto the broken stone in its place, planting its feet, shell catching the floodlights.\n\n\"Smart,\" Milo grunts, \"but Graveler doesn't fold that easy.\" He points. \"Rock Throw!\"\n\nGraveler wrenches a chunk of the battlefield loose and hurls it in a flat, brutal arc. But Wartortle's already moving, ducking low behind a natural outcrop for a half-second of cover before the shard clips its shoulder — a glancing hit, more noise than damage.\n\n\"Wartortle, Water Gun — full power!\"\n\nWartortle plants both feet and unleashes a roaring jet of water straight into Graveler's chest. It's not a graceful hit, it's a drowning one — the stream slams Graveler backward, hissing against hot rock, and the four-armed boulder staggers, one knee grinding into the dirt before it forces itself back upright, dripping and furious.\n\n\"That's the matchup I didn't want to see,\" Milo mutters, though he's grinning like he half-expected it.\n\nGraveler shakes itself off, glaring, arms flexing back into a battle-ready stance. It's hurt — badly — but still standing, and still dangerous at close range.\n\nThe referee's flag stays down. It's still your move.",
"createdAt": "2026-08-28T12:25:00.242852",
"stateAfter": {},
"worldStateAfter": {
"world": {
"round": 1,
"turn": 1
},
"player": {
"potions": 3,
"active_pokemon": "Wartortle"
},
"npc": {
"pidgeotto": {
"hp": 84,
"status": "none"
},
"ivysaur": {
"hp": 96,
"status": "none"
},
"wartortle": {
"hp": 84,
"status": "none"
},
"machoke": {
"hp": 104,
"status": "none"
},
"pikachu": {
"hp": 76,
"status": "none"
},
"milo": {
"active_pokemon": "Graveler",
"active_hp": 52,
"active_status": "none",
"pokemon_fainted": 0
}
},
"milestones": {},
"flags": {
"sandstorm_active": false
},
"_meta": {
"last_changed": {
"world.turn": 2,
"player.active_pokemon": 2,
"npc.wartortle.hp": 2,
"npc.milo.active_hp": 2
}
}
},
"worldDelta": {
"delta": {
"world.turn": 1,
"player.active_pokemon": "Wartortle",
"npc.wartortle.hp": -8,
"npc.milo.active_hp": -46,
"flags.type_advantage_used": true
},
"applied": [
{
"path": "world.turn",
"old": 0,
"new": 1
},
{
"path": "player.active_pokemon",
"old": "Pidgeotto",
"new": "Wartortle"
},
{
"path": "npc.wartortle.hp",
"old": 92,
"new": 84
},
{
"path": "npc.milo.active_hp",
"old": 98,
"new": 52
}
],
"clamped": [],
"rejected": [
{
"path": "flags.type_advantage_used",
"reason": "unknown flag",
"fix": "There is no flag `type_advantage_used`. The flags are: `sandstorm_active`."
}
]
}
},
{
"branch": 0,
"depth": 3,
"live": true,
"type": "do",
"text": "> You Wartortle hits Graveler with everything it has left, a point-blank Hydro Pump to finish it.",
"createdAt": "2026-08-28T12:25:21.664571",
"stateAfter": {},
"worldStateAfter": {
"world": {
"round": 1,
"turn": 1
},
"player": {
"potions": 3,
"active_pokemon": "Wartortle"
},
"npc": {
"pidgeotto": {
"hp": 84,
"status": "none"
},
"ivysaur": {
"hp": 96,
"status": "none"
},
"wartortle": {
"hp": 84,
"status": "none"
},
"machoke": {
"hp": 104,
"status": "none"
},
"pikachu": {
"hp": 76,
"status": "none"
},
"milo": {
"active_pokemon": "Graveler",
"active_hp": 52,
"active_status": "none",
"pokemon_fainted": 0
}
},
"milestones": {},
"flags": {
"sandstorm_active": false
},
"_meta": {
"last_changed": {
"world.turn": 2,
"player.active_pokemon": 2,
"npc.wartortle.hp": 2,
"npc.milo.active_hp": 2
}
}
}
},
{
"branch": 0,
"depth": 4,
"live": true,
"type": "ai",
"text": "Wartortle plants its stubby legs against the cracked stone, shell rattling as it draws breath for the biggest attack it has. \"Finish it — Hydro Pump!\" you shout.\n\nThe blast that follows isn't a stream, it's a cannon — a roaring column of pressurized water that closes the gap before Graveler can even brace. It catches the rock-type square in the chest and drives it backward across the broken terrain, gouging a furrow through the stone floor. Graveler's arms windmill once, twice, searching for purchase that isn't there, and then it simply goes down, crashing onto its back in a spreading puddle, swirl-eyed and still.\n\nMilo's jaw tightens, but he's already reaching for his second ball. \"Didn't want to see that either. Fine — you earned that one.\" He hurls it out. \"Onix, let's go!\"\n\nThe ground itself seems to shudder as Onix erupts from the earth in a towering column of jointed rock, easily three stories high, boulder head swinging around to fix its small eyes on Wartortle. Dust and loose gravel rain down from the impact of its arrival, pattering off your shoulders even from across the field.\n\n\"Rock/Ground,\" you murmur to yourself, but Wartortle doesn't flinch — Water still chews through that typing just as easily, and the referee's flag is already dropping to signal the next exchange.\n\nWartortle shakes water from its shell, tired from the Hydro Pump but still standing tall, eyeing the massive serpent of stone now looming over it.",
"createdAt": "2026-08-28T12:25:34.346927",
"stateAfter": {},
"worldStateAfter": {
"world": {
"round": 1,
"turn": 2
},
"player": {
"potions": 3,
"active_pokemon": "Wartortle"
},
"npc": {
"pidgeotto": {
"hp": 84,
"status": "none"
},
"ivysaur": {
"hp": 96,
"status": "none"
},
"wartortle": {
"hp": 84,
"status": "none"
},
"machoke": {
"hp": 104,
"status": "none"
},
"pikachu": {
"hp": 76,
"status": "none"
},
"milo": {
"active_pokemon": "Onix",
"active_hp": 90,
"active_status": "none",
"pokemon_fainted": 1
}
},
"milestones": {
"graveler_defeated": {
"reached": true,
"at": 4
}
},
"flags": {
"sandstorm_active": false
},
"_meta": {
"last_changed": {
"world.turn": 4,
"player.active_pokemon": 2,
"npc.wartortle.hp": 2,
"npc.milo.active_hp": 4,
"npc.milo.pokemon_fainted": 4,
"npc.milo.active_pokemon": 4
}
}
},
"worldDelta": {
"delta": {
"world.turn": 1,
"npc.milo.active_hp": -52,
"npc.milo.pokemon_fainted": 1,
"npc.milo.active_pokemon": "Onix",
"npc.milo.active_status": "none",
"milestones.graveler_defeated": true
},
"applied": [
{
"path": "world.turn",
"old": 1,
"new": 2
},
{
"path": "npc.milo.active_hp",
"old": 52,
"new": 0
},
{
"path": "npc.milo.pokemon_fainted",
"old": 0,
"new": 1
},
{
"path": "npc.milo.active_pokemon",
"old": "Graveler",
"new": "Onix"
},
{
"path": "milestones.graveler_defeated",
"old": false,
"new": true
}
],
"clamped": [],
"rejected": []
}
}
]
}
backend/app/tree.py
+412
View File
@@ -0,0 +1,412 @@
"""Phase 14: writes nodes onto the story tree.
This module is the write half of the tree. It decides which branch a new node
goes on, what depth the node gets, and where the adventure's head points. All
three decisions live here because a mistake in any of them is silent. A node
written without a branch is invisible to every read, and nothing raises an
error.
The read half is `context/lineage.py`. It turns a branch into the set of nodes
that make up one story.
Until forking ships, each adventure has one branch and `depth` mirrors `index`,
so nothing here changes observable behavior yet. That is intentional. By the
time reads depend on these columns, every row already has them, including the
rows written between the two deploys that no migration visits.
SP2 added `place_new_nodes`, which runs on every flush. Wiring up individual
call sites worked while nothing read the columns. Now that reads filter on them,
relying on each writer to remember would also mean relying on every fixture,
script, and test. A missed call produces a story with missing turns, so the
flush enforces the rule instead.
"""
import copy
from sqlalchemy import func, insert, update
from sqlalchemy.orm import Session
from . import models
from .context import lineage
# Head depth of an adventure that has no actions. Using -1 keeps the rule "the
# next node goes at head_depth + 1" true without a special case. This matches
# `migrations.NO_DEPTH`.
NO_DEPTH = -1
def root_branch(db: Session, adventure: models.Adventure) -> models.Branch:
"""Returns the adventure's root branch, creating it on first use.
This function gets or creates the branch instead of creating it alongside
the adventure. The adventures that need a root branch are usually ones that
already exist: an imported bundle, a fixture built through the ORM, or a
database migrated before this code shipped.
"""
branch = (
db.query(models.Branch)
.filter(
models.Branch.adventure_id == adventure.id,
models.Branch.parent_branch_id.is_(None),
)
.order_by(models.Branch.id)
.first()
)
if branch is not None:
return branch
# Use a Core insert instead of the ORM. `place_new_nodes` can call this
# during a flush, and a nested ORM flush raises an error. Both paths share
# one transaction. The lineage refers to the branch's own id, so it needs a
# second statement, which runs once per adventure.
new_id = db.execute(
insert(models.Branch).values(
adventure_id=adventure.id,
parent_branch_id=None,
fork_depth=None,
lineage=[],
created_at=models.utcnow(),
)
).inserted_primary_key[0]
db.execute(
update(models.Branch)
.where(models.Branch.id == new_id)
.values(lineage=[[new_id, None]])
)
return db.get(models.Branch, new_id)
def head_branch(db: Session, adventure: models.Adventure) -> models.Branch:
"""Returns the branch that new nodes are played onto."""
if adventure.head_branch_id is not None:
branch = db.get(models.Branch, adventure.head_branch_id)
if branch is not None:
return branch
# The head points at a branch that no longer exists, which means a bug
# elsewhere. Fall back to the root instead of refusing to play,
# otherwise the adventure becomes unusable.
branch = root_branch(db, adventure)
adventure.head_branch_id = branch.id
return branch
def fork(db: Session, adventure: models.Adventure, node: models.Action) -> models.Branch:
"""Moves `node` onto a new branch so the story can continue from it.
`node` is a discarded attempt at a turn that the story has already moved
past. Making it live where it stands would orphan every turn played after
it, because those turns continue the attempt that won. Instead, `node` moves
to a new branch that forks from the depth just before it. The parent branch
keeps its story unchanged. The new branch inherits everything up to the fork
and owns this one node.
This function inserts one row and moves one row. It copies nothing, so a
fork costs one `branches` row plus the ancestry cached on it, regardless of
how long the story is.
Derived data stays on the parent, by design. A memory attaches to the node
its block ends on, and that node does not move. From the new branch, the
lineage caps the parent at `fork_depth`, so the memory sits one depth past
the border. Neither retrieval nor the cursors can see it, and the block is
summarized again from the text this branch contains.
The other attempts at this turn also stay on the parent, because they are
still takes on the parent's turn. If none of them is live, the oldest one
becomes live, so the parent keeps the line it was written on.
"""
parent = db.get(models.Branch, node.branch_id)
if parent is None or node.depth is None:
raise ValueError("cannot fork from a node that is not on a branch")
fork_depth = node.depth - 1
# Read the sibling attempts before moving the node. The session does not
# autoflush, so a later read still finds the node here and hands the live
# flag back to it.
remaining = [
row for row in db.query(models.Action)
.filter(
models.Action.adventure_id == adventure.id,
models.Action.branch_id == parent.id,
models.Action.depth == node.depth,
)
.order_by(models.Action.id)
.all()
if row is not node
]
# The parent's ancestry, with every entry capped at the fork depth. Only the
# first entry can change in practice, because older entries are already
# capped at a shallower depth. Capping all of them states the invariant
# directly.
inherited = [
[branch_id, fork_depth if cap is None else min(cap, fork_depth)]
for branch_id, cap in lineage.entries_of(parent)
]
# Core insert with the lineage written second, for the reason given in
# `root_branch`. This code can run inside a flush, and the lineage refers to
# the new row's own id.
new_id = db.execute(
insert(models.Branch).values(
adventure_id=adventure.id,
parent_branch_id=parent.id,
fork_depth=fork_depth,
lineage=[],
created_at=models.utcnow(),
)
).inserted_primary_key[0]
db.execute(
update(models.Branch)
.where(models.Branch.id == new_id)
.values(lineage=[[new_id, None]] + inherited)
)
depth = node.depth
node.branch_id = new_id
node.live = True
# The group the node left needs a live attempt again. Taking the oldest is
# arbitrary, and it has to be somebody: a coordinate with no live attempt
# disappears from the story on the branch it was left on.
if remaining and not any(row.live for row in remaining):
remaining[0].live = True
adventure.head_branch_id = new_id
adventure.head_depth = depth
return db.get(models.Branch, new_id)
def branch_at(
db: Session, adventure: models.Adventure, fork_depth: int
) -> models.Branch:
"""Creates an empty branch that leaves the current path at `fork_depth`.
`fork` moves an existing node onto its own branch. This function creates the
same kind of branch with no nodes on it yet, for the case where the take
that will live there does not exist. A player asking for another take of a
turn the story has moved past reaches this path (SP9).
The head lands at `fork_depth`, so the next node written becomes the new
take. That node gets the same depth as the original and the same parent,
which `place_action` derives from the path.
This function does not modify the branch being left. That branch keeps its
node at that depth, the node stays live, and every turn played after it
stays in place.
"""
parent = head_branch(db, adventure)
inherited = [
[branch_id, fork_depth if cap is None else min(cap, fork_depth)]
for branch_id, cap in lineage.entries_of(parent)
]
# Core insert with the lineage written second, for the reason given in
# `root_branch`. This code can run inside a flush, and the lineage refers to
# the new row's own id.
new_id = db.execute(
insert(models.Branch).values(
adventure_id=adventure.id,
parent_branch_id=parent.id,
fork_depth=fork_depth,
lineage=[],
created_at=models.utcnow(),
)
).inserted_primary_key[0]
db.execute(
update(models.Branch)
.where(models.Branch.id == new_id)
.values(lineage=[[new_id, None]] + inherited)
)
adventure.head_branch_id = new_id
adventure.head_depth = fork_depth
return db.get(models.Branch, new_id)
def place_action(
db: Session,
adventure: models.Adventure,
action: models.Action,
branch: models.Branch | None = None,
parent: models.Action | None = None,
) -> models.Branch:
"""Puts `action` on the head branch and moves the head to it.
An action with no `depth` of its own goes one step past the tip, which is
where the next turn belongs. The opening of a new adventure lands at 0 that
way, because an adventure with nothing played has a head depth of
`NO_DEPTH`.
Pass `branch` when you have already resolved the head and are placing
several nodes at once. See `place_new_nodes` for why that is worth doing.
`parent` is the take that this node was played after (SP9), and it is what
groups the takes of one turn. If you omit it, this function derives it from
the path, which is the correct default: a node written now follows the story
the player is reading now. If you place several nodes in one flush, chain
`parent` explicitly, because the database has not seen any of them yet.
"""
branch = branch or head_branch(db, adventure)
action.branch_id = branch.id
if action.depth is None:
action.depth = adventure.head_depth + 1
if parent is not None:
action.parent_id = parent.id
elif action.parent_id is None and action.depth:
action.parent_id = _preceding_id(db, adventure, branch, action.depth)
adventure.head_branch_id = branch.id
if action.depth is not None and action.depth > adventure.head_depth:
adventure.head_depth = action.depth
return branch
def _preceding_id(
db: Session, adventure: models.Adventure, branch: models.Branch, depth: int
) -> int | None:
"""Returns the id of the live node one step back along `branch`'s path.
The query searches the whole lineage instead of `branch` alone, because a
branch inherits the story before its fork point. The node in front of a
forked branch's first turn lives on an ancestor, and that node is the parent
a pager needs in order to find siblings.
"""
path = lineage.Path(lineage.entries_of(branch))
return (
db.query(models.Action.id)
.filter(
models.Action.adventure_id == adventure.id,
models.Action.depth == depth - 1,
models.Action.live.is_(True),
path.clause(),
)
.order_by(models.Action.id)
.limit(1)
.scalar()
)
def place_memory(
db: Session,
adventure: models.Adventure,
memory: models.Memory,
branch: models.Branch | None = None,
) -> models.Branch:
"""Attaches a memory to the node it belongs to.
`source_end` is the index of the last action the memory summarizes, which is
that node's depth. A hand-written memory summarizes no actions, so it uses
the head instead. That records the story the author was reading at the time.
Giving every memory a coordinate is what makes its scope unambiguous (SP7).
Before SP7, hand-written memories had a NULL depth and belonged to the
adventure rather than to a path. A fork cannot cap a NULL, so those memories
followed the reader onto branches whose story they did not describe. Now the
question "is this memory part of the story I am reading?" has one answer for
every row in the bank, and it is the same answer the lineage gives for
nodes.
"""
branch = branch or head_branch(db, adventure)
memory.branch_id = branch.id
if memory.depth is None:
memory.depth = (
memory.source_end if memory.source_end is not None
else adventure.head_depth
)
return branch
def attach_memory(memory: models.Memory, node: models.Action) -> None:
"""Attaches a memory to the node it was derived from.
This follows the general rule for derived data, and the memory bank is the
first case of it. Anything derived from the story attaches to the node that
produced it, and is then visible from exactly the paths that contain that
node. A fork inherits its ancestors' memories because it inherits their
nodes, so nothing is copied. A memory made on a sibling branch is not
visible, because that node is not on this path.
This function differs from `place_memory` because it takes the branch from
`node`, which is not always the head. A block of story can end before the
fork that created the current branch, and the memory belongs where that
block is.
"""
memory.branch_id = node.branch_id
memory.depth = node.depth
def stamp_outcome(adventure: models.Adventure, action: models.Action) -> None:
"""Records the state a node left behind, if the writer did not record it.
This is the fallback under `attempts.snapshot_outcome`, and it exists for
the same reason `place_action` has one. Since SP4, undo and retry cannot
roll back past a node that has no outcome, and the failure is silent: the
script state and world state stay where they were. The turn engine records
the outcome itself, so this function skips those rows. It catches fixtures,
scripts, and imports that write a story directly through the ORM.
The values written are the state as of the flush. That is correct, because a
writer that changes no state between two nodes leaves the same state behind
both of them.
"""
if action.state_after is None:
state = adventure.script_state if isinstance(adventure.script_state, dict) else {}
action.state_after = copy.deepcopy(state)
if action.world_state_after is None:
world = adventure.world_state if isinstance(adventure.world_state, dict) else {}
action.world_state_after = copy.deepcopy(world)
def place_new_nodes(session: Session) -> None:
"""Places every unplaced node that is about to be inserted.
This runs on every flush. Call sites still call `place_action` and
`place_memory` directly, and they should. Placing a node at the call site
happens before the surrounding code reads the row back, and the explicit
call makes that ordering visible. This function is the fallback under those
calls, so a fixture, script, test, or a call site added later still gets a
branch without knowing the tree exists.
Nodes whose adventure has not been inserted yet are skipped, because there
is no id to attach a branch to. This case does not arise in practice, since
an Action needs `adventure_id` before it can be written at all.
The head is resolved once per adventure per flush and cached in `heads`.
This saves more than a dictionary lookup. The identity map holds weak
references, so a branch row that nothing else refers to is collected between
two nodes and read from the database again for the next one. Resolving the
head per node turned a fixture that wrote 200 actions in one flush into 200
separate queries.
"""
heads: dict[int, models.Branch] = {}
for obj in list(session.new):
if isinstance(obj, models.Action):
place = place_action
elif isinstance(obj, models.Memory):
place = place_memory
else:
continue
if obj.adventure_id is None:
continue
adventure = session.get(models.Adventure, obj.adventure_id)
if adventure is None:
continue
if isinstance(obj, models.Action):
stamp_outcome(adventure, obj)
if obj.branch_id is not None:
continue # already placed at its call site
head = heads.get(adventure.id)
if head is None:
head = heads[adventure.id] = head_branch(session, adventure)
place(session, adventure, obj, head)
def refresh_head(db: Session, adventure: models.Adventure) -> None:
"""Recomputes the head depth after nodes are removed by undo or delete.
A branch with no nodes of its own sits at its fork point, because that is
the last node its story contains. The node is inherited from the parent, but
it is still the tip. A root branch with no nodes has no story at all.
"""
branch = head_branch(db, adventure)
tip = (
db.query(func.max(models.Action.depth))
.filter(models.Action.branch_id == branch.id)
.scalar()
)
if tip is None:
tip = branch.fork_depth if branch.fork_depth is not None else NO_DEPTH
adventure.head_depth = tip
backend/app/vectors.py
+52
View File
@@ -0,0 +1,52 @@
"""Storing and comparing embedding vectors.
A 1536-dimension vector written as a JSON list is about 31 kB, because every
component is written as a decimal string of about seventeen digits. The same
vector as packed float32 is 6,144 bytes, which is five times smaller, and the
memory bank is read in full on every turn, so those bytes are paid repeatedly.
Float32 is not an approximation here. The embedding endpoints return vectors
computed in float32 and render them into JSON as the shortest decimal string
that round-trips through a double. Converting that back to float32 recovers the
original bits exactly. Nothing that was present is lost, which is why the
conversion needs no re-embedding and carries no risk to retrieval quality.
Dimensions are deliberately unchanged. Dropping to 512 or 768 would have saved
another 3x and cost an API call per stored memory to re-embed, against a bank
that the packing and the in-process cache together already make cheap.
"""
import math
import struct
import sys
from array import array
def pack(vector) -> bytes:
"""A vector as little-endian float32."""
return struct.pack(f"<{len(vector)}f", *vector)
def unpack(blob: bytes) -> array:
"""The inverse of `pack`. Length is implied: four bytes per component.
Returns an `array("f")` rather than a list, because these are held in
memory between turns: the array is the same 4 bytes a component the column
is, where a list of Python floats is eight times that. It indexes, zips and
lens like a list, which is all the ranking needs.
"""
vector = array("f")
vector.frombytes(blob)
if sys.byteorder != "little":
vector.byteswap()
return vector
def cosine(a: list[float], b: list[float]) -> float:
# Different lengths means the embedding model changed since this vector was
# stored; zip() would silently score garbage.
if len(a) != len(b):
return 0.0
dot = sum(x * y for x, y in zip(a, b))
norm = math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b))
return dot / norm if norm else 0.0
backend/app/worldstate/__init__.py
+55
View File
@@ -0,0 +1,55 @@
"""Phase 12: RPG world state.
The scenario carries a `stat_schema`, which is the template: which stats exist,
their bands and rules, and the milestones. An adventure carries a live
`world_state` instantiated from it. Each turn the AI proposes a delta holding
only what changed. `apply_delta` decides what the delta is allowed to do. It
clamps values to min and max, caps the change per turn, enforces cooldowns, and
makes milestones sticky.
Nothing here raises on bad AI output. A malformed delta returns `{}` and the turn
continues, the same way a broken script never breaks a turn.
The work is split four ways, and each module reads on its own:
schema what a scenario's `stat_schema` allows, and building state from it
parse the block the model writes, and reading it back
apply every write to world state, and the limits it is held to
render turning state and schema into prompt text
Import the names from this package rather than from those modules. The split is
an implementation detail, and the names below are the interface.
"""
from .apply import apply_delta, apply_override
from .parse import (
EMIT_REMINDER,
EMIT_RULE,
applied_delta,
extract_delta,
refusals,
render_delta_block,
render_refusals,
)
from .render import render_reference, render_state_section
from .schema import band_label, has_schema, instantiate, npc_name, npc_triggers, reconcile
__all__ = [
"EMIT_REMINDER",
"EMIT_RULE",
"applied_delta",
"apply_delta",
"apply_override",
"band_label",
"extract_delta",
"has_schema",
"instantiate",
"npc_name",
"npc_triggers",
"reconcile",
"refusals",
"render_delta_block",
"render_reference",
"render_refusals",
"render_state_section",
]
backend/app/worldstate/apply.py
+407
View File
@@ -0,0 +1,407 @@
"""Applying a delta or an override to world state, within the schema's limits.
Every write to world state goes through here, so the clamping rules live in one
place. A proposed change that breaks a limit is recorded as refused rather than
dropped, because the player and the model both need to see that it did not land.
"""
import copy
from typing import NamedTuple
from .schema import STAT_SECTIONS, _initials, instantiate, npc_name
class _Target(NamedTuple):
"""Where one path writes, and what the schema says about the value there."""
kind: str # "flag", "milestone", or "stat"
section: str # "flags", "milestones", "player", "world", or "npc"
key: str # the key inside the container
stat_def: dict | None # the stat's definition, or None for a flag or milestone
npc_id: str | None # the character, for `npc.<id>.<stat>`, else None
npc_stats: dict | None # that character's stat defs, used to build a new block
def container(self, ws: dict) -> dict:
"""Returns the dict this path writes into, and creates it if it is missing.
Call this only when you are about to write. Creating the container is a
side effect, and a rejected path must not leave an empty section behind.
"""
if self.npc_id is None:
return ws.setdefault(self.section, {})
return ws.setdefault("npc", {}).setdefault(self.npc_id, _initials(self.npc_stats))
def _resolve(path: str, stat_schema: dict) -> tuple[_Target | None, dict | None]:
"""Routes one path to the value it names, without deciding what happens to it.
Returns `(target, None)` when the schema defines the path, and
`(None, rejection)` when it does not. The rejection is the entry that goes
into a report's `rejected` list, with its reason and its fix already worded.
`apply_delta` and `apply_override` route paths identically and differ only in
what they write, so the routing lives here and each function keeps its own
write rule.
"""
parts = path.split(".")
# `flags.<name>` is a boolean the scenario declares.
if parts[0] == "flags" and len(parts) == 2:
flag_defs = stat_schema.get("flags") or {}
if parts[1] not in flag_defs:
return None, {
"path": path, "reason": "unknown flag",
"fix": f"There is no flag `{parts[1]}`. {_names_phrase('flag', flag_defs)}",
}
return _Target("flag", "flags", parts[1], None, None, None), None
# `milestones.<id>` records that a milestone is reached.
if parts[0] == "milestones" and len(parts) == 2:
milestones = stat_schema.get("milestones") or {}
if parts[1] not in milestones:
return None, {
"path": path, "reason": "unknown milestone",
"fix": f"There is no milestone `{parts[1]}`. "
f"{_names_phrase('milestone', milestones)}",
}
return _Target("milestone", "milestones", parts[1], None, None, None), None
# `world.<stat>` and `player.<stat>`.
if parts[0] in STAT_SECTIONS and len(parts) == 2:
stat_defs = stat_schema.get(parts[0]) or {}
stat_def = stat_defs.get(parts[1])
if not isinstance(stat_def, dict):
return None, {
"path": path, "reason": "unknown stat",
"fix": f"`{parts[0]}` has no stat `{parts[1]}`. "
f"{_names_phrase('stat', stat_defs)}",
}
return _Target("stat", parts[0], parts[1], stat_def, None, None), None
# `npc.<id>.<stat>`. Each character carries its own stat definitions.
if parts[0] == "npc" and len(parts) == 3:
npcs = stat_schema.get("npcs") or {}
ndef = npcs.get(parts[1])
if not isinstance(ndef, dict):
return None, {
"path": path, "reason": "unknown npc",
"fix": f"There is no character `{parts[1]}`. "
f"{_names_phrase('character', npcs)}",
}
stat_defs = ndef.get("stats") or {}
stat_def = stat_defs.get(parts[2])
if not isinstance(stat_def, dict):
return None, {
"path": path, "reason": "unknown npc stat",
"fix": f"`{npc_name(ndef, parts[1])}` has no stat `{parts[2]}`. "
f"{_names_phrase('stat', stat_defs)}",
}
return _Target("stat", "npc", parts[2], stat_def, parts[1], stat_defs), None
return None, {
"path": path, "reason": "unknown path",
"fix": f"`{path}` is not a tracked value. Use player.<stat>, "
f"world.<stat>, npc.<id>.<stat>, flags.<name> or milestones.<id>.",
}
def _coerce_number(value):
if isinstance(value, bool): # `bool` is an `int` subclass, so reject it here.
return None
if isinstance(value, (int, float)):
return value
if isinstance(value, str):
try:
return float(value.strip())
except ValueError:
return None
return None
def _names_phrase(kind: str, defs: dict, limit: int = 12) -> str:
"""Lists what the model could have written instead, for a wrong name.
A rejection that only says a name is unknown leaves the model guessing
again. Naming the alternatives turns it into a correction it can act on.
"""
names = [k for k in (defs or {}) if isinstance(k, str)]
if not names:
return f"This scenario tracks no {kind}."
shown = ", ".join(f"`{n}`" for n in names[:limit])
more = f", and {len(names) - limit} more" if len(names) > limit else ""
plural = f"{kind}s" if not kind.endswith("s") else kind
return f"The {plural} are: {shown}{more}."
def _limits_phrase(stat_def: dict) -> str:
"""Names a stat's numeric limits, for a correction sent back to the model."""
lo, hi = stat_def.get("min"), stat_def.get("max")
cap = stat_def.get("max_delta_per_turn")
bits = []
if isinstance(lo, (int, float)) and isinstance(hi, (int, float)):
bits.append(f"it runs from {lo} to {hi}")
elif isinstance(lo, (int, float)):
bits.append(f"it never goes below {lo}")
elif isinstance(hi, (int, float)):
bits.append(f"it never goes above {hi}")
if isinstance(cap, (int, float)):
bits.append(f"it moves at most {cap} per turn")
return "; ".join(bits)
def _apply_stat(container: dict, key: str, stat_def: dict, change,
path: str, action_index: int, meta: dict, report: dict) -> None:
delta = _coerce_number(change)
if delta is None:
report["rejected"].append({
"path": path, "reason": "not a number",
"fix": f"`{path}` takes a number, written as a change such as -5 or 8.",
})
return
cooldown = stat_def.get("cooldown") or 0
last = meta["last_changed"].get(path)
if cooldown and last is not None and action_index - last < cooldown:
waited = action_index - last
report["rejected"].append({
"path": path, "reason": "cooldown",
"fix": f"`{path}` changed {waited} turn(s) ago and cannot change again "
f"until {cooldown} turns have passed.",
})
return
if stat_def.get("type") == "counter" and delta < 0:
report["rejected"].append({
"path": path, "reason": "counter can't decrease",
"fix": f"`{path}` only counts up. Send a positive change such as 1, "
f"never a negative and never the running total.",
})
return
clamped = False
cap = stat_def.get("max_delta_per_turn")
if cap is not None and abs(delta) > cap:
delta = cap if delta > 0 else -cap
clamped = True
old = container.get(key, stat_def.get("initial", 0))
new = old + delta
lo, hi = stat_def.get("min"), stat_def.get("max")
if lo is not None and new < lo:
new, clamped = lo, True
if hi is not None and new > hi:
new, clamped = hi, True
# Keep ints integral for display.
if isinstance(old, int) and float(new).is_integer():
new = int(new)
container[key] = new
meta["last_changed"][path] = action_index
entry = {"path": path, "old": old, "new": new}
if clamped and new == old:
# The clamp cancelled the change. Nothing moved, so the model needs the
# same correction a rejection gets: without it the only evidence is a
# value that stayed put, which reads as the change never being asked for.
edge = "maximum" if hi is not None and new == hi else "minimum"
entry["fix"] = (
f"`{path}` did not move. It is already at its {edge} of {new}"
+ (f" ({_limits_phrase(stat_def)})." if _limits_phrase(stat_def) else ".")
)
report["applied"].append(entry)
if clamped:
report["clamped"].append(entry)
def _apply_text_stat(container: dict, key: str, stat_def: dict, change,
path: str, action_index: int, meta: dict, report: dict) -> None:
"""Applies a free-text stat, which replaces rather than adds.
The AI sends the new value in full rather than a delta. No clamping and no
bands apply. Only an optional cooldown and an optional `max_length`
truncation apply.
"""
if not isinstance(change, str):
report["rejected"].append({
"path": path, "reason": "not a string",
"fix": f"`{path}` holds text. Send its new value in full, not a number "
f"and not a change.",
})
return
cooldown = stat_def.get("cooldown") or 0
last = meta["last_changed"].get(path)
if cooldown and last is not None and action_index - last < cooldown:
waited = action_index - last
report["rejected"].append({
"path": path, "reason": "cooldown",
"fix": f"`{path}` changed {waited} turn(s) ago and cannot change again "
f"until {cooldown} turns have passed.",
})
return
new = change.strip()
max_len = stat_def.get("max_length")
if isinstance(max_len, int) and max_len > 0 and len(new) > max_len:
new = new[:max_len]
old = container.get(key, stat_def.get("initial", ""))
if new == old:
return # Nothing changed, so do nothing.
container[key] = new
meta["last_changed"][path] = action_index
report["applied"].append({"path": path, "old": old, "new": new})
def apply_override(world_state: dict, stat_schema: dict, overrides: dict) -> tuple[dict, dict]:
"""Sets live values directly, as a manual author edit rather than an AI turn.
This differs from `apply_delta` in three ways. Numeric stats are set rather
than added to. `cooldown`, `max_delta_per_turn`, and the rule that a counter
cannot decrease are all ignored, because this is a deliberate correction
rather than an AI move to check. Milestones can be toggled in both
directions rather than only marked reached.
Values are still validated against the schema, so an unknown path or a wrong
type is rejected, and numeric values still clamp to min and max.
"""
ws = copy.deepcopy(world_state) if isinstance(world_state, dict) else {}
if not ws:
ws = instantiate(stat_schema)
report: dict = {"applied": [], "rejected": []}
if not isinstance(overrides, dict):
return ws, report
def set_stat(container: dict, key: str, stat_def: dict, value, path: str) -> None:
if stat_def.get("type") == "text":
if not isinstance(value, str):
report["rejected"].append({"path": path, "reason": "not a string"})
return
new = value.strip()
max_len = stat_def.get("max_length")
if isinstance(max_len, int) and max_len > 0 and len(new) > max_len:
new = new[:max_len]
old = container.get(key, stat_def.get("initial", ""))
container[key] = new
report["applied"].append({"path": path, "old": old, "new": new})
return
num = _coerce_number(value)
if num is None:
report["rejected"].append({"path": path, "reason": "not a number"})
return
old = container.get(key, stat_def.get("initial", 0))
lo, hi = stat_def.get("min"), stat_def.get("max")
if lo is not None and num < lo:
num = lo
if hi is not None and num > hi:
num = hi
if isinstance(old, int) and float(num).is_integer():
num = int(num)
container[key] = num
report["applied"].append({"path": path, "old": old, "new": num})
for raw_path, value in overrides.items():
path = str(raw_path)
target, rejection = _resolve(path, stat_schema)
if rejection is not None:
report["rejected"].append(rejection)
continue
if target.kind == "flag":
if not isinstance(value, bool):
report["rejected"].append({
"path": path, "reason": "not a boolean",
"fix": f"`{path}` takes true or false.",
})
continue
flags = target.container(ws)
old = bool(flags.get(target.key, False))
flags[target.key] = value
report["applied"].append({"path": path, "old": old, "new": value})
elif target.kind == "milestone":
# An override toggles a milestone in both directions, so unlike a
# delta it takes false as well as true.
if not isinstance(value, bool):
report["rejected"].append({
"path": path, "reason": "not a boolean",
"fix": f"`{path}` takes true or false.",
})
continue
reached = target.container(ws)
old = bool(reached.get(target.key, {}).get("reached"))
if value:
reached[target.key] = {"reached": True}
else:
reached.pop(target.key, None)
report["applied"].append({"path": path, "old": old, "new": value})
else:
set_stat(target.container(ws), target.key, target.stat_def, value, path)
return ws, report
def apply_delta(world_state: dict, stat_schema: dict, delta: dict,
action_index: int) -> tuple[dict, dict]:
"""Validates and clamps `delta` against `stat_schema`, then applies it.
The delta is applied to a copy of `world_state`. The return value is
`(new_world_state, report)`.
"""
ws = copy.deepcopy(world_state) if isinstance(world_state, dict) else {}
if not ws:
ws = instantiate(stat_schema)
ws.setdefault("_meta", {}).setdefault("last_changed", {})
meta = ws["_meta"]
report: dict = {"applied": [], "clamped": [], "rejected": []}
if not isinstance(delta, dict):
return ws, report
for raw_path, change in delta.items():
path = str(raw_path)
target, rejection = _resolve(path, stat_schema)
if rejection is not None:
report["rejected"].append(rejection)
continue
if target.kind == "flag":
# A flag goes both ways, so either value is accepted.
if not isinstance(change, bool):
report["rejected"].append({
"path": path, "reason": "not a boolean",
"fix": f"`{path}` takes true or false.",
})
continue
flags = target.container(ws)
old = bool(flags.get(target.key, False))
if change != old:
flags[target.key] = change
report["applied"].append({"path": path, "old": old, "new": change})
elif target.kind == "milestone":
# A milestone is sticky, so only true is accepted.
if change is not True:
report["rejected"].append({
"path": path, "reason": "not true",
"fix": f"`{path}` can only be set to true. A milestone is "
f"reached once and never taken back.",
})
continue
reached = target.container(ws)
if reached.get(target.key, {}).get("reached"):
continue # Already reached, so do nothing.
reached[target.key] = {"reached": True, "at": action_index}
report["applied"].append({"path": path, "old": False, "new": True})
elif target.stat_def.get("type") == "text":
_apply_text_stat(target.container(ws), target.key, target.stat_def, change,
path, action_index, meta, report)
else:
_apply_stat(target.container(ws), target.key, target.stat_def, change,
path, action_index, meta, report)
return ws, report
backend/app/worldstate/parse.py
+165
View File
@@ -0,0 +1,165 @@
"""The delta block: the rule the model is given, and reading back what it sent.
The model writes one fenced block per turn. This module holds the instruction
text, the parser that survives the ways a model gets the format wrong, and the
report rendering that tells a player which changes were refused.
"""
import json
import re
# Appended once to the system prompt, so the model knows how to report
# changes.
EMIT_RULE = (
"You maintain a numeric world state. Treat your own narration as authoritative: "
"whenever what you write implies a change to any tracked value — health or resources "
"going up or down, time passing, a relationship or mood shifting, a status turning on "
"or off, progress toward a goal, an item or piece of information gained or lost — you "
"MUST record it, including the numbers, not just on/off flags. After your narration, "
"append a fenced code block labelled `state` with a JSON object of the CHANGES ONLY, "
"as deltas (not new totals). Update every value the scene affected this turn, not only "
"the obvious ones. Read the range and band labels shown for each stat and keep every "
"change proportionate to the moment: an ordinary or minor event nudges a value slightly, "
"while a large change — or reaching a stat's minimum or maximum — is reserved for a "
"genuinely pivotal, defining moment (a passing remark shifts a relationship a little; a "
"lasting act of loyalty or betrayal shifts it a lot). Do not move a value across most of "
'its range in a single ordinary turn. Every stat is listed by its exact path in the '
'stat guide and again beside its live value — copy a path from there rather than '
'building one out of a name. The shapes are "player.<stat>", '
'"world.<stat>", "npc.<id>.<stat>" (use the id in parentheses, e.g. npc.gwen.trust, '
'not the display name); "flags.<name>": true or false to toggle an on/off state; and '
'"milestones.<id>": true when an objective is completed. Some stats marked (free text) in '
"the stat guide hold a short string instead of a number — for those, send the new value "
"in full (not a delta), e.g. what the player is now wearing or holding; only send it when "
"it actually changed. Send only things that actually "
"changed and never restate unchanged values; if truly nothing changed, omit the block. "
"Example:\n"
'```state\n{"player.hp": -15, "npc.gwen.trust": 5, "milestones.escaped": true, '
'"player.outfit": "torn traveling cloak"}\n```'
)
# A short reminder placed at the end of the prompt, which is the strongest
# recency position, so the emit rule is close to where the model generates.
EMIT_REMINDER = (
"[Reminder: end your reply with a ```state block of the changes this turn "
"(deltas only), or omit it if truly nothing changed.]"
)
def render_delta_block(delta: dict) -> str:
"""Renders a stored delta back into the fenced `state` block the AI emitted.
The caller replays past turns into the context with this, so the model copies
the format. An empty delta returns an empty string, which means the turn
changed nothing.
"""
if not isinstance(delta, dict) or not delta:
return ""
return "```state\n" + json.dumps(delta, ensure_ascii=False) + "\n```"
def applied_delta(world_delta: dict | None) -> dict:
"""Returns the changes the engine accepted, shaped as the AI sends them.
Replaying the delta the AI sent would show it a refused change standing as
though it had been applied, contradicted by the live values in the same
prompt. The model has no way to read that as a correction, so it repeats
the change. Replaying what was accepted removes the contradiction.
A numeric change that ended where it started is omitted, because it moved
nothing and a zero in the replayed block reads as a value worth sending.
"""
if not isinstance(world_delta, dict):
return {}
out: dict = {}
for entry in world_delta.get("applied") or []:
if not isinstance(entry, dict):
continue
path = str(entry.get("path", ""))
old, new = entry.get("old"), entry.get("new")
if path.startswith("milestones."):
out[path] = True
elif path.startswith("flags."):
out[path] = bool(new)
elif isinstance(old, (int, float)) and isinstance(new, (int, float)):
if new != old:
out[path] = new - old
else:
out[path] = new
return out
def refusals(world_delta: dict | None) -> list[str]:
"""Returns a correction line for each change the engine did not carry out.
Covers the changes that were lost: a rejection, and a clamp that left the
value where it started. A clamp that reduced a change but still moved the
value is deliberately absent. The model's intent landed in that case, and
reporting the shortfall invites it to send the remainder on the next turn,
which is the swing `max_delta_per_turn` exists to prevent.
"""
if not isinstance(world_delta, dict):
return []
lines = []
for entry in world_delta.get("rejected") or []:
if isinstance(entry, dict) and entry.get("fix"):
lines.append(str(entry["fix"]))
for entry in world_delta.get("clamped") or []:
if isinstance(entry, dict) and entry.get("fix"):
lines.append(str(entry["fix"]))
return lines
def render_refusals(world_delta: dict | None) -> str:
"""Renders `refusals` as the note appended after the most recent AI turn."""
lines = refusals(world_delta)
if not lines:
return ""
body = "\n".join(f"- {ln}" for ln in lines)
return ("[Part of your last state block was not applied. Correct it in this "
f"turn's block:\n{body}]")
# ```state { ... } ``` (also tolerates ```json or an unlabelled fence); DOTALL.
_FENCE_RE = re.compile(r"```(?:state|json)?\s*(\{.*?\})\s*```", re.DOTALL | re.IGNORECASE)
# Fallback: a bare JSON object hugging the end of the text.
_TRAILING_RE = re.compile(r"(\{[^{}]*\})\s*$", re.DOTALL)
def _tolerant_load(blob: str) -> dict:
# Strip trailing commas and leading + on numbers, both of which weaker
# free models emit and strict JSON rejects.
cleaned = re.sub(r",(\s*[}\]])", r"\1", blob)
cleaned = re.sub(r"(:\s*)\+(\d)", r"\1\2", cleaned)
try:
parsed = json.loads(cleaned)
except (json.JSONDecodeError, ValueError):
return {}
return parsed if isinstance(parsed, dict) else {}
def extract_delta(text: str) -> tuple[str, dict]:
"""Removes the trailing state block from an AI response.
The return value is `(clean_text, delta)`. `delta` is `{}` when there is no
block or the block cannot be parsed, and `clean_text` has the block removed.
A bare trailing object is stripped only when it parses to a delta, so
ordinary prose that ends in `}` is left alone.
"""
matches = list(_FENCE_RE.finditer(text))
if matches:
m = matches[-1]
delta = _tolerant_load(m.group(1))
clean = (text[: m.start()] + text[m.end():]).strip()
return clean, delta
m = _TRAILING_RE.search(text)
if m:
delta = _tolerant_load(m.group(1))
if delta and all("." in str(k) for k in delta):
clean = text[: m.start()].strip()
return clean, delta
return text.strip(), {}
backend/app/worldstate/render.py
+187
View File
@@ -0,0 +1,187 @@
"""Turning world state into the text the model and the player read.
`render_state_section` writes the current state into the prompt.
`render_reference` writes the schema itself, so the model knows which stats
exist and what they are called.
"""
from .schema import STAT_SECTIONS, _initials, band_label, npc_name
def _stat_line(defs: dict, values: dict) -> str:
parts = []
for name, d in defs.items():
if not isinstance(d, dict):
continue
if d.get("type") == "text":
val = values.get(name, d.get("initial", ""))
parts.append(f'{name} "{val}"' if val else f"{name} (unset)")
continue
val = values.get(name, d.get("initial", 0))
hi = d.get("max")
shown = f"{val}/{hi}" if hi is not None else f"{val}"
label = band_label(d, val)
parts.append(f"{name} {shown}" + (f" ({label})" if label else ""))
return ", ".join(parts)
def render_state_section(world_state: dict, stat_schema: dict,
visible_npcs: dict[str, str],
player_name: str = "") -> str:
"""Returns the compact context block that every turn includes.
`visible_npcs` maps a card id to a display name, for the NPCs currently in
the scene.
`player_name` is the persona's name (Phase 18), used to label the player's
stat block. It is empty for an adventure with no persona, and the block then
reads `You:` as it did before personas existed.
"""
ws = world_state if isinstance(world_state, dict) else {}
lines: list[str] = []
world_defs = stat_schema.get("world") or {}
world_line = _stat_line(world_defs, ws.get("world") or {})
header = "World state" + (f" — {world_line}." if world_line else ".")
lines.append(header)
player_defs = stat_schema.get("player") or {}
player_line = _stat_line(player_defs, ws.get("player") or {})
if player_line:
# Name the block after the persona, and show the path beside it exactly
# as the NPC lines below do. The model reads the name in the narration,
# so without the path in view it writes `kaelen.hp` and the delta is
# refused as an unknown path.
label = f"{player_name} (player)" if player_name else "You"
lines.append(f"{label}: {player_line}.")
npcs = stat_schema.get("npcs") or {}
npc_state = ws.get("npc") or {}
for npc_key, name in visible_npcs.items():
ndef = npcs.get(npc_key) or {}
stat_defs = ndef.get("stats") or {}
values = npc_state.get(npc_key) or _initials(stat_defs)
npc_line = _stat_line(stat_defs, values)
if npc_line:
# Show the id so the AI can address it as npc.<id>.<stat>.
lines.append(f"{name} (npc.{npc_key}): {npc_line}.")
flag_defs = stat_schema.get("flags") or {}
flag_state = ws.get("flags") or {}
flag_parts = [
f"{name} {'yes' if flag_state.get(name, bool(d.get('initial', False))) else 'no'}"
for name, d in flag_defs.items() if isinstance(d, dict)
]
if flag_parts:
lines.append("Flags: " + ", ".join(flag_parts) + ".")
# Show the id beside each goal, the same as NPCs and flags. The AI marks a
# milestone as `milestones.<id>`, and `apply_delta` rejects an id the schema
# does not define, so a goal listed by description alone gives the model no
# way to name it and it can only guess.
milestones = stat_schema.get("milestones") or {}
reached = ws.get("milestones") or {}
goals = [f"{mid} — {d.get('desc', mid)}" for mid, d in milestones.items()
if not reached.get(mid, {}).get("reached")]
done = [f"{mid} — {d.get('desc', mid)}" for mid, d in milestones.items()
if reached.get(mid, {}).get("reached")]
if goals:
lines.append("Goals (mark with milestones.<id>): " + "; ".join(goals) + ".")
if done:
lines.append("Achieved: " + "; ".join(done) + ".")
return "\n".join(lines)
def _describe_stat(name: str, d: dict) -> str | None:
"""Returns one reference line for a stat, named by the path the AI writes.
`name` is that path. The guide is the model's only complete list of what
exists, so a stat named any other way leaves it to work the path out from
the live values, and those cover only what is on screen this turn.
The description and the band ladder are independent, and each is included
only when present, so a stat may have either, both, or neither.
"""
is_text = d.get("type") == "text"
# `EMIT_RULE` sends the model here to find out which stats take a whole
# value instead of a change, and it names this marker, so the two have to
# be written the same way.
label = f"{name} (free text)" if is_text else name
bits: list[str] = []
desc = d.get("desc")
if isinstance(desc, str) and desc.strip():
# Fragments are joined with "; " and end with a single ".", so remove
# any trailing period the author put on the description.
bits.append(desc.strip().rstrip("."))
if is_text:
# Free text has no range and no bands, so the marker is the whole
# entry. It still earns a line without a description, because the
# marker is what stops the model sending a delta.
return f"{label} — {'; '.join(bits)}." if bits else f"{label}."
lo, hi = d.get("min"), d.get("max")
if isinstance(lo, (int, float)) and isinstance(hi, (int, float)):
bits.append(f"range {lo}–{hi}")
bands = d.get("bands")
if isinstance(bands, list) and bands:
ladder = ", ".join(
f"{b[0]}–{b[1]} {b[2]}"
for b in bands if isinstance(b, list) and len(b) == 3
)
if ladder:
bits.append(f"bands: {ladder}")
return f"{label} — {'; '.join(bits)}." if bits else None
def render_reference(stat_schema: dict, player_name: str = "") -> str:
"""Returns a fixed, per-scenario legend for the stats.
Each line gives what a stat means, from its `desc`, and its band ladder. The
legend does not change from turn to turn, and it is separate from the live
values.
`player_name` names the persona, so that the guide ties the name the model
reads in the story to the `player.` paths it has to write. The persona's
description is deliberately not repeated here: it has its own section, and
this guide is about paths.
"""
lines: list[str] = []
if player_name:
lines.append(
f"Protagonist {player_name} — the player character; their stats are "
f"addressed as player.<stat>."
)
for section in STAT_SECTIONS:
for name, d in (stat_schema.get(section) or {}).items():
if isinstance(d, dict):
row = _describe_stat(f"{section}.{name}", d)
if row:
lines.append(row)
for npc_key, ndef in (stat_schema.get("npcs") or {}).items():
if not isinstance(ndef, dict):
continue
name = npc_name(ndef, npc_key)
desc = ndef.get("desc")
# The header is written even for an NPC with no description, because it
# is the only line that ties a display name to the id the AI has to
# address. The live values state it too, but only for the NPCs a scene
# has mentioned, so without this an NPC off screen can only be guessed
# at — and a guess is refused as a character or a stat that does not
# exist.
head = f"NPC {name} (npc.{npc_key})"
lines.append(
f"{head} — {desc.strip().rstrip('.')}."
if isinstance(desc, str) and desc.strip() else f"{head}."
)
for sname, sdef in (ndef.get("stats") or {}).items():
if isinstance(sdef, dict):
row = _describe_stat(f"npc.{npc_key}.{sname}", sdef)
if row:
lines.append(row)
for name, d in (stat_schema.get("flags") or {}).items():
if isinstance(d, dict):
desc = d.get("desc")
if isinstance(desc, str) and desc.strip():
lines.append(f"flags.{name} — {desc.strip().rstrip('.')}.")
if not lines:
return ""
return "Stat guide (fixed reference):\n" + "\n".join(f"- {ln}" for ln in lines)
backend/app/worldstate/schema.py
+198
View File
@@ -0,0 +1,198 @@
"""Reading a scenario's `stat_schema` and building state that matches it.
Nothing here decides what a turn changes. These functions answer what the schema
allows, what a fresh state looks like, and how an edited schema maps onto state
that already exists.
"""
import copy
# The `stat_schema` top-level sections that hold stat definitions.
STAT_SECTIONS = ("world", "player")
def has_schema(stat_schema: dict | None) -> bool:
"""Returns `True` when a scenario defines an RPG layer."""
if not isinstance(stat_schema, dict):
return False
return any(
isinstance(stat_schema.get(k), dict) and stat_schema[k]
for k in (*STAT_SECTIONS, "npcs", "milestones", "flags")
)
def npc_name(ndef: dict, key: str) -> str:
name = ndef.get("name")
return name.strip() if isinstance(name, str) and name.strip() else key
def npc_triggers(ndef: dict, key: str) -> list[str]:
"""Returns the lowercased trigger words that detect an NPC in a scene.
The words come from the NPC's `keys` field, or from its display name when
`keys` is empty.
"""
raw = ndef.get("keys") or npc_name(ndef, key)
return [k.strip().lower() for k in str(raw).split(",") if k.strip()]
def _initials(defs: dict) -> dict:
return {
name: d.get("initial", "" if d.get("type") == "text" else 0)
for name, d in defs.items()
if isinstance(d, dict)
}
def instantiate(stat_schema: dict | None) -> dict:
"""Builds a fresh live `world_state` from a schema, using initial values only."""
if not has_schema(stat_schema):
return {}
ws: dict = {}
for section in STAT_SECTIONS:
ws[section] = _initials(stat_schema.get(section) or {})
# Each defined NPC gets its own stat block from its own `stats` defs.
ws["npc"] = {
key: _initials(ndef.get("stats") or {})
for key, ndef in (stat_schema.get("npcs") or {}).items()
if isinstance(ndef, dict)
}
ws["milestones"] = {} # only reached ones are stored
ws["flags"] = {
name: bool(d.get("initial", False))
for name, d in (stat_schema.get("flags") or {}).items()
if isinstance(d, dict)
}
ws["_meta"] = {"last_changed": {}}
return ws
def reconcile(world_state: dict | None, stat_schema: dict | None) -> tuple[dict, dict]:
"""Brings a live `world_state` back in line with an edited schema.
This is not `instantiate`. A value the schema still defines keeps whatever it
reached in play, because re-instantiating would restore the player to full
health and clear their milestones. Only the difference is applied. Stats,
NPCs, flags, and milestones the schema gained appear at their initial value,
and ones it no longer defines are removed along with their `last_changed`
bookkeeping. The return value is `(new_state, report)`, where the report
lists paths under `added` and `removed`, so the UI can show what a refresh
would do.
The additions are mostly cosmetic. Rendering and delta application both fall
back to a stat definition's `initial` when the live state has no value for
it, so a newly added stat already behaves correctly. This function stores the
value, and unlike those read-through paths it also removes what the schema
dropped.
"""
report: dict = {"added": [], "removed": []}
if not has_schema(stat_schema):
# The scenario dropped its RPG layer, so the adventure drops it too.
stale = bool(world_state)
if stale:
report["removed"].append("(all world state)")
return {}, report
ws = copy.deepcopy(world_state) if isinstance(world_state, dict) else {}
if not ws:
return instantiate(stat_schema), report
def sync_section(container: dict, defs: dict, prefix: str) -> dict:
out = {}
for name, d in defs.items():
if not isinstance(d, dict):
continue
if name in container:
out[name] = container[name]
else:
out[name] = d.get("initial", "" if d.get("type") == "text" else 0)
report["added"].append(f"{prefix}.{name}")
for name in container:
if name not in out:
report["removed"].append(f"{prefix}.{name}")
return out
for section in STAT_SECTIONS:
ws[section] = sync_section(
ws.get(section) if isinstance(ws.get(section), dict) else {},
stat_schema.get(section) or {},
section,
)
npc_defs = stat_schema.get("npcs") or {}
old_npcs = ws.get("npc") if isinstance(ws.get("npc"), dict) else {}
new_npcs = {}
for key, ndef in npc_defs.items():
if not isinstance(ndef, dict):
continue
old = old_npcs.get(key) if isinstance(old_npcs.get(key), dict) else {}
new_npcs[key] = sync_section(old, ndef.get("stats") or {}, f"npc.{key}")
for key in old_npcs:
if key not in new_npcs:
report["removed"].append(f"npc.{key}")
ws["npc"] = new_npcs
flag_defs = stat_schema.get("flags") or {}
old_flags = ws.get("flags") if isinstance(ws.get("flags"), dict) else {}
new_flags = {}
for name, d in flag_defs.items():
if not isinstance(d, dict):
continue
if name in old_flags:
new_flags[name] = bool(old_flags[name])
else:
new_flags[name] = bool(d.get("initial", False))
report["added"].append(f"flags.{name}")
for name in old_flags:
if name not in new_flags:
report["removed"].append(f"flags.{name}")
ws["flags"] = new_flags
# Milestones store only the ones reached, so there is nothing to add here.
# An unreached milestone is absent. Drop reached milestones the scenario no
# longer defines, because they would otherwise stay in "Achieved" with no
# label.
milestone_defs = stat_schema.get("milestones") or {}
reached = ws.get("milestones") if isinstance(ws.get("milestones"), dict) else {}
ws["milestones"] = {k: v for k, v in reached.items() if k in milestone_defs}
for k in reached:
if k not in milestone_defs:
report["removed"].append(f"milestones.{k}")
# Cooldown bookkeeping for paths that no longer exist is never read, but it
# accumulates in every stored snapshot, so remove it with the rest.
meta = ws.setdefault("_meta", {})
last_changed = meta.get("last_changed")
if isinstance(last_changed, dict):
removed = set(report["removed"])
meta["last_changed"] = {
path: at for path, at in last_changed.items()
if not any(path == r or path.startswith(f"{r}.") for r in removed)
}
else:
meta["last_changed"] = {}
return ws, report
def band_label(stat_def: dict, value) -> str | None:
"""Returns the word label for `value` from a stat definition's bands, if any.
A band is `[lo, hi, label]` and matches when `lo <= value < hi`. The top band
includes its upper bound, so a stat at its maximum still gets a label.
"""
bands = stat_def.get("bands")
if not isinstance(bands, list) or not isinstance(value, (int, float)):
return None
last_hi = None
for band in bands:
if not (isinstance(band, list) and len(band) == 3):
continue
lo, hi, label = band
last_hi = hi
if lo <= value < hi:
return str(label)
# Inclusive top edge.
if bands and value == last_hi:
return str(bands[-1][2])
return None
backend/requirements-dev.txt
+4
View File
@@ -0,0 +1,4 @@
# Dev/test-only dependencies (not installed on the Render deploy).
# pip install -r requirements.txt -r requirements-dev.txt
# python -m pytest tests/
pytest>=8.0
backend/requirements.txt
+9
View File
@@ -0,0 +1,9 @@
fastapi>=0.115
uvicorn[standard]>=0.30
sqlalchemy>=2.0
pydantic>=2.7
httpx>=0.27
tiktoken>=0.7
quickjs>=1.19
cryptography>=42
psycopg[binary]>=3.2
backend/seed_demo.py
+267
View File
@@ -0,0 +1,267 @@
"""Seed a demo scenario + adventure with sample scripts for testing.
Run from the backend folder: .venv\\Scripts\\python.exe seed_demo.py
Safe to rerun: it deletes any previous rows titled "[Demo] ..." first.
Phase 8: the scenario is seeded as PUBLIC (user_id NULL + is_public), so in
multi-user mode every guest sees it as read-only starter content. The sample
adventure and script-library copies belong to the local user (only relevant
on single-user installs).
"""
from app import auth, models, migrations, tree
from app.database import SessionLocal, engine
# create_all + user_version stamp; plain create_all would leave a fresh DB at
# version 0 and the server would replay every ALTER TABLE migration on start.
migrations.bootstrap(engine)
DEMO_PREFIX = "[Demo]"
# ---------------------------------------------------------------------------
# Sample scripts. The AI Dungeon contract is to define `modifier(text)` and call
# it last.
# ---------------------------------------------------------------------------
DICE_ROLLER = dict(
name=f"{DEMO_PREFIX} Dice Roller",
description=(
"Input hook + shared library. Type '!roll 2d6' or '!roll d20' in a Do/Say/"
"Story action and the command is replaced with the rolled result."
),
library_js="""\
// Shared library: available to every hook of this script.
function rollDice(count, sides) {
var total = 0, rolls = [];
for (var i = 0; i < count; i++) {
var r = Math.floor(Math.random() * sides) + 1;
rolls.push(r);
total += r;
}
return { total: total, rolls: rolls };
}
""",
input_js="""\
const modifier = (text) => {
// Replace every "!roll NdS" (N optional) with the roll result.
var out = text.replace(/!roll\\s+(\\d*)d(\\d+)/gi, function (m, n, s) {
var count = parseInt(n || "1", 10);
var sides = parseInt(s, 10);
var res = rollDice(count, sides);
log("Rolled " + count + "d" + sides + ": [" + res.rolls.join(", ") + "] = " + res.total);
return "(rolled " + count + "d" + sides + ": " + res.total + ")";
});
return { text: out };
};
modifier(text);
""",
)
TURN_TRACKER = dict(
name=f"{DEMO_PREFIX} Turn & HP Tracker",
description=(
"Demonstrates persistent state. Counts turns; '!hp -3' or '!hp +5' in input "
"adjusts HP (starts at 20). Current stats appear in state.message."
),
input_js="""\
const modifier = (text) => {
if (state.hp === undefined) state.hp = 20;
state.turns = (state.turns || 0) + 1;
var out = text.replace(/!hp\\s*([+-]\\d+)/gi, function (m, delta) {
state.hp += parseInt(delta, 10);
return "";
});
state.message = "Turn " + state.turns + " | HP: " + state.hp + "/20";
log(state.message);
if (state.hp <= 0) {
// stop:true ends the turn before the AI is called.
return { text: out + "\\n\\nYou have fallen. (HP reached 0 — turn stopped by script.)", stop: true };
}
return { text: out };
};
modifier(text);
""",
)
CONTEXT_INSPECTOR = dict(
name=f"{DEMO_PREFIX} Context Inspector",
description=(
"Context hook: logs the size of the assembled context each turn and appends "
"a style directive. Check the logs/context in the action's context snapshot."
),
context_js="""\
const modifier = (text) => {
log("Context size: " + text.length + " chars, actions so far: " + info.actionCount
+ ", story cards: " + storyCards.length);
// Anything returned here replaces what is sent to the model.
return { text: text + "\\n[Style: keep the response under three paragraphs.]" };
};
modifier(text);
""",
)
OUTPUT_POLISH = dict(
name=f"{DEMO_PREFIX} Output Polish + Card Discovery",
description=(
"Output hook: trims a trailing incomplete sentence from the AI reply, and "
"auto-creates a story card the first time the ghost Vharos is mentioned."
),
output_js="""\
const modifier = (text) => {
var out = text;
// Drop a trailing sentence fragment (no ending punctuation).
var m = out.match(/^([\\s\\S]*[.!?"'\\u2026])[^.!?"'\\u2026]*$/);
if (m && m[1].length > 40) {
if (m[1].length < out.length) log("Trimmed incomplete final sentence.");
out = m[1];
}
// Demonstrate script-created story cards.
if (/vharos/i.test(out)) {
var added = addStoryCard(
"Vharos, ghost, spirit",
"Vharos was the crypt's architect, now a restless ghost bound to the amulet he was buried with. He speaks in echoes and cannot lie.",
"character"
);
if (added !== false) log("Vharos mentioned — story card created.");
}
return { text: out };
};
modifier(text);
""",
)
SCRIPTS = [DICE_ROLLER, TURN_TRACKER, CONTEXT_INSPECTOR, OUTPUT_POLISH]
# ---------------------------------------------------------------------------
# Scenario content
# ---------------------------------------------------------------------------
SCENARIO = dict(
title=f"{DEMO_PREFIX} The Sunken Crypt of Vharos",
description=(
"A short dungeon-crawl demo scenario with story cards and one of each "
"script hook, for testing the app end to end."
),
prompt=(
"Rain hammers the moors as you descend the moss-slick steps beneath the "
"ruined chapel. Your torch gutters in the stale air. Below, the Sunken "
"Crypt of Vharos waits — its iron door ajar, as if someone (or something) "
"expected you. Mira's warning rings in your ears: bring back the Ember "
"Amulet before nightfall, or the village of Hollowmere burns.\n\n"
"You stand before the iron door, water pooling around your boots."
),
memory=(
"The player is an adventurer hired by Mira, blacksmith of Hollowmere, to "
"retrieve the Ember Amulet from the Sunken Crypt of Vharos before "
"nightfall. The crypt is flooded, dark, and haunted. Tone: classic D&D "
"dungeon crawl, dangerous but fair."
),
authors_note="Keep scenes tense and grounded; offer clear choices; consequences matter.",
ai_instructions=(
"Write in second person, present tense. End each response at a moment "
"where the player can act."
),
tags="demo, dungeon, fantasy, short",
)
STORY_CARDS = [
dict(
type="character",
name="Mira the Blacksmith",
keys="Mira, blacksmith",
entry=(
"Mira is Hollowmere's blacksmith: broad-shouldered, gray-braided, "
"practical. She hired the player and paid half up front. She knows "
"more about the crypt than she has admitted."
),
notes="Secretly a descendant of Vharos.",
),
dict(
type="location",
name="The Sunken Crypt",
keys="crypt, tomb, Vharos",
entry=(
"A flooded burial complex beneath a ruined chapel. Knee-deep black "
"water, collapsed pillars, and phosphorescent moss. Three chambers: "
"the Drowned Hall, the Ossuary, and the sealed Reliquary where the "
"Ember Amulet rests."
),
notes="",
),
dict(
type="item",
name="The Ember Amulet",
keys="amulet, ember",
entry=(
"A fist-sized garnet on a bronze chain that glows like a coal. It "
"keeps Hollowmere's protective hearth-ward burning. Touching it bare-"
"handed brands the flesh but does no lasting harm."
),
notes="",
),
]
# ---------------------------------------------------------------------------
db = SessionLocal()
try:
owner = auth.local_user(db)
# Remove earlier demo rows so reruns stay clean.
for adv in db.query(models.Adventure).filter(models.Adventure.title.like(f"{DEMO_PREFIX}%")):
db.delete(adv)
for sc in db.query(models.Scenario).filter(models.Scenario.title.like(f"{DEMO_PREFIX}%")):
db.delete(sc)
for s in db.query(models.Script).filter(models.Script.name.like(f"{DEMO_PREFIX}%")):
db.delete(s)
db.commit()
# Scripts attached to the public scenario are unowned (user_id NULL) so
# they ship with it everywhere; they're copied into each adventure at
# creation, so they never need to appear in anyone's script library.
scripts = [models.Script(**s) for s in SCRIPTS]
db.add_all(scripts)
# A scenario with cards and scripts attached, as public starter content.
scenario = models.Scenario(**SCENARIO, is_public=True)
scenario.scripts = scripts
db.add(scenario)
db.flush()
for card in STORY_CARDS:
db.add(models.StoryCard(scenario_id=scenario.id, **card))
# Adventure created from the scenario, mirroring POST /api/adventures
adventure = models.Adventure(
user_id=owner.id,
scenario_id=scenario.id,
title=scenario.title,
memory=scenario.memory,
authors_note=scenario.authors_note,
ai_instructions=scenario.ai_instructions,
)
db.add(adventure)
db.flush()
for card in STORY_CARDS:
db.add(models.StoryCard(adventure_id=adventure.id, **card))
for position, s in enumerate(SCRIPTS):
db.add(models.AdventureScript(adventure_id=adventure.id, position=position, **s))
opening = models.Action(
adventure_id=adventure.id, type="start", text=scenario.prompt
)
# Through the same door create_adventure uses, so the seeded adventure has a
# story tree like every other one.
tree.place_action(db, adventure, opening)
db.add(opening)
db.commit()
print(f"Scenario id={scenario.id}: {scenario.title}")
print(f"Adventure id={adventure.id}: {adventure.title}")
print(f"Scripts: {', '.join(s.name for s in scripts)}")
finally:
db.close()
backend/tests/conftest.py
+57
View File
@@ -0,0 +1,57 @@
"""Shared setup for the test suite.
pytest imports this file before it imports any test module, which is the only
reason the database redirection below works. `app.database` reads
`AIDND_DB_PATH` at import and builds `engine` from it once, so the variable has
to be set before the first `from app...` line anywhere in the suite.
Every test module used to carry its own copy of that redirection. Only the first
one to be imported ever took effect, because the engine already existed by the
time the second one ran. The other copies created a temp file that nothing
opened and nothing deleted. One copy here does the job, and it cleans up after
itself.
The tests share one database. That is not new: they already did. Each `client`
fixture calls `Base.metadata.create_all` on setup and `drop_all` on teardown, so
no test sees another test's rows.
"""
import os
import tempfile
_tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
_tmp.close()
os.environ["AIDND_DB_PATH"] = _tmp.name
# A real `AIDND_DATABASE_URL` or `DATABASE_URL` in the developer's shell points
# at Postgres, and `app.database` prefers either over the SQLite path above.
# Clear both, so running the suite never touches a server database.
os.environ.pop("AIDND_DATABASE_URL", None)
os.environ.pop("DATABASE_URL", None)
import pytest # noqa: E402 Import order is load-bearing; see above.
from fakes import ScriptedProvider # noqa: E402
@pytest.fixture(autouse=True)
def reset_scripted_provider():
"""Clears the fake provider's state between tests.
`ScriptedProvider` keeps its replies and its call count on the class, because
the code under test constructs the provider itself and a test cannot reach
the instance. Class state outlives a test, so reset it here rather than
trusting every fixture to remember.
"""
ScriptedProvider.replies = []
ScriptedProvider.calls = 0
ScriptedProvider.prompts = []
yield
def pytest_sessionfinish(session, exitstatus):
"""Deletes the temporary database once the run ends."""
try:
os.unlink(_tmp.name)
except OSError:
# The file is already gone, or Windows still holds a handle on it. It is
# in the temp directory either way, so leaving it costs nothing.
pass
backend/tests/fakes.py
+41
View File
@@ -0,0 +1,41 @@
"""Stand-ins for the parts of the app a test must not really call.
Import these rather than writing another copy. Nine test modules each carried
their own `ScriptedProvider`, and the copies had drifted into four different
feature sets, so a test that needed to raise a provider error had to be written
in one of the files whose copy supported it.
"""
class ScriptedProvider:
"""Streams canned replies in place of `OpenAICompatibleProvider`.
Set `replies` to the texts the model returns, one per call. The last entry
repeats once the list runs out, so a test that plays more turns than it
scripted still gets text. To drive the provider-error path, put an
`Exception` in the list. It is raised rather than streamed.
State lives on the class, not on the instance, because the turn engine
constructs the provider itself and a test never sees the object. The autouse
`reset_scripted_provider` fixture in `conftest.py` clears it between tests.
`prompts` records every assembled `(system, story)` pair, which is what a
test asserts on to check what the model was shown.
"""
last_usage = None
replies: list = []
calls = 0
prompts: list = []
def __init__(self, *a, **k):
pass
async def generate(self, parts, *, temperature, max_tokens):
index = min(ScriptedProvider.calls, len(ScriptedProvider.replies) - 1)
ScriptedProvider.calls += 1
ScriptedProvider.prompts.append((parts.system, parts.story))
reply = ScriptedProvider.replies[index]
if isinstance(reply, Exception):
raise reply
yield ("text", reply)
backend/tests/schema_rewind.py
+47
View File
@@ -0,0 +1,47 @@
"""Make a database look like an older schema version, so a migration can run.
`create_all` always builds the current schema. A test that wants to watch a
migration run must remove the newer columns first and then stamp an older
version. Otherwise the migration finds a column that already exists and
fails on a duplicate.
Rewinding the stamp alone worked for a while, so two test files did exactly
that. It stopped working when another `ADD COLUMN` migration landed. Without
this rewind, the replay runs migrations the tests never intended to
exercise, against columns `create_all` already added. This module rewinds
properly in one place. Adding a migration now means adding its inverse here,
instead of tracking down failures in three unrelated test files.
This module supports SQLite only. Every test that replays migrations runs on
a temp file, and SQLite stores the stamp in `PRAGMA user_version`.
Migrations that change a column's type (43-45, JSON to compressed bytes)
have no clean inverse, so this list omits them. Those migrations replay
as-is, which is what the tests using them already expect.
"""
from sqlalchemy import text
from sqlalchemy.engine import Engine
# Each entry is (version that added the column, statements that remove it).
# The list is ordered newest first.
#
# Phase 14's `branch_id` columns are missing on purpose. SQLite refuses to
# drop a column that a foreign key references, so a current-schema database
# cannot be rewound past them. `migrations._column_already_there` handles
# this case. It skips DDL that already ran, so the tree migrations run their
# backfill against a schema that already has the columns.
_UNDO: list[tuple[int, tuple[str, ...]]] = [
# Packed float32 vectors and the flag beside them.
(39, ("ALTER TABLE memories DROP COLUMN embedded",)),
(38, ("ALTER TABLE memories DROP COLUMN embedding_blob",)),
]
def rewind_to(engine: Engine, version: int) -> None:
"""Drop everything added after `version`, then stamp the database at it."""
with engine.begin() as conn:
for added_at, statements in _UNDO:
if added_at > version:
for sql in statements:
conn.execute(text(sql))
conn.execute(text(f"PRAGMA user_version = {version}"))
backend/tests/test_accesslog.py
+208
View File
@@ -0,0 +1,208 @@
"""The access log: app/accesslog.py and GET /api/analytics/access.
This is the half of the analytics work that identifies people on purpose,
so these tests pin the details that would quietly make it wrong. The
address recorded must be the hardened one, not a header a client chose.
Session rows must be thinned instead of written on every page load. And a
row must outlive the account it describes, because guest cleanup deletes
accounts on a schedule, and a log that deletes itself is not a log.
python -m pytest tests/test_accesslog.py -v
"""
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from app import accesslog, auth, limits, models, security
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
EDGE = "198.51.100.77" # what the trusted proxy appended
SPOOF = "10.0.0.1" # what a client put in front of it
@pytest.fixture(autouse=True)
def clean_state():
accesslog._last_session.clear()
yield
accesslog._last_session.clear()
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
owner = models.User(is_guest=False, email="owner@example.com")
member = models.User(
is_guest=False, email="player@example.com",
password_hash=security.hash_password("hunter2long"),
)
setup.add_all([owner, member])
setup.commit()
ids = {"owner": owner.id, "member": member.id}
setup.close()
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
monkeypatch.setattr(limits, "check_login_allowed", lambda *a, **k: None)
monkeypatch.setattr(auth, "MULTI_USER", True)
monkeypatch.setattr(auth, "ANALYTICS_EMAILS", {"owner@example.com"})
# /auth/me resolves its own session, so the cookie flow below is the real
# one. Every other endpoint goes through get_current_user, and `act_as`
# decides who that is.
acting = {"id": ids["owner"]}
def _current(db=Depends(get_db)):
return db.get(models.User, acting["id"])
app.dependency_overrides[auth.get_current_user] = _current
try:
test_client = TestClient(app)
test_client.ids = ids
test_client.act_as = lambda user_id: acting.update(id=user_id)
yield test_client
finally:
app.dependency_overrides.clear()
Base.metadata.drop_all(bind=engine)
def visit(client, ip=EDGE, ua="Mozilla/5.0 (Windows NT 10.0; Win64; x64)"):
return client.get(
"/api/auth/me",
headers={"x-forwarded-for": f"{SPOOF}, {ip}", "user-agent": ua},
)
def rows(kind=None):
db = SessionLocal()
try:
query = db.query(models.AccessEvent).order_by(models.AccessEvent.id)
if kind:
query = query.filter_by(kind=kind)
return query.all()
finally:
db.close()
def read_log(client, **params):
return client.get("/api/analytics/access", params=params)
# ---------- Writing ----------
def test_a_new_session_is_logged(client):
visit(client)
logged = rows()
assert len(logged) == 1
entry = logged[0]
assert entry.kind == accesslog.SESSION
assert entry.is_guest and entry.who.startswith("Guest #")
assert entry.device == "desktop"
def test_the_address_is_the_hardened_one_not_the_clients(client):
visit(client)
# The client prepended its own value. Only the hop the edge appended counts.
# Recording the leftmost value would make every row forgeable, which is
# worse for a log than having no log at all.
assert rows()[0].ip == EDGE
def test_session_rows_are_thinned_to_one_per_day_per_address(client):
for _ in range(4):
visit(client)
assert len(rows(accesslog.SESSION)) == 1
def test_a_changed_address_writes_a_new_row(client):
visit(client)
visit(client, ip="203.0.113.9")
logged = rows(accesslog.SESSION)
assert [entry.ip for entry in logged] == [EDGE, "203.0.113.9"]
# Same session throughout, so both rows name the same visitor.
assert logged[0].who == logged[1].who
def test_sign_in_and_failure_are_both_logged(client):
client.post("/api/auth/login", json={"email": "player@example.com", "password": "wrong"},
headers={"x-forwarded-for": EDGE})
client.post("/api/auth/login", json={"email": "player@example.com", "password": "hunter2long"},
headers={"x-forwarded-for": EDGE})
kinds = [entry.kind for entry in rows()]
assert accesslog.LOGIN_FAILED in kinds and accesslog.LOGIN in kinds
failure = rows(accesslog.LOGIN_FAILED)[0]
# This records the address that was tried, not the account it belongs to.
# A failed attempt against an address with no matching account is
# exactly what this row exists to capture.
assert failure.who == "player@example.com"
assert failure.user_id is None
assert rows(accesslog.LOGIN)[0].user_id == client.ids["member"]
def test_registering_is_logged_against_the_upgraded_account(client):
visit(client) # creates the guest whose session then registers
client.act_as(rows()[0].user_id)
client.post("/api/auth/register", json={"email": "new@example.com", "password": "hunter2long"})
entry = rows(accesslog.REGISTER)[0]
assert entry.who == "new@example.com" and not entry.is_guest
def test_a_row_outlives_the_account_it_describes(client):
visit(client)
entry = rows()[0]
db = SessionLocal()
try:
db.delete(db.get(models.User, entry.user_id))
db.commit()
finally:
db.close()
# There is no foreign key, and `who` is a snapshot. Guest cleanup deletes
# accounts on a schedule, and a log that vanishes along with them is not
# a log.
survivor = rows()[0]
assert survivor.who == entry.who and survivor.ip == EDGE
def test_a_long_user_agent_is_truncated(client):
visit(client, ua="Mozilla/" + "x" * 500)
assert len(rows()[0].user_agent) == accesslog.MAX_UA
def test_a_logging_failure_does_not_break_the_request(client, monkeypatch):
monkeypatch.setattr(accesslog, "_client_ip", lambda request: 1 / 0)
# The log observes sign-in. A logging failure must not block the request.
assert visit(client).status_code == 200
# ---------- Reading ----------
def test_the_log_is_invisible_to_everyone_but_the_owner(client):
visit(client)
assert read_log(client).status_code == 200
client.act_as(client.ids["member"])
assert read_log(client).status_code == 404
def test_the_log_reads_newest_first_and_pages_backwards(client):
for index in range(5):
visit(client, ip=f"203.0.113.{index}")
first = read_log(client, limit=2).json()
assert [event["ip"] for event in first["events"]] == ["203.0.113.4", "203.0.113.3"]
assert first["has_more"]
older = read_log(client, limit=2, before_id=first["events"][-1]["id"]).json()
assert [event["ip"] for event in older["events"]] == ["203.0.113.2", "203.0.113.1"]
def test_the_log_filters_by_kind_and_searches(client):
visit(client)
client.post("/api/auth/login", json={"email": "player@example.com", "password": "hunter2long"},
headers={"x-forwarded-for": "203.0.113.44"})
assert len(read_log(client, kind="login").json()["events"]) == 1
by_email = read_log(client, q="player@example.com").json()["events"]
assert len(by_email) == 1 and by_email[0]["kind"] == "login"
by_ip = read_log(client, q="203.0.113.44").json()["events"]
assert len(by_ip) == 1
assert read_log(client, q="nobody@example.com").json()["events"] == []
backend/tests/test_action_paging.py
+246
View File
@@ -0,0 +1,246 @@
"""Opening an adventure fetches a window, not the whole story.
A story only ever gets longer. Production's longest is 607 actions and
589.5 kB in one response, and that number never decreases on its own. The
page load returns the newest `ACTION_PAGE` window, and the reader pages
upward from there.
The paging anchors on an action id rather than an offset, and these tests
cover why. An offset counted back from the newest shifts every older
position the moment a turn lands, which is exactly when a reader is likely
to be scrolling. An anchor means the same thing before and after.
python -m pytest tests/test_action_paging.py -v
"""
import re
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from app import auth, limits, models
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
from app.routers.adventures import ACTION_PAGE
from tools import dbmeter
TOTAL = ACTION_PAGE * 3 + 7 # deliberately not a whole number of pages
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
user = models.User(is_guest=False, email="paging@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="m"))
adventure = models.Adventure(user_id=user.id, title="Cave", script_state={})
setup.add(adventure)
setup.flush()
for i in range(TOTAL):
setup.add(models.Action(
adventure_id=adventure.id,
type="start" if i == 0 else ("ai" if i % 2 else "do"),
text=f"Action {i}." + "word " * 200,
))
setup.commit()
adv_id, user_id = adventure.id, user.id
setup.close()
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
def _current_user(db=Depends(get_db)):
return db.get(models.User, user_id)
app.dependency_overrides[auth.get_current_user] = _current_user
c = TestClient(app)
c.adv_id = adv_id
try:
yield c
finally:
app.dependency_overrides.clear()
Base.metadata.drop_all(bind=engine)
def ordinal(action) -> int:
"""Returns which turn this is, read out of the fixture's own text.
The payload used to carry `index`, a story-wide turn number that SP8
dropped. Nothing replaced it: a depth is a position along one branch, and
the pager keys on ids. The fixture numbers its own actions, so these tests
read the number back rather than reintroduce one.
"""
return int(re.match(r"Action (\d+)\.", action["text"]).group(1))
def page(client, before_id=None, limit=None):
params = {}
if before_id is not None:
params["before_id"] = before_id
if limit is not None:
params["limit"] = limit
r = client.get(f"/api/adventures/{client.adv_id}/actions", params=params)
assert r.status_code == 200, r.text
return r.json()
def add_action(client, text="A new turn.") -> int:
db = SessionLocal()
try:
action = models.Action(
adventure_id=client.adv_id, type="ai", text=text
)
db.add(action)
db.commit()
return action.id
finally:
db.close()
# ------------------------------------------------------------- the page load
def test_the_page_load_returns_only_the_newest_window(client):
r = client.get(f"/api/adventures/{client.adv_id}")
assert r.status_code == 200
body = r.json()
assert len(body["actions"]) == ACTION_PAGE
assert body["action_count"] == TOTAL
# It is the newest window, ending on the last action.
assert ordinal(body["actions"][-1]) == TOTAL - 1
assert ordinal(body["actions"][0]) == TOTAL - ACTION_PAGE
def test_a_short_story_is_returned_whole(client):
db = SessionLocal()
try:
db.query(models.Action).filter(models.Action.depth >= 5).delete()
db.commit()
finally:
db.close()
body = client.get(f"/api/adventures/{client.adv_id}").json()
assert len(body["actions"]) == 5
assert body["action_count"] == 5
def test_the_page_load_does_not_grow_with_the_story(client):
"""Confirm that opening a story costs a window, regardless of the
story's length."""
meter = dbmeter.Meter()
meter.attach(engine)
try:
with meter.scope("page load"):
client.get(f"/api/adventures/{client.adv_id}")
windowed = meter.scopes[-1].total.fetched
finally:
meter.detach()
# Each action carries about 1 KB of text, and there are 187 of them. A
# window holds 60 actions. This ceiling is generous but still far below
# the size of the whole story.
assert windowed < ACTION_PAGE * 2_000, f"{windowed:,} B for one window"
assert windowed < TOTAL * 500, (
f"{windowed:,} B — that is the whole story, not a window"
)
# ------------------------------------------------------------------ paging up
def test_the_first_page_is_the_newest(client):
body = page(client)
assert len(body["actions"]) == ACTION_PAGE
assert body["total"] == TOTAL
assert body["has_more"] is True
assert ordinal(body["actions"][-1]) == TOTAL - 1
def test_paging_up_covers_the_whole_story_exactly_once(client):
seen = []
body = page(client)
seen = [ordinal(a) for a in body["actions"]]
guard = 0
while body["has_more"]:
guard += 1
assert guard < 20, "paging did not terminate"
body = page(client, before_id=body["actions"][0]["id"])
seen = [ordinal(a) for a in body["actions"]] + seen
assert seen == list(range(TOTAL)), "gap, duplicate or reordering while paging"
def test_has_more_is_false_at_the_beginning_of_the_story(client):
body = page(client)
while body["has_more"]:
body = page(client, before_id=body["actions"][0]["id"])
assert ordinal(body["actions"][0]) == 0
def test_each_page_is_ordered_oldest_first(client):
body = page(client)
indices = [ordinal(a) for a in body["actions"]]
assert indices == sorted(indices)
# ------------------------------------------------- the reason for the anchor
def test_a_turn_arriving_mid_scroll_does_not_shift_the_next_page(client):
"""Reproduce the failure an offset-based scheme would have. Read the
newest page, let a turn land, then page up. The reader must get exactly
what precedes the actions they already hold, with no duplicate and no
skipped action."""
first = page(client)
oldest_held = first["actions"][0]
add_action(client)
older = page(client, before_id=oldest_held["id"])
assert ordinal(older["actions"][-1]) == ordinal(oldest_held) - 1, (
"the page shifted when a turn landed"
)
assert all(ordinal(a) < ordinal(oldest_held) for a in older["actions"])
# The new turn changes the total, which is expected. It must not move the window.
assert older["total"] == TOTAL + 1
def test_a_deleted_anchor_reports_the_end_rather_than_a_duplicate_page(client):
"""Undo can remove the action a slow scroll was anchored to. The endpoint
must stop instead of returning a page the reader already has."""
body = page(client)
anchor = body["actions"][0]
db = SessionLocal()
try:
db.query(models.Action).filter(models.Action.id == anchor["id"]).delete()
db.commit()
finally:
db.close()
after = page(client, before_id=anchor["id"])
assert after["actions"] == []
assert after["has_more"] is False
# ------------------------------------------------------------------- limits
def test_limit_is_honoured_and_capped(client):
assert len(page(client, limit=5)["actions"]) == 5
# A client that asks for the whole story cannot bypass the paging cap.
assert len(page(client, limit=100_000)["actions"]) <= ACTION_PAGE * 4
def test_a_nonsense_limit_still_returns_something(client):
assert len(page(client, limit=0)["actions"]) >= 1
assert len(page(client, limit=-5)["actions"]) >= 1
# --------------------------------------------------------------------- undo
def test_undo_returns_a_window_not_the_story(client):
r = client.post(f"/api/adventures/{client.adv_id}/undo")
assert r.status_code == 200, r.text
body = r.json()
assert len(body["actions"]) == ACTION_PAGE
assert body["total"] == TOTAL - 1
assert body["has_more"] is True
backend/tests/test_analytics.py
+385
View File
@@ -0,0 +1,385 @@
"""Visit analytics: app/analytics.py and the two endpoints in front of it.
This file tests three things, and the rest is arithmetic. The counters must
survive the buffer/UPSERT round trip: a flush adds to what is already
stored instead of replacing it, or every number would show only the last
minute. The funnel counts people rather than clicks, which is the only
reason the visitor-day table exists. The gate holds: a stranger cannot read
the dashboard, and cannot inflate what it reports beyond hitting the page.
python -m pytest tests/test_analytics.py -v
"""
from datetime import timedelta
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from app import analytics, auth, limits, models
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
@pytest.fixture(autouse=True)
def clean_buffer():
"""The buffer is process-wide, so a test that leaves counts in it would
show up inside the next one's flush."""
analytics._counts.clear()
analytics._visits.clear()
analytics._labels_seen.clear()
yield
analytics._counts.clear()
analytics._visits.clear()
analytics._labels_seen.clear()
@pytest.fixture()
def db():
Base.metadata.create_all(bind=engine)
session = SessionLocal()
try:
yield session
finally:
session.close()
Base.metadata.drop_all(bind=engine)
def counter(db, metric, label):
row = (
db.query(models.AnalyticsDaily)
.filter_by(metric=metric, label=label)
.one_or_none()
)
return row.hits if row else 0
def make_user(db, email=None):
user = models.User(is_guest=email is None, email=email)
db.add(user)
db.commit()
return user
# ---------- The buffer and its flush ----------
def test_counts_accumulate_across_flushes(db):
analytics.record(analytics.M_PAGE, "/")
analytics.record(analytics.M_PAGE, "/")
analytics.flush(db)
analytics.record(analytics.M_PAGE, "/")
analytics.flush(db)
# The second flush has to find the existing row and add to it. Replacing it
# would leave every counter showing only the newest minute of traffic.
assert counter(db, analytics.M_PAGE, "/") == 3
def test_flush_is_a_no_op_when_nothing_happened(db):
analytics.flush(db)
assert db.query(models.AnalyticsDaily).count() == 0
def test_a_failed_flush_keeps_the_counts(db, monkeypatch):
analytics.record(analytics.M_PAGE, "/")
monkeypatch.setattr(analytics, "_write_counts", lambda *a: 1 / 0)
analytics.flush(db) # must not raise
monkeypatch.undo()
analytics.flush(db)
assert counter(db, analytics.M_PAGE, "/") == 1
def test_label_cardinality_is_capped(db):
for i in range(analytics.MAX_LABELS_PER_METRIC + 25):
analytics.record(analytics.M_REFERRER, f"host{i}.example")
analytics.flush(db)
labels = db.query(models.AnalyticsDaily).filter_by(metric=analytics.M_REFERRER).count()
# Everything past the cap is folded into one bucket, so a referrer flood
# cannot create unlimited rows.
assert labels == analytics.MAX_LABELS_PER_METRIC + 1
assert counter(db, analytics.M_REFERRER, analytics.OTHER) == 25
# ---------- Visitors ----------
def test_visitor_id_is_stable_and_keyed(db, monkeypatch):
user = make_user(db)
handle = analytics.visitor_id(user)
assert handle == analytics.visitor_id(user) # a returning visitor
assert handle != analytics.visitor_id(make_user(db)) # is still one visitor
assert len(handle) == 32 and int(handle, 16) >= 0 # opaque hex, not an id
# Keyed on the app secret, not a bare hash of the user id. Otherwise
# anyone holding this table could rebuild the mapping by hashing
# sequential ids.
monkeypatch.setattr(analytics.security, "SECRET_KEY", b"a-different-secret")
assert analytics.visitor_id(user) != handle
def test_a_repeat_visitor_is_new_only_once(db):
user = make_user(db)
analytics.record_visit(user)
analytics.flush(db)
rows = db.query(models.AnalyticsVisitorDay).all()
assert len(rows) == 1 and rows[0].is_new
# Same visitor, a later day: seen before, so not new. The row is not
# merged into the first day's row either.
tomorrow = (models.utcnow().date() + timedelta(days=1)).isoformat()
analytics._visits[(tomorrow, analytics.visitor_id(user))] = set()
analytics.flush(db)
rows = db.query(models.AnalyticsVisitorDay).order_by(models.AnalyticsVisitorDay.day).all()
assert [row.is_new for row in rows] == [True, False]
def test_one_row_per_visitor_per_day_however_much_they_do(db):
user = make_user(db)
for _ in range(5):
analytics.record_event(analytics.EV_ADVENTURE, user)
analytics.flush(db)
assert db.query(models.AnalyticsVisitorDay).count() == 1
assert counter(db, analytics.M_EVENT, analytics.EV_ADVENTURE) == 5
def test_funnel_flags_only_ever_turn_on(db):
user = make_user(db)
analytics.record_event(analytics.EV_TURN, user)
analytics.flush(db)
# A later visit that reaches no funnel step must not clear the earlier one.
analytics.record_visit(user)
analytics.flush(db)
row = db.query(models.AnalyticsVisitorDay).one()
assert row.played and not row.created
def test_purge_drops_only_rows_past_the_horizon(db):
old = (models.utcnow().date() - timedelta(days=analytics.RETENTION_DAYS + 1)).isoformat()
db.add(models.AnalyticsVisitorDay(day=old, visitor="a" * 32))
db.add(models.AnalyticsVisitorDay(day=analytics._today(), visitor="b" * 32))
db.commit()
assert analytics.purge_old_visitor_days(db) == 1
assert [r.visitor for r in db.query(models.AnalyticsVisitorDay)] == ["b" * 32]
# ---------- Normalizing what a browser claims ----------
@pytest.mark.parametrize("path, expected", [
("/", "/"),
("/adventures", "/adventures"),
("/adventures/", "/adventures"),
("/play/12?x=1", "/play/:id"),
("/scenarios/9#top", "/scenarios/:id"),
("/wp-admin", "(other)"),
("/play/../../etc", "(other)"),
("", "/"),
])
def test_route_normalization(path, expected):
assert analytics.normalize_route(path) == expected
@pytest.mark.parametrize("referrer, expected", [
("", "(direct)"),
("https://news.ycombinator.com/item?id=1", "news.ycombinator.com"),
("https://www.google.com/", "google.com"),
("https://ai-dnd.example/scenarios", ""), # our own host: not a referral
("javascript:alert(1)", "(other)"),
("https://" + "x" * 200 + ".com", "(other)"),
])
def test_referrer_normalization(referrer, expected):
assert analytics.normalize_referrer(referrer, "ai-dnd.example") == expected
@pytest.mark.parametrize("ua, expected", [
("Mozilla/5.0 (iPhone; CPU iPhone OS 17_0) AppleWebKit", "mobile"),
("Mozilla/5.0 (iPad; CPU OS 17_0) AppleWebKit", "tablet"),
("Mozilla/5.0 (Windows NT 10.0; Win64; x64)", "desktop"),
("Googlebot/2.1", "bot"),
("", "(unknown)"),
])
def test_device_detection(ua, expected):
assert analytics.device_of(ua) == expected
def test_only_iso_looking_country_headers_are_trusted():
assert analytics.country_of({"cf-ipcountry": "de"}) == "DE"
assert analytics.country_of({"cf-ipcountry": "Norway"}) == analytics.UNKNOWN
assert analytics.country_of({"cf-ipcountry": "XX"}) == analytics.UNKNOWN
assert analytics.country_of({}) == analytics.UNKNOWN
def test_error_labels_use_the_route_not_the_path():
class Route:
path = "/api/adventures/{adventure_id}"
assert analytics.api_route_label({"route": Route()}, 500) == "500 /api/adventures/{adventure_id}"
# An unmatched path is entirely attacker-chosen, so it never becomes a label.
assert analytics.api_route_label({}, 404) == "404 (unmatched)"
# ---------- The summary ----------
def test_summary_counts_people_once_per_step(db):
one, two = make_user(db), make_user(db)
for _ in range(3):
analytics.record_event(analytics.EV_SCENARIO_OPEN, one)
analytics.record_event(analytics.EV_TURN, one)
analytics.record_event(analytics.EV_SCENARIO_OPEN, two)
result = analytics.summary(db, days=7)
steps = {row["step"]: row["count"] for row in result["funnel"]}
assert steps["Visited"] == 2
assert steps["Opened a scenario"] == 2
assert steps["Played a turn"] == 1 # not 3, because one person made three turns
assert steps["Signed up"] == 0
# Raw event totals still count every occurrence.
assert result["totals"]["turns"] == 3
assert result["totals"]["visitors"] == 2
def test_summary_series_covers_every_day_including_empty_ones(db):
analytics.record(analytics.M_PAGE, "/")
result = analytics.summary(db, days=7)
assert len(result["series"]) == 7
assert result["series"][-1]["day"] == models.utcnow().date().isoformat()
assert result["series"][-1]["pageviews"] == 1
assert result["series"][0]["pageviews"] == 0
def test_summary_flushes_before_reading(db):
analytics.record(analytics.M_EVENT, analytics.EV_TURN)
# Never flushed by hand: the dashboard must not be up to a minute stale.
assert analytics.summary(db, days=1)["totals"]["turns"] == 1
def test_summary_reports_pages_referrers_and_errors(db):
analytics.record(analytics.M_PAGE, "/play/:id", n=4)
analytics.record(analytics.M_REFERRER, "news.ycombinator.com", n=2)
analytics.record(analytics.M_ERROR, "500 /api/adventures/{adventure_id}")
result = analytics.summary(db, days=30)
assert result["pages"][0] == {"label": "/play/:id", "hits": 4}
assert result["referrers"][0]["label"] == "news.ycombinator.com"
assert result["totals"]["errors"] == 1
# ---------- The endpoints ----------
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
visitor = models.User(is_guest=True)
owner = models.User(is_guest=False, email="owner@example.com")
setup.add_all([visitor, owner])
setup.commit()
ids = {"visitor": visitor.id, "owner": owner.id}
setup.close()
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
# Multi-user is what makes the gate mean anything: local mode trusts
# whoever is at the keyboard, because it is the operator's own machine.
monkeypatch.setattr(auth, "MULTI_USER", True)
monkeypatch.setattr(auth, "ANALYTICS_EMAILS", {"owner@example.com"})
current = {"id": ids["visitor"]}
def _current_user(db=Depends(get_db)):
return db.get(models.User, current["id"])
app.dependency_overrides[auth.get_current_user] = _current_user
monkeypatch.setattr(
auth, "resolve_session_user", lambda request, db: db.get(models.User, current["id"])
)
try:
client = TestClient(app)
client.ids, client.current = ids, current
yield client
finally:
app.dependency_overrides.clear()
Base.metadata.drop_all(bind=engine)
def read_summary(client, days=30):
return client.get(f"/api/analytics/summary?days={days}")
def test_dashboard_is_invisible_to_everyone_but_the_owner(client):
assert read_summary(client).status_code == 404
client.current["id"] = client.ids["owner"]
assert read_summary(client).status_code == 200
def test_collect_records_a_pageview_and_the_visit(client):
resp = client.post("/api/analytics/collect", json={"path": "/play/7", "first": True,
"referrer": "https://news.ycombinator.com/"})
assert resp.status_code == 204
client.current["id"] = client.ids["owner"]
body = read_summary(client).json()
assert body["pages"][0] == {"label": "/play/:id", "hits": 1}
assert body["referrers"][0]["label"] == "news.ycombinator.com"
assert body["totals"]["visitors"] == 1
def test_referrer_and_device_are_recorded_once_per_visit_not_per_view(client):
for path in ("/", "/scenarios", "/adventures"):
client.post("/api/analytics/collect", json={"path": path, "first": path == "/"})
client.current["id"] = client.ids["owner"]
body = read_summary(client).json()
assert body["totals"]["pageviews"] == 3
# Three views, one visit: the referral and the device are facts about the
# visit, so counting them per view would multiply every one of them.
assert sum(row["hits"] for row in body["devices"]) == 1
def test_the_owners_own_visits_are_not_traffic(client):
client.current["id"] = client.ids["owner"]
client.post("/api/analytics/collect", json={"path": "/", "first": True})
assert read_summary(client).json()["totals"]["pageviews"] == 0
def test_a_client_cannot_invent_pages_or_events(client):
client.post("/api/analytics/collect", json={"path": "/../../admin", "first": True})
# There is no field for it, so a made-up event is not even expressible.
client.post("/api/analytics/collect", json={"path": "/", "event": "signup"})
client.current["id"] = client.ids["owner"]
body = read_summary(client).json()
assert {row["label"] for row in body["pages"]} == {"(other)", "/"}
assert body["totals"]["signups"] == 0
def test_api_errors_are_counted_by_route(client):
client.get("/api/adventures/999999")
client.current["id"] = client.ids["owner"]
errors = read_summary(client).json()["errors"]
assert errors and errors[0]["label"].startswith("404 /api/adventures/")
# ---------- The dialect the tests never run on ----------
def test_the_upserts_compile_for_postgres():
"""Prod runs on Neon, but these tests run on SQLite, and a failed flush
is caught and logged instead of raised. A dialect mistake would
therefore stay invisible until the dashboard quietly stayed empty. This
test compiles both statements against Postgres without connecting to
one.
"""
from sqlalchemy import create_engine
from sqlalchemy.dialects import postgresql
from sqlalchemy.orm import sessionmaker
session = sessionmaker(bind=create_engine("postgresql+psycopg://u:p@localhost/db"))()
compiled = []
def capture(statement, *args, **kwargs):
compiled.append(str(statement.compile(dialect=postgresql.dialect())))
session.execute = capture
session.scalars = lambda *a, **k: []
analytics._write_counts(session, {("2026-01-01", "pageview", "/"): 2})
analytics._write_visits(session, {("2026-01-01", "f" * 32): {"played"}})
counts, visits = compiled
assert "ON CONFLICT (day, metric, label) DO UPDATE" in counts
assert "analytics_daily.hits + excluded.hits" in counts
assert "ON CONFLICT (day, visitor) DO UPDATE" in visits
assert "analytics_visitor_days.played OR excluded.played" in visits
# is_new is settled by the first write of a visitor's first day and must
# not be in the update clause at all.
assert "is_new" not in visits.split("DO UPDATE")[1]
backend/tests/test_attempt_siblings.py
+444
View File
@@ -0,0 +1,444 @@
"""Phase 14 SP4: a retry writes a sibling node instead of rewriting a row.
`test_retry_variants.py` is the behavioral contract from before the tree
existed, and it still passes unchanged: the same URLs, the same payload
shape, the same outcomes. This file asserts the things that are true only of
the new storage. A turn can be several rows, exactly one of them is the
story, and the arrangement costs neither an extra prompt nor an extra turn.
python -m pytest tests/test_attempt_siblings.py -v
"""
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from sqlalchemy.orm import undefer
from app import attempts, auth, limits, models, tree
from app.context import cursors, history
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
from app.routers import adventures
from fakes import ScriptedProvider
SCHEMA = {"player": {"hp": {"min": 0, "max": 100, "initial": 100}}}
GOLD_SCRIPT = """
const modifier = (text) => {
state.gold = (state.gold || 0) + 10;
return { text };
};
modifier(text);
"""
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
user = models.User(is_guest=False, email="siblings@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="test-model"))
scenario = models.Scenario(user_id=user.id, title="S", stat_schema=SCHEMA)
setup.add(scenario)
setup.flush()
adv = models.Adventure(
user_id=user.id, title="Cave", scenario_id=scenario.id,
script_state={}, world_state={"player": {"hp": 100}},
)
setup.add(adv)
setup.flush()
setup.add(models.Action(adventure_id=adv.id, type="start", text="You enter a cave."))
setup.add(models.AdventureScript(
adventure_id=adv.id, position=0, enabled=True, name="Gold", output_js=GOLD_SCRIPT,
))
setup.commit()
adv_id, user_id = adv.id, user.id
setup.close()
ScriptedProvider.replies = ["Attempt one."]
ScriptedProvider.calls = 0
ScriptedProvider.prompts = []
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
monkeypatch.setattr(auth, "resolve_provider_config", lambda s: auth.ProviderConfig(
"http://fake", "k", "test-model", False))
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
def _current_user(db=Depends(get_db)):
return db.get(models.User, user_id)
app.dependency_overrides[auth.get_current_user] = _current_user
c = TestClient(app)
c.adv_id = adv_id
try:
yield c
finally:
app.dependency_overrides.clear()
adventures.turns._active_turns.clear()
Base.metadata.drop_all(bind=engine)
def _play(client, text="look around", type="do"):
r = client.post(f"/api/adventures/{client.adv_id}/actions",
json={"type": type, "text": text})
assert r.status_code == 200, r.text
def _retry(client):
r = client.post(f"/api/adventures/{client.adv_id}/retry")
assert r.status_code == 200, r.text
def _page(client) -> dict:
return client.get(f"/api/adventures/{client.adv_id}").json()
def _rows(adv_id) -> list[models.Action]:
"""Every action row of the adventure, story or not, live or not.
The query undefers these columns because the session closes before the
caller reads the result. This file specifically tests the columns that a
page load never loads.
"""
db = SessionLocal()
try:
return (
db.query(models.Action)
.filter(models.Action.adventure_id == adv_id)
.options(
undefer(models.Action.state_after),
undefer(models.Action.world_state_after),
undefer(models.Action.context_snapshot),
)
.order_by(models.Action.depth, models.Action.id)
.all()
)
finally:
db.close()
# ------------------------------------------------------------- the sibling
def test_a_retry_writes_a_second_row_at_the_same_coordinate(client):
ScriptedProvider.replies = ["Attempt one.", "Attempt two."]
_play(client)
_retry(client)
rows = _rows(client.adv_id)
ai = [a for a in rows if a.type == "ai"]
assert len(ai) == 2, "a retry is a node, not a rewrite"
assert {(a.branch_id, a.depth) for a in ai} == {(ai[0].branch_id, ai[0].depth)}
assert [a.text for a in ai] == ["Attempt one.", "Attempt two."]
# Exactly one of them is the story, and it is the newer take.
assert [a.live for a in ai] == [False, True]
# The discarded attempt is untouched, not a copy of anything.
assert ai[0].state_after is not None
def test_the_story_shows_and_counts_the_turn_once(client):
ScriptedProvider.replies = ["Attempt one.", "Attempt two."]
_play(client)
before = _page(client)["action_count"]
_retry(client)
after = _page(client)
assert after["action_count"] == before, "a discarded attempt is not a turn"
assert [a["type"] for a in after["actions"]] == ["start", "do", "ai"]
assert after["actions"][-1]["text"] == "Attempt two."
def test_a_discarded_attempt_never_reaches_the_prompt(client):
"""This is the failure case the branch clause exists to prevent, at
sibling scale. The losing attempt sits at the same branch and depth as
the live one, so anything reading the story by coordinate alone would
replay both."""
ScriptedProvider.replies = ["Attempt one.", "Attempt two.", "Next turn."]
_play(client)
_retry(client)
_play(client, "go deeper")
story = ScriptedProvider.prompts[-1][1]
assert "Attempt two." in story
assert "Attempt one." not in story
def test_switching_moves_the_story_onto_the_other_row(client):
ScriptedProvider.replies = [
"A scratch.\n```state\n{\"player.hp\": -5}\n```",
"A beating.\n```state\n{\"player.hp\": -40}\n```",
]
_play(client)
_retry(client)
newest_id = _page(client)["actions"][-1]["id"]
r = client.post(
f"/api/adventures/{client.adv_id}/actions/{newest_id}/variant", json={"index": 0})
assert r.status_code == 200, r.text
# A different row answers the request. That is the only change.
assert r.json()["id"] != newest_id
assert r.json()["text"].startswith("A scratch")
rows = _rows(client.adv_id)
ai = [a for a in rows if a.type == "ai"]
assert [a.live for a in ai] == [True, False]
# Both takes remain unchanged in the database.
assert [a.text.split(".")[0] for a in ai] == ["A scratch", "A beating"]
def test_the_assembled_prompt_is_stored_once_per_turn(client):
"""A snapshot holds about 160 kB of prompt that every attempt at a turn
shares. Giving each sibling its own copy would make retry multiply the
size of the largest column in the database. Instead, the prompt moves
with the live flag."""
ScriptedProvider.replies = ["Attempt one.", "Attempt two."]
_play(client)
_retry(client)
def holders():
return [
a.id for a in _rows(client.adv_id)
if a.type == "ai" and "sections" in (a.context_snapshot or {})
]
live_holder = holders()
assert len(live_holder) == 1
newest = _page(client)["actions"][-1]
assert live_holder == [newest["id"]]
client.post(f"/api/adventures/{client.adv_id}/actions/{newest['id']}/variant",
json={"index": 0})
moved = holders()
assert len(moved) == 1 and moved != live_holder, "the prompt follows the story"
# ------------------------------------------------------- removing the turn
def test_undo_takes_every_attempt_with_it(client):
ScriptedProvider.replies = ["One.", "Two.", "Three."]
_play(client)
_retry(client)
_retry(client)
assert len([a for a in _rows(client.adv_id) if a.type == "ai"]) == 3
r = client.post(f"/api/adventures/{client.adv_id}/undo")
assert r.status_code == 200, r.text
assert [a.type for a in _rows(client.adv_id)] == ["start"]
def test_deleting_a_retried_turn_deletes_its_attempts(client):
ScriptedProvider.replies = ["One.", "Two."]
_play(client)
_retry(client)
newest = _page(client)["actions"][-1]
r = client.delete(f"/api/adventures/{client.adv_id}/actions/{newest['id']}")
assert r.status_code == 204, r.text
assert [a.type for a in _rows(client.adv_id)] == ["start", "do"]
def test_deleting_a_turn_through_a_discarded_attempt_still_takes_the_turn(client):
"""The pager hands out whichever id it last saw, and a switch changes which
row that is. Deleting through the losing sibling must not leave the story
holding a turn with no attempts."""
ScriptedProvider.replies = ["One.", "Two."]
_play(client)
_retry(client)
discarded = [a for a in _rows(client.adv_id) if a.type == "ai" and not a.live][0]
r = client.delete(f"/api/adventures/{client.adv_id}/actions/{discarded.id}")
assert r.status_code == 204, r.text
assert [a.type for a in _rows(client.adv_id)] == ["start", "do"]
# ------------------------------------------- what the holdback used to cover
def test_retrying_withdraws_the_memory_the_turn_produced(client):
"""Why summarization no longer holds the newest action back.
A memory covering the newest turn used to be unreachable by
construction. The summarizer stopped one action short, because a retry
rewrote the row under a mark that had already moved past it. Now the
mark and the memory both name the node, so replacing what a node says
withdraws them. Undo and delete already had this repair; the holdback
was the only gap a retry still needed to close.
"""
ScriptedProvider.replies = ["One.", "Two."]
_play(client)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
newest = history.newest(adventure)
memory = models.Memory(
adventure_id=adventure.id, text="You looked around.",
source_start=1, source_end=newest.depth,
)
tree.attach_memory(memory, newest)
db.add(memory)
cursors.MEMORY.anchor_at(adventure, newest)
cursors.SUMMARY.anchor_at(adventure, newest)
db.commit()
covered_depth = newest.depth
finally:
db.close()
_retry(client)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
assert db.query(models.Memory).count() == 0, "the withdrawn memory is gone"
# The depth range it covered is released, so the block is
# summarized again from where it began instead of being skipped.
assert cursors.MEMORY.depth(db, adventure) == 0
assert cursors.SUMMARY.depth(db, adventure) == 0
assert covered_depth > 0
finally:
db.close()
def test_a_memory_on_an_earlier_turn_survives_a_retry(client):
"""Only the coordinate whose text changed is withdrawn."""
ScriptedProvider.replies = ["One.", "Two.", "Three."]
_play(client)
_play(client, "go deeper")
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
earlier = history.tail(adventure, 3)[0]
memory = models.Memory(
adventure_id=adventure.id, text="An earlier block.",
source_start=0, source_end=earlier.depth,
)
tree.attach_memory(memory, earlier)
db.add(memory)
db.commit()
finally:
db.close()
_retry(client)
db = SessionLocal()
try:
assert [m.text for m in db.query(models.Memory).all()] == ["An earlier block."]
finally:
db.close()
# -------------------------------------------------------------- the group
def test_the_group_grows_in_the_order_the_attempts_arrive(client):
"""The attempts page in the order they were made, and the pager counts them.
Ordering used to come from `variant_index`, an explicit ordinal that SP8
dropped. `id` carries the same order, because a row is inserted when its
attempt is made.
"""
ScriptedProvider.replies = ["One.", "Two.", "Three."]
_play(client)
assert _page(client)["actions"][-1]["take_count"] == 1 # never retried
_retry(client)
_retry(client)
ai = [a for a in _rows(client.adv_id) if a.type == "ai"]
assert [a.text for a in ai] == ["One.", "Two.", "Three."]
assert _page(client)["actions"][-1]["take_count"] == 3
def test_attempts_module_agrees_with_the_endpoint(client):
ScriptedProvider.replies = ["One.", "Two."]
_play(client)
_retry(client)
newest = _page(client)["actions"][-1]
listed = client.get(
f"/api/adventures/{client.adv_id}/actions/{newest['id']}/variants").json()
db = SessionLocal()
try:
node = db.get(models.Action, newest["id"])
group = attempts.group(db, node)
assert [a.text for a in group] == [v["text"] for v in listed]
assert attempts.live_in(group).id == newest["id"]
finally:
db.close()
def test_export_carries_every_attempt_as_its_own_node(client):
"""SP6 changed the answer here, for the same reasons as the rest of that
subphase.
A v1 bundle had one entry per turn and folded the group back into a
`variants` array, because the format had nowhere else to put a second
take. A v2 bundle has coordinates, so an attempt is a node in the file
exactly as it is a node in the database, and `live` says which one is
the story.
"""
ScriptedProvider.replies = ["One.", "Two."]
_play(client)
_retry(client)
bundle = client.get(f"/api/adventures/{client.adv_id}/export").json()
ai = [a for a in bundle["actions"] if a["type"] == "ai"]
assert [(a["text"], a["live"]) for a in ai] == [("One.", False), ("Two.", True)]
assert len({(a["branch"], a["depth"]) for a in ai}) == 1, "one turn, two takes"
assert "variants" not in ai[0], "nothing writes the repeating group any more"
# Importing the bundle puts the group back exactly as it stood.
imported = client.post("/api/adventures/import", json=bundle).json()["id"]
rows = _rows(imported)
ai_rows = [a for a in rows if a.type == "ai"]
assert [(a.text, a.live) for a in ai_rows] == [("One.", False), ("Two.", True)]
assert len({(a.branch_id, a.depth) for a in ai_rows}) == 1
def test_a_retry_after_switching_back_files_the_new_attempt_last(client):
"""The group stays in the order the attempts were made.
`add_attempt` used to number a new take one past the take it replaced.
That numbering is correct only when the story is standing on the newest
take. Switch a three-take turn back to the first and retry, and the new
attempt collided with take 2, which put it between takes 2 and 3. The
group orders by `id` now, so a new attempt is always last.
"""
ScriptedProvider.replies = ["One.", "Two.", "Three.", "Four."]
_play(client)
_retry(client)
_retry(client)
assert [a.text for a in _rows(client.adv_id) if a.type == "ai"] == [
"One.", "Two.", "Three."]
live = _page(client)["actions"][-1]
r = client.post(
f"/api/adventures/{client.adv_id}/actions/{live['id']}/variant",
json={"index": 0})
assert r.status_code == 200, r.text
_retry(client)
ai = [a for a in _rows(client.adv_id) if a.type == "ai"]
assert [a.text for a in ai] == ["One.", "Two.", "Three.", "Four."]
assert attempts.live_in(ai).text == "Four."
def test_the_adventure_list_quotes_the_take_the_story_tells(client):
"""The index screen and the story have to agree.
Siblings share a depth, and the newest of them has the highest id. A
snippet ordered by `(depth, id)` alone quotes whichever attempt was
written last. After switching back, that attempt is the one the player
discarded.
"""
ScriptedProvider.replies = ["One.", "Two."]
_play(client)
_retry(client)
live = _page(client)["actions"][-1]
assert live["text"] == "Two."
client.post(f"/api/adventures/{client.adv_id}/actions/{live['id']}/variant",
json={"index": 0})
listed = client.get("/api/adventures").json()
row = [a for a in listed if a["id"] == client.adv_id][0]
assert "One." in row["snippet"]
assert "Two." not in row["snippet"]
backend/tests/test_branch_clause.py
+558
View File
@@ -0,0 +1,558 @@
"""Phase 14 SP2: a read sees one story, and knows which one.
These tests build the fork by hand. Three branch rows and their nodes are
written straight to the database, arranged as the design doc's own worked
example. That was the only way to build one when this file was written,
because nothing forked until SP5. It stays that way now that `tree.fork`
exists, because a fixture built with the same code under test could not
catch that code being wrong. `test_branch_forking.py` checks the two
against each other.
branch C, tip at depth 7, lineage [(C, 7), (B, 5), (A, 3)]
-> A0 A1 A2 A3 B4 B5 C6 C7
The point of building the fixture by hand is that every read in the app
must go through one module. A forgotten clause does not raise an error. It
silently shows a story assembled out of two different branches. The
fixture deliberately leaves nodes where a forgotten clause would pick them
up: A kept playing past the fork (A4, A5), B kept playing past its own
(B6), and a second adventure holds a whole story of its own. None of these
nodes may appear on C.
python -m pytest tests/test_branch_clause.py -v
"""
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from sqlalchemy import event
from app import auth, limits, models, tree
from app.context import history, lineage
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
from app.scripting import ScriptPipeline
from tools import dbmeter
# --------------------------------------------------------------- the fixture
def make_branch(db, adventure, parent=None, fork_depth=None):
"""A branch row whose lineage is its parent's, capped, plus itself.
This function performs the same computation SP5 does at fork time. It
is written out here so the fixture cannot pass by repeating a bug in
the code under test.
"""
branch = models.Branch(
adventure_id=adventure.id,
parent_branch_id=parent.id if parent else None,
fork_depth=fork_depth,
lineage=[],
)
db.add(branch)
db.flush()
inherited = []
if parent is not None:
for ancestor_id, cap in lineage.entries_of(parent):
capped = fork_depth if cap is None else min(cap, fork_depth)
inherited.append([ancestor_id, capped])
branch.lineage = [[branch.id, None]] + inherited
db.flush()
return branch
def add_node(db, adventure, branch, depth, label, index=None):
action = models.Action(
adventure_id=adventure.id,
branch_id=branch.id,
depth=depth,
type="start" if depth == 0 else ("ai" if depth % 2 else "do"),
text=f"{label}{depth}",
)
db.add(action)
return action
def make_adventure(db, user, title):
adventure = models.Adventure(user_id=user.id, title=title, script_state={})
db.add(adventure)
db.flush()
return adventure
@pytest.fixture()
def forked():
"""The worked example, plus everything a forgotten clause would expose."""
Base.metadata.create_all(bind=engine)
db = SessionLocal()
user = models.User(is_guest=False, email="branch@example.com")
db.add(user)
db.flush()
db.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="m"))
adventure = make_adventure(db, user, "Forked")
a = make_branch(db, adventure)
b = make_branch(db, adventure, parent=a, fork_depth=3)
c = make_branch(db, adventure, parent=b, fork_depth=5)
for depth in range(4):
add_node(db, adventure, a, depth, "A")
# A did not stop when B forked off it: these two are siblings of B4/B5.
for depth in (4, 5):
add_node(db, adventure, a, depth, "A", index=100 + depth)
for depth in (4, 5):
add_node(db, adventure, b, depth, "B")
add_node(db, adventure, b, 6, "B", index=200 + 6) # B's own continuation
for depth in (6, 7):
add_node(db, adventure, c, depth, "C")
# A second adventure, so "does the clause remember its adventure?" has an
# answer. A database holding one adventure cannot tell you.
other = make_adventure(db, user, "Elsewhere")
other_branch = make_branch(db, other)
for depth in range(4):
add_node(db, other, other_branch, depth, "X")
adventure.head_branch_id = c.id
adventure.head_depth = 7
other.head_branch_id = other_branch.id
other.head_depth = 3
db.commit()
ids = {"a": a.id, "b": b.id, "c": c.id, "adventure": adventure.id,
"other": other.id, "user": user.id}
try:
yield db, adventure, ids
finally:
db.close()
Base.metadata.drop_all(bind=engine)
def switch_to(db, adventure, branch_id, tip):
"""Move the head, the way SP7's branch picker will."""
adventure.head_branch_id = branch_id
adventure.head_depth = tip
db.commit()
def labels(actions):
return [a.text for a in actions]
# ------------------------------------------------------------- the story read
def test_the_worked_example_reads_back_as_the_design_doc_says(forked):
db, adventure, _ = forked
assert labels(history.story_actions(adventure)) == [
"A0", "A1", "A2", "A3", "B4", "B5", "C6", "C7"
]
def test_a_siblings_nodes_are_invisible(forked):
db, adventure, _ = forked
seen = labels(history.story_actions(adventure))
# A4 and A5 are A's own continuation past B's fork. B6 is B's own
# continuation past C's fork.
assert "A4" not in seen and "A5" not in seen and "B6" not in seen
# Nothing from the other adventure appears either.
assert not [text for text in seen if text.startswith("X")]
def test_each_branch_reads_its_own_story(forked):
db, adventure, ids = forked
switch_to(db, adventure, ids["a"], 5)
assert labels(history.story_actions(adventure)) == [
"A0", "A1", "A2", "A3", "A4", "A5"
]
switch_to(db, adventure, ids["b"], 6)
assert labels(history.story_actions(adventure)) == [
"A0", "A1", "A2", "A3", "B4", "B5", "B6"
]
switch_to(db, adventure, ids["c"], 7)
assert labels(history.story_actions(adventure)) == [
"A0", "A1", "A2", "A3", "B4", "B5", "C6", "C7"
]
def test_the_ancestors_shared_nodes_are_shared_not_copied(forked):
db, adventure, ids = forked
# A0..A3 appear on all three stories and exist exactly once in the table.
rows = (
db.query(models.Action)
.filter(models.Action.adventure_id == adventure.id,
models.Action.text == "A2")
.all()
)
assert len(rows) == 1
assert rows[0].branch_id == ids["a"]
def test_count_and_tail_and_slice_agree_with_the_path(forked):
db, adventure, _ = forked
assert history.count(adventure) == 8
assert labels(history.tail(adventure, 3)) == ["B5", "C6", "C7"]
assert labels(history.tail_range(adventure, 2, 2)) == ["B4", "B5"]
assert labels(history.slice_(adventure, 3, 3)) == ["A3", "B4", "B5"]
def test_a_window_that_reaches_past_the_fork_still_reads_in_order(forked):
db, adventure, _ = forked
# 32 is history.WINDOW_START: more than the whole story, so the read has to
# widen through all three lineage entries and still come back in order.
assert labels(history.window_covering(adventure, 10 ** 6, len)) == [
"A0", "A1", "A2", "A3", "B4", "B5", "C6", "C7"
]
# ------------------------------------------- the loaded-collection short cut
def test_an_already_loaded_collection_is_cut_down_to_the_path(forked):
"""Tests `history._from_memory`'s shortcut, the highest-risk line here.
`adventure.actions` returns every branch's actions. Slicing it without
the path would assemble a prompt out of two different stories, and
nothing would raise an error. This test loads the collection
deliberately and checks that the answer is still the path.
"""
db, adventure, _ = forked
loaded = list(adventure.actions) # every branch, ordered by index
assert len(loaded) == 11 # the path's 8, plus A4, A5 and B6
assert labels(history.story_actions(adventure)) == [
"A0", "A1", "A2", "A3", "B4", "B5", "C6", "C7"
]
assert history.count(adventure) == 8
assert labels(history.tail(adventure, 3)) == ["B5", "C6", "C7"]
def test_user_scripts_are_handed_the_path(forked):
"""The same risk one layer up, in code visible to users:
`pipeline._history()` is the documented scripting history API."""
db, adventure, _ = forked
list(adventure.actions) # the pipeline's caller has usually loaded these
pipeline = ScriptPipeline(adventure, db)
assert [h["text"] for h in pipeline._history()] == [
"A0", "A1", "A2", "A3", "B4", "B5", "C6", "C7"
]
assert pipeline._info()["actionCount"] == 8
# ------------------------------------------------------------ over the wire
@pytest.fixture()
def client(forked, monkeypatch):
db, adventure, ids = forked
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
def _current_user(session=Depends(get_db)):
return session.get(models.User, ids["user"])
app.dependency_overrides[auth.get_current_user] = _current_user
c = TestClient(app)
try:
yield c, db, adventure, ids
finally:
app.dependency_overrides.clear()
def test_the_page_the_reader_opens_is_the_path(client):
c, db, adventure, ids = client
r = c.get(f"/api/adventures/{adventure.id}")
assert r.status_code == 200, r.text
body = r.json()
assert [a["text"] for a in body["actions"]] == [
"A0", "A1", "A2", "A3", "B4", "B5", "C6", "C7"
]
# `action_count` tells the reader whether more actions exist above. It
# counts the path too: 8, not the 13 rows the adventure holds.
assert body["action_count"] == 8
def test_paging_up_walks_the_path_across_the_forks(client):
c, db, adventure, ids = client
first = c.get(f"/api/adventures/{adventure.id}/actions", params={"limit": 3}).json()
assert [a["text"] for a in first["actions"]] == ["B5", "C6", "C7"]
assert first["has_more"] is True
older = c.get(
f"/api/adventures/{adventure.id}/actions",
params={"limit": 3, "before_id": first["actions"][0]["id"]},
).json()
assert [a["text"] for a in older["actions"]] == ["A2", "A3", "B4"]
oldest = c.get(
f"/api/adventures/{adventure.id}/actions",
params={"limit": 3, "before_id": older["actions"][0]["id"]},
).json()
assert [a["text"] for a in oldest["actions"]] == ["A0", "A1"]
assert oldest["has_more"] is False
def test_paging_from_an_anchor_off_the_path_reports_the_end(client):
"""A stale client holding an id from another branch gets an empty page,
not that branch's story."""
c, db, adventure, ids = client
off_path = (
db.query(models.Action)
.filter(models.Action.adventure_id == adventure.id,
models.Action.text == "A5")
.one()
)
page = c.get(
f"/api/adventures/{adventure.id}/actions",
params={"before_id": off_path.id},
).json()
assert page["actions"] == []
assert page["has_more"] is False
def test_the_index_screen_quotes_the_branch_being_played(client):
c, db, adventure, ids = client
listed = {row["id"]: row for row in c.get("/api/adventures").json()}
# The newest *narrated* node on C: do/say are the player's voice, and the
# fixture alternates types, so C7 is the one that reads as story.
assert listed[adventure.id]["snippet"] == "C7"
switch_to(db, adventure, ids["a"], 5)
listed = {row["id"]: row for row in c.get("/api/adventures").json()}
assert listed[adventure.id]["snippet"] == "A5"
assert listed[ids["other"]]["snippet"] == "X3"
# ------------------------------------------------------- the flush guard
def test_a_node_written_without_a_branch_is_placed_anyway(forked):
"""SP1 wired the writers. Since SP2, an unplaced node is an invisible one.
This behavior lets a fixture, a script, or a test built straight through
the ORM keep working. It is also why the baseline contract still passes
with its actions written directly to the database.
"""
db, adventure, ids = forked
written = models.Action(
adventure_id=adventure.id, type="do", text="C8"
)
db.add(written)
db.commit()
assert written.branch_id == ids["c"]
# The node lands one step past the tip of C. SP8 dropped `index`, which is
# where a caller used to name its own depth, so an unplaced node now always
# follows the head.
assert written.depth == adventure.head_depth
assert adventure.head_depth == 8
assert labels(history.tail(adventure, 2)) == ["C7", "C8"]
def test_a_memory_written_without_a_branch_is_placed_anyway(forked):
db, adventure, ids = forked
memory = models.Memory(
adventure_id=adventure.id, text="The cave was cold.", source_start=0, source_end=3
)
db.add(memory)
db.commit()
assert memory.branch_id == ids["c"]
assert memory.depth == 3
def test_placing_a_flush_of_nodes_reads_the_branch_once(forked, emitted_sql):
"""The guard resolves the head once per flush, not once per node.
The identity map holds weak references. A branch row with no strong
reference gets collected between two nodes and read back again for the
next one. Writing two hundred actions in one flush ran two hundred
SELECTs on `branches` before the head lookup moved outside the loop.
The test result alone would not have shown this.
"""
db, adventure, _ = forked
emitted_sql.clear()
for i in range(50):
db.add(models.Action(
adventure_id=adventure.id, type="do", text=f"bulk {i}"
))
db.commit()
branch_reads = [s for s in emitted_sql if s.startswith("SELECT") and "FROM branches" in s]
assert len(branch_reads) <= 2, (
f"{len(branch_reads)} reads of `branches` to place 50 nodes"
)
def test_an_adventure_with_no_branch_at_all_reads_as_empty(forked):
"""A missing branch must fail loudly: nothing, rather than everything.
A row with no branch cannot be shown without guessing which story it
belongs to, and a wrong guess here puts a sibling's turns into a
prompt.
"""
db, adventure, ids = forked
stray = make_adventure(db, db.get(models.User, ids["user"]), "Stray")
db.query(models.Action).filter(models.Action.text == "A0").update(
{"adventure_id": stray.id}, synchronize_session=False
)
db.commit()
db.expire_all()
stray = db.get(models.Adventure, stray.id)
assert lineage.path_of(db, stray).entries == []
assert history.story_actions(stray) == []
assert history.count(stray) == 0
# --------------------------------------------------- the cost of forking
@pytest.fixture()
def deeply_forked():
"""A story forked twenty times, then played forty turns past the last one.
This shape tests the design's core assumption: reading the tail of this
story must cost the same as reading the tail of an unforked story,
because the window is covered long before the ancestry runs out.
"""
Base.metadata.create_all(bind=engine)
db = SessionLocal()
user = models.User(is_guest=False, email="deep@example.com")
db.add(user)
db.flush()
adventure = make_adventure(db, user, "Deep")
branch = make_branch(db, adventure)
depth = 0
for _ in range(4):
add_node(db, adventure, branch, depth, "n")
depth += 1
for fork in range(20):
branch = make_branch(db, adventure, parent=branch, fork_depth=depth - 1)
for _ in range(2):
add_node(db, adventure, branch, depth, "n")
depth += 1
tip_branch = make_branch(db, adventure, parent=branch, fork_depth=depth - 1)
for _ in range(40):
add_node(db, adventure, tip_branch, depth, "n")
depth += 1
adventure.head_branch_id = tip_branch.id
adventure.head_depth = depth - 1
db.commit()
try:
yield db, adventure
finally:
db.close()
Base.metadata.drop_all(bind=engine)
@pytest.fixture()
def emitted_sql():
"""Every statement the connection runs, so a clause can be counted."""
seen = []
def on_execute(conn, cursor, statement, parameters, context, executemany):
seen.append(statement)
event.listen(engine, "before_cursor_execute", on_execute)
try:
yield seen
finally:
event.remove(engine, "before_cursor_execute", on_execute)
def branch_terms(statement: str) -> int:
return statement.count("actions.branch_id =")
def test_a_tail_read_names_one_branch_however_many_forks_there_were(
deeply_forked, emitted_sql
):
db, adventure = deeply_forked
assert len(lineage.path_of(db, adventure)) == 22 # the whole ancestry
emitted_sql.clear()
rows = history.tail(adventure, 32)
assert len(rows) == 32
selects = [s for s in emitted_sql if "FROM actions" in s and branch_terms(s)]
assert selects, "no action read was emitted"
# Clause count is bounded by the context window, not by fork count: the
# newest branch alone holds forty turns, so one entry covers a window of
# thirty-two and the other twenty-one are never named.
assert max(branch_terms(s) for s in selects) == 1
def test_a_window_reaching_past_the_forks_names_only_what_it_needs(
deeply_forked, emitted_sql
):
db, adventure = deeply_forked
emitted_sql.clear()
rows = history.tail(adventure, 41) # 40 on the tip branch, one older
assert len(rows) == 41
selects = [s for s in emitted_sql if "FROM actions" in s and branch_terms(s)]
# Two lineage entries reach 41 deep (40 + 2). The other twenty stay
# unnamed. Every fork past the window costs the query nothing.
assert max(branch_terms(s) for s in selects) == 2
def test_the_estimate_is_arithmetic_not_a_query(deeply_forked):
db, adventure = deeply_forked
path = lineage.path_of(db, adventure)
assert path.prefix_covering(1) == 1
assert path.prefix_covering(40) == 1
assert path.prefix_covering(41) == 2
assert path.prefix_covering(43) == 3
assert path.prefix_covering(10 ** 6) == len(path)
def test_forking_twenty_times_costs_the_same_bytes_as_never_forking(
deeply_forked,
):
"""The design's cost assumption, measured in bytes.
Two stories of the same length, one played straight through and one
forked twenty times, cost the same to read their newest window. The
window is covered by the newest lineage entry either way, and the
ancestry is never named. The forked read pays for one extra row: the
branch it reads the lineage from.
"""
db, forked_adventure = deeply_forked
flat = make_adventure(db, db.get(models.User, forked_adventure.user_id), "Flat")
branch = make_branch(db, flat)
for depth in range(84): # the same 84 nodes the forked story is long
add_node(db, flat, branch, depth, "n")
flat.head_branch_id = branch.id
flat.head_depth = 83
flat_id, forked_id = flat.id, forked_adventure.id
# Commit and release the connection. The meter wraps the connection
# pool's factory, so a connection checked out before the meter attaches
# is never visible to it. Building the fixture is a write path the test
# does not measure, and it is not charged to either scope.
db.commit()
db.expire_all()
meter = dbmeter.Meter()
meter.attach(engine)
try:
with meter.scope("flat"):
assert len(history.tail(db.get(models.Adventure, flat_id), 32)) == 32
flat_bytes = meter.scopes[-1].total.fetched
with meter.scope("forked"):
assert len(history.tail(db.get(models.Adventure, forked_id), 32)) == 32
forked_bytes = meter.scopes[-1].total.fetched
finally:
meter.detach()
assert flat_bytes > 0, "the meter saw nothing; it is measuring the wrong connection"
assert forked_bytes < flat_bytes * 1.25, (
f"reading a 20-fork story cost {forked_bytes:,} B against the "
f"{flat_bytes:,} B an unforked one of the same length cost"
)
def test_a_gap_in_the_story_widens_the_read_rather_than_shortening_it(
deeply_forked, emitted_sql
):
"""The estimate counts depths, and a deleted action leaves a depth with
no row behind it. The read must notice it came up short and widen."""
db, adventure = deeply_forked
victim = (
db.query(models.Action)
.filter(models.Action.branch_id == adventure.head_branch_id)
.order_by(models.Action.depth)
.first()
)
db.delete(victim)
db.commit()
emitted_sql.clear()
rows = history.tail(adventure, 40)
assert len(rows) == 40 # 39 on the tip branch, one borrowed from above
selects = [s for s in emitted_sql if "FROM actions" in s and branch_terms(s)]
assert max(branch_terms(s) for s in selects) == len(lineage.path_of(db, adventure))
backend/tests/test_branch_forking.py
+562
View File
@@ -0,0 +1,562 @@
"""Phase 14 SP5: continuing from a discarded attempt forks a branch.
SP4 made every attempt at a turn a node. While the attempts sit at the
tip, they are leaves and cost nothing: switching between them just moves
the `live` flag. The moment the player continues from an attempt the line
has already moved past, the two futures must coexist. That is a branch.
This file tests the claim the whole design rests on: a fork inserts one
row and moves one row, no matter how large the story behind it is.
Everything before the fork is borrowed, not copied, and the arithmetic
that makes borrowing possible lives in `lineage`.
python -m pytest tests/test_branch_forking.py -v
"""
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from app import auth, limits, models
from app.context import cursors, lineage
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
from app.routers import adventures
from fakes import ScriptedProvider
# `hp` moves freely. `mana` has a cooldown of 2 turns, so an incorrect
# advance shows up as a change the referee should have rejected.
SCHEMA = {
"player": {
"hp": {"min": 0, "max": 100, "initial": 100},
"mana": {"min": 0, "max": 50, "initial": 50, "cooldown": 2},
}
}
GOLD_SCRIPT = """
const modifier = (text) => {
state.gold = (state.gold || 0) + 10;
return { text };
};
modifier(text);
"""
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
user = models.User(is_guest=False, email="fork@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="test-model"))
scenario = models.Scenario(user_id=user.id, title="S", stat_schema=SCHEMA)
setup.add(scenario)
setup.flush()
adv = models.Adventure(
user_id=user.id, title="Cave", scenario_id=scenario.id,
script_state={}, world_state={"player": {"hp": 100, "mana": 50}},
)
setup.add(adv)
setup.flush()
setup.add(models.Action(adventure_id=adv.id, type="start", text="You enter a cave."))
setup.add(models.AdventureScript(
adventure_id=adv.id, position=0, enabled=True, name="Gold", output_js=GOLD_SCRIPT,
))
setup.commit()
adv_id, user_id = adv.id, user.id
setup.close()
ScriptedProvider.replies = ["A reply."]
ScriptedProvider.calls = 0
ScriptedProvider.prompts = []
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
monkeypatch.setattr(auth, "resolve_provider_config", lambda s: auth.ProviderConfig(
"http://fake", "k", "test-model", False))
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
def _current_user(db=Depends(get_db)):
return db.get(models.User, user_id)
app.dependency_overrides[auth.get_current_user] = _current_user
c = TestClient(app)
c.adv_id = adv_id
try:
yield c
finally:
app.dependency_overrides.clear()
adventures.turns._active_turns.clear()
Base.metadata.drop_all(bind=engine)
# ------------------------------------------------------------------ helpers
def _play(client, text="look around", type="do"):
r = client.post(f"/api/adventures/{client.adv_id}/actions",
json={"type": type, "text": text})
assert r.status_code == 200, r.text
def _retry(client):
r = client.post(f"/api/adventures/{client.adv_id}/retry")
assert r.status_code == 200, r.text
def _texts(client) -> list[str]:
return [a["text"] for a in client.get(f"/api/adventures/{client.adv_id}").json()["actions"]]
def _branches(client) -> list[dict]:
r = client.get(f"/api/adventures/{client.adv_id}/branches")
assert r.status_code == 200, r.text
return r.json()
def _fork(client, action_id):
return client.post(f"/api/adventures/{client.adv_id}/actions/{action_id}/fork")
def _state(adv_id):
db = SessionLocal()
try:
adv = db.get(models.Adventure, adv_id)
return adv.script_state, adv.world_state
finally:
db.close()
def _rows(adv_id) -> list[models.Action]:
db = SessionLocal()
try:
return (
db.query(models.Action)
.filter(models.Action.adventure_id == adv_id)
.order_by(models.Action.depth, models.Action.id)
.all()
)
finally:
db.close()
def _divergent_story(client):
"""A story that retried turn 2, continued from the newer take, and left
the older one behind as a leaf.
start > do > [attempt one | ATTEMPT TWO] > do > next turn
Returns the id of the discarded attempt.
"""
ScriptedProvider.replies = ["Attempt one.", "Attempt two.", "Next turn."]
_play(client)
_retry(client)
_play(client, "go deeper")
return [a.id for a in _rows(client.adv_id) if a.type == "ai" and not a.live][0]
# ---------------------------------------------------------------- the fork
def test_a_fork_inserts_one_branch_row_and_copies_no_actions(client):
discarded = _divergent_story(client)
rows_before = [a.id for a in _rows(client.adv_id)]
assert len(_branches(client)) == 1
r = _fork(client, discarded)
assert r.status_code == 200, r.text
branches = _branches(client)
assert len(branches) == 2, "exactly one branch row per divergence built on"
assert [a.id for a in _rows(client.adv_id)] == rows_before, "a fork copies nothing"
forked = [b for b in branches if b["parent_branch_id"] is not None][0]
assert forked["is_head"] is True
assert forked["own_actions"] == 1, "the promoted attempt, and nothing else"
# The fork point is the depth just before the attempt. The code stores
# this value instead of inferring it from where two branches first
# differ, because that inference would guess wrong as soon as an
# attempt repeats its parent's text.
assert forked["fork_depth"] == forked["depth"] - 1
def test_the_lineage_is_capped_at_the_fork_depth(client):
discarded = _divergent_story(client)
_fork(client, discarded)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
branch = lineage.branch_of(db, adventure)
entries = lineage.entries_of(branch)
assert entries[0] == (branch.id, None), "its own nodes, to the tip"
assert entries[1] == (branch.parent_branch_id, branch.fork_depth)
assert len(entries) == 2
finally:
db.close()
def test_both_branches_read_independently(client):
discarded = _divergent_story(client)
parent = _branches(client)[0]["id"]
_fork(client, discarded)
# The fork's story: everything up to the divergence, then the other take.
assert _texts(client) == [
"You enter a cave.", "> You look around.", "Attempt one.",
]
# The branch it left behind is unchanged, including turns after the fork.
r = client.post(f"/api/adventures/{client.adv_id}/branches/{parent}/switch")
assert r.status_code == 200, r.text
assert _texts(client) == [
"You enter a cave.", "> You look around.", "Attempt two.",
"> You go deeper.", "Next turn.",
]
def test_the_parent_keeps_a_live_attempt_where_the_fork_left(client):
"""Promoting the other attempt must not leave the parent with a gap in
its story. A coordinate with no live node is a turn that disappears
from the read."""
discarded = _divergent_story(client)
_fork(client, discarded)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
parent_id = db.query(models.Branch).filter_by(
adventure_id=adventure.id, parent_branch_id=None).one().id
per_coordinate = {}
for row in _rows(client.adv_id):
per_coordinate.setdefault((row.branch_id, row.depth), []).append(row)
for (branch_id, depth), group in per_coordinate.items():
live = [a for a in group if a.live]
assert len(live) == 1, f"branch {branch_id} depth {depth}"
# The parent's turn 2 is now a single take, so the pager no longer
# offers a page through attempts that diverged onto another branch.
parent_turn = per_coordinate[(parent_id, 2)]
assert len(parent_turn) == 1
finally:
db.close()
def test_playing_on_a_fork_continues_that_branchs_depths(client):
"""A depth is a position along this story. Numbering the next node from
the adventure-wide index would leave a gap where the other branch's
turns are, and every windowing estimate would then have to work around
that gap."""
discarded = _divergent_story(client)
_fork(client, discarded)
ScriptedProvider.replies = ["Onward."]
_play(client, "turn back")
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
path = lineage.path_of(db, adventure)
rows = [a for a in _rows(client.adv_id) if path.contains(a)]
assert [a.depth for a in sorted(rows, key=lambda a: a.depth)] == [0, 1, 2, 3, 4]
finally:
db.close()
assert _texts(client)[-1] == "Onward."
# ------------------------------------------------------------- not a fork
def test_forking_at_the_tip_switches_without_making_a_branch(client):
"""Attempts nobody has built on stay leaves. This is what keeps the
lineage a list of divergences instead of a list of every retry."""
ScriptedProvider.replies = ["Attempt one.", "Attempt two."]
_play(client)
_retry(client)
discarded = [a.id for a in _rows(client.adv_id) if a.type == "ai" and not a.live][0]
r = _fork(client, discarded)
assert r.status_code == 200, r.text
assert len(_branches(client)) == 1, "no branch for an attempt at the tip"
assert _texts(client)[-1] == "Attempt one."
def test_forking_the_attempt_already_in_the_story_does_nothing(client):
"""This call is idempotent. A client that has lost track of which take
is live must not create a new branch on every click."""
discarded = _divergent_story(client)
_fork(client, discarded)
promoted = [a.id for a in _rows(client.adv_id) if a.type == "ai" and a.live
and a.branch_id != _branches(client)[0]["id"]][0]
before = _texts(client)
for _ in range(3):
r = _fork(client, promoted)
assert r.status_code == 200, r.text
assert len(_branches(client)) == 2
assert _texts(client) == before
def test_forking_a_turn_that_is_already_the_story_is_a_no_op(client):
ScriptedProvider.replies = ["Only take."]
_play(client)
only = [a.id for a in _rows(client.adv_id) if a.type == "ai"][0]
r = _fork(client, only)
assert r.status_code == 200, r.text
assert len(_branches(client)) == 1
def test_forking_a_live_node_on_another_branch_is_refused(client):
"""A live node off the path belongs to another branch's story, not to a
spare attempt on this one. The refusal names the tool that actually
switches branches. It used to answer "only one take", which was true
of the attempt group but useless here: the caller does not want
another take, it wants the branch this node is on."""
discarded = _divergent_story(client)
_fork(client, discarded)
parent_id = [b for b in _branches(client) if b["parent_branch_id"] is None][0]["id"]
stranded = [a.id for a in _rows(client.adv_id)
if a.branch_id == parent_id and a.depth == 2][0]
r = _fork(client, stranded)
assert r.status_code == 400
assert "another branch" in r.json()["detail"]
# Refusing must leave the tree alone. The bug this guards against is a
# fork that promotes a sibling on the branch it was called against.
assert len(_branches(client)) == 2
# -------------------------------------------------------------- the state
def test_switching_restores_the_script_and_world_state(client):
ScriptedProvider.replies = [
"A scratch.\n```state\n{\"player.hp\": -5}\n```",
"A beating.\n```state\n{\"player.hp\": -40}\n```",
"Onward.",
]
_play(client)
_retry(client)
_play(client, "go deeper")
parent = _branches(client)[0]["id"]
on_parent = _state(client.adv_id)
discarded = [a.id for a in _rows(client.adv_id) if a.type == "ai" and not a.live][0]
_fork(client, discarded)
script_state, world_state = _state(client.adv_id)
assert world_state["player"]["hp"] == 95, "the attempt this branch tells"
assert script_state == {"gold": 10}, "one turn of gold, not three"
client.post(f"/api/adventures/{client.adv_id}/branches/{parent}/switch")
assert _state(client.adv_id) == on_parent
def test_the_cooldown_clock_travels_with_the_branch(client):
"""The world-state clock is a depth, and depths repeat across branches,
so it can only be correct if each branch carries its own. It does,
without extra work: the clock lives inside `_meta.last_changed`, which
is part of the world state a switch restores."""
ScriptedProvider.replies = [
"Drained.\n```state\n{\"player.mana\": -10}\n```",
"Untouched.",
"Onward.",
]
_play(client)
_retry(client)
_play(client, "go deeper")
discarded = [a.id for a in _rows(client.adv_id) if a.type == "ai" and not a.live][0]
on_parent = _state(client.adv_id)[1]
assert on_parent["_meta"]["last_changed"].get("player.mana") is None
_fork(client, discarded)
forked = _state(client.adv_id)[1]
assert forked["player"]["mana"] == 40
assert forked["_meta"]["last_changed"]["player.mana"] == 2
parent = [b for b in _branches(client) if b["parent_branch_id"] is None][0]["id"]
client.post(f"/api/adventures/{client.adv_id}/branches/{parent}/switch")
assert _state(client.adv_id)[1] == on_parent
def test_a_retry_does_not_advance_the_cooldown_clock(client):
"""SP5's one carried-over open item. A retry re-runs the same turn, so
the clock the cooldown rules read must not move. The reused `index`
used to guarantee this; the reused depth guarantees it now."""
ScriptedProvider.replies = [
"Drained.\n```state\n{\"player.mana\": -10}\n```",
"Drained again.\n```state\n{\"player.mana\": -10}\n```",
]
_play(client)
first = _state(client.adv_id)[1]["_meta"]["last_changed"]["player.mana"]
_retry(client)
assert _state(client.adv_id)[1]["_meta"]["last_changed"]["player.mana"] == first
# The second attempt's drain must land, instead of being rejected for a
# cooldown it was never actually subject to.
assert _state(client.adv_id)[1]["player"]["mana"] == 40
# --------------------------------------------------------- derived work
def test_a_memory_on_the_line_left_behind_is_out_of_range_on_the_fork(client):
"""Nothing is moved or removed when a branch forks. The memory attaches
to the coordinate the parent's attempt still occupies, and the lineage
caps the parent one depth short of it. The fork therefore cannot see
this memory, and resummarizes that span from the text it actually
contains."""
from app import tree
discarded = _divergent_story(client)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
winner = db.query(models.Action).filter_by(
adventure_id=adventure.id, type="ai", live=True).order_by(
models.Action.depth).first()
memory = models.Memory(
adventure_id=adventure.id, text="Attempt two happened.",
source_start=0, source_end=winner.depth,
)
tree.attach_memory(memory, winner)
db.add(memory)
cursors.MEMORY.anchor_at(adventure, winner)
db.commit()
finally:
db.close()
_fork(client, discarded)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
# The memory is still there, untouched, because it describes the
# parent's story, which is unchanged.
assert [m.text for m in db.query(models.Memory).all()] == ["Attempt two happened."]
path = lineage.path_of(db, adventure)
visible = db.query(models.Memory).filter(
models.Memory.adventure_id == adventure.id,
path.clause(models.Memory),
).all()
assert visible == [], "a sibling's memory reached this branch"
# The cursor reads one depth short of the memory, so this branch
# treats the block as due again instead of silently marking it read.
assert cursors.MEMORY.depth(db, adventure) == 1
finally:
db.close()
# ------------------------------------------------------------------- undo
def test_undo_stops_at_the_fork(client):
"""Undoing a turn on a fork must never reach into the branch it forked
from. Those turns belong to that branch's story too."""
discarded = _divergent_story(client)
_fork(client, discarded)
rows_before = len(_rows(client.adv_id))
r = client.post(f"/api/adventures/{client.adv_id}/undo")
assert r.status_code == 200, r.text
# The promoted attempt is removed, and the player action before it
# stays, because that action belongs to the parent and the parent
# still has it.
assert len(_rows(client.adv_id)) == rows_before - 1
assert _texts(client) == ["You enter a cave.", "> You look around."]
# Nothing is left of this branch's own turns, so undo must refuse
# instead of removing the parent's turns.
r = client.post(f"/api/adventures/{client.adv_id}/undo")
assert r.status_code == 400
assert "forked from" in r.json()["detail"]
# ----------------------------------------------------------- the tree view
def test_the_branch_list_is_the_tree(client):
discarded = _divergent_story(client)
_fork(client, discarded)
ScriptedProvider.replies = ["Onward."]
_play(client, "turn back")
branches = _branches(client)
root = [b for b in branches if b["parent_branch_id"] is None][0]
forked = [b for b in branches if b["parent_branch_id"] == root["id"]][0]
assert root["fork_depth"] is None and root["depth"] == 4
assert forked["fork_depth"] == 1 and forked["depth"] == 4
assert root["own_actions"] == 5 and forked["own_actions"] == 3
assert forked["is_head"] is True and root["is_head"] is False
def test_fork_agrees_with_the_lineage_computed_by_hand(client):
"""`test_branch_clause.make_branch` has computed a fork's lineage by
hand since SP2, precisely so the fixture could not pass by repeating a
bug in the code under test. SP5 introduces that code, so this test
checks the two against each other instead of letting them drift
apart."""
from tests.test_branch_clause import make_branch
discarded = _divergent_story(client)
_fork(client, discarded)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
real = lineage.branch_of(db, adventure)
parent = db.get(models.Branch, real.parent_branch_id)
by_hand = make_branch(db, adventure, parent=parent, fork_depth=real.fork_depth)
# The two branches share a shape but not an id, so compare only the
# ancestry, which is the arithmetic part rather than the allocated id.
assert lineage.entries_of(real)[1:] == lineage.entries_of(by_hand)[1:]
assert lineage.entries_of(real)[0] == (real.id, None)
finally:
db.close()
def test_a_deep_fork_chain_reads_for_what_one_branch_costs(client):
"""Clause count is bounded by the window, not by fork count. This is
the property the whole lineage cache exists for, now measured through
real forks instead of hand-built rows."""
from tools import dbmeter
ScriptedProvider.replies = ["First take.", "Second take.", "Onward."]
_play(client)
for _ in range(8):
_retry(client)
_play(client, "onward")
discarded = [
a.id for a in _rows(client.adv_id)
if a.type == "ai" and not a.live
]
if discarded:
assert _fork(client, discarded[-1]).status_code == 200
_play(client, "onward")
branches = _branches(client)
assert len(branches) > 4, "the fixture did not actually fork"
meter = dbmeter.Meter()
meter.attach(engine)
try:
with meter.scope("page"):
client.get(f"/api/adventures/{client.adv_id}")
page_bytes = meter.scopes[-1].total.fetched
finally:
meter.detach()
assert page_bytes > 0, "the meter saw nothing"
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
entries = lineage.entries_of(lineage.branch_of(db, adventure))
# The whole ancestry is available to be named.
assert len(entries) == len(branches)
# The windowed read names as few of them as the window needs.
path = lineage.path_of(db, adventure)
assert path.prefix_covering(60) <= len(entries)
finally:
db.close()
def test_switching_to_a_branch_of_another_adventure_is_a_404(client):
"""A branch id names one adventure, so the two ids in the URL must
agree. Otherwise a guessed number reads somebody else's story."""
other = client.post("/api/adventures", json={"title": "Elsewhere"}).json()["id"]
ScriptedProvider.replies = ["Elsewhere."]
r = client.post(f"/api/adventures/{other}/actions", json={"type": "do", "text": "wait"})
assert r.status_code == 200, r.text
stranger = client.get(f"/api/adventures/{other}/branches").json()
assert len(stranger) == 1
assert client.post(
f"/api/adventures/{client.adv_id}/branches/{stranger[0]['id']}/switch"
).status_code == 404
backend/tests/test_branch_management.py
+494
View File
@@ -0,0 +1,494 @@
"""Phase 14 SP7: renaming a branch and deleting one.
SP5 gave the tree a fork and a switch. Neither operation removes anything,
and nothing in the design prunes a tree automatically, so an adventure that
is retried and forked enough grows without a limit. Delete is what limits
that growth. It ships with the same view that first makes a fork reachable,
rather than in a later subphase.
Two rules carry most of this file:
* A name is chosen, so the database stores it. A label is derived, so the
database does not. An unnamed branch keeps NULL, and the client draws the
label from its fork depth. A generated label such as "branch 4" stored in
the column would become wrong the moment branch 3 is deleted.
* Delete must never remove a branch the reader depends on. Refusing to
delete the head is the obvious case. Refusing to delete an ancestor of the
head is the same rule applied one level up: deleting it would leave
`head_branch_id` pointing at a row the cascade removed.
python -m pytest tests/test_branch_management.py -v
"""
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from sqlalchemy import select
from app import auth, limits, models, schemas
from app.context import cursors, lineage
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
from app.routers import adventures
from fakes import ScriptedProvider
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
user = models.User(is_guest=False, email="branches@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="test-model"))
adv = models.Adventure(
user_id=user.id, title="Cave", script_state={}, world_state={},
)
setup.add(adv)
setup.flush()
setup.add(models.Action(
adventure_id=adv.id, type="start", text="You enter a cave."))
setup.commit()
adv_id, user_id = adv.id, user.id
setup.close()
ScriptedProvider.replies = ["Attempt one.", "Attempt two.", "Next turn."]
ScriptedProvider.calls = 0
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
monkeypatch.setattr(auth, "resolve_provider_config", lambda s: auth.ProviderConfig(
"http://fake", "k", "test-model", False))
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
def _current_user(db=Depends(get_db)):
return db.get(models.User, user_id)
app.dependency_overrides[auth.get_current_user] = _current_user
c = TestClient(app)
c.adv_id = adv_id
try:
yield c
finally:
app.dependency_overrides.clear()
adventures.turns._active_turns.clear()
Base.metadata.drop_all(bind=engine)
# ------------------------------------------------------------------ helpers
def _play(client, text="look around"):
r = client.post(f"/api/adventures/{client.adv_id}/actions",
json={"type": "do", "text": text})
assert r.status_code == 200, r.text
def _retry(client):
r = client.post(f"/api/adventures/{client.adv_id}/retry")
assert r.status_code == 200, r.text
def _branches(client) -> list[dict]:
r = client.get(f"/api/adventures/{client.adv_id}/branches")
assert r.status_code == 200, r.text
return r.json()
def _rename(client, branch_id, name):
return client.patch(f"/api/adventures/{client.adv_id}/branches/{branch_id}",
json={"name": name})
def _delete(client, branch_id):
return client.delete(f"/api/adventures/{client.adv_id}/branches/{branch_id}")
def _switch(client, branch_id):
return client.post(f"/api/adventures/{client.adv_id}/branches/{branch_id}/switch")
def _texts(client) -> list[str]:
return [
a["text"]
for a in client.get(f"/api/adventures/{client.adv_id}").json()["actions"]
]
def _discarded_on(adv_id, branch_id=None) -> int:
"""An AI attempt with no turn built on it, optionally restricted to one branch."""
db = SessionLocal()
try:
q = db.query(models.Action).filter(
models.Action.adventure_id == adv_id,
models.Action.type == "ai",
models.Action.live.is_(False),
)
if branch_id is not None:
q = q.filter(models.Action.branch_id == branch_id)
return q.order_by(models.Action.id).first().id
finally:
db.close()
def _forked(client):
"""A story with one fork. Returns (root id, forked id). The fork is head.
start · do · [attempt one | ATTEMPT TWO] · do · next turn
└── forked here
"""
_play(client)
_retry(client)
_play(client, "go deeper")
root = _branches(client)[0]["id"]
r = client.post(
f"/api/adventures/{client.adv_id}/actions/{_discarded_on(client.adv_id)}/fork")
assert r.status_code == 200, r.text
forked = [b for b in _branches(client) if b["id"] != root][0]["id"]
return root, forked
def _counts(adv_id, branch_ids):
db = SessionLocal()
try:
return (
db.query(models.Action)
.filter(models.Action.branch_id.in_(branch_ids)).count(),
db.query(models.Memory)
.filter(models.Memory.branch_id.in_(branch_ids)).count(),
)
finally:
db.close()
# ------------------------------------------------------------------ naming
def test_a_branch_starts_unnamed(client):
"""NULL, not a generated label. The client draws a label from the fork depth.
A name stored here would go stale the moment a branch before it is
deleted and the ordinals shift.
"""
root, forked = _forked(client)
assert [b["name"] for b in _branches(client)] == [None, None]
def test_a_name_is_stored_and_read_back(client):
root, forked = _forked(client)
r = _rename(client, forked, "the cellar")
assert r.status_code == 200, r.text
assert r.json()["name"] == "the cellar"
assert {b["id"]: b["name"] for b in _branches(client)} == {
root: None, forked: "the cellar",
}
def test_a_blank_name_goes_back_to_unnamed(client):
"""A name of spaces is not a name anyone chose.
Storing it would give the client an empty label instead of falling back
to the fork depth. The branch would look nameless and appear broken.
"""
root, forked = _forked(client)
_rename(client, forked, "briefly named")
assert _rename(client, forked, " ").json()["name"] is None
assert _rename(client, forked, None).json()["name"] is None
def test_a_name_longer_than_the_column_is_refused(client):
"""422 here rather than a 500 at INSERT: Postgres enforces VARCHAR(80)."""
root, forked = _forked(client)
assert _rename(client, forked, "x" * (schemas.BRANCH_NAME_MAX + 1)).status_code == 422
assert _rename(client, forked, "x" * schemas.BRANCH_NAME_MAX).status_code == 200
def test_a_rename_hands_back_the_row_the_listing_would_give(client):
"""Renaming a branch does not change how many turns are on it.
`own_actions` was hard-coded to 0 in this response. That bug stayed
invisible because the panel discards the response body and refetches.
Anything that trusted the reply would show a branch that had just lost
its turns.
"""
root, forked = _forked(client)
listed = {b["id"]: b for b in _branches(client)}
renamed = _rename(client, forked, "the cellar").json()
assert renamed["own_actions"] == listed[forked]["own_actions"]
assert renamed["own_actions"] > 0, "the fixture put turns on this branch"
assert renamed["depth"] == listed[forked]["depth"]
assert renamed["is_head"] == listed[forked]["is_head"]
def test_naming_a_branch_of_another_adventure_is_a_404(client):
root, forked = _forked(client)
db = SessionLocal()
try:
other = models.Adventure(
user_id=db.get(models.Adventure, client.adv_id).user_id,
title="Elsewhere", script_state={}, world_state={},
)
db.add(other)
db.commit()
other_id = other.id
finally:
db.close()
r = client.patch(f"/api/adventures/{other_id}/branches/{forked}", json={"name": "x"})
assert r.status_code == 404
# ----------------------------------------------------------------- deleting
def test_the_root_branch_cannot_be_deleted(client):
"""It holds the turns every other branch borrows."""
root, forked = _forked(client)
r = _delete(client, root)
assert r.status_code == 400
assert "adventure" in r.json()["detail"].lower()
assert len(_branches(client)) == 2
def test_the_branch_being_read_cannot_be_deleted(client):
root, forked = _forked(client)
assert [b["is_head"] for b in _branches(client) if b["id"] == forked] == [True]
r = _delete(client, forked)
assert r.status_code == 400
assert "switch" in r.json()["detail"].lower()
def test_an_ancestor_of_the_branch_being_read_cannot_be_deleted(client):
"""Deleting an ancestor of the head is the same mistake as deleting the head.
`parent_branch_id` cascades, so deleting a branch the head was forked
from would delete the head too, and leave `head_branch_id` pointing at
nothing.
"""
root, forked = _forked(client)
# A fork of the fork, so `forked` is an ancestor of the head rather than
# the head itself.
_retry(client)
_play(client, "press on")
nested = _discarded_on(client.adv_id, branch_id=forked)
r = client.post(f"/api/adventures/{client.adv_id}/actions/{nested}/fork")
assert r.status_code == 200, r.text
assert len(_branches(client)) == 3
r = _delete(client, forked)
assert r.status_code == 400
assert "forked from it" in r.json()["detail"]
assert len(_branches(client)) == 3
def test_deleting_a_branch_leaves_the_line_it_forked_from_untouched(client):
root, forked = _forked(client)
_switch(client, root)
kept = _texts(client)
assert _delete(client, forked).status_code == 204
assert [b["id"] for b in _branches(client)] == [root]
assert _texts(client) == kept, "the parent keeps every turn it had"
def test_deleting_a_branch_takes_its_nodes_and_its_descendants(client):
"""One statement deletes the whole subtree, regardless of depth. The
cascade performs the traversal."""
root, forked = _forked(client)
_retry(client)
_play(client, "press on")
nested_attempt = _discarded_on(client.adv_id, branch_id=forked)
client.post(f"/api/adventures/{client.adv_id}/actions/{nested_attempt}/fork")
nested = [b["id"] for b in _branches(client) if b["id"] not in (root, forked)][0]
doomed_actions, _ = _counts(client.adv_id, [forked, nested])
assert doomed_actions > 0
root_actions_before, _ = _counts(client.adv_id, [root])
_switch(client, root)
assert _delete(client, forked).status_code == 204
assert [b["id"] for b in _branches(client)] == [root]
assert _counts(client.adv_id, [forked, nested]) == (0, 0)
assert _counts(client.adv_id, [root])[0] == root_actions_before
def test_deleting_a_branch_clears_a_cursor_that_stood_on_it(client):
"""Harmless on Postgres, a real bug on SQLite.
Postgres never reuses a branch id, so a stale anchor simply never
resolves. SQLite assigns the freed id to the next fork. The anchor then
resolves onto a branch it never saw, and reports a stretch of story as
already summarized. That stretch is then permanently excluded from the
memories.
"""
root, forked = _forked(client)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
cursors.MEMORY.anchor(adventure, forked, 3)
cursors.SUMMARY.anchor(adventure, root, 1)
db.commit()
finally:
db.close()
_switch(client, root)
assert _delete(client, forked).status_code == 204
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
assert cursors.MEMORY.stored(adventure) == (None, cursors.NO_DEPTH)
# The cursor on a branch that still exists is left exactly where it was.
assert cursors.SUMMARY.stored(adventure) == (root, 1)
finally:
db.close()
def test_deleting_an_unknown_branch_is_a_404(client):
root, forked = _forked(client)
assert _delete(client, forked + 9999).status_code == 404
# --------------------------------------------------------- the bank vs a path
def _memories(client) -> list[dict]:
r = client.get(f"/api/adventures/{client.adv_id}/memories")
assert r.status_code == 200, r.text
return r.json()
def _add_memory(client, text):
r = client.post(f"/api/adventures/{client.adv_id}/memories", json={"text": text})
assert r.status_code == 201, r.text
return r.json()["id"]
def test_a_hand_written_memory_is_anchored_where_it_was_written(client):
"""It takes the head, so it is a memory *of a story* rather than of an
adventure. A NULL depth is a coordinate no fork can cap."""
root, forked = _forked(client)
memory_id = _add_memory(client, "Took the other door.")
db = SessionLocal()
try:
memory = db.get(models.Memory, memory_id)
adventure = db.get(models.Adventure, client.adv_id)
assert memory.branch_id == forked, "the branch being read"
assert memory.depth is not None, "never NULL again"
assert memory.depth == adventure.head_depth
finally:
db.close()
def test_the_drawer_shows_the_path_being_read_and_nothing_else(client):
"""The memories a reader can see match the memories the model can retrieve.
An adventure-wide list would include memories from branches this story
never took. Those memories are never retrieved, and a reader could not
distinguish them from the ones actually in play.
"""
root, forked = _forked(client)
on_the_fork = _add_memory(client, "Took the other door.")
_switch(client, root)
on_the_root = _add_memory(client, "Went the long way instead.")
assert {m["id"] for m in _memories(client)} == {on_the_root}, \
"the fork's memory is not on this story"
# Switching to the fork shows its own memory and the root's. A fork
# borrows its ancestors up to the point where it diverged from them.
# The relationship is asymmetric on purpose: the parent never took the
# fork's path.
_switch(client, forked)
listed = {m["id"] for m in _memories(client)}
assert on_the_fork in listed
assert on_the_root not in listed, "written after the fork left this branch"
def test_the_drawer_and_retrieval_agree_on_what_is_visible(client):
"""One predicate decides visibility, so a memory can never be listed but
unretrievable, or the reverse. Two separate definitions of "on this
path" would eventually diverge."""
root, forked = _forked(client)
_add_memory(client, "Took the other door.")
_switch(client, root)
_add_memory(client, "Went the long way instead.")
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
retrievable = {
row[0] for row in db.execute(
select(models.Memory.id).where(
models.Memory.adventure_id == adventure.id,
lineage.path_of(db, adventure).clause(models.Memory),
)
)
}
finally:
db.close()
assert {m["id"] for m in _memories(client)} == retrievable
def test_deleting_a_branch_deletes_the_memories_written_on_it(client):
"""Deleting a branch removes its memories from the database, not just
from view: the cascade deletes the row along with the branch. This
keeps a hidden memory from becoming unreachable in a different way: a
memory you cannot currently see is on a branch you can still switch to,
and deleting that branch deletes the memory permanently."""
root, forked = _forked(client)
doomed = _add_memory(client, "Took the other door.")
_switch(client, root)
db = SessionLocal()
try:
assert db.get(models.Memory, doomed) is not None, "still on its own branch"
finally:
db.close()
assert _delete(client, forked).status_code == 204
db = SessionLocal()
try:
assert db.get(models.Memory, doomed) is None, "gone with the branch"
finally:
db.close()
# ------------------------------------------------------------------- backup
def test_a_bundle_carries_the_name_a_player_chose(client):
"""A name is a decision, so the export includes it. This is the rule the
v2 format is built on.
`lineage` and the head depth stay out of the export because the
importer can compute them from what the file already carries. A name
is computed from nothing, so the export must carry it.
"""
root, forked = _forked(client)
_rename(client, root, "the long way")
_rename(client, forked, "the cellar")
exported = client.get(f"/api/adventures/{client.adv_id}/export").json()
assert [b.get("name") for b in exported["branches"]] == ["the long way", "the cellar"]
r = client.post("/api/adventures/import", json=exported)
assert r.status_code == 201, r.text
restored = client.get(f"/api/adventures/{r.json()['id']}/branches").json()
assert [b["name"] for b in restored] == ["the long way", "the cellar"]
def test_an_unnamed_tree_exports_no_name_key(client):
"""Unchanged from the file SP6 wrote, for a tree nobody has named."""
_forked(client)
exported = client.get(f"/api/adventures/{client.adv_id}/export").json()
assert all("name" not in b for b in exported["branches"])
def test_a_bundle_naming_a_branch_with_a_number_is_refused(client):
"""400 from the planner, not a database error three branches in."""
_forked(client)
exported = client.get(f"/api/adventures/{client.adv_id}/export").json()
exported["branches"][1]["name"] = 7
r = client.post("/api/adventures/import", json=exported)
assert r.status_code == 400
assert "not text" in r.json()["detail"]
backend/tests/test_bundle_v2.py
+635
View File
@@ -0,0 +1,635 @@
"""Phase 14 SP6: the export bundle carries the tree.
A bundle is the only part of this phase a migration can never reach: the
file already exists on somebody's disk. So there are two formats, and the
two halves of this file check different properties.
v2 must be lossless for a story that forked. v1 could not be, because it
stored one list for two stories, interleaved by `index`, which read back
as a mangled story. Losslessness here means the tree: every branch, the
fork point it left on its parent, which attempt at each turn is the
story, and what each node left behind. That last item is what a branch
switch restores, and a tree nobody can switch inside is not the tree
that was exported.
v1 must still import, because a backup that stops importing is not a
backup.
Both formats follow one rule: a bundle carries what was chosen and never
what is derived. The lineage, the head depth, the legacy `index`, and the
variant ordinals are all rebuilt on the way in, so a hand-edited file
cannot disagree with itself. The tests that matter most here hand the
importer a file that does disagree with itself.
python -m pytest tests/test_bundle_v2.py -v
"""
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from app import auth, bundle, limits, models
from app.context import lineage
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
from app.routers import adventures
from fakes import ScriptedProvider
SCHEMA = {"player": {"hp": {"min": 0, "max": 100, "initial": 100}}}
# Ten gold a turn, so the stored gold total tells how many turns the
# story behind it played. This makes an after-snapshot visible from outside.
GOLD_SCRIPT = """
const modifier = (text) => {
state.gold = (state.gold || 0) + 10;
return { text };
};
modifier(text);
"""
OPENING = "You enter a cave."
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
user = models.User(is_guest=False, email="bundle@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="test-model"))
scenario = models.Scenario(user_id=user.id, title="S", stat_schema=SCHEMA)
setup.add(scenario)
setup.flush()
adv = models.Adventure(
user_id=user.id, title="Cave", scenario_id=scenario.id,
script_state={}, world_state={"player": {"hp": 100}},
)
setup.add(adv)
setup.flush()
setup.add(models.Action(adventure_id=adv.id, type="start", text=OPENING))
setup.add(models.AdventureScript(
adventure_id=adv.id, position=0, enabled=True, name="Gold", output_js=GOLD_SCRIPT,
))
setup.commit()
adv_id, user_id = adv.id, user.id
setup.close()
ScriptedProvider.replies = ["A reply."]
ScriptedProvider.calls = 0
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
monkeypatch.setattr(auth, "resolve_provider_config", lambda s: auth.ProviderConfig(
"http://fake", "k", "test-model", False))
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
def _current_user(db=Depends(get_db)):
return db.get(models.User, user_id)
app.dependency_overrides[auth.get_current_user] = _current_user
c = TestClient(app)
c.adv_id = adv_id
try:
yield c
finally:
app.dependency_overrides.clear()
adventures.turns._active_turns.clear()
Base.metadata.drop_all(bind=engine)
# ------------------------------------------------------------------ helpers
def _play(client, adv_id, text="look around", type="do"):
r = client.post(f"/api/adventures/{adv_id}/actions", json={"type": type, "text": text})
assert r.status_code == 200, r.text
def _retry(client, adv_id):
assert client.post(f"/api/adventures/{adv_id}/retry").status_code == 200
def _export(client, adv_id) -> dict:
r = client.get(f"/api/adventures/{adv_id}/export")
assert r.status_code == 200, r.text
return r.json()
def _import(client, payload):
return client.post("/api/adventures/import", json=payload)
def _imported(client, payload) -> int:
r = _import(client, payload)
assert r.status_code == 201, r.text
return r.json()["id"]
def _branches(client, adv_id) -> list[dict]:
r = client.get(f"/api/adventures/{adv_id}/branches")
assert r.status_code == 200, r.text
return r.json()
def _switch(client, adv_id, branch_id):
r = client.post(f"/api/adventures/{adv_id}/branches/{branch_id}/switch")
assert r.status_code == 200, r.text
def _texts(client, adv_id) -> list[str]:
return [a["text"] for a in client.get(f"/api/adventures/{adv_id}").json()["actions"]]
def _every_branch_story(client, adv_id) -> list[list[str]]:
"""What each branch tells, in branch order. This is the whole tree as text."""
stories = []
for branch in _branches(client, adv_id):
_switch(client, adv_id, branch["id"])
stories.append(_texts(client, adv_id))
return stories
def _adventure_count() -> int:
db = SessionLocal()
try:
return db.query(models.Adventure).count()
finally:
db.close()
def _rows(adv_id) -> list[models.Action]:
db = SessionLocal()
try:
return (
db.query(models.Action)
.filter(models.Action.adventure_id == adv_id)
.order_by(models.Action.branch_id, models.Action.depth,
models.Action.id)
.all()
)
finally:
db.close()
def _branch_rows(adv_id) -> list[models.Branch]:
db = SessionLocal()
try:
return (
db.query(models.Branch)
.filter(models.Branch.adventure_id == adv_id)
.order_by(models.Branch.id)
.all()
)
finally:
db.close()
def _script_state(adv_id) -> dict:
db = SessionLocal()
try:
return db.get(models.Adventure, adv_id).script_state
finally:
db.close()
def _forked_story(client) -> int:
"""A story that went two ways and stayed both.
root: start > do > [attempt two] > do > next turn
fork: [ATTEMPT ONE] > do > elsewhere
The discarded attempt is the one this function promotes, because a
fork moves the attempt being left for and leaves the line it came
from untouched.
Returns the adventure id, with the head on the fork.
"""
adv_id = client.adv_id
ScriptedProvider.replies = ["Attempt one.", "Attempt two.", "Next turn.", "Elsewhere."]
_play(client, adv_id)
_retry(client, adv_id)
_play(client, adv_id, "go deeper")
discarded = [a.id for a in _rows(adv_id) if a.type == "ai" and not a.live][0]
assert client.post(f"/api/adventures/{adv_id}/actions/{discarded}/fork").status_code == 200
_play(client, adv_id, "go sideways")
return adv_id
# ------------------------------------------------------- the round trip (v2)
def test_a_forked_story_survives_the_round_trip(client):
"""The main claim: both futures come back, and both are still readable.
This is the thing v1 could not do. The check is not "the same rows",
since the ids are new, but "the same stories", read the way a player
reads them: by switching to a branch and looking at what it says.
"""
original = _forked_story(client)
before = _every_branch_story(client, original)
assert len(before) == 2, "the fixture forked"
assert before[0] != before[1], "and the two branches tell different stories"
copy = _imported(client, _export(client, original))
assert copy != original
assert _every_branch_story(client, copy) == before
def test_the_fork_point_comes_back_where_it_was_put(client):
"""`fork_depth` is stored, never inferred, including through a file.
Inferring it from where two branches' nodes first differ would guess
at how the story was played, and that guess fails as soon as an
attempt happens to repeat its parent's text.
"""
original = _forked_story(client)
before = [(b["parent_branch_id"] is None, b["fork_depth"]) for b in _branches(client, original)]
copy = _imported(client, _export(client, original))
assert [(b["parent_branch_id"] is None, b["fork_depth"])
for b in _branches(client, copy)] == before
def test_the_head_comes_back_on_the_branch_it_was_left_on(client):
original = _forked_story(client)
head_before = [b["is_head"] for b in _branches(client, original)]
assert head_before == [False, True], "the fixture left the head on the fork"
copy = _imported(client, _export(client, original))
assert [b["is_head"] for b in _branches(client, copy)] == head_before
# The tip it sits at is derived from the nodes that arrived, not
# read from the file. The bundle never states how deep a branch goes.
assert _texts(client, copy) == _texts(client, original)
def test_a_switch_in_the_copy_restores_what_that_branch_left_behind(client):
"""This test justifies why the bundle carries after-snapshots.
The gold script adds ten a turn, so the stored gold total counts the
turns behind it. A bundle that carried the actions but not the
outcomes would import a tree that reads correctly but switches to the
wrong state.
"""
original = _forked_story(client)
# Play one more turn on the fork, so the two tips end up at
# genuinely different totals. Turn for turn, both branches earn the
# same gold, so a switch that restored nothing would still look right.
ScriptedProvider.replies = ["Further still."]
_play(client, original, "press on")
per_branch = []
for branch in _branches(client, original):
_switch(client, original, branch["id"])
per_branch.append(_script_state(original).get("gold"))
assert len(set(per_branch)) == len(per_branch), "the tips are at different totals"
copy = _imported(client, _export(client, original))
restored = []
for branch in _branches(client, copy):
_switch(client, copy, branch["id"])
restored.append(_script_state(copy).get("gold"))
assert restored == per_branch
def test_a_memory_comes_back_on_the_node_it_hangs_off(client):
"""Derived work is addressed by coordinate, so the coordinate is carried."""
original = _forked_story(client)
tip = [a for a in _rows(original) if a.live][-1]
db = SessionLocal()
try:
db.add(models.Memory(
adventure_id=original, text="They met a goblin.",
source_start=0, source_end=tip.depth,
branch_id=tip.branch_id, depth=tip.depth,
))
db.commit()
finally:
db.close()
exported = _export(client, original)
assert [(m["branch"], m["depth"]) for m in exported["memories"]] == [(1, tip.depth)]
copy = _imported(client, exported)
db = SessionLocal()
try:
memories = db.query(models.Memory).filter(
models.Memory.adventure_id == copy).all()
branches = [b.id for b in _branch_rows(copy)]
assert [(branches.index(m.branch_id), m.depth) for m in memories] == [(1, tip.depth)]
finally:
db.close()
# --------------------------------------------------- what is not in the file
def test_the_lineage_is_rebuilt_rather_than_carried(client):
"""A cache of `parent` plus `fork_depth` is not a second thing to ship.
The file states where each branch forked. The ancestry that makes the
fork readable is computed from that value on the way in, capped at
the fork exactly as `tree.fork` caps it. Shipping the cache too would
put two sources of truth for one fact in a file anybody can hand-edit.
"""
original = _forked_story(client)
exported = _export(client, original)
assert all("lineage" not in b for b in exported["branches"])
root, forked = _branch_rows(_imported(client, exported))
assert root.lineage == [[root.id, None]]
assert forked.lineage == [[forked.id, None], [root.id, forked.fork_depth]]
# This is the arithmetic the reader depends on. The parent is capped
# one depth short of the attempt that was promoted, so the fork
# cannot see it.
assert lineage.entries_of(forked) == [(forked.id, None), (root.id, forked.fork_depth)]
def test_two_branches_each_keep_their_own_node_at_one_depth(client):
"""A depth describes a path, not the adventure.
The fork and the branch it left both hold a turn at depth 2, and they are
not the same turn. The import used to issue a global `index` per turn as
well, so that every coordinate had a number nothing else held. SP8 dropped
that column, so the coordinate is all there is, and it has to survive the
round trip on its own.
"""
copy = _imported(client, _export(client, _forked_story(client)))
rows = _rows(copy)
at_depth_2 = [r for r in rows if r.depth == 2]
assert len({r.branch_id for r in at_depth_2}) == 2
# ------------------------------------------------------- a file that is wrong
def test_a_node_naming_a_branch_the_file_does_not_list_is_refused(client):
"""The import must refuse this file rather than half-apply it. A tree
missing a branch is a story that silently stops, which is the failure
this whole phase exists to prevent."""
payload = _export(client, _forked_story(client))
payload["branches"] = payload["branches"][:1]
before = _adventure_count()
r = _import(client, payload)
assert r.status_code == 400, r.text
assert "branch" in r.json()["detail"].lower()
assert _adventure_count() == before, "nothing was created"
def test_a_branch_forking_from_one_listed_after_it_is_refused(client):
"""The ordering rule guarantees no cycles at the cost of one
comparison. Without it, a cycle would produce an import that never
returns instead of one that fails cleanly."""
payload = _export(client, _forked_story(client))
payload["branches"] = [{"parent": 1, "forkDepth": 0}, {"parent": None, "forkDepth": None}]
before = _adventure_count()
r = _import(client, payload)
assert r.status_code == 400, r.text
assert _adventure_count() == before
def test_a_fork_with_no_depth_is_refused(client):
payload = _export(client, _forked_story(client))
payload["branches"][1].pop("forkDepth")
before = _adventure_count()
r = _import(client, payload)
assert r.status_code == 400, r.text
assert "depth" in r.json()["detail"].lower()
assert _adventure_count() == before
def test_more_branches_than_the_cap_is_refused(client, monkeypatch):
monkeypatch.setattr(auth, "MULTI_USER", True)
payload = {
"format": bundle.FORMAT, "title": "Too many",
"branches": [{"parent": None, "forkDepth": None}]
* (limits.MAX_BRANCHES_PER_ADVENTURE + 1),
"actions": [],
}
before = _adventure_count()
r = _import(client, payload)
assert r.status_code == 409, r.text
assert _adventure_count() == before
def test_a_turn_the_file_gives_no_live_attempt_still_tells_one(client):
"""A coordinate with nothing live is a turn no read can see.
The file is allowed to be wrong about this, since it is a text file
someone can edit. The import picks the first attempt instead of
importing a story with a gap.
"""
payload = _export(client, _forked_story(client))
for node in payload["actions"]:
node["live"] = False
copy = _imported(client, payload)
assert _texts(client, copy), "the story is readable"
live = [(r.branch_id, r.depth) for r in _rows(copy) if r.live]
assert len(live) == len(set(live)), "exactly one live attempt per coordinate"
assert len(live) == len({(r.branch_id, r.depth) for r in _rows(copy)})
# ------------------------------------------------------------- the v1 reader
def test_a_v1_bundle_still_imports(client):
"""The v1 reader must remain even after the v1 writer is gone, because
those files already exist and are saved."""
payload = {
"format": bundle.LEGACY_FORMAT,
"title": "Old backup",
"memoryCursor": 0, "summaryCursor": 0,
"actions": [
{"index": 0, "type": "start", "text": OPENING},
{"index": 1, "type": "do", "text": "> You go north."},
{
"index": 2, "type": "ai", "text": "Two.",
"variants": [{"text": "One."}, {"text": "Two."}],
"variantIndex": 1,
},
],
}
copy = _imported(client, payload)
assert _texts(client, copy) == [OPENING, "> You go north.", "Two."]
# The import produces one branch, and the `variants` array splits
# back into the sibling group it always described.
assert len(_branches(client, copy)) == 1
ai = [r for r in _rows(copy) if r.type == "ai"]
assert [(r.text, r.live) for r in ai] == [("One.", False), ("Two.", True)]
assert len({(r.branch_id, r.depth) for r in ai}) == 1
def test_a_v1_bundle_with_a_cursor_lands_it_on_a_node(client):
"""v1 counts covered actions. The tree anchors them to a node instead.
The translation needs the nodes to exist first, so it runs after they
are written."""
payload = {
"format": bundle.LEGACY_FORMAT, "title": "Old backup",
"memoryCursor": 2, "summaryCursor": 2,
"actions": [
{"index": 0, "type": "start", "text": OPENING},
{"index": 1, "type": "story", "text": "A corridor."},
{"index": 2, "type": "story", "text": "A door."},
],
}
copy = _imported(client, payload)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, copy)
assert adventure.memory_cursor_branch_id is not None
assert adventure.memory_cursor_depth == 1, "the second story action"
finally:
db.close()
def test_a_v2_bundle_brings_its_anchors_back(client):
"""The other direction: v2 carries the anchor directly."""
original = _forked_story(client)
tip = [a for a in _rows(original) if a.live][-1]
db = SessionLocal()
try:
adventure = db.get(models.Adventure, original)
adventure.memory_cursor_branch_id = tip.branch_id
adventure.memory_cursor_depth = tip.depth
db.commit()
finally:
db.close()
exported = _export(client, original)
assert exported["memoryCursor"] == {"branch": 1, "depth": tip.depth}
copy = _imported(client, exported)
db = SessionLocal()
try:
adventure = db.get(models.Adventure, copy)
branches = [b.id for b in _branch_rows(copy)]
assert branches.index(adventure.memory_cursor_branch_id) == 1
assert adventure.memory_cursor_depth == tip.depth
finally:
db.close()
def test_a_v1_memory_that_summarises_nothing_lands_on_the_root(client):
"""The import must answer the question migration 62 already answered.
A v1 file has no depths, and a memory the player typed has no
`sourceEnd` to derive one from. It used to come back with a NULL
depth, the exact state SP7 removed from the schema.
`Path._entry_clause` compares `depth <= max_depth`, and a NULL fails
that comparison. The memory would read fine until the imported
adventure forked, then vanish from the new branch.
"""
payload = {
"format": bundle.LEGACY_FORMAT, "title": "Old backup",
"memoryCursor": 0, "summaryCursor": 0,
"actions": [
{"index": 0, "type": "start", "text": OPENING},
{"index": 1, "type": "story", "text": "A corridor."},
],
"memories": [
{"text": "Kira is the innkeeper's daughter"}, # typed
{"text": "The corridor, summarised", "sourceStart": 1, "sourceEnd": 1},
],
}
copy = _imported(client, payload)
db = SessionLocal()
try:
rows = {m.text: m for m in db.query(models.Memory)
.filter(models.Memory.adventure_id == copy).all()}
assert rows["Kira is the innkeeper's daughter"].depth == 0, (
"a typed memory anchors at the root, which every branch can see"
)
assert rows["The corridor, summarised"].depth == 1, "derived from its range"
assert all(m.depth is not None for m in rows.values())
assert all(m.branch_id is not None for m in rows.values())
finally:
db.close()
def test_the_action_cap_counts_the_rows_a_v1_file_expands_into(client, monkeypatch):
"""The cap must count what gets written, not what the file lists.
A v1 turn carries its retries in a `variants` array, and SP4 made
every attempt a row, so one entry can expand into ten. Counting
entries instead of rows would let a file inside the cap write a
multiple of it. The body-size limit does not help here: the text is
tiny, and the row count is the actual cost.
"""
monkeypatch.setattr(auth, "MULTI_USER", True)
monkeypatch.setattr(limits, "MAX_ACTIONS_PER_ADVENTURE", 6)
monkeypatch.setattr(limits, "_BUNDLE_LIST_CAPS",
{**limits._BUNDLE_LIST_CAPS, "actions": 6})
payload = {
"format": bundle.LEGACY_FORMAT, "title": "Small file, many rows",
"memoryCursor": 0, "summaryCursor": 0,
"actions": [{"index": 0, "type": "start", "text": OPENING}] + [
{
"index": i, "type": "ai", "text": "Take four.",
"variants": [{"text": f"Take {n}."} for n in range(4)],
"variantIndex": 3,
}
for i in range(1, 4)
],
}
assert len(payload["actions"]) <= 6, "the file itself is inside the cap"
before = _adventure_count()
r = _import(client, payload)
assert r.status_code == 409, r.text
assert _adventure_count() == before, "and nothing was written"
def test_an_unknown_format_is_refused(client):
r = _import(client, {"format": "ai-dnd-adventure-v3", "title": "From the future"})
assert r.status_code == 400, r.text
assert bundle.FORMAT in r.json()["detail"]
# ------------------------------------------------------- the persona (Phase 18)
def test_the_persona_survives_the_round_trip(client):
adv_id = client.adv_id
assert client.patch(f"/api/adventures/{adv_id}", json={
"persona_name": "Kaelen",
"persona_pronouns": "he/him",
"persona_desc": "A half-elf ranger, exiled from the northern holds.",
}).status_code == 200
payload = _export(client, adv_id)
assert payload["persona"] == {
"name": "Kaelen",
"pronouns": "he/him",
"desc": "A half-elf ranger, exiled from the northern holds.",
}
copy_id = _imported(client, payload)
r = client.get(f"/api/adventures/{copy_id}")
assert r.status_code == 200, r.text
copy = r.json()
assert copy["persona_name"] == "Kaelen"
assert copy["persona_pronouns"] == "he/him"
assert copy["persona_desc"].startswith("A half-elf ranger")
def test_a_bundle_written_before_personas_still_imports(client):
"""The key is read with `.get`, so a v2 file from before Phase 18 lands
with an empty persona rather than failing. No FORMAT bump was needed."""
payload = _export(client, client.adv_id)
del payload["persona"]
copy = client.get(f"/api/adventures/{_imported(client, payload)}").json()
assert copy["persona_name"] == ""
assert copy["persona_pronouns"] == ""
assert copy["persona_desc"] == ""
def test_an_over_long_persona_in_a_raw_bundle_is_truncated(client):
"""A raw-dict import bypasses the schemas, so the widths the columns
declare are enforced here instead. Postgres would reject the INSERT."""
payload = _export(client, client.adv_id)
payload["persona"] = {"name": "K" * 500, "pronouns": "p" * 500, "desc": "d"}
copy = client.get(f"/api/adventures/{_imported(client, payload)}").json()
assert len(copy["persona_name"]) == 80
assert len(copy["persona_pronouns"]) == 40
backend/tests/test_cast_brief.py
+234
View File
@@ -0,0 +1,234 @@
"""Phase 18b: what the summarizer is told about who the characters are.
Before this, `_create_due_memories` sent six actions of second-person prose and
nothing else. The model had no way to know who "you" was or who "she" was, so
neither did any memory it wrote, and the summary built from those memories
inherited the problem.
The rules this file holds in place:
* The brief carries fixed descriptions only. A live stat in it would make the
same event, summarized twice, come out framed differently — the exact fault
the change exists to remove.
* Places and items are not topped up into the roster. Only characters are.
* An adventure with no persona and no cards still summarizes, with the prompt
it had before.
python -m pytest tests/test_cast_brief.py -v
"""
import asyncio
import pytest
from app import memorybank, models
from app.database import Base, SessionLocal, engine
GWEN = "A loyal ranger and the player's ally. Quick with a bow, fiercely protective."
LEADER = "Scarred, patient, and the one holding the strongbox key."
TAVERN = "A tavern three days south of the camp."
class StubSummarizer:
"""Records every (system, user) pair handed to the summarizer."""
def __init__(self):
self.calls: list[tuple[str, str]] = []
async def complete(self, system, user, **kwargs):
self.calls.append((system, user))
return f"Memory {len(self.calls)}."
@pytest.fixture()
def db():
Base.metadata.create_all(bind=engine)
session = SessionLocal()
try:
yield session
finally:
session.close()
Base.metadata.drop_all(bind=engine)
def make_adventure(db, *, actions=0, persona=True, cards=True, memory=True):
user = models.User(is_guest=False, email="cast@example.com")
db.add(user)
db.flush()
db.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="test-model"))
adventure = models.Adventure(
user_id=user.id, title="Camp", script_state={}, auto_summarize=True,
memory=("The player and Gwen are raiding a bandit camp to recover a "
"stolen strongbox.") if memory else "",
persona_name="Kaelen" if persona else "",
persona_pronouns="he/him" if persona else "",
persona_desc="A half-elf ranger, exiled from the northern holds." if persona else "",
)
db.add(adventure)
db.flush()
if cards:
db.add(models.StoryCard(adventure_id=adventure.id, name="Gwen",
keys="Gwen, ranger, her", type="character", entry=GWEN))
db.add(models.StoryCard(adventure_id=adventure.id, name="Bandit Leader",
keys="Bandit Leader, leader", type="character", entry=LEADER))
db.add(models.StoryCard(adventure_id=adventure.id, name="The Rusted Tankard",
keys="tankard, tavern", type="location", entry=TAVERN))
for i in range(actions):
db.add(models.Action(adventure_id=adventure.id,
type="ai" if i % 2 else "do",
text=f"You walk on. Action {i}."))
db.commit()
db.refresh(adventure)
return adventure
# ------------------------------------------------------------ the brief itself
def test_the_protagonist_is_named_and_marked_as_such(db):
adventure = make_adventure(db)
brief = memorybank.cast_brief(adventure, "You walk into the camp.")
assert "- Kaelen (he/him) — the protagonist." in brief
assert "A half-elf ranger" in brief
def test_a_named_character_in_the_text_is_in_the_roster(db):
adventure = make_adventure(db)
brief = memorybank.cast_brief(adventure, "Gwen checks her bowstring.")
assert "- Gwen — " in brief and "Quick with a bow" in brief
def test_a_character_referred_to_only_by_pronoun_is_still_in_the_roster(db):
"""The block that most needs a cast is the one written in bare pronouns.
Keyword matching alone finds nothing here, so the roster is topped up."""
adventure = make_adventure(db)
brief = memorybank.cast_brief(adventure, "She grabs your arm before you step through.")
assert "- Gwen — " in brief
assert "- Bandit Leader — " in brief
def test_places_are_not_topped_up(db):
"""An unmentioned tavern is not who "she" was."""
adventure = make_adventure(db)
brief = memorybank.cast_brief(adventure, "She grabs your arm.")
assert "Rusted Tankard" not in brief
def test_a_place_that_is_mentioned_does_appear(db):
"""Topping up is limited to characters. Matching is not."""
adventure = make_adventure(db)
brief = memorybank.cast_brief(adventure, "You push into the tavern, breathless.")
assert "The Rusted Tankard" in brief
def test_the_setting_is_the_plot_essentials(db):
adventure = make_adventure(db)
brief = memorybank.cast_brief(adventure, "You walk on.")
assert "Setting:\nThe player and Gwen are raiding a bandit camp" in brief
def test_no_persona_and_no_cards_gives_no_brief(db):
adventure = make_adventure(db, persona=False, cards=False, memory=False)
assert memorybank.cast_brief(adventure, "You walk on.") == ""
def test_a_persona_alone_is_enough_for_a_brief(db):
adventure = make_adventure(db, cards=False, memory=False)
brief = memorybank.cast_brief(adventure, "You walk on.")
assert brief.startswith("Cast:\n- Kaelen (he/him) — the protagonist.")
def test_an_unnamed_protagonist_is_called_the_player(db):
adventure = make_adventure(db, cards=False, memory=False)
adventure.persona_name = ""
adventure.persona_pronouns = ""
db.commit()
assert "- The player — the protagonist." in memorybank.cast_brief(adventure, "x")
def test_the_roster_is_capped(db, monkeypatch):
monkeypatch.setattr(memorybank, "MAX_CAST_MEMBERS", 2)
adventure = make_adventure(db)
brief = memorybank.cast_brief(adventure, "She grabs your arm.")
assert len([ln for ln in brief.splitlines() if ln.startswith("- ")]) == 2
def test_a_long_description_is_trimmed(db):
adventure = make_adventure(db, cards=False, memory=False)
adventure.persona_desc = "word " * 400
db.commit()
brief = memorybank.cast_brief(adventure, "x")
assert "…" in brief
assert len(brief) < 600
def test_a_character_is_never_listed_twice(db):
"""The persona and a card can carry the same name."""
adventure = make_adventure(db, cards=False, memory=False)
db.add(models.StoryCard(adventure_id=adventure.id, name="Kaelen",
keys="Kaelen", type="character", entry="Also Kaelen."))
db.commit()
db.refresh(adventure)
brief = memorybank.cast_brief(adventure, "Kaelen walks on.")
assert brief.lower().count("kaelen") == 1
def test_the_brief_holds_no_live_values(db):
"""Fixed descriptions only. A stat here would frame the same event two ways
depending on when it happened to be summarized."""
adventure = make_adventure(db)
brief = memorybank.cast_brief(adventure, "Gwen checks her bowstring.")
for live in ("hp", "trust", "/100", "wary", "healthy"):
assert live not in brief.lower()
# ------------------------------------------------- what actually gets sent
def _run_memory_pass(db, adventure, monkeypatch, stub):
monkeypatch.setattr(memorybank, "MEMORY_START", 0)
monkeypatch.setattr(memorybank, "MEMORY_INTERVAL", 4)
monkeypatch.setattr(memorybank, "MAX_MEMORIES_PER_RUN", 1)
monkeypatch.setattr(memorybank, "summary_provider", lambda s: stub)
settings = db.query(models.Settings).first()
asyncio.run(memorybank._create_due_memories(adventure, settings, db))
assert stub.calls, "the summarizer was never called"
return stub.calls[0]
def test_the_memory_prompt_carries_the_brief_above_the_excerpt(db, monkeypatch):
adventure = make_adventure(db, actions=6)
system, user = _run_memory_pass(db, adventure, monkeypatch, StubSummarizer())
assert user.index("Cast:") < user.index("Story excerpt:")
assert "Kaelen (he/him) — the protagonist" in user
assert "Setting:" in user
def test_the_memory_prompt_demands_third_person_and_names(db, monkeypatch):
adventure = make_adventure(db, actions=6)
system, _ = _run_memory_pass(db, adventure, monkeypatch, StubSummarizer())
assert "third person" in system
assert 'never as "you"' in system
def test_an_adventure_with_nothing_to_say_sends_the_old_prompt(db, monkeypatch):
"""No persona, no cards, no plot essentials: the user message is exactly
what it was before this change, with no stray blank lines."""
adventure = make_adventure(db, actions=6, persona=False, cards=False, memory=False)
_, user = _run_memory_pass(db, adventure, monkeypatch, StubSummarizer())
assert user.startswith("Story excerpt:")
assert "Cast:" not in user
def test_the_summary_prompt_carries_the_brief_too(db, monkeypatch):
"""It builds from the memories, so it inherits their framing — but the
fallback hands it raw second-person story text, which needs the brief."""
adventure = make_adventure(db, actions=20)
stub = StubSummarizer()
monkeypatch.setattr(memorybank, "SUMMARY_INTERVAL", 1)
monkeypatch.setattr(memorybank, "summary_provider", lambda s: stub)
settings = db.query(models.Settings).first()
asyncio.run(memorybank._update_story_summary(adventure, settings, db))
assert stub.calls, "the summarizer was never called"
system, user = stub.calls[0]
assert user.index("Cast:") < user.index("Current story summary:")
assert "Kaelen (he/him) — the protagonist" in user
assert "third person" in system
assert adventure.story_summary == "Memory 1."
backend/tests/test_change_visibility.py
+261
View File
@@ -0,0 +1,261 @@
"""Tests for the changes the engine refuses, and for naming a milestone.
A refused change used to leave no trace a player could see. The turn summary
read `world_delta["applied"]` alone, so a value the model pushed past its
ceiling came back with a delta of 0 and rendered as an ordinary chip. The
engine had recorded the refusal and nothing showed it.
The milestone half is the same shape. `apply_delta` matches a milestone by its
schema id, and the context named goals by description only, so the model had no
way to learn the id it was being asked to send.
python -m pytest tests/test_change_visibility.py -v
"""
import json
import pathlib
from app import models
from app import worldstate as w
from app.routers.adventures import world_delta_of
SEED = pathlib.Path(__file__).resolve().parents[1] / "app" / "seed_data"
SCHEMA = {
"world": {"day": {"type": "counter", "min": 1, "initial": 1}},
"player": {
"hp": {"min": 0, "max": 100, "initial": 100, "max_delta_per_turn": 30},
# Starts at its own ceiling and only ever falls, which is the shape that
# turns a wrong-signed delta into a change of nothing.
"arrows": {"min": 0, "max": 6, "initial": 6, "max_delta_per_turn": 1},
},
"npcs": {"gwen": {"name": "Gwen", "stats": {"trust": {"min": -100, "max": 100, "initial": 0}}}},
"flags": {"has_key": {"desc": "Holds the key", "initial": False}},
"milestones": {
"rescue_gwen": {"desc": "Rescue Gwen"},
"escape_keep": {"desc": "Escape the keep"},
},
}
def action(delta, index=1):
"""Returns an unsaved Action carrying the report `apply_delta` produced.
The column is filled through `world_delta_of`, the same function the turn
endpoint uses, rather than by assembling the dict here. An earlier version
of this helper built the shape by hand with every report list present. That
hid a live bug: `world_delta_of` stored `applied` alone, so no refusal ever
reached a chip or the next prompt while all of these tests passed.
"""
ws, report = w.apply_delta(w.instantiate(SCHEMA), SCHEMA, delta, index)
snapshot = {"world_state": {"delta": delta, "report": report, "state": ws}}
return models.Action(world_delta=world_delta_of(snapshot))
def test_world_delta_of_keeps_every_report_list():
"""The stored column must carry the refusals, not just the successes.
`Action.world_changes` marks a clamped chip from `clamped` and builds its
refusal chips from `rejected`, and `worldstate.refusals` reads both. Dropping
either list leaves every consumer unable to tell a refused change from one
that worked.
"""
_, report = w.apply_delta(
w.instantiate(SCHEMA), SCHEMA,
{"player.arrows": 3, "player.nonesuch": 1, "player.hp": -5}, 1,
)
stored = world_delta_of({"world_state": {"delta": {}, "report": report}})
assert [e["path"] for e in stored["applied"]] == ["player.arrows", "player.hp"]
assert [e["path"] for e in stored["clamped"]] == ["player.arrows"]
assert [e["path"] for e in stored["rejected"]] == ["player.nonesuch"]
def test_world_delta_of_survives_a_snapshot_with_no_world_state():
assert world_delta_of({"story": "s"}) is None
assert world_delta_of(None) is None
# --------------------------------------------------------------------------- #
# The goals line names the id the AI has to send
# --------------------------------------------------------------------------- #
def test_goals_line_names_the_milestone_id():
line = w.render_state_section(w.instantiate(SCHEMA), SCHEMA, {})
assert "rescue_gwen — Rescue Gwen" in line
assert "escape_keep — Escape the keep" in line
# The path is spelled out, because the id alone does not say how to send it.
assert "milestones.<id>" in line
def test_achieved_line_names_the_id_too():
ws, _ = w.apply_delta(w.instantiate(SCHEMA), SCHEMA, {"milestones.rescue_gwen": True}, 1)
line = w.render_state_section(ws, SCHEMA, {})
assert "Achieved: rescue_gwen — Rescue Gwen." in line
# A reached milestone leaves the goals list.
assert "rescue_gwen — Rescue Gwen;" not in line
def test_every_milestone_in_the_demo_is_named_to_the_model():
"""The demo that exposed this must not lose the ids again."""
schema = json.loads((SEED / "05-league-championship.json").read_text(encoding="utf-8"))["stat_schema"]
line = w.render_state_section(w.instantiate(schema), schema, {})
for mid in schema["milestones"]:
assert mid in line
# --------------------------------------------------------------------------- #
# A refused change reaches the turn summary
# --------------------------------------------------------------------------- #
def test_a_clamp_that_changes_nothing_is_still_reported():
"""The Pokémon bug: a positive delta on a stat already at its ceiling.
`arrows` sits at 6 of a maximum 6, so `+2` caps to `+1`, reaches 7, and
clamps back to 6. The value never moves, and the summary has to say so.
"""
chips = action({"player.arrows": 2}).world_changes
arrows = [c for c in chips if c["label"] == "arrows"]
assert len(arrows) == 1
assert arrows[0]["delta"] == 0
assert arrows[0]["clamped"] is True
def test_a_clamp_that_still_moves_the_value_is_marked():
chips = action({"player.hp": -80}).world_changes
hp = [c for c in chips if c["label"] == "hp"][0]
assert hp["delta"] == -30 # max_delta_per_turn
assert hp["clamped"] is True
def test_an_accepted_change_is_not_marked():
chips = action({"player.hp": -10}).world_changes
hp = [c for c in chips if c["label"] == "hp"][0]
assert hp["delta"] == -10
assert hp["clamped"] is False
def test_a_refusal_becomes_its_own_entry_carrying_the_reason():
chips = action({"world.day": -1, "player.stamina": 5}).world_changes
refused = {c["label"]: c["reason"] for c in chips if c["kind"] == "rejected"}
assert refused == {"day": "counter can't decrease", "stamina": "unknown stat"}
def test_a_refused_npc_stat_keeps_the_npc_in_its_label():
chips = action({"npc.gwen.bogus": 5}).world_changes
refused = [c for c in chips if c["kind"] == "rejected"][0]
assert refused["label"] == "gwen bogus"
def test_accepted_and_refused_changes_appear_together():
chips = action({"player.hp": -10, "world.day": -1}).world_changes
kinds = [c["kind"] for c in chips]
assert "stat" in kinds and "rejected" in kinds
def test_a_turn_that_changed_nothing_still_has_no_chips():
assert models.Action(world_delta=None).world_changes == []
def test_flags_and_milestones_are_unchanged_by_the_new_fields():
chips = action({"flags.has_key": True, "milestones.rescue_gwen": True}).world_changes
assert {"kind": "flag", "label": "has_key", "on": True} in chips
assert {"kind": "milestone", "label": "rescue_gwen"} in chips
# --------------------------------------------------------------------------- #
# The demo stat counts up, so a wrong sign is refused rather than absorbed
# --------------------------------------------------------------------------- #
def test_milos_faint_counter_counts_up_from_zero():
"""A stat that starts at its ceiling cannot report a wrong sign.
`pokemon_left` began at 3 of a maximum 3, so the model sending the count it
had left rather than a delta clamped to no change at all. Counting the
faints up from 0 puts the wrong direction on the counter rule, which
refuses it out loud.
"""
schema = json.loads((SEED / "05-league-championship.json").read_text(encoding="utf-8"))["stat_schema"]
stat = schema["npcs"]["milo"]["stats"]["pokemon_fainted"]
assert stat["initial"] == 0 and stat["type"] == "counter"
ws = w.instantiate(schema)
after, report = w.apply_delta(ws, schema, {"npc.milo.pokemon_fainted": 1}, 1)
assert after["npc"]["milo"]["pokemon_fainted"] == 1
_, report = w.apply_delta(ws, schema, {"npc.milo.pokemon_fainted": -1}, 1)
assert len(report["rejected"]) == 1
refused = report["rejected"][0]
assert refused["path"] == "npc.milo.pokemon_fainted"
assert refused["reason"] == "counter can't decrease"
def test_the_demo_no_longer_mentions_the_old_stat():
"""The instructions name the stat, so a rename has to reach them too."""
raw = (SEED / "05-league-championship.json").read_text(encoding="utf-8")
assert "pokemon_left" not in raw
assert "pokemon_fainted" in json.loads(raw)["ai_instructions"]
def test_the_demo_asks_for_the_turn_counter():
"""`world.turn` sat at 0 for a whole playtest: nothing told the model to
move it. The schema defining a stat is not an instruction to update it."""
d = json.loads((SEED / "05-league-championship.json").read_text(encoding="utf-8"))
assert "world.turn" in d["stat_schema"]["world"] or "turn" in d["stat_schema"]["world"]
assert "world.turn" in d["ai_instructions"]
# --------------------------------------------------------------------------- #
# What the model is told about its own refused changes
# --------------------------------------------------------------------------- #
def test_history_replays_what_was_accepted_not_what_was_sent():
"""The contradiction that taught the model to repeat itself.
`arrows` is at its ceiling, so `+2` changes nothing. Replaying the sent
delta showed the model a change the live values disagreed with.
"""
from app.context.builder import _history_text
a = action({"player.arrows": 2, "player.hp": -10})
a.text = "The arrow flies."
replayed = _history_text(a)
assert '"player.hp": -10' in replayed
assert "arrows" not in replayed
def test_history_replay_keeps_flags_and_milestones_and_text():
from app.context.builder import _history_text
a = action({"flags.has_key": True, "milestones.rescue_gwen": True})
a.text = "The lock gives."
replayed = _history_text(a)
assert '"flags.has_key": true' in replayed
assert '"milestones.rescue_gwen": true' in replayed
def test_a_refusal_reaches_the_model_with_the_valid_names():
note = w.render_refusals(action({"milestones.bogus": True}).world_delta)
assert "There is no milestone `bogus`" in note
# The correction has to name what it could have sent instead.
assert "`rescue_gwen`" in note and "`escape_keep`" in note
def test_a_wrong_sign_on_a_counter_is_explained():
note = w.render_refusals(action({"world.day": -1}).world_delta)
assert "only counts up" in note
def test_a_clamp_that_moved_nothing_quotes_the_limit():
note = w.render_refusals(action({"player.arrows": 2}).world_delta)
assert "did not move" in note
assert "maximum of 6" in note and "it runs from 0 to 6" in note
def test_a_clamp_that_still_moved_the_value_says_nothing():
"""Reporting a trimmed change invites the model to send the remainder next
turn, which is the swing `max_delta_per_turn` exists to prevent."""
assert w.render_refusals(action({"player.hp": -80}).world_delta) == ""
def test_a_clean_turn_adds_no_note():
assert w.render_refusals(action({"player.hp": -10}).world_delta) == ""
assert w.render_refusals(None) == ""
backend/tests/test_chat.py
+248
View File
@@ -0,0 +1,248 @@
"""HTTP tests for the AI Chat scratchpad (power users only).
Covers the access gate, the streamed reply, and the demo-key model pinning.
This pinning must not let a public visitor reach paid models through this
page.
python -m pytest tests/test_chat.py -v
"""
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from app import auth, limits, models
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
from app.routers import chat
class FakeProvider:
"""Records what it was constructed with, then streams a fixed reply. Stands
in for the real egress point, so asserting on last_key/last_model is
asserting on exactly what would have gone over the wire."""
last_usage = None
last_model = None
last_key = None
last_endpoint = None
last_messages = None
def __init__(self, endpoint_url, api_key, model, api_mode="chat", reasoning_max_tokens=0):
FakeProvider.last_model = model
FakeProvider.last_key = api_key
FakeProvider.last_endpoint = endpoint_url
async def chat(self, messages, *, temperature, max_tokens):
FakeProvider.last_messages = messages
yield ("reasoning", "hmm")
yield ("text", "Hello back.")
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
user = models.User(is_guest=False, email="power@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="test-model"))
setup.commit()
user_id = user.id
setup.close()
monkeypatch.setattr(chat, "OpenAICompatibleProvider", FakeProvider)
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
# Multi-user mode is what makes the power-user gate meaningful, because
# local mode trusts everyone. The allowlist is set per test.
monkeypatch.setattr(auth, "MULTI_USER", True)
monkeypatch.setattr(auth, "POWER_USERS", {"power@example.com"})
# These tests deliberately do not stub resolve_provider_config. The
# point is to exercise the real BYOK-vs-demo decision, since that
# decision is what keeps the shared key off paid models. Each test
# picks a mode with _byok/_demo below.
def _current_user(db=Depends(get_db)):
return db.get(models.User, user_id)
app.dependency_overrides[auth.get_current_user] = _current_user
try:
yield TestClient(app)
finally:
app.dependency_overrides.clear()
Base.metadata.drop_all(bind=engine)
def _send(client, **extra):
return client.post("/api/chat/stream", json={"messages": [{"role": "user", "content": "hi"}], **extra})
def _byok(monkeypatch):
"""The user brought their own key: no demo key in play, any model allowed."""
monkeypatch.setattr(auth, "demo_enabled", lambda: False)
db = SessionLocal()
try:
settings = db.query(models.Settings).first()
settings.api_key = "sk-my-own-key" # legacy-plaintext path: used as-is
db.commit()
finally:
db.close()
def _demo(monkeypatch, whitelist=("free/allowed",)):
"""The user has no key, so turns run on the server-funded demo key."""
monkeypatch.setattr(auth, "demo_enabled", lambda: True)
monkeypatch.setattr(auth, "DEMO_API_KEY", "demo-key")
monkeypatch.setattr(auth, "DEMO_ENDPOINT_URL", "http://demo")
monkeypatch.setattr(auth, "DEMO_MODELS", list(whitelist))
def test_non_power_user_gets_404(client, monkeypatch):
monkeypatch.setattr(auth, "POWER_USERS", set())
assert _send(client).status_code == 404
assert client.get("/api/chat/config").status_code == 404
def test_power_user_streams_a_reply(client, monkeypatch):
_byok(monkeypatch)
resp = _send(client)
assert resp.status_code == 200, resp.text
assert '"type": "reasoning"' in resp.text
assert "Hello back." in resp.text
assert '"type": "done"' in resp.text
assert FakeProvider.last_messages == [{"role": "user", "content": "hi"}]
def test_system_prompt_and_model_override_are_honoured(client, monkeypatch):
_byok(monkeypatch)
resp = client.post("/api/chat/stream", json={
"messages": [
{"role": "system", "content": "Be terse."},
{"role": "user", "content": "hi"},
],
"model": "some/other-model",
})
assert resp.status_code == 200, resp.text
# BYOK: any model the user names is passed straight through, on their key.
assert FakeProvider.last_model == "some/other-model"
assert FakeProvider.last_key == "sk-my-own-key"
assert FakeProvider.last_messages[0] == {"role": "system", "content": "Be terse."}
def test_demo_key_pins_model_to_whitelist(client, monkeypatch):
_demo(monkeypatch)
resp = _send(client, model="expensive/paid-model")
assert resp.status_code == 200, resp.text
# Refused visibly: the whitelisted model runs instead, with a note. The
# paid slug must never reach the wire alongside the server-funded key.
assert FakeProvider.last_model == "free/allowed"
assert FakeProvider.last_key == "demo-key"
assert '"type": "note"' in resp.text
# A whitelisted model is still selectable on the demo key.
_demo(monkeypatch, ["free/allowed", "free/second"])
resp = _send(client, model="free/second")
assert resp.status_code == 200, resp.text
assert FakeProvider.last_model == "free/second"
def test_demo_key_ignores_an_off_whitelist_settings_model(client, monkeypatch):
"""The override is not the only untrusted input. `Settings.model` is
also user-set, and it must be pinned the same way when there is no
BYOK key."""
_demo(monkeypatch)
db = SessionLocal()
try:
db.query(models.Settings).first().model = "expensive/paid-model"
db.commit()
finally:
db.close()
resp = _send(client)
assert resp.status_code == 200, resp.text
assert FakeProvider.last_model == "free/allowed"
def test_demo_key_endpoint_cannot_be_redirected(client, monkeypatch):
"""A user-controlled `endpoint_url` would leak the key itself, which is
worse than spending it. The demo branch pins the URL too."""
_demo(monkeypatch)
db = SessionLocal()
try:
db.query(models.Settings).first().endpoint_url = "http://attacker.example/v1"
db.commit()
finally:
db.close()
assert _send(client).status_code == 200
assert FakeProvider.last_endpoint == "http://demo"
assert FakeProvider.last_key == "demo-key"
def test_provider_config_refuses_server_funded_paid_model(monkeypatch):
"""The structural backstop: a hand-built config (a future code path that
forgets to go through resolve_provider_config) cannot run a
server-funded turn on an off-whitelist model."""
monkeypatch.setattr(auth, "DEMO_API_KEY", "demo-key")
monkeypatch.setattr(auth, "DEMO_MODELS", ["free/allowed"])
with pytest.raises(ValueError):
auth.ProviderConfig("http://demo", "demo-key", "expensive/paid-model", True)
auth.ProviderConfig("http://demo", "demo-key", "free/allowed", True) # whitelisted: fine
# The user's own key with any model stays fine.
auth.ProviderConfig("http://any", "sk-mine", "expensive/paid-model", False)
def test_byok_user_may_reuse_the_demo_keys_value(client, monkeypatch):
"""Regression: the demo key is just an OpenRouter key, so a user can paste
that same value into their own Settings. That is still BYOK, because
the user is paying, and it must not trip the guard. It used to raise
on every resolution, which returned a 500 from `GET /auth/me` and
broke the entire SPA (no nav, no chat)."""
monkeypatch.setattr(auth, "demo_enabled", lambda: True)
monkeypatch.setattr(auth, "DEMO_API_KEY", "shared-key")
monkeypatch.setattr(auth, "DEMO_ENDPOINT_URL", "http://demo")
monkeypatch.setattr(auth, "DEMO_MODELS", ["free/allowed"])
db = SessionLocal()
try:
settings = db.query(models.Settings).first()
settings.api_key = "shared-key" # same value, but supplied by the user
settings.model = "expensive/paid-model" # their spend, their choice
db.commit()
finally:
db.close()
assert client.get("/api/auth/me").status_code == 200
assert client.get("/api/chat/config").status_code == 200
resp = _send(client)
assert resp.status_code == 200, resp.text
assert FakeProvider.last_model == "expensive/paid-model"
assert FakeProvider.last_key == "shared-key"
def test_resolve_provider_config_is_the_single_choke_point(monkeypatch):
"""Turns, AI Chat, and the connection test all resolve through this one
function, so pinning it here pins every caller. No DB or HTTP needed."""
monkeypatch.setattr(auth, "demo_enabled", lambda: True)
monkeypatch.setattr(auth, "DEMO_API_KEY", "demo-key")
monkeypatch.setattr(auth, "DEMO_ENDPOINT_URL", "http://demo")
monkeypatch.setattr(auth, "DEMO_MODELS", ["free/allowed"])
# No key of their own: both endpoint and model are pinned, regardless
# of what they set.
no_key = models.Settings(endpoint_url="http://mine/v1", api_key="", model="expensive/paid")
assert auth.resolve_provider_config(no_key) == auth.ProviderConfig(
"http://demo", "demo-key", "free/allowed", True)
assert auth.resolve_provider_config(
no_key, model_override="expensive/paid").model == "free/allowed"
assert auth.resolve_provider_config(
no_key, model_override="free/allowed").model == "free/allowed"
# Their own key: their endpoint, their key, their choice of model.
byok = models.Settings(endpoint_url="http://mine/v1", api_key="sk-mine", model="expensive/paid")
assert auth.resolve_provider_config(byok) == auth.ProviderConfig(
"http://mine/v1", "sk-mine", "expensive/paid", False)
def test_oversized_conversation_is_refused(client, monkeypatch):
_byok(monkeypatch)
huge = "x" * 90_000
resp = client.post("/api/chat/stream", json={
"messages": [{"role": "user", "content": huge} for _ in range(5)],
})
assert resp.status_code == 413, resp.text
backend/tests/test_delete_state.py
+214
View File
@@ -0,0 +1,214 @@
"""Deleting a turn puts the shared state back.
`script_state` and `world_state` belong to the adventure, not to the node
that changed them. Undo, retry, a take and a branch switch all restore them;
the delete endpoint did not. Deleting an AI turn removed the text and left
everything the turn did to the numbers standing.
The visible symptom was the cooldown clock. It lives in
`world_state._meta.last_changed` and it holds a depth. A deleted turn left
its depth there, and the turn played in its place is played at that same
depth, so the referee refused the change as one that had happened this very
turn — on a turn the story no longer contains. Delete the AI reply because
you did not like the stat change it proposed, press Continue, and the same
change comes back refused as "changed too recently".
python -m pytest tests/test_delete_state.py -v
"""
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from app import auth, limits, models
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
from app.routers import adventures
from fakes import ScriptedProvider
# `mana` carries a cooldown, so a clock that was not rolled back shows up as
# a refusal rather than as a number that is merely off.
SCHEMA = {
"player": {
"hp": {"min": 0, "max": 100, "initial": 100},
"mana": {"min": 0, "max": 50, "initial": 50, "cooldown": 2},
}
}
# Ten gold a turn. A total that only ever climbs makes a missing rollback
# obvious: it is off by exactly one turn's worth.
GOLD_SCRIPT = """
const modifier = (text) => {
state.gold = (state.gold || 0) + 10;
return { text };
};
modifier(text);
"""
DRAIN = 'Drained.\n```state\n{"player.mana": -10}\n```'
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
user = models.User(is_guest=False, email="delstate@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="test-model"))
scenario = models.Scenario(user_id=user.id, title="S", stat_schema=SCHEMA)
setup.add(scenario)
setup.flush()
adv = models.Adventure(
user_id=user.id, title="Tower", scenario_id=scenario.id,
script_state={}, world_state={"player": {"hp": 100, "mana": 50}},
)
setup.add(adv)
setup.flush()
setup.add(models.Action(adventure_id=adv.id, type="start", text="You begin."))
setup.add(models.AdventureScript(
adventure_id=adv.id, position=0, enabled=True, name="Gold", output_js=GOLD_SCRIPT,
))
setup.commit()
adv_id, user_id = adv.id, user.id
setup.close()
ScriptedProvider.replies = [DRAIN]
ScriptedProvider.calls = 0
monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", ScriptedProvider)
monkeypatch.setattr(auth, "resolve_provider_config", lambda s: auth.ProviderConfig(
"http://fake", "k", "test-model", False))
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
def _current_user(db=Depends(get_db)):
return db.get(models.User, user_id)
app.dependency_overrides[auth.get_current_user] = _current_user
c = TestClient(app)
c.adv_id = adv_id
try:
yield c
finally:
app.dependency_overrides.clear()
adventures.turns._active_turns.clear()
Base.metadata.drop_all(bind=engine)
# ------------------------------------------------------------------ helpers
def _play(client, text="look around"):
r = client.post(f"/api/adventures/{client.adv_id}/actions",
json={"type": "do", "text": text})
assert r.status_code == 200, r.text
def _continue(client):
r = client.post(f"/api/adventures/{client.adv_id}/actions",
json={"type": "continue", "text": ""})
assert r.status_code == 200, r.text
def _delete(client, action_id):
return client.delete(f"/api/adventures/{client.adv_id}/actions/{action_id}")
def _state(adv_id):
db = SessionLocal()
try:
adv = db.get(models.Adventure, adv_id)
return adv.script_state, adv.world_state
finally:
db.close()
def _ai_rows(adv_id):
db = SessionLocal()
try:
return (
db.query(models.Action)
.filter(models.Action.adventure_id == adv_id, models.Action.type == "ai")
.order_by(models.Action.depth, models.Action.id)
.all()
)
finally:
db.close()
def _last_changes(adv_id):
db = SessionLocal()
try:
adv = db.get(models.Adventure, adv_id)
return adv.actions[-1].world_changes
finally:
db.close()
# ------------------------------------------------------- the reported bug
def test_deleting_the_ai_turn_rewinds_the_world_state(client):
_play(client)
assert _state(client.adv_id)[1]["player"]["mana"] == 40
_delete(client, _ai_rows(client.adv_id)[-1].id)
_, world = _state(client.adv_id)
assert world["player"]["mana"] == 50, "the drain went with the turn"
assert not (world.get("_meta") or {}).get("last_changed"), "and so did its clock"
def test_the_next_turn_is_not_refused_for_a_deleted_turn_s_cooldown(client):
"""The bug as a player meets it: delete the reply, press Continue, and
the change it proposes is refused as one that already happened."""
_play(client)
_delete(client, _ai_rows(client.adv_id)[-1].id)
_continue(client)
assert _state(client.adv_id)[1]["player"]["mana"] == 40, "the drain lands"
assert [c for c in _last_changes(client.adv_id) if c["kind"] == "rejected"] == []
def test_deleting_the_ai_turn_rewinds_the_script_state(client):
"""The same restore, on the other half of the shared state. Without it a
replayed turn stacks its script run on top of the deleted one's."""
_play(client)
assert _state(client.adv_id)[0] == {"gold": 10}
_delete(client, _ai_rows(client.adv_id)[-1].id)
assert _state(client.adv_id)[0] == {}
_continue(client)
assert _state(client.adv_id)[0] == {"gold": 10}, "one turn of gold, not two"
# ------------------------------------------------- deleting further back
def test_deleting_a_turn_the_story_moved_past_leaves_the_tip_alone(client):
"""A restore reads the tip's own outcome, not the deleted node's
neighbour, so removing a turn from the middle of the story does not roll
the numbers back to that point. The text goes; the state stays."""
_play(client)
_play(client, "press on")
before = _state(client.adv_id)
assert before[0] == {"gold": 20}
first_ai = _ai_rows(client.adv_id)[0]
assert _delete(client, first_ai.id).status_code == 204
assert _state(client.adv_id) == before
def test_delete_is_blocked_while_a_turn_is_generating(client):
"""The endpoint writes the shared state now, so it takes the same lock
undo takes rather than racing the turn that is about to write it."""
_play(client)
action_id = _ai_rows(client.adv_id)[-1].id
adventures.turns.acquire_turn_lock(client.adv_id) # a turn is "generating"
try:
assert _delete(client, action_id).status_code == 409
# The refused delete must not have released someone else's lock.
assert client.adv_id in adventures.turns._active_turns
finally:
adventures.turns._active_turns.discard(client.adv_id)
assert len(_ai_rows(client.adv_id)) == 1, "and the turn is still there"
backend/tests/test_egress.py
+463
View File
@@ -0,0 +1,463 @@
"""Guards on how much the database is asked for.
`context_snapshot` holds the entire assembled prompt for a turn. That is 163 kB
per row averaged over production, 232 kB on the longest adventure, and 89% of the
database. It used to be fetched for every action on every adventure load and
every turn, to read two small values out of it. These tests fail if that
returns.
Two kinds of guard live here, and both are needed:
* Column guards assert which columns a statement names. That is the shape both
of this project's egress regressions took: one query carrying a column nobody
read.
* Byte ceilings assert what a request costs. Every column guard would still pass
if a response grew tenfold within the columns it is allowed to read, which is
what a story that keeps getting longer does.
python -m pytest tests/test_egress.py -v
"""
import json
import random
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from sqlalchemy import event, text
from sqlalchemy.orm import undefer
from app import auth, limits, migrations, models
from app.context import history
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
from tools import dbmeter
from tools.fakeprose import prose
# A stand-in for the real thing: the assembled prompt, which is what makes the
# column enormous, plus the small world_state slice the UI actually needs.
#
# Varied text, not `"x" * 20_000`. The column is stored compressed now
# by migration 43, and a repeated character compresses about a thousandfold.
# That would make the byte ceilings below pass against a fixture that costs
# nothing, which tests nothing. Prose-shaped filler compresses like the prompts
# it stands in for.
_SNAPSHOT_RNG = random.Random(20_260_817)
BIG_SNAPSHOT = {
"system": prose(_SNAPSHOT_RNG, 20_000),
"story": prose(_SNAPSHOT_RNG, 40_000),
"world_state": {
"delta": {"player.hp": -15},
"report": {"applied": [{"path": "player.hp", "old": 100, "new": 85}]},
"state": {"player": {"hp": 85}},
},
}
# Retry history: each discarded attempt keeps its full narration, so an action
# retried a few times carries several KB that a list response only ever counts.
BIG_VARIANTS = [
{"text": "z" * 4_000, "reasoning": None, "script_state": {},
"created_at": "2026-01-01T00:00:00"}
for _ in range(3)
]
@pytest.fixture()
def sql_log():
"""Every statement the ORM sends, for asserting on what was selected."""
statements: list[str] = []
def record(conn, cursor, statement, parameters, context, executemany):
statements.append(statement)
event.listen(engine, "before_cursor_execute", record)
try:
yield statements
finally:
event.remove(engine, "before_cursor_execute", record)
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
setup = SessionLocal()
user = models.User(is_guest=False, email="egress@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(user_id=user.id, api_key="enc:dummy", model="test-model"))
adventure = models.Adventure(user_id=user.id, title="Cave", script_state={})
setup.add(adventure)
setup.flush()
for i in range(12):
setup.add(models.Action(
adventure_id=adventure.id,
type="ai" if i % 2 else "do", text=f"Action {i}.",
context_snapshot=BIG_SNAPSHOT,
world_delta={"delta": {"player.hp": -15},
"applied": [{"path": "player.hp", "old": 100, "new": 85}]},
))
setup.commit()
adv_id, user_id = adventure.id, user.id
setup.close()
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
def _current_user(db=Depends(get_db)):
return db.get(models.User, user_id)
app.dependency_overrides[auth.get_current_user] = _current_user
c = TestClient(app)
c.adv_id = adv_id
try:
yield c
finally:
app.dependency_overrides.clear()
Base.metadata.drop_all(bind=engine)
def action_selects(statements: list[str]) -> list[str]:
return [s for s in statements if "FROM actions" in s and s.lstrip().upper().startswith("SELECT")]
# ------------------------------------------------------- the deferred columns
def test_loading_an_adventure_does_not_fetch_context_snapshot(client, sql_log):
r = client.get(f"/api/adventures/{client.adv_id}")
assert r.status_code == 200, r.text
assert len(r.json()["actions"]) == 12
selects = action_selects(sql_log)
assert selects, "expected at least one SELECT against actions"
offenders = [s for s in selects if "context_snapshot" in s]
assert offenders == [], f"context_snapshot was fetched in bulk:\n{offenders[0][:400]}"
def test_the_state_snapshots_are_not_fetched_in_bulk(client, sql_log):
"""Both are rollback snapshots, only ever needed for the single node being
undone, retried past or switched to. A page load must pay for neither.
"""
client.get(f"/api/adventures/{client.adv_id}")
selects = action_selects(sql_log)
for column in ("state_after", "world_state_after"):
offenders = [s for s in selects if column in s]
assert offenders == [], f"{column} was fetched in bulk"
def test_world_changes_still_works_without_the_snapshot(client):
"""The chips under an AI message must survive the snapshot being deferred."""
r = client.get(f"/api/adventures/{client.adv_id}")
ai = [a for a in r.json()["actions"] if a["type"] == "ai"]
assert ai, "fixture should have AI actions"
assert ai[0]["world_changes"] == [
{"kind": "stat", "label": "hp", "delta": -15, "value": 85, "clamped": False}
]
def test_counting_actions_does_not_name_the_deferred_columns(client, sql_log):
"""A count that wraps the entity select in a subquery names every column in
the emitted SQL.
No bytes come back, but the database still reads them, and the guard above
cannot distinguish that from a real bulk fetch.
"""
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
sql_log.clear()
assert history.count(adventure) == 12
counts = [s for s in sql_log if "count" in s.lower()]
assert counts, "expected a COUNT to be emitted"
for column in ("context_snapshot", "state_after", "world_state_after"):
assert not any(column in s for s in counts), (
f"{column} is named by the count query:\n{counts[0][:400]}"
)
finally:
db.close()
def test_snapshot_is_still_reachable_on_demand(client):
"""Deferred means lazy rather than absent. Insights still gets the whole
snapshot."""
r = client.get(f"/api/adventures/{client.adv_id}")
action_id = r.json()["actions"][0]["id"]
r = client.get(f"/api/adventures/{client.adv_id}/actions/{action_id}/context")
assert r.status_code == 200, r.text
# Round-tripped through zlib and back to a dict, byte for byte.
assert r.json()["system"] == BIG_SNAPSHOT["system"]
assert r.json()["story"] == BIG_SNAPSHOT["story"]
# ------------------------------------------------------------------ backfill
def as_json_snapshot_column(db) -> None:
"""Put actions.context_snapshot back as JSON, the way it was before 43.
Migration 36 lifts world_delta out of the snapshot with SQL JSON
functions, so it can only run while the column still *is* JSON. In a real
upgrade it always is, because 36 runs seven migrations before 43 compresses
the column into a BLOB. `create_all` builds today's schema, so a test that
calls that backfill has to rebuild the schema it was written against.
"""
db.execute(text("ALTER TABLE actions DROP COLUMN context_snapshot"))
db.execute(text("ALTER TABLE actions ADD COLUMN context_snapshot JSON"))
db.execute(
text("UPDATE actions SET context_snapshot = :snapshot"),
{"snapshot": json.dumps(BIG_SNAPSHOT)},
)
db.commit()
def test_backfill_populates_world_delta_from_existing_snapshots(client):
"""Migration 36 lifts the slice out server-side, without reading the
snapshots into Python."""
db = SessionLocal()
try:
as_json_snapshot_column(db)
db.execute(text("UPDATE actions SET world_delta = NULL"))
db.commit()
assert db.query(models.Action).filter(models.Action.world_delta.isnot(None)).count() == 0
with engine.begin() as conn:
migrations._backfill_world_delta(conn)
db.expire_all()
actions = db.query(models.Action).all()
assert all(a.world_delta is not None for a in actions)
assert actions[0].world_delta["delta"] == {"player.hp": -15}
assert actions[0].world_delta["applied"] == [
{"path": "player.hp", "old": 100, "new": 85}
]
finally:
db.close()
def test_backfill_populates_variant_count_from_existing_variants(client):
"""Migration 37 counts the lists on the server.
Reading them into Python to count them would fetch the column over the wire
once in order to stop fetching it on every request.
Migration 68 drops `variant_count` and 66 drops `variants`, so this test
puts both columns back before it calls the pass, the same way
`as_json_snapshot_column` above rebuilds the column migration 36 needs. The
assertions read raw SQL, because the model no longer has either attribute.
"""
db = SessionLocal()
try:
db.execute(text("ALTER TABLE actions ADD COLUMN variants JSON"))
db.execute(text(
"ALTER TABLE actions ADD COLUMN variant_count INTEGER NOT NULL DEFAULT 0"
))
db.execute(
text("UPDATE actions SET variants = :v WHERE type = 'ai'"),
{"v": json.dumps(BIG_VARIANTS)},
)
db.commit()
with engine.begin() as conn:
migrations._backfill_variant_count(conn)
rows = db.execute(text(
"SELECT type, variant_count FROM actions ORDER BY id"
)).all()
assert rows, "fixture should have actions"
for kind, count in rows:
assert count == (len(BIG_VARIANTS) if kind == "ai" else 0)
finally:
db.close()
def test_backfill_leaves_actions_without_world_state_alone(client):
db = SessionLocal()
try:
db.execute(text("UPDATE actions SET world_delta = NULL, context_snapshot = '{\"story\": \"s\"}'"))
db.commit()
with engine.begin() as conn:
migrations._backfill_world_delta(conn)
db.expire_all()
assert all(a.world_delta is None for a in db.query(models.Action).all())
finally:
db.close()
# ---------------------------------------------------------------- byte ceilings
#
# The tests above assert which columns a statement names, which is the shape both
# of this project's egress regressions took. They would all still pass if a
# response grew tenfold within the columns it is allowed to read, and a story
# that keeps getting longer does that. These tests put a number on it.
#
# The ceilings are per action rather than absolute, so they mean the same thing
# whatever size the fixture is, and they are generous. They exist to catch a
# tenfold regression rather than to freeze today's byte count.
ACTIONS_IN_FIXTURE = 12
# 3 kB an action against a real 994 B, measured on production 2026-08-17.
# Anything that pulls a deferred column blows past this by two orders of
# magnitude. See `test_the_ceiling_discriminates` below.
PAGE_LOAD_BYTES_PER_ACTION = 3_000
@pytest.fixture()
def meter():
"""A byte meter on the shared engine, removed again afterwards.
A test requests this after `client` in its arguments, so that building the
fixture, which is a write path no player takes, is not charged to any
scope.
"""
m = dbmeter.Meter()
m.attach(engine)
try:
yield m
finally:
m.detach()
def fetched(meter) -> int:
return meter.scopes[-1].total.fetched
def test_page_load_stays_under_its_byte_ceiling(client, meter):
with meter.scope("page load"):
r = client.get(f"/api/adventures/{client.adv_id}")
assert r.status_code == 200
budget = ACTIONS_IN_FIXTURE * PAGE_LOAD_BYTES_PER_ACTION
assert fetched(meter) < budget, (
f"page load fetched {fetched(meter):,} B for {ACTIONS_IN_FIXTURE} "
f"actions, over the {budget:,} B budget"
)
def test_the_action_list_stays_under_its_byte_ceiling(client, meter):
with meter.scope("action list"):
r = client.get(f"/api/adventures/{client.adv_id}/actions")
assert r.status_code == 200
budget = ACTIONS_IN_FIXTURE * PAGE_LOAD_BYTES_PER_ACTION
assert fetched(meter) < budget, (
f"the action list fetched {fetched(meter):,} B, over {budget:,} B"
)
def test_reading_one_action_does_not_cost_the_whole_story(client, meter):
"""The snapshot is reachable on demand, and that request pays for one row
rather than for the whole adventure."""
db = SessionLocal()
try:
action_id = db.query(models.Action.id).order_by(models.Action.id).first()[0]
finally:
db.close()
with meter.scope("one snapshot"):
r = client.get(f"/api/adventures/{client.adv_id}/actions/{action_id}/context")
assert r.status_code == 200, r.text
one_snapshot = len(json.dumps(BIG_SNAPSHOT))
assert fetched(meter) < one_snapshot * 2, (
f"fetching one action's snapshot cost {fetched(meter):,} B; one "
f"snapshot is {one_snapshot:,} B"
)
def _fat_adventures(user_id: int, count: int = 5, body: int = 20_000) -> None:
"""Adventures whose bodies are heavy and whose index cards are not.
script_state, world_state and story_summary belong to the play screen. The
index shows a title, a stamp and a snippet, and used to load all of it.
"""
db = SessionLocal()
try:
for i in range(count):
db.add(models.Adventure(
user_id=user_id,
title=f"Adventure {i}",
script_state={"log": "s" * body},
world_state={"player": {"notes": "w" * body}},
story_summary="y" * body,
memory="m" * body,
))
db.commit()
finally:
db.close()
def test_the_index_does_not_read_the_adventure_body(client, sql_log):
db = SessionLocal()
try:
user_id = db.query(models.User.id).first()[0]
finally:
db.close()
_fat_adventures(user_id)
r = client.get("/api/adventures")
assert r.status_code == 200
assert len(r.json()) == 6 # the fixture's one, plus five
listing = [
s for s in sql_log
if "FROM adventures" in s and s.lstrip().upper().startswith("SELECT")
]
assert listing, "expected a listing query"
for column in ("script_state", "world_state", "story_summary", "memory",
"authors_note", "ai_instructions", "placeholders"):
assert not any(column in s for s in listing), (
f"the index read adventures.{column}, which nothing on that "
f"screen displays"
)
def test_the_index_stays_under_its_byte_ceiling(client, meter):
db = SessionLocal()
try:
user_id = db.query(models.User.id).first()[0]
finally:
db.close()
_fat_adventures(user_id)
with meter.scope("index"):
r = client.get("/api/adventures")
assert r.status_code == 200
# Six adventures carrying 80 kB of body each. A card is a title, a stamp
# and a 220-character snippet; 4 kB apiece is already generous.
budget = 6 * 4_000
assert fetched(meter) < budget, (
f"the index fetched {fetched(meter):,} B for six adventures, over "
f"{budget:,} B — it is reading the bodies again"
)
def test_the_ceiling_discriminates(client, meter):
"""A ceiling is only worth having if the thing it excludes would breach it.
This is the regression the byte tests exist to catch, performed on purpose:
undefer the snapshot and the same twelve rows cost several times the whole
budget. If this ever stops exceeding it, the fixture has gone too small for
the tests above to mean anything.
The margin used to be a hundredfold and is now about six. The guard has not
weakened. Migration 43 compresses the column, and the fixture text is
prose-shaped, so it compresses like a real prompt rather than like a repeated
character.
"""
budget = ACTIONS_IN_FIXTURE * PAGE_LOAD_BYTES_PER_ACTION
db = SessionLocal()
try:
with meter.scope("undeferred"):
rows = (
db.query(models.Action)
.options(undefer(models.Action.context_snapshot))
.all()
)
assert len(rows) == ACTIONS_IN_FIXTURE
finally:
db.close()
assert fetched(meter) > budget * 3, (
"undeferring the snapshot cost only "
f"{fetched(meter):,} B against a {budget:,} B budget — the fixture is "
"too small for the byte ceilings above to catch anything"
)
backend/tests/test_embedding_blob.py
+300
View File
@@ -0,0 +1,300 @@
"""Migration 38: embeddings move from a JSON list to packed float32.
The conversion has to be exact, because nothing re-embeds. A memory whose
vector shifts is silently ranked wrong forever, with no error anywhere to
report it. These tests check that the numbers survive the round trip bit
for bit, and that the migration reaches every row, however many there are.
python -m pytest tests/test_embedding_blob.py -v
"""
import struct
import json
import random
import pytest
from sqlalchemy import text
from app import memorybank, migrations, models, vectors
from app.database import Base, SessionLocal, engine
from tests import schema_rewind
def float32(value: float) -> float:
"""`value` as the double nearest to its float32 truncation. This is
what an embedding endpoint's JSON actually holds."""
return struct.unpack("<f", struct.pack("<f", value))[0]
def sample_vector(rng: random.Random, dims: int = 1536) -> list[float]:
return [float32(rng.uniform(-1.0, 1.0)) for _ in range(dims)]
@pytest.fixture()
def db():
Base.metadata.create_all(bind=engine)
session = SessionLocal()
try:
yield session
finally:
session.close()
Base.metadata.drop_all(bind=engine)
@pytest.fixture()
def adventure(db):
user = models.User(is_guest=False, email="vectors@example.com")
db.add(user)
db.flush()
adv = models.Adventure(user_id=user.id, title="Cave", script_state={})
db.add(adv)
db.commit()
return adv
# ------------------------------------------------------------------- packing
def test_pack_round_trips_exactly():
"""Not "close enough": embedding endpoints compute in float32 and render
that into JSON, so packing back to float32 must be lossless."""
rng = random.Random(1)
vector = sample_vector(rng)
assert list(vectors.unpack(vectors.pack(vector))) == vector
def test_packed_vector_is_four_bytes_per_dimension():
"""1536 dims takes 6 KB packed, against about 31 KB as JSON."""
vector = sample_vector(random.Random(2))
blob = vectors.pack(vector)
assert len(blob) == 1536 * 4
assert len(blob) < len(json.dumps(vector).encode()) / 4
def test_pack_handles_the_extremes():
vector = [float32(v) for v in (0.0, -0.0, 1.0, -1.0, 3.4028234663852886e38, 1e-38)]
assert list(vectors.unpack(vectors.pack(vector))) == vector
def test_unpack_returns_a_compact_array():
"""These vectors stay in memory between turns, so the container type
matters. An array("f") stores each component in 4 bytes, the same width
as the column. A list of Python floats uses eight times that."""
vector = sample_vector(random.Random(9))
unpacked = vectors.unpack(vectors.pack(vector))
assert unpacked.typecode == "f"
assert unpacked.itemsize == 4
assert len(unpacked) == len(vector)
def test_pack_rounds_a_value_float32_cannot_hold():
"""The exactness guarantee applies only to vectors that came from an
embedding model, which computes in float32, not to arbitrary doubles.
This test pins down that boundary, because it is where the round-trip
claim holds."""
assert vectors.unpack(vectors.pack([1e-38]))[0] != 1e-38
assert vectors.unpack(vectors.pack([1e-38]))[0] == pytest.approx(1e-38)
def test_cosine_moved_but_still_reachable_from_memorybank():
"""Callers import it from memorybank. The math lives in vectors."""
assert memorybank.cosine is vectors.cosine
assert vectors.cosine([1.0, 0.0], [1.0, 0.0]) == pytest.approx(1.0)
assert vectors.cosine([1.0, 0.0], [0.0, 1.0]) == pytest.approx(0.0)
assert vectors.cosine([1.0, 0.0], [1.0, 0.0, 0.0]) == 0.0 # length mismatch
# -------------------------------------------------------------- set_vector
def test_set_vector_writes_the_blob_and_the_flag(db, adventure):
"""The two columns that describe a vector move together, or a reader that
trusts `embedded` gets a NULL blob."""
memory = models.Memory(adventure_id=adventure.id, text="a fact")
db.add(memory)
db.commit()
vector = sample_vector(random.Random(3), dims=8)
memorybank.set_vector(memory, vector)
db.commit()
db.expire_all()
assert list(vectors.unpack(memory.embedding_blob)) == vector
assert memory.embedded is True
def test_set_vector_none_clears_both(db, adventure):
"""Editing a memory's text drops its vector so the next pass re-embeds. A
blob left behind would keep ranking the old text."""
memory = models.Memory(adventure_id=adventure.id, text="a fact")
db.add(memory)
memorybank.set_vector(memory, sample_vector(random.Random(4), dims=8))
db.commit()
memorybank.set_vector(memory, None)
db.commit()
db.expire_all()
assert memory.embedding_blob is None
assert memory.embedded is False
# --------------------------------------------------------------- the backfill
def add_legacy_json_column(db) -> None:
"""Put `memories.embedding` back for the length of a test.
Migration 42 dropped it, and the model no longer declares it, so
`create_all` does not produce it. Everything below tests the upgrade
from a database that still has the column, which is the only state
where the backfill has any work to do. Re-adding it by hand keeps these
tests honest about the schema they claim to start from.
"""
db.execute(text("ALTER TABLE memories ADD COLUMN embedding JSON"))
db.commit()
def seed_json_only(db, adventure, count: int, dims: int = 64) -> dict[int, list[float]]:
"""Memories as they exist before the migration: JSON vector, no blob."""
add_legacy_json_column(db)
rng = random.Random(count)
expected = {}
for i in range(count):
vector = sample_vector(rng, dims)
memory = models.Memory(adventure_id=adventure.id, text=f"fact {i}")
db.add(memory)
db.flush()
# Raw, because the ORM no longer knows this column exists.
db.execute(
text("UPDATE memories SET embedding = :v WHERE id = :id"),
{"v": json.dumps(vector), "id": memory.id},
)
expected[memory.id] = vector
db.commit()
db.execute(text("UPDATE memories SET embedding_blob = NULL, embedded = false"))
db.commit()
return expected
def test_backfill_converts_every_existing_vector(db, adventure):
expected = seed_json_only(db, adventure, count=5)
with engine.begin() as conn:
migrations._backfill_embedding_blob(conn)
db.expire_all()
for memory in db.query(models.Memory).all():
assert list(vectors.unpack(memory.embedding_blob)) == expected[memory.id]
def test_backfill_reaches_past_one_batch(db, adventure):
"""It loops on id, and an off-by-one there would silently leave the
tail of a big bank unconverted. That failure reads as "not embedded
yet"."""
count = migrations.BACKFILL_BATCH * 2 + 3
expected = seed_json_only(db, adventure, count=count, dims=4)
with engine.begin() as conn:
migrations._backfill_embedding_blob(conn)
db.expire_all()
memories = db.query(models.Memory).all()
assert len(memories) == count
assert all(m.embedding_blob is not None for m in memories)
assert all(list(vectors.unpack(m.embedding_blob)) == expected[m.id] for m in memories)
def test_backfill_leaves_unembedded_memories_alone(db, adventure):
add_legacy_json_column(db)
db.add(models.Memory(adventure_id=adventure.id, text="not embedded yet"))
db.commit()
with engine.begin() as conn:
migrations._backfill_embedding_blob(conn)
db.expire_all()
assert db.query(models.Memory).one().embedding_blob is None
def test_backfill_is_idempotent(db, adventure):
"""It runs once from bootstrap, but a half-finished run must be safe to
repeat, and rows already converted must not be rewritten."""
seed_json_only(db, adventure, count=3)
with engine.begin() as conn:
migrations._backfill_embedding_blob(conn)
db.expire_all()
first = {m.id: m.embedding_blob for m in db.query(models.Memory).all()}
with engine.begin() as conn:
migrations._backfill_embedding_blob(conn)
db.expire_all()
assert {m.id: m.embedding_blob for m in db.query(models.Memory).all()} == first
def test_backfill_skips_a_malformed_row_without_stopping(db, adventure):
"""One bad row must not strand every row after it. The loop orders by
id, so an exception here would leave the rest of the bank unconverted."""
expected = seed_json_only(db, adventure, count=2)
broken = models.Memory(adventure_id=adventure.id, text="broken")
db.add(broken)
db.commit()
db.execute(
text("UPDATE memories SET embedding = :bad WHERE id = :id"),
{"bad": '"not a list"', "id": broken.id},
)
db.commit()
with engine.begin() as conn:
migrations._backfill_embedding_blob(conn)
db.expire_all()
assert db.get(models.Memory, broken.id).embedding_blob is None
for memory_id, vector in expected.items():
blob = db.get(models.Memory, memory_id).embedding_blob
assert list(vectors.unpack(blob)) == vector
# ------------------------------------------------------- the upgrade in full
def test_bootstrap_adds_the_columns_and_backfills_them(db, adventure):
"""The path a deployed database actually takes: sitting at 37 with neither
new column, then started on this build."""
expected = seed_json_only(db, adventure, count=4)
unembedded = models.Memory(adventure_id=adventure.id, text="no vector yet")
db.add(unembedded)
db.commit()
unembedded_id = unembedded.id
db.close()
schema_rewind.rewind_to(engine, migrations.EMBEDDING_BLOB_VERSION - 1)
migrations.bootstrap(engine)
with engine.begin() as conn:
assert conn.execute(text("PRAGMA user_version")).scalar() == migrations.LATEST_VERSION
rows = conn.execute(text("SELECT id, embedding_blob, embedded FROM memories")).all()
by_id = {row[0]: (row[1], row[2]) for row in rows}
assert len(by_id) == len(expected) + 1
for memory_id, vector in expected.items():
blob, embedded = by_id[memory_id]
assert list(vectors.unpack(blob)) == vector
assert embedded
# The flag has to follow the vector, not the row: a memory that was never
# embedded must still read as not embedded afterwards.
assert by_id[unembedded_id] == (None, False)
# Migration 42, at the end of the same run, removes the JSON column.
# Ordering matters: 38 reads it, 42 drops it, and an upgrade that ran
# them in the other order would arrive with an empty bank.
with engine.begin() as conn:
columns = {row[1] for row in conn.execute(text("PRAGMA table_info(memories)"))}
assert "embedding" not in columns
assert {"embedding_blob", "embedded"} <= columns
def test_migration_38_is_spelled_for_both_dialects():
"""Every Postgres deploy replays migrations from 24 on, so a SQLite-only
ALTER here would break the live database and nothing else would notice."""
sql = dict(migrations.MIGRATIONS)[migrations.EMBEDDING_BLOB_VERSION]
assert "BLOB" in migrations._for_dialect(sql, "sqlite")
assert "BYTEA" in migrations._for_dialect(sql, "postgresql")
backend/tests/test_embedding_model_switch.py
+161
View File
@@ -0,0 +1,161 @@
"""Switching embedding models must re-embed the bank.
Vectors from two different models are not comparable. They live in
different spaces and often have different widths. Changing the model must
discard the stored vectors and let the post-turn pass rebuild them.
This worked while the vectors lived in `memories.embedding`. The settings
route nulled that column, and the embed queue picked up the rows. Migration
38 moved the vectors to `embedding_blob` and added an `embedded` flag beside
them, but the bulk clear kept nulling only the old column. The blob
survived, the flag stayed true, and `_embed_pending` (which filters on
`embedded IS FALSE`) never saw the rows. The bank kept ranking against the
previous model's vectors.
Nothing reports this failure. `cosine` returns 0.0 on a width mismatch, so a
different-width model scores every memory zero, and retrieval silently
returns whichever rows sort first. A same-width model scores plausible
garbage instead.
python -m pytest tests/test_embedding_model_switch.py -v
"""
import asyncio
import pytest
from fastapi import Depends
from fastapi.testclient import TestClient
from app import auth, limits, memorybank, models
from app.database import Base, SessionLocal, engine, get_db
from app.main import app
DIMS = 8
@pytest.fixture()
def client(monkeypatch):
Base.metadata.create_all(bind=engine)
memorybank._vector_cache.clear()
setup = SessionLocal()
user = models.User(is_guest=False, email="switch@example.com")
setup.add(user)
setup.flush()
setup.add(models.Settings(
user_id=user.id, api_key="enc:dummy", model="test-model",
embedding_model="model-a",
))
adventure = models.Adventure(
user_id=user.id, title="Cave", script_state={}, memory_bank_enabled=True
)
setup.add(adventure)
setup.flush()
for i in range(5):
memory = models.Memory(adventure_id=adventure.id, text=f"Memory {i}")
memorybank.set_vector(memory, [float(i)] + [0.0] * (DIMS - 1))
setup.add(memory)
setup.commit()
adv_id, user_id = adventure.id, user.id
setup.close()
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
monkeypatch.setattr(limits, "check_row_cap", lambda *a, **k: None)
def _current_user(db=Depends(get_db)):
return db.get(models.User, user_id)
app.dependency_overrides[auth.get_current_user] = _current_user
c = TestClient(app)
c.adv_id = adv_id
try:
yield c
finally:
app.dependency_overrides.clear()
memorybank._vector_cache.clear()
Base.metadata.drop_all(bind=engine)
def memories(db):
return db.query(models.Memory).order_by(models.Memory.id).all()
def test_the_bank_starts_embedded(client):
db = SessionLocal()
try:
rows = memories(db)
assert len(rows) == 5
assert all(m.embedded for m in rows)
assert all(m.embedding_blob for m in rows)
finally:
db.close()
def test_changing_the_model_clears_every_vector(client):
r = client.put("/api/settings", json={"embedding_model": "model-b"})
assert r.status_code == 200, r.text
db = SessionLocal()
try:
rows = memories(db)
assert [m.embedding_blob for m in rows] == [None] * 5, \
"the blob survived the model change"
assert not any(m.embedded for m in rows), \
"`embedded` stayed true, so nothing will ever re-embed these"
finally:
db.close()
def test_cleared_memories_are_queued_for_re_embedding(client):
"""The `embedded` flag is not cosmetic. It is the only condition
`_embed_pending` filters on, so this test confirms the bank actually
recovers."""
client.put("/api/settings", json={"embedding_model": "model-b"})
db = SessionLocal()
try:
pending = (
db.query(models.Memory)
.filter(models.Memory.embedded.is_(False),
models.Memory.forgotten.is_(False))
.all()
)
assert len(pending) == 5
finally:
db.close()
def test_retrieval_uses_no_stale_vector_after_the_switch(client, monkeypatch):
"""Until the re-embed runs, the bank must return nothing rather than
ranking against the old model's vectors."""
client.put("/api/settings", json={"embedding_model": "model-b"})
class Embedder:
async def embed(self, texts):
return [[1.0] + [0.0] * (DIMS - 1) for _ in texts]
monkeypatch.setattr(memorybank, "embedding_provider", lambda s: Embedder())
db = SessionLocal()
try:
adventure = db.get(models.Adventure, client.adv_id)
settings = db.query(models.Settings).first()
result = asyncio.run(
memorybank.retrieve_memories(adventure, settings, update_stats=False)
)
assert result["used"] == []
finally:
db.close()
def test_an_unrelated_settings_change_keeps_the_vectors(client):
"""Only an embedding-model change may clear the bank. Re-embedding costs
an API call per memory."""
r = client.put("/api/settings", json={"model": "some-other-chat-model"})
assert r.status_code == 200, r.text
db = SessionLocal()
try:
rows = memories(db)
assert all(m.embedded for m in rows)
assert all(m.embedding_blob for m in rows)
finally:
db.close()
backend/tests/test_guest_cleanup.py
+205
View File
@@ -0,0 +1,205 @@
"""Guest retention policy: app/cleanup.py.
These tests cover the two things that matter. Idle guests and their whole
data graph must actually be deleted, and nothing else must ever be
deleted.
python -m pytest tests/test_guest_cleanup.py -v
"""
from datetime import timedelta
import pytest
from sqlalchemy import create_engine, event
from sqlalchemy.orm import sessionmaker
from app import cleanup, models
from app.database import Base
from app.migrations import bootstrap
@pytest.fixture()
def db(tmp_path):
engine = create_engine(f"sqlite:///{tmp_path/'t.db'}", connect_args={"check_same_thread": False})
@event.listens_for(engine, "connect")
def _fk(dbapi_connection, _record):
# The whole policy relies on ON DELETE CASCADE. SQLite ignores every
# one of them unless this is set, the same as database.py does.
cur = dbapi_connection.cursor()
cur.execute("PRAGMA foreign_keys=ON")
cur.close()
bootstrap(engine)
Base.metadata.create_all(bind=engine)
session = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False)()
yield session
session.close()
NOW = models.utcnow().replace(tzinfo=None)
def make_user(db, *, days_idle=None, days_old=0, guest=True, email=None):
"""A user last seen `days_idle` ago (None = never seen, only created)."""
user = models.User(
is_guest=guest,
email=email,
password_hash=None if email is None else "x",
created_at=NOW - timedelta(days=days_old),
last_seen_at=None if days_idle is None else NOW - timedelta(days=days_idle),
)
db.add(user)
db.commit()
return user
def sweep(db):
return cleanup.delete_stale_guests(db, now=NOW)
def alive(db, user_id):
# A count, not db.get: the sweep deletes with synchronize_session=False, so
# the session's identity map still holds the object and db.get would answer
# from memory without ever asking the database.
return db.query(models.User).filter(models.User.id == user_id).count() == 1
# ---------- what goes ----------
def test_deletes_guest_idle_past_the_window(db):
user = make_user(db, days_idle=6)
assert sweep(db) == 1
assert not alive(db, user.id)
def test_keeps_guest_inside_the_window(db):
user = make_user(db, days_idle=4)
assert sweep(db) == 0
assert alive(db, user.id)
def test_boundary_is_not_yet_stale(db):
# Exactly 5 days survives. The comparison is strict.
user = make_user(db, days_idle=cleanup.RETENTION_DAYS)
assert sweep(db) == 0
assert alive(db, user.id)
def test_never_seen_guest_falls_back_to_created_at(db):
"""last_seen_at is NULL until a guest's second request (auth._touch runs
hourly), so a coalesce-less query would delete brand-new visitors."""
fresh = make_user(db, days_idle=None, days_old=0)
stale = make_user(db, days_idle=None, days_old=9)
assert sweep(db) == 1
assert alive(db, fresh.id)
assert not alive(db, stale.id)
def test_recent_visit_beats_an_old_created_at(db):
# A long-standing guest who came back yesterday stays.
user = make_user(db, days_idle=1, days_old=90)
assert sweep(db) == 0
assert alive(db, user.id)
# ---------- what must never go ----------
def test_spares_registered_users(db):
"""Registering upgrades the guest row in place. An idle account here is
a real user with real data, which is what signing up is meant to
protect."""
user = make_user(db, days_idle=400, guest=False, email="a@b.com")
assert sweep(db) == 0
assert alive(db, user.id)
def test_spares_the_local_mode_user(db):
# email NULL but is_guest False: local mode's implicit owner of everything.
user = make_user(db, days_idle=400, guest=False)
assert sweep(db) == 0
assert alive(db, user.id)
def test_spares_a_guest_flagged_row_that_has_an_email(db):
# This row should not exist, but both clauses are checked so it cannot be collected.
user = make_user(db, days_idle=400, guest=True, email="odd@b.com")
assert sweep(db) == 0
assert alive(db, user.id)
def test_leaves_seeded_public_scenarios_alone(db):
"""Seeded demo content has user_id NULL, so it is outside the filter."""
seeded = models.Scenario(user_id=None, is_public=True, title="Demo")
db.add(seeded)
make_user(db, days_idle=30)
db.commit()
assert sweep(db) == 1
assert db.query(models.Scenario).filter(models.Scenario.id == seeded.id).count() == 1
def test_disabled_when_retention_is_zero(db, monkeypatch):
monkeypatch.setattr(cleanup, "RETENTION_DAYS", 0)
user = make_user(db, days_idle=999)
assert sweep(db) == 0
assert alive(db, user.id)
def test_enabled_requires_multi_user(monkeypatch):
from app import auth
monkeypatch.setattr(auth, "MULTI_USER", False)
assert cleanup.enabled() is False
monkeypatch.setattr(auth, "MULTI_USER", True)
monkeypatch.setattr(cleanup, "RETENTION_DAYS", 5)
assert cleanup.enabled() is True
monkeypatch.setattr(cleanup, "RETENTION_DAYS", 0)
assert cleanup.enabled() is False
# ---------- the cascade ----------
def test_deletes_the_whole_data_graph(db):
"""One DELETE must remove the adventure, its actions and memories, the
story cards, and the settings row. Nothing is loaded into Python, so if
the FK cascade does not reach a table, its rows are silently orphaned,
or the statement fails, instead of being removed."""
user = make_user(db, days_idle=30)
scenario = models.Scenario(user_id=user.id, title="S")
db.add(scenario)
db.commit()
adventure = models.Adventure(user_id=user.id, scenario_id=scenario.id, title="A")
db.add(adventure)
db.commit()
db.add_all([
models.Action(adventure_id=adventure.id, type="ai", text="t"),
models.Memory(adventure_id=adventure.id, text="m", source_start=0, source_end=0),
models.StoryCard(adventure_id=adventure.id, name="c"),
models.Settings(user_id=user.id),
])
db.commit()
assert sweep(db) == 1
for model in (models.Scenario, models.Adventure, models.Action,
models.Memory, models.StoryCard, models.Settings):
assert db.query(model).count() == 0, f"{model.__name__} rows survived"
def test_one_users_cleanup_does_not_touch_another(db):
keeper = make_user(db, days_idle=1)
keep_adv = models.Adventure(user_id=keeper.id, title="mine")
goner = make_user(db, days_idle=30)
db.add_all([keep_adv, models.Adventure(user_id=goner.id, title="theirs")])
db.commit()
assert sweep(db) == 1
remaining = db.query(models.Adventure).all()
assert [a.title for a in remaining] == ["mine"]
def test_sweep_swallows_errors(monkeypatch):
"""A broken cleanup must not take the app down (same rule as seeding)."""
monkeypatch.setattr(cleanup.auth, "MULTI_USER", True)
monkeypatch.setattr(cleanup, "delete_stale_guests",
lambda *a, **k: (_ for _ in ()).throw(RuntimeError("boom")))
assert cleanup.sweep() == 0

Some files were not shown because too many files have changed in this diff Show More