A hosted demo raises a question a local app never does: is anyone using it, and do they reach the part that matters? `/analytics` answers it — visitors, pages, referrers, countries, devices, which shared scenarios get played, turns and demo-key spend, API and turn errors, and a funnel from visited to played a turn to signed up. Not a third-party script, for reasons specific to this one. The CSP allows `script-src 'self'`, so a tracker means loosening it; adblockers eat the popular ones, which silently biases exactly the technical audience this project gets shown to; and none of them can see the measurement that actually matters here, which is a turn, not a pageview. **A visit is a write and never a read.** After the 189x egress fix it would be perverse to add a feature that reads rows per request, so counts accumulate in a process-local dict and flush every 60s as UPSERTs. Storage is a generic `(day, metric, label) -> hits` counter, so measuring something new later costs a constant rather than a migration, plus one row per visitor per day for the funnel flags. Every dashboard query is a GROUP BY returning tens of rows however much traffic sits behind it; a month reads back in a few kilobytes. The buffer's cost is that a hard restart can lose up to a minute — the flusher also runs on shutdown, and a tier that sleeps when idle sleeps on an empty buffer anyway. **The counters are anonymous; the access log beside them is not, on purpose.** A visitor is `HMAC(secret, "visitor:<user id>")` truncated to 32 chars — one-way, so `analytics_daily` and `analytics_visitor_days` cannot be joined back to `users`, and keyed, so no client can compute one. Story content never reaches that module, and the only content it ever names is a seeded public scenario's title; a player's own titles are theirs. `accesslog.py` is the identifying half and is a separate module writing a separate table so that separation is a property of the code rather than a convention: `access_events` records sessions, sign-ins, registrations and failed attempts with address, email and device, read on a second tab of the same page behind the same gate. Both halves are gated on `AIDND_ANALYTICS_EMAILS`, not `POWER_USERS`. An unmetered tester is not automatically someone who should see the traffic. The route 404s and the nav link is absent for everyone else, the same treatment AI Chat gets; unset in a hosted deploy means nobody sees it, including me. Three things came out of building it that a test would not have suggested. **A failed turn is an HTTP 200 with a bad ending.** The status-code middleware cannot see one, so a demo whose model had started refusing every request would look perfectly healthy from outside. All five SSE error paths in `_generate_turn` now go through a `turn_error()` helper that counts on the way out. Error buckets elsewhere are labelled by the matched route template rather than the requested path — one bucket per endpoint instead of one per adventure id, and, the reason it isn't merely tidier, an unmatched path is entirely attacker-chosen, so labelling by it would let anyone mint rows. **The funnel counts people, not clicks.** A player who starts six adventures is one person who started an adventure. That is the whole reason the per-visitor-day table exists; its flags only ever turn on, and `is_new` is settled by the first write of a visitor's first day. **The tests run on SQLite and production is Neon.** A flush that raises is caught and logged, so a dialect mistake in the UPSERTs would have stayed invisible until the dashboard quietly never filled. `test_the_upserts_compile_for_postgres` compiles both statements against the Postgres dialect without connecting to one. Two things this leans on elsewhere. `limits._client_ip` is now public `client_ip`: the access log needs the same answer, and two functions both deciding which hop is the caller's is how one of them ends up trusting a header it shouldn't. And the cleanup sweeper now starts if *either* job has work — a deployment can keep every guest forever and still want its visitor-day rows aged out. No migration. Both tables are new and `bootstrap()` calls `create_all` on existing databases too, the route `branches` took in Phase 14, so `LATEST_VERSION` is still 64. 497 tests green, frontend lint and build clean, driven by hand against a synthetic 90-day fixture at 1568px. The narrow-screen layout follows the existing 720px block but is unverified: `resize_window` is ignored on a maximized Chrome and `frame-ancestors 'none'` rules out checking it in a sized iframe. Also repaired here: a rename in test_ratelimit_hardening.py had run through the test names themselves, leaving `testclient_ip_*` — still collected by pytest, which is why it passed unnoticed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DfMCsN1KBLsTqMkj5hSgrY
321 lines
13 KiB
Python
321 lines
13 KiB
Python
"""Phase 9 — abuse guards for hosted (multi-user) deployments.
|
|
|
|
Rate limits and row caps are no-ops in local mode: a single local player
|
|
should never be throttled by their own app. Values are hardcoded on purpose —
|
|
generous enough that a legitimate player never notices, tight enough that a
|
|
hostile visitor can't burn the demo key, peg the CPU, or bloat the database.
|
|
"""
|
|
|
|
import json
|
|
import os
|
|
import threading
|
|
import time
|
|
from collections import defaultdict, deque
|
|
|
|
from fastapi import HTTPException, Request
|
|
from sqlalchemy import func
|
|
from sqlalchemy.orm import Session
|
|
|
|
from . import auth, models
|
|
|
|
# ---------- Rate limiting ----------
|
|
# Fixed windows per (scope, caller). In-memory: fine for the single-process
|
|
# deployment this app targets (and the worst case after a restart is a brief
|
|
# extra allowance).
|
|
|
|
# scope -> (max requests, window seconds)
|
|
RATE_LIMITS: dict[str, tuple[int, int]] = {
|
|
"turn": (10, 60), # AI turn generation (demo key also has a daily cap)
|
|
"chat": (30, 60), # AI Chat scratchpad (power users only)
|
|
"script-test": (30, 60), # sandboxed, but each run costs up to 2s CPU
|
|
"connection-test": (10, 60), # outbound HTTP to a user-supplied URL
|
|
"import": (30, 60), # large writes
|
|
"auth": (10, 300), # register/login attempts, per IP
|
|
"guest": (30, 300), # new guest users, per IP (each is a DB row)
|
|
# Pageview beacons. Generous — a real reader clicking around a SPA fires a
|
|
# handful a minute — but low enough that nobody can inflate the traffic
|
|
# numbers faster than they could by actually reloading the page.
|
|
"analytics": (120, 60),
|
|
}
|
|
|
|
_windows: dict[tuple[str, str], deque] = defaultdict(deque)
|
|
_windows_guard = threading.Lock()
|
|
|
|
|
|
# How many proxy hops sit between the app and the real client. On Render (and
|
|
# most PaaS) that's one: the platform's edge appends the connecting IP to the
|
|
# RIGHT of X-Forwarded-For. A client can prepend anything it likes to the left,
|
|
# but it cannot push a value past the edge's own append — so the trustworthy
|
|
# client IP is the (hops)-th entry from the right, NOT uvicorn's leftmost pick.
|
|
# Trusting the leftmost let anyone rotate X-Forwarded-For to mint a fresh
|
|
# rate-limit bucket per request and bypass the auth/guest limits entirely.
|
|
# Override with AIDND_TRUSTED_PROXY_HOPS if the deployment adds more hops.
|
|
TRUSTED_PROXY_HOPS = max(1, int(os.environ.get("AIDND_TRUSTED_PROXY_HOPS", "1") or 1))
|
|
|
|
|
|
def client_ip(request: Request) -> str:
|
|
"""The real client IP, resistant to a spoofed X-Forwarded-For. Takes the
|
|
hop the trusted edge appended (rightmost minus any extra trusted hops);
|
|
falls back to the socket peer when no forwarded header is present
|
|
(local/dev, or a direct connection).
|
|
|
|
Public because the access log needs the same answer, and two functions that
|
|
both decide "which address is the caller's" is how one of them ends up
|
|
trusting a header it shouldn't."""
|
|
forwarded = request.headers.get("x-forwarded-for")
|
|
if forwarded:
|
|
parts = [p.strip() for p in forwarded.split(",") if p.strip()]
|
|
if parts:
|
|
return parts[-min(TRUSTED_PROXY_HOPS, len(parts))]
|
|
return request.client.host if request.client else "unknown"
|
|
|
|
|
|
def rate_limit(scope: str, request: Request, user: models.User | None = None) -> None:
|
|
"""429 when the caller exceeds the scope's window. Keyed per user when one
|
|
is known (accounts survive IP changes), per IP otherwise."""
|
|
if not auth.MULTI_USER:
|
|
return
|
|
limit, window_seconds = RATE_LIMITS[scope]
|
|
key = (scope, f"u{user.id}" if user else f"ip{client_ip(request)}")
|
|
now = time.time()
|
|
with _windows_guard:
|
|
window = _windows[key]
|
|
while window and window[0] < now - window_seconds:
|
|
window.popleft()
|
|
if len(window) >= limit:
|
|
raise HTTPException(
|
|
429, "You're doing that too fast — wait a minute and try again."
|
|
)
|
|
window.append(now)
|
|
if len(_windows) > 10_000:
|
|
_prune(now)
|
|
|
|
|
|
# ---------- Per-account login throttle ----------
|
|
# Defense in depth beside the per-IP `auth` limit: that one can be diluted by a
|
|
# botnet (many real source IPs, one bucket each), so it can't by itself stop a
|
|
# distributed guessing run against a single account. This cap keys on the target
|
|
# email instead of the caller, so guessing ONE account's password stays
|
|
# expensive regardless of how many addresses it comes from. Failures only — a
|
|
# correct password clears the record — and it's a short sliding window, not a
|
|
# hard lock, so a user mistyping a few times recovers on their own in minutes.
|
|
# Tradeoff: an attacker can keep a known account throttled (a nuisance), which
|
|
# is strictly preferable to letting it be brute-forced.
|
|
LOGIN_FAIL_LIMIT = 8 # failed attempts per account...
|
|
LOGIN_FAIL_WINDOW = 900 # ...within this many seconds (15 min)
|
|
|
|
_login_fails: dict[str, deque] = defaultdict(deque)
|
|
_login_guard = threading.Lock()
|
|
|
|
|
|
def check_login_allowed(email: str) -> None:
|
|
"""429 when an account has too many recent failed logins. Call before
|
|
verifying the password so guesses don't even reach the hash."""
|
|
if not auth.MULTI_USER:
|
|
return
|
|
now = time.time()
|
|
with _login_guard:
|
|
window = _login_fails[email]
|
|
while window and window[0] < now - LOGIN_FAIL_WINDOW:
|
|
window.popleft()
|
|
if len(window) >= LOGIN_FAIL_LIMIT:
|
|
raise HTTPException(
|
|
429,
|
|
"Too many failed sign-in attempts for this account — "
|
|
"wait a few minutes and try again.",
|
|
)
|
|
|
|
|
|
def note_login_failure(email: str) -> None:
|
|
"""Record one failed attempt against `email`."""
|
|
if not auth.MULTI_USER:
|
|
return
|
|
now = time.time()
|
|
with _login_guard:
|
|
_login_fails[email].append(now)
|
|
if len(_login_fails) > 10_000: # bound the map on a flood of unique emails
|
|
stale = [
|
|
key for key, window in _login_fails.items()
|
|
if not window or window[-1] < now - LOGIN_FAIL_WINDOW
|
|
]
|
|
for key in stale:
|
|
del _login_fails[key]
|
|
|
|
|
|
def note_login_success(email: str) -> None:
|
|
"""A correct password wipes the account's failure streak."""
|
|
with _login_guard:
|
|
_login_fails.pop(email, None)
|
|
|
|
|
|
def _prune(now: float) -> None:
|
|
"""Drop callers whose whole window has expired (call with guard held) so
|
|
the per-IP dict can't grow without bound."""
|
|
longest = max(seconds for _, seconds in RATE_LIMITS.values())
|
|
stale = [key for key, window in _windows.items()
|
|
if not window or window[-1] < now - longest]
|
|
for key in stale:
|
|
del _windows[key]
|
|
|
|
|
|
# ---------- Per-user row caps ----------
|
|
|
|
MAX_ADVENTURES_PER_USER = 100
|
|
MAX_SCENARIOS_PER_USER = 200
|
|
MAX_SCRIPTS_PER_USER = 200
|
|
MAX_STORY_CARDS_PER_OWNER = 200 # per scenario or adventure
|
|
MAX_MEMORIES_PER_ADVENTURE = 1000
|
|
MAX_ACTIONS_PER_ADVENTURE = 5000
|
|
# Phase 14, SP6. A branch per divergence somebody built a story on, so a tree
|
|
# with more of them than a story has turns is a file, not a game. Import-only
|
|
# for now: forking is a POST that adds one row and has no cap of its own, and
|
|
# the cap that matters there is `MAX_ACTIONS_PER_ADVENTURE` above it.
|
|
MAX_BRANCHES_PER_ADVENTURE = 1000
|
|
|
|
|
|
def check_row_cap(
|
|
kind: str,
|
|
db: Session,
|
|
user: models.User,
|
|
*,
|
|
adventure: models.Adventure | None = None,
|
|
scenario_id: int | None = None,
|
|
adventure_id: int | None = None,
|
|
) -> None:
|
|
"""409 with a friendly message when creating one more row of `kind` would
|
|
exceed its cap. Ownership of the passed scenario/adventure has already
|
|
been checked by the caller."""
|
|
if not auth.MULTI_USER:
|
|
return
|
|
if kind == "adventures":
|
|
count = _count(db, models.Adventure, models.Adventure.user_id == user.id)
|
|
cap, subject, hint = (
|
|
MAX_ADVENTURES_PER_USER, "adventures",
|
|
"delete one you no longer play to make room",
|
|
)
|
|
elif kind == "scenarios":
|
|
count = _count(db, models.Scenario, models.Scenario.user_id == user.id)
|
|
cap, subject, hint = (
|
|
MAX_SCENARIOS_PER_USER, "scenarios", "delete one to make room"
|
|
)
|
|
elif kind == "scripts":
|
|
count = _count(db, models.Script, models.Script.user_id == user.id)
|
|
cap, subject, hint = (
|
|
MAX_SCRIPTS_PER_USER, "scripts", "delete one to make room"
|
|
)
|
|
elif kind == "story_cards":
|
|
owner_filter = (
|
|
models.StoryCard.scenario_id == scenario_id
|
|
if scenario_id is not None
|
|
else models.StoryCard.adventure_id == adventure_id
|
|
)
|
|
count = _count(db, models.StoryCard, owner_filter)
|
|
cap, subject, hint = (
|
|
MAX_STORY_CARDS_PER_OWNER, "story cards here", "delete one to make room"
|
|
)
|
|
elif kind == "memories":
|
|
count = _count(db, models.Memory, models.Memory.adventure_id == adventure.id)
|
|
cap, subject, hint = (
|
|
MAX_MEMORIES_PER_ADVENTURE, "memories in this adventure",
|
|
"delete some to make room",
|
|
)
|
|
elif kind == "actions":
|
|
# Every action of the adventure — the whole tree, not the path being
|
|
# played. That is the number that costs storage, and nothing is ever
|
|
# auto-pruned, so it is the right one to cap on. It does mean a heavily
|
|
# branched adventure reaches the cap while its *story* is shorter than
|
|
# the cap, which is why the message counts "actions in this adventure"
|
|
# rather than turns.
|
|
count = _count(db, models.Action, models.Action.adventure_id == adventure.id)
|
|
cap, subject, hint = (
|
|
MAX_ACTIONS_PER_ADVENTURE, "actions in this adventure",
|
|
"export it and continue in a new adventure",
|
|
)
|
|
else: # pragma: no cover — programming error, not user input
|
|
raise ValueError(f"Unknown row cap kind: {kind}")
|
|
if count >= cap:
|
|
raise HTTPException(409, f"You've reached the limit of {cap} {subject} — {hint}.")
|
|
|
|
|
|
def _count(db: Session, model, condition) -> int:
|
|
return db.query(func.count(model.id)).filter(condition).scalar() or 0
|
|
|
|
|
|
_BUNDLE_LIST_CAPS = {
|
|
"story_cards": MAX_STORY_CARDS_PER_OWNER,
|
|
"memories": MAX_MEMORIES_PER_ADVENTURE,
|
|
"actions": MAX_ACTIONS_PER_ADVENTURE,
|
|
"branches": MAX_BRANCHES_PER_ADVENTURE,
|
|
}
|
|
|
|
|
|
def check_bundle_lists(**lists) -> None:
|
|
"""409 when an import bundle's lists exceed the same caps live creation
|
|
enforces (kwargs: story_cards=, memories=, actions=, branches=)."""
|
|
if not auth.MULTI_USER:
|
|
return
|
|
for name, value in lists.items():
|
|
cap = _BUNDLE_LIST_CAPS[name]
|
|
if isinstance(value, list) and len(value) > cap:
|
|
noun = name.replace("_", " ")
|
|
raise HTTPException(
|
|
409, f"This file contains {len(value)} {noun} — the limit is {cap}."
|
|
)
|
|
|
|
|
|
# ---------- Request body size ----------
|
|
# Generous enough for the biggest legitimate payload (an adventure export with
|
|
# thousands of actions), applied in every mode — no honest request comes close.
|
|
|
|
MAX_BODY_BYTES = 2 * 1024 * 1024
|
|
MAX_IMPORT_BODY_BYTES = 20 * 1024 * 1024
|
|
|
|
|
|
class BodySizeLimitMiddleware:
|
|
"""Rejects oversized request bodies by declared Content-Length. Pure ASGI
|
|
(not BaseHTTPMiddleware) so SSE responses stream through untouched.
|
|
Chunked uploads without a length are refused — every real client of this
|
|
API (browser fetch, curl with a file) sends Content-Length."""
|
|
|
|
def __init__(self, app):
|
|
self.app = app
|
|
|
|
async def __call__(self, scope, receive, send):
|
|
if scope["type"] == "http" and scope.get("method") in ("POST", "PUT", "PATCH"):
|
|
headers = {k.decode("latin-1").lower(): v.decode("latin-1")
|
|
for k, v in scope.get("headers", [])}
|
|
limit = (
|
|
MAX_IMPORT_BODY_BYTES
|
|
if scope.get("path", "").endswith("/import")
|
|
else MAX_BODY_BYTES
|
|
)
|
|
length = headers.get("content-length")
|
|
problem = None
|
|
if length is None:
|
|
if "chunked" in headers.get("transfer-encoding", "").lower():
|
|
problem = (411, "Content-Length is required.")
|
|
else:
|
|
try:
|
|
if int(length) > limit:
|
|
problem = (
|
|
413,
|
|
f"Request too large (limit {limit // (1024 * 1024)} MB).",
|
|
)
|
|
except ValueError:
|
|
problem = (400, "Invalid Content-Length.")
|
|
if problem:
|
|
await _send_json_error(send, *problem)
|
|
return
|
|
await self.app(scope, receive, send)
|
|
|
|
|
|
async def _send_json_error(send, status: int, detail: str) -> None:
|
|
body = json.dumps({"detail": detail}).encode()
|
|
await send({
|
|
"type": "http.response.start",
|
|
"status": status,
|
|
"headers": [(b"content-type", b"application/json"),
|
|
(b"content-length", str(len(body)).encode())],
|
|
})
|
|
await send({"type": "http.response.body", "body": body})
|