Count the visits, and say whether anyone got anywhere
A hosted demo raises a question a local app never does: is anyone using it, and do they reach the part that matters? `/analytics` answers it — visitors, pages, referrers, countries, devices, which shared scenarios get played, turns and demo-key spend, API and turn errors, and a funnel from visited to played a turn to signed up. Not a third-party script, for reasons specific to this one. The CSP allows `script-src 'self'`, so a tracker means loosening it; adblockers eat the popular ones, which silently biases exactly the technical audience this project gets shown to; and none of them can see the measurement that actually matters here, which is a turn, not a pageview. **A visit is a write and never a read.** After the 189x egress fix it would be perverse to add a feature that reads rows per request, so counts accumulate in a process-local dict and flush every 60s as UPSERTs. Storage is a generic `(day, metric, label) -> hits` counter, so measuring something new later costs a constant rather than a migration, plus one row per visitor per day for the funnel flags. Every dashboard query is a GROUP BY returning tens of rows however much traffic sits behind it; a month reads back in a few kilobytes. The buffer's cost is that a hard restart can lose up to a minute — the flusher also runs on shutdown, and a tier that sleeps when idle sleeps on an empty buffer anyway. **The counters are anonymous; the access log beside them is not, on purpose.** A visitor is `HMAC(secret, "visitor:<user id>")` truncated to 32 chars — one-way, so `analytics_daily` and `analytics_visitor_days` cannot be joined back to `users`, and keyed, so no client can compute one. Story content never reaches that module, and the only content it ever names is a seeded public scenario's title; a player's own titles are theirs. `accesslog.py` is the identifying half and is a separate module writing a separate table so that separation is a property of the code rather than a convention: `access_events` records sessions, sign-ins, registrations and failed attempts with address, email and device, read on a second tab of the same page behind the same gate. Both halves are gated on `AIDND_ANALYTICS_EMAILS`, not `POWER_USERS`. An unmetered tester is not automatically someone who should see the traffic. The route 404s and the nav link is absent for everyone else, the same treatment AI Chat gets; unset in a hosted deploy means nobody sees it, including me. Three things came out of building it that a test would not have suggested. **A failed turn is an HTTP 200 with a bad ending.** The status-code middleware cannot see one, so a demo whose model had started refusing every request would look perfectly healthy from outside. All five SSE error paths in `_generate_turn` now go through a `turn_error()` helper that counts on the way out. Error buckets elsewhere are labelled by the matched route template rather than the requested path — one bucket per endpoint instead of one per adventure id, and, the reason it isn't merely tidier, an unmatched path is entirely attacker-chosen, so labelling by it would let anyone mint rows. **The funnel counts people, not clicks.** A player who starts six adventures is one person who started an adventure. That is the whole reason the per-visitor-day table exists; its flags only ever turn on, and `is_new` is settled by the first write of a visitor's first day. **The tests run on SQLite and production is Neon.** A flush that raises is caught and logged, so a dialect mistake in the UPSERTs would have stayed invisible until the dashboard quietly never filled. `test_the_upserts_compile_for_postgres` compiles both statements against the Postgres dialect without connecting to one. Two things this leans on elsewhere. `limits._client_ip` is now public `client_ip`: the access log needs the same answer, and two functions both deciding which hop is the caller's is how one of them ends up trusting a header it shouldn't. And the cleanup sweeper now starts if *either* job has work — a deployment can keep every guest forever and still want its visitor-day rows aged out. No migration. Both tables are new and `bootstrap()` calls `create_all` on existing databases too, the route `branches` took in Phase 14, so `LATEST_VERSION` is still 64. 497 tests green, frontend lint and build clean, driven by hand against a synthetic 90-day fixture at 1568px. The narrow-screen layout follows the existing 720px block but is unverified: `resize_window` is ignored on a maximized Chrome and `frame-ancestors 'none'` rules out checking it in a sized iframe. Also repaired here: a rename in test_ratelimit_hardening.py had run through the test names themselves, leaving `testclient_ip_*` — still collected by pytest, which is why it passed unnoticed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DfMCsN1KBLsTqMkj5hSgrY
This commit is contained in:
co-authored by
Claude Opus 5
parent
3b9e6b3d50
commit
041f9e25f3
@@ -151,7 +151,7 @@ player input
|
|||||||
|
|
||||||
```
|
```
|
||||||
frontend/ React + Vite SPA ──HTTP/SSE──► backend/ FastAPI
|
frontend/ React + Vite SPA ──HTTP/SSE──► backend/ FastAPI
|
||||||
├─ routers/ auth, scenarios, adventures, story cards, scripts, chat, settings, debug
|
├─ routers/ auth, scenarios, adventures, story cards, scripts, chat, settings, analytics, debug
|
||||||
├─ models.py SQLAlchemy: User, Scenario, Adventure, Branch, Action, StoryCard, Script, Settings, Memory
|
├─ models.py SQLAlchemy: User, Scenario, Adventure, Branch, Action, StoryCard, Script, Settings, Memory
|
||||||
├─ migrations.py hand-rolled, versioned via PRAGMA user_version (64 and counting)
|
├─ migrations.py hand-rolled, versioned via PRAGMA user_version (64 and counting)
|
||||||
├─ auth.py guest/registered users, sessions, shared demo key
|
├─ auth.py guest/registered users, sessions, shared demo key
|
||||||
@@ -162,6 +162,7 @@ frontend/ React + Vite SPA ──HTTP/SSE──► backend/ FastAPI
|
|||||||
├─ worldstate/ the stat engine: clamps, cooldowns, bands, milestones
|
├─ worldstate/ the stat engine: clamps, cooldowns, bands, milestones
|
||||||
├─ scripting/ quickjs sandbox + AI Dungeon API surface
|
├─ scripting/ quickjs sandbox + AI Dungeon API surface
|
||||||
├─ memorybank.py auto-summarization + embedding retrieval
|
├─ memorybank.py auto-summarization + embedding retrieval
|
||||||
|
├─ analytics.py buffered visit counters + the owner's dashboard query
|
||||||
├─ bundle.py the export/import formats, v2 (tree) and a v1 reader
|
├─ bundle.py the export/import formats, v2 (tree) and a v1 reader
|
||||||
├─ providers/ OpenAI-compatible adapter, streaming
|
├─ providers/ OpenAI-compatible adapter, streaming
|
||||||
└─ data.db SQLite (path overridable via AIDND_DB_PATH)
|
└─ data.db SQLite (path overridable via AIDND_DB_PATH)
|
||||||
@@ -172,7 +173,7 @@ development Vite proxies `/api` to FastAPI.
|
|||||||
|
|
||||||
## Tests
|
## Tests
|
||||||
|
|
||||||
440 backend tests — unit plus full HTTP integration through the real quickjs scripting engine,
|
497 backend tests — unit plus full HTTP integration through the real quickjs scripting engine,
|
||||||
with the LLM provider mocked. CI runs them on every push, alongside the frontend lint/build and
|
with the LLM provider mocked. CI runs them on every push, alongside the frontend lint/build and
|
||||||
a Docker image build.
|
a Docker image build.
|
||||||
|
|
||||||
@@ -202,6 +203,21 @@ the most interesting engineering in the repo:
|
|||||||
stay cheap because the lineage is windowed like the history is, so the number of SQL clauses is
|
stay cheap because the lineage is windowed like the history is, so the number of SQL clauses is
|
||||||
bounded by the context window rather than by the number of forks.
|
bounded by the context window rather than by the number of forks.
|
||||||
|
|
||||||
|
## Visit analytics
|
||||||
|
|
||||||
|
The hosted demo keeps its own analytics: an owner-only dashboard at `/analytics` showing
|
||||||
|
traffic, which shared scenarios get played, turns and demo-key spend, errors, and a funnel
|
||||||
|
from *visited* to *played a turn* to *signed up*. It is visible only to the emails listed in
|
||||||
|
`AIDND_ANALYTICS_EMAILS`, and the route 404s for everyone else.
|
||||||
|
|
||||||
|
Built into the app rather than bolted on with a third-party script, for reasons specific to
|
||||||
|
this one: the CSP allows `script-src 'self'`, adblockers eat the popular trackers, and none
|
||||||
|
of them can see the measurement that actually matters here — a turn. Counts are aggregated
|
||||||
|
in memory and flushed as UPSERTs, so a visit is a write and never a read, and every dashboard
|
||||||
|
query is a `GROUP BY` returning tens of rows however much traffic sits behind it. That last
|
||||||
|
part is not incidental; see the egress note above for what reading rows per request costs on
|
||||||
|
this stack.
|
||||||
|
|
||||||
## Deploy (Render)
|
## Deploy (Render)
|
||||||
|
|
||||||
The repo ships a [`render.yaml`](render.yaml) blueprint: one Docker web service that
|
The repo ships a [`render.yaml`](render.yaml) blueprint: one Docker web service that
|
||||||
@@ -211,7 +227,8 @@ serves the SPA and API same-origin, backed by external [Neon](https://neon.tech)
|
|||||||
1. Create a **Neon** project and copy its pooled connection string.
|
1. Create a **Neon** project and copy its pooled connection string.
|
||||||
2. In Render: **New → Blueprint**, point it at this repo. Render reads `render.yaml`.
|
2. In Render: **New → Blueprint**, point it at this repo. Render reads `render.yaml`.
|
||||||
3. Fill the secrets it prompts for (`sync: false` vars): `AIDND_DATABASE_URL` (the Neon
|
3. Fill the secrets it prompts for (`sync: false` vars): `AIDND_DATABASE_URL` (the Neon
|
||||||
string) and, to offer a no-signup demo, `AIDND_DEMO_API_KEY` / `AIDND_DEMO_MODELS`.
|
string); to offer a no-signup demo, `AIDND_DEMO_API_KEY` / `AIDND_DEMO_MODELS`; and to
|
||||||
|
see the Visitors dashboard, `AIDND_ANALYTICS_EMAILS` (your own account's email).
|
||||||
`AIDND_SECRET_KEY` is generated automatically and kept stable across deploys.
|
`AIDND_SECRET_KEY` is generated automatically and kept stable across deploys.
|
||||||
4. Deploy. Pushes to `main` auto-deploy thereafter. Health check: `/api/health`.
|
4. Deploy. Pushes to `main` auto-deploy thereafter. Health check: `/api/health`.
|
||||||
|
|
||||||
|
|||||||
@@ -66,6 +66,20 @@ AIDND_DEMO_TURNS_PER_DAY=
|
|||||||
# Local (single-user) installs are always treated as power users.
|
# Local (single-user) installs are always treated as power users.
|
||||||
AIDND_POWER_USERS=
|
AIDND_POWER_USERS=
|
||||||
|
|
||||||
|
# --- Visit analytics ---
|
||||||
|
# Comma-separated emails allowed to see the Visitors dashboard (/analytics) and
|
||||||
|
# its nav link. Deliberately separate from AIDND_POWER_USERS: a trusted tester
|
||||||
|
# gets unmetered turns, which is no reason to hand them the traffic numbers.
|
||||||
|
# Unset = nobody sees it in a hosted deploy. Local installs always can, and are
|
||||||
|
# the only mode where the viewer's own visits are still counted (excluding them
|
||||||
|
# would leave the page permanently empty on the machine it's developed on).
|
||||||
|
# Collection itself is always on; only the dashboard is gated.
|
||||||
|
AIDND_ANALYTICS_EMAILS=
|
||||||
|
# Days to keep the one-row-per-visitor-per-day table that makes the funnel
|
||||||
|
# count people rather than clicks. The daily counters are aggregate and kept
|
||||||
|
# forever. Default: 400. Set 0 to keep visitor-days forever.
|
||||||
|
AIDND_ANALYTICS_RETENTION_DAYS=
|
||||||
|
|
||||||
# --- Guest retention (only active when AIDND_MULTI_USER=1) ---
|
# --- Guest retention (only active when AIDND_MULTI_USER=1) ---
|
||||||
# Every first visit mints a guest account, so a public demo collects one row
|
# Every first visit mints a guest account, so a public demo collects one row
|
||||||
# per visitor. A guest with no activity for this many days is deleted along
|
# per visitor. A guest with no activity for this many days is deleted along
|
||||||
|
|||||||
@@ -0,0 +1,160 @@
|
|||||||
|
"""The access log: who arrived, when, and from where.
|
||||||
|
|
||||||
|
The deliberate opposite of analytics.py. That module counts and stores nothing
|
||||||
|
that points at a person; this one records addresses, email addresses and
|
||||||
|
devices, because an access log that cannot identify the access is not an access
|
||||||
|
log. They are kept in separate modules and separate tables on purpose — the
|
||||||
|
anonymity of the counters is then a property of the code rather than of a
|
||||||
|
convention someone has to remember.
|
||||||
|
|
||||||
|
Owner-only, and never shown to the people it records.
|
||||||
|
|
||||||
|
Four kinds of row:
|
||||||
|
|
||||||
|
- `session` a browser that has a session made a request — for a guest,
|
||||||
|
their first visit;
|
||||||
|
- `login` an existing account signed in;
|
||||||
|
- `register` a guest upgraded to an account;
|
||||||
|
- `login_failed` a password attempt that didn't match, with the address tried.
|
||||||
|
|
||||||
|
Session rows are the only ones that need thinning: `/auth/me` runs on every
|
||||||
|
page load, and a row per load would be noise rather than a log. One is written
|
||||||
|
when the day or the address changes for that user, which is the granularity a
|
||||||
|
log is actually read at — "seen on the 3rd from 1.2.3.4" — and it still catches
|
||||||
|
someone moving networks mid-day.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import threading
|
||||||
|
|
||||||
|
from sqlalchemy import desc, or_, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from . import analytics, models
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
SESSION = "session"
|
||||||
|
LOGIN = "login"
|
||||||
|
REGISTER = "register"
|
||||||
|
LOGIN_FAILED = "login_failed"
|
||||||
|
|
||||||
|
MAX_UA = 200
|
||||||
|
|
||||||
|
# user id -> (day, ip) of the last session row written for them. Process-local
|
||||||
|
# like the rate limiter's windows, and for the same reason: this is a single
|
||||||
|
# process, and the worst case after a restart is one redundant row per user.
|
||||||
|
_last_session: dict[int, tuple[str, str]] = {}
|
||||||
|
_guard = threading.Lock()
|
||||||
|
_MAX_TRACKED = 10_000
|
||||||
|
|
||||||
|
|
||||||
|
def _client_ip(request) -> str:
|
||||||
|
# Deferred: limits imports auth, which is imported by the routers that call
|
||||||
|
# this, so a module-level import here would close the loop. The spoof
|
||||||
|
# resistance lives there and must not be reimplemented — a second, laxer
|
||||||
|
# copy of "what is the client's address" is exactly how one of them ends up
|
||||||
|
# trusting a header it shouldn't.
|
||||||
|
from . import limits
|
||||||
|
|
||||||
|
return limits.client_ip(request)
|
||||||
|
|
||||||
|
|
||||||
|
def describe(user: models.User) -> str:
|
||||||
|
"""How a user is named in the log. Guests have no email, and their id is
|
||||||
|
the only handle anyone has for them. The third case is a local install's
|
||||||
|
implicit single user, which is also email-less but is the operator rather
|
||||||
|
than a visitor — calling that one "Guest #1" would be a small lie in the
|
||||||
|
one row they are certain to read."""
|
||||||
|
if user.email:
|
||||||
|
return user.email
|
||||||
|
return f"Guest #{user.id}" if user.is_guest else f"Local user #{user.id}"
|
||||||
|
|
||||||
|
|
||||||
|
def _country(request) -> str:
|
||||||
|
"""The edge's country header, or "" when there isn't one. Blank rather than
|
||||||
|
the counters' "(unknown)" label: a table column reads better as an em dash
|
||||||
|
than as a word, and an empty string is the honest value for "not known"."""
|
||||||
|
country = analytics.country_of(request.headers)
|
||||||
|
return "" if country == analytics.UNKNOWN else country
|
||||||
|
|
||||||
|
|
||||||
|
def record(
|
||||||
|
db: Session,
|
||||||
|
kind: str,
|
||||||
|
request,
|
||||||
|
*,
|
||||||
|
user: models.User | None = None,
|
||||||
|
who: str | None = None,
|
||||||
|
) -> None:
|
||||||
|
"""Write one row. Never raises: the log watches sign-in, it doesn't guard
|
||||||
|
it, and a logging failure must not be able to lock anyone out."""
|
||||||
|
try:
|
||||||
|
event = models.AccessEvent(
|
||||||
|
kind=kind,
|
||||||
|
user_id=user.id if user is not None else None,
|
||||||
|
who=(who if who is not None else describe(user) if user else "")[:320],
|
||||||
|
is_guest=bool(user.is_guest) if user is not None else False,
|
||||||
|
ip=_client_ip(request)[:45],
|
||||||
|
country=_country(request),
|
||||||
|
device=analytics.device_of(request.headers.get("user-agent", "")),
|
||||||
|
user_agent=(request.headers.get("user-agent") or "")[:MAX_UA],
|
||||||
|
)
|
||||||
|
db.add(event)
|
||||||
|
db.commit()
|
||||||
|
except Exception: # pragma: no cover - defensive
|
||||||
|
db.rollback()
|
||||||
|
logger.exception("Access log write failed; continuing.")
|
||||||
|
|
||||||
|
|
||||||
|
def note_session(db: Session, user: models.User, request) -> None:
|
||||||
|
"""A session made a request. Thinned to one row per day per address."""
|
||||||
|
try:
|
||||||
|
today = analytics._today()
|
||||||
|
ip = _client_ip(request)
|
||||||
|
with _guard:
|
||||||
|
if _last_session.get(user.id) == (today, ip):
|
||||||
|
return
|
||||||
|
_last_session[user.id] = (today, ip)
|
||||||
|
if len(_last_session) > _MAX_TRACKED:
|
||||||
|
# Nothing here is worth persisting; dropping the map costs at
|
||||||
|
# most one extra row per active user.
|
||||||
|
_last_session.clear()
|
||||||
|
_last_session[user.id] = (today, ip)
|
||||||
|
except Exception: # pragma: no cover - defensive
|
||||||
|
logger.exception("Access log session check failed; continuing.")
|
||||||
|
return
|
||||||
|
record(db, SESSION, request, user=user)
|
||||||
|
|
||||||
|
|
||||||
|
def recent(
|
||||||
|
db: Session,
|
||||||
|
*,
|
||||||
|
limit: int = 50,
|
||||||
|
before_id: int | None = None,
|
||||||
|
kind: str | None = None,
|
||||||
|
query: str | None = None,
|
||||||
|
) -> dict:
|
||||||
|
"""A page of the log, newest first.
|
||||||
|
|
||||||
|
Anchored on a row id rather than an offset, like the story pager: rows keep
|
||||||
|
arriving while it is being read, and an offset would shift the page under
|
||||||
|
whoever is reading it.
|
||||||
|
"""
|
||||||
|
statement = select(models.AccessEvent).order_by(desc(models.AccessEvent.id))
|
||||||
|
if before_id is not None:
|
||||||
|
statement = statement.where(models.AccessEvent.id < before_id)
|
||||||
|
if kind:
|
||||||
|
statement = statement.where(models.AccessEvent.kind == kind)
|
||||||
|
if query:
|
||||||
|
like = f"%{query.strip()}%"
|
||||||
|
statement = statement.where(or_(
|
||||||
|
models.AccessEvent.who.ilike(like),
|
||||||
|
models.AccessEvent.ip.ilike(like),
|
||||||
|
models.AccessEvent.country.ilike(like),
|
||||||
|
))
|
||||||
|
# One extra row answers "is there more" without a second COUNT over the
|
||||||
|
# whole table.
|
||||||
|
rows = list(db.scalars(statement.limit(limit + 1)))
|
||||||
|
has_more = len(rows) > limit
|
||||||
|
return {"events": rows[:limit], "has_more": has_more}
|
||||||
@@ -0,0 +1,556 @@
|
|||||||
|
"""Visit analytics for the hosted demo.
|
||||||
|
|
||||||
|
A small self-hosted counter answering "did anyone visit, and did they play?",
|
||||||
|
built into the app rather than bolted on with a third-party script: the CSP in
|
||||||
|
main.py allows scripts from 'self' only, adblockers eat the popular trackers,
|
||||||
|
and none of them can see the things actually worth knowing here (turns taken,
|
||||||
|
demo-key spend, which seeded scenario people pick).
|
||||||
|
|
||||||
|
Three rules shaped it:
|
||||||
|
|
||||||
|
1. **Nothing personal is stored.** No IP addresses, no user agents, no user
|
||||||
|
ids, no title of anything a player wrote. A visitor appears only as an HMAC
|
||||||
|
of their user id — one-way and salted with the app's secret key, so these
|
||||||
|
tables cannot be joined back to an account even by someone holding the
|
||||||
|
database. Story content never reaches this module at all. What a *specific*
|
||||||
|
person did is deliberately unanswerable; only totals are.
|
||||||
|
2. **Egress is the budget.** Neon bills for bytes leaving the database and this
|
||||||
|
project has already paid for forgetting that once. So counts are aggregated
|
||||||
|
in memory and flushed as UPSERTs — a visit is a write, never a read — and
|
||||||
|
every dashboard query is a GROUP BY returning tens of rows, never per-visit
|
||||||
|
rows. A month of traffic costs a few kilobytes to read back.
|
||||||
|
3. **The numbers are the server's, not the browser's.** The client reports one
|
||||||
|
thing: which page was viewed. Everything that *means* something ("a turn
|
||||||
|
happened", "an account was created") is recorded by the code that does it,
|
||||||
|
where it can be neither faked by a stranger nor blocked by an extension.
|
||||||
|
|
||||||
|
Storage is two tables, both bounded. `analytics_daily` is one row per (day,
|
||||||
|
metric, label) counter — a few dozen a day. `analytics_visitor_days` is one row
|
||||||
|
per visitor per day carrying the funnel flags, which is what makes the funnel
|
||||||
|
count *people* rather than clicks; it is the only table that grows with traffic
|
||||||
|
and cleanup ages it out.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import hmac
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import threading
|
||||||
|
from datetime import timedelta
|
||||||
|
from hashlib import sha256
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
from sqlalchemy import case, func, or_, select
|
||||||
|
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||||
|
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from . import models, security
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# ---------- Metrics ----------
|
||||||
|
# `metric` is the family, `label` the bucket within it. One generic counter
|
||||||
|
# table beats a column per thing measured: adding a new question later is a
|
||||||
|
# constant, not a migration.
|
||||||
|
|
||||||
|
M_PAGE = "pageview"
|
||||||
|
M_EVENT = "event"
|
||||||
|
M_REFERRER = "referrer"
|
||||||
|
M_DEVICE = "device"
|
||||||
|
M_COUNTRY = "country"
|
||||||
|
M_SCENARIO = "scenario" # which seeded/public scenario got played
|
||||||
|
M_ERROR = "error" # "<status> <route>" for 4xx/5xx on /api
|
||||||
|
|
||||||
|
EV_SCENARIO_OPEN = "scenario_opened"
|
||||||
|
EV_ADVENTURE = "adventure_created"
|
||||||
|
EV_IMPORT = "adventure_imported"
|
||||||
|
EV_TURN = "turn"
|
||||||
|
EV_DEMO_TURN = "demo_turn" # a turn billed to the shared demo key
|
||||||
|
EV_TURN_ERROR = "turn_error"
|
||||||
|
EV_SIGNUP = "signup"
|
||||||
|
EV_LOGIN = "login"
|
||||||
|
|
||||||
|
# Events that are also funnel steps: recording one flips a flag on the
|
||||||
|
# visitor's row for the day, so the funnel counts distinct people-days instead
|
||||||
|
# of repeat clicks. The name -> column map is the whole definition of the
|
||||||
|
# funnel; the dashboard reads it back in this order.
|
||||||
|
FUNNEL_FLAGS = {
|
||||||
|
EV_SCENARIO_OPEN: "opened",
|
||||||
|
EV_ADVENTURE: "created",
|
||||||
|
EV_TURN: "played",
|
||||||
|
EV_SIGNUP: "signed_up",
|
||||||
|
}
|
||||||
|
|
||||||
|
OTHER = "(other)"
|
||||||
|
NONE_LABEL = "(direct)"
|
||||||
|
UNKNOWN = "(unknown)"
|
||||||
|
|
||||||
|
# ---------- Bounds ----------
|
||||||
|
# Everything below exists so a hostile visitor can add rows to these tables no
|
||||||
|
# faster than an honest one. The only label a client can influence is the
|
||||||
|
# referrer, and these caps together mean the worst it can do is fill one day's
|
||||||
|
# referrer list with junk and then be folded into "(other)".
|
||||||
|
|
||||||
|
MAX_LABEL_LEN = 80
|
||||||
|
MAX_LABELS_PER_METRIC = 200 # distinct labels per metric per day, then OTHER
|
||||||
|
MAX_PENDING = 4000 # buffered entries before an inline flush
|
||||||
|
FLUSH_INTERVAL_SECONDS = 60
|
||||||
|
|
||||||
|
# How long the per-visitor-day rows are kept. The daily counters are tiny and
|
||||||
|
# kept forever; these are the ones that scale with traffic. A visitor whose
|
||||||
|
# last visit falls off the end counts as new again — a fair trade at this
|
||||||
|
# horizon, and it keeps the table from being a permanent record of anyone.
|
||||||
|
RETENTION_DAYS = int(os.environ.get("AIDND_ANALYTICS_RETENTION_DAYS", "400") or 400)
|
||||||
|
|
||||||
|
_HOST_OK = re.compile(r"^[a-z0-9.-]+$")
|
||||||
|
_COUNTRY_OK = re.compile(r"^[A-Z]{2}$")
|
||||||
|
_NUMERIC_SEGMENT = re.compile(r"^\d+$")
|
||||||
|
|
||||||
|
# SPA routes, in the shape the dashboard should show them. Anything else a
|
||||||
|
# client claims to have viewed becomes OTHER, so the page list can neither be
|
||||||
|
# polluted nor accidentally record which adventure someone is reading.
|
||||||
|
KNOWN_ROUTES = {
|
||||||
|
"/", "/adventures", "/scenarios", "/scenarios/:id", "/play/:id",
|
||||||
|
"/scripts", "/scripts/:id", "/settings", "/chat", "/analytics",
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------- In-process buffer ----------
|
||||||
|
# Single-process deployment (same assumption as limits.py), so a plain dict
|
||||||
|
# under a lock is the whole design. Losing up to a minute of counts to a hard
|
||||||
|
# restart is acceptable for traffic numbers, and the flusher also runs on
|
||||||
|
# shutdown; on Render's free tier the service is idle when it sleeps, so the
|
||||||
|
# buffer it sleeps on is empty anyway.
|
||||||
|
|
||||||
|
_counts: dict[tuple[str, str, str], int] = {}
|
||||||
|
_visits: dict[tuple[str, str], set[str]] = {} # (day, visitor) -> flags
|
||||||
|
_labels_seen: dict[tuple[str, str], set[str]] = {} # (day, metric) -> labels
|
||||||
|
_guard = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
|
def _today() -> str:
|
||||||
|
return models.utcnow().date().isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def record(metric: str, label: str = "", *, n: int = 1) -> None:
|
||||||
|
"""Add `n` to one counter. Never raises: analytics must not be able to
|
||||||
|
fail a request it is only watching."""
|
||||||
|
try:
|
||||||
|
day = _today()
|
||||||
|
label = (label or "").strip()[:MAX_LABEL_LEN]
|
||||||
|
with _guard:
|
||||||
|
seen = _labels_seen.setdefault((day, metric), set())
|
||||||
|
if label not in seen:
|
||||||
|
if len(seen) >= MAX_LABELS_PER_METRIC:
|
||||||
|
label = OTHER
|
||||||
|
else:
|
||||||
|
seen.add(label)
|
||||||
|
key = (day, metric, label)
|
||||||
|
_counts[key] = _counts.get(key, 0) + n
|
||||||
|
pending = len(_counts) + len(_visits)
|
||||||
|
except Exception: # pragma: no cover - defensive
|
||||||
|
logger.exception("Analytics counter failed; continuing.")
|
||||||
|
return
|
||||||
|
if pending >= MAX_PENDING:
|
||||||
|
flush()
|
||||||
|
|
||||||
|
|
||||||
|
def visitor_id(user: models.User) -> str:
|
||||||
|
"""A stable but one-way handle for one visitor.
|
||||||
|
|
||||||
|
HMAC of the user id under the app's secret key. Stable, so a returning
|
||||||
|
visitor can be told from a new one; one-way, so nothing in the analytics
|
||||||
|
tables points back at an account; keyed, so a client cannot compute one and
|
||||||
|
claim to be somebody else. One consequence worth knowing: rotating
|
||||||
|
AIDND_SECRET_KEY makes every returning visitor look new again.
|
||||||
|
"""
|
||||||
|
digest = hmac.new(security.SECRET_KEY, f"visitor:{user.id}".encode(), sha256)
|
||||||
|
return digest.hexdigest()[:32]
|
||||||
|
|
||||||
|
|
||||||
|
def record_visit(user: models.User | None, *, flag: str | None = None) -> None:
|
||||||
|
"""Note that this visitor was here today, optionally flipping one funnel
|
||||||
|
flag. A no-op without a user: a page loaded before a session exists still
|
||||||
|
counts as a pageview, just not as a person."""
|
||||||
|
if user is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
with _guard:
|
||||||
|
flags = _visits.setdefault((_today(), visitor_id(user)), set())
|
||||||
|
if flag:
|
||||||
|
flags.add(flag)
|
||||||
|
except Exception: # pragma: no cover - defensive
|
||||||
|
logger.exception("Analytics visit failed; continuing.")
|
||||||
|
|
||||||
|
|
||||||
|
def record_event(name: str, user: models.User | None = None) -> None:
|
||||||
|
"""One thing that happened: counted, and — if it is a funnel step —
|
||||||
|
credited to the visitor's day. This is the call sites' whole interface."""
|
||||||
|
record(M_EVENT, name)
|
||||||
|
record_visit(user, flag=FUNNEL_FLAGS.get(name))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Normalizing what the browser reports ----------
|
||||||
|
|
||||||
|
def normalize_route(path: str) -> str:
|
||||||
|
"""A client-reported path, reduced to one of KNOWN_ROUTES.
|
||||||
|
|
||||||
|
Numeric segments become ":id" — both to bound the label count and because
|
||||||
|
*which* adventure someone opened is their business, not a statistic.
|
||||||
|
"""
|
||||||
|
path = (path or "/").split("?")[0].split("#")[0]
|
||||||
|
if not path.startswith("/"):
|
||||||
|
path = "/" + path
|
||||||
|
if len(path) > 1:
|
||||||
|
path = path.rstrip("/")
|
||||||
|
parts = [":id" if _NUMERIC_SEGMENT.match(p) else p for p in path.split("/")]
|
||||||
|
route = "/".join(parts) or "/"
|
||||||
|
return route if route in KNOWN_ROUTES else OTHER
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_referrer(referrer: str, own_host: str = "") -> str:
|
||||||
|
"""The sending site as a bare host. Our own host means an internal
|
||||||
|
navigation, which is not a referral — "" tells the caller to skip it."""
|
||||||
|
if not referrer:
|
||||||
|
return NONE_LABEL
|
||||||
|
host = (urlsplit(referrer).hostname or "").lower().lstrip(".")
|
||||||
|
if not host or not _HOST_OK.match(host) or len(host) > MAX_LABEL_LEN:
|
||||||
|
return OTHER
|
||||||
|
if host == (own_host or "").lower() or host in ("localhost", "127.0.0.1"):
|
||||||
|
return ""
|
||||||
|
return host[4:] if host.startswith("www.") else host
|
||||||
|
|
||||||
|
|
||||||
|
def api_route_label(scope: dict, status: int) -> str:
|
||||||
|
"""An error bucket like "500 /api/adventures/{adventure_id}".
|
||||||
|
|
||||||
|
The route *template* is used, never the request path: it keeps one bucket
|
||||||
|
per endpoint instead of one per adventure id, and — the reason it is not
|
||||||
|
merely tidier — an unmatched path is entirely attacker-chosen, so labelling
|
||||||
|
by it would let anyone mint rows by requesting nonsense.
|
||||||
|
"""
|
||||||
|
template = getattr(scope.get("route"), "path", None)
|
||||||
|
return f"{status} {template}" if template else f"{status} (unmatched)"
|
||||||
|
|
||||||
|
|
||||||
|
def device_of(user_agent: str) -> str:
|
||||||
|
"""Mobile / tablet / desktop, and nothing finer. The UA string itself is
|
||||||
|
never stored — it is a fingerprint, and the answer worth having is one
|
||||||
|
word."""
|
||||||
|
ua = (user_agent or "").lower()
|
||||||
|
if not ua:
|
||||||
|
return UNKNOWN
|
||||||
|
if any(bot in ua for bot in ("bot", "crawler", "spider", "headless", "preview")):
|
||||||
|
return "bot"
|
||||||
|
if "ipad" in ua or "tablet" in ua or ("android" in ua and "mobile" not in ua):
|
||||||
|
return "tablet"
|
||||||
|
if any(m in ua for m in ("mobi", "iphone", "ipod", "android", "phone")):
|
||||||
|
return "mobile"
|
||||||
|
return "desktop"
|
||||||
|
|
||||||
|
|
||||||
|
# Geo headers an edge may add. Render fronts services with a CDN that can set
|
||||||
|
# cf-ipcountry; the others cost nothing to look for. A value is trusted only if
|
||||||
|
# it looks like an ISO code, since a client can send any header it likes — the
|
||||||
|
# worst case is therefore a wrong country, never an unbounded label.
|
||||||
|
_GEO_HEADERS = ("cf-ipcountry", "x-vercel-ip-country", "x-geo-country", "x-country-code")
|
||||||
|
|
||||||
|
|
||||||
|
def country_of(headers) -> str:
|
||||||
|
for name in _GEO_HEADERS:
|
||||||
|
value = (headers.get(name) or "").strip().upper()
|
||||||
|
if _COUNTRY_OK.match(value) and value != "XX":
|
||||||
|
return value
|
||||||
|
return UNKNOWN
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Flushing ----------
|
||||||
|
|
||||||
|
def _insert(db: Session):
|
||||||
|
return sqlite_insert if db.get_bind().dialect.name == "sqlite" else pg_insert
|
||||||
|
|
||||||
|
|
||||||
|
def _drain() -> tuple[dict, dict]:
|
||||||
|
with _guard:
|
||||||
|
counts, visits = _counts.copy(), _visits.copy()
|
||||||
|
_counts.clear()
|
||||||
|
_visits.clear()
|
||||||
|
# The label sets only bound cardinality within a day, so let yesterday's
|
||||||
|
# go rather than growing a map that never shrinks.
|
||||||
|
today = _today()
|
||||||
|
for key in [k for k in _labels_seen if k[0] != today]:
|
||||||
|
del _labels_seen[key]
|
||||||
|
return counts, visits
|
||||||
|
|
||||||
|
|
||||||
|
def _restore(counts: dict, visits: dict) -> None:
|
||||||
|
"""Put a failed flush's work back so the next one retries it."""
|
||||||
|
with _guard:
|
||||||
|
for key, n in counts.items():
|
||||||
|
_counts[key] = _counts.get(key, 0) + n
|
||||||
|
for key, flags in visits.items():
|
||||||
|
_visits.setdefault(key, set()).update(flags)
|
||||||
|
|
||||||
|
|
||||||
|
def flush(db: Session | None = None) -> None:
|
||||||
|
"""Write the buffer out. Safe to call from anywhere; never raises."""
|
||||||
|
counts, visits = _drain()
|
||||||
|
if not counts and not visits:
|
||||||
|
return
|
||||||
|
own_session = db is None
|
||||||
|
if own_session:
|
||||||
|
from .database import SessionLocal
|
||||||
|
db = SessionLocal()
|
||||||
|
try:
|
||||||
|
_write_counts(db, counts)
|
||||||
|
_write_visits(db, visits)
|
||||||
|
db.commit()
|
||||||
|
except Exception:
|
||||||
|
db.rollback()
|
||||||
|
_restore(counts, visits)
|
||||||
|
logger.exception("Analytics flush failed; counts held for the next one.")
|
||||||
|
finally:
|
||||||
|
if own_session:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _write_counts(db: Session, counts: dict) -> None:
|
||||||
|
if not counts:
|
||||||
|
return
|
||||||
|
table = models.AnalyticsDaily.__table__
|
||||||
|
rows = [
|
||||||
|
{"day": day, "metric": metric, "label": label, "hits": hits}
|
||||||
|
for (day, metric, label), hits in counts.items()
|
||||||
|
]
|
||||||
|
stmt = _insert(db)(table).values(rows)
|
||||||
|
db.execute(stmt.on_conflict_do_update(
|
||||||
|
index_elements=["day", "metric", "label"],
|
||||||
|
set_={"hits": table.c.hits + stmt.excluded.hits},
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def _write_visits(db: Session, visits: dict) -> None:
|
||||||
|
if not visits:
|
||||||
|
return
|
||||||
|
table = models.AnalyticsVisitorDay.__table__
|
||||||
|
ids = {visitor for _, visitor in visits}
|
||||||
|
# One indexed lookup settles new-vs-returning for the whole batch. It is
|
||||||
|
# the only read this module does off the dashboard, and it returns short
|
||||||
|
# hashes for visitors who are active right now — bounded by the batch.
|
||||||
|
known = set(db.scalars(
|
||||||
|
select(models.AnalyticsVisitorDay.visitor)
|
||||||
|
.where(models.AnalyticsVisitorDay.visitor.in_(ids))
|
||||||
|
.distinct()
|
||||||
|
))
|
||||||
|
rows = [
|
||||||
|
{
|
||||||
|
"day": day,
|
||||||
|
"visitor": visitor,
|
||||||
|
"is_new": visitor not in known,
|
||||||
|
**{column: column in flags for column in FUNNEL_FLAGS.values()},
|
||||||
|
}
|
||||||
|
for (day, visitor), flags in visits.items()
|
||||||
|
]
|
||||||
|
stmt = _insert(db)(table).values(rows)
|
||||||
|
db.execute(stmt.on_conflict_do_update(
|
||||||
|
index_elements=["day", "visitor"],
|
||||||
|
# Flags only ever turn on, and `is_new` is deliberately absent: the
|
||||||
|
# first write of a visitor's first day is the one that decided it.
|
||||||
|
set_={
|
||||||
|
column: or_(table.c[column], stmt.excluded[column])
|
||||||
|
for column in FUNNEL_FLAGS.values()
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def purge_old_visitor_days(db: Session) -> int:
|
||||||
|
"""Drop visitor-day rows past the retention horizon. Called by the cleanup
|
||||||
|
sweeper; the daily counters are never purged — they are aggregate, tiny,
|
||||||
|
and a portfolio project wants to keep its history."""
|
||||||
|
if RETENTION_DAYS <= 0:
|
||||||
|
return 0
|
||||||
|
cutoff = (models.utcnow().date() - timedelta(days=RETENTION_DAYS)).isoformat()
|
||||||
|
removed = db.query(models.AnalyticsVisitorDay).filter(
|
||||||
|
models.AnalyticsVisitorDay.day < cutoff
|
||||||
|
).delete(synchronize_session=False)
|
||||||
|
db.commit()
|
||||||
|
return removed or 0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Reading it back ----------
|
||||||
|
# Every query below is an aggregate: the database does the counting and ships
|
||||||
|
# back tens of rows, whatever the traffic behind them. Nothing here can return
|
||||||
|
# a row that belongs to one visitor.
|
||||||
|
|
||||||
|
TOP_N = 12
|
||||||
|
|
||||||
|
|
||||||
|
def _top(rows: list[dict], limit: int = TOP_N) -> list[dict]:
|
||||||
|
return rows[:limit]
|
||||||
|
|
||||||
|
|
||||||
|
def summary(db: Session, days: int = 30) -> dict:
|
||||||
|
"""Everything the dashboard shows, for the last `days` days (today
|
||||||
|
included). Flushes first so the numbers include the last minute."""
|
||||||
|
flush(db)
|
||||||
|
today = models.utcnow().date()
|
||||||
|
since = (today - timedelta(days=days - 1)).isoformat()
|
||||||
|
daily = models.AnalyticsDaily
|
||||||
|
visitor = models.AnalyticsVisitorDay
|
||||||
|
|
||||||
|
# 1. Every counter in the window, folded to (metric, label) totals: the
|
||||||
|
# page/referrer/country/device/scenario/error tables all come from this
|
||||||
|
# one pass rather than a query each.
|
||||||
|
by_metric: dict[str, list[dict]] = {}
|
||||||
|
for metric, label, hits in db.execute(
|
||||||
|
select(daily.metric, daily.label, func.sum(daily.hits))
|
||||||
|
.where(daily.day >= since)
|
||||||
|
.group_by(daily.metric, daily.label)
|
||||||
|
):
|
||||||
|
by_metric.setdefault(metric, []).append({"label": label, "hits": int(hits)})
|
||||||
|
for rows in by_metric.values():
|
||||||
|
rows.sort(key=lambda row: -row["hits"])
|
||||||
|
events = {row["label"]: row["hits"] for row in by_metric.get(M_EVENT, [])}
|
||||||
|
|
||||||
|
# 2. Two per-day series worth drawing.
|
||||||
|
pageviews_by_day = {
|
||||||
|
day: int(hits)
|
||||||
|
for day, hits in db.execute(
|
||||||
|
select(daily.day, func.sum(daily.hits))
|
||||||
|
.where(daily.day >= since, daily.metric == M_PAGE)
|
||||||
|
.group_by(daily.day)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
turns_by_day = {
|
||||||
|
day: int(hits)
|
||||||
|
for day, hits in db.execute(
|
||||||
|
select(daily.day, func.sum(daily.hits))
|
||||||
|
.where(daily.day >= since, daily.metric == M_EVENT, daily.label == EV_TURN)
|
||||||
|
.group_by(daily.day)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
# 3. People, per day. One row per visitor per day means COUNT(*) is already
|
||||||
|
# the day's unique visitors — no DISTINCT needed here.
|
||||||
|
visitors_by_day: dict[str, dict] = {}
|
||||||
|
for day, total, fresh in db.execute(
|
||||||
|
select(
|
||||||
|
visitor.day,
|
||||||
|
func.count(),
|
||||||
|
func.sum(case((visitor.is_new, 1), else_=0)),
|
||||||
|
)
|
||||||
|
.where(visitor.day >= since)
|
||||||
|
.group_by(visitor.day)
|
||||||
|
):
|
||||||
|
visitors_by_day[day] = {"visitors": int(total), "new": int(fresh or 0)}
|
||||||
|
|
||||||
|
# 4. The funnel, over the whole window, counting *people* once each:
|
||||||
|
# COUNT(DISTINCT CASE WHEN flag THEN visitor END) ignores the NULLs the
|
||||||
|
# CASE leaves for everyone who didn't reach that step.
|
||||||
|
unique, unique_new, *reached = db.execute(
|
||||||
|
select(
|
||||||
|
func.count(func.distinct(visitor.visitor)),
|
||||||
|
func.count(func.distinct(case((visitor.is_new, visitor.visitor)))),
|
||||||
|
*[
|
||||||
|
func.count(func.distinct(case((visitor.__table__.c[column], visitor.visitor))))
|
||||||
|
for column in FUNNEL_FLAGS.values()
|
||||||
|
],
|
||||||
|
).where(visitor.day >= since)
|
||||||
|
).one()
|
||||||
|
|
||||||
|
series = []
|
||||||
|
for offset in range(days):
|
||||||
|
day = (today - timedelta(days=days - 1 - offset)).isoformat()
|
||||||
|
counted = visitors_by_day.get(day, {})
|
||||||
|
series.append({
|
||||||
|
"day": day,
|
||||||
|
"visitors": counted.get("visitors", 0),
|
||||||
|
"new": counted.get("new", 0),
|
||||||
|
"pageviews": pageviews_by_day.get(day, 0),
|
||||||
|
"turns": turns_by_day.get(day, 0),
|
||||||
|
})
|
||||||
|
|
||||||
|
visits = sum(row["visitors"] for row in series)
|
||||||
|
pageviews = sum(pageviews_by_day.values())
|
||||||
|
turns = events.get(EV_TURN, 0)
|
||||||
|
errors = by_metric.get(M_ERROR, [])
|
||||||
|
return {
|
||||||
|
"days": days,
|
||||||
|
"since": since,
|
||||||
|
"until": today.isoformat(),
|
||||||
|
"generated_at": models.utcnow().isoformat(),
|
||||||
|
"totals": {
|
||||||
|
# `visitors` counts each person once for the window; `visits` counts
|
||||||
|
# them once per day they came back, which is the closest honest
|
||||||
|
# thing to "sessions" without tracking sessions.
|
||||||
|
"visitors": int(unique),
|
||||||
|
"new_visitors": int(unique_new),
|
||||||
|
"visits": visits,
|
||||||
|
"pageviews": pageviews,
|
||||||
|
"turns": turns,
|
||||||
|
"demo_turns": events.get(EV_DEMO_TURN, 0),
|
||||||
|
"adventures": events.get(EV_ADVENTURE, 0),
|
||||||
|
"signups": events.get(EV_SIGNUP, 0),
|
||||||
|
"logins": events.get(EV_LOGIN, 0),
|
||||||
|
"turn_errors": events.get(EV_TURN_ERROR, 0),
|
||||||
|
"errors": sum(row["hits"] for row in errors),
|
||||||
|
"turns_per_visit": round(turns / visits, 1) if visits else 0,
|
||||||
|
"pages_per_visit": round(pageviews / visits, 1) if visits else 0,
|
||||||
|
},
|
||||||
|
"series": series,
|
||||||
|
# Step 0 is everyone who showed up, so the drop-off between it and
|
||||||
|
# "Opened a scenario" is visible as a step like any other.
|
||||||
|
"funnel": [{"step": "Visited", "count": int(unique)}] + [
|
||||||
|
{"step": step, "count": int(count)}
|
||||||
|
for step, count in zip(
|
||||||
|
["Opened a scenario", "Started an adventure", "Played a turn", "Signed up"],
|
||||||
|
reached,
|
||||||
|
)
|
||||||
|
],
|
||||||
|
"pages": _top(by_metric.get(M_PAGE, [])),
|
||||||
|
"referrers": _top(by_metric.get(M_REFERRER, [])),
|
||||||
|
"countries": _top(by_metric.get(M_COUNTRY, [])),
|
||||||
|
"devices": by_metric.get(M_DEVICE, []),
|
||||||
|
"scenarios": _top(by_metric.get(M_SCENARIO, [])),
|
||||||
|
"errors": _top(errors),
|
||||||
|
"events": by_metric.get(M_EVENT, []),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Background flusher ----------
|
||||||
|
# Mirrors cleanup's start/stop pair so main.py's lifespan reads the same way
|
||||||
|
# for both. The interval is what bounds how much a hard restart can lose.
|
||||||
|
|
||||||
|
async def _flush_loop() -> None:
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
from starlette.concurrency import run_in_threadpool
|
||||||
|
|
||||||
|
while True:
|
||||||
|
await asyncio.sleep(FLUSH_INTERVAL_SECONDS)
|
||||||
|
# Blocking DB work: keep it off the event loop, which is also serving
|
||||||
|
# SSE turn streams.
|
||||||
|
await run_in_threadpool(flush)
|
||||||
|
|
||||||
|
|
||||||
|
def start_flusher():
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
return asyncio.create_task(_flush_loop())
|
||||||
|
|
||||||
|
|
||||||
|
async def stop_flusher(task) -> None:
|
||||||
|
"""Cancel the loop and write out whatever it was holding — a deploy is the
|
||||||
|
one restart that is both frequent and predictable, so it should not be the
|
||||||
|
thing that loses a minute of counts."""
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
from starlette.concurrency import run_in_threadpool
|
||||||
|
|
||||||
|
if task is not None:
|
||||||
|
task.cancel()
|
||||||
|
try:
|
||||||
|
await task
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
pass
|
||||||
|
await run_in_threadpool(flush)
|
||||||
@@ -66,6 +66,16 @@ POWER_USERS = {
|
|||||||
if e.strip()
|
if e.strip()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Who can see the visit analytics. Deliberately its own list rather than
|
||||||
|
# POWER_USERS: a trusted tester gets unmetered turns and the AI Chat page,
|
||||||
|
# which is not a reason to hand them the site's traffic numbers. Empty (the
|
||||||
|
# default) means nobody sees the dashboard in a hosted deployment.
|
||||||
|
ANALYTICS_EMAILS = {
|
||||||
|
e.strip().lower()
|
||||||
|
for e in os.environ.get("AIDND_ANALYTICS_EMAILS", "").split(",")
|
||||||
|
if e.strip()
|
||||||
|
}
|
||||||
|
|
||||||
DEMO_CAP_MESSAGE = (
|
DEMO_CAP_MESSAGE = (
|
||||||
f"You've used all {DEMO_TURNS_PER_DAY} free demo turns for today. "
|
f"You've used all {DEMO_TURNS_PER_DAY} free demo turns for today. "
|
||||||
"Add your own API key in Settings to keep playing (it resets tomorrow)."
|
"Add your own API key in Settings to keep playing (it resets tomorrow)."
|
||||||
@@ -148,6 +158,15 @@ def is_power_user(user: models.User) -> bool:
|
|||||||
return bool(user.email) and user.email.lower() in POWER_USERS
|
return bool(user.email) and user.email.lower() in POWER_USERS
|
||||||
|
|
||||||
|
|
||||||
|
def is_owner(user: models.User) -> bool:
|
||||||
|
"""May this user see the visit analytics? Local installs always can — it is
|
||||||
|
the operator's own machine and their own visits, same reasoning as the
|
||||||
|
provider debug log; hosted deployments check AIDND_ANALYTICS_EMAILS."""
|
||||||
|
if not MULTI_USER:
|
||||||
|
return True
|
||||||
|
return bool(user.email) and user.email.lower() in ANALYTICS_EMAILS
|
||||||
|
|
||||||
|
|
||||||
def demo_turns_left(user: models.User) -> int:
|
def demo_turns_left(user: models.User) -> int:
|
||||||
# Power users are never capped; report the full cap so the banner reads
|
# Power users are never capped; report the full cap so the banner reads
|
||||||
# "N of N" rather than a decrementing count.
|
# "N of N" rather than a decrementing count.
|
||||||
|
|||||||
+26
-10
@@ -42,7 +42,7 @@ from sqlalchemy import delete, func
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
from starlette.concurrency import run_in_threadpool
|
from starlette.concurrency import run_in_threadpool
|
||||||
|
|
||||||
from . import auth, models
|
from . import analytics, auth, models
|
||||||
from .database import SessionLocal
|
from .database import SessionLocal
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@@ -72,6 +72,13 @@ def enabled() -> bool:
|
|||||||
return auth.MULTI_USER and RETENTION_DAYS > 0
|
return auth.MULTI_USER and RETENTION_DAYS > 0
|
||||||
|
|
||||||
|
|
||||||
|
def anything_to_sweep() -> bool:
|
||||||
|
"""Whether the periodic task is worth starting at all. The two jobs it runs
|
||||||
|
are independent: a deployment can keep every guest forever and still want
|
||||||
|
its analytics rows aged out, and vice versa."""
|
||||||
|
return enabled() or analytics.RETENTION_DAYS > 0
|
||||||
|
|
||||||
|
|
||||||
def delete_stale_guests(db: Session, *, now: datetime | None = None) -> int:
|
def delete_stale_guests(db: Session, *, now: datetime | None = None) -> int:
|
||||||
"""Delete guests idle for RETENTION_DAYS or more. Returns the row count.
|
"""Delete guests idle for RETENTION_DAYS or more. Returns the row count.
|
||||||
|
|
||||||
@@ -105,12 +112,19 @@ def delete_stale_guests(db: Session, *, now: datetime | None = None) -> int:
|
|||||||
|
|
||||||
def sweep() -> int:
|
def sweep() -> int:
|
||||||
"""One pass, with its own session. Never raises: a failed cleanup must not
|
"""One pass, with its own session. Never raises: a failed cleanup must not
|
||||||
be able to take the app down (same rule as seeding)."""
|
be able to take the app down (same rule as seeding). Returns the guest
|
||||||
if not enabled():
|
count, which is the number worth logging about."""
|
||||||
|
if not anything_to_sweep():
|
||||||
return 0
|
return 0
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
removed = delete_stale_guests(db)
|
# Ages out the per-visitor analytics rows, on its own terms: it is not
|
||||||
|
# about guests, and it must still happen on a deployment that has
|
||||||
|
# chosen to keep every account it ever minted.
|
||||||
|
aged = analytics.purge_old_visitor_days(db)
|
||||||
|
if aged:
|
||||||
|
logger.info("Aged out %d analytics visitor-day row(s).", aged)
|
||||||
|
removed = delete_stale_guests(db) if enabled() else 0
|
||||||
if removed:
|
if removed:
|
||||||
logger.info(
|
logger.info(
|
||||||
"Cleaned up %d guest account(s) idle for %d+ days.",
|
"Cleaned up %d guest account(s) idle for %d+ days.",
|
||||||
@@ -135,16 +149,18 @@ async def _sweep_loop() -> None:
|
|||||||
|
|
||||||
|
|
||||||
def start_sweeper() -> asyncio.Task | None:
|
def start_sweeper() -> asyncio.Task | None:
|
||||||
"""Kick off the periodic sweep; None when the policy is off."""
|
"""Kick off the periodic sweep; None when there is nothing to sweep."""
|
||||||
if not enabled():
|
if not enabled():
|
||||||
logger.info("Guest cleanup disabled (multi_user=%s, retention_days=%d).",
|
logger.info("Guest cleanup disabled (multi_user=%s, retention_days=%d).",
|
||||||
auth.MULTI_USER, RETENTION_DAYS)
|
auth.MULTI_USER, RETENTION_DAYS)
|
||||||
|
else:
|
||||||
|
logger.info(
|
||||||
|
"Guest cleanup on: deleting guests idle %d+ days, every %d hour(s).",
|
||||||
|
RETENTION_DAYS,
|
||||||
|
SWEEP_INTERVAL_SECONDS // 3600,
|
||||||
|
)
|
||||||
|
if not anything_to_sweep():
|
||||||
return None
|
return None
|
||||||
logger.info(
|
|
||||||
"Guest cleanup on: deleting guests idle %d+ days, every %d hour(s).",
|
|
||||||
RETENTION_DAYS,
|
|
||||||
SWEEP_INTERVAL_SECONDS // 3600,
|
|
||||||
)
|
|
||||||
return asyncio.create_task(_sweep_loop())
|
return asyncio.create_task(_sweep_loop())
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+14
-6
@@ -32,6 +32,10 @@ RATE_LIMITS: dict[str, tuple[int, int]] = {
|
|||||||
"import": (30, 60), # large writes
|
"import": (30, 60), # large writes
|
||||||
"auth": (10, 300), # register/login attempts, per IP
|
"auth": (10, 300), # register/login attempts, per IP
|
||||||
"guest": (30, 300), # new guest users, per IP (each is a DB row)
|
"guest": (30, 300), # new guest users, per IP (each is a DB row)
|
||||||
|
# Pageview beacons. Generous — a real reader clicking around a SPA fires a
|
||||||
|
# handful a minute — but low enough that nobody can inflate the traffic
|
||||||
|
# numbers faster than they could by actually reloading the page.
|
||||||
|
"analytics": (120, 60),
|
||||||
}
|
}
|
||||||
|
|
||||||
_windows: dict[tuple[str, str], deque] = defaultdict(deque)
|
_windows: dict[tuple[str, str], deque] = defaultdict(deque)
|
||||||
@@ -49,11 +53,15 @@ _windows_guard = threading.Lock()
|
|||||||
TRUSTED_PROXY_HOPS = max(1, int(os.environ.get("AIDND_TRUSTED_PROXY_HOPS", "1") or 1))
|
TRUSTED_PROXY_HOPS = max(1, int(os.environ.get("AIDND_TRUSTED_PROXY_HOPS", "1") or 1))
|
||||||
|
|
||||||
|
|
||||||
def _client_ip(request: Request) -> str:
|
def client_ip(request: Request) -> str:
|
||||||
"""The real client IP for rate-limit keying, resistant to a spoofed
|
"""The real client IP, resistant to a spoofed X-Forwarded-For. Takes the
|
||||||
X-Forwarded-For. Takes the hop the trusted edge appended (rightmost minus
|
hop the trusted edge appended (rightmost minus any extra trusted hops);
|
||||||
any extra trusted hops); falls back to the socket peer when no forwarded
|
falls back to the socket peer when no forwarded header is present
|
||||||
header is present (local/dev, or a direct connection)."""
|
(local/dev, or a direct connection).
|
||||||
|
|
||||||
|
Public because the access log needs the same answer, and two functions that
|
||||||
|
both decide "which address is the caller's" is how one of them ends up
|
||||||
|
trusting a header it shouldn't."""
|
||||||
forwarded = request.headers.get("x-forwarded-for")
|
forwarded = request.headers.get("x-forwarded-for")
|
||||||
if forwarded:
|
if forwarded:
|
||||||
parts = [p.strip() for p in forwarded.split(",") if p.strip()]
|
parts = [p.strip() for p in forwarded.split(",") if p.strip()]
|
||||||
@@ -68,7 +76,7 @@ def rate_limit(scope: str, request: Request, user: models.User | None = None) ->
|
|||||||
if not auth.MULTI_USER:
|
if not auth.MULTI_USER:
|
||||||
return
|
return
|
||||||
limit, window_seconds = RATE_LIMITS[scope]
|
limit, window_seconds = RATE_LIMITS[scope]
|
||||||
key = (scope, f"u{user.id}" if user else f"ip{_client_ip(request)}")
|
key = (scope, f"u{user.id}" if user else f"ip{client_ip(request)}")
|
||||||
now = time.time()
|
now = time.time()
|
||||||
with _windows_guard:
|
with _windows_guard:
|
||||||
window = _windows[key]
|
window = _windows[key]
|
||||||
|
|||||||
+44
-2
@@ -7,12 +7,15 @@ from fastapi.middleware.cors import CORSMiddleware
|
|||||||
from fastapi.staticfiles import StaticFiles
|
from fastapi.staticfiles import StaticFiles
|
||||||
from starlette.exceptions import HTTPException as StarletteHTTPException
|
from starlette.exceptions import HTTPException as StarletteHTTPException
|
||||||
|
|
||||||
from . import cleanup
|
from . import analytics, cleanup
|
||||||
from .auth import MULTI_USER
|
from .auth import MULTI_USER
|
||||||
from .database import engine
|
from .database import engine
|
||||||
from .limits import BodySizeLimitMiddleware
|
from .limits import BodySizeLimitMiddleware
|
||||||
from .migrations import bootstrap
|
from .migrations import bootstrap
|
||||||
from .routers import adventures, auth, chat, debug, scenarios, scripts, settings, story_cards
|
from .routers import (
|
||||||
|
adventures, analytics as analytics_router, auth, chat, debug, scenarios, scripts,
|
||||||
|
settings, story_cards,
|
||||||
|
)
|
||||||
from .seed import seed_public_scenarios
|
from .seed import seed_public_scenarios
|
||||||
|
|
||||||
bootstrap(engine)
|
bootstrap(engine)
|
||||||
@@ -32,10 +35,15 @@ async def lifespan(_app: FastAPI):
|
|||||||
# trigger on Render's free tier, where the service sleeps after ~15
|
# trigger on Render's free tier, where the service sleeps after ~15
|
||||||
# minutes and a long-running timer rarely gets to fire.
|
# minutes and a long-running timer rarely gets to fire.
|
||||||
sweeper = cleanup.start_sweeper()
|
sweeper = cleanup.start_sweeper()
|
||||||
|
# Visit counters are buffered in memory and written in batches; this is
|
||||||
|
# what turns them into rows, and stop_flusher writes out the last batch so
|
||||||
|
# a deploy doesn't drop it.
|
||||||
|
flusher = analytics.start_flusher()
|
||||||
try:
|
try:
|
||||||
yield
|
yield
|
||||||
finally:
|
finally:
|
||||||
await cleanup.stop_sweeper(sweeper)
|
await cleanup.stop_sweeper(sweeper)
|
||||||
|
await analytics.stop_flusher(flusher)
|
||||||
|
|
||||||
|
|
||||||
# The interactive API docs stay local-only: in multi-user mode they just hand
|
# The interactive API docs stay local-only: in multi-user mode they just hand
|
||||||
@@ -95,6 +103,39 @@ class SecurityHeadersMiddleware:
|
|||||||
await self.app(scope, receive, send_with_headers)
|
await self.app(scope, receive, send_with_headers)
|
||||||
|
|
||||||
|
|
||||||
|
class ApiErrorMiddleware:
|
||||||
|
"""Counts failed API responses for the analytics dashboard.
|
||||||
|
|
||||||
|
Here rather than in an exception handler because it sees what the client
|
||||||
|
actually got: a 429 from a rate limiter, a 404 from routing, a 500 from a
|
||||||
|
handler that never returned, all the same way. Pure ASGI for the same
|
||||||
|
reason as the headers above — an SSE turn must not be buffered on its way
|
||||||
|
out. Only /api is watched; a 404 on the SPA mount is a page load, not a
|
||||||
|
fault.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, app):
|
||||||
|
self.app = app
|
||||||
|
|
||||||
|
async def __call__(self, scope, receive, send):
|
||||||
|
if scope["type"] != "http" or not scope.get("path", "").startswith("/api"):
|
||||||
|
return await self.app(scope, receive, send)
|
||||||
|
|
||||||
|
async def send_counting(message):
|
||||||
|
if message["type"] == "http.response.start" and message["status"] >= 400:
|
||||||
|
# The router has already put the matched route on the scope by
|
||||||
|
# the time a response starts, so the label can name the
|
||||||
|
# endpoint rather than the caller's path.
|
||||||
|
analytics.record(
|
||||||
|
analytics.M_ERROR,
|
||||||
|
analytics.api_route_label(scope, message["status"]),
|
||||||
|
)
|
||||||
|
await send(message)
|
||||||
|
|
||||||
|
await self.app(scope, receive, send_counting)
|
||||||
|
|
||||||
|
|
||||||
|
app.add_middleware(ApiErrorMiddleware)
|
||||||
app.add_middleware(SecurityHeadersMiddleware)
|
app.add_middleware(SecurityHeadersMiddleware)
|
||||||
|
|
||||||
app.include_router(auth.router)
|
app.include_router(auth.router)
|
||||||
@@ -105,6 +146,7 @@ app.include_router(scripts.router)
|
|||||||
app.include_router(settings.router)
|
app.include_router(settings.router)
|
||||||
app.include_router(chat.router)
|
app.include_router(chat.router)
|
||||||
app.include_router(debug.router)
|
app.include_router(debug.router)
|
||||||
|
app.include_router(analytics_router.router)
|
||||||
|
|
||||||
|
|
||||||
@app.get("/api/health")
|
@app.get("/api/health")
|
||||||
|
|||||||
+88
-1
@@ -2,7 +2,7 @@ from datetime import datetime, timezone
|
|||||||
|
|
||||||
from sqlalchemy import (
|
from sqlalchemy import (
|
||||||
JSON, Boolean, Column, DateTime, Float, ForeignKey, Index, Integer, LargeBinary,
|
JSON, Boolean, Column, DateTime, Float, ForeignKey, Index, Integer, LargeBinary,
|
||||||
String, Table, Text, event,
|
String, Table, Text, UniqueConstraint, event,
|
||||||
)
|
)
|
||||||
from sqlalchemy.orm import Mapped, Session, mapped_column, relationship
|
from sqlalchemy.orm import Mapped, Session, mapped_column, relationship
|
||||||
|
|
||||||
@@ -575,6 +575,93 @@ class Settings(Base):
|
|||||||
return security.decrypt_secret(self.api_key)
|
return security.decrypt_secret(self.api_key)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Visit analytics (see analytics.py) ----------
|
||||||
|
# Two deliberately dumb tables. Neither can hold anything a player wrote, and
|
||||||
|
# neither can be joined back to a `users` row: the visitor column is an HMAC,
|
||||||
|
# with no foreign key, so guest cleanup deleting an account leaves the history
|
||||||
|
# it contributed to intact and anonymous.
|
||||||
|
|
||||||
|
|
||||||
|
class AnalyticsDaily(Base):
|
||||||
|
"""One counter: how many times `label` happened within `metric` on `day`.
|
||||||
|
|
||||||
|
A generic (metric, label, hits) triple rather than a column per statistic,
|
||||||
|
so measuring something new later costs a constant instead of a migration.
|
||||||
|
Written only by UPSERT, from a buffer — see analytics.flush.
|
||||||
|
"""
|
||||||
|
|
||||||
|
__tablename__ = "analytics_daily"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(primary_key=True)
|
||||||
|
day: Mapped[str] = mapped_column(String(10), index=True) # YYYY-MM-DD, UTC
|
||||||
|
metric: Mapped[str] = mapped_column(String(32))
|
||||||
|
label: Mapped[str] = mapped_column(String(80), default="")
|
||||||
|
hits: Mapped[int] = mapped_column(Integer, default=0)
|
||||||
|
|
||||||
|
# The upsert target: one row per bucket per day, created or incremented.
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint("day", "metric", "label", name="uq_analytics_daily_bucket"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class AnalyticsVisitorDay(Base):
|
||||||
|
"""One visitor, one day, and which funnel steps they reached on it.
|
||||||
|
|
||||||
|
Exists so the funnel counts people rather than clicks — a player who starts
|
||||||
|
six adventures is one person who started an adventure. `is_new` is set when
|
||||||
|
the visitor has no earlier row, which is also why the visitor column is
|
||||||
|
indexed on its own.
|
||||||
|
"""
|
||||||
|
|
||||||
|
__tablename__ = "analytics_visitor_days"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(primary_key=True)
|
||||||
|
day: Mapped[str] = mapped_column(String(10))
|
||||||
|
# HMAC of the user id under the app secret; not reversible, not a key.
|
||||||
|
visitor: Mapped[str] = mapped_column(String(32))
|
||||||
|
is_new: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
|
opened: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
|
created: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
|
played: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
|
signed_up: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
|
|
||||||
|
__table_args__ = (
|
||||||
|
UniqueConstraint("day", "visitor", name="uq_analytics_visitor_day"),
|
||||||
|
Index("ix_analytics_visitor", "visitor"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class AccessEvent(Base):
|
||||||
|
"""One sign-in, registration, failed attempt, or session first-seen.
|
||||||
|
|
||||||
|
The counterpart to the two tables above, and deliberately not mixed in with
|
||||||
|
them: this one identifies people on purpose — address, email, device — so
|
||||||
|
keeping it in its own table (and its own module) means the anonymity of the
|
||||||
|
counters stays a property of the code rather than of a convention.
|
||||||
|
|
||||||
|
`user_id` is a plain integer with no foreign key. An access log that
|
||||||
|
disappeared when the account did would not be an access log, and guest
|
||||||
|
cleanup deletes accounts on a schedule; `who` and `is_guest` are snapshots
|
||||||
|
for the same reason, so a row still reads correctly afterwards.
|
||||||
|
"""
|
||||||
|
|
||||||
|
__tablename__ = "access_events"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(primary_key=True)
|
||||||
|
at: Mapped[datetime] = mapped_column(DateTime, default=utcnow, index=True)
|
||||||
|
# session | login | register | login_failed
|
||||||
|
kind: Mapped[str] = mapped_column(String(16))
|
||||||
|
user_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||||
|
# Email for a registered account, "Guest #12" otherwise; for a failed
|
||||||
|
# sign-in, the address that was tried — which is the point of the row.
|
||||||
|
who: Mapped[str] = mapped_column(String(320), default="")
|
||||||
|
is_guest: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||||
|
ip: Mapped[str] = mapped_column(String(45), default="") # 45 = max IPv6
|
||||||
|
country: Mapped[str] = mapped_column(String(16), default="")
|
||||||
|
device: Mapped[str] = mapped_column(String(16), default="")
|
||||||
|
user_agent: Mapped[str] = mapped_column(String(200), default="")
|
||||||
|
|
||||||
|
|
||||||
# Phase 14 — the floor under `tree.place_action`.
|
# Phase 14 — the floor under `tree.place_action`.
|
||||||
#
|
#
|
||||||
# From SP2 a read selects on (branch_id, depth): a node written without them is
|
# From SP2 a read selects on (branch_id, depth): a node written without them is
|
||||||
|
|||||||
@@ -9,8 +9,8 @@ from sqlalchemy.orm import Session, load_only, undefer
|
|||||||
from sqlalchemy.orm.attributes import set_committed_value
|
from sqlalchemy.orm.attributes import set_committed_value
|
||||||
|
|
||||||
from .. import (
|
from .. import (
|
||||||
attempts, auth, bundle, images, limits, memorybank, models, schemas, tree,
|
analytics, attempts, auth, bundle, images, limits, memorybank, models, schemas,
|
||||||
worldstate,
|
tree, worldstate,
|
||||||
)
|
)
|
||||||
from ..context import build_context, cursors
|
from ..context import build_context, cursors
|
||||||
from ..context import history as context_history
|
from ..context import history as context_history
|
||||||
@@ -411,6 +411,12 @@ def create_adventure(
|
|||||||
|
|
||||||
db.commit()
|
db.commit()
|
||||||
db.refresh(adventure)
|
db.refresh(adventure)
|
||||||
|
analytics.record_event(analytics.EV_ADVENTURE, user)
|
||||||
|
# Which of the shared scenarios people actually pick — the one piece of
|
||||||
|
# content this module ever names, and only ever a seeded/public one. A
|
||||||
|
# player's own scenario titles are theirs.
|
||||||
|
if scenario is not None and scenario.is_public:
|
||||||
|
analytics.record(analytics.M_SCENARIO, scenario.title)
|
||||||
return adventure
|
return adventure
|
||||||
|
|
||||||
|
|
||||||
@@ -589,6 +595,15 @@ def sse(obj: dict) -> str:
|
|||||||
return f"data: {json.dumps(obj)}\n\n"
|
return f"data: {json.dumps(obj)}\n\n"
|
||||||
|
|
||||||
|
|
||||||
|
def turn_error(detail: str, **extra) -> str:
|
||||||
|
"""An SSE error for a turn that could not be produced, counted on the way
|
||||||
|
out. Worth its own event: a failed turn is an HTTP 200 with a bad ending,
|
||||||
|
so the middleware's status-code tally cannot see it — a demo whose model
|
||||||
|
has started refusing every request looks perfectly healthy from outside."""
|
||||||
|
analytics.record(analytics.M_EVENT, analytics.EV_TURN_ERROR)
|
||||||
|
return sse({"type": "error", "detail": detail, **extra})
|
||||||
|
|
||||||
|
|
||||||
# no-cache defeats any intermediary caching; X-Accel-Buffering makes
|
# no-cache defeats any intermediary caching; X-Accel-Buffering makes
|
||||||
# nginx-style reverse proxies (hosted deploys) flush each event immediately
|
# nginx-style reverse proxies (hosted deploys) flush each event immediately
|
||||||
# instead of buffering the stream.
|
# instead of buffering the stream.
|
||||||
@@ -745,7 +760,7 @@ async def _generate_turn(
|
|||||||
chunks.append(chunk)
|
chunks.append(chunk)
|
||||||
yield sse({"type": "chunk", "text": chunk})
|
yield sse({"type": "chunk", "text": chunk})
|
||||||
except ProviderError as exc:
|
except ProviderError as exc:
|
||||||
yield sse({"type": "error", "detail": str(exc)})
|
yield turn_error(str(exc))
|
||||||
return
|
return
|
||||||
|
|
||||||
text = "".join(chunks).strip()
|
text = "".join(chunks).strip()
|
||||||
@@ -763,13 +778,13 @@ async def _generate_turn(
|
|||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
detail = "The AI returned an empty response."
|
detail = "The AI returned an empty response."
|
||||||
yield sse({"type": "error", "detail": detail})
|
yield turn_error(detail)
|
||||||
return
|
return
|
||||||
|
|
||||||
# onOutput
|
# onOutput
|
||||||
text, _ = pipeline.run("output", text)
|
text, _ = pipeline.run("output", text)
|
||||||
if not text.strip():
|
if not text.strip():
|
||||||
yield sse({"type": "error", "detail": "A script's output modifier returned empty text."})
|
yield turn_error("A script's output modifier returned empty text.")
|
||||||
return
|
return
|
||||||
snapshot["script"] = snapshot["script"] | pipeline.report()
|
snapshot["script"] = snapshot["script"] | pipeline.report()
|
||||||
|
|
||||||
@@ -785,7 +800,7 @@ async def _generate_turn(
|
|||||||
if worldstate.has_schema(stat_schema):
|
if worldstate.has_schema(stat_schema):
|
||||||
text, delta = worldstate.extract_delta(text)
|
text, delta = worldstate.extract_delta(text)
|
||||||
if not text.strip():
|
if not text.strip():
|
||||||
yield sse({"type": "error", "detail": "The AI returned only a state update and no story text."})
|
yield turn_error("The AI returned only a state update and no story text.")
|
||||||
return
|
return
|
||||||
new_world_state, ws_report = worldstate.apply_delta(
|
new_world_state, ws_report = worldstate.apply_delta(
|
||||||
adventure.world_state, stat_schema, delta, ai_depth
|
adventure.world_state, stat_schema, delta, ai_depth
|
||||||
@@ -832,6 +847,12 @@ async def _generate_turn(
|
|||||||
# in the endpoint); failed provider calls above don't reach here.
|
# in the endpoint); failed provider calls above don't reach here.
|
||||||
auth.count_demo_turn(user)
|
auth.count_demo_turn(user)
|
||||||
db.commit()
|
db.commit()
|
||||||
|
# Counted here, past every way the turn could still have failed, so the
|
||||||
|
# number means "stories advanced" rather than "requests attempted". The
|
||||||
|
# demo tally is those same turns seen as spend on the server-funded key.
|
||||||
|
analytics.record_event(analytics.EV_TURN, user)
|
||||||
|
if cfg.using_demo:
|
||||||
|
analytics.record(analytics.M_EVENT, analytics.EV_DEMO_TURN)
|
||||||
db.refresh(ai_action)
|
db.refresh(ai_action)
|
||||||
yield _SAVED
|
yield _SAVED
|
||||||
yield sse({"type": "done", "action": action_json(ai_action, db), "script": pipeline.report()})
|
yield sse({"type": "done", "action": action_json(ai_action, db), "script": pipeline.report()})
|
||||||
@@ -876,8 +897,8 @@ async def run_player_turn(
|
|||||||
)
|
)
|
||||||
modified, stop = pipeline.run("input", formatted)
|
modified, stop = pipeline.run("input", formatted)
|
||||||
if not modified.strip():
|
if not modified.strip():
|
||||||
yield sse({"type": "error", "detail": "A script's input modifier returned empty text.",
|
yield turn_error("A script's input modifier returned empty text.",
|
||||||
"script": pipeline.report()})
|
script=pipeline.report())
|
||||||
return
|
return
|
||||||
player_action = models.Action(
|
player_action = models.Action(
|
||||||
adventure_id=adventure.id,
|
adventure_id=adventure.id,
|
||||||
@@ -1800,6 +1821,10 @@ def import_adventure(
|
|||||||
|
|
||||||
db.commit()
|
db.commit()
|
||||||
db.refresh(adventure)
|
db.refresh(adventure)
|
||||||
|
# Not a funnel step: importing a bundle is something a returning player
|
||||||
|
# does, not a sign a first-time visitor got anywhere. Counted anyway,
|
||||||
|
# because it is the clearest evidence anyone is using the export format.
|
||||||
|
analytics.record_event(analytics.EV_IMPORT, user)
|
||||||
return adventure
|
return adventure
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,137 @@
|
|||||||
|
"""Visit analytics: one endpoint the browser writes to, one the owner reads.
|
||||||
|
|
||||||
|
The split matters. `/collect` is public and takes exactly one fact — which
|
||||||
|
page was viewed — because anything a stranger can POST is a number a stranger
|
||||||
|
can invent. Everything the dashboard actually relies on (turns, adventures,
|
||||||
|
sign-ups, demo spend, errors) is recorded server-side by the code performing
|
||||||
|
it, so those counts are as trustworthy as the app itself.
|
||||||
|
|
||||||
|
The two reading endpoints are owner-only and 404 for everyone else, the same
|
||||||
|
way the AI Chat router does: a feature nobody else can use is better off not
|
||||||
|
appearing to exist. `/summary` serves the anonymous counters (analytics.py,
|
||||||
|
which stores nothing that points at a person) and `/access` serves the access
|
||||||
|
log (accesslog.py, which identifies people on purpose).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from .. import accesslog, analytics, auth, limits, models
|
||||||
|
from ..database import get_db
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/api/analytics", tags=["analytics"])
|
||||||
|
|
||||||
|
|
||||||
|
def owner(
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
user: models.User = Depends(auth.get_current_user),
|
||||||
|
) -> models.User:
|
||||||
|
"""Gate for the reading half. 404, not 403 — see the module docstring."""
|
||||||
|
if not auth.is_owner(user):
|
||||||
|
raise HTTPException(404, "Not found")
|
||||||
|
return user
|
||||||
|
|
||||||
|
|
||||||
|
Owner = Depends(owner)
|
||||||
|
|
||||||
|
|
||||||
|
class Pageview(BaseModel):
|
||||||
|
"""What the SPA reports on a page load or a route change.
|
||||||
|
|
||||||
|
`first` marks a real page load rather than a client-side navigation: the
|
||||||
|
things that describe a *visit* rather than a *view* — where it came from,
|
||||||
|
on what kind of device, from which country — are recorded only then, so a
|
||||||
|
visitor who clicks around five pages is still one referral.
|
||||||
|
"""
|
||||||
|
|
||||||
|
path: str = Field("", max_length=300)
|
||||||
|
referrer: str = Field("", max_length=500)
|
||||||
|
first: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/collect", status_code=204)
|
||||||
|
def collect(
|
||||||
|
payload: Pageview,
|
||||||
|
request: Request,
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
) -> Response:
|
||||||
|
"""Record one pageview. Always 204, even when nothing was counted: the
|
||||||
|
browser has no business knowing whether it was."""
|
||||||
|
limits.rate_limit("analytics", request)
|
||||||
|
# Resolved by hand rather than through get_current_user: a pageview that
|
||||||
|
# arrives before /auth/me has minted a session should still be counted as a
|
||||||
|
# view, not turned into a 401 the SPA has to handle.
|
||||||
|
user = (
|
||||||
|
auth.resolve_session_user(request, db)
|
||||||
|
if auth.MULTI_USER
|
||||||
|
else auth.local_user(db)
|
||||||
|
)
|
||||||
|
# The operator's own clicking is not traffic. Only in multi-user mode —
|
||||||
|
# locally everyone is the owner, and excluding them would leave the
|
||||||
|
# dashboard permanently empty on the machine it is developed on.
|
||||||
|
if auth.MULTI_USER and user is not None and auth.is_owner(user):
|
||||||
|
return Response(status_code=204)
|
||||||
|
|
||||||
|
analytics.record(analytics.M_PAGE, analytics.normalize_route(payload.path))
|
||||||
|
analytics.record_visit(user)
|
||||||
|
if payload.first:
|
||||||
|
referrer = analytics.normalize_referrer(
|
||||||
|
payload.referrer, request.url.hostname or ""
|
||||||
|
)
|
||||||
|
if referrer: # "" means same-origin, which is not a referral
|
||||||
|
analytics.record(analytics.M_REFERRER, referrer)
|
||||||
|
analytics.record(
|
||||||
|
analytics.M_DEVICE,
|
||||||
|
analytics.device_of(request.headers.get("user-agent", "")),
|
||||||
|
)
|
||||||
|
analytics.record(analytics.M_COUNTRY, analytics.country_of(request.headers))
|
||||||
|
return Response(status_code=204)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/summary")
|
||||||
|
def summary(
|
||||||
|
days: int = Query(30, ge=1, le=365),
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
_user: models.User = Owner,
|
||||||
|
) -> dict:
|
||||||
|
"""The whole dashboard in one aggregate response — a few kilobytes however
|
||||||
|
much traffic sits behind it."""
|
||||||
|
return analytics.summary(db, days)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/access")
|
||||||
|
def access_log(
|
||||||
|
limit: int = Query(50, ge=1, le=200),
|
||||||
|
before_id: int | None = Query(None),
|
||||||
|
kind: str | None = Query(None),
|
||||||
|
q: str | None = Query(None, max_length=120),
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
_user: models.User = Owner,
|
||||||
|
) -> dict:
|
||||||
|
"""A page of the access log, newest first.
|
||||||
|
|
||||||
|
Unlike /summary this returns rows about people, which is the whole point of
|
||||||
|
it — so it is behind the same owner gate, paged rather than dumped, and has
|
||||||
|
no counterpart the people it describes can reach.
|
||||||
|
"""
|
||||||
|
page = accesslog.recent(
|
||||||
|
db, limit=limit, before_id=before_id, kind=kind, query=q
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"id": event.id,
|
||||||
|
"at": event.at.isoformat(),
|
||||||
|
"kind": event.kind,
|
||||||
|
"who": event.who,
|
||||||
|
"is_guest": event.is_guest,
|
||||||
|
"ip": event.ip,
|
||||||
|
"country": event.country,
|
||||||
|
"device": event.device,
|
||||||
|
"user_agent": event.user_agent,
|
||||||
|
}
|
||||||
|
for event in page["events"]
|
||||||
|
],
|
||||||
|
"has_more": page["has_more"],
|
||||||
|
}
|
||||||
@@ -3,7 +3,7 @@ import re
|
|||||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response
|
from fastapi import APIRouter, Depends, HTTPException, Request, Response
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from .. import auth, cleanup, limits, models, schemas, security
|
from .. import accesslog, analytics, auth, cleanup, limits, models, schemas, security
|
||||||
from ..database import get_db
|
from ..database import get_db
|
||||||
from .settings import get_settings
|
from .settings import get_settings
|
||||||
|
|
||||||
@@ -34,6 +34,8 @@ def me_payload(user: models.User, db: Session) -> dict:
|
|||||||
"is_guest": user.is_guest,
|
"is_guest": user.is_guest,
|
||||||
# Trusted testers: unmetered demo turns, plus the AI Chat scratchpad.
|
# Trusted testers: unmetered demo turns, plus the AI Chat scratchpad.
|
||||||
"power_user": auth.is_power_user(user),
|
"power_user": auth.is_power_user(user),
|
||||||
|
# Separate allowlist: shows the visit-analytics page and its nav link.
|
||||||
|
"analytics": auth.is_owner(user),
|
||||||
# How long an idle guest is kept before cleanup deletes it (None when
|
# How long an idle guest is kept before cleanup deletes it (None when
|
||||||
# the policy is off). Served rather than hardcoded in the UI so the
|
# the policy is off). Served rather than hardcoded in the UI so the
|
||||||
# number a guest is shown is the number actually enforced.
|
# number a guest is shown is the number actually enforced.
|
||||||
@@ -65,6 +67,9 @@ def me(request: Request, response: Response, db: Session = Depends(get_db)):
|
|||||||
db.add(user)
|
db.add(user)
|
||||||
db.commit()
|
db.commit()
|
||||||
_set_session_cookie(response, user.id)
|
_set_session_cookie(response, user.id)
|
||||||
|
# This endpoint is the SPA's bootstrap call, so it is where a session first
|
||||||
|
# shows itself; accesslog thins the rows down to one per day per address.
|
||||||
|
accesslog.note_session(db, user, request)
|
||||||
return me_payload(user, db)
|
return me_payload(user, db)
|
||||||
|
|
||||||
|
|
||||||
@@ -93,6 +98,8 @@ def register(
|
|||||||
user.password_hash = security.hash_password(payload.password)
|
user.password_hash = security.hash_password(payload.password)
|
||||||
user.is_guest = False
|
user.is_guest = False
|
||||||
db.commit()
|
db.commit()
|
||||||
|
analytics.record_event(analytics.EV_SIGNUP, user)
|
||||||
|
accesslog.record(db, accesslog.REGISTER, request, user=user)
|
||||||
return me_payload(user, db)
|
return me_payload(user, db)
|
||||||
|
|
||||||
|
|
||||||
@@ -119,9 +126,15 @@ def login(
|
|||||||
or not security.verify_password(payload.password, user.password_hash)
|
or not security.verify_password(payload.password, user.password_hash)
|
||||||
):
|
):
|
||||||
limits.note_login_failure(email)
|
limits.note_login_failure(email)
|
||||||
|
# Logged with the address that was tried, not the account that owns it:
|
||||||
|
# a guessing run against an address that has no account is exactly the
|
||||||
|
# thing worth being able to see.
|
||||||
|
accesslog.record(db, accesslog.LOGIN_FAILED, request, who=email)
|
||||||
raise HTTPException(401, "Incorrect email or password.")
|
raise HTTPException(401, "Incorrect email or password.")
|
||||||
limits.note_login_success(email)
|
limits.note_login_success(email)
|
||||||
_set_session_cookie(response, user.id)
|
_set_session_cookie(response, user.id)
|
||||||
|
analytics.record_event(analytics.EV_LOGIN, user)
|
||||||
|
accesslog.record(db, accesslog.LOGIN, request, user=user)
|
||||||
return me_payload(user, db)
|
return me_payload(user, db)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ from fastapi.responses import Response
|
|||||||
from sqlalchemy import or_
|
from sqlalchemy import or_
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from .. import auth, images, limits, models, schemas
|
from .. import analytics, auth, images, limits, models, schemas
|
||||||
from ..database import get_db
|
from ..database import get_db
|
||||||
|
|
||||||
router = APIRouter(prefix="/api/scenarios", tags=["scenarios"])
|
router = APIRouter(prefix="/api/scenarios", tags=["scenarios"])
|
||||||
@@ -53,7 +53,13 @@ def get_scenario(
|
|||||||
db: Session = Depends(get_db),
|
db: Session = Depends(get_db),
|
||||||
user: models.User = Depends(auth.get_current_user),
|
user: models.User = Depends(auth.get_current_user),
|
||||||
):
|
):
|
||||||
return get_scenario_or_404(scenario_id, db, user)
|
scenario = get_scenario_or_404(scenario_id, db, user)
|
||||||
|
# Funnel step. Only for shared scenarios: opening one is the first sign a
|
||||||
|
# visitor is interested, whereas someone editing their own is already past
|
||||||
|
# this point — and their titles are theirs, not a statistic.
|
||||||
|
if scenario.is_public:
|
||||||
|
analytics.record_event(analytics.EV_SCENARIO_OPEN, user)
|
||||||
|
return scenario
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{scenario_id}/image")
|
@router.get("/{scenario_id}/image")
|
||||||
|
|||||||
@@ -0,0 +1,215 @@
|
|||||||
|
"""The access log — app/accesslog.py and GET /api/analytics/access.
|
||||||
|
|
||||||
|
This is the half of the analytics work that identifies people on purpose, so
|
||||||
|
the things worth pinning are the ones that would quietly make it wrong: that
|
||||||
|
the address recorded is the hardened one and not a header a client chose, that
|
||||||
|
session rows are thinned instead of written per page load, and that a row
|
||||||
|
outlives the account it describes — guest cleanup runs on a schedule, and a log
|
||||||
|
that deletes itself is not a log.
|
||||||
|
|
||||||
|
python -m pytest tests/test_accesslog.py -v
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
|
||||||
|
_tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
|
||||||
|
_tmp.close()
|
||||||
|
os.environ["AIDND_DB_PATH"] = _tmp.name
|
||||||
|
os.environ.pop("AIDND_DATABASE_URL", None)
|
||||||
|
os.environ.pop("DATABASE_URL", None)
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi import Depends
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from app import accesslog, auth, limits, models, security
|
||||||
|
from app.database import Base, SessionLocal, engine, get_db
|
||||||
|
from app.main import app
|
||||||
|
|
||||||
|
EDGE = "198.51.100.77" # what the trusted proxy appended
|
||||||
|
SPOOF = "10.0.0.1" # what a client put in front of it
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def clean_state():
|
||||||
|
accesslog._last_session.clear()
|
||||||
|
yield
|
||||||
|
accesslog._last_session.clear()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def client(monkeypatch):
|
||||||
|
Base.metadata.create_all(bind=engine)
|
||||||
|
setup = SessionLocal()
|
||||||
|
owner = models.User(is_guest=False, email="owner@example.com")
|
||||||
|
member = models.User(
|
||||||
|
is_guest=False, email="player@example.com",
|
||||||
|
password_hash=security.hash_password("hunter2long"),
|
||||||
|
)
|
||||||
|
setup.add_all([owner, member])
|
||||||
|
setup.commit()
|
||||||
|
ids = {"owner": owner.id, "member": member.id}
|
||||||
|
setup.close()
|
||||||
|
|
||||||
|
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
|
||||||
|
monkeypatch.setattr(limits, "check_login_allowed", lambda *a, **k: None)
|
||||||
|
monkeypatch.setattr(auth, "MULTI_USER", True)
|
||||||
|
monkeypatch.setattr(auth, "ANALYTICS_EMAILS", {"owner@example.com"})
|
||||||
|
|
||||||
|
# /auth/me resolves its own session, so the cookie flow below is the real
|
||||||
|
# one; every other endpoint goes through get_current_user, and `act_as`
|
||||||
|
# decides who that is.
|
||||||
|
acting = {"id": ids["owner"]}
|
||||||
|
|
||||||
|
def _current(db=Depends(get_db)):
|
||||||
|
return db.get(models.User, acting["id"])
|
||||||
|
|
||||||
|
app.dependency_overrides[auth.get_current_user] = _current
|
||||||
|
try:
|
||||||
|
test_client = TestClient(app)
|
||||||
|
test_client.ids = ids
|
||||||
|
test_client.act_as = lambda user_id: acting.update(id=user_id)
|
||||||
|
yield test_client
|
||||||
|
finally:
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
Base.metadata.drop_all(bind=engine)
|
||||||
|
|
||||||
|
|
||||||
|
def visit(client, ip=EDGE, ua="Mozilla/5.0 (Windows NT 10.0; Win64; x64)"):
|
||||||
|
return client.get(
|
||||||
|
"/api/auth/me",
|
||||||
|
headers={"x-forwarded-for": f"{SPOOF}, {ip}", "user-agent": ua},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def rows(kind=None):
|
||||||
|
db = SessionLocal()
|
||||||
|
try:
|
||||||
|
query = db.query(models.AccessEvent).order_by(models.AccessEvent.id)
|
||||||
|
if kind:
|
||||||
|
query = query.filter_by(kind=kind)
|
||||||
|
return query.all()
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def read_log(client, **params):
|
||||||
|
return client.get("/api/analytics/access", params=params)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Writing ----------
|
||||||
|
|
||||||
|
def test_a_new_session_is_logged(client):
|
||||||
|
visit(client)
|
||||||
|
logged = rows()
|
||||||
|
assert len(logged) == 1
|
||||||
|
entry = logged[0]
|
||||||
|
assert entry.kind == accesslog.SESSION
|
||||||
|
assert entry.is_guest and entry.who.startswith("Guest #")
|
||||||
|
assert entry.device == "desktop"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_address_is_the_hardened_one_not_the_clients(client):
|
||||||
|
visit(client)
|
||||||
|
# The client prepended its own value; only the hop the edge appended counts.
|
||||||
|
# Recording the leftmost would make every row forgeable, which for a log is
|
||||||
|
# worse than having no log.
|
||||||
|
assert rows()[0].ip == EDGE
|
||||||
|
|
||||||
|
|
||||||
|
def test_session_rows_are_thinned_to_one_per_day_per_address(client):
|
||||||
|
for _ in range(4):
|
||||||
|
visit(client)
|
||||||
|
assert len(rows(accesslog.SESSION)) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_changed_address_writes_a_new_row(client):
|
||||||
|
visit(client)
|
||||||
|
visit(client, ip="203.0.113.9")
|
||||||
|
logged = rows(accesslog.SESSION)
|
||||||
|
assert [entry.ip for entry in logged] == [EDGE, "203.0.113.9"]
|
||||||
|
# Same session throughout, so both rows name the same visitor.
|
||||||
|
assert logged[0].who == logged[1].who
|
||||||
|
|
||||||
|
|
||||||
|
def test_sign_in_and_failure_are_both_logged(client):
|
||||||
|
client.post("/api/auth/login", json={"email": "player@example.com", "password": "wrong"},
|
||||||
|
headers={"x-forwarded-for": EDGE})
|
||||||
|
client.post("/api/auth/login", json={"email": "player@example.com", "password": "hunter2long"},
|
||||||
|
headers={"x-forwarded-for": EDGE})
|
||||||
|
kinds = [entry.kind for entry in rows()]
|
||||||
|
assert accesslog.LOGIN_FAILED in kinds and accesslog.LOGIN in kinds
|
||||||
|
|
||||||
|
failure = rows(accesslog.LOGIN_FAILED)[0]
|
||||||
|
# The address tried, not the account that owns it: a run against an address
|
||||||
|
# with no account behind it is exactly what this row is for.
|
||||||
|
assert failure.who == "player@example.com"
|
||||||
|
assert failure.user_id is None
|
||||||
|
assert rows(accesslog.LOGIN)[0].user_id == client.ids["member"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_registering_is_logged_against_the_upgraded_account(client):
|
||||||
|
visit(client) # mints the guest whose session registers
|
||||||
|
client.act_as(rows()[0].user_id)
|
||||||
|
client.post("/api/auth/register", json={"email": "new@example.com", "password": "hunter2long"})
|
||||||
|
entry = rows(accesslog.REGISTER)[0]
|
||||||
|
assert entry.who == "new@example.com" and not entry.is_guest
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_row_outlives_the_account_it_describes(client):
|
||||||
|
visit(client)
|
||||||
|
entry = rows()[0]
|
||||||
|
db = SessionLocal()
|
||||||
|
try:
|
||||||
|
db.delete(db.get(models.User, entry.user_id))
|
||||||
|
db.commit()
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
# No foreign key, and `who` is a snapshot — guest cleanup deletes accounts
|
||||||
|
# on a schedule, and a log that vanishes with them is not a log.
|
||||||
|
survivor = rows()[0]
|
||||||
|
assert survivor.who == entry.who and survivor.ip == EDGE
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_long_user_agent_is_truncated(client):
|
||||||
|
visit(client, ua="Mozilla/" + "x" * 500)
|
||||||
|
assert len(rows()[0].user_agent) == accesslog.MAX_UA
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_logging_failure_does_not_break_the_request(client, monkeypatch):
|
||||||
|
monkeypatch.setattr(accesslog, "_client_ip", lambda request: 1 / 0)
|
||||||
|
# The log watches sign-in; it must not be able to stand in its way.
|
||||||
|
assert visit(client).status_code == 200
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Reading ----------
|
||||||
|
|
||||||
|
def test_the_log_is_invisible_to_everyone_but_the_owner(client):
|
||||||
|
visit(client)
|
||||||
|
assert read_log(client).status_code == 200
|
||||||
|
client.act_as(client.ids["member"])
|
||||||
|
assert read_log(client).status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_log_reads_newest_first_and_pages_backwards(client):
|
||||||
|
for index in range(5):
|
||||||
|
visit(client, ip=f"203.0.113.{index}")
|
||||||
|
first = read_log(client, limit=2).json()
|
||||||
|
assert [event["ip"] for event in first["events"]] == ["203.0.113.4", "203.0.113.3"]
|
||||||
|
assert first["has_more"]
|
||||||
|
|
||||||
|
older = read_log(client, limit=2, before_id=first["events"][-1]["id"]).json()
|
||||||
|
assert [event["ip"] for event in older["events"]] == ["203.0.113.2", "203.0.113.1"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_log_filters_by_kind_and_searches(client):
|
||||||
|
visit(client)
|
||||||
|
client.post("/api/auth/login", json={"email": "player@example.com", "password": "hunter2long"},
|
||||||
|
headers={"x-forwarded-for": "203.0.113.44"})
|
||||||
|
|
||||||
|
assert len(read_log(client, kind="login").json()["events"]) == 1
|
||||||
|
by_email = read_log(client, q="player@example.com").json()["events"]
|
||||||
|
assert len(by_email) == 1 and by_email[0]["kind"] == "login"
|
||||||
|
by_ip = read_log(client, q="203.0.113.44").json()["events"]
|
||||||
|
assert len(by_ip) == 1
|
||||||
|
assert read_log(client, q="nobody@example.com").json()["events"] == []
|
||||||
@@ -0,0 +1,392 @@
|
|||||||
|
"""Visit analytics — app/analytics.py and the two endpoints in front of it.
|
||||||
|
|
||||||
|
Three things are worth testing here and the rest is arithmetic. That the
|
||||||
|
counters survive the buffer/UPSERT round trip (a flush must add to what is
|
||||||
|
already stored, not replace it, or every number is only ever the last minute).
|
||||||
|
That the funnel counts *people* rather than clicks, which is the only reason
|
||||||
|
the visitor-day table exists. And that the gate holds: a stranger cannot read
|
||||||
|
the dashboard, and cannot inflate what it says beyond hitting the page.
|
||||||
|
|
||||||
|
python -m pytest tests/test_analytics.py -v
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
|
||||||
|
_tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
|
||||||
|
_tmp.close()
|
||||||
|
os.environ["AIDND_DB_PATH"] = _tmp.name
|
||||||
|
os.environ.pop("AIDND_DATABASE_URL", None)
|
||||||
|
os.environ.pop("DATABASE_URL", None)
|
||||||
|
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi import Depends
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from app import analytics, auth, limits, models
|
||||||
|
from app.database import Base, SessionLocal, engine, get_db
|
||||||
|
from app.main import app
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def clean_buffer():
|
||||||
|
"""The buffer is process-wide, so a test that leaves counts in it would
|
||||||
|
show up inside the next one's flush."""
|
||||||
|
analytics._counts.clear()
|
||||||
|
analytics._visits.clear()
|
||||||
|
analytics._labels_seen.clear()
|
||||||
|
yield
|
||||||
|
analytics._counts.clear()
|
||||||
|
analytics._visits.clear()
|
||||||
|
analytics._labels_seen.clear()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def db():
|
||||||
|
Base.metadata.create_all(bind=engine)
|
||||||
|
session = SessionLocal()
|
||||||
|
try:
|
||||||
|
yield session
|
||||||
|
finally:
|
||||||
|
session.close()
|
||||||
|
Base.metadata.drop_all(bind=engine)
|
||||||
|
|
||||||
|
|
||||||
|
def counter(db, metric, label):
|
||||||
|
row = (
|
||||||
|
db.query(models.AnalyticsDaily)
|
||||||
|
.filter_by(metric=metric, label=label)
|
||||||
|
.one_or_none()
|
||||||
|
)
|
||||||
|
return row.hits if row else 0
|
||||||
|
|
||||||
|
|
||||||
|
def make_user(db, email=None):
|
||||||
|
user = models.User(is_guest=email is None, email=email)
|
||||||
|
db.add(user)
|
||||||
|
db.commit()
|
||||||
|
return user
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- The buffer and its flush ----------
|
||||||
|
|
||||||
|
def test_counts_accumulate_across_flushes(db):
|
||||||
|
analytics.record(analytics.M_PAGE, "/")
|
||||||
|
analytics.record(analytics.M_PAGE, "/")
|
||||||
|
analytics.flush(db)
|
||||||
|
analytics.record(analytics.M_PAGE, "/")
|
||||||
|
analytics.flush(db)
|
||||||
|
# The second flush has to find the existing row and add to it. Replacing it
|
||||||
|
# would leave every counter showing only the newest minute of traffic.
|
||||||
|
assert counter(db, analytics.M_PAGE, "/") == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_flush_is_a_no_op_when_nothing_happened(db):
|
||||||
|
analytics.flush(db)
|
||||||
|
assert db.query(models.AnalyticsDaily).count() == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_failed_flush_keeps_the_counts(db, monkeypatch):
|
||||||
|
analytics.record(analytics.M_PAGE, "/")
|
||||||
|
monkeypatch.setattr(analytics, "_write_counts", lambda *a: 1 / 0)
|
||||||
|
analytics.flush(db) # must not raise
|
||||||
|
monkeypatch.undo()
|
||||||
|
analytics.flush(db)
|
||||||
|
assert counter(db, analytics.M_PAGE, "/") == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_label_cardinality_is_capped(db):
|
||||||
|
for i in range(analytics.MAX_LABELS_PER_METRIC + 25):
|
||||||
|
analytics.record(analytics.M_REFERRER, f"host{i}.example")
|
||||||
|
analytics.flush(db)
|
||||||
|
labels = db.query(models.AnalyticsDaily).filter_by(metric=analytics.M_REFERRER).count()
|
||||||
|
# Everything past the cap is folded into one bucket, so a referrer flood
|
||||||
|
# cannot mint rows without limit.
|
||||||
|
assert labels == analytics.MAX_LABELS_PER_METRIC + 1
|
||||||
|
assert counter(db, analytics.M_REFERRER, analytics.OTHER) == 25
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Visitors ----------
|
||||||
|
|
||||||
|
def test_visitor_id_is_stable_and_keyed(db, monkeypatch):
|
||||||
|
user = make_user(db)
|
||||||
|
handle = analytics.visitor_id(user)
|
||||||
|
assert handle == analytics.visitor_id(user) # a returning visitor
|
||||||
|
assert handle != analytics.visitor_id(make_user(db)) # is still one visitor
|
||||||
|
assert len(handle) == 32 and int(handle, 16) >= 0 # opaque hex, not an id
|
||||||
|
# Keyed on the app secret, not a bare hash of the user id: otherwise anyone
|
||||||
|
# holding this table could rebuild the mapping by hashing 1, 2, 3, …
|
||||||
|
monkeypatch.setattr(analytics.security, "SECRET_KEY", b"a-different-secret")
|
||||||
|
assert analytics.visitor_id(user) != handle
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_repeat_visitor_is_new_only_once(db):
|
||||||
|
user = make_user(db)
|
||||||
|
analytics.record_visit(user)
|
||||||
|
analytics.flush(db)
|
||||||
|
rows = db.query(models.AnalyticsVisitorDay).all()
|
||||||
|
assert len(rows) == 1 and rows[0].is_new
|
||||||
|
|
||||||
|
# Same visitor, a later day: seen before, so not new — and not merged into
|
||||||
|
# the first day's row either.
|
||||||
|
tomorrow = (models.utcnow().date() + timedelta(days=1)).isoformat()
|
||||||
|
analytics._visits[(tomorrow, analytics.visitor_id(user))] = set()
|
||||||
|
analytics.flush(db)
|
||||||
|
rows = db.query(models.AnalyticsVisitorDay).order_by(models.AnalyticsVisitorDay.day).all()
|
||||||
|
assert [row.is_new for row in rows] == [True, False]
|
||||||
|
|
||||||
|
|
||||||
|
def test_one_row_per_visitor_per_day_however_much_they_do(db):
|
||||||
|
user = make_user(db)
|
||||||
|
for _ in range(5):
|
||||||
|
analytics.record_event(analytics.EV_ADVENTURE, user)
|
||||||
|
analytics.flush(db)
|
||||||
|
assert db.query(models.AnalyticsVisitorDay).count() == 1
|
||||||
|
assert counter(db, analytics.M_EVENT, analytics.EV_ADVENTURE) == 5
|
||||||
|
|
||||||
|
|
||||||
|
def test_funnel_flags_only_ever_turn_on(db):
|
||||||
|
user = make_user(db)
|
||||||
|
analytics.record_event(analytics.EV_TURN, user)
|
||||||
|
analytics.flush(db)
|
||||||
|
# A later visit that reaches no funnel step must not clear the earlier one.
|
||||||
|
analytics.record_visit(user)
|
||||||
|
analytics.flush(db)
|
||||||
|
row = db.query(models.AnalyticsVisitorDay).one()
|
||||||
|
assert row.played and not row.created
|
||||||
|
|
||||||
|
|
||||||
|
def test_purge_drops_only_rows_past_the_horizon(db):
|
||||||
|
old = (models.utcnow().date() - timedelta(days=analytics.RETENTION_DAYS + 1)).isoformat()
|
||||||
|
db.add(models.AnalyticsVisitorDay(day=old, visitor="a" * 32))
|
||||||
|
db.add(models.AnalyticsVisitorDay(day=analytics._today(), visitor="b" * 32))
|
||||||
|
db.commit()
|
||||||
|
assert analytics.purge_old_visitor_days(db) == 1
|
||||||
|
assert [r.visitor for r in db.query(models.AnalyticsVisitorDay)] == ["b" * 32]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Normalizing what a browser claims ----------
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("path, expected", [
|
||||||
|
("/", "/"),
|
||||||
|
("/adventures", "/adventures"),
|
||||||
|
("/adventures/", "/adventures"),
|
||||||
|
("/play/12?x=1", "/play/:id"),
|
||||||
|
("/scenarios/9#top", "/scenarios/:id"),
|
||||||
|
("/wp-admin", "(other)"),
|
||||||
|
("/play/../../etc", "(other)"),
|
||||||
|
("", "/"),
|
||||||
|
])
|
||||||
|
def test_route_normalization(path, expected):
|
||||||
|
assert analytics.normalize_route(path) == expected
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("referrer, expected", [
|
||||||
|
("", "(direct)"),
|
||||||
|
("https://news.ycombinator.com/item?id=1", "news.ycombinator.com"),
|
||||||
|
("https://www.google.com/", "google.com"),
|
||||||
|
("https://ai-dnd.example/scenarios", ""), # our own host: not a referral
|
||||||
|
("javascript:alert(1)", "(other)"),
|
||||||
|
("https://" + "x" * 200 + ".com", "(other)"),
|
||||||
|
])
|
||||||
|
def test_referrer_normalization(referrer, expected):
|
||||||
|
assert analytics.normalize_referrer(referrer, "ai-dnd.example") == expected
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("ua, expected", [
|
||||||
|
("Mozilla/5.0 (iPhone; CPU iPhone OS 17_0) AppleWebKit", "mobile"),
|
||||||
|
("Mozilla/5.0 (iPad; CPU OS 17_0) AppleWebKit", "tablet"),
|
||||||
|
("Mozilla/5.0 (Windows NT 10.0; Win64; x64)", "desktop"),
|
||||||
|
("Googlebot/2.1", "bot"),
|
||||||
|
("", "(unknown)"),
|
||||||
|
])
|
||||||
|
def test_device_detection(ua, expected):
|
||||||
|
assert analytics.device_of(ua) == expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_only_iso_looking_country_headers_are_trusted():
|
||||||
|
assert analytics.country_of({"cf-ipcountry": "de"}) == "DE"
|
||||||
|
assert analytics.country_of({"cf-ipcountry": "Norway"}) == analytics.UNKNOWN
|
||||||
|
assert analytics.country_of({"cf-ipcountry": "XX"}) == analytics.UNKNOWN
|
||||||
|
assert analytics.country_of({}) == analytics.UNKNOWN
|
||||||
|
|
||||||
|
|
||||||
|
def test_error_labels_use_the_route_not_the_path():
|
||||||
|
class Route:
|
||||||
|
path = "/api/adventures/{adventure_id}"
|
||||||
|
|
||||||
|
assert analytics.api_route_label({"route": Route()}, 500) == "500 /api/adventures/{adventure_id}"
|
||||||
|
# An unmatched path is entirely attacker-chosen, so it never becomes a label.
|
||||||
|
assert analytics.api_route_label({}, 404) == "404 (unmatched)"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- The summary ----------
|
||||||
|
|
||||||
|
def test_summary_counts_people_once_per_step(db):
|
||||||
|
one, two = make_user(db), make_user(db)
|
||||||
|
for _ in range(3):
|
||||||
|
analytics.record_event(analytics.EV_SCENARIO_OPEN, one)
|
||||||
|
analytics.record_event(analytics.EV_TURN, one)
|
||||||
|
analytics.record_event(analytics.EV_SCENARIO_OPEN, two)
|
||||||
|
|
||||||
|
result = analytics.summary(db, days=7)
|
||||||
|
steps = {row["step"]: row["count"] for row in result["funnel"]}
|
||||||
|
assert steps["Visited"] == 2
|
||||||
|
assert steps["Opened a scenario"] == 2
|
||||||
|
assert steps["Played a turn"] == 1 # not 3 — one person, three turns
|
||||||
|
assert steps["Signed up"] == 0
|
||||||
|
# Raw event totals still count every occurrence.
|
||||||
|
assert result["totals"]["turns"] == 3
|
||||||
|
assert result["totals"]["visitors"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_series_covers_every_day_including_empty_ones(db):
|
||||||
|
analytics.record(analytics.M_PAGE, "/")
|
||||||
|
result = analytics.summary(db, days=7)
|
||||||
|
assert len(result["series"]) == 7
|
||||||
|
assert result["series"][-1]["day"] == models.utcnow().date().isoformat()
|
||||||
|
assert result["series"][-1]["pageviews"] == 1
|
||||||
|
assert result["series"][0]["pageviews"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_flushes_before_reading(db):
|
||||||
|
analytics.record(analytics.M_EVENT, analytics.EV_TURN)
|
||||||
|
# Never flushed by hand: the dashboard must not be up to a minute stale.
|
||||||
|
assert analytics.summary(db, days=1)["totals"]["turns"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_reports_pages_referrers_and_errors(db):
|
||||||
|
analytics.record(analytics.M_PAGE, "/play/:id", n=4)
|
||||||
|
analytics.record(analytics.M_REFERRER, "news.ycombinator.com", n=2)
|
||||||
|
analytics.record(analytics.M_ERROR, "500 /api/adventures/{adventure_id}")
|
||||||
|
result = analytics.summary(db, days=30)
|
||||||
|
assert result["pages"][0] == {"label": "/play/:id", "hits": 4}
|
||||||
|
assert result["referrers"][0]["label"] == "news.ycombinator.com"
|
||||||
|
assert result["totals"]["errors"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- The endpoints ----------
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def client(monkeypatch):
|
||||||
|
Base.metadata.create_all(bind=engine)
|
||||||
|
setup = SessionLocal()
|
||||||
|
visitor = models.User(is_guest=True)
|
||||||
|
owner = models.User(is_guest=False, email="owner@example.com")
|
||||||
|
setup.add_all([visitor, owner])
|
||||||
|
setup.commit()
|
||||||
|
ids = {"visitor": visitor.id, "owner": owner.id}
|
||||||
|
setup.close()
|
||||||
|
|
||||||
|
monkeypatch.setattr(limits, "rate_limit", lambda *a, **k: None)
|
||||||
|
# Multi-user is what makes the gate mean anything: local mode trusts
|
||||||
|
# whoever is at the keyboard, because it is the operator's own machine.
|
||||||
|
monkeypatch.setattr(auth, "MULTI_USER", True)
|
||||||
|
monkeypatch.setattr(auth, "ANALYTICS_EMAILS", {"owner@example.com"})
|
||||||
|
|
||||||
|
current = {"id": ids["visitor"]}
|
||||||
|
|
||||||
|
def _current_user(db=Depends(get_db)):
|
||||||
|
return db.get(models.User, current["id"])
|
||||||
|
|
||||||
|
app.dependency_overrides[auth.get_current_user] = _current_user
|
||||||
|
monkeypatch.setattr(
|
||||||
|
auth, "resolve_session_user", lambda request, db: db.get(models.User, current["id"])
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
client = TestClient(app)
|
||||||
|
client.ids, client.current = ids, current
|
||||||
|
yield client
|
||||||
|
finally:
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
Base.metadata.drop_all(bind=engine)
|
||||||
|
|
||||||
|
|
||||||
|
def read_summary(client, days=30):
|
||||||
|
return client.get(f"/api/analytics/summary?days={days}")
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_is_invisible_to_everyone_but_the_owner(client):
|
||||||
|
assert read_summary(client).status_code == 404
|
||||||
|
client.current["id"] = client.ids["owner"]
|
||||||
|
assert read_summary(client).status_code == 200
|
||||||
|
|
||||||
|
|
||||||
|
def test_collect_records_a_pageview_and_the_visit(client):
|
||||||
|
resp = client.post("/api/analytics/collect", json={"path": "/play/7", "first": True,
|
||||||
|
"referrer": "https://news.ycombinator.com/"})
|
||||||
|
assert resp.status_code == 204
|
||||||
|
client.current["id"] = client.ids["owner"]
|
||||||
|
body = read_summary(client).json()
|
||||||
|
assert body["pages"][0] == {"label": "/play/:id", "hits": 1}
|
||||||
|
assert body["referrers"][0]["label"] == "news.ycombinator.com"
|
||||||
|
assert body["totals"]["visitors"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_referrer_and_device_are_recorded_once_per_visit_not_per_view(client):
|
||||||
|
for path in ("/", "/scenarios", "/adventures"):
|
||||||
|
client.post("/api/analytics/collect", json={"path": path, "first": path == "/"})
|
||||||
|
client.current["id"] = client.ids["owner"]
|
||||||
|
body = read_summary(client).json()
|
||||||
|
assert body["totals"]["pageviews"] == 3
|
||||||
|
# Three views, one visit: the referral and the device are facts about the
|
||||||
|
# visit, so counting them per view would multiply every one of them.
|
||||||
|
assert sum(row["hits"] for row in body["devices"]) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_owners_own_visits_are_not_traffic(client):
|
||||||
|
client.current["id"] = client.ids["owner"]
|
||||||
|
client.post("/api/analytics/collect", json={"path": "/", "first": True})
|
||||||
|
assert read_summary(client).json()["totals"]["pageviews"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_client_cannot_invent_pages_or_events(client):
|
||||||
|
client.post("/api/analytics/collect", json={"path": "/../../admin", "first": True})
|
||||||
|
# There is no field for it, so a made-up event is not even expressible.
|
||||||
|
client.post("/api/analytics/collect", json={"path": "/", "event": "signup"})
|
||||||
|
client.current["id"] = client.ids["owner"]
|
||||||
|
body = read_summary(client).json()
|
||||||
|
assert {row["label"] for row in body["pages"]} == {"(other)", "/"}
|
||||||
|
assert body["totals"]["signups"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_errors_are_counted_by_route(client):
|
||||||
|
client.get("/api/adventures/999999")
|
||||||
|
client.current["id"] = client.ids["owner"]
|
||||||
|
errors = read_summary(client).json()["errors"]
|
||||||
|
assert errors and errors[0]["label"].startswith("404 /api/adventures/")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- The dialect the tests never run on ----------
|
||||||
|
|
||||||
|
def test_the_upserts_compile_for_postgres():
|
||||||
|
"""Prod is Neon; these tests are SQLite, and a failed flush is caught and
|
||||||
|
logged rather than raised. A dialect mistake would therefore be invisible
|
||||||
|
until the dashboard quietly stayed empty — so compile both statements
|
||||||
|
against Postgres without ever connecting to one.
|
||||||
|
"""
|
||||||
|
from sqlalchemy import create_engine
|
||||||
|
from sqlalchemy.dialects import postgresql
|
||||||
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
|
session = sessionmaker(bind=create_engine("postgresql+psycopg://u:p@localhost/db"))()
|
||||||
|
compiled = []
|
||||||
|
|
||||||
|
def capture(statement, *args, **kwargs):
|
||||||
|
compiled.append(str(statement.compile(dialect=postgresql.dialect())))
|
||||||
|
|
||||||
|
session.execute = capture
|
||||||
|
session.scalars = lambda *a, **k: []
|
||||||
|
|
||||||
|
analytics._write_counts(session, {("2026-01-01", "pageview", "/"): 2})
|
||||||
|
analytics._write_visits(session, {("2026-01-01", "f" * 32): {"played"}})
|
||||||
|
|
||||||
|
counts, visits = compiled
|
||||||
|
assert "ON CONFLICT (day, metric, label) DO UPDATE" in counts
|
||||||
|
assert "analytics_daily.hits + excluded.hits" in counts
|
||||||
|
assert "ON CONFLICT (day, visitor) DO UPDATE" in visits
|
||||||
|
assert "analytics_visitor_days.played OR excluded.played" in visits
|
||||||
|
# is_new is settled by the first write of a visitor's first day and must
|
||||||
|
# not be in the update clause at all.
|
||||||
|
assert "is_new" not in visits.split("DO UPDATE")[1]
|
||||||
@@ -3,7 +3,7 @@ per-account login throttle added to close it.
|
|||||||
|
|
||||||
Background: uvicorn's --forwarded-allow-ips "*" trusted the LEFTMOST
|
Background: uvicorn's --forwarded-allow-ips "*" trusted the LEFTMOST
|
||||||
X-Forwarded-For entry, which the client controls, so rotating the header
|
X-Forwarded-For entry, which the client controls, so rotating the header
|
||||||
handed out a fresh rate-limit bucket per request. _client_ip now reads the
|
handed out a fresh rate-limit bucket per request. client_ip now reads the
|
||||||
hop the trusted edge appends (rightmost), and login has an email-keyed throttle
|
hop the trusted edge appends (rightmost), and login has an email-keyed throttle
|
||||||
that no IP trick can dilute.
|
that no IP trick can dilute.
|
||||||
|
|
||||||
@@ -31,22 +31,22 @@ class _Req:
|
|||||||
self.client = None if peer is None else type("C", (), {"host": peer})()
|
self.client = None if peer is None else type("C", (), {"host": peer})()
|
||||||
|
|
||||||
|
|
||||||
# ---------- _client_ip: the spoof-resistant hop ----------
|
# ---------- client_ip: the spoof-resistant hop ----------
|
||||||
|
|
||||||
def test_client_ip_takes_appended_rightmost_hop(monkeypatch):
|
def test_client_ip_takes_appended_rightmost_hop(monkeypatch):
|
||||||
monkeypatch.setattr(limits, "TRUSTED_PROXY_HOPS", 1)
|
monkeypatch.setattr(limits, "TRUSTED_PROXY_HOPS", 1)
|
||||||
# Attacker prepends a fake IP; the edge appends the real one on the right.
|
# Attacker prepends a fake IP; the edge appends the real one on the right.
|
||||||
req = _Req("203.0.113.9, 198.51.100.77")
|
req = _Req("203.0.113.9, 198.51.100.77")
|
||||||
assert limits._client_ip(req) == "198.51.100.77"
|
assert limits.client_ip(req) == "198.51.100.77"
|
||||||
|
|
||||||
|
|
||||||
def test_client_ip_ignores_spoofed_leftmost(monkeypatch):
|
def test_client_ip_ignores_spoofed_leftmost(monkeypatch):
|
||||||
monkeypatch.setattr(limits, "TRUSTED_PROXY_HOPS", 1)
|
monkeypatch.setattr(limits, "TRUSTED_PROXY_HOPS", 1)
|
||||||
# Whatever the client stuffs to the left, the keyed IP stays the real hop —
|
# Whatever the client stuffs to the left, the keyed IP stays the real hop —
|
||||||
# so rotating it no longer mints a new bucket.
|
# so rotating it no longer mints a new bucket.
|
||||||
a = limits._client_ip(_Req("1.1.1.1, 198.51.100.77"))
|
a = limits.client_ip(_Req("1.1.1.1, 198.51.100.77"))
|
||||||
b = limits._client_ip(_Req("2.2.2.2, 198.51.100.77"))
|
b = limits.client_ip(_Req("2.2.2.2, 198.51.100.77"))
|
||||||
c = limits._client_ip(_Req("evil, junk, 198.51.100.77"))
|
c = limits.client_ip(_Req("evil, junk, 198.51.100.77"))
|
||||||
assert a == b == c == "198.51.100.77"
|
assert a == b == c == "198.51.100.77"
|
||||||
|
|
||||||
|
|
||||||
@@ -54,12 +54,12 @@ def test_client_ip_honours_extra_trusted_hops(monkeypatch):
|
|||||||
monkeypatch.setattr(limits, "TRUSTED_PROXY_HOPS", 2)
|
monkeypatch.setattr(limits, "TRUSTED_PROXY_HOPS", 2)
|
||||||
# Two trusted hops: real client is second from the right.
|
# Two trusted hops: real client is second from the right.
|
||||||
req = _Req("9.9.9.9, 203.0.113.5, 198.51.100.77")
|
req = _Req("9.9.9.9, 203.0.113.5, 198.51.100.77")
|
||||||
assert limits._client_ip(req) == "203.0.113.5"
|
assert limits.client_ip(req) == "203.0.113.5"
|
||||||
|
|
||||||
|
|
||||||
def test_client_ip_falls_back_to_socket_peer():
|
def test_client_ip_falls_back_to_socket_peer():
|
||||||
assert limits._client_ip(_Req(None, peer="172.16.0.4")) == "172.16.0.4"
|
assert limits.client_ip(_Req(None, peer="172.16.0.4")) == "172.16.0.4"
|
||||||
assert limits._client_ip(_Req(None, peer=None)) == "unknown"
|
assert limits.client_ip(_Req(None, peer=None)) == "unknown"
|
||||||
|
|
||||||
|
|
||||||
# ---------- per-account login throttle ----------
|
# ---------- per-account login throttle ----------
|
||||||
|
|||||||
@@ -23,6 +23,7 @@ what makes it a service rather than a demo. Part 4 is the web plumbing, kept sho
|
|||||||
- [1.8 Why there is no agent framework](#18-why-there-is-no-agent-framework)
|
- [1.8 Why there is no agent framework](#18-why-there-is-no-agent-framework)
|
||||||
- [Part 2 — Data and correctness](#part-2--data-and-correctness)
|
- [Part 2 — Data and correctness](#part-2--data-and-correctness)
|
||||||
- [Part 3 — Production concerns](#part-3--production-concerns)
|
- [Part 3 — Production concerns](#part-3--production-concerns)
|
||||||
|
- [3.6 Counting visits](#36-counting-visits)
|
||||||
- [Part 4 — The web plumbing, briefly](#part-4--the-web-plumbing-briefly)
|
- [Part 4 — The web plumbing, briefly](#part-4--the-web-plumbing-briefly)
|
||||||
- [Part 5 — Measured results and known limitations](#part-5--measured-results-and-known-limitations)
|
- [Part 5 — Measured results and known limitations](#part-5--measured-results-and-known-limitations)
|
||||||
|
|
||||||
@@ -1048,6 +1049,43 @@ Two things worth knowing about the free tier:
|
|||||||
CI runs the backend tests, the frontend lint and build, and a Docker image build on every
|
CI runs the backend tests, the frontend lint and build, and a Docker image build on every
|
||||||
push.
|
push.
|
||||||
|
|
||||||
|
## 3.6 Counting visits
|
||||||
|
|
||||||
|
A hosted demo raises a question a local app never does: is anyone using it, and do they get
|
||||||
|
anywhere? The answer is an owner-only dashboard at `/analytics`, gated on
|
||||||
|
`AIDND_ANALYTICS_EMAILS` — a list kept separate from `AIDND_POWER_USERS`, since an unmetered
|
||||||
|
tester is not automatically someone who should see the traffic.
|
||||||
|
|
||||||
|
**Why it isn't a third-party script.** The CSP allows `script-src 'self'`, so a tracker would
|
||||||
|
mean loosening it; adblockers eat the popular ones, which silently biases exactly the
|
||||||
|
technical audience this project is shown to; and none of them can see the measurement that
|
||||||
|
matters here — a *turn*. The interesting funnel step is not a pageview.
|
||||||
|
|
||||||
|
**Egress is the budget.** After the 189x fix (§2.5) it would be perverse to add a feature
|
||||||
|
that reads rows per request. So counts accumulate in a process-local dict and flush every 60
|
||||||
|
seconds as UPSERTs: **a visit is a write and never a read**. Storage is a generic
|
||||||
|
`(day, metric, label) -> hits` counter plus one row per visitor per day for the funnel flags.
|
||||||
|
Every dashboard query is a `GROUP BY` that returns tens of rows regardless of the traffic
|
||||||
|
behind it, so a month costs a few kilobytes to read back. The cost of the buffer is that a
|
||||||
|
hard restart can lose up to a minute; the flusher also runs on shutdown, and on a tier that
|
||||||
|
sleeps when idle the buffer it sleeps on is empty anyway.
|
||||||
|
|
||||||
|
**The numbers are the server's, not the browser's.** The client reports one fact — which page
|
||||||
|
was viewed — and even that is normalized to a route (`/play/12` → `/play/:id`) against a
|
||||||
|
whitelist, so the page list cannot be polluted by anything a stranger posts. Everything that
|
||||||
|
means something — a turn, an adventure, a sign-up — is recorded by the code that performs it.
|
||||||
|
That also fixes a blind spot: a failed turn is an HTTP 200 with a bad ending, so a
|
||||||
|
status-code tally cannot see it, and a demo whose model has started refusing looks perfectly
|
||||||
|
healthy from outside. `turn_error` is counted where the SSE error is written.
|
||||||
|
|
||||||
|
The funnel counts **people, not clicks** — a player who starts six adventures is one person
|
||||||
|
who started an adventure — which is the entire reason the per-visitor-day table exists.
|
||||||
|
|
||||||
|
One smaller decision worth naming: error buckets are labelled by the matched *route template*,
|
||||||
|
never the requested path. That gives one bucket per endpoint instead of one per adventure id,
|
||||||
|
and — the reason it isn't merely tidier — an unmatched path is entirely attacker-chosen, so
|
||||||
|
labelling by it would let anyone mint rows.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Part 4 — The web plumbing, briefly
|
# Part 4 — The web plumbing, briefly
|
||||||
|
|||||||
+25
-2
@@ -1,5 +1,5 @@
|
|||||||
import { useEffect, useState } from 'react'
|
import { useEffect, useRef, useState } from 'react'
|
||||||
import { NavLink, Outlet } from 'react-router-dom'
|
import { NavLink, Outlet, useLocation } from 'react-router-dom'
|
||||||
import { api } from './api'
|
import { api } from './api'
|
||||||
import { AuthModal, ToastHost } from './components'
|
import { AuthModal, ToastHost } from './components'
|
||||||
import Embers from './Embers.jsx'
|
import Embers from './Embers.jsx'
|
||||||
@@ -9,11 +9,28 @@ export default function App() {
|
|||||||
const [me, setMe] = useState(null)
|
const [me, setMe] = useState(null)
|
||||||
const [authMode, setAuthMode] = useState(null) // 'register' | 'login' | null
|
const [authMode, setAuthMode] = useState(null) // 'register' | 'login' | null
|
||||||
const [navOpen, setNavOpen] = useState(false) // mobile hamburger menu
|
const [navOpen, setNavOpen] = useState(false) // mobile hamburger menu
|
||||||
|
const location = useLocation()
|
||||||
|
const lastPath = useRef(null)
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
api.getMe().then(setMe).catch(() => {})
|
api.getMe().then(setMe).catch(() => {})
|
||||||
}, [])
|
}, [])
|
||||||
|
|
||||||
|
// One pageview per route the reader actually lands on. Guarded on the path
|
||||||
|
// rather than fired on every render: StrictMode runs effects twice in dev,
|
||||||
|
// and a re-render for unrelated state is not a new page.
|
||||||
|
useEffect(() => {
|
||||||
|
if (lastPath.current === location.pathname) return
|
||||||
|
const first = lastPath.current === null
|
||||||
|
lastPath.current = location.pathname
|
||||||
|
// document.referrer survives client-side navigation, so it is only honest
|
||||||
|
// on the first view — after that this was our own page, not a referral.
|
||||||
|
api.trackPageview(location.pathname, {
|
||||||
|
referrer: first ? document.referrer : '',
|
||||||
|
first,
|
||||||
|
})
|
||||||
|
}, [location.pathname])
|
||||||
|
|
||||||
const onAuthed = (newMe, mode) => {
|
const onAuthed = (newMe, mode) => {
|
||||||
setAuthMode(null)
|
setAuthMode(null)
|
||||||
if (mode === 'login') {
|
if (mode === 'login') {
|
||||||
@@ -64,6 +81,12 @@ export default function App() {
|
|||||||
AI Chat
|
AI Chat
|
||||||
</NavLink>
|
</NavLink>
|
||||||
)}
|
)}
|
||||||
|
{/* Owner only: the site's own traffic, on its own allowlist. */}
|
||||||
|
{me?.analytics && (
|
||||||
|
<NavLink to="/analytics" className={({ isActive }) => `navlink${isActive ? ' active' : ''}`}>
|
||||||
|
Visitors
|
||||||
|
</NavLink>
|
||||||
|
)}
|
||||||
{me?.multi_user && (
|
{me?.multi_user && (
|
||||||
<div className="nav-account">
|
<div className="nav-account">
|
||||||
{me.is_guest ? (
|
{me.is_guest ? (
|
||||||
|
|||||||
@@ -61,6 +61,28 @@ async function streamSSE(path, payload, onEvent, signal, isRetry = false) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export const api = {
|
export const api = {
|
||||||
|
// Analytics. The beacon is deliberately not a `request()`: it must never
|
||||||
|
// retry, never bootstrap a session, and never surface an error — a counter
|
||||||
|
// that can interrupt the app it is counting is worse than no counter.
|
||||||
|
trackPageview: (path, { referrer = '', first = false } = {}) => {
|
||||||
|
try {
|
||||||
|
fetch('/api/analytics/collect', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ path, referrer, first }),
|
||||||
|
keepalive: true,
|
||||||
|
}).catch(() => {})
|
||||||
|
} catch { /* no beacon, no problem */ }
|
||||||
|
},
|
||||||
|
getAnalytics: (days) => request(`/analytics/summary?days=${days}`),
|
||||||
|
getAccessLog: ({ beforeId, kind, q, limit = 50 } = {}) => {
|
||||||
|
const params = new URLSearchParams({ limit })
|
||||||
|
if (beforeId != null) params.set('before_id', beforeId)
|
||||||
|
if (kind) params.set('kind', kind)
|
||||||
|
if (q) params.set('q', q)
|
||||||
|
return request(`/analytics/access?${params}`)
|
||||||
|
},
|
||||||
|
|
||||||
// Auth (Phase 8 — no-ops in local mode beyond getMe)
|
// Auth (Phase 8 — no-ops in local mode beyond getMe)
|
||||||
getMe: () => request('/auth/me'),
|
getMe: () => request('/auth/me'),
|
||||||
register: (email, password) =>
|
register: (email, password) =>
|
||||||
|
|||||||
@@ -13,6 +13,13 @@
|
|||||||
--accent-glow: rgba(212, 169, 78, 0.25);
|
--accent-glow: rgba(212, 169, 78, 0.25);
|
||||||
--danger: #d06565;
|
--danger: #d06565;
|
||||||
--player: #9fc7d1;
|
--player: #9fc7d1;
|
||||||
|
/* Chart hues (analytics dashboard). Deeper and more saturated than --accent
|
||||||
|
and --player, which are tuned for text and borders and turn muddy once
|
||||||
|
they are a 10px bar: these are checked against --bg-panel for lightness,
|
||||||
|
chroma, contrast and colour-blind separation as a set. */
|
||||||
|
--chart-1: #b58a30;
|
||||||
|
--chart-2: #3d8ac4;
|
||||||
|
--chart-3: #a8608f;
|
||||||
--font-display: 'Cinzel', Georgia, serif;
|
--font-display: 'Cinzel', Georgia, serif;
|
||||||
--font-story: 'Crimson Pro', Georgia, 'Times New Roman', serif;
|
--font-story: 'Crimson Pro', Georgia, 'Times New Roman', serif;
|
||||||
--font-ui: 'Inter', 'Segoe UI', system-ui, sans-serif;
|
--font-ui: 'Inter', 'Segoe UI', system-ui, sans-serif;
|
||||||
@@ -2333,6 +2340,293 @@ button.primary.compact { padding: 3px 12px; font-size: 0.76rem; margin-left: aut
|
|||||||
.card.tome:hover { transform: none; }
|
.card.tome:hover { transform: none; }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ---------- Visit analytics (owner dashboard) ---------- */
|
||||||
|
|
||||||
|
/* Wider than the reading pages: this one is a grid of small figures, not
|
||||||
|
prose, and 960px puts two charts and eight tiles into a column. */
|
||||||
|
.an-page { max-width: 1180px; }
|
||||||
|
.an-ranges { display: flex; gap: 6px; }
|
||||||
|
|
||||||
|
/* Four across, not auto-fit: there are eight tiles, and letting them flow
|
||||||
|
leaves a single orphan on the second row at most widths. */
|
||||||
|
.an-tiles {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(4, minmax(0, 1fr));
|
||||||
|
gap: 12px;
|
||||||
|
margin-bottom: 18px;
|
||||||
|
}
|
||||||
|
@media (max-width: 900px) {
|
||||||
|
.an-tiles { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
||||||
|
}
|
||||||
|
.an-tile {
|
||||||
|
background: var(--bg-panel);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 10px;
|
||||||
|
padding: 14px 16px;
|
||||||
|
}
|
||||||
|
.an-tile-value {
|
||||||
|
font-family: var(--font-display);
|
||||||
|
font-size: 1.7rem;
|
||||||
|
line-height: 1.1;
|
||||||
|
color: var(--accent-bright);
|
||||||
|
}
|
||||||
|
.an-tile-label { font-size: 0.82rem; color: var(--text); margin-top: 4px; }
|
||||||
|
.an-tile-hint { font-size: 0.72rem; color: var(--text-dim); margin-top: 2px; }
|
||||||
|
|
||||||
|
.an-grid {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(340px, 1fr));
|
||||||
|
gap: 16px;
|
||||||
|
margin-bottom: 16px;
|
||||||
|
}
|
||||||
|
.an-card {
|
||||||
|
position: relative;
|
||||||
|
background: var(--bg-panel);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 12px;
|
||||||
|
padding: 16px 18px 14px;
|
||||||
|
margin-bottom: 16px;
|
||||||
|
}
|
||||||
|
.an-grid .an-card { margin-bottom: 0; }
|
||||||
|
.an-card-head {
|
||||||
|
display: flex;
|
||||||
|
align-items: baseline;
|
||||||
|
justify-content: space-between;
|
||||||
|
gap: 12px;
|
||||||
|
margin-bottom: 14px;
|
||||||
|
}
|
||||||
|
.an-card-head h2 {
|
||||||
|
margin: 0;
|
||||||
|
font-family: var(--font-display);
|
||||||
|
font-size: 0.95rem;
|
||||||
|
letter-spacing: 0.06em;
|
||||||
|
color: var(--accent);
|
||||||
|
}
|
||||||
|
.an-note { font-size: 0.72rem; color: var(--text-dim); }
|
||||||
|
.an-empty { color: var(--text-dim); font-style: italic; padding: 18px 0; text-align: center; }
|
||||||
|
|
||||||
|
.an-legend { display: flex; gap: 12px; font-size: 0.72rem; color: var(--text-dim); }
|
||||||
|
.an-legend-item { display: inline-flex; align-items: center; gap: 5px; }
|
||||||
|
.an-swatch { width: 9px; height: 9px; border-radius: 2px; display: inline-block; flex: none; }
|
||||||
|
|
||||||
|
/* ----- Day columns ----- */
|
||||||
|
|
||||||
|
.an-plot { position: relative; height: 170px; margin-top: 12px; }
|
||||||
|
.an-gridline {
|
||||||
|
position: absolute;
|
||||||
|
left: 0;
|
||||||
|
right: 0;
|
||||||
|
border-top: 1px dashed var(--border);
|
||||||
|
pointer-events: none;
|
||||||
|
}
|
||||||
|
.an-gridline span {
|
||||||
|
position: absolute;
|
||||||
|
right: 0;
|
||||||
|
top: -0.62em;
|
||||||
|
font-size: 0.68rem;
|
||||||
|
color: var(--text-dim);
|
||||||
|
background: var(--bg-panel);
|
||||||
|
padding: 0 4px;
|
||||||
|
/* The dashed rule belongs behind the bars; its value does not. Columns are
|
||||||
|
positioned, so without this the tallest bar swallows the label. */
|
||||||
|
z-index: 3;
|
||||||
|
}
|
||||||
|
.an-columns { display: flex; align-items: flex-end; gap: 2px; height: 100%; }
|
||||||
|
.an-column {
|
||||||
|
position: relative;
|
||||||
|
flex: 1 1 0;
|
||||||
|
min-width: 0;
|
||||||
|
height: 100%;
|
||||||
|
display: flex;
|
||||||
|
align-items: flex-end;
|
||||||
|
border-radius: 4px 4px 0 0;
|
||||||
|
outline: none;
|
||||||
|
}
|
||||||
|
.an-column.hot { background: rgba(212, 169, 78, 0.07); }
|
||||||
|
.an-stack {
|
||||||
|
width: 100%;
|
||||||
|
/* Capped so a 7-day range draws bars and not slabs; the column around it
|
||||||
|
stays full width, so the hover target does not shrink with the mark. */
|
||||||
|
max-width: 44px;
|
||||||
|
margin: 0 auto;
|
||||||
|
height: 100%;
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
justify-content: flex-end;
|
||||||
|
/* A 2px gap of surface between stacked segments: the boundary reads as a
|
||||||
|
boundary without a border, which would eat a thin bar entirely. */
|
||||||
|
gap: 2px;
|
||||||
|
}
|
||||||
|
.an-bar { width: 100%; transition: filter 0.15s; }
|
||||||
|
.an-stack > .an-bar:first-child { border-radius: 4px 4px 0 0; }
|
||||||
|
.an-column.hot .an-bar { filter: brightness(1.2); }
|
||||||
|
.an-tip {
|
||||||
|
position: absolute;
|
||||||
|
bottom: calc(100% + 8px);
|
||||||
|
left: 50%;
|
||||||
|
transform: translateX(-50%);
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 3px;
|
||||||
|
padding: 8px 10px;
|
||||||
|
background: var(--bg-panel);
|
||||||
|
border: 1px solid var(--border-bright);
|
||||||
|
border-radius: 8px;
|
||||||
|
box-shadow: 0 10px 26px rgba(0, 0, 0, 0.55);
|
||||||
|
font-size: 0.74rem;
|
||||||
|
white-space: nowrap;
|
||||||
|
pointer-events: none;
|
||||||
|
z-index: 4;
|
||||||
|
}
|
||||||
|
.an-tip strong { font-weight: 600; color: var(--accent-bright); }
|
||||||
|
.an-tip span { display: inline-flex; align-items: center; gap: 6px; color: var(--text-dim); }
|
||||||
|
|
||||||
|
.an-axis { display: flex; gap: 2px; margin-top: 7px; }
|
||||||
|
.an-axis span {
|
||||||
|
flex: 1 1 0;
|
||||||
|
min-width: 0;
|
||||||
|
font-size: 0.68rem;
|
||||||
|
color: var(--text-dim);
|
||||||
|
text-align: center;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
.an-axis span:first-child { text-align: left; }
|
||||||
|
.an-axis span:last-child { text-align: right; }
|
||||||
|
|
||||||
|
.an-overlay-empty {
|
||||||
|
position: absolute;
|
||||||
|
inset: 0;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
color: var(--text-dim);
|
||||||
|
font-style: italic;
|
||||||
|
font-size: 0.85rem;
|
||||||
|
pointer-events: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ----- Funnel ----- */
|
||||||
|
|
||||||
|
.an-funnel { display: flex; flex-direction: column; gap: 10px; }
|
||||||
|
.an-funnel-row {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 168px 1fr 96px;
|
||||||
|
align-items: center;
|
||||||
|
gap: 12px;
|
||||||
|
}
|
||||||
|
.an-funnel-label { font-size: 0.84rem; }
|
||||||
|
.an-funnel-track { height: 14px; background: var(--bg-input); border-radius: 7px; overflow: hidden; }
|
||||||
|
.an-funnel-bar {
|
||||||
|
height: 100%;
|
||||||
|
min-width: 2px;
|
||||||
|
background: var(--chart-1);
|
||||||
|
border-radius: 7px;
|
||||||
|
transition: width 0.4s ease;
|
||||||
|
}
|
||||||
|
.an-funnel-value {
|
||||||
|
text-align: right;
|
||||||
|
font-size: 0.86rem;
|
||||||
|
display: flex;
|
||||||
|
justify-content: flex-end;
|
||||||
|
align-items: baseline;
|
||||||
|
gap: 8px;
|
||||||
|
}
|
||||||
|
.an-funnel-share { font-size: 0.72rem; color: var(--text-dim); }
|
||||||
|
|
||||||
|
/* ----- Ranked lists ----- */
|
||||||
|
|
||||||
|
.an-list { list-style: none; margin: 0; padding: 0; display: flex; flex-direction: column; gap: 3px; }
|
||||||
|
.an-list li {
|
||||||
|
position: relative;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
gap: 12px;
|
||||||
|
padding: 7px 10px;
|
||||||
|
border-radius: 6px;
|
||||||
|
font-size: 0.84rem;
|
||||||
|
}
|
||||||
|
/* The bar sits behind the row rather than beside it, so a long label keeps the
|
||||||
|
full width and the magnitude is still visible at a glance. */
|
||||||
|
.an-list-fill {
|
||||||
|
position: absolute;
|
||||||
|
top: 0;
|
||||||
|
bottom: 0;
|
||||||
|
left: 0;
|
||||||
|
background: rgba(181, 138, 48, 0.2);
|
||||||
|
border-radius: 6px;
|
||||||
|
}
|
||||||
|
.an-list-label {
|
||||||
|
position: relative;
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
.an-list-value { position: relative; color: var(--text-dim); flex: none; }
|
||||||
|
|
||||||
|
/* ----- Tabs, and the access log table ----- */
|
||||||
|
|
||||||
|
.an-tabs { display: flex; gap: 4px; margin-bottom: 16px; border-bottom: 1px solid var(--border); }
|
||||||
|
.an-tabs button {
|
||||||
|
background: none;
|
||||||
|
border: none;
|
||||||
|
border-bottom: 2px solid transparent;
|
||||||
|
border-radius: 0;
|
||||||
|
padding: 8px 14px;
|
||||||
|
color: var(--text-dim);
|
||||||
|
font-family: var(--font-display);
|
||||||
|
font-size: 0.82rem;
|
||||||
|
letter-spacing: 0.06em;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.an-tabs button:hover { color: var(--text); }
|
||||||
|
.an-tabs button.active { color: var(--accent-bright); border-bottom-color: var(--accent); }
|
||||||
|
|
||||||
|
.an-log-head { flex-wrap: wrap; row-gap: 10px; }
|
||||||
|
.an-log-search { max-width: 260px; }
|
||||||
|
|
||||||
|
/* The table is the one thing here with a minimum width — six columns of real
|
||||||
|
data don't compress. It scrolls inside its own box so the page never does. */
|
||||||
|
.an-table-scroll { overflow-x: auto; }
|
||||||
|
.an-table { width: 100%; min-width: 620px; border-collapse: collapse; font-size: 0.82rem; }
|
||||||
|
.an-table th {
|
||||||
|
text-align: left;
|
||||||
|
font-weight: 500;
|
||||||
|
font-size: 0.72rem;
|
||||||
|
letter-spacing: 0.08em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
color: var(--text-dim);
|
||||||
|
padding: 0 10px 8px;
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
.an-table td { padding: 8px 10px; border-bottom: 1px solid var(--border); white-space: nowrap; }
|
||||||
|
.an-table tr:last-child td { border-bottom: none; }
|
||||||
|
.an-table tbody tr:hover { background: rgba(212, 169, 78, 0.05); }
|
||||||
|
/* A failed attempt is the row you are scanning for; it gets the only colour in
|
||||||
|
the table, on a border rather than the text, which stays readable. */
|
||||||
|
.an-table tr.failed td:first-child { box-shadow: inset 2px 0 var(--danger); }
|
||||||
|
.an-table tr.failed td:nth-child(3) { color: var(--danger); }
|
||||||
|
.an-cell-dim { color: var(--text-dim); }
|
||||||
|
.an-cell-mono { font-family: ui-monospace, 'Cascadia Code', Consolas, monospace; }
|
||||||
|
.an-tag {
|
||||||
|
margin-left: 8px;
|
||||||
|
padding: 1px 6px;
|
||||||
|
border: 1px solid var(--border-bright);
|
||||||
|
border-radius: 10px;
|
||||||
|
font-size: 0.66rem;
|
||||||
|
color: var(--text-dim);
|
||||||
|
}
|
||||||
|
.an-more { display: flex; justify-content: center; padding-top: 14px; }
|
||||||
|
|
||||||
|
.an-footnote {
|
||||||
|
color: var(--text-dim);
|
||||||
|
font-size: 0.74rem;
|
||||||
|
line-height: 1.6;
|
||||||
|
margin: 22px 0 0;
|
||||||
|
max-width: 70ch;
|
||||||
|
}
|
||||||
|
|
||||||
/* ============================================================
|
/* ============================================================
|
||||||
Mobile / narrow screens (≤ 720px)
|
Mobile / narrow screens (≤ 720px)
|
||||||
The desktop Play screen is a horizontal row of up to four columns
|
The desktop Play screen is a horizontal row of up to four columns
|
||||||
@@ -2536,4 +2830,19 @@ button.primary.compact { padding: 3px 12px; font-size: 0.76rem; margin-left: aut
|
|||||||
.branch-map-canvas { padding: 4px 6px 8px; }
|
.branch-map-canvas { padding: 4px 6px 8px; }
|
||||||
.branch-map-detail { padding: 12px 16px 16px; }
|
.branch-map-detail { padding: 12px 16px 16px; }
|
||||||
.branch-map-hint { padding: 0 16px 10px; }
|
.branch-map-hint { padding: 0 16px 10px; }
|
||||||
|
|
||||||
|
/* ---------- Analytics dashboard ---------- */
|
||||||
|
.an-page .page-header { flex-direction: column; align-items: flex-start; gap: 10px; }
|
||||||
|
.an-tiles { gap: 8px; }
|
||||||
|
.an-tile { padding: 10px 12px; }
|
||||||
|
.an-tile-value { font-size: 1.35rem; }
|
||||||
|
.an-grid { grid-template-columns: 1fr; }
|
||||||
|
.an-plot { height: 140px; }
|
||||||
|
.an-tabs button { padding: 8px 10px; font-size: 0.78rem; }
|
||||||
|
.an-log-search { max-width: none; width: 100%; }
|
||||||
|
/* Three columns don't fit; the label goes above its own bar instead. */
|
||||||
|
.an-funnel-row { grid-template-columns: 1fr auto; gap: 4px 10px; }
|
||||||
|
.an-funnel-track { grid-column: 1; }
|
||||||
|
.an-funnel-label { grid-column: 1 / -1; }
|
||||||
|
.an-funnel-value { grid-column: 2; }
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import Scripts from './pages/Scripts.jsx'
|
|||||||
import ScriptEditor from './pages/ScriptEditor.jsx'
|
import ScriptEditor from './pages/ScriptEditor.jsx'
|
||||||
import Settings from './pages/Settings.jsx'
|
import Settings from './pages/Settings.jsx'
|
||||||
import Chat from './pages/Chat.jsx'
|
import Chat from './pages/Chat.jsx'
|
||||||
|
import Analytics from './pages/Analytics.jsx'
|
||||||
import { trackKeyboardInset } from './keyboard.js'
|
import { trackKeyboardInset } from './keyboard.js'
|
||||||
import './index.css'
|
import './index.css'
|
||||||
|
|
||||||
@@ -31,6 +32,8 @@ const router = createBrowserRouter([
|
|||||||
{ path: 'settings', element: <Settings /> },
|
{ path: 'settings', element: <Settings /> },
|
||||||
// Power users only — the page redirects home and the API 404s otherwise.
|
// Power users only — the page redirects home and the API 404s otherwise.
|
||||||
{ path: 'chat', element: <Chat /> },
|
{ path: 'chat', element: <Chat /> },
|
||||||
|
// Owner only, by a separate allowlist; same redirect-and-404 treatment.
|
||||||
|
{ path: 'analytics', element: <Analytics /> },
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
])
|
])
|
||||||
|
|||||||
@@ -0,0 +1,476 @@
|
|||||||
|
/* Visit analytics — the owner's view of who came by and what they did.
|
||||||
|
|
||||||
|
Everything on this page arrives in a single aggregate response (see
|
||||||
|
backend/app/analytics.py), so changing the range is one small request, not a
|
||||||
|
scan of anything. The page is hidden from everyone else: the nav link is
|
||||||
|
gated on `me.analytics`, this component bounces, and the API 404s.
|
||||||
|
|
||||||
|
Charts are plain HTML — a flex row of columns, a row of bars — rather than
|
||||||
|
SVG or a charting library. At this size that is less code, responsive for
|
||||||
|
free, and keeps the CSP as tight as it is. */
|
||||||
|
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
|
||||||
|
import { useNavigate, useOutletContext } from 'react-router-dom'
|
||||||
|
import { api } from '../api'
|
||||||
|
|
||||||
|
const RANGES = [
|
||||||
|
{ days: 7, label: '7 days' },
|
||||||
|
{ days: 30, label: '30 days' },
|
||||||
|
{ days: 90, label: '90 days' },
|
||||||
|
]
|
||||||
|
|
||||||
|
const nf = new Intl.NumberFormat()
|
||||||
|
|
||||||
|
// Weekday + day for a short range, day + month for a long one: a 90-day axis
|
||||||
|
// has no room for "Mon".
|
||||||
|
function dayLabel(iso, days) {
|
||||||
|
const date = new Date(`${iso}T00:00:00Z`)
|
||||||
|
const opts = days <= 14
|
||||||
|
? { weekday: 'short', timeZone: 'UTC' }
|
||||||
|
: { day: 'numeric', month: 'short', timeZone: 'UTC' }
|
||||||
|
return date.toLocaleDateString(undefined, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
function fullDate(iso) {
|
||||||
|
return new Date(`${iso}T00:00:00Z`)
|
||||||
|
.toLocaleDateString(undefined, { dateStyle: 'medium', timeZone: 'UTC' })
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- Pieces ---------- */
|
||||||
|
|
||||||
|
function StatTile({ label, value, hint }) {
|
||||||
|
return (
|
||||||
|
<div className="an-tile" title={hint || undefined}>
|
||||||
|
<div className="an-tile-value">{nf.format(value ?? 0)}</div>
|
||||||
|
<div className="an-tile-label">{label}</div>
|
||||||
|
{hint && <div className="an-tile-hint">{hint}</div>}
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/* A day-by-day column chart. `series` names the stacked segments bottom-up;
|
||||||
|
one segment means one plain bar and no legend, since the title already says
|
||||||
|
what it is. */
|
||||||
|
function DayChart({ title, data, series, days }) {
|
||||||
|
const [hover, setHover] = useState(null)
|
||||||
|
const max = Math.max(1, ...data.map((d) => series.reduce((sum, s) => sum + (d[s.key] || 0), 0)))
|
||||||
|
// Only ever three x labels. Every column labelled is unreadable at 90 days
|
||||||
|
// and redundant at 7 — the tooltip carries the exact date either way.
|
||||||
|
const ticks = new Set([0, Math.floor((data.length - 1) / 2), data.length - 1])
|
||||||
|
const empty = data.every((d) => series.every((s) => !d[s.key]))
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section className="an-card an-chart">
|
||||||
|
<header className="an-card-head">
|
||||||
|
<h2>{title}</h2>
|
||||||
|
{series.length > 1 && (
|
||||||
|
<div className="an-legend">
|
||||||
|
{[...series].reverse().map((s) => (
|
||||||
|
<span key={s.key} className="an-legend-item">
|
||||||
|
<i className="an-swatch" style={{ background: s.color }} />
|
||||||
|
{s.label}
|
||||||
|
</span>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</header>
|
||||||
|
<div className="an-plot" onMouseLeave={() => setHover(null)}>
|
||||||
|
<div className="an-gridline" style={{ bottom: '100%' }}><span>{nf.format(max)}</span></div>
|
||||||
|
<div className="an-gridline" style={{ bottom: '50%' }}><span>{nf.format(Math.round(max / 2))}</span></div>
|
||||||
|
<div className="an-columns">
|
||||||
|
{data.map((point, i) => {
|
||||||
|
const total = series.reduce((sum, s) => sum + (point[s.key] || 0), 0)
|
||||||
|
return (
|
||||||
|
<div
|
||||||
|
key={point.day}
|
||||||
|
className={`an-column${hover === i ? ' hot' : ''}`}
|
||||||
|
onMouseEnter={() => setHover(i)}
|
||||||
|
onFocus={() => setHover(i)}
|
||||||
|
tabIndex={0}
|
||||||
|
aria-label={`${fullDate(point.day)}: ${total}`}
|
||||||
|
>
|
||||||
|
<div className="an-stack">
|
||||||
|
{[...series].reverse().map((s) => (
|
||||||
|
(point[s.key] || 0) > 0 && (
|
||||||
|
<div
|
||||||
|
key={s.key}
|
||||||
|
className="an-bar"
|
||||||
|
style={{
|
||||||
|
height: `${((point[s.key] || 0) / max) * 100}%`,
|
||||||
|
background: s.color,
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
)
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
{hover === i && (
|
||||||
|
<div className="an-tip">
|
||||||
|
<strong>{fullDate(point.day)}</strong>
|
||||||
|
{series.map((s) => (
|
||||||
|
<span key={s.key}>
|
||||||
|
<i className="an-swatch" style={{ background: s.color }} />
|
||||||
|
{s.label}: {nf.format(point[s.key] || 0)}
|
||||||
|
</span>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="an-axis">
|
||||||
|
{data.map((point, i) => (
|
||||||
|
<span key={point.day}>{ticks.has(i) ? dayLabel(point.day, days) : ''}</span>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
{empty && <div className="an-overlay-empty">Nothing recorded in this range</div>}
|
||||||
|
</section>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The funnel. Each step's bar is drawn against the first step, so the shape of
|
||||||
|
the drop-off is the picture; the percentage beside it is of the step above,
|
||||||
|
which is the number you act on. */
|
||||||
|
function Funnel({ steps }) {
|
||||||
|
const top = steps[0]?.count || 0
|
||||||
|
return (
|
||||||
|
<section className="an-card">
|
||||||
|
<header className="an-card-head">
|
||||||
|
<h2>Where visitors get to</h2>
|
||||||
|
<span className="an-note">people, counted once each</span>
|
||||||
|
</header>
|
||||||
|
{top === 0 ? (
|
||||||
|
<div className="an-empty">No visitors in this range.</div>
|
||||||
|
) : (
|
||||||
|
<div className="an-funnel">
|
||||||
|
{steps.map((step, i) => {
|
||||||
|
const previous = i === 0 ? step.count : steps[i - 1].count
|
||||||
|
const share = previous ? Math.round((step.count / previous) * 100) : 0
|
||||||
|
return (
|
||||||
|
<div key={step.step} className="an-funnel-row">
|
||||||
|
<div className="an-funnel-label">{step.step}</div>
|
||||||
|
<div className="an-funnel-track">
|
||||||
|
<div
|
||||||
|
className="an-funnel-bar"
|
||||||
|
style={{ width: `${top ? (step.count / top) * 100 : 0}%` }}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<div className="an-funnel-value">
|
||||||
|
{nf.format(step.count)}
|
||||||
|
{i > 0 && <span className="an-funnel-share">{share}%</span>}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</section>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
function TopList({ title, rows, note, empty = 'Nothing yet' }) {
|
||||||
|
const max = Math.max(1, ...rows.map((r) => r.hits))
|
||||||
|
return (
|
||||||
|
<section className="an-card">
|
||||||
|
<header className="an-card-head">
|
||||||
|
<h2>{title}</h2>
|
||||||
|
{note && <span className="an-note">{note}</span>}
|
||||||
|
</header>
|
||||||
|
{rows.length === 0 ? (
|
||||||
|
<div className="an-empty">{empty}</div>
|
||||||
|
) : (
|
||||||
|
<ul className="an-list">
|
||||||
|
{rows.map((row) => (
|
||||||
|
<li key={row.label}>
|
||||||
|
{/* The bar is the row's own background, so a long label stays
|
||||||
|
readable on top of it instead of being squeezed beside it. */}
|
||||||
|
<span className="an-list-fill" style={{ width: `${(row.hits / max) * 100}%` }} />
|
||||||
|
<span className="an-list-label" title={row.label}>{row.label}</span>
|
||||||
|
<span className="an-list-value">{nf.format(row.hits)}</span>
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
)}
|
||||||
|
</section>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- Access log ---------- */
|
||||||
|
|
||||||
|
const KINDS = [
|
||||||
|
{ key: '', label: 'Everything' },
|
||||||
|
{ key: 'session', label: 'Sessions' },
|
||||||
|
{ key: 'login', label: 'Sign-ins' },
|
||||||
|
{ key: 'register', label: 'Registrations' },
|
||||||
|
{ key: 'login_failed', label: 'Failed' },
|
||||||
|
]
|
||||||
|
|
||||||
|
const KIND_LABEL = {
|
||||||
|
session: 'Session',
|
||||||
|
login: 'Signed in',
|
||||||
|
register: 'Registered',
|
||||||
|
login_failed: 'Failed sign-in',
|
||||||
|
}
|
||||||
|
|
||||||
|
function when(iso) {
|
||||||
|
const date = new Date(iso.endsWith('Z') ? iso : `${iso}Z`)
|
||||||
|
return date.toLocaleString(undefined, { dateStyle: 'medium', timeStyle: 'short' })
|
||||||
|
}
|
||||||
|
|
||||||
|
function AccessLog() {
|
||||||
|
const [kind, setKind] = useState('')
|
||||||
|
const [search, setSearch] = useState('')
|
||||||
|
const [query, setQuery] = useState('')
|
||||||
|
const [page, setPage] = useState(null) // { events, has_more }
|
||||||
|
const [error, setError] = useState(null)
|
||||||
|
const [busy, setBusy] = useState(false)
|
||||||
|
|
||||||
|
// Typing shouldn't fire a request per keystroke against a table scan.
|
||||||
|
useEffect(() => {
|
||||||
|
const timer = setTimeout(() => setQuery(search.trim()), 350)
|
||||||
|
return () => clearTimeout(timer)
|
||||||
|
}, [search])
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
let live = true
|
||||||
|
setPage(null)
|
||||||
|
api.getAccessLog({ kind, q: query }).then(
|
||||||
|
(result) => { if (live) { setPage(result); setError(null) } },
|
||||||
|
(err) => { if (live) setError(err.message) },
|
||||||
|
)
|
||||||
|
return () => { live = false }
|
||||||
|
}, [kind, query])
|
||||||
|
|
||||||
|
const loadMore = useCallback(async () => {
|
||||||
|
if (!page?.events.length || busy) return
|
||||||
|
setBusy(true)
|
||||||
|
try {
|
||||||
|
// Anchored on the oldest row already on screen, so rows arriving while
|
||||||
|
// this is open can't shift the next page.
|
||||||
|
const next = await api.getAccessLog({
|
||||||
|
kind, q: query, beforeId: page.events[page.events.length - 1].id,
|
||||||
|
})
|
||||||
|
setPage({ events: [...page.events, ...next.events], has_more: next.has_more })
|
||||||
|
} catch (err) {
|
||||||
|
setError(err.message)
|
||||||
|
} finally {
|
||||||
|
setBusy(false)
|
||||||
|
}
|
||||||
|
}, [page, kind, query, busy])
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section className="an-card">
|
||||||
|
<header className="an-card-head an-log-head">
|
||||||
|
<div className="an-ranges">
|
||||||
|
{KINDS.map((option) => (
|
||||||
|
<button
|
||||||
|
key={option.key}
|
||||||
|
className={kind === option.key ? 'primary' : ''}
|
||||||
|
onClick={() => setKind(option.key)}
|
||||||
|
>
|
||||||
|
{option.label}
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
className="search-input an-log-search"
|
||||||
|
placeholder="Search email, IP, country…"
|
||||||
|
value={search}
|
||||||
|
onChange={(event) => setSearch(event.target.value)}
|
||||||
|
/>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
{error && <div className="an-empty">Couldn’t load the log: {error}</div>}
|
||||||
|
{!page && !error && <div className="an-empty">Reading…</div>}
|
||||||
|
|
||||||
|
{page && (page.events.length === 0 ? (
|
||||||
|
<div className="an-empty">
|
||||||
|
{query || kind ? 'Nothing matches that.' : 'Nothing logged yet.'}
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<div className="an-table-scroll">
|
||||||
|
<table className="an-table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>When</th><th>Who</th><th>Event</th>
|
||||||
|
<th>IP</th><th>Country</th><th>Device</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{page.events.map((event) => (
|
||||||
|
<tr key={event.id} className={event.kind === 'login_failed' ? 'failed' : undefined}>
|
||||||
|
<td className="an-cell-dim">{when(event.at)}</td>
|
||||||
|
<td>
|
||||||
|
{event.who}
|
||||||
|
{event.is_guest && <span className="an-tag">guest</span>}
|
||||||
|
</td>
|
||||||
|
<td>{KIND_LABEL[event.kind] || event.kind}</td>
|
||||||
|
<td className="an-cell-mono">{event.ip || '—'}</td>
|
||||||
|
<td>{event.country || '—'}</td>
|
||||||
|
{/* The full user-agent is a wall of text; it lives on the
|
||||||
|
hover instead of in a column that would push the rest
|
||||||
|
of the table off screen. */}
|
||||||
|
<td className="an-cell-dim" title={event.user_agent || undefined}>
|
||||||
|
{event.device || '—'}
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
))}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
{page.has_more && (
|
||||||
|
<div className="an-more">
|
||||||
|
<button onClick={loadMore} disabled={busy}>
|
||||||
|
{busy ? 'Loading…' : 'Load older'}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</>
|
||||||
|
))}
|
||||||
|
</section>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- Page ---------- */
|
||||||
|
|
||||||
|
export default function Analytics() {
|
||||||
|
const { me } = useOutletContext() ?? {}
|
||||||
|
const navigate = useNavigate()
|
||||||
|
const [tab, setTab] = useState('overview')
|
||||||
|
const [days, setDays] = useState(30)
|
||||||
|
const [data, setData] = useState(null)
|
||||||
|
const [error, setError] = useState(null)
|
||||||
|
const loaded = useRef(false)
|
||||||
|
|
||||||
|
// me is null until /auth/me resolves; only bounce once we know.
|
||||||
|
useEffect(() => {
|
||||||
|
if (me && !me.analytics) navigate('/', { replace: true })
|
||||||
|
}, [me, navigate])
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (tab !== 'overview') return undefined
|
||||||
|
let live = true
|
||||||
|
api.getAnalytics(days).then(
|
||||||
|
(result) => { if (live) { setData(result); setError(null); loaded.current = true } },
|
||||||
|
(err) => { if (live) setError(err.message) },
|
||||||
|
)
|
||||||
|
return () => { live = false }
|
||||||
|
}, [days, tab])
|
||||||
|
|
||||||
|
const totals = data?.totals ?? {}
|
||||||
|
const visitorSeries = useMemo(() => ([
|
||||||
|
{ key: 'returning', label: 'Returning', color: 'var(--chart-2)' },
|
||||||
|
{ key: 'new', label: 'New', color: 'var(--chart-1)' },
|
||||||
|
]), [])
|
||||||
|
// The server sends new-vs-total; the chart stacks, so it wants the remainder.
|
||||||
|
const visitorDays = useMemo(
|
||||||
|
() => (data?.series ?? []).map((d) => ({ ...d, returning: d.visitors - d.new })),
|
||||||
|
[data],
|
||||||
|
)
|
||||||
|
|
||||||
|
if (me && !me.analytics) return null
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="page an-page">
|
||||||
|
<div className="page-header">
|
||||||
|
<h1>Visitors</h1>
|
||||||
|
{tab === 'overview' && (
|
||||||
|
<div className="an-ranges">
|
||||||
|
{RANGES.map((range) => (
|
||||||
|
<button
|
||||||
|
key={range.days}
|
||||||
|
className={days === range.days ? 'primary' : ''}
|
||||||
|
onClick={() => setDays(range.days)}
|
||||||
|
>
|
||||||
|
{range.label}
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="an-tabs">
|
||||||
|
<button
|
||||||
|
className={tab === 'overview' ? 'active' : ''}
|
||||||
|
onClick={() => setTab('overview')}
|
||||||
|
>
|
||||||
|
Overview
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
className={tab === 'access' ? 'active' : ''}
|
||||||
|
onClick={() => setTab('access')}
|
||||||
|
>
|
||||||
|
Access log
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{tab === 'access' && <AccessLog />}
|
||||||
|
|
||||||
|
{tab === 'overview' && error && (
|
||||||
|
<div className="an-empty">Couldn’t load analytics: {error}</div>
|
||||||
|
)}
|
||||||
|
{tab === 'overview' && !data && !error && <div className="an-empty">Counting…</div>}
|
||||||
|
|
||||||
|
{tab === 'overview' && data && (
|
||||||
|
<>
|
||||||
|
<div className="an-tiles">
|
||||||
|
<StatTile label="Visitors" value={totals.visitors}
|
||||||
|
hint={`${nf.format(totals.new_visitors || 0)} first-time`} />
|
||||||
|
<StatTile label="Visits" value={totals.visits}
|
||||||
|
hint={`${totals.pages_per_visit || 0} pages each`} />
|
||||||
|
<StatTile label="Pageviews" value={totals.pageviews} />
|
||||||
|
<StatTile label="Adventures started" value={totals.adventures} />
|
||||||
|
<StatTile label="Turns played" value={totals.turns}
|
||||||
|
hint={`${totals.turns_per_visit || 0} per visit`} />
|
||||||
|
<StatTile label="Sign-ups" value={totals.signups} />
|
||||||
|
<StatTile label="Demo-key turns" value={totals.demo_turns}
|
||||||
|
hint="billed to the shared key" />
|
||||||
|
<StatTile label="Failed turns" value={totals.turn_errors}
|
||||||
|
hint={`${nf.format(totals.errors || 0)} API errors`} />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="an-grid">
|
||||||
|
<DayChart
|
||||||
|
title="Visitors per day"
|
||||||
|
data={visitorDays}
|
||||||
|
series={visitorSeries}
|
||||||
|
days={days}
|
||||||
|
/>
|
||||||
|
<DayChart
|
||||||
|
title="Turns played per day"
|
||||||
|
data={data.series}
|
||||||
|
series={[{ key: 'turns', label: 'Turns', color: 'var(--chart-3)' }]}
|
||||||
|
days={days}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<Funnel steps={data.funnel} />
|
||||||
|
|
||||||
|
<div className="an-grid">
|
||||||
|
<TopList title="Pages" rows={data.pages} />
|
||||||
|
<TopList
|
||||||
|
title="Where they came from"
|
||||||
|
rows={data.referrers}
|
||||||
|
note="per visit"
|
||||||
|
empty="No referrals yet — every visit was typed or bookmarked."
|
||||||
|
/>
|
||||||
|
<TopList title="Scenarios started" rows={data.scenarios}
|
||||||
|
note="shared scenarios only" empty="No adventures started yet." />
|
||||||
|
<TopList title="Countries" rows={data.countries} />
|
||||||
|
<TopList title="Devices" rows={data.devices} />
|
||||||
|
<TopList title="API errors" rows={data.errors} empty="None — clean run." />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<p className="an-footnote">
|
||||||
|
{fullDate(data.since)} – {fullDate(data.until)}, UTC. Your own visits aren’t
|
||||||
|
counted here. These totals are anonymous — visitors are counted as one-way
|
||||||
|
hashes, and nothing on this tab can be traced back to a player or their
|
||||||
|
stories. The access log tab is the separate, identifying record.
|
||||||
|
</p>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
+69
-1
@@ -3,7 +3,7 @@
|
|||||||
Read this first when picking the project back up. Updated at the end of a working
|
Read this first when picking the project back up. Updated at the end of a working
|
||||||
session; the per-phase plan files hold the detail, this holds the thread.
|
session; the per-phase plan files hold the detail, this holds the thread.
|
||||||
|
|
||||||
**Last updated: 2026-08-20.**
|
**Last updated: 2026-08-21.**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -195,6 +195,74 @@ an automated version; see SP7's entry in `plan/14`.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## What happened on 2026-08-21 — visit analytics
|
||||||
|
|
||||||
|
The hosted demo can now answer whether anyone is using it. `/analytics` is a dashboard —
|
||||||
|
visitors, pages, referrers, countries, devices, which shared scenarios get played, turns and
|
||||||
|
demo-key spend, API and turn errors, and a funnel from *visited* to *played a turn* to
|
||||||
|
*signed up*. **Built, green (497 tests), driven by hand against a synthetic 90-day fixture.
|
||||||
|
Not committed, not deployed.**
|
||||||
|
|
||||||
|
**It is gated on its own allowlist, `AIDND_ANALYTICS_EMAILS`** — not `AIDND_POWER_USERS`.
|
||||||
|
An unmetered tester is not automatically someone who sees the traffic numbers. The route
|
||||||
|
404s and the nav link is absent for everyone else, same treatment as AI Chat. Set the var in
|
||||||
|
the Render dashboard or the page is invisible to everybody, including you.
|
||||||
|
|
||||||
|
**The counters are anonymous; the access log beside them is not, on purpose.** A visitor in
|
||||||
|
`analytics_daily`/`analytics_visitor_days` is `HMAC(secret, "visitor:<user id>")` truncated
|
||||||
|
to 32 chars — one-way, so those two tables cannot be joined back to `users`, and keyed, so no
|
||||||
|
client can compute one. Story content never reaches that module. The operator's own visits
|
||||||
|
are not counted there (multi-user only — excluding them locally would leave the page
|
||||||
|
permanently empty on the machine it is developed on).
|
||||||
|
|
||||||
|
**`accesslog.py` is the identifying half, added the same week.** `access_events` records
|
||||||
|
sessions, sign-ins, registrations and failed attempts with address, email (or `Guest #n`),
|
||||||
|
country and device, read on an "Access log" tab of the same page and behind the same owner
|
||||||
|
gate. It is a separate module and a separate table so the anonymity of the counters stays a
|
||||||
|
property of the code rather than a convention. Three things it leans on: the address comes
|
||||||
|
from `limits.client_ip` — now public, and the only place that decides which hop to trust, so
|
||||||
|
a spoofed `X-Forwarded-For` cannot forge a row; `user_id` carries **no foreign key** and
|
||||||
|
`who`/`is_guest` are snapshots, so a row outlives the guest cleanup that deletes the account;
|
||||||
|
and session rows are thinned to one per day per address, since `/auth/me` runs on every page
|
||||||
|
load. Nothing is purged — that was the deliberate choice. The published docs describe the
|
||||||
|
analytics generally and do not enumerate this.
|
||||||
|
|
||||||
|
**A visit is a write and never a read.** Counts accumulate in a process-local dict and flush
|
||||||
|
every 60s as UPSERTs into `analytics_daily` — a generic `(day, metric, label) -> hits`
|
||||||
|
counter — plus one row per visitor per day in `analytics_visitor_days` for the funnel flags.
|
||||||
|
Every dashboard query is a `GROUP BY` returning tens of rows however much traffic sits
|
||||||
|
behind it. Deliberate, given §2.5: adding a feature that reads rows per request would have
|
||||||
|
undone the egress work.
|
||||||
|
|
||||||
|
**No migration was needed.** Both tables are new, and `bootstrap()` calls `create_all` on
|
||||||
|
existing databases too — the same route `branches` took in Phase 14. Nothing was appended to
|
||||||
|
`MIGRATIONS`, so `LATEST_VERSION` is still 64.
|
||||||
|
|
||||||
|
Three things worth remembering out of building it:
|
||||||
|
|
||||||
|
- **The funnel counts people, not clicks.** A player who starts six adventures is one person
|
||||||
|
who started an adventure. That is the entire reason the per-visitor-day table exists; its
|
||||||
|
flags only ever turn on, and `is_new` is settled by the first write of a visitor's first
|
||||||
|
day and never updated after.
|
||||||
|
- **A failed turn is an HTTP 200 with a bad ending.** The status-code middleware cannot see
|
||||||
|
one, so a demo whose model started refusing every request would look perfectly healthy.
|
||||||
|
`turn_error` is counted in a new `turn_error()` helper that all five SSE error paths in
|
||||||
|
`_generate_turn` now go through.
|
||||||
|
- **The tests run on SQLite; production is Neon.** A flush that raises is caught and logged,
|
||||||
|
so a dialect mistake in the UPSERTs would have been invisible until the dashboard quietly
|
||||||
|
stayed empty. `test_the_upserts_compile_for_postgres` compiles both statements against the
|
||||||
|
Postgres dialect without connecting to one.
|
||||||
|
|
||||||
|
**Still not verified: the narrow-screen layout.** The CSS follows the existing `max-width:
|
||||||
|
720px` block (single-column grids, funnel label above its bar) but `resize_window` is
|
||||||
|
ignored on a maximized Chrome, and the app sends `frame-ancestors 'none'` so it cannot be
|
||||||
|
checked in a sized iframe either. Desktop was driven by hand at 1568px.
|
||||||
|
|
||||||
|
**`docs/guide.html` is now behind `docs/GUIDE.md`,** which gained §3.6. Nothing in the repo
|
||||||
|
regenerates it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## What happened on 2026-08-20 — the branch map
|
## What happened on 2026-08-20 — the branch map
|
||||||
|
|
||||||
The Branches panel gained a **⌗ See the tree** button opening a full-screen map: one
|
The Branches panel gained a **⌗ See the tree** button opening a full-screen map: one
|
||||||
|
|||||||
@@ -51,6 +51,13 @@ services:
|
|||||||
- key: AIDND_POWER_USERS
|
- key: AIDND_POWER_USERS
|
||||||
sync: false
|
sync: false
|
||||||
|
|
||||||
|
# Comma-separated emails allowed to see the Visitors dashboard
|
||||||
|
# (/analytics). Separate from AIDND_POWER_USERS on purpose — an unmetered
|
||||||
|
# tester is not automatically someone who sees the traffic. Leave unset
|
||||||
|
# and the page is invisible to everyone. Set it in the dashboard.
|
||||||
|
- key: AIDND_ANALYTICS_EMAILS
|
||||||
|
sync: false
|
||||||
|
|
||||||
# Guest retention: one account is minted per first-time visitor, so idle
|
# Guest retention: one account is minted per first-time visitor, so idle
|
||||||
# ones are collected (with their adventures) to keep the free-tier
|
# ones are collected (with their adventures) to keep the free-tier
|
||||||
# Postgres from filling with abandoned demo data. Registered accounts are
|
# Postgres from filling with abandoned demo data. Registered accounts are
|
||||||
|
|||||||
Reference in New Issue
Block a user