JesseMarkowitzandClaude Opus 5
480414efe0
A campaign can import local .txt and .md files as Canon, Reference or Inspiration, and the class is load-bearing rather than a label: it decides the words a passage is framed with in the prompt, the weight it carries when passages are ranked, and which budget it competes in when the context is tight. This is a separate subsystem, which is the Phase 0B decision (IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification, provenance, content identity, chunking, an index or a lifecycle, and they were not promoted into something that does. Nothing here reads or writes one. The subsystem, in backend/app/knowledge/: classes the three classes, their weights, and the prompt framing chunking deterministic, heading-aware, 60-800 tokens, no overlap fts SQLite FTS5 with porter stemming; scoped and bounded in SQL importer validate, hash, store, chunk, index — in one transaction embeddings local Ollama vectors through the shared provider retrieval query construction, hybrid merge, rerank inject the budgeted cut and the rendered prompt sections Relevance admission is a separate stage from ranking, and that separation is the milestone's most expensive lesson. An independent review found the first implementation deciding relevance with a floor expressed as a share of the best candidate — which the best clears by construction — so a passage was admitted on every turn regardless of the scene. A query about tide tables and container tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden Canon among them. So the pipeline is now: candidate generation -> admission -> ranking -> class weighting -> budget Admission reads raw, candidate-set-independent signals: the cosine the model returned, and how many distinct meaningful query terms a passage contains. Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero is not zero. Normalization decides order among things that matched; it can never decide whether anything matched. Authority is applied after admission, so a class orders what matched and never rescues what did not. Retrieval may therefore return nothing, and on a scene unrelated to the library it does. The other decisions that each replaced an obvious wrong one: - The class multiplies relevance rather than adding to it. An additive bonus satisfies "Canon outranks Reference" and makes "do not include irrelevant Canon" impossible, because a large enough constant wins on its own. - The semantic floor is measured, not guessed: 113 production-path pairs against nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at 0.36-0.56, and 0.58 sits between them. Because it is a property of that model and not of cosine similarity, it is keyed to the model rather than applied to whatever is configured: an embedding model with no measured calibration in this build does not borrow the number. Semantic admission is skipped, the campaign retrieves lexically, and the reason is stated in the knowledge status and in the turn's provenance. Degrading to lexical keeps the library usable; lending the threshold to an unmeasured model is how the admitted-everything defect would return. - One lexical term is not evidence. Two distinct meaningful terms, or one that is neither a standing campaign entity nor a negligible share of the query. The stop list grew from 42 words to 261, all function words — no subject matter, because a stop list that removes subject matter stops finding "The Silver Key". - Lexical retrieval is a production path, not a fallback. It finds the proper nouns and invented terms a setting bible is made of, and the library is fully usable with no embedding model configured. Safety is structural rather than filtered. Imported text reaches the prompt whole, inside a section that says what it is, under a rule stating the authority order in words and refusing every instruction inside it. No endpoint accepts a filesystem path, so H08 has no mechanism to escape from. Nothing renders imported content as HTML, so a script tag is five visible characters and a remote image is never fetched. Import, chunking, indexing, retrieval and a turn open no socket at all; only embeddings do, through the endpoint allowlist the memory bank already uses. Provenance is the rendered text, not a foreign key: deleting a source cannot turn a historical turn's evidence into dangling ids. Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5 virtual table attached to knowledge_chunks as a DDL hook so it is created and dropped with the table it indexes. Migration 92. A pre-M7 database opens unchanged and needs no sources to play. Bundle: the source content and the reader's judgements about it travel; the passages, index rows and vectors are rebuilt on import, so a restored campaign is searchable immediately without a reindex step. One runtime dependency: python-multipart, Starlette's multipart parser. It is what makes the upload surface possible, and the upload surface is why no pathname is ever accepted. The test doubles were the reason the defect shipped, so they were corrected too. The retrieval stub scored unrelated text at 0.06-0.20 where the real model scores it at 0.43-0.44, and its docstring said it had deliberately removed the constant component that "would put a similarity floor under every pair" — which is exactly the property real models have. The stub now has that floor, one test fails if it is ever removed, and another reproduces the superseded rule and asserts it is still fooled by the same fixture. Run against the pre-corrective implementation, the new suite fails 13 of 18. Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of which mocks nothing between itself and Ollama and re-measures the similarity separation on every run. 43/43 checks in a real Firefox, reproduced. Docker build clean. Four other defects found by review or by the browser run were fixed here rather than carried: an unreachable relevance constant that appeared to enforce something and did not; acceptance tests using the wrong fixture files, so G07's trap was never exercised; a bidirectional override surviving into displayed filenames; and, from the implementation pass, the Insights panel showing M5's two state sections as raw keys and the source inspector refetching on every keystroke. M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK REQUIRED. Both blocking findings are closed, and closeout resolved the embedding-model calibration boundary the corrective pass had left as debt. planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective closeout and the closeout verification in sequence, none overwriting another. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b
8.6 KiB
Provenance
This repository is the Adventure Storyteller production fork. Its application
code comes from AI-DnD, and its planning package (planning/) is original
to this project.
Upstream
| Project | AI-DnD |
| Repository | https://github.com/parththakkar106/AI-DnD |
| Commit | d72f7c1bda0f34fccd84afb7a25c34eb01c901de |
| Subject | Stop paying twice for a block a retry can still throw away |
| Author date | Mon 31 Aug 2026 16:14:24 +0000 |
| Position | tip of upstream/main on 1 Sep 2026, when the fork was taken |
| License | MIT, © 2026 Parth Thakkar |
The commit is the one pinned by planning/DECISIONS/009-ai-dnd-production-base.md
after Phase 0B. It was not substituted for a newer upstream commit.
How the fork is wired
Upstream history is in this repository rather than copied out of it. The
import is a merge of the pinned commit with --allow-unrelated-histories, so:
git log d72f7c1bda0f34fccd84afb7a25c34eb01c901deshows the real upstream history, not a squashed snapshot;- upstream code paths are unchanged (
backend/,frontend/, …), so a later upstream commit can still be fetched and cherry-picked against matching files. Upstream's own documentation trees,plan/anddocs/, were removed on 2026-09-03: they described the hosted, scripted, multi-user product this fork is not. They remain in this repository's history and in upstream; - the planning package that predates the fork keeps its own history on the other parent of the merge.
To re-verify from a fresh clone:
git remote add upstream https://github.com/parththakkar106/AI-DnD.git
git fetch --no-tags upstream
git cat-file -t d72f7c1bda0f34fccd84afb7a25c34eb01c901de # -> commit
git merge-base --is-ancestor d72f7c1bda0f34fccd84afb7a25c34eb01c901de HEAD && echo "in this history"
License
Upstream is MIT. LICENSE is upstream's file, unmodified, and the copyright
notice stays with it. The MIT terms require that the notice travel with the
code and with substantial portions of it; keep LICENSE in place in any
redistribution of this fork, including a packaged build.
Work done in this repository after the fork is a derivative of that MIT-licensed code.
Vendored third-party assets
Both were added by Milestone M1 to remove a runtime Internet dependency. Each is redistributable and each has a regeneration path in the tree, so neither is an opaque binary nobody can rebuild.
backend/app/context/vendor/cl100k_base.tiktoken
The BPE merge table for OpenAI's cl100k_base tokenizer, used for context
budgeting only — no model of OpenAI's is ever called.
- Source:
https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken - SHA-256:
223921b76ee99bde995b7ff738513eef100fb51d18c93597a113bcffe865b2a7, which is the digesttiktokenitself pins for that URL, and whichbackend/app/context/encoding.pyre-checks every time it builds the encoding. - Published by OpenAI for use with
tiktoken(MIT).
frontend/public/fonts/*.woff2
Cinzel, Crimson Pro and Inter, Latin and Latin Extended subsets, as variable
fonts. All three are licensed under the SIL Open Font License 1.1; the license
text ships beside them as OFL-cinzel.txt, OFL-crimsonpro.txt and
OFL-inter.txt, which is what the OFL requires of a redistribution.
Regenerate with python3 frontend/tools/vendor_fonts.py, which also rewrites
frontend/src/styles/fonts.css.
What this fork changed in Milestone M7
M7 is additive. It builds the imported knowledge library the specification asks
for as a separate first-class subsystem, which is the Phase 0B decision
recorded in planning/IMPORTED-KNOWLEDGE-DESIGN.md §73: AI-DnD's Story Cards do
not carry the classification, provenance, chunking, index, lifecycle or
inspection an imported-knowledge system needs, and they were not promoted into
one. Story Cards are untouched and still work exactly as upstream left them;
nothing in the new subsystem reads or writes one.
backend/app/knowledge/(new) — the whole subsystem: the three classes and their prompt framing, a deterministic heading-aware chunker, the SQLite FTS5 lexical index, local Ollama embeddings, hybrid retrieval and reranking, and the budgeted injection into the prompt.backend/app/routers/adventures/knowledge.py(new) — import, list, inspect, reclassify, enable/disable, delete, reindex and status. The import surface is a multipart upload; no endpoint anywhere accepts a filesystem path.backend/app/models.py— three new tables (knowledge_sources,knowledge_chunks,knowledge_embeddings) and the DDL hook that carries the FTS5 virtual table with the table it indexes.backend/app/migrations.py— version 92.backend/app/context/builder.py— the knowledge sections, their budget, and the provenance record in the context snapshot.backend/app/bundle.py— the export carries source content and the reader's judgements about it; passages, index rows and vectors are rebuilt on import.backend/app/derived.py,backend/app/memorybank.py— aknowledgekind of derived work, and the post-turn pass that catches up vectors an import could not build.frontend/src/pages/Play/panels/KnowledgePanel.jsx(new),frontend/src/styles/knowledge.css(new), and additions to the Insights panel — a utilitarian browser surface for the whole lifecycle. Imported text is displayed as inert text and is never rendered as HTML.- One new runtime dependency,
python-multipart— Starlette's multipart parser, pure Python, Apache-2.0, no dependencies of its own. It is what makes the upload surface possible and is the reason no path is ever accepted.
No network path was added. Embeddings go through the same
OpenAICompatibleProvider the memory bank uses, so the endpoint allowlist, the
request-time re-check and the OS/private-CA trust union all apply unchanged
(ADR 011). Lexical indexing is local SQLite and touches no socket at all.
What this fork changed in Milestone M2
M2 is subtractive. It reduced the inherited application to the intended single-user, local-first trust boundary. Nothing was added that upstream did not have, except the endpoint policy and the tests that hold these removals in place.
Removed in full: campaign scripting and the QuickJS sandbox; multi-user accounts, guest sessions, login, registration and the shared demo key; the visitor analytics tables, dashboard and beacon; the access log; per-IP and per-user rate limiting and quotas; Render deployment config; Postgres/Neon support; cloud inference providers and the API-key field; session-cookie signing and API-key encryption at rest.
Added: backend/app/endpoints.py, which decides what an inference endpoint may
be, and a configurable model timeout.
Three database tables (scripts, adventure_scripts, analytics_daily,
analytics_visitor_days, access_log) and four columns (adventures.script_state,
settings.api_key, users.demo_turns_used, users.demo_turns_date) are left
in place, unmapped or inert, so that an existing M1 campaign database opens
unchanged. They are not product functionality and nothing reads or writes them.
What this fork changed in Milestone M1
Nothing was removed from upstream. The changes are the offline/locality
hardening M1 called for; see planning/archive/milestone-reports/M1-BASELINE-REPORT.md for the
evidence.
backend/app/context/encoding.py(new) andbackend/app/context/builder.py— buildcl100k_basefrom the vendored table instead of downloading it on first use.frontend/index.html,frontend/src/index.css,frontend/src/styles/fonts.css(new),frontend/public/fonts/(new),frontend/tools/vendor_fonts.py(new) — self-hosted fonts in place of the Google Fonts link.backend/app/main.py— CSP narrowed to same-origin, with the two Google hosts dropped andobject-src/base-uri/form-actionadded;woff2registered so the self-hosted fonts are served with their real media type.backend/app/tlstrust.py(new),backend/app/providers/openai_compatible.py,backend/app/routers/settings.py— outbound HTTPS verifies against the machine's own CA store as well as certifi's, so a trusted-LAN Ollama with a locally-issued certificate works. Verification is not relaxed.start.sh,start.ps1,docker-compose.yml— the storyteller listener is explicitly loopback-bound.backend/requirements.lock(new) — the exact tested dependency closure.backend/tests/test_offline_assets.pyandbackend/tests/test_tls_trust.py(new) — regression tests for the above.DEVELOPMENT.md(new) — environment setup and Ollama configuration.