Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
480414efe082a4bfe0600a19fe23961f6bddd925
interactive-story/PROVENANCE.md
T
JesseMarkowitzandClaude Opus 5 480414efe0 M7: a first-class imported knowledge library 
A campaign can import local .txt and .md files as Canon, Reference or
Inspiration, and the class is load-bearing rather than a label: it decides the
words a passage is framed with in the prompt, the weight it carries when
passages are ranked, and which budget it competes in when the context is tight.

This is a separate subsystem, which is the Phase 0B decision
(IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification,
provenance, content identity, chunking, an index or a lifecycle, and they were
not promoted into something that does. Nothing here reads or writes one.

The subsystem, in backend/app/knowledge/:

  classes      the three classes, their weights, and the prompt framing
  chunking     deterministic, heading-aware, 60-800 tokens, no overlap
  fts          SQLite FTS5 with porter stemming; scoped and bounded in SQL
  importer     validate, hash, store, chunk, index — in one transaction
  embeddings   local Ollama vectors through the shared provider
  retrieval    query construction, hybrid merge, rerank
  inject       the budgeted cut and the rendered prompt sections

Relevance admission is a separate stage from ranking, and that separation is
the milestone's most expensive lesson. An independent review found the first
implementation deciding relevance with a floor expressed as a share of the best
candidate — which the best clears by construction — so a passage was admitted on
every turn regardless of the scene. A query about tide tables and container
tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden
Canon among them.

So the pipeline is now:

  candidate generation -> admission -> ranking -> class weighting -> budget

Admission reads raw, candidate-set-independent signals: the cosine the model
returned, and how many distinct meaningful query terms a passage contains.
Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero
is not zero. Normalization decides order among things that matched; it can never
decide whether anything matched. Authority is applied after admission, so a
class orders what matched and never rescues what did not.

Retrieval may therefore return nothing, and on a scene unrelated to the library
it does.

The other decisions that each replaced an obvious wrong one:

- The class multiplies relevance rather than adding to it. An additive bonus
  satisfies "Canon outranks Reference" and makes "do not include irrelevant
  Canon" impossible, because a large enough constant wins on its own.
- The semantic floor is measured, not guessed: 113 production-path pairs against
  nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at
  0.36-0.56, and 0.58 sits between them. Because it is a property of that model
  and not of cosine similarity, it is keyed to the model rather than applied to
  whatever is configured: an embedding model with no measured calibration in
  this build does not borrow the number. Semantic admission is skipped, the
  campaign retrieves lexically, and the reason is stated in the knowledge status
  and in the turn's provenance. Degrading to lexical keeps the library usable;
  lending the threshold to an unmeasured model is how the admitted-everything
  defect would return.
- One lexical term is not evidence. Two distinct meaningful terms, or one that
  is neither a standing campaign entity nor a negligible share of the query.
  The stop list grew from 42 words to 261, all function words — no subject
  matter, because a stop list that removes subject matter stops finding "The
  Silver Key".
- Lexical retrieval is a production path, not a fallback. It finds the proper
  nouns and invented terms a setting bible is made of, and the library is fully
  usable with no embedding model configured.

Safety is structural rather than filtered. Imported text reaches the prompt
whole, inside a section that says what it is, under a rule stating the authority
order in words and refusing every instruction inside it. No endpoint accepts a
filesystem path, so H08 has no mechanism to escape from. Nothing renders
imported content as HTML, so a script tag is five visible characters and a
remote image is never fetched. Import, chunking, indexing, retrieval and a turn
open no socket at all; only embeddings do, through the endpoint allowlist the
memory bank already uses.

Provenance is the rendered text, not a foreign key: deleting a source cannot
turn a historical turn's evidence into dangling ids.

Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5
virtual table attached to knowledge_chunks as a DDL hook so it is created and
dropped with the table it indexes. Migration 92. A pre-M7 database opens
unchanged and needs no sources to play.

Bundle: the source content and the reader's judgements about it travel; the
passages, index rows and vectors are rebuilt on import, so a restored campaign
is searchable immediately without a reindex step.

One runtime dependency: python-multipart, Starlette's multipart parser. It is
what makes the upload surface possible, and the upload surface is why no
pathname is ever accepted.

The test doubles were the reason the defect shipped, so they were corrected too.
The retrieval stub scored unrelated text at 0.06-0.20 where the real model
scores it at 0.43-0.44, and its docstring said it had deliberately removed the
constant component that "would put a similarity floor under every pair" — which
is exactly the property real models have. The stub now has that floor, one test
fails if it is ever removed, and another reproduces the superseded rule and
asserts it is still fooled by the same fixture. Run against the pre-corrective
implementation, the new suite fails 13 of 18.

Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of
which mocks nothing between itself and Ollama and re-measures the similarity
separation on every run. 43/43 checks in a real Firefox, reproduced.
Docker build clean.

Four other defects found by review or by the browser run were fixed here rather
than carried: an unreachable relevance constant that appeared to enforce
something and did not; acceptance tests using the wrong fixture files, so G07's
trap was never exercised; a bidirectional override surviving into displayed
filenames; and, from the implementation pass, the Insights panel showing M5's
two state sections as raw keys and the source inspector refetching on every
keystroke.

M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK
REQUIRED. Both blocking findings are closed, and closeout resolved the
embedding-model calibration boundary the corrective pass had left as debt.
planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective
closeout and the closeout verification in sequence, none overwriting another.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b
2026-09-06 15:40:13 -04:00

173 lines
8.6 KiB
Markdown
Raw Blame History

# Provenance
This repository is the Adventure Storyteller production fork. Its application
code comes from **AI-DnD**, and its planning package (`planning/`) is original
to this project.
## Upstream
| | |
| --- | --- |
| Project | AI-DnD |
| Repository | <https://github.com/parththakkar106/AI-DnD> |
| Commit | `d72f7c1bda0f34fccd84afb7a25c34eb01c901de` |
| Subject | Stop paying twice for a block a retry can still throw away |
| Author date | Mon 31 Aug 2026 16:14:24 +0000 |
| Position | tip of `upstream/main` on 1 Sep 2026, when the fork was taken |
| License | MIT, © 2026 Parth Thakkar |
The commit is the one pinned by `planning/DECISIONS/009-ai-dnd-production-base.md`
after Phase 0B. It was not substituted for a newer upstream commit.
## How the fork is wired
Upstream history is *in* this repository rather than copied out of it. The
import is a merge of the pinned commit with `--allow-unrelated-histories`, so:
- `git log d72f7c1bda0f34fccd84afb7a25c34eb01c901de` shows the real upstream
history, not a squashed snapshot;
- upstream code paths are unchanged (`backend/`, `frontend/`, …), so a later
upstream commit can still be fetched and cherry-picked against matching
files. Upstream's own documentation trees, `plan/` and `docs/`, were removed
on 2026-09-03: they described the hosted, scripted, multi-user product this
fork is not. They remain in this repository's history and in upstream;
- the planning package that predates the fork keeps its own history on the
other parent of the merge.
To re-verify from a fresh clone:
```bash
git remote add upstream https://github.com/parththakkar106/AI-DnD.git
git fetch --no-tags upstream
git cat-file -t d72f7c1bda0f34fccd84afb7a25c34eb01c901de # -> commit
git merge-base --is-ancestor d72f7c1bda0f34fccd84afb7a25c34eb01c901de HEAD && echo "in this history"
```
## License
Upstream is MIT. `LICENSE` is upstream's file, unmodified, and the copyright
notice stays with it. The MIT terms require that the notice travel with the
code and with substantial portions of it; keep `LICENSE` in place in any
redistribution of this fork, including a packaged build.
Work done in this repository after the fork is a derivative of that MIT-licensed
code.
## Vendored third-party assets
Both were added by Milestone M1 to remove a runtime Internet dependency. Each
is redistributable and each has a regeneration path in the tree, so neither is
an opaque binary nobody can rebuild.
### `backend/app/context/vendor/cl100k_base.tiktoken`
The BPE merge table for OpenAI's `cl100k_base` tokenizer, used for context
budgeting only — no model of OpenAI's is ever called.
- Source: `https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken`
- SHA-256: `223921b76ee99bde995b7ff738513eef100fb51d18c93597a113bcffe865b2a7`,
which is the digest `tiktoken` itself pins for that URL, and which
`backend/app/context/encoding.py` re-checks every time it builds the encoding.
- Published by OpenAI for use with `tiktoken` (MIT).
### `frontend/public/fonts/*.woff2`
Cinzel, Crimson Pro and Inter, Latin and Latin Extended subsets, as variable
fonts. All three are licensed under the SIL Open Font License 1.1; the license
text ships beside them as `OFL-cinzel.txt`, `OFL-crimsonpro.txt` and
`OFL-inter.txt`, which is what the OFL requires of a redistribution.
Regenerate with `python3 frontend/tools/vendor_fonts.py`, which also rewrites
`frontend/src/styles/fonts.css`.
## What this fork changed in Milestone M7
M7 is additive. It builds the imported knowledge library the specification asks
for as a **separate first-class subsystem**, which is the Phase 0B decision
recorded in `planning/IMPORTED-KNOWLEDGE-DESIGN.md` §73: AI-DnD's Story Cards do
not carry the classification, provenance, chunking, index, lifecycle or
inspection an imported-knowledge system needs, and they were not promoted into
one. Story Cards are untouched and still work exactly as upstream left them;
nothing in the new subsystem reads or writes one.
- `backend/app/knowledge/` (new) — the whole subsystem: the three classes and
their prompt framing, a deterministic heading-aware chunker, the SQLite FTS5
lexical index, local Ollama embeddings, hybrid retrieval and reranking, and the
budgeted injection into the prompt.
- `backend/app/routers/adventures/knowledge.py` (new) — import, list, inspect,
reclassify, enable/disable, delete, reindex and status. The import surface is a
multipart upload; **no endpoint anywhere accepts a filesystem path**.
- `backend/app/models.py` — three new tables (`knowledge_sources`,
`knowledge_chunks`, `knowledge_embeddings`) and the DDL hook that carries the
FTS5 virtual table with the table it indexes.
- `backend/app/migrations.py` — version 92.
- `backend/app/context/builder.py` — the knowledge sections, their budget, and
the provenance record in the context snapshot.
- `backend/app/bundle.py` — the export carries source content and the reader's
judgements about it; passages, index rows and vectors are rebuilt on import.
- `backend/app/derived.py`, `backend/app/memorybank.py` — a `knowledge` kind of
derived work, and the post-turn pass that catches up vectors an import could
not build.
- `frontend/src/pages/Play/panels/KnowledgePanel.jsx` (new),
`frontend/src/styles/knowledge.css` (new), and additions to the Insights panel
— a utilitarian browser surface for the whole lifecycle. Imported text is
displayed as inert text and is never rendered as HTML.
- **One new runtime dependency**, `python-multipart` — Starlette's multipart
parser, pure Python, Apache-2.0, no dependencies of its own. It is what makes
the upload surface possible and is the reason no path is ever accepted.
No network path was added. Embeddings go through the same
`OpenAICompatibleProvider` the memory bank uses, so the endpoint allowlist, the
request-time re-check and the OS/private-CA trust union all apply unchanged
(ADR 011). Lexical indexing is local SQLite and touches no socket at all.
## What this fork changed in Milestone M2
M2 is subtractive. It reduced the inherited application to the intended
single-user, local-first trust boundary. **Nothing was added that upstream did
not have, except the endpoint policy and the tests that hold these removals in
place.**
Removed in full: campaign scripting and the QuickJS sandbox; multi-user
accounts, guest sessions, login, registration and the shared demo key; the
visitor analytics tables, dashboard and beacon; the access log; per-IP and
per-user rate limiting and quotas; Render deployment config; Postgres/Neon
support; cloud inference providers and the API-key field; session-cookie
signing and API-key encryption at rest.
Added: `backend/app/endpoints.py`, which decides what an inference endpoint may
be, and a configurable model timeout.
Three database tables (`scripts`, `adventure_scripts`, `analytics_daily`,
`analytics_visitor_days`, `access_log`) and four columns (`adventures.script_state`,
`settings.api_key`, `users.demo_turns_used`, `users.demo_turns_date`) are left
in place, unmapped or inert, so that an existing M1 campaign database opens
unchanged. They are not product functionality and nothing reads or writes them.
## What this fork changed in Milestone M1
Nothing was removed from upstream. The changes are the offline/locality
hardening M1 called for; see `planning/archive/milestone-reports/M1-BASELINE-REPORT.md` for the
evidence.
- `backend/app/context/encoding.py` (new) and `backend/app/context/builder.py` —
build `cl100k_base` from the vendored table instead of downloading it on
first use.
- `frontend/index.html`, `frontend/src/index.css`,
`frontend/src/styles/fonts.css` (new), `frontend/public/fonts/` (new),
`frontend/tools/vendor_fonts.py` (new) — self-hosted fonts in place of the
Google Fonts link.
- `backend/app/main.py` — CSP narrowed to same-origin, with the two Google
hosts dropped and `object-src` / `base-uri` / `form-action` added; `woff2`
registered so the self-hosted fonts are served with their real media type.
- `backend/app/tlstrust.py` (new), `backend/app/providers/openai_compatible.py`,
`backend/app/routers/settings.py` — outbound HTTPS verifies against the
machine's own CA store as well as certifi's, so a trusted-LAN Ollama with a
locally-issued certificate works. Verification is not relaxed.
- `start.sh`, `start.ps1`, `docker-compose.yml` — the storyteller listener is
explicitly loopback-bound.
- `backend/requirements.lock` (new) — the exact tested dependency closure.
- `backend/tests/test_offline_assets.py` and `backend/tests/test_tls_trust.py`
(new) — regression tests for the above.
- `DEVELOPMENT.md` (new) — environment setup and Ollama configuration.
Reference in New Issue View Git Blame Copy Permalink