M7: a first-class imported knowledge library
A campaign can import local .txt and .md files as Canon, Reference or Inspiration, and the class is load-bearing rather than a label: it decides the words a passage is framed with in the prompt, the weight it carries when passages are ranked, and which budget it competes in when the context is tight. This is a separate subsystem, which is the Phase 0B decision (IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification, provenance, content identity, chunking, an index or a lifecycle, and they were not promoted into something that does. Nothing here reads or writes one. The subsystem, in backend/app/knowledge/: classes the three classes, their weights, and the prompt framing chunking deterministic, heading-aware, 60-800 tokens, no overlap fts SQLite FTS5 with porter stemming; scoped and bounded in SQL importer validate, hash, store, chunk, index — in one transaction embeddings local Ollama vectors through the shared provider retrieval query construction, hybrid merge, rerank inject the budgeted cut and the rendered prompt sections Relevance admission is a separate stage from ranking, and that separation is the milestone's most expensive lesson. An independent review found the first implementation deciding relevance with a floor expressed as a share of the best candidate — which the best clears by construction — so a passage was admitted on every turn regardless of the scene. A query about tide tables and container tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden Canon among them. So the pipeline is now: candidate generation -> admission -> ranking -> class weighting -> budget Admission reads raw, candidate-set-independent signals: the cosine the model returned, and how many distinct meaningful query terms a passage contains. Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero is not zero. Normalization decides order among things that matched; it can never decide whether anything matched. Authority is applied after admission, so a class orders what matched and never rescues what did not. Retrieval may therefore return nothing, and on a scene unrelated to the library it does. The other decisions that each replaced an obvious wrong one: - The class multiplies relevance rather than adding to it. An additive bonus satisfies "Canon outranks Reference" and makes "do not include irrelevant Canon" impossible, because a large enough constant wins on its own. - The semantic floor is measured, not guessed: 113 production-path pairs against nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at 0.36-0.56, and 0.58 sits between them. Because it is a property of that model and not of cosine similarity, it is keyed to the model rather than applied to whatever is configured: an embedding model with no measured calibration in this build does not borrow the number. Semantic admission is skipped, the campaign retrieves lexically, and the reason is stated in the knowledge status and in the turn's provenance. Degrading to lexical keeps the library usable; lending the threshold to an unmeasured model is how the admitted-everything defect would return. - One lexical term is not evidence. Two distinct meaningful terms, or one that is neither a standing campaign entity nor a negligible share of the query. The stop list grew from 42 words to 261, all function words — no subject matter, because a stop list that removes subject matter stops finding "The Silver Key". - Lexical retrieval is a production path, not a fallback. It finds the proper nouns and invented terms a setting bible is made of, and the library is fully usable with no embedding model configured. Safety is structural rather than filtered. Imported text reaches the prompt whole, inside a section that says what it is, under a rule stating the authority order in words and refusing every instruction inside it. No endpoint accepts a filesystem path, so H08 has no mechanism to escape from. Nothing renders imported content as HTML, so a script tag is five visible characters and a remote image is never fetched. Import, chunking, indexing, retrieval and a turn open no socket at all; only embeddings do, through the endpoint allowlist the memory bank already uses. Provenance is the rendered text, not a foreign key: deleting a source cannot turn a historical turn's evidence into dangling ids. Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5 virtual table attached to knowledge_chunks as a DDL hook so it is created and dropped with the table it indexes. Migration 92. A pre-M7 database opens unchanged and needs no sources to play. Bundle: the source content and the reader's judgements about it travel; the passages, index rows and vectors are rebuilt on import, so a restored campaign is searchable immediately without a reindex step. One runtime dependency: python-multipart, Starlette's multipart parser. It is what makes the upload surface possible, and the upload surface is why no pathname is ever accepted. The test doubles were the reason the defect shipped, so they were corrected too. The retrieval stub scored unrelated text at 0.06-0.20 where the real model scores it at 0.43-0.44, and its docstring said it had deliberately removed the constant component that "would put a similarity floor under every pair" — which is exactly the property real models have. The stub now has that floor, one test fails if it is ever removed, and another reproduces the superseded rule and asserts it is still fooled by the same fixture. Run against the pre-corrective implementation, the new suite fails 13 of 18. Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of which mocks nothing between itself and Ollama and re-measures the similarity separation on every run. 43/43 checks in a real Firefox, reproduced. Docker build clean. Four other defects found by review or by the browser run were fixed here rather than carried: an unreachable relevance constant that appeared to enforce something and did not; acceptance tests using the wrong fixture files, so G07's trap was never exercised; a bidirectional override surviving into displayed filenames; and, from the implementation pass, the Insights panel showing M5's two state sections as raw keys and the source inspector refetching on every keystroke. M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK REQUIRED. Both blocking findings are closed, and closeout resolved the embedding-model calibration boundary the corrective pass had left as debt. planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective closeout and the closeout verification in sequence, none overwriting another. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 5
Claude Opus 5
parent
a6e9c7a32b
commit
480414efe0
@@ -0,0 +1,381 @@
|
||||
// M7: the imported knowledge library — import, classify, inspect, disable, delete.
|
||||
//
|
||||
// Functional rather than finished. M8 owns the designed knowledge surface; what
|
||||
// this has to do is make every M7 behaviour reachable in a browser without
|
||||
// anyone opening the database, which is the milestone's own standard.
|
||||
//
|
||||
// **Nothing here renders imported text as HTML.** Source text and passage text
|
||||
// both go into a `<pre>` as React children, which React escapes — so a
|
||||
// `<script>` in a file is five visible characters and a `javascript:` URL is
|
||||
// never an href, on first inspection and on every reopen (H06, H07). A remote
|
||||
// Markdown image reference is likewise just characters: no `<img>` is created,
|
||||
// so no request is made (G09). Adding a Markdown renderer would buy appearance
|
||||
// and cost exactly those three properties. `SECURITY-THREAT-MODEL.md` §14 names
|
||||
// sanitized presentation text as the safer default; rendering no markup at all
|
||||
// is one step safer still.
|
||||
|
||||
import { useCallback, useEffect, useRef, useState } from 'react'
|
||||
import { api } from '../../../api'
|
||||
|
||||
const CLASSES = [
|
||||
{ value: 'canon', label: 'Canon', hint: 'Authoritative truth for this campaign.' },
|
||||
{ value: 'reference', label: 'Reference', hint: 'Supporting information; does not establish story truth.' },
|
||||
{ value: 'inspiration', label: 'Inspiration', hint: 'Creative and style influence only.' },
|
||||
]
|
||||
|
||||
const CLASS_LABEL = Object.fromEntries(CLASSES.map((c) => [c.value, c.label]))
|
||||
|
||||
function bytes(n) {
|
||||
if (n < 1024) return `${n} B`
|
||||
if (n < 1024 * 1024) return `${(n / 1024).toFixed(1)} kB`
|
||||
return `${(n / 1024 / 1024).toFixed(2)} MB`
|
||||
}
|
||||
|
||||
function when(iso) {
|
||||
if (!iso) return ''
|
||||
const d = new Date(iso)
|
||||
return Number.isNaN(d.getTime()) ? '' : d.toLocaleString()
|
||||
}
|
||||
|
||||
function SourceDetail({ advId, source, onError }) {
|
||||
const [detail, setDetail] = useState(null)
|
||||
const [chunks, setChunks] = useState(null)
|
||||
const [showText, setShowText] = useState(false)
|
||||
const [showChunks, setShowChunks] = useState(false)
|
||||
// The error reporter reaches this component through a ref rather than
|
||||
// through the effect's dependencies. It arrives as a fresh arrow function on
|
||||
// every render of the Play screen — which re-renders on every keystroke in
|
||||
// the story box — so depending on it would refetch the source, and the
|
||||
// source text, once per character typed. The ref keeps the *current*
|
||||
// reporter without making it a reason to re-run.
|
||||
const report = useRef(onError)
|
||||
report.current = onError
|
||||
|
||||
useEffect(() => {
|
||||
let stale = false // a slow earlier request must not clobber a newer one
|
||||
setDetail(null)
|
||||
api.getKnowledgeSource(advId, source.id)
|
||||
.then((r) => { if (!stale) setDetail(r) })
|
||||
.catch((e) => { if (!stale) report.current(e.message) })
|
||||
return () => { stale = true }
|
||||
}, [advId, source.id])
|
||||
|
||||
const loadChunks = () => {
|
||||
setShowChunks((open) => !open)
|
||||
if (chunks === null) {
|
||||
api.getKnowledgeChunks(advId, source.id)
|
||||
.then(setChunks).catch((e) => report.current(e.message))
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="knowledge-detail">
|
||||
<div className="dim knowledge-facts">
|
||||
<div>File: {source.original_filename}</div>
|
||||
<div>Imported: {when(source.imported_at)}</div>
|
||||
<div>Size: {bytes(source.byte_size)} · {source.media_type}</div>
|
||||
<div>Passages: {source.chunk_count} · embedded {source.embedded_count}</div>
|
||||
<div>Parser v{source.parser_version} · chunking v{source.chunking_version}</div>
|
||||
{/* The content identity, in full rather than shortened: it is here to be
|
||||
compared against another copy, and half a digest compares nothing. */}
|
||||
<div className="knowledge-hash">SHA-256: {source.content_hash}</div>
|
||||
</div>
|
||||
<div className="knowledge-detail-buttons">
|
||||
<button onClick={() => setShowText((open) => !open)}>
|
||||
{showText ? 'Hide source text' : 'Inspect source text'}
|
||||
</button>
|
||||
<button onClick={loadChunks}>
|
||||
{showChunks ? 'Hide passages' : `Inspect passages (${source.chunk_count})`}
|
||||
</button>
|
||||
</div>
|
||||
{showText && (
|
||||
detail
|
||||
? <pre className="knowledge-text" data-testid="knowledge-source-text">{detail.content}</pre>
|
||||
: <div className="empty">Loading…</div>
|
||||
)}
|
||||
{showChunks && (
|
||||
chunks
|
||||
? chunks.map((chunk) => (
|
||||
<div key={chunk.id} className="knowledge-chunk">
|
||||
<div className="dim">
|
||||
passage {chunk.chunk_index + 1}
|
||||
{chunk.heading_path && ` · ${chunk.heading_path}`}
|
||||
{' '}· {chunk.token_count} tok
|
||||
{chunk.embedded ? ` · embedded (${chunk.embedding_model})` : ' · not embedded'}
|
||||
</div>
|
||||
<pre className="knowledge-text">{chunk.text}</pre>
|
||||
</div>
|
||||
))
|
||||
: <div className="empty">Loading…</div>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function SourceRow({ advId, source, onChange, onDelete, onError }) {
|
||||
const [open, setOpen] = useState(false)
|
||||
const [confirming, setConfirming] = useState(false)
|
||||
|
||||
return (
|
||||
<div className={`knowledge-row ${source.enabled ? '' : 'disabled'}`}
|
||||
data-source-id={source.id} data-classification={source.classification}>
|
||||
<div className="knowledge-head">
|
||||
<button className="knowledge-title" onClick={() => setOpen((o) => !o)}>
|
||||
{open ? '▾' : '▸'} {source.title}
|
||||
</button>
|
||||
{/* The filename beside the title, not only inside the detail. A title
|
||||
defaults to the filename without its extension, so two files that
|
||||
differ only by type would otherwise be indistinguishable in the
|
||||
list — and the filename is the name the reader knows the file by. */}
|
||||
{source.original_filename && source.original_filename !== source.title && (
|
||||
<span className="dim knowledge-filename">{source.original_filename}</span>
|
||||
)}
|
||||
<span className={`knowledge-class knowledge-class-${source.classification}`}>
|
||||
{CLASS_LABEL[source.classification]}
|
||||
</span>
|
||||
{!source.enabled && <span className="knowledge-badge">disabled</span>}
|
||||
{source.visibility === 'hidden' && (
|
||||
<span className="knowledge-badge" title="Given to the narrator; the protagonist does not know it">
|
||||
narrator only
|
||||
</span>
|
||||
)}
|
||||
{source.always_include && <span className="knowledge-badge">always included</span>}
|
||||
{source.index_state === 'failed' && (
|
||||
<span className="knowledge-badge failed" title={source.index_detail}>index failed</span>
|
||||
)}
|
||||
{source.embed_state === 'failed' && (
|
||||
<span className="knowledge-badge failed" title={source.embed_detail}>
|
||||
semantic failed — lexical search still works
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="knowledge-controls">
|
||||
<label>
|
||||
Class{' '}
|
||||
<select value={source.classification}
|
||||
onChange={(e) => onChange(source, { classification: e.target.value })}>
|
||||
{CLASSES.map((c) => <option key={c.value} value={c.value}>{c.label}</option>)}
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<input type="checkbox" checked={source.enabled}
|
||||
onChange={(e) => onChange(source, { enabled: e.target.checked })} />
|
||||
{' '}Enabled
|
||||
</label>
|
||||
<label>
|
||||
<input type="checkbox" checked={source.visibility === 'hidden'}
|
||||
onChange={(e) => onChange(source, {
|
||||
visibility: e.target.checked ? 'hidden' : 'normal',
|
||||
})} />
|
||||
{' '}Narrator only
|
||||
</label>
|
||||
{/* Canon's alone. The server enforces it too — this only stops the
|
||||
control offering something that would be silently ignored. */}
|
||||
{source.classification === 'canon' && (
|
||||
<label>
|
||||
<input type="checkbox" checked={source.always_include}
|
||||
onChange={(e) => onChange(source, { always_include: e.target.checked })} />
|
||||
{' '}Always include
|
||||
</label>
|
||||
)}
|
||||
{confirming ? (
|
||||
<span className="knowledge-confirm">
|
||||
Delete “{source.title}”? It stops being used from now on. Turns that
|
||||
already used it keep their record of what they were given.{' '}
|
||||
<button className="danger" onClick={() => onDelete(source)}>Delete</button>
|
||||
<button onClick={() => setConfirming(false)}>Cancel</button>
|
||||
</span>
|
||||
) : (
|
||||
<button className="danger" onClick={() => setConfirming(true)}>Delete</button>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{open && (
|
||||
<SourceDetail advId={advId} source={source} onError={onError} />
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
// A stable no-op for the `onError`-less case. Defined once, at module scope,
|
||||
// because an inline `onError || (() => {})` would hand every row a different
|
||||
// function on every render — which is the same identity problem `SourceDetail`
|
||||
// guards against, arriving from the other side.
|
||||
const NO_OP = () => {}
|
||||
|
||||
function KnowledgePanel({ advId, refreshKey, onError }) {
|
||||
const [sources, setSources] = useState(null)
|
||||
const [status, setStatus] = useState(null)
|
||||
const [file, setFile] = useState(null)
|
||||
const [classification, setClassification] = useState('canon')
|
||||
const [title, setTitle] = useState('')
|
||||
const [hidden, setHidden] = useState(false)
|
||||
const [busy, setBusy] = useState(false)
|
||||
const [notice, setNotice] = useState(null)
|
||||
const [duplicate, setDuplicate] = useState(null)
|
||||
|
||||
const load = useCallback(() => {
|
||||
api.listKnowledge(advId).then(setSources).catch(() => setSources([]))
|
||||
api.getKnowledgeStatus(advId).then(setStatus).catch(() => setStatus(null))
|
||||
}, [advId])
|
||||
|
||||
useEffect(() => { load() }, [load, refreshKey])
|
||||
|
||||
const doImport = async (allowDuplicate = false) => {
|
||||
if (!file) return
|
||||
setBusy(true)
|
||||
setNotice(null)
|
||||
try {
|
||||
await api.importKnowledge(advId, file, {
|
||||
classification,
|
||||
title,
|
||||
visibility: hidden ? 'hidden' : 'normal',
|
||||
allow_duplicate: allowDuplicate,
|
||||
})
|
||||
setFile(null)
|
||||
setTitle('')
|
||||
setDuplicate(null)
|
||||
// The input is uncontrolled (a file input cannot be controlled), so it is
|
||||
// cleared through the DOM. Without this, re-picking the same file after a
|
||||
// failed import fires no change event and the button does nothing.
|
||||
const input = document.getElementById('knowledge-file')
|
||||
if (input) input.value = ''
|
||||
setNotice('Imported.')
|
||||
load()
|
||||
} catch (err) {
|
||||
if (err.conflict) setDuplicate(err.conflict)
|
||||
setNotice(err.message)
|
||||
} finally {
|
||||
setBusy(false)
|
||||
}
|
||||
}
|
||||
|
||||
const change = async (source, data) => {
|
||||
try {
|
||||
const updated = await api.updateKnowledgeSource(advId, source.id, data)
|
||||
setSources((prev) => prev.map((s) => (s.id === source.id ? updated : s)))
|
||||
} catch (err) { onError?.(err.message) }
|
||||
}
|
||||
|
||||
const remove = async (source) => {
|
||||
try {
|
||||
await api.deleteKnowledgeSource(advId, source.id)
|
||||
setSources((prev) => prev.filter((s) => s.id !== source.id))
|
||||
load()
|
||||
} catch (err) { onError?.(err.message) }
|
||||
}
|
||||
|
||||
const reindex = async () => {
|
||||
setBusy(true)
|
||||
setNotice(null)
|
||||
try {
|
||||
const out = await api.reindexKnowledge(advId)
|
||||
setNotice(
|
||||
`Rebuilt ${out.chunks} passage${out.chunks === 1 ? '' : 's'} across `
|
||||
+ `${out.sources} source${out.sources === 1 ? '' : 's'}`
|
||||
+ (out.semantic ? `, embedded ${out.embedded}.` : '. Semantic index not configured.')
|
||||
)
|
||||
load()
|
||||
} catch (err) {
|
||||
setNotice(err.message)
|
||||
} finally { setBusy(false) }
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="knowledge-panel">
|
||||
<p className="dim">
|
||||
Local <code>.txt</code> and <code>.md</code> files this campaign can draw on.
|
||||
Nothing is uploaded anywhere, no link in a file is ever fetched, and text
|
||||
in a source is never treated as an instruction to the application.
|
||||
</p>
|
||||
|
||||
<div className="knowledge-import">
|
||||
<input id="knowledge-file" type="file" accept=".txt,.md,text/plain,text/markdown"
|
||||
onChange={(e) => { setFile(e.target.files?.[0] || null); setDuplicate(null) }} />
|
||||
<label>
|
||||
Class{' '}
|
||||
<select value={classification} onChange={(e) => setClassification(e.target.value)}>
|
||||
{CLASSES.map((c) => <option key={c.value} value={c.value}>{c.label}</option>)}
|
||||
</select>
|
||||
</label>
|
||||
<div className="dim">{CLASSES.find((c) => c.value === classification)?.hint}</div>
|
||||
<input type="text" placeholder="Title (optional)" value={title}
|
||||
onChange={(e) => setTitle(e.target.value)} />
|
||||
<label>
|
||||
<input type="checkbox" checked={hidden} onChange={(e) => setHidden(e.target.checked)} />
|
||||
{' '}Narrator only — the protagonist does not know this
|
||||
</label>
|
||||
<button className="primary" id="knowledge-import" disabled={!file || busy}
|
||||
onClick={() => doImport(false)}>
|
||||
{busy ? 'Working…' : 'Import file'}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{notice && <div className="knowledge-notice" id="knowledge-notice">{notice}</div>}
|
||||
{duplicate && (
|
||||
<div className="knowledge-notice">
|
||||
Already imported as “{duplicate.title}” ({CLASS_LABEL[duplicate.classification]}).{' '}
|
||||
<button onClick={() => doImport(true)}>Import a second copy anyway</button>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{status && (
|
||||
<div className="dim knowledge-status">
|
||||
{status.sources} source{status.sources === 1 ? '' : 's'},
|
||||
{' '}{status.enabled_sources} enabled ·{' '}
|
||||
{status.semantic_enabled
|
||||
? `semantic search on (${status.embedding_model})`
|
||||
+ (status.pending_embeddings
|
||||
? `, ${status.pending_embeddings} passage(s) still to embed`
|
||||
: '')
|
||||
: 'semantic search off — lexical search only, which is a supported setup'}
|
||||
{/* A configured but uncalibrated model is neither "on" nor simply
|
||||
"off": the reader chose a model and it is not being used for
|
||||
retrieval, so the reason has to be visible. */}
|
||||
{status.embedding_model && !status.semantic_calibrated && (
|
||||
<div className="dropped" data-testid="knowledge-uncalibrated">
|
||||
⚠ {status.semantic_note} Calibrated in this build:{' '}
|
||||
{(status.calibrated_models || []).join(', ')}.
|
||||
</div>
|
||||
)}
|
||||
{status.failed_index.length > 0 && (
|
||||
<div className="dropped">
|
||||
⚠ {status.failed_index.length} source(s) failed to index and cannot be retrieved.
|
||||
</div>
|
||||
)}
|
||||
{status.failed_embedding.length > 0 && (
|
||||
<div className="dropped">
|
||||
⚠ {status.failed_embedding.length} source(s) failed to embed. Lexical
|
||||
retrieval still works for them; Rebuild indexes to retry.
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="page-header" style={{ marginTop: 16 }}>
|
||||
<h3 style={{ margin: 0 }}>Sources {sources && `(${sources.length})`}</h3>
|
||||
<span>
|
||||
<button onClick={reindex} disabled={busy} title="Rebuild passages and search indexes from the stored text. Story history is untouched.">
|
||||
Rebuild indexes
|
||||
</button>
|
||||
<button onClick={load} style={{ marginLeft: 6 }}>Refresh</button>
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{!sources && <div className="empty" style={{ padding: '12px 0' }}>Loading…</div>}
|
||||
{sources && sources.length === 0 && (
|
||||
<div className="empty" style={{ padding: '12px 0' }}>
|
||||
Nothing imported yet. Add a setting bible, character notes, a research
|
||||
file or a passage you want the prose to feel like.
|
||||
</div>
|
||||
)}
|
||||
{sources?.map((source) => (
|
||||
<SourceRow key={source.id} advId={advId} source={source}
|
||||
onChange={change} onDelete={remove} onError={onError || NO_OP} />
|
||||
))}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export { KnowledgePanel }
|
||||
Reference in New Issue
Block a user