Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
480414efe082a4bfe0600a19fe23961f6bddd925
interactive-story/planning/DATA-MODEL.md
T
JesseMarkowitzandClaude Opus 5 480414efe0 M7: a first-class imported knowledge library 
A campaign can import local .txt and .md files as Canon, Reference or
Inspiration, and the class is load-bearing rather than a label: it decides the
words a passage is framed with in the prompt, the weight it carries when
passages are ranked, and which budget it competes in when the context is tight.

This is a separate subsystem, which is the Phase 0B decision
(IMPORTED-KNOWLEDGE-DESIGN.md §73). Story Cards do not carry classification,
provenance, content identity, chunking, an index or a lifecycle, and they were
not promoted into something that does. Nothing here reads or writes one.

The subsystem, in backend/app/knowledge/:

  classes      the three classes, their weights, and the prompt framing
  chunking     deterministic, heading-aware, 60-800 tokens, no overlap
  fts          SQLite FTS5 with porter stemming; scoped and bounded in SQL
  importer     validate, hash, store, chunk, index — in one transaction
  embeddings   local Ollama vectors through the shared provider
  retrieval    query construction, hybrid merge, rerank
  inject       the budgeted cut and the rendered prompt sections

Relevance admission is a separate stage from ranking, and that separation is
the milestone's most expensive lesson. An independent review found the first
implementation deciding relevance with a floor expressed as a share of the best
candidate — which the best clears by construction — so a passage was admitted on
every turn regardless of the scene. A query about tide tables and container
tonnage retrieved all five sources of a fantasy campaign, narrator-only hidden
Canon among them.

So the pipeline is now:

  candidate generation -> admission -> ranking -> class weighting -> budget

Admission reads raw, candidate-set-independent signals: the cosine the model
returned, and how many distinct meaningful query terms a passage contains.
Ranking reads normalized ones, because bm25 has no fixed range and cosine's zero
is not zero. Normalization decides order among things that matched; it can never
decide whether anything matched. Authority is applied after admission, so a
class orders what matched and never rescues what did not.

Retrieval may therefore return nothing, and on a scene unrelated to the library
it does.

The other decisions that each replaced an obvious wrong one:

- The class multiplies relevance rather than adding to it. An additive bonus
  satisfies "Canon outranks Reference" and makes "do not include irrelevant
  Canon" impossible, because a large enough constant wins on its own.
- The semantic floor is measured, not guessed: 113 production-path pairs against
  nomic-embed-text put targeted matches at 0.55-0.85 and off-topic pairs at
  0.36-0.56, and 0.58 sits between them. Because it is a property of that model
  and not of cosine similarity, it is keyed to the model rather than applied to
  whatever is configured: an embedding model with no measured calibration in
  this build does not borrow the number. Semantic admission is skipped, the
  campaign retrieves lexically, and the reason is stated in the knowledge status
  and in the turn's provenance. Degrading to lexical keeps the library usable;
  lending the threshold to an unmeasured model is how the admitted-everything
  defect would return.
- One lexical term is not evidence. Two distinct meaningful terms, or one that
  is neither a standing campaign entity nor a negligible share of the query.
  The stop list grew from 42 words to 261, all function words — no subject
  matter, because a stop list that removes subject matter stops finding "The
  Silver Key".
- Lexical retrieval is a production path, not a fallback. It finds the proper
  nouns and invented terms a setting bible is made of, and the library is fully
  usable with no embedding model configured.

Safety is structural rather than filtered. Imported text reaches the prompt
whole, inside a section that says what it is, under a rule stating the authority
order in words and refusing every instruction inside it. No endpoint accepts a
filesystem path, so H08 has no mechanism to escape from. Nothing renders
imported content as HTML, so a script tag is five visible characters and a
remote image is never fetched. Import, chunking, indexing, retrieval and a turn
open no socket at all; only embeddings do, through the endpoint allowlist the
memory bank already uses.

Provenance is the rendered text, not a foreign key: deleting a source cannot
turn a historical turn's evidence into dangling ids.

Schema: knowledge_sources, knowledge_chunks, knowledge_embeddings, and an FTS5
virtual table attached to knowledge_chunks as a DDL hook so it is created and
dropped with the table it indexes. Migration 92. A pre-M7 database opens
unchanged and needs no sources to play.

Bundle: the source content and the reader's judgements about it travel; the
passages, index rows and vectors are rebuilt on import, so a restored campaign
is searchable immediately without a reindex step.

One runtime dependency: python-multipart, Starlette's multipart parser. It is
what makes the upload surface possible, and the upload surface is why no
pathname is ever accepted.

The test doubles were the reason the defect shipped, so they were corrected too.
The retrieval stub scored unrelated text at 0.06-0.20 where the real model
scores it at 0.43-0.44, and its docstring said it had deliberately removed the
constant component that "would put a similarity floor under every pair" — which
is exactly the property real models have. The stub now has that floor, one test
fails if it is ever removed, and another reproduces the superseded rule and
asserts it is still fooled by the same fixture. Run against the pre-corrective
implementation, the new suite fails 13 of 18.

Tests: 939 passed, 14 skipped (836/7 at M6). 110 new across seven files, one of
which mocks nothing between itself and Ollama and re-measures the similarity
separation on every run. 43/43 checks in a real Firefox, reproduced.
Docker build clean.

Four other defects found by review or by the browser run were fixed here rather
than carried: an unreachable relevance constant that appeared to enforce
something and did not; acceptance tests using the wrong fixture files, so G07's
trap was never exercised; a bidirectional override surviving into displayed
filenames; and, from the implementation pass, the Insights panel showing M5's
two state sections as raw keys and the source inspector refetching on every
keystroke.

M7 was independently reviewed, which returned PASS WITH CORRECTIVE WORK
REQUIRED. Both blocking findings are closed, and closeout resolved the
embedding-model calibration boundary the corrective pass had left as debt.
planning/reports/M7-IMPLEMENTATION-REPORT.md carries the review, the corrective
closeout and the closeout verification in sequence, none overwriting another.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HdaXiFbscatQaLS7dJk6b
2026-09-06 15:40:13 -04:00

997 lines
31 KiB
Markdown
Raw Blame History

# Adventure Storyteller — Data Model
**Status:** v1.0 conceptual model aligned to Phase 0B decisions
**Purpose:** Define the persistent information the application must represent, independent of the final fork or database implementation.
## 1. Design Goals
The data model must support persistent interactive stories, complete authoritative history, non-destructive branching, checkpoints and rollback, genre-independent narrative state, long-term memory, imported local knowledge, prompt/context provenance, future image/video/audio/TTS/STT generation, export/restore, and local-only operation.
The same core schema should work for fantasy, science fiction, mystery, horror, historical fiction, westerns, and other narrative genres.
## 2. Core Principles
### 2.1 Application-owned authority
The database is the source of truth. The model may propose narration and state changes, but those proposals become authoritative only after application validation.
### 2.2 Non-destructive history
Accepted turns are historical records. Going backward should move the active story head or create a branch, not silently delete accepted history.
### 2.3 Historical reconstruction
The system must answer both:
- What is true now?
- What was true at a particular earlier turn on a particular branch?
### 2.4 Genre neutrality
Avoid fantasy- or science-fiction-specific core fields. Use generic concepts such as characters, locations, organizations, items, vehicles, facts, relationships, conditions, scenes, and story threads.
## 3. Conceptual Entity Map
```text
Campaign
├── Branches
│ └── Turns
│ ├── Prompt Snapshot
│ ├── State Version
│ ├── Scene Snapshot
│ └── Retrieval Records
├── Checkpoints
├── Narrative Entities
├── Facts
├── Relationships
├── Story Threads
├── Memories
├── Summaries
├── Knowledge Sources
│ └── Knowledge Chunks
└── Media
├── Media Jobs
└── Media Assets
```
The exact SQL schema may differ from this conceptual model.
## 4. Campaign
A campaign is the top-level story container.
```yaml
campaign:
id: uuid
title: string
created_at: timestamp
updated_at: timestamp
active_branch_id: uuid
active_head_turn_id: optional uuid
status: active | archived
story_profile:
genre: string
subgenre: optional string
tone: optional string
style: optional string
point_of_view: optional string
tense: optional string
```
The active head is **stored on the campaign, not derived** from its newest turn.
This is the concept M3 implemented; the current implementation carries it as a
branch reference plus a depth on that branch rather than as a turn id, which is
an equivalent coordinate and is what the export format records. What matters
conceptually is that the position is a decision the campaign remembers: two
campaigns holding identical turns can be being read at different places, and
nothing about the turns themselves can tell them apart.
Campaigns also store durable narrator rules and model configuration.
Potential model roles:
- narrator,
- state extractor,
- summarizer,
- embedding model.
For v1, all model roles should use local Ollama-compatible models.
## 5. Branch
A branch represents one valid continuation of story history.
```yaml
branch:
id: uuid
campaign_id: uuid
name: string
created_at: timestamp
created_from_branch_id: optional uuid
fork_turn_id: optional uuid
tip_turn_id: optional uuid
disposition: active | retained | disposable
status: active | archived
```
Rules:
- branches may share ancestral turns,
- shared history should not be duplicated unnecessarily,
- creating a branch must not modify the source branch,
- the campaign active head may sit behind the retained branch tip after Undo,
- Redo moves the active head forward while the prior continuation remains selected,
- a new write below the retained tip creates a new continuation and leaves the old future retained/disposable.
`disposition` above is conceptual. As implemented in M3 it is stored as **the
fact that produced it** rather than as a word: a branch records the depth a
divergent write left it at, and when. No value means active; a value means the
story past that depth is retained history no active head is reading. The
shallowest departure wins if a branch is left more than once.
Two properties of that representation are deliberate and worth carrying in this
document:
- **Nothing reads it to decide behavior.** Whether Redo is available, what the
transcript shows, and which continuation a write belongs to are all decided by
the lineage. A stale or hand-edited disposition therefore cannot make the story
wrong; it can only mislead a cleanup or recovery feature about what is
abandoned.
- **It survives export and import.** Every row of an abandoned line is exported
either way, so the disposition is the only thing distinguishing it from an
active one in a restored campaign.
Detailed behavior will be defined separately in `STORY-BRANCH-SEMANTICS.md`;
the architecture is recorded in ADR 012.
## 6. Turn
A turn is one accepted story interaction:
```text
user input -> narrator response -> accepted state transition
```
```yaml
turn:
id: uuid
campaign_id: uuid
branch_id: uuid
parent_turn_id: optional uuid
created_at: timestamp
input:
mode: action | dialogue | direction | continue
text: string
output:
narration: string
generation:
provider: ollama
model: string
generation_settings: object
prompt_snapshot_id: uuid
state_before_id: uuid
state_after_id: uuid
scene_snapshot_id: optional uuid
status: pending | accepted | failed | superseded
```
The root turn has no parent.
An accepted historical turn should not silently disappear if another continuation is chosen.
## 7. Alternate Take
The system may need to distinguish:
- a different narrator response to the same user action,
- a genuinely different story branch.
Conceptual form:
```yaml
take:
id: uuid
turn_request_id: uuid
output_text: string
model_metadata: object
selected: boolean
```
Physical representation remains implementation-specific. The selected AI-DnD base already models alternate takes within its lineage machinery; production should retain that approach if it satisfies the required Retry/select/retention semantics without forcing a separate table.
## 8. Checkpoint
```yaml
checkpoint:
id: uuid
campaign_id: uuid
turn_id: uuid
name: string
notes: optional string
created_at: timestamp
```
A checkpoint is a named pointer to a recoverable story position. It should normally remain tied to the turn where it was created. Restoring it moves the campaign active head; it does not delete later retained history. A new branch is created on the first divergent write after restore, not merely because the checkpoint was opened.
As implemented in M4, the pointer is a **coordinate rather than a turn id**:
`(branch_id, depth)`, which is the same pair §4 records as the campaign's active
head and which `head.node_at` resolves. This is the equivalence §4 already draws
between a turn reference and a branch-plus-depth, applied to the same position
from the other end, and it is not a shortcut — it is the more correct pointer of
the two for this data model:
- **A coordinate follows a retry; a row id does not.** One coordinate holds
every attempt at a turn and exactly one of them is live (§7). A Save Point
names the turn, so it must land on whichever take the story currently tells.
Pinning the row would leave the pointer on a superseded attempt the reader
cannot see.
- **The user-facing term is Save Point**; `checkpoint` remains the internal name
(`BROWSER-UX-SPEC.md` §23).
The row carries the name, an optional note, the coordinate, and its timestamps.
It carries **no** copy of the transcript, the state, the prompt, a memory, a
summary, or a branch's contents. Everything a restore produces comes from the
retained history the coordinate points into.
The retry case is what settles the coordinate-versus-turn-id question, and M4
closeout measured it rather than arguing it. A Save Point named a turn whose
live row was id 17; retrying that turn made id 17 dead and id 18 live at the
same coordinate; the Save Point resolved to id 18 and restored correctly. A row
id would have pinned a take the story no longer tells. **A Save Point names a
story position, not a particular take of it.**
Three further properties of the implemented model, recorded so they are decided
rather than incidental:
- **Names are not unique**, and nothing requires them to be. No product
requirement asks for uniqueness, and two names for one moment is a reasonable
thing for a player to want.
- **Several Save Points may name the same position.** Same reason.
- **Ordinary list presentation is newest-created first.** Story order is not
something the list can honestly claim: depths on lines that have parted
company are not comparable, so ordering by depth would draw a sequence that no
reading of the story passes through. When each was made is a fact about all of
them.
None of this is genre-specific. A coordinate is a position in a story; what the
state at that position *contains* is M5's question, and changing it does not
change what a Save Point is.
Two consequences worth recording here:
- **Restore reuses the campaign's one head-movement mechanism.** It resolves the
coordinate and moves the head; nothing is reconstructed and nothing is
deleted. The branch half of the head moves only when the coordinate is not on
the path being read, which is what makes a Save Point on a departed line
restorable at all — and what keeps a Save Point in a shared prefix from
dragging the reader off the line they chose. See `TECHNICAL-DESIGN.md` §8.8.
- **A checkpoint is durable against everything but its own explicit deletion.**
No pass removes one for going stale, sitting behind the head, or naming a line
the story left (`STORY-BRANCH-SEMANTICS.md` §19). Deleting a *branch* does not
remove one either: the deletion is refused while a checkpoint names any
position in the subtree, and the user deletes the checkpoint first
(§19.1). Deleting the whole campaign removes them, which is what deleting a
campaign means.
## 9. Narrative Entity
An entity is a persistent thing or concept in the fictional world.
```yaml
entity:
id: uuid
campaign_id: uuid
type: string
name: string
aliases: [string]
description: string
status: active | inactive | destroyed | dead | unknown
created_turn_id: optional uuid
metadata: object
```
Recommended built-in categories:
- character,
- location,
- organization,
- item,
- vehicle,
- creature,
- structure,
- concept,
- other.
These are descriptive categories, not separate game systems.
## 10. Character
```yaml
character:
entity_id: uuid
role: optional string
description: string
current_location_id: optional uuid
condition: [string]
personality_notes: [string]
goals: [string]
secrets: [string]
```
Optional visual continuity fields:
```yaml
visual_profile:
apparent_age: optional string
build: optional string
hair: optional string
eyes: optional string
clothing: optional string
distinctive_features: [string]
continuity_notes: [string]
```
## 11. Location
```yaml
location:
entity_id: uuid
description: string
parent_location_id: optional uuid
current_status: optional string
visual_profile:
architecture: optional string
environment: optional string
lighting: optional string
signature_features: [string]
continuity_notes: [string]
```
## 12. Organization
Organizations may represent factions, governments, companies, guilds, military units, religious organizations, or informal groups.
```yaml
organization:
entity_id: uuid
purpose: optional string
current_status: optional string
```
Specific characteristics should usually live in facts and relationships.
## 13. Items and Vehicles
Items and vehicles remain generic entities.
Examples:
- silver key,
- longsword,
- encrypted data crystal,
- survey ship,
- horse-drawn carriage.
Possession and location should normally be represented as relationships or facts:
```text
Aldric --possesses--> Silver Key
Persephone --docked_at--> Ceres Station
```
## 14. Fact
Facts represent assertions about the story world.
```yaml
fact:
id: uuid
campaign_id: uuid
subject_entity_id: optional uuid
predicate: string
object_entity_id: optional uuid
value: optional scalar_or_object
authority: string
source_type: string
source_id: optional uuid
created_turn_id: optional uuid
invalidated_turn_id: optional uuid
status: active | superseded | disputed
```
Examples:
- Mara knows Aldric has the silver key.
- The Persephone cannot travel faster than light.
- Edrin disappeared three weeks before the campaign began.
- The eastern bridge collapsed during Turn 47.
Minimum authority categories:
- campaign_canon,
- accepted_story,
- current_state,
- imported_canon,
- reference,
- heuristic,
- inspiration.
Exact context behavior will be defined in `CONTEXT-AND-MEMORY.md`.
## 15. Relationship
```yaml
relationship:
id: uuid
campaign_id: uuid
source_entity_id: uuid
target_entity_id: uuid
type: string
status: string
description: optional string
created_turn_id: optional uuid
ended_turn_id: optional uuid
```
Examples:
- Aldric -> trusts -> Mara
- Mara -> member_of -> Circle of Ash
- Silver Key -> belongs_to -> Aldric
- Persephone -> docked_at -> Ceres Station
Relationship types should remain extensible.
## 16. Story Thread
Story threads track unresolved or resolved narrative business.
```yaml
story_thread:
id: uuid
campaign_id: uuid
title: string
description: string
status: open | dormant | resolved | abandoned
importance: optional number_or_label
opened_turn_id: optional uuid
resolved_turn_id: optional uuid
```
These are narrative continuity tools, not RPG quests.
## 16A. Derived Context Tables (M6, as implemented)
Two tables and one column carry M6's derived context. All three are derived
data: deleting them changes no accepted history, no authoritative state and no
head position.
```text
summaries
id, adventure_id
text
branch_id, depth the coordinate of the last node covered
source_start, source_end the stretch of story summarized
trigger "interval" (generated) or "manual" (reader-written)
model_name, created_at
derived_status one row per (adventure, kind)
kind "memory" | "summary" | "embedding"
status "ok" (did work) | "idle" (nothing pending) | "failed"
detail, failures
last_attempt_at, last_success_at
memories.authority "accepted_story" | "heuristic"
```
`summaries` mirrors the shape `memories` already had, deliberately: both are
derived rows anchored to a coordinate on a path, and both are filtered by the
same lineage clause — and both are also the *input* to the next round of derived
work, which is why summary generation reads `summaries.current` rather than any
campaign-global field.
`adventures.story_summary` is retained as the reader's edit surface and the
export field, mirroring whichever summary is eligible. It carries no lineage of
its own and nothing authoritative reads it.
## 17. State Version
The system must reconstruct authoritative state at any retained turn.
```yaml
state_version:
id: uuid
campaign_id: uuid
turn_id: optional uuid
parent_state_version_id: optional uuid
created_at: timestamp
state_hash: optional string
```
Possible implementation models:
### A. Full snapshots
Simple restore, but duplicates data.
### B. Event sourcing
Excellent auditability, but requires replay.
### C. Hybrid
Validated events plus periodic/current snapshots.
**Selected for v1: Hybrid.** Store validated authoritative events plus efficient state snapshots/cache for normal reads and restore.
## 18. State Change Event
If the hybrid/event model is selected:
```yaml
state_event:
id: uuid
campaign_id: uuid
turn_id: uuid
event_type: string
payload: object
sequence: integer
```
Potential event types:
- entity_created,
- set_entity_status,
- set_entity_attribute,
- fact_added,
- fact_invalidated,
- relationship_added,
- relationship_ended,
- thread_opened,
- thread_resolved,
- current_location_set,
- possession_set,
- scene_set.
Event semantics must be explicit and typed. Prefer unambiguous absolute assignments for mutable values. If incremental operations are ever needed, encode the operation explicitly (for example `increment_value`) rather than relying on one numeric field whose interpretation is implicit.
Events must be schema-validated, semantically checked where deterministic rules exist, and accepted by the application before commit.
## 19. State Proposal
The model's extracted state proposal must be distinct from accepted state.
```yaml
state_proposal:
id: uuid
turn_id: uuid
model: string
raw_output: string
parsed_payload: object
validation_status: accepted | partially_accepted | rejected | repair_required
```
The model must never write directly to authoritative state tables. The production protocol must not depend on AI-DnD-style ambiguous relative deltas; see ADR 010.
## 20. Scene Snapshot
A scene snapshot captures the immediate narrative situation.
```yaml
scene:
id: uuid
campaign_id: uuid
branch_id: uuid
source_turn_start_id: optional uuid
source_turn_end_id: optional uuid
location_id: optional uuid
time_description: optional string
mood: optional string
environment: optional string
participants: [uuid]
significant_objects: [uuid]
current_actions: [string]
visual_notes: [string]
continuity_notes: [string]
```
Scene snapshots support:
- current context,
- narrative continuity,
- future image generation,
- future multi-turn video/storyboard generation.
## 21. Summary
Summaries compress history but never replace authoritative history.
```yaml
summary:
id: uuid
campaign_id: uuid
branch_id: uuid
type: campaign | arc | rolling | turn_range
source_start_turn_id: uuid
source_end_turn_id: uuid
text: string
created_at: timestamp
model: optional string
```
Summaries are derived data. Branch changes must invalidate or lineage-filter incompatible summaries.
## 22. Memory
```yaml
memory:
id: uuid
campaign_id: uuid
branch_scope: optional uuid
source_turn_id: optional uuid
type: string
text: string
authority: string
importance: optional number
embedding_ref: optional string
created_at: timestamp
```
Potential types:
- event,
- character,
- relationship,
- location,
- promise,
- discovery,
- conflict,
- heuristic.
A retrieved memory does not automatically become canon.
## 23. Knowledge Source
A knowledge source is a user-imported local file or manually authored campaign document.
```yaml
knowledge_source:
id: uuid
campaign_id: optional uuid
title: string
source_type: file | manual
classification: canon | reference | inspiration
original_filename: optional string
content_hash: string
enabled: boolean
imported_at: timestamp
metadata: object
```
Initial supported file types should be `.txt` and `.md`.
## 24. Knowledge Chunk
```yaml
knowledge_chunk:
id: uuid
source_id: uuid
sequence: integer
text: string
heading_path: optional string
token_count: optional integer
embedding_ref: optional string
metadata: object
```
Requirements:
- preserve provenance,
- preserve chunk order,
- permit re-indexing,
- never execute imported content.
## 24A. Imported Knowledge Tables (M7, as implemented)
Three tables, and the boundary between them is the boundary between what the
reader gave the campaign and what the machine derived from it.
```text
knowledge_sources the file, and the reader's judgements about it
id, adventure_id campaign-scoped; no branch coordinate, deliberately
title, original_filename the filename is metadata and is never a path
classification "canon" | "reference" | "inspiration"
enabled out of retrieval without being deleted
visibility "normal" | "hidden" (narrator-only)
always_include Canon only: in force whatever the scene is
content the accepted text, as decoded
content_hash SHA-256 of the normalized text; the duplicate test
byte_size, media_type
parser_version what produced the passages now on disk
chunking_version
index_state, index_detail "pending" | "ready" | "failed" — the lexical half
embed_state, embed_detail "idle" | "pending" | "ok" | "failed" — the semantic half
notes, imported_at, updated_at
knowledge_chunks derived: a deterministic function of the content
id, source_id, adventure_id
chunk_index, heading_path
text, token_count, content_hash
knowledge_embeddings derived: rebuildable, and its own table so that
id, chunk_id, adventure_id "rebuild the semantic index" is one DELETE
vector packed float32, as `memories.embedding_blob` is
model, dimensions what makes a stale vector detectable
parser_version, chunking_version, created_at
knowledge_fts a SQLite FTS5 virtual table over heading + text,
keyed by chunk id. Not describable in SQLAlchemy
metadata, so it is attached to `knowledge_chunks`
as a DDL hook and travels with it.
```
**Only the first two columns of a source are not derivable**: its content and
its classification. Everything else about a source is metadata describing one of
those two, and everything in the other two tables is rebuilt from the content by
a deterministic chunker. That is what lets the export carry the source alone
(§29) and what makes a reindex safe.
There is deliberately **no branch coordinate** anywhere here. An imported file is
campaign source material and does not become a different file because the story
forked (`CONTEXT-AND-MEMORY.md` §39). The rule that abandoned story content must
not reach the prompt is met at the *query* instead: the retrieval query is built
from the head-capped lineage and the authoritative state at the position being
read, never from the uncapped action table. A future knowledge record *derived*
from story history would need a coordinate; M7 introduces no such record.
## 25. Retrieval Record
Every turn should record which memories or knowledge chunks were supplied to the narrator.
```yaml
retrieval_record:
id: uuid
turn_id: uuid
source_kind: memory | knowledge | summary | fact
source_id: uuid
retrieval_method: lexical | semantic | hybrid | forced
score: optional number
rank: integer
```
This lets the prompt inspector answer:
> Why did the narrator know this?
### As implemented (M6 for memories, M7 for imported knowledge)
There is no `retrieval_record` table. The record lives in the turn's own context
snapshot, which every turn already stores, and it carries the **rendered text**
alongside the identifiers:
```text
context_snapshot.knowledge
used[] source_id, title, filename, classification, visibility,
chunk_id, chunk_index, heading_path, always_include,
mode ("lexical" | "semantic" | "hybrid" | "always"),
lexical, semantic, cosine, score, tokens,
text, rendered, prompt_tokens
dropped[] the same, plus why there was no budget for it
suppressed[] the same, plus the passage it repeated
terms, considered, floor, budget, spent
semantic_used, semantic_note, scan_truncated
```
Carrying the text rather than a foreign key is the whole point. A separate table
of ids would turn every historical turn's evidence into dangling references the
moment a source were deleted, and §49-50 of `IMPORTED-KNOWLEDGE-DESIGN.md`
require the opposite: a turn must go on being able to say what it was given.
## 26. Prompt Snapshot
```yaml
prompt_snapshot:
id: uuid
turn_id: uuid
created_at: timestamp
system_instructions: text
campaign_context: text_or_structured
state_context: text_or_structured
summary_context: text_or_structured
memory_context: text_or_structured
knowledge_context: text_or_structured
recent_history: text_or_structured
user_input: text
token_accounting: object
```
The physical representation may be compressed or normalized. Reproducibility and inspection are the requirements.
## 27. Media Job
Not required for v1 behavior, but the data model should not prevent it.
```yaml
media_job:
id: uuid
campaign_id: uuid
scene_id: optional uuid
source_turn_start_id: optional uuid
source_turn_end_id: optional uuid
type: image | video | audio | tts | stt
provider: string
model: string
status: queued | running | completed | failed
request_payload: object
created_at: timestamp
completed_at: optional timestamp
```
## 28. Media Asset
```yaml
media_asset:
id: uuid
campaign_id: uuid
media_job_id: optional uuid
scene_id: optional uuid
type: image | video | audio | tts | stt
file_path: string
metadata: object
created_at: timestamp
```
Potential metadata includes prompt, seed, model, workflow, dimensions, duration, character references, and source turn range.
## 29. Export Package
A campaign export should be capable of preserving:
- campaign configuration,
- branches,
- turn graph,
- checkpoints,
- state/events,
- entities,
- facts,
- relationships,
- story threads,
- summaries,
- memories,
- scene snapshots,
- knowledge-source metadata,
- knowledge chunks/source files if selected,
- prompt provenance if selected,
- media metadata,
- media files if selected.
The physical container format remains an implementation choice, but the export must preserve the exact active branch **and active head position**, even when the head is behind a retained tip after Undo. A ZIP containing a database plus manifest remains a strong candidate.
As implemented in M3, the export carries the active branch, the active head
position on it, and each branch's disposition, alongside the whole retained turn
graph. **M4 added the checkpoints**, by the same rule: a position someone chose
to name cannot be recomputed from the turns, because nothing about a turn records
that it was bookmarked. The head and the checkpoints stay independent on import —
a campaign opens where its head says, never at a checkpoint merely because one is
in the file. The governing rule for this package is that an export carries what was
*chosen* and recomputes what is *derived* — and the active head moved from the
second category to the first, because once Undo stops deleting, two campaigns
with identical turns can be being read at different positions and no import can
tell which. An export written before the field existed is opened at its retained
tip, which is the position such a file recorded.
**M7 added the imported knowledge library**, by the same rule and no other. The
bundle carries each source's content, classification, enabled state, visibility,
always-include flag, title, filename, notes, import timestamp and content hash —
everything the reader chose, and one derived value whose only purpose is to be
checked against what arrived. It carries no passages, no FTS rows and no
vectors: those are a deterministic function of the content, and the import
rebuilds the passages and the lexical index before it returns, so an imported
campaign is searchable immediately with no reindex step. Vectors rebuild
separately against whatever embedding model *this* machine has, which is the
right answer and the reason exporting them would have been the wrong one.
A source that fails to rebuild is recorded as failed rather than refusing the
import: by that point the story, its tree, its head and its Save Points are
already written, and the index is the cheap half. A malformed *knowledge section*
— an unknown classification, missing content — does refuse the import, because a
campaign whose imported Canon quietly did not arrive is a campaign whose narrator
has stopped being told the rules, with nothing to notice.
A bundle written before M7 has no knowledge section and imports with an empty
library, which is what such a campaign had.
## 30. Deletion vs Archival
The system must distinguish:
- archive,
- detach/disable,
- permanent delete.
Retry, Undo, Restore, and branch switching must not silently perform permanent deletion.
## 31. Authoritative vs Derived Data
### Authoritative
- accepted turns,
- branch lineage,
- campaign configuration,
- accepted facts,
- accepted relationships,
- accepted state events,
- checkpoints.
### Derived
- summaries,
- embeddings,
- semantic indexes,
- lexical indexes,
- some automatically produced scene descriptions.
Derived data should be rebuildable where practical.
## 32. Provenance
Important information should answer:
> Where did this come from?
Potential provenance:
- campaign setup,
- manual user edit,
- accepted narrator turn,
- state extraction,
- imported canon,
- imported reference,
- imported inspiration,
- derived inference.
## 33. Phase 0B Decisions Applied
Phase 0B resolved the foundational data-model questions:
- AI-DnD story lineage/alternate-take machinery is the production starting point.
- The active head is distinct from retained tip history.
- Undo/Redo use head movement; destructive deletion is not part of ordinary history operations.
- Named checkpoints are durable pointers to recoverable head positions.
- State uses a hybrid event + snapshot/cache model.
- State proposals use explicit typed operations with unambiguous value semantics.
- Memories/summaries must be lineage-filtered or lineage-anchored.
- Imported knowledge requires separate source/chunk/index tables rather than overloading Story Cards.
- Scene/media records remain optional derived extensions and must carry source lineage.
- Export/import must preserve active head position as well as the retained history graph.
## 34. Acceptance Criteria
The final v1 data model must support all of these without destructive hacks:
- close/restart/resume exact story,
- Undo/Redo without deleting accepted turns,
- active head behind retained tip,
- branch from an earlier turn while retaining the original future,
- mark abandoned futures/takes retained/disposable,
- create and restore named checkpoints,
- know current characters/locations/relationships/story threads,
- reconstruct earlier authoritative state,
- validate typed state proposals before committing events,
- retrieve old events outside the active context window,
- identify which imported passages informed a turn,
- reconstruct what was sent to Ollama,
- export/import an undone campaign without silently redoing it,
- run fantasy and science-fiction campaigns without schema changes,
- attach future image/video/audio/TTS/STT metadata to scenes or turn ranges without making media authoritative.
## 35. Selected Conceptual Model
```text
Retained Turn/Take Lineage
|
+--> Active branch + movable active head
|
+--> Validated typed state events
| |
| +--> state snapshot/cache
|
+--> lineage-safe summaries/memories
|
+--> prompt/retrieval provenance
|
+--> scene snapshot
|
+--> future optional media records
Separate campaign knowledge subsystem
+--> sources
+--> chunks
+--> FTS/local embeddings
+--> authority/provenance
```
This combines AI-DnD's retained story lineage and snapshot infrastructure with ai-adventure-style explicit event/commit discipline while preserving the project's own specification as the authority.
Reference in New Issue View Git Blame Copy Permalink