interactive-story/planning/DATA-MODEL.md

# Adventure Storyteller — Data Model

**Status:** v1.0 conceptual model aligned to Phase 0B decisions  
**Purpose:** Define the persistent information the application must represent, independent of the final fork or database implementation.

## 1. Design Goals

The data model must support persistent interactive stories, complete authoritative history, non-destructive branching, checkpoints and rollback, genre-independent narrative state, long-term memory, imported local knowledge, prompt/context provenance, future image/video/audio/TTS/STT generation, export/restore, and local-only operation.

The same core schema should work for fantasy, science fiction, mystery, horror, historical fiction, westerns, and other narrative genres.

## 2. Core Principles

### 2.1 Application-owned authority
The database is the source of truth. The model may propose narration and state changes, but those proposals become authoritative only after application validation.

### 2.2 Non-destructive history
Accepted turns are historical records. Going backward should move the active story head or create a branch, not silently delete accepted history.

### 2.3 Historical reconstruction
The system must answer both:
- What is true now?
- What was true at a particular earlier turn on a particular branch?

### 2.4 Genre neutrality
Avoid fantasy- or science-fiction-specific core fields. Use generic concepts such as characters, locations, organizations, items, vehicles, facts, relationships, conditions, scenes, and story threads.

## 3. Conceptual Entity Map

```text
Campaign
 ├── Branches
 │    └── Turns
 │         ├── Prompt Snapshot
 │         ├── State Version
 │         ├── Scene Snapshot
 │         └── Retrieval Records
 ├── Checkpoints
 ├── Narrative Entities
 ├── Facts
 ├── Relationships
 ├── Story Threads
 ├── Memories
 ├── Summaries
 ├── Knowledge Sources
 │    └── Knowledge Chunks
 └── Media
      ├── Media Jobs
      └── Media Assets
```

The exact SQL schema may differ from this conceptual model.

## 4. Campaign

A campaign is the top-level story container.

```yaml
campaign:
  id: uuid
  title: string
  created_at: timestamp
  updated_at: timestamp
  active_branch_id: uuid
  active_head_turn_id: optional uuid
  status: active | archived

story_profile:
  genre: string
  subgenre: optional string
  tone: optional string
  style: optional string
  point_of_view: optional string
  tense: optional string
```

The active head is **stored on the campaign, not derived** from its newest turn.
This is the concept M3 implemented; the current implementation carries it as a
branch reference plus a depth on that branch rather than as a turn id, which is
an equivalent coordinate and is what the export format records. What matters
conceptually is that the position is a decision the campaign remembers: two
campaigns holding identical turns can be being read at different places, and
nothing about the turns themselves can tell them apart.

Campaigns also store durable narrator rules and model configuration.

Potential model roles:
- narrator,
- state extractor,
- summarizer,
- embedding model.

For v1, all model roles should use local Ollama-compatible models.

## 5. Branch

A branch represents one valid continuation of story history.

```yaml
branch:
  id: uuid
  campaign_id: uuid
  name: string
  created_at: timestamp
  created_from_branch_id: optional uuid
  fork_turn_id: optional uuid
  tip_turn_id: optional uuid
  disposition: active | retained | disposable
  status: active | archived
```

Rules:
- branches may share ancestral turns,
- shared history should not be duplicated unnecessarily,
- creating a branch must not modify the source branch,
- the campaign active head may sit behind the retained branch tip after Undo,
- Redo moves the active head forward while the prior continuation remains selected,
- a new write below the retained tip creates a new continuation and leaves the old future retained/disposable.

`disposition` above is conceptual. As implemented in M3 it is stored as **the
fact that produced it** rather than as a word: a branch records the depth a
divergent write left it at, and when. No value means active; a value means the
story past that depth is retained history no active head is reading. The
shallowest departure wins if a branch is left more than once.

Two properties of that representation are deliberate and worth carrying in this
document:

- **Nothing reads it to decide behavior.** Whether Redo is available, what the
  transcript shows, and which continuation a write belongs to are all decided by
  the lineage. A stale or hand-edited disposition therefore cannot make the story
  wrong; it can only mislead a cleanup or recovery feature about what is
  abandoned.
- **It survives export and import.** Every row of an abandoned line is exported
  either way, so the disposition is the only thing distinguishing it from an
  active one in a restored campaign.

Detailed behavior will be defined separately in `STORY-BRANCH-SEMANTICS.md`;
the architecture is recorded in ADR 012.

## 6. Turn

A turn is one accepted story interaction:

```text
user input -> narrator response -> accepted state transition
```

```yaml
turn:
  id: uuid
  campaign_id: uuid
  branch_id: uuid
  parent_turn_id: optional uuid
  created_at: timestamp

input:
  mode: action | dialogue | direction | continue
  text: string

output:
  narration: string

generation:
  provider: ollama
  model: string
  generation_settings: object
  prompt_snapshot_id: uuid

state_before_id: uuid
state_after_id: uuid
scene_snapshot_id: optional uuid
status: pending | accepted | failed | superseded
```

The root turn has no parent.

An accepted historical turn should not silently disappear if another continuation is chosen.

## 7. Alternate Take

The system may need to distinguish:
- a different narrator response to the same user action,
- a genuinely different story branch.

Conceptual form:

```yaml
take:
  id: uuid
  turn_request_id: uuid
  output_text: string
  model_metadata: object
  selected: boolean
```

Physical representation remains implementation-specific. The selected AI-DnD base already models alternate takes within its lineage machinery; production should retain that approach if it satisfies the required Retry/select/retention semantics without forcing a separate table.

## 8. Checkpoint

```yaml
checkpoint:
  id: uuid
  campaign_id: uuid
  turn_id: uuid
  name: string
  notes: optional string
  created_at: timestamp
```

A checkpoint is a named pointer to a recoverable story position. It should normally remain tied to the turn where it was created. Restoring it moves the campaign active head; it does not delete later retained history. A new branch is created on the first divergent write after restore, not merely because the checkpoint was opened.

As implemented in M4, the pointer is a **coordinate rather than a turn id**:
`(branch_id, depth)`, which is the same pair §4 records as the campaign's active
head and which `head.node_at` resolves. This is the equivalence §4 already draws
between a turn reference and a branch-plus-depth, applied to the same position
from the other end, and it is not a shortcut — it is the more correct pointer of
the two for this data model:

- **A coordinate follows a retry; a row id does not.** One coordinate holds
  every attempt at a turn and exactly one of them is live (§7). A Save Point
  names the turn, so it must land on whichever take the story currently tells.
  Pinning the row would leave the pointer on a superseded attempt the reader
  cannot see.
- **The user-facing term is Save Point**; `checkpoint` remains the internal name
  (`BROWSER-UX-SPEC.md` §23).

The row carries the name, an optional note, the coordinate, and its timestamps.
It carries **no** copy of the transcript, the state, the prompt, a memory, a
summary, or a branch's contents. Everything a restore produces comes from the
retained history the coordinate points into.

The retry case is what settles the coordinate-versus-turn-id question, and M4
closeout measured it rather than arguing it. A Save Point named a turn whose
live row was id 17; retrying that turn made id 17 dead and id 18 live at the
same coordinate; the Save Point resolved to id 18 and restored correctly. A row
id would have pinned a take the story no longer tells. **A Save Point names a
story position, not a particular take of it.**

Three further properties of the implemented model, recorded so they are decided
rather than incidental:

- **Names are not unique**, and nothing requires them to be. No product
  requirement asks for uniqueness, and two names for one moment is a reasonable
  thing for a player to want.
- **Several Save Points may name the same position.** Same reason.
- **Ordinary list presentation is newest-created first.** Story order is not
  something the list can honestly claim: depths on lines that have parted
  company are not comparable, so ordering by depth would draw a sequence that no
  reading of the story passes through. When each was made is a fact about all of
  them.

None of this is genre-specific. A coordinate is a position in a story; what the
state at that position *contains* is M5's question, and changing it does not
change what a Save Point is.

Two consequences worth recording here:

- **Restore reuses the campaign's one head-movement mechanism.** It resolves the
  coordinate and moves the head; nothing is reconstructed and nothing is
  deleted. The branch half of the head moves only when the coordinate is not on
  the path being read, which is what makes a Save Point on a departed line
  restorable at all — and what keeps a Save Point in a shared prefix from
  dragging the reader off the line they chose. See `TECHNICAL-DESIGN.md` §8.8.
- **A checkpoint is durable against everything but its own explicit deletion.**
  No pass removes one for going stale, sitting behind the head, or naming a line
  the story left (`STORY-BRANCH-SEMANTICS.md` §19). Deleting a *branch* does not
  remove one either: the deletion is refused while a checkpoint names any
  position in the subtree, and the user deletes the checkpoint first
  (§19.1). Deleting the whole campaign removes them, which is what deleting a
  campaign means.

## 9. Narrative Entity

An entity is a persistent thing or concept in the fictional world.

```yaml
entity:
  id: uuid
  campaign_id: uuid
  type: string
  name: string
  aliases: [string]
  description: string
  status: active | inactive | destroyed | dead | unknown
  created_turn_id: optional uuid
  metadata: object
```

Recommended built-in categories:
- character,
- location,
- organization,
- item,
- vehicle,
- creature,
- structure,
- concept,
- other.

These are descriptive categories, not separate game systems.

## 10. Character

```yaml
character:
  entity_id: uuid
  role: optional string
  description: string
  current_location_id: optional uuid
  condition: [string]
  personality_notes: [string]
  goals: [string]
  secrets: [string]
```

Optional visual continuity fields:

```yaml
visual_profile:
  apparent_age: optional string
  build: optional string
  hair: optional string
  eyes: optional string
  clothing: optional string
  distinctive_features: [string]
  continuity_notes: [string]
```

## 11. Location

```yaml
location:
  entity_id: uuid
  description: string
  parent_location_id: optional uuid
  current_status: optional string

visual_profile:
  architecture: optional string
  environment: optional string
  lighting: optional string
  signature_features: [string]
  continuity_notes: [string]
```

## 12. Organization

Organizations may represent factions, governments, companies, guilds, military units, religious organizations, or informal groups.

```yaml
organization:
  entity_id: uuid
  purpose: optional string
  current_status: optional string
```

Specific characteristics should usually live in facts and relationships.

## 13. Items and Vehicles

Items and vehicles remain generic entities.

Examples:
- silver key,
- longsword,
- encrypted data crystal,
- survey ship,
- horse-drawn carriage.

Possession and location should normally be represented as relationships or facts:

```text
Aldric --possesses--> Silver Key
Persephone --docked_at--> Ceres Station
```

## 14. Fact

Facts represent assertions about the story world.

```yaml
fact:
  id: uuid
  campaign_id: uuid
  subject_entity_id: optional uuid
  predicate: string
  object_entity_id: optional uuid
  value: optional scalar_or_object
  authority: string
  source_type: string
  source_id: optional uuid
  created_turn_id: optional uuid
  invalidated_turn_id: optional uuid
  status: active | superseded | disputed
```

Examples:
- Mara knows Aldric has the silver key.
- The Persephone cannot travel faster than light.
- Edrin disappeared three weeks before the campaign began.
- The eastern bridge collapsed during Turn 47.

Minimum authority categories:
- campaign_canon,
- accepted_story,
- current_state,
- imported_canon,
- reference,
- heuristic,
- inspiration.

Exact context behavior will be defined in `CONTEXT-AND-MEMORY.md`.

## 15. Relationship

```yaml
relationship:
  id: uuid
  campaign_id: uuid
  source_entity_id: uuid
  target_entity_id: uuid
  type: string
  status: string
  description: optional string
  created_turn_id: optional uuid
  ended_turn_id: optional uuid
```

Examples:
- Aldric -> trusts -> Mara
- Mara -> member_of -> Circle of Ash
- Silver Key -> belongs_to -> Aldric
- Persephone -> docked_at -> Ceres Station

Relationship types should remain extensible.

## 16. Story Thread

Story threads track unresolved or resolved narrative business.

```yaml
story_thread:
  id: uuid
  campaign_id: uuid
  title: string
  description: string
  status: open | dormant | resolved | abandoned
  importance: optional number_or_label
  opened_turn_id: optional uuid
  resolved_turn_id: optional uuid
```

These are narrative continuity tools, not RPG quests.

## 16A. Derived Context Tables (M6, as implemented)

Two tables and one column carry M6's derived context. All three are derived
data: deleting them changes no accepted history, no authoritative state and no
head position.

```text
summaries
  id, adventure_id
  text
  branch_id, depth          the coordinate of the last node covered
  source_start, source_end  the stretch of story summarized
  trigger                   "interval" (generated) or "manual" (reader-written)
  model_name, created_at

derived_status              one row per (adventure, kind)
  kind                      "memory" | "summary" | "embedding"
  status                    "ok" (did work) | "idle" (nothing pending) | "failed"
  detail, failures
  last_attempt_at, last_success_at

memories.authority          "accepted_story" | "heuristic"
```

`summaries` mirrors the shape `memories` already had, deliberately: both are
derived rows anchored to a coordinate on a path, and both are filtered by the
same lineage clause — and both are also the *input* to the next round of derived
work, which is why summary generation reads `summaries.current` rather than any
campaign-global field.

`adventures.story_summary` is retained as the reader's edit surface and the
export field, mirroring whichever summary is eligible. It carries no lineage of
its own and nothing authoritative reads it.

## 17. State Version

The system must reconstruct authoritative state at any retained turn.

```yaml
state_version:
  id: uuid
  campaign_id: uuid
  turn_id: optional uuid
  parent_state_version_id: optional uuid
  created_at: timestamp
  state_hash: optional string
```

Possible implementation models:

### A. Full snapshots
Simple restore, but duplicates data.

### B. Event sourcing
Excellent auditability, but requires replay.

### C. Hybrid
Validated events plus periodic/current snapshots.

**Selected for v1: Hybrid.** Store validated authoritative events plus efficient state snapshots/cache for normal reads and restore.

## 18. State Change Event

If the hybrid/event model is selected:

```yaml
state_event:
  id: uuid
  campaign_id: uuid
  turn_id: uuid
  event_type: string
  payload: object
  sequence: integer
```

Potential event types:
- entity_created,
- set_entity_status,
- set_entity_attribute,
- fact_added,
- fact_invalidated,
- relationship_added,
- relationship_ended,
- thread_opened,
- thread_resolved,
- current_location_set,
- possession_set,
- scene_set.

Event semantics must be explicit and typed. Prefer unambiguous absolute assignments for mutable values. If incremental operations are ever needed, encode the operation explicitly (for example `increment_value`) rather than relying on one numeric field whose interpretation is implicit.

Events must be schema-validated, semantically checked where deterministic rules exist, and accepted by the application before commit.

## 19. State Proposal

The model's extracted state proposal must be distinct from accepted state.

```yaml
state_proposal:
  id: uuid
  turn_id: uuid
  model: string
  raw_output: string
  parsed_payload: object
  validation_status: accepted | partially_accepted | rejected | repair_required
```

The model must never write directly to authoritative state tables. The production protocol must not depend on AI-DnD-style ambiguous relative deltas; see ADR 010.

## 20. Scene Snapshot

A scene snapshot captures the immediate narrative situation.

```yaml
scene:
  id: uuid
  campaign_id: uuid
  branch_id: uuid
  source_turn_start_id: optional uuid
  source_turn_end_id: optional uuid
  location_id: optional uuid
  time_description: optional string
  mood: optional string
  environment: optional string
  participants: [uuid]
  significant_objects: [uuid]
  current_actions: [string]
  visual_notes: [string]
  continuity_notes: [string]
```

Scene snapshots support:
- current context,
- narrative continuity,
- future image generation,
- future multi-turn video/storyboard generation.

## 21. Summary

Summaries compress history but never replace authoritative history.

```yaml
summary:
  id: uuid
  campaign_id: uuid
  branch_id: uuid
  type: campaign | arc | rolling | turn_range
  source_start_turn_id: uuid
  source_end_turn_id: uuid
  text: string
  created_at: timestamp
  model: optional string
```

Summaries are derived data. Branch changes must invalidate or lineage-filter incompatible summaries.

## 22. Memory

```yaml
memory:
  id: uuid
  campaign_id: uuid
  branch_scope: optional uuid
  source_turn_id: optional uuid
  type: string
  text: string
  authority: string
  importance: optional number
  embedding_ref: optional string
  created_at: timestamp
```

Potential types:
- event,
- character,
- relationship,
- location,
- promise,
- discovery,
- conflict,
- heuristic.

A retrieved memory does not automatically become canon.

## 23. Knowledge Source

A knowledge source is a user-imported local file or manually authored campaign document.

```yaml
knowledge_source:
  id: uuid
  campaign_id: optional uuid
  title: string
  source_type: file | manual
  classification: canon | reference | inspiration
  original_filename: optional string
  content_hash: string
  enabled: boolean
  imported_at: timestamp
  metadata: object
```

Initial supported file types should be `.txt` and `.md`.

## 24. Knowledge Chunk

```yaml
knowledge_chunk:
  id: uuid
  source_id: uuid
  sequence: integer
  text: string
  heading_path: optional string
  token_count: optional integer
  embedding_ref: optional string
  metadata: object
```

Requirements:
- preserve provenance,
- preserve chunk order,
- permit re-indexing,
- never execute imported content.

## 25. Retrieval Record

Every turn should record which memories or knowledge chunks were supplied to the narrator.

```yaml
retrieval_record:
  id: uuid
  turn_id: uuid
  source_kind: memory | knowledge | summary | fact
  source_id: uuid
  retrieval_method: lexical | semantic | hybrid | forced
  score: optional number
  rank: integer
```

This lets the prompt inspector answer:

> Why did the narrator know this?

## 26. Prompt Snapshot

```yaml
prompt_snapshot:
  id: uuid
  turn_id: uuid
  created_at: timestamp
  system_instructions: text
  campaign_context: text_or_structured
  state_context: text_or_structured
  summary_context: text_or_structured
  memory_context: text_or_structured
  knowledge_context: text_or_structured
  recent_history: text_or_structured
  user_input: text
  token_accounting: object
```

The physical representation may be compressed or normalized. Reproducibility and inspection are the requirements.

## 27. Media Job

Not required for v1 behavior, but the data model should not prevent it.

```yaml
media_job:
  id: uuid
  campaign_id: uuid
  scene_id: optional uuid
  source_turn_start_id: optional uuid
  source_turn_end_id: optional uuid
  type: image | video | audio | tts | stt
  provider: string
  model: string
  status: queued | running | completed | failed
  request_payload: object
  created_at: timestamp
  completed_at: optional timestamp
```

## 28. Media Asset

```yaml
media_asset:
  id: uuid
  campaign_id: uuid
  media_job_id: optional uuid
  scene_id: optional uuid
  type: image | video | audio | tts | stt
  file_path: string
  metadata: object
  created_at: timestamp
```

Potential metadata includes prompt, seed, model, workflow, dimensions, duration, character references, and source turn range.

## 29. Export Package

A campaign export should be capable of preserving:

- campaign configuration,
- branches,
- turn graph,
- checkpoints,
- state/events,
- entities,
- facts,
- relationships,
- story threads,
- summaries,
- memories,
- scene snapshots,
- knowledge-source metadata,
- knowledge chunks/source files if selected,
- prompt provenance if selected,
- media metadata,
- media files if selected.

The physical container format remains an implementation choice, but the export must preserve the exact active branch **and active head position**, even when the head is behind a retained tip after Undo. A ZIP containing a database plus manifest remains a strong candidate.

As implemented in M3, the export carries the active branch, the active head
position on it, and each branch's disposition, alongside the whole retained turn
graph. **M4 added the checkpoints**, by the same rule: a position someone chose
to name cannot be recomputed from the turns, because nothing about a turn records
that it was bookmarked. The head and the checkpoints stay independent on import —
a campaign opens where its head says, never at a checkpoint merely because one is
in the file. The governing rule for this package is that an export carries what was
*chosen* and recomputes what is *derived* — and the active head moved from the
second category to the first, because once Undo stops deleting, two campaigns
with identical turns can be being read at different positions and no import can
tell which. An export written before the field existed is opened at its retained
tip, which is the position such a file recorded.

## 30. Deletion vs Archival

The system must distinguish:
- archive,
- detach/disable,
- permanent delete.

Retry, Undo, Restore, and branch switching must not silently perform permanent deletion.

## 31. Authoritative vs Derived Data

### Authoritative
- accepted turns,
- branch lineage,
- campaign configuration,
- accepted facts,
- accepted relationships,
- accepted state events,
- checkpoints.

### Derived
- summaries,
- embeddings,
- semantic indexes,
- lexical indexes,
- some automatically produced scene descriptions.

Derived data should be rebuildable where practical.

## 32. Provenance

Important information should answer:

> Where did this come from?

Potential provenance:
- campaign setup,
- manual user edit,
- accepted narrator turn,
- state extraction,
- imported canon,
- imported reference,
- imported inspiration,
- derived inference.

## 33. Phase 0B Decisions Applied

Phase 0B resolved the foundational data-model questions:

- AI-DnD story lineage/alternate-take machinery is the production starting point.
- The active head is distinct from retained tip history.
- Undo/Redo use head movement; destructive deletion is not part of ordinary history operations.
- Named checkpoints are durable pointers to recoverable head positions.
- State uses a hybrid event + snapshot/cache model.
- State proposals use explicit typed operations with unambiguous value semantics.
- Memories/summaries must be lineage-filtered or lineage-anchored.
- Imported knowledge requires separate source/chunk/index tables rather than overloading Story Cards.
- Scene/media records remain optional derived extensions and must carry source lineage.
- Export/import must preserve active head position as well as the retained history graph.

## 34. Acceptance Criteria

The final v1 data model must support all of these without destructive hacks:

- close/restart/resume exact story,
- Undo/Redo without deleting accepted turns,
- active head behind retained tip,
- branch from an earlier turn while retaining the original future,
- mark abandoned futures/takes retained/disposable,
- create and restore named checkpoints,
- know current characters/locations/relationships/story threads,
- reconstruct earlier authoritative state,
- validate typed state proposals before committing events,
- retrieve old events outside the active context window,
- identify which imported passages informed a turn,
- reconstruct what was sent to Ollama,
- export/import an undone campaign without silently redoing it,
- run fantasy and science-fiction campaigns without schema changes,
- attach future image/video/audio/TTS/STT metadata to scenes or turn ranges without making media authoritative.

## 35. Selected Conceptual Model

```text
Retained Turn/Take Lineage
        |
        +--> Active branch + movable active head
        |
        +--> Validated typed state events
        |       |
        |       +--> state snapshot/cache
        |
        +--> lineage-safe summaries/memories
        |
        +--> prompt/retrieval provenance
        |
        +--> scene snapshot
        |
        +--> future optional media records

Separate campaign knowledge subsystem
        +--> sources
        +--> chunks
        +--> FTS/local embeddings
        +--> authority/provenance
```

This combines AI-DnD's retained story lineage and snapshot infrastructure with ai-adventure-style explicit event/commit discipline while preserving the project's own specification as the authority.