interactive-story/planning/DATA-MODEL.md

Adventure Storyteller — Data Model

Status: Draft v0.1
Purpose: Define the persistent information the application must represent, independent of the final fork or database implementation.

1. Design Goals

The data model must support persistent interactive stories, complete authoritative history, non-destructive branching, checkpoints and rollback, genre-independent narrative state, long-term memory, imported local knowledge, prompt/context provenance, future image/video generation, export/restore, and local-only operation.

The same core schema should work for fantasy, science fiction, mystery, horror, historical fiction, westerns, and other narrative genres.

2. Core Principles

2.1 Application-owned authority

The database is the source of truth. The model may propose narration and state changes, but those proposals become authoritative only after application validation.

2.2 Non-destructive history

Accepted turns are historical records. Going backward should move the active story head or create a branch, not silently delete accepted history.

2.3 Historical reconstruction

The system must answer both:

• What is true now?
• What was true at a particular earlier turn on a particular branch?

2.4 Genre neutrality

Avoid fantasy- or science-fiction-specific core fields. Use generic concepts such as characters, locations, organizations, items, vehicles, facts, relationships, conditions, scenes, and story threads.

3. Conceptual Entity Map

Campaign
 ├── Branches
 │    └── Turns
 │         ├── Prompt Snapshot
 │         ├── State Version
 │         ├── Scene Snapshot
 │         └── Retrieval Records
 ├── Checkpoints
 ├── Narrative Entities
 ├── Facts
 ├── Relationships
 ├── Story Threads
 ├── Memories
 ├── Summaries
 ├── Knowledge Sources
 │    └── Knowledge Chunks
 └── Media
      ├── Media Jobs
      └── Media Assets

The exact SQL schema may differ from this conceptual model.

4. Campaign

A campaign is the top-level story container.

campaign:
  id: uuid
  title: string
  created_at: timestamp
  updated_at: timestamp
  active_branch_id: uuid
  status: active | archived

story_profile:
  genre: string
  subgenre: optional string
  tone: optional string
  style: optional string
  point_of_view: optional string
  tense: optional string

Campaigns also store durable narrator rules and model configuration.

Potential model roles:

• narrator,
• state extractor,
• summarizer,
• embedding model.

For v1, all model roles should use local Ollama-compatible models.

5. Branch

A branch represents one valid continuation of story history.

branch:
  id: uuid
  campaign_id: uuid
  name: string
  created_at: timestamp
  created_from_branch_id: optional uuid
  fork_turn_id: optional uuid
  head_turn_id: optional uuid
  status: active | archived

Rules:

• branches may share ancestral turns,
• shared history should not be duplicated unnecessarily,
• creating a branch must not modify the source branch.

Detailed behavior will be defined separately in STORY-BRANCH-SEMANTICS.md.

6. Turn

A turn is one accepted story interaction:

user input -> narrator response -> accepted state transition

turn:
  id: uuid
  campaign_id: uuid
  branch_id: uuid
  parent_turn_id: optional uuid
  created_at: timestamp

input:
  mode: action | dialogue | direction | continue
  text: string

output:
  narration: string

generation:
  provider: ollama
  model: string
  generation_settings: object
  prompt_snapshot_id: uuid

state_before_id: uuid
state_after_id: uuid
scene_snapshot_id: optional uuid
status: pending | accepted | failed | superseded

The root turn has no parent.

An accepted historical turn should not silently disappear if another continuation is chosen.

7. Alternate Take

The system may need to distinguish:

• a different narrator response to the same user action,
• a genuinely different story branch.

Conceptual form:

take:
  id: uuid
  turn_request_id: uuid
  output_text: string
  model_metadata: object
  selected: boolean

Whether Take becomes a separate table or sibling turn nodes will be decided after Phase 0B.

8. Checkpoint

checkpoint:
  id: uuid
  campaign_id: uuid
  turn_id: uuid
  name: string
  notes: optional string
  created_at: timestamp

A checkpoint is a named pointer to a recoverable story position. It should normally remain tied to the turn where it was created.

9. Narrative Entity

An entity is a persistent thing or concept in the fictional world.

entity:
  id: uuid
  campaign_id: uuid
  type: string
  name: string
  aliases: [string]
  description: string
  status: active | inactive | destroyed | dead | unknown
  created_turn_id: optional uuid
  metadata: object

Recommended built-in categories:

• character,
• location,
• organization,
• item,
• vehicle,
• creature,
• structure,
• concept,
• other.

These are descriptive categories, not separate game systems.

10. Character

character:
  entity_id: uuid
  role: optional string
  description: string
  current_location_id: optional uuid
  condition: [string]
  personality_notes: [string]
  goals: [string]
  secrets: [string]

Optional visual continuity fields:

visual_profile:
  apparent_age: optional string
  build: optional string
  hair: optional string
  eyes: optional string
  clothing: optional string
  distinctive_features: [string]
  continuity_notes: [string]

11. Location

location:
  entity_id: uuid
  description: string
  parent_location_id: optional uuid
  current_status: optional string

visual_profile:
  architecture: optional string
  environment: optional string
  lighting: optional string
  signature_features: [string]
  continuity_notes: [string]

12. Organization

Organizations may represent factions, governments, companies, guilds, military units, religious organizations, or informal groups.

organization:
  entity_id: uuid
  purpose: optional string
  current_status: optional string

Specific characteristics should usually live in facts and relationships.

13. Items and Vehicles

Items and vehicles remain generic entities.

Examples:

• silver key,
• longsword,
• encrypted data crystal,
• survey ship,
• horse-drawn carriage.

Possession and location should normally be represented as relationships or facts:

Aldric --possesses--> Silver Key
Persephone --docked_at--> Ceres Station

14. Fact

Facts represent assertions about the story world.

fact:
  id: uuid
  campaign_id: uuid
  subject_entity_id: optional uuid
  predicate: string
  object_entity_id: optional uuid
  value: optional scalar_or_object
  authority: string
  source_type: string
  source_id: optional uuid
  created_turn_id: optional uuid
  invalidated_turn_id: optional uuid
  status: active | superseded | disputed

Examples:

• Mara knows Aldric has the silver key.
• The Persephone cannot travel faster than light.
• Edrin disappeared three weeks before the campaign began.
• The eastern bridge collapsed during Turn 47.

Minimum authority categories:

• campaign_canon,
• accepted_story,
• current_state,
• imported_canon,
• reference,
• heuristic,
• inspiration.

Exact context behavior will be defined in CONTEXT-AND-MEMORY.md.

15. Relationship

relationship:
  id: uuid
  campaign_id: uuid
  source_entity_id: uuid
  target_entity_id: uuid
  type: string
  status: string
  description: optional string
  created_turn_id: optional uuid
  ended_turn_id: optional uuid

Examples:

• Aldric -> trusts -> Mara
• Mara -> member_of -> Circle of Ash
• Silver Key -> belongs_to -> Aldric
• Persephone -> docked_at -> Ceres Station

Relationship types should remain extensible.

16. Story Thread

Story threads track unresolved or resolved narrative business.

story_thread:
  id: uuid
  campaign_id: uuid
  title: string
  description: string
  status: open | dormant | resolved | abandoned
  importance: optional number_or_label
  opened_turn_id: optional uuid
  resolved_turn_id: optional uuid

These are narrative continuity tools, not RPG quests.

17. State Version

The system must reconstruct authoritative state at any retained turn.

state_version:
  id: uuid
  campaign_id: uuid
  turn_id: optional uuid
  parent_state_version_id: optional uuid
  created_at: timestamp
  state_hash: optional string

Possible implementation models:

A. Full snapshots

Simple restore, but duplicates data.

B. Event sourcing

Excellent auditability, but requires replay.

C. Hybrid

Validated events plus periodic/current snapshots.

Current preference: Hybrid, pending Phase 0B.

18. State Change Event

If the hybrid/event model is selected:

state_event:
  id: uuid
  campaign_id: uuid
  turn_id: uuid
  event_type: string
  payload: object
  sequence: integer

Potential event types:

• entity_created,
• entity_updated,
• fact_added,
• fact_invalidated,
• relationship_added,
• relationship_ended,
• thread_opened,
• thread_resolved,
• location_changed,
• possession_changed,
• scene_changed.

Events must be validated before commit.

19. State Proposal

The model's extracted state proposal must be distinct from accepted state.

state_proposal:
  id: uuid
  turn_id: uuid
  model: string
  raw_output: string
  parsed_payload: object
  validation_status: accepted | partially_accepted | rejected | repair_required

The model must never write directly to authoritative state tables.

20. Scene Snapshot

A scene snapshot captures the immediate narrative situation.

scene:
  id: uuid
  campaign_id: uuid
  branch_id: uuid
  source_turn_start_id: optional uuid
  source_turn_end_id: optional uuid
  location_id: optional uuid
  time_description: optional string
  mood: optional string
  environment: optional string
  participants: [uuid]
  significant_objects: [uuid]
  current_actions: [string]
  visual_notes: [string]
  continuity_notes: [string]

Scene snapshots support:

• current context,
• narrative continuity,
• future image generation,
• future multi-turn video/storyboard generation.

21. Summary

Summaries compress history but never replace authoritative history.

summary:
  id: uuid
  campaign_id: uuid
  branch_id: uuid
  type: campaign | arc | rolling | turn_range
  source_start_turn_id: uuid
  source_end_turn_id: uuid
  text: string
  created_at: timestamp
  model: optional string

Summaries are derived data. Branch changes must invalidate or lineage-filter incompatible summaries.

22. Memory

memory:
  id: uuid
  campaign_id: uuid
  branch_scope: optional uuid
  source_turn_id: optional uuid
  type: string
  text: string
  authority: string
  importance: optional number
  embedding_ref: optional string
  created_at: timestamp

Potential types:

• event,
• character,
• relationship,
• location,
• promise,
• discovery,
• conflict,
• heuristic.

A retrieved memory does not automatically become canon.

23. Knowledge Source

A knowledge source is a user-imported local file or manually authored campaign document.

knowledge_source:
  id: uuid
  campaign_id: optional uuid
  title: string
  source_type: file | manual
  classification: canon | reference | inspiration
  original_filename: optional string
  content_hash: string
  enabled: boolean
  imported_at: timestamp
  metadata: object

Initial supported file types should be .txt and .md.

24. Knowledge Chunk

knowledge_chunk:
  id: uuid
  source_id: uuid
  sequence: integer
  text: string
  heading_path: optional string
  token_count: optional integer
  embedding_ref: optional string
  metadata: object

Requirements:

• preserve provenance,
• preserve chunk order,
• permit re-indexing,
• never execute imported content.

25. Retrieval Record

Every turn should record which memories or knowledge chunks were supplied to the narrator.

retrieval_record:
  id: uuid
  turn_id: uuid
  source_kind: memory | knowledge | summary | fact
  source_id: uuid
  retrieval_method: lexical | semantic | hybrid | forced
  score: optional number
  rank: integer

This lets the prompt inspector answer:

Why did the narrator know this?

26. Prompt Snapshot

prompt_snapshot:
  id: uuid
  turn_id: uuid
  created_at: timestamp
  system_instructions: text
  campaign_context: text_or_structured
  state_context: text_or_structured
  summary_context: text_or_structured
  memory_context: text_or_structured
  knowledge_context: text_or_structured
  recent_history: text_or_structured
  user_input: text
  token_accounting: object

The physical representation may be compressed or normalized. Reproducibility and inspection are the requirements.

27. Media Job

Not required for v1 behavior, but the data model should not prevent it.

media_job:
  id: uuid
  campaign_id: uuid
  scene_id: optional uuid
  source_turn_start_id: optional uuid
  source_turn_end_id: optional uuid
  type: image | video | audio
  provider: string
  model: string
  status: queued | running | completed | failed
  request_payload: object
  created_at: timestamp
  completed_at: optional timestamp

28. Media Asset

media_asset:
  id: uuid
  campaign_id: uuid
  media_job_id: optional uuid
  scene_id: optional uuid
  type: image | video | audio
  file_path: string
  metadata: object
  created_at: timestamp

Potential metadata includes prompt, seed, model, workflow, dimensions, duration, character references, and source turn range.

29. Export Package

A campaign export should be capable of preserving:

• campaign configuration,
• branches,
• turn graph,
• checkpoints,
• state/events,
• entities,
• facts,
• relationships,
• story threads,
• summaries,
• memories,
• scene snapshots,
• knowledge-source metadata,
• knowledge chunks/source files if selected,
• prompt provenance if selected,
• media metadata,
• media files if selected.

Exact format remains open. A ZIP containing a database plus manifest is a strong candidate.

30. Deletion vs Archival

The system must distinguish:

• archive,
• detach/disable,
• permanent delete.

Retry, Undo, Restore, and branch switching must not silently perform permanent deletion.

31. Authoritative vs Derived Data

Authoritative

• accepted turns,
• branch lineage,
• campaign configuration,
• accepted facts,
• accepted relationships,
• accepted state events,
• checkpoints.

Derived

• summaries,
• embeddings,
• semantic indexes,
• lexical indexes,
• some automatically produced scene descriptions.

Derived data should be rebuildable where practical.

32. Provenance

Important information should answer:

Where did this come from?

Potential provenance:

• campaign setup,
• manual user edit,
• accepted narrator turn,
• state extraction,
• imported canon,
• imported reference,
• imported inspiration,
• derived inference.

33. Open Questions for Phase 0B

1. Does AI-DnD already model alternate takes separately from branch nodes in a reusable way?
2. Can its state snapshots hold generic narrative JSON without major redesign?
3. Is its branch lineage compatible with immutable turns?
4. Should checkpoints be branch-independent pointers to turns?
5. Should memories be physically branch-scoped or lineage-filtered at query time?
6. Should knowledge sources be reusable across campaigns in v1?
7. Should media tables physically exist in v1 or only interfaces/types?
8. How should manual edits to canon/state be versioned?
9. Which state needs full historical reconstruction versus only current-state storage?
10. Can ai-adventure's event/replay discipline be adopted without overcomplicating AI-DnD?

34. Acceptance Criteria

The final v1 data model must support all of these without destructive hacks:

• close/restart/resume exact story,
• branch from an earlier turn while retaining the original future,
• create and restore named checkpoints,
• know current characters/locations/relationships/story threads,
• reconstruct earlier authoritative state,
• retrieve old events outside the active context window,
• identify which imported passages informed a turn,
• reconstruct what was sent to Ollama,
• run fantasy and science-fiction campaigns without schema changes,
• attach future image/video assets to scenes or turn ranges.

35. Current Recommendation

The target conceptual model should be:

Immutable Turn Graph
        |
        +--> validated state events
        |
        +--> state snapshot/cache
        |
        +--> scene snapshot
        |
        +--> prompt/retrieval provenance

This combines the strongest observed concepts from:

• AI-DnD's story tree and snapshots,
• ai-adventure's append-only event/replay discipline,
• Open Dungeon's simple story UX and visual continuity.

The physical implementation remains provisional until Phase 0B validates the preferred production base.