Add initial planning files from ChatGPT research here

2026-09-01 12:09:38 -04:00
commit f011362494
37 changed files with 14601 additions and 0 deletions
@@ -0,0 +1,745 @@
+# Adventure Storyteller — Data Model
+
+**Status:** Draft v0.1  
+**Purpose:** Define the persistent information the application must represent, independent of the final fork or database implementation.
+
+## 1. Design Goals
+
+The data model must support persistent interactive stories, complete authoritative history, non-destructive branching, checkpoints and rollback, genre-independent narrative state, long-term memory, imported local knowledge, prompt/context provenance, future image/video generation, export/restore, and local-only operation.
+
+The same core schema should work for fantasy, science fiction, mystery, horror, historical fiction, westerns, and other narrative genres.
+
+## 2. Core Principles
+
+### 2.1 Application-owned authority
+The database is the source of truth. The model may propose narration and state changes, but those proposals become authoritative only after application validation.
+
+### 2.2 Non-destructive history
+Accepted turns are historical records. Going backward should move the active story head or create a branch, not silently delete accepted history.
+
+### 2.3 Historical reconstruction
+The system must answer both:
+- What is true now?
+- What was true at a particular earlier turn on a particular branch?
+
+### 2.4 Genre neutrality
+Avoid fantasy- or science-fiction-specific core fields. Use generic concepts such as characters, locations, organizations, items, vehicles, facts, relationships, conditions, scenes, and story threads.
+
+## 3. Conceptual Entity Map
+
+```text
+Campaign
+ ├── Branches
+ │    └── Turns
+ │         ├── Prompt Snapshot
+ │         ├── State Version
+ │         ├── Scene Snapshot
+ │         └── Retrieval Records
+ ├── Checkpoints
+ ├── Narrative Entities
+ ├── Facts
+ ├── Relationships
+ ├── Story Threads
+ ├── Memories
+ ├── Summaries
+ ├── Knowledge Sources
+ │    └── Knowledge Chunks
+ └── Media
+      ├── Media Jobs
+      └── Media Assets
+```
+
+The exact SQL schema may differ from this conceptual model.
+
+## 4. Campaign
+
+A campaign is the top-level story container.
+
+```yaml
+campaign:
+  id: uuid
+  title: string
+  created_at: timestamp
+  updated_at: timestamp
+  active_branch_id: uuid
+  status: active | archived
+
+story_profile:
+  genre: string
+  subgenre: optional string
+  tone: optional string
+  style: optional string
+  point_of_view: optional string
+  tense: optional string
+```
+
+Campaigns also store durable narrator rules and model configuration.
+
+Potential model roles:
+- narrator,
+- state extractor,
+- summarizer,
+- embedding model.
+
+For v1, all model roles should use local Ollama-compatible models.
+
+## 5. Branch
+
+A branch represents one valid continuation of story history.
+
+```yaml
+branch:
+  id: uuid
+  campaign_id: uuid
+  name: string
+  created_at: timestamp
+  created_from_branch_id: optional uuid
+  fork_turn_id: optional uuid
+  head_turn_id: optional uuid
+  status: active | archived
+```
+
+Rules:
+- branches may share ancestral turns,
+- shared history should not be duplicated unnecessarily,
+- creating a branch must not modify the source branch.
+
+Detailed behavior will be defined separately in `STORY-BRANCH-SEMANTICS.md`.
+
+## 6. Turn
+
+A turn is one accepted story interaction:
+
+```text
+user input -> narrator response -> accepted state transition
+```
+
+```yaml
+turn:
+  id: uuid
+  campaign_id: uuid
+  branch_id: uuid
+  parent_turn_id: optional uuid
+  created_at: timestamp
+
+input:
+  mode: action | dialogue | direction | continue
+  text: string
+
+output:
+  narration: string
+
+generation:
+  provider: ollama
+  model: string
+  generation_settings: object
+  prompt_snapshot_id: uuid
+
+state_before_id: uuid
+state_after_id: uuid
+scene_snapshot_id: optional uuid
+status: pending | accepted | failed | superseded
+```
+
+The root turn has no parent.
+
+An accepted historical turn should not silently disappear if another continuation is chosen.
+
+## 7. Alternate Take
+
+The system may need to distinguish:
+- a different narrator response to the same user action,
+- a genuinely different story branch.
+
+Conceptual form:
+
+```yaml
+take:
+  id: uuid
+  turn_request_id: uuid
+  output_text: string
+  model_metadata: object
+  selected: boolean
+```
+
+Whether `Take` becomes a separate table or sibling turn nodes will be decided after Phase 0B.
+
+## 8. Checkpoint
+
+```yaml
+checkpoint:
+  id: uuid
+  campaign_id: uuid
+  turn_id: uuid
+  name: string
+  notes: optional string
+  created_at: timestamp
+```
+
+A checkpoint is a named pointer to a recoverable story position. It should normally remain tied to the turn where it was created.
+
+## 9. Narrative Entity
+
+An entity is a persistent thing or concept in the fictional world.
+
+```yaml
+entity:
+  id: uuid
+  campaign_id: uuid
+  type: string
+  name: string
+  aliases: [string]
+  description: string
+  status: active | inactive | destroyed | dead | unknown
+  created_turn_id: optional uuid
+  metadata: object
+```
+
+Recommended built-in categories:
+- character,
+- location,
+- organization,
+- item,
+- vehicle,
+- creature,
+- structure,
+- concept,
+- other.
+
+These are descriptive categories, not separate game systems.
+
+## 10. Character
+
+```yaml
+character:
+  entity_id: uuid
+  role: optional string
+  description: string
+  current_location_id: optional uuid
+  condition: [string]
+  personality_notes: [string]
+  goals: [string]
+  secrets: [string]
+```
+
+Optional visual continuity fields:
+
+```yaml
+visual_profile:
+  apparent_age: optional string
+  build: optional string
+  hair: optional string
+  eyes: optional string
+  clothing: optional string
+  distinctive_features: [string]
+  continuity_notes: [string]
+```
+
+## 11. Location
+
+```yaml
+location:
+  entity_id: uuid
+  description: string
+  parent_location_id: optional uuid
+  current_status: optional string
+
+visual_profile:
+  architecture: optional string
+  environment: optional string
+  lighting: optional string
+  signature_features: [string]
+  continuity_notes: [string]
+```
+
+## 12. Organization
+
+Organizations may represent factions, governments, companies, guilds, military units, religious organizations, or informal groups.
+
+```yaml
+organization:
+  entity_id: uuid
+  purpose: optional string
+  current_status: optional string
+```
+
+Specific characteristics should usually live in facts and relationships.
+
+## 13. Items and Vehicles
+
+Items and vehicles remain generic entities.
+
+Examples:
+- silver key,
+- longsword,
+- encrypted data crystal,
+- survey ship,
+- horse-drawn carriage.
+
+Possession and location should normally be represented as relationships or facts:
+
+```text
+Aldric --possesses--> Silver Key
+Persephone --docked_at--> Ceres Station
+```
+
+## 14. Fact
+
+Facts represent assertions about the story world.
+
+```yaml
+fact:
+  id: uuid
+  campaign_id: uuid
+  subject_entity_id: optional uuid
+  predicate: string
+  object_entity_id: optional uuid
+  value: optional scalar_or_object
+  authority: string
+  source_type: string
+  source_id: optional uuid
+  created_turn_id: optional uuid
+  invalidated_turn_id: optional uuid
+  status: active | superseded | disputed
+```
+
+Examples:
+- Mara knows Aldric has the silver key.
+- The Persephone cannot travel faster than light.
+- Edrin disappeared three weeks before the campaign began.
+- The eastern bridge collapsed during Turn 47.
+
+Minimum authority categories:
+- campaign_canon,
+- accepted_story,
+- current_state,
+- imported_canon,
+- reference,
+- heuristic,
+- inspiration.
+
+Exact context behavior will be defined in `CONTEXT-AND-MEMORY.md`.
+
+## 15. Relationship
+
+```yaml
+relationship:
+  id: uuid
+  campaign_id: uuid
+  source_entity_id: uuid
+  target_entity_id: uuid
+  type: string
+  status: string
+  description: optional string
+  created_turn_id: optional uuid
+  ended_turn_id: optional uuid
+```
+
+Examples:
+- Aldric -> trusts -> Mara
+- Mara -> member_of -> Circle of Ash
+- Silver Key -> belongs_to -> Aldric
+- Persephone -> docked_at -> Ceres Station
+
+Relationship types should remain extensible.
+
+## 16. Story Thread
+
+Story threads track unresolved or resolved narrative business.
+
+```yaml
+story_thread:
+  id: uuid
+  campaign_id: uuid
+  title: string
+  description: string
+  status: open | dormant | resolved | abandoned
+  importance: optional number_or_label
+  opened_turn_id: optional uuid
+  resolved_turn_id: optional uuid
+```
+
+These are narrative continuity tools, not RPG quests.
+
+## 17. State Version
+
+The system must reconstruct authoritative state at any retained turn.
+
+```yaml
+state_version:
+  id: uuid
+  campaign_id: uuid
+  turn_id: optional uuid
+  parent_state_version_id: optional uuid
+  created_at: timestamp
+  state_hash: optional string
+```
+
+Possible implementation models:
+
+### A. Full snapshots
+Simple restore, but duplicates data.
+
+### B. Event sourcing
+Excellent auditability, but requires replay.
+
+### C. Hybrid
+Validated events plus periodic/current snapshots.
+
+**Current preference: Hybrid**, pending Phase 0B.
+
+## 18. State Change Event
+
+If the hybrid/event model is selected:
+
+```yaml
+state_event:
+  id: uuid
+  campaign_id: uuid
+  turn_id: uuid
+  event_type: string
+  payload: object
+  sequence: integer
+```
+
+Potential event types:
+- entity_created,
+- entity_updated,
+- fact_added,
+- fact_invalidated,
+- relationship_added,
+- relationship_ended,
+- thread_opened,
+- thread_resolved,
+- location_changed,
+- possession_changed,
+- scene_changed.
+
+Events must be validated before commit.
+
+## 19. State Proposal
+
+The model's extracted state proposal must be distinct from accepted state.
+
+```yaml
+state_proposal:
+  id: uuid
+  turn_id: uuid
+  model: string
+  raw_output: string
+  parsed_payload: object
+  validation_status: accepted | partially_accepted | rejected | repair_required
+```
+
+The model must never write directly to authoritative state tables.
+
+## 20. Scene Snapshot
+
+A scene snapshot captures the immediate narrative situation.
+
+```yaml
+scene:
+  id: uuid
+  campaign_id: uuid
+  branch_id: uuid
+  source_turn_start_id: optional uuid
+  source_turn_end_id: optional uuid
+  location_id: optional uuid
+  time_description: optional string
+  mood: optional string
+  environment: optional string
+  participants: [uuid]
+  significant_objects: [uuid]
+  current_actions: [string]
+  visual_notes: [string]
+  continuity_notes: [string]
+```
+
+Scene snapshots support:
+- current context,
+- narrative continuity,
+- future image generation,
+- future multi-turn video/storyboard generation.
+
+## 21. Summary
+
+Summaries compress history but never replace authoritative history.
+
+```yaml
+summary:
+  id: uuid
+  campaign_id: uuid
+  branch_id: uuid
+  type: campaign | arc | rolling | turn_range
+  source_start_turn_id: uuid
+  source_end_turn_id: uuid
+  text: string
+  created_at: timestamp
+  model: optional string
+```
+
+Summaries are derived data. Branch changes must invalidate or lineage-filter incompatible summaries.
+
+## 22. Memory
+
+```yaml
+memory:
+  id: uuid
+  campaign_id: uuid
+  branch_scope: optional uuid
+  source_turn_id: optional uuid
+  type: string
+  text: string
+  authority: string
+  importance: optional number
+  embedding_ref: optional string
+  created_at: timestamp
+```
+
+Potential types:
+- event,
+- character,
+- relationship,
+- location,
+- promise,
+- discovery,
+- conflict,
+- heuristic.
+
+A retrieved memory does not automatically become canon.
+
+## 23. Knowledge Source
+
+A knowledge source is a user-imported local file or manually authored campaign document.
+
+```yaml
+knowledge_source:
+  id: uuid
+  campaign_id: optional uuid
+  title: string
+  source_type: file | manual
+  classification: canon | reference | inspiration
+  original_filename: optional string
+  content_hash: string
+  enabled: boolean
+  imported_at: timestamp
+  metadata: object
+```
+
+Initial supported file types should be `.txt` and `.md`.
+
+## 24. Knowledge Chunk
+
+```yaml
+knowledge_chunk:
+  id: uuid
+  source_id: uuid
+  sequence: integer
+  text: string
+  heading_path: optional string
+  token_count: optional integer
+  embedding_ref: optional string
+  metadata: object
+```
+
+Requirements:
+- preserve provenance,
+- preserve chunk order,
+- permit re-indexing,
+- never execute imported content.
+
+## 25. Retrieval Record
+
+Every turn should record which memories or knowledge chunks were supplied to the narrator.
+
+```yaml
+retrieval_record:
+  id: uuid
+  turn_id: uuid
+  source_kind: memory | knowledge | summary | fact
+  source_id: uuid
+  retrieval_method: lexical | semantic | hybrid | forced
+  score: optional number
+  rank: integer
+```
+
+This lets the prompt inspector answer:
+
+> Why did the narrator know this?
+
+## 26. Prompt Snapshot
+
+```yaml
+prompt_snapshot:
+  id: uuid
+  turn_id: uuid
+  created_at: timestamp
+  system_instructions: text
+  campaign_context: text_or_structured
+  state_context: text_or_structured
+  summary_context: text_or_structured
+  memory_context: text_or_structured
+  knowledge_context: text_or_structured
+  recent_history: text_or_structured
+  user_input: text
+  token_accounting: object
+```
+
+The physical representation may be compressed or normalized. Reproducibility and inspection are the requirements.
+
+## 27. Media Job
+
+Not required for v1 behavior, but the data model should not prevent it.
+
+```yaml
+media_job:
+  id: uuid
+  campaign_id: uuid
+  scene_id: optional uuid
+  source_turn_start_id: optional uuid
+  source_turn_end_id: optional uuid
+  type: image | video | audio
+  provider: string
+  model: string
+  status: queued | running | completed | failed
+  request_payload: object
+  created_at: timestamp
+  completed_at: optional timestamp
+```
+
+## 28. Media Asset
+
+```yaml
+media_asset:
+  id: uuid
+  campaign_id: uuid
+  media_job_id: optional uuid
+  scene_id: optional uuid
+  type: image | video | audio
+  file_path: string
+  metadata: object
+  created_at: timestamp
+```
+
+Potential metadata includes prompt, seed, model, workflow, dimensions, duration, character references, and source turn range.
+
+## 29. Export Package
+
+A campaign export should be capable of preserving:
+
+- campaign configuration,
+- branches,
+- turn graph,
+- checkpoints,
+- state/events,
+- entities,
+- facts,
+- relationships,
+- story threads,
+- summaries,
+- memories,
+- scene snapshots,
+- knowledge-source metadata,
+- knowledge chunks/source files if selected,
+- prompt provenance if selected,
+- media metadata,
+- media files if selected.
+
+Exact format remains open. A ZIP containing a database plus manifest is a strong candidate.
+
+## 30. Deletion vs Archival
+
+The system must distinguish:
+- archive,
+- detach/disable,
+- permanent delete.
+
+Retry, Undo, Restore, and branch switching must not silently perform permanent deletion.
+
+## 31. Authoritative vs Derived Data
+
+### Authoritative
+- accepted turns,
+- branch lineage,
+- campaign configuration,
+- accepted facts,
+- accepted relationships,
+- accepted state events,
+- checkpoints.
+
+### Derived
+- summaries,
+- embeddings,
+- semantic indexes,
+- lexical indexes,
+- some automatically produced scene descriptions.
+
+Derived data should be rebuildable where practical.
+
+## 32. Provenance
+
+Important information should answer:
+
+> Where did this come from?
+
+Potential provenance:
+- campaign setup,
+- manual user edit,
+- accepted narrator turn,
+- state extraction,
+- imported canon,
+- imported reference,
+- imported inspiration,
+- derived inference.
+
+## 33. Open Questions for Phase 0B
+
+1. Does AI-DnD already model alternate takes separately from branch nodes in a reusable way?
+2. Can its state snapshots hold generic narrative JSON without major redesign?
+3. Is its branch lineage compatible with immutable turns?
+4. Should checkpoints be branch-independent pointers to turns?
+5. Should memories be physically branch-scoped or lineage-filtered at query time?
+6. Should knowledge sources be reusable across campaigns in v1?
+7. Should media tables physically exist in v1 or only interfaces/types?
+8. How should manual edits to canon/state be versioned?
+9. Which state needs full historical reconstruction versus only current-state storage?
+10. Can ai-adventure's event/replay discipline be adopted without overcomplicating AI-DnD?
+
+## 34. Acceptance Criteria
+
+The final v1 data model must support all of these without destructive hacks:
+
+- close/restart/resume exact story,
+- branch from an earlier turn while retaining the original future,
+- create and restore named checkpoints,
+- know current characters/locations/relationships/story threads,
+- reconstruct earlier authoritative state,
+- retrieve old events outside the active context window,
+- identify which imported passages informed a turn,
+- reconstruct what was sent to Ollama,
+- run fantasy and science-fiction campaigns without schema changes,
+- attach future image/video assets to scenes or turn ranges.
+
+## 35. Current Recommendation
+
+The target conceptual model should be:
+
+```text
+Immutable Turn Graph
+        |
+        +--> validated state events
+        |
+        +--> state snapshot/cache
+        |
+        +--> scene snapshot
+        |
+        +--> prompt/retrieval provenance
+```
+
+This combines the strongest observed concepts from:
+
+- AI-DnD's story tree and snapshots,
+- ai-adventure's append-only event/replay discipline,
+- Open Dungeon's simple story UX and visual continuity.
+
+The physical implementation remains provisional until Phase 0B validates the preferred production base.