feat: v0.6.0 — collapse policy, session limiter, Claude Code provider, prune, browser auth, media downloads

fix: persist created_at in cache so Joplin note titles get date prefix
mark_exported() was discarding created_at from the metadata dict because it wasn't in the hardcoded stored-key list, so the joplin sync always saw an empty date and omitted the prefix. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 18:26:14 -04:00 · 2026-05-05 11:36:21 -04:00 · 2026-05-05 11:05:39 -04:00 · 2026-05-05 09:25:55 -04:00 · 2026-05-04 23:17:18 -04:00 · 2026-04-19 22:32:01 -04:00
32 changed files with 8059 additions and 321 deletions
@@ -6,9 +6,19 @@
 # --- ChatGPT ---
 # How to get: open chatgpt.com in Chrome → F12 → Application tab
-# → Cookies → https://chatgpt.com → find "__Secure-next-auth.session-token" → copy Value
+# → Cookies → https://chatgpt.com → find the two cookie chunks:
-# Token type: JWT (starts with "eyJ"). Typically valid for ~7 days.
+#   __Secure-next-auth.session-token.0  (starts with "eyJ") → CHATGPT_SESSION_TOKEN
 #   __Secure-next-auth.session-token.1  (the remainder)     → CHATGPT_SESSION_TOKEN_1
 # Token type: JWE. Typically valid for ~7 days.
 CHATGPT_SESSION_TOKEN=
 CHATGPT_SESSION_TOKEN_1=
 # ChatGPT Projects (optional): comma-separated list of project gizmo IDs.
 # Project conversations are NOT included in the default /conversations listing.
 # How to find: open chatgpt.com → click a Project → look at the browser URL:
 #   https://chatgpt.com/g/g-p-<ID>-<slug>/project  →  copy "g-p-<ID>"
 # Example: CHATGPT_PROJECT_IDS=g-p-68c2b2b3037c8191890036fb4ae3ed9f,g-p-anotherproject
 CHATGPT_PROJECT_IDS=
 # --- Claude ---
 # How to get: open claude.ai in Chrome → F12 → Application tab
@@ -26,10 +36,47 @@ EXPORT_DIR=./exports
 #   provider/year          → exports/claude/2024/file.md (ignores projects)
 OUTPUT_STRUCTURE=provider/project/year
 # What to do with content that was invisible in the provider's web UI
 # (file-retrieval tool dumps, hidden context like Custom Instructions).
 # These dumps can be 90% of a conversation's bytes. Options:
 #   placeholder  (default) → one-line placeholder with tool name and size
 #   full                   → keep everything (pre-v0.6.0 behavior)
 #   omit                   → drop entirely (still counted in the run summary)
 EXPORTER_HIDDEN_CONTENT=placeholder
 # Download conversation assets (images, audio) next to the Markdown, into a
 # media/ folder, and inline them. Options:
 #   images  (default) → images only
 #   all               → also audio/voice clips and other files
 #   off               → keep text placeholders, download nothing
 # Downloaded media is uploaded to Joplin as resources on the next `joplin` run.
 EXPORTER_DOWNLOAD_MEDIA=images
 # Cap how many conversations are downloaded per export run (per provider).
 # Runs are resumable — a capped run continues where it stopped next time.
 # Keeps big backfills from looking like scraper traffic. Unset = unlimited.
 #MAX_CONVERSATIONS_PER_RUN=25
 # Seconds between consecutive API requests (small random jitter is added).
 # Default 1.0; set 0 to disable pacing.
 #REQUEST_DELAY=1.0
 # --- Joplin ---
 # Automate importing exported conversations into Joplin as notes.
 # Requires Joplin desktop running with the Web Clipper service enabled.
 # How to get the token:
 #   Joplin → Tools → Options → Web Clipper → copy "Authorization token"
 JOPLIN_API_TOKEN=
 # API URL (default port is 41184; change only if you've customised it)
 JOPLIN_API_URL=http://localhost:41184
 # Request timeout in seconds (default: 30). Increase if Joplin times out on
 # large conversations. Example: JOPLIN_REQUEST_TIMEOUT=60
 # JOPLIN_REQUEST_TIMEOUT=30
 # --- Cache ---
-# Where the sync manifest and logs are stored (default: ~/.ai-chat-exporter)
+# Where the sync manifest is stored (default: ./cache, inside the install directory)
-CACHE_DIR=~/.ai-chat-exporter
+CACHE_DIR=./cache
 # --- Logging ---
 # Log file path. Set to "none" to disable file logging.
-LOG_FILE=~/.ai-chat-exporter/logs/exporter.log
+LOG_FILE=./cache/logs/exporter.log
@@ -25,10 +25,14 @@ exports/
 !CHANGELOG.md
 # Cache and logs
 cache/
 .ai-chat-exporter/
 logs/
 *.log
 # Test tracking
 test-plan.csv
 # Editor / OS
 .DS_Store
 .idea/
@@ -36,3 +40,6 @@ logs/
 *.swp
 *.swo
 Thumbs.db
 # HTTP traffic captures — may contain auth cookies and session tokens
 *.har
@@ -3,9 +3,66 @@
 All notable changes to this project will be documented here.
 Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
-## [0.1.0] - Unreleased
+## [0.6.0] - 2026-06-12
 First tagged release. The project was developed through several internal
 milestones (0.1.0–0.5.0) that were never published; their changes are
 consolidated here.
 ### Added
- Initial implementation: ChatGPT and Claude export via internal web APIs
+
- Markdown and JSON exporters
+**Core export & sync**
- Local cache/manifest for incremental sync
+- ChatGPT and Claude export via their internal web APIs, with a local cache/manifest for incremental, resumable sync (only new or updated conversations are fetched; each is written to the manifest immediately).
- CLI with export, list, cache, doctor, and auth commands
+- Markdown and JSON exporters. Markdown rendering happens at exporter-write time (providers produce typed blocks; exporters render them), which keeps the door open for future Obsidian/HTML output.
 - CLI: `export`, `list`, `cache`, `doctor`, `auth`, `joplin`, `prune`.
 - `--project` filter on `export`/`list`/`joplin` (case-insensitive substring, or `none`).
 - ChatGPT Projects support via `CHATGPT_PROJECT_IDS` (project conversations are fetched separately from the default listing).
 **Joplin integration**
 - `joplin` command syncs exported Markdown to Joplin as notes; notebooks are created automatically and nested per provider+project. Re-running is safe — the Joplin note ID is stored in the manifest, so notes are updated, not duplicated.
 - `JOPLIN_API_TOKEN`, `JOPLIN_API_URL`, `JOPLIN_REQUEST_TIMEOUT` config, with actionable error messages on timeout/connection failure.
 **Rich content (typed blocks)**
 - Messages carry an ordered `blocks` list: text, code, thinking, tool_use, tool_result, citation, image_placeholder, file_placeholder, unknown.
 - ChatGPT voice mode: `audio_transcription` parts render as text; audio asset pointers render as `📎 File attached` placeholders with size/duration. Custom Instructions (`user_editable_context`/`model_editable_context`) now appear (were silently dropped) with a `> ℹ️ Hidden context` marker.
 - ChatGPT `execution_output`, `system_error`, and `tether_browsing_display` render as `tool_result` blocks (with `tool_name`, `is_error`, optional `summary`); transient browse spinners skip silently.
 - Defensive Claude block extraction (text/thinking/tool_use/tool_result/image, including nested-block flattening).
 - `_safe_fence` picks a backtick fence longer than any run in the content, so embedded triple-backticks can't corrupt rendering (verified live in Joplin).
 - Data-loss visibility: a `LossReport` summary at the end of every `export` run breaks down `unknown blocks` and `extraction failures` by raw type; `unknown` blocks render as visible `> ⚠️ Unsupported content` with the raw type and observed keys.
 **Invisible-content collapse (`EXPORTER_HIDDEN_CONTENT=full|placeholder|omit`, default `placeholder`; `export --hidden-content`)**
 - Messages that were invisible in the ChatGPT web UI are collapsed to one-line placeholders instead of exported in full. Two triggers: (a) tool-role retrieval dumps identified by `author.name` (`file_search`, `myfiles_browser`) — ChatGPT re-injects the full text of attached/project files on every tool run, measured at 86% of all content bytes across the three largest conversations and **not** hidden-flagged; (b) messages flagged `is_visually_hidden_from_conversation` (Custom Instructions). Code-execution results, web-search output, and all dialogue are untouched.
 - New `collapsed` block renders as `> 🔧 **Tool output** — file_search (15.0 KB) — omitted (EXPORTER_HIDDEN_CONTENT=full to keep)` (ℹ️ Hidden context variant for hidden-flagged messages). The `LossReport` gains a `collapsed by policy` section (count per origin + total KB) so the omission stays visible.
 **Claude Code session provider (`--provider claude-code`)**
 - Archives local agent transcripts from `~/.claude/projects/*/*.jsonl` (override with `CLAUDE_CODE_DIR`) — no tokens, no rate limits, no ToS exposure. Prose-only by default per the hidden-content policy: dialogue and deliverables kept; tool traffic grouped into one placeholder per activity run (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`); thinking dropped (counted in the summary); subagent (`isSidechain`) and harness (`isMeta`, snapshots, command tags) records stripped. Titles from the last `ai-title` record; project from the working-directory basename. Joplin notebooks nest under the `AI-Claude` parent. Measured: 19MB of JSONL → 804KB of Markdown across 25 sessions.
 **Binary/image downloads (`EXPORTER_DOWNLOAD_MEDIA=images|all|off`, default `images`; `export --download-media`)**
 - Downloads ChatGPT conversation assets into a `media/` folder beside each export and inlines them — images become real `![](media/…)` embeds, files become local links. Two-hop fetch via `/backend-api/files/{id}/download` → signed URL (verified live). `all` also pulls voice-mode audio (≈162MB of clips in this archive, transcripts already in text — hence the images-only default). Idempotent (skips assets already on disk), atomic writes, `600` perms. Expired/missing assets (old generated images 404) keep their placeholder and are counted as `media failed` — never fatal.
 - Joplin resource upload: the `joplin` command uploads downloaded media as resources and rewrites `media/…` links to `:/resourceId`, so images render inside notes (verified live: resources created with correct mime/size and linked to the note). Resource IDs are tracked per-conversation in the manifest, so re-syncs reuse them instead of duplicating.
 **Token setup & throughput**
 - `auth --from-browser [brave|chrome|chromium|edge|firefox]`: extracts ChatGPT session-token cookies and the Claude `sessionKey` straight from a local browser's cookie store (via `browser-cookie3`; defaults to Brave), validates each against the live API, and writes `.env`. Tokens that fail validation never overwrite a working entry; clear fallback when the browser is absent or the cookie DB is locked.
 - Session limiter: `--max-conversations N` / `MAX_CONVERSATIONS_PER_RUN` cap downloads per run (per provider); capped-out conversations are reported as "deferred" and resumed next run.
 - Polite pacing: `REQUEST_DELAY` (default 1.0s, ±25% jitter, `0` disables) between consecutive API requests, with the existing 429 backoff as the reactive net.
 **Archive hygiene**
 - `prune` deletes export files not referenced by the manifest (old-layout trees, no-ID orphans), with listing, confirmation, `--dry-run`, `-y`, and empty-directory sweep. Refuses to run when the manifest references no files (so `cache --clear` + `prune` can't wipe the archive). First live run removed 420 stale files (9.4 MB).
 - `doctor` verifies manifest ↔ disk integrity: every recorded `file_path` must exist on disk.
 ### Changed
 - ChatGPT role filter that dropped `tool`/`system` messages is **lifted**; all roles route through normal extraction (truly empty messages skip via the empty-content guard).
 - `BaseProvider.normalize_conversation` accepts an optional `LossReport` parameter.
 - `ChatGPTProvider.normalize_conversation` reads `conversation_id` as a fallback for `id` (live detail responses use `conversation_id`; fixtures use `id`).
 - `Config` validates `EXPORTER_HIDDEN_CONTENT`, `EXPORTER_DOWNLOAD_MEDIA`, `MAX_CONVERSATIONS_PER_RUN`, and `REQUEST_DELAY`, and logs the active values at startup.
 - New dependency: `browser-cookie3==0.20.1`.
 ### Fixed
 - Custom Instructions (`user_editable_context`/`model_editable_context`) were silently dropped from every conversation (parts-vs-direct-fields mismatch).
 ### Migration
 - JSON exports: messages contain typed `blocks` and may omit the legacy `content` field — external consumers should prefer `blocks`.
 - To re-render existing exports with the current rendering and collapse policy: `python -m src.main cache --clear` then `python -m src.main export`, followed by `joplin` to update notes (and upload media resources). Per-conversation message counts may increase as previously-dropped Custom Instructions, image-only turns, and tool-only turns now appear.
 ### Test suite
 - 264 tests, all passing.
@@ -1,9 +1,308 @@
 # Planned Future Work
-These items are explicitly out of scope for v0.1.0 but have been designed for.
+Items completed in each release are moved to the changelog. Items here are
-The codebase is structured to make each of these additions straightforward.
+designed for but not yet implemented. The codebase is structured to make each
 of these additions straightforward.
 **Completed:**
 - v0.1.0 — Core export: ChatGPT + Claude, incremental sync, Markdown + JSON output
 - v0.2.0 — Joplin import automation (`joplin` command, create/update notes, notebook auto-creation)
 - v0.4.0 — Rich content support: typed message blocks (text, code, thinking, tool_use, tool_result, image_placeholder, file_placeholder, unknown); ChatGPT voice transcripts as text + audio placeholders; Custom Instructions extraction; data-loss visibility via `LossReport` summary and visible `unknown` blocks
 - v0.5.0 — Nested Joplin notebooks, date-prefixed note titles, flat year folders
 - v0.6.0 — Collapse tool retrieval dumps & hidden context (`EXPORTER_HIDDEN_CONTENT` policy; roadmap item 1); session limiter + request pacing (`MAX_CONVERSATIONS_PER_RUN`, `REQUEST_DELAY`; roadmap item 6)
 ---
 # Roadmap (decided 2026-06-12)
 Priorities reflect the tool's primary purpose — a trustworthy backup so that
 conversation data is not lost if a provider account is ever closed — plus the
 day-to-day friction of weekly ChatGPT token refresh, and the long-term goal
 of running headless as a StartOS service.
 **Now (in order):**
 1. ~~Collapse tool retrieval dumps & hidden context~~ — **shipped in v0.6.0**
   (full-archive re-export still pending; see entry below)
 2. ~~Brave cookie auto-extraction~~ — **shipped in v0.6.0** (note: runs on
   whichever machine hosts the browser — Brave is not on this Linux box)
 3. ~~Claude Code session provider~~ — **shipped in v0.6.0**
 4. ~~Archive hygiene — `prune` + `doctor` integrity check~~ — **shipped in v0.6.0**
 **Soon, but later:**
 5. ~~Binary content downloads~~ — **shipped in v0.6.0**
 6. ~~Per-session download limiter + polite pacing~~ — **shipped in v0.6.0**
 7. Scheduled / watch mode → StartOS service packaging (long-term destination)
 **Deprioritized** (entries kept at the bottom of this file; revisit on
 demand): `--force` flags, per-conversation cache reset, official export-ZIP
 fallback, o1/o3 reasoning reclassification, Obsidian output, token expiry
 notifications (largely superseded by cookie auto-extraction), search command.
 Additional web providers (Gemini/Grok/Perplexity) are explicitly out of
 scope — no significant usage to archive.
 ---
 ## 1. Collapse Tool Retrieval Dumps & Hidden Context — SHIPPED v0.6.0
 **Implemented 2026-06-12** as designed below, with one scoping correction
 from live recon: the retrieval dumps are NOT flagged
 `is_visually_hidden_from_conversation` — `author.name == "file_search"` is
 the discriminator (the hidden flag only marks Custom Instructions and small
 system stubs). Verified live: worst files shrink 93% (524KB → 36KB).
 Remaining step: full-archive re-export (`cache --clear` + `export`) +
 Joplin re-sync, at the user's chosen time.
 **Problem (measured 2026-06-12 against a full fresh export):** 45% of the
 entire 11.2MB archive (260 files) is tool-role messages; 29 files are
 majority tool-dump. Worst case: a 524KB conversation where 495KB (94%, 64
 messages) is ChatGPT's file-retrieval tool re-injecting the full text of
 the user's own attached documents, the same files dumped dozens of times
 per conversation. These messages were invisible in the ChatGPT web UI;
 they appear in exports because v0.4.0 lifted the role filter to fix silent
 data loss. Custom Instructions hidden-context blocks are a minor secondary
 case (~2KB, once per conversation) — the originally planned
 `EXPORTER_INCLUDE_HIDDEN_CONTEXT` toggle alone would not help: the worst
 file contains zero hidden-context blocks.
 **Fix: collapse, don't drop** (consistent with the no-silent-drop rule):
 - `EXPORTER_HIDDEN_CONTENT=full|placeholder|omit` env var, default
  `placeholder`, plus a `--hidden-content` CLI override on `export`.
 - `placeholder` renders affected messages as one line with type and size:
  `> 🔧 Tool output (file_search, 24KB) — omitted
  (EXPORTER_HIDDEN_CONTENT=full to keep)`. Expected effect: archive
  roughly halves; worst files shrink ~90%.
 - **Scope — collapse:** (a) tool-role retrieval dumps, identified by raw
  `author.name` (`file_search`, `myfiles_browser`, …) in the API response
  (the rendered Markdown only shows a generic "🔧 Tool" label, so the
  decision must happen in the provider, not the renderer); (b) messages
  flagged `is_visually_hidden_from_conversation`, including
  `user_editable_context` / `model_editable_context` (Custom
  Instructions) — subsumes the old suppress-hidden-context idea.
 - **Scope — keep at full size:** code-execution `tool_result` blocks and
  web-search results; those are usually content the user wants.
 - Count collapsed messages in the post-export summary so the omission
  stays visible (mirror the LossReport presentation, but as intentional
  policy, not loss).
 Re-export workflow after shipping: `cache --clear` + `export` (same as
 the v0.4.0 migration).
 ## 2. Brave Cookie Auto-Extraction — SHIPPED v0.6.0
 **Implemented 2026-06-12**: `auth --from-browser [browser]`
 (`src/browser_tokens.py`, browser-cookie3 dependency, defaults to brave;
 chrome/chromium/edge/firefox also supported). Extracted tokens are validated
 against the live API before `.env` is touched. **Discovered during
 implementation: Brave is not installed on this machine** — the logged-in
 browser lives elsewhere, so the flag only helps when the exporter runs on
 that machine (verified the graceful-failure path here; the happy path is
 covered by mocked tests). This strengthens the case for the StartOS
 token-push mechanism (entry 8).
 Pull session tokens directly from the Brave browser profile instead of the
 weekly manual DevTools dance. `auth` gains an "extract from browser" path
 (with the manual flow kept as fallback); a later iteration could let
 `doctor` or a 401 handler suggest/perform a re-extract automatically.
 - Brave on Linux follows the Chromium pattern: cookies SQLite at
  `~/.config/BraveSoftware/Brave-Browser/Default/Cookies`, values
  AES-128-CBC encrypted with a key derived from the OS keyring secret
  ("Brave Safe Storage" via SecretService/kwallet, or the `peanuts` v10
  fallback). Well-trodden territory — evaluate a small dependency
  (`browser_cookie3`-style) vs. a focused in-repo implementation.
 - Extract `__Secure-next-auth.session-token.0` / `.1` from `chatgpt.com`
  and `sessionKey` from `claude.ai`; write to `.env` through the existing
  auth-wizard write path (tokens never echoed).
 - Brave must be the user's logged-in browser (it is); detect a locked /
  unreadable cookie DB (Brave running with the DB exclusively locked) and
  fall back to the manual flow with a clear message.
 - Does **not** solve headless/StartOS — no browser on the server. That
  needs a token-push mechanism, tracked under the StartOS entry.
 ## 3. Claude Code Session Provider — SHIPPED v0.6.0
 **Implemented 2026-06-12** as designed below (`src/providers/claude_code.py`,
 `--provider claude-code`, `CLAUDE_CODE_DIR` override). Additional findings
 during implementation: `isSidechain` records are subagent transcripts (skipped),
 `isMeta` marks harness-generated user records (skipped), and listing/normalized
 `updated_at` must both use file mtime or the cache would re-export every
 session every run. First export: 25 sessions, 19MB JSONL → 804KB Markdown.
 Archive local Claude Code session transcripts. No tokens, no rate limits,
 no ToS risk — the data is already on disk but lives in a single JSONL per
 session that Claude Code may clean up, and it contains deliverables
 (reviews, plans, analyses) that exist nowhere else.
 Decisions (2026-06-12):
 - **Rendering: prose-only.** Keep user prompts and assistant text
  (including full deliverable write-ups); collapse tool activity to
  one-line placeholders (`> 🔧 Tool activity — 14 calls (Read ×9, Bash ×2),
  86KB — omitted`); **exclude thinking blocks**.
 - **Joplin: sync enabled.** Each coding project becomes a notebook nested
  under an **"AI-Claude"** parent notebook (nested-notebook support shipped
  in v0.5.0).
 Data facts (measured 2026-06-12):
 - Source: `~/.claude/projects/<munged-cwd>/<session-uuid>.jsonl`.
  Currently 29 sessions, 18.8MB total, largest 3.8MB.
 - Representative 2.1MB session: tool_result 436KB, tool_use 122KB,
  thinking 104KB, dialogue prose only ~24KB (~4%) — collapsing tool
  activity is what makes these exports readable.
 - Record types: `user` / `assistant` (Anthropic-style `message.content`
  block arrays) plus harness records: `ai-title` (use for note title and
  filename slug), `last-prompt`, `file-history-snapshot`, `attachment`,
  `permission-mode`, `system` (skip). Strip harness noise from user
  messages (`<local-command-caveat>`, `<command-name>` blocks).
 Implementation shape: new `src/providers/claude_code.py` implementing the
 `BaseProvider` interface — `list_conversations` scans project dirs,
 `get_conversation` parses the JSONL, `normalize_conversation` maps onto the
 existing block schema (content is already block-shaped: text / tool_use /
 tool_result / thinking). Incremental sync via file mtime/size recorded in
 the existing manifest. Project name derives from the munged cwd dirname.
 ## 4. Archive Hygiene: `prune` Command + Manifest Integrity — SHIPPED v0.6.0
 **Implemented 2026-06-12** as designed below, plus an empty-manifest guard
 (refuses to prune right after `cache --clear`). First live run removed 420
 stale files (9.4 MB, old-layout trees + `_.md` orphans); doctor now reports
 manifest↔disk integrity (293/293 after the run).
 A backup is only trustworthy if the on-disk tree matches the manifest.
 Observed 2026-06-12: pre-v0.5.0 layout trees (`tspc-expertcouncil/2025/`)
 and `_.md` no-ID orphans (from the empty-conversation-id bug fixed in
 v0.4.1) sit alongside current exports and would double-sync into Joplin.
 - `prune` command: delete export files not referenced by the manifest.
  `--dry-run` (default off, but always print the list before deleting)
  shows what would be removed and why (old layout / orphan / unknown).
 - `doctor` extension: verify every manifest entry's `file_path` exists on
  disk; report missing files (re-export candidates) and unreferenced files
  (prune candidates).
 ## 5. Binary Content Downloads — SHIPPED v0.6.0
 **Implemented 2026-06-12** (`src/media.py`, `EXPORTER_DOWNLOAD_MEDIA`,
 `download_asset`/`parse_asset_file_id` on the ChatGPT provider, Joplin
 `create_resource` + `upload_media_and_rewrite`). Live recon settled the
 download mechanism: `GET /backend-api/files/{id}/download` returns a signed
 `download_url`; a second GET yields the bytes (works for user uploads;
 older AI-generated images 404 — expired server-side, handled gracefully).
 Asset refs come in three shapes — `sediment://file_…`,
 `sediment://<hash>#file_…#p_N.png` (generated), `file-service://…`. Archive
 scan: 14 images (4 uploads / 10 generated) + 556 audio clips ≈162MB, so
 images-only is the default and audio is opt-in via `all`.
 Original notes below.
 **Priority note (2026-06-12): "later, but soon" — under the
 backup-if-account-closes goal, embedded images are part of the data that
 would be lost; placeholders alone don't preserve them.**
 v0.4.0 ships placeholders for images and audio assets but does not download
 the binary content. The `_safe_fence`-wrapped placeholders include the asset
 reference (`sediment://...` or `file-service://...`), MIME type, size, and
 duration where available; the actual bytes are not preserved.
 Next steps:
 - Download attached images alongside the Markdown export, save under a
  `media/` sibling directory with a stable filename derived from the asset
  reference.
 - Replace `image_placeholder` rendering with an inline `![](relative/path)`
  reference once the file is on disk.
 - Joplin integration: upload binaries as Joplin resources via `POST /resources`,
  rewrite the rendered Markdown to use `:/resourceId` references, and track
  the resource ID in the cache manifest so re-syncs stay idempotent.
 - DALL-E images on the assistant side: not observed in this user's data; the
  code path exists (`source = "model_generated"`) but is untested.
 The block-level schema is already in place — only the file-fetch + rewrite
 layer needs to be added. See the `image_placeholder` and `file_placeholder`
 block definitions in `src/blocks.py`.
 ## 6. Per-Session Download Limiter — SHIPPED v0.6.0
 **Implemented 2026-06-12** as designed below: `--max-conversations N` /
 `MAX_CONVERSATIONS_PER_RUN` session cap with deferred-count reporting, and
 `REQUEST_DELAY` pacing (default 1.0s ±25% jitter) in `BaseProvider._request`.
 Verified live: a 2-pending run with cap 1 exported one, deferred one, and
 the re-run picked it up.
 Cap how many conversations are downloaded in a single `export` run so the
 tool never hammers the ChatGPT/Claude internal APIs with a large burst —
 most importantly on the very first export, which otherwise fetches the
 entire conversation history in one session. Because every run is resumable
 (the manifest records each conversation immediately), a capped run simply
 exports the first N pending conversations and the next run picks up where
 it left off. This keeps traffic looking like a human-paced session rather
 than a scraper, reducing the risk of rate limiting or account flags.
 Two complementary pieces:
 1. **Session cap** — `--max-conversations N` flag (and
   `MAX_CONVERSATIONS_PER_RUN` env default). Implementation: in the
   `export` command, slice the pending list after the cache filter:
   `to_export = to_export[:n]`. On exit, print exported-vs-remaining
   counts (reuse the message format from the 429 early-exit path) and
   remind the user to re-run to continue.
 2. **Polite pacing** — `REQUEST_DELAY` env var (seconds, with small
   random jitter) slept between per-conversation detail fetches in
   `BaseProvider`, so even a capped run doesn't fire requests
   back-to-back. The existing 429 backoff in `_request` stays as the
   reactive safety net.
 Note: the conversation *listing* (paginated, 100/page) still runs in full
 each time so the cache comparison works — the cap applies to the heavy
 per-conversation detail fetches, which dominate request volume.
 Together with watch mode, this is a stepping stone to the StartOS service:
 a scheduled, capped, politely-paced export is the traffic profile a
 headless deployment needs.
 ## 7. Scheduled / Watch Mode
 Add a `watch` command (or cron integration helper) to run exports automatically
 on a schedule:
 ```bash
 python -m src.main watch --interval 6h   # poll every 6 hours
 ```
 This would run `export` + `joplin` in sequence, then sleep. Alternatively,
 provide a `cron` command that prints the correct crontab line for the user's
 setup.
 Implementation: simple loop with `time.sleep()`, or emit a crontab entry
 string that calls the export and joplin commands in sequence. A `--once`
 flag would do a single run then exit (useful for cron itself).
 Stepping stone to the StartOS service (below).
 ## 8. StartOS Service Packaging (long-term destination)
 Run the exporter headless as a StartOS service: scheduled export + sync to
 a Joplin Server instance, so the backup happens without a desktop in the
 loop.
 - Building blocks needed first: watch mode (#7), session limiter (#6),
  and a Joplin Server (vs. desktop Web Clipper) sync target.
 - **Hard problem, flagged early:** session-token freshness without a
  browser. ChatGPT tokens last ~7 days; there is no browser on the server
  to re-extract from. Candidate mechanisms: a service config action the
  user pastes fresh tokens into periodically, or a future browser
  extension that pushes tokens to the server. Cookie auto-extraction (#2)
  only solves the desktop case.
 ---
 # Deprioritized
 Kept for reference; not on the active roadmap.
 ## Export `--force` Flag
 ## Export --force Flag (v0.1.x)
 Add `--force` to the `export` command to re-export already-cached conversations
 without permanently clearing the entire manifest. Useful for re-generating files
 after changing the Markdown template or output structure.
@@ -13,30 +312,25 @@ returns all conversations regardless of cache state when force is True.
 Current workaround: `python -m src.main cache --clear` then re-run export.
-## Joplin Integration (v0.2.0)
+## Joplin `--force` Flag
 Automate importing exported Markdown files into Joplin as new notes.
 Joplin exposes a local REST API (requires Joplin desktop running with Web Clipper enabled).
-Approach: after export, iterate exported files and POST each to
+Similarly, add `--force` to the `joplin` command to re-sync all cached
-`http://localhost:41184/notes` with the appropriate notebook ID.
+conversations to Joplin regardless of whether they've been synced before.
 Useful after making formatting changes to the Markdown exporter.
-The output folder structure maps directly to Joplin notebooks:
+Implementation: in `get_joplin_pending()`, return all entries that have a
- exports/chatgpt/my-project/    → Joplin notebook "ChatGPT - My Project"
+`file_path` when `force=True`, ignoring `joplin_synced_at`.
 - exports/claude/my-project/     → Joplin notebook "Claude - My Project"
 - exports/chatgpt/no-project/    → Joplin notebook "ChatGPT - No Project"
 - exports/claude/no-project/     → Joplin notebook "Claude - No Project"
-Prerequisites:
+## Per-Conversation Cache Reset
 - Joplin desktop must be running with Web Clipper enabled
 - `JOPLIN_API_TOKEN` env var (get from Joplin → Tools → Web Clipper Options)
 - The Joplin import script will need to create notebooks if they don't exist,
  then POST each note into the correct notebook
-Note: The default OUTPUT_STRUCTURE of provider/project/year is assumed when
+Add `cache --reset --conversation <id>` to force re-export or re-sync of a
-implementing the import script. If the user has changed OUTPUT_STRUCTURE,
+single conversation without clearing the entire provider cache.
-the import script will need updating accordingly.
+
 Current workaround: manually edit `~/.ai-chat-exporter/manifest.json` and
 delete the entry, then re-run export.
 ## Official API Fallback
 ## Official API Migration (v0.3.0)
 If the unofficial internal web API approach breaks, migrate to official export
 file parsing as a fallback:
 - ChatGPT: parse `conversations.json` from Settings → Export Data
@@ -44,29 +338,61 @@ file parsing as a fallback:
 The `BaseProvider` abstract class is intentionally designed so that a
 `FileProvider` subclass can implement the same interface
-(list_conversations, get_conversation, normalize_conversation)
+(`list_conversations`, `get_conversation`, `normalize_conversation`)
 without any changes to cache, exporters, or CLI code.
 To add this: implement `src/providers/file_chatgpt.py` and
 `src/providers/file_claude.py`, then add `--input-file` flag to the
 export command to accept a pre-downloaded export ZIP or JSON.
-## Rich Content Support (v0.4.0)
+Deprioritized 2026-06-12: the official ChatGPT export does not cover what
-Currently only text content is exported. Future versions should handle:
+this user needs (project data), so it isn't a real fallback here.
-### Claude
+## Reclassify o1/o3 Reasoning Subparts
 - Artifacts (code, documents, HTML) — export as separate files, link from Markdown
 - Uploaded images — download and embed or link
 - Extended thinking/reasoning blocks — include as collapsible sections
 - Tool call results and web search citations — include as footnotes or appendices
-### ChatGPT
+v0.4.0 leaves dict parts inside `text` content_type messages with shape
- DALL-E generated images — download and embed or link
+`{"summary": ..., "content": ...}` rendered as plain text (defensive — the
- Code Interpreter outputs — export code and results
+shape was inferred from a code comment, not captured live). Once a real
- File attachments — download and reference
+reasoning conversation is captured, reclassify these as `thinking` blocks.
 - Voice transcripts — include as text
-Implementation note: the normalized message schema already includes a
+## Obsidian Vault Output
-`content_type` field placeholder. When this work begins, extend the schema
+
-rather than replacing it. In v0.1.0, log a WARNING whenever non-text content
+Add an `obsidian` command (or `--target obsidian` flag) to sync exported
-is encountered so users know what was skipped.
+conversations into an Obsidian vault directory. The current Markdown format
 is already largely compatible; the main differences are:
 - Obsidian uses YAML frontmatter `properties` (same format, already supported)
 - Tags should use `#tag` inline or `tags:` list in frontmatter (already done)
 - Wikilinks (`[[Title]]`) instead of Markdown links — optional, Obsidian
  supports both
 Implementation: the existing `MarkdownExporter` output is already valid in
 Obsidian. An `ObsidianSyncer` class (mirroring `JoplinClient`) would simply
 copy files to the vault directory and maintain a flat or nested folder
 structure matching the user's Obsidian setup. No API needed — just file I/O.
 ## Token Expiry Notifications
 Proactively warn when a token is close to expiry (within 48h for ChatGPT),
 rather than only surfacing the warning at startup. Options:
 - Add an `expiry` subcommand that prints token status and exits non-zero if
  any token is expired or expiring soon (useful in scripts/cron)
 - Send a desktop notification via `notify-send` (Linux) or `osascript` (macOS)
  when a token is within 24h of expiry
 Largely superseded by Brave cookie auto-extraction (#2): once refresh is
 one command (or automatic), expiry warnings matter much less.
 ## Search Command
 Add a `search` command to full-text search across all exported Markdown files:
 ```bash
 python -m src.main search "kubernetes ingress"
 python -m src.main search "kubernetes ingress" --provider claude --project devops
 ```
 Implementation: `grep`/`ripgrep` over `EXPORT_DIR`, display results with
 conversation title, date, and a snippet. No index needed — Markdown files are
 small enough to grep directly.
@@ -0,0 +1,507 @@
 # AI Chat Exporter
 A personal backup tool for ChatGPT and Claude conversation history. Exports your chats to Markdown files and syncs them to [Joplin](https://joplinapp.org/) as notes. Each conversation becomes a single `.md` file with YAML frontmatter, organised into folders that map directly to Joplin notebooks.
 Supports incremental sync — only new or updated conversations are exported on each run. Every run is resumable: if interrupted, re-running picks up exactly where it left off.
 ---
 ## ⚠️ Terms of Service Warning
 **Read this before using this tool.**
 This tool works by accessing **unofficial, undocumented internal web API endpoints** used by the ChatGPT and Claude web apps. These endpoints are not publicly supported by OpenAI or Anthropic and are subject to change or removal without notice.
 **Use of this tool may conflict with their Terms of Service:**
 - OpenAI: https://openai.com/policies/terms-of-use
 - Anthropic: https://www.anthropic.com/legal/consumer-terms
 **By using this tool, you accept that:**
 - You are using it entirely at your own risk
 - Your account could potentially be suspended for automated or scripted access
 - The internal APIs this tool relies on may break at any time without notice
 - This tool is for **personal archival use only** — not commercial use
 This tool is designed for a single user backing up their own conversations. Do not use it to scrape data at scale or for any commercial purpose.
 ---
 ## Installation
 ### Linux / macOS
 ```bash
 git clone <repo-url>
 cd ai-chat-exporter
 python3 -m venv .venv
 source .venv/bin/activate
 pip install -e ".[dev]"
 ```
 ### Windows
 No admin access required. Run these in **Command Prompt** (`cmd.exe`) — it's the simplest option on Windows because it doesn't have PowerShell's script execution policy restrictions.
 ```bat
 git clone <repo-url>
 cd ai-chat-exporter
 python -m venv .venv
 .venv\Scripts\activate
 pip install -e ".[dev]"
 ```
 All `ai-chat-exporter` commands work identically in Command Prompt.
 **Using PowerShell instead?** If you prefer PowerShell, you may need to allow script execution first (one-time, current user only):
 ```powershell
 Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
 ```
 Then activate the venv and run commands the same way.
 **Prerequisites:**
 - Python 3.11 or later — install from [python.org](https://www.python.org/downloads/windows/). During installation, tick **"Add Python to PATH"**.
 - Git — install from [git-scm.com](https://git-scm.com/) if not already present.
 **Notes:**
 - The cache manifest and logs are stored in `cache\` inside the install directory — the same as on Linux.
 - File permission hardening (`chmod 600`) is silently ignored on Windows — not a concern for single-user desktop use.
 - Joplin Web Clipper runs on `localhost:41184` on all platforms; no configuration changes needed.
 ---
 ## First Run: Run Doctor
 Before anything else, validate your setup:
 ```bash
 ai-chat-exporter doctor
 ```
 This checks token presence, format, expiry, directory permissions, disk space, and live API connectivity. Fix any failures before proceeding.
 ---
 ## Getting Your Session Tokens
 Session tokens are how your browser stays logged in. This tool uses them to access your chat history on your behalf.
 ### The fast way: extract from your browser
 If the browser you're logged in with runs on the same machine as this tool:
 ```bash
 ai-chat-exporter auth --from-browser            # Brave (default)
 ai-chat-exporter auth --from-browser firefox    # or chrome, chromium, edge
 ```
 This reads the session cookies straight from the browser's cookie store (decrypted via the OS keyring/DPAPI/Keychain), validates each token against the live API, and writes `.env` — no DevTools, no copy-pasting. A token that fails validation never overwrites a working one. If extraction fails (browser not installed here, cookie DB locked, logged out), fall back to the manual flow below.
 ### Token Lifetimes
 | Provider | Cookie Name | Lifetime | Expiry Detection |
 |----------|-------------|----------|-----------------|
 | ChatGPT | `__Secure-next-auth.session-token.0` + `.1` | ~7 days | JWT `exp` claim (decoded automatically) |
 | Claude | `sessionKey` | ~30 days | Only detectable via 401 response |
 ### Finding Tokens in Chrome DevTools
 1. Open the provider's website and make sure you're logged in
 2. Press **F12** (Windows/Linux) or **Cmd+Option+I** (macOS) to open DevTools
 3. Click the **Application** tab
 4. In the left panel, expand **Cookies** and click the site URL
 5. Find the cookie by name and copy its **Value**
 **ChatGPT:** go to `https://chatgpt.com` → find **two** cookies:
 - `__Secure-next-auth.session-token.0` — copy Value (starts with `eyJ`) → `CHATGPT_SESSION_TOKEN`
 - `__Secure-next-auth.session-token.1` — copy Value → `CHATGPT_SESSION_TOKEN_1`
 ChatGPT splits large session tokens across two cookies to stay under the browser's 4KB cookie limit. Both are required.
 **Claude:** go to `https://claude.ai` → find `sessionKey` → copy Value
 ### When Tokens Expire
 When a token expires you'll see a `401 Unauthorized` error. To refresh:
 - Re-run the `auth` wizard: `ai-chat-exporter auth`
 - Or manually update the value in your `.env` file
 ---
 ## The `auth` Command
 The easiest way to configure tokens is the interactive wizard:
 ```bash
 ai-chat-exporter auth
 ```
 This walks you through finding your token, validates it, shows the expiry date (ChatGPT only), and offers to write it to your `.env` automatically. Tokens are never echoed to the terminal.
 ---
 ## `.env` Setup
 Copy `.env.example` to `.env` and fill in your values:
 ```bash
 cp .env.example .env
 ```
 ### Provider tokens
 | Variable | Description |
 |----------|-------------|
 | `CHATGPT_SESSION_TOKEN` | ChatGPT session token chunk `.0` (starts with `eyJ…`) |
 | `CHATGPT_SESSION_TOKEN_1` | ChatGPT session token chunk `.1` (the remainder) |
 | `CHATGPT_PROJECT_IDS` | Comma-separated ChatGPT project IDs (see below) |
 | `CLAUDE_SESSION_KEY` | Your Claude session key |
 ### Output
 | Variable | Default | Description |
 |----------|---------|-------------|
 | `EXPORT_DIR` | `./exports` | Where to write exported Markdown files |
 | `OUTPUT_STRUCTURE` | `provider/project/year` | Folder structure (see below) |
 | `EXPORTER_HIDDEN_CONTENT` | `placeholder` | What to do with content invisible in the provider's web UI (file-retrieval tool dumps, Custom Instructions): `placeholder` collapses to a one-line note with size, `full` keeps everything, `omit` drops it. Dumps can be 90% of a conversation's bytes. |
 | `EXPORTER_DOWNLOAD_MEDIA` | `images` | Download conversation assets into a `media/` folder beside each export and inline them: `images` (images only), `all` (also audio/voice clips), `off` (text placeholders only). Downloaded media is uploaded to Joplin as resources on the next `joplin` run. |
 | `MAX_CONVERSATIONS_PER_RUN` | unlimited | Cap downloads per export run (per provider). Runs are resumable, so re-running continues where the cap stopped — useful to spread a large first export across several sessions. |
 | `REQUEST_DELAY` | `1.0` | Seconds between consecutive API requests, with small random jitter, so traffic stays human-paced. Set `0` to disable. |
 ### Joplin
 | Variable | Default | Description |
 |----------|---------|-------------|
 | `JOPLIN_API_TOKEN` | — | Authorization token from Joplin Web Clipper settings |
 | `JOPLIN_API_URL` | `http://localhost:41184` | Joplin API URL (change only if you've customised the port) |
 | `JOPLIN_REQUEST_TIMEOUT` | `30` | Seconds before an API call times out. Increase for very large conversations. |
 ### Cache & logging
 | Variable | Default | Description |
 |----------|---------|-------------|
 | `CACHE_DIR` | `./cache` | Where to store the sync manifest |
 | `LOG_FILE` | `./cache/logs/exporter.log` | Log file path (`none` to disable) |
 ---
 ## ChatGPT Projects
 ChatGPT project conversations are stored separately from your main conversation list and require extra configuration.
 ### Finding your project IDs
 1. Open ChatGPT and click a Project in the left sidebar
 2. Look at the browser URL — it will look like:
   `https://chatgpt.com/g/g-p-68c2b2b3037c8191890036fb4ae3ed9f-my-project/project`
 3. Copy the `g-p-…` part (everything up to but not including the slug after the second `-`)
 Add all your project IDs to `.env` as a comma-separated list:
 ```
 CHATGPT_PROJECT_IDS=g-p-68c2b2b3037c8191890036fb4ae3ed9f,g-p-anotherprojectid
 ```
 The `auth` wizard can also guide you through this step interactively.
 ---
 ## Claude Code Sessions
 The `claude-code` provider archives your local [Claude Code](https://claude.com/claude-code) agent transcripts — no tokens, no API, no ToS exposure. Sessions are read from `~/.claude/projects/` (override with `CLAUDE_CODE_DIR` in `.env`).
 ```bash
 ai-chat-exporter export --provider claude-code
 ai-chat-exporter joplin --provider claude-code
 ```
 Exports are prose-only by default: your prompts and Claude's write-ups are kept, tool activity is grouped into one-line placeholders (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`), and internal reasoning is dropped (counted in the run summary). Set `EXPORTER_HIDDEN_CONTENT=full` to keep everything. Each coding project becomes a Joplin notebook under the `AI-Claude` parent. The provider is included in `--provider all` whenever the sessions directory exists.
 ---
 ## Output Structure
 All exported files go under `EXPORT_DIR`. The folder structure maps directly to Joplin notebooks.
 ### Default: `provider/project/year`
 ```
 exports/
 ├── chatgpt/
 │   ├── no-project/
 │   │   └── 2024/
 │   │       └── 2024-03-15_my-conversation_abc12345.md
 │   └── learning-python/
 │       └── 2024/
 │           └── 2024-03-15_async-tutorial_def67890.md
 └── claude/
    ├── no-project/
    │   └── 2024/
    │       └── 2024-06-01_docker-explained_ghi11111.md
    └── startos-packaging/
        └── 2024/
            └── 2024-06-10_manifest-setup_jkl22222.md
 ```
 ### Joplin Notebook Mapping
 Each provider+project combination maps to a flat Joplin notebook created automatically by the `joplin` command:
 | Export folder | Joplin notebook |
 |---------------|-----------------|
 | `exports/chatgpt/learning-python/` | `ChatGPT - Learning Python` |
 | `exports/claude/startos-packaging/` | `Claude - Startos Packaging` |
 | `exports/chatgpt/no-project/` | `ChatGPT - No Project` |
 | `exports/claude/no-project/` | `Claude - No Project` |
 ### Other `OUTPUT_STRUCTURE` options
 | Value | Result |
 |-------|--------|
 | `provider/project/year` (default) | `exports/claude/my-project/2024/file.md` |
 | `provider/project` | `exports/claude/my-project/file.md` |
 | `provider/year` | `exports/claude/2024/file.md` (projects ignored) |
 ### Filename format
 `YYYY-MM-DD_{title-slug}_{id[:8]}.md` — e.g. `2024-06-10_manifest-setup_jkl22222.md`
 ---
 ## CLI Reference
 ### Global flags
 ```
 --verbose / -v     DEBUG output to console
 --quiet / -q       WARNING and above only
 --debug            DEBUG + full tracebacks + redacted API response bodies
 --no-log-file      Disable file logging
 --version          Print version and exit
 ```
 ### `auth` — Interactive token setup
 ```bash
 ai-chat-exporter auth                  # manual wizard (DevTools flow)
 ai-chat-exporter auth --from-browser   # extract from Brave's cookie store
 ```
 Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut. `--from-browser [brave|chrome|chromium|edge|firefox]` skips the wizard and pulls tokens directly from a local browser, validating them live before writing `.env`.
 ### `doctor` — Health check
 ```bash
 ai-chat-exporter doctor
 ```
 Checks: token presence, JWT validity and expiry, directory permissions, disk space, live API reachability. Exits with code 0 if all pass, 1 if any fail.
 ### `export` — Export conversations
 ```bash
 # Export everything (new/updated only)
 ai-chat-exporter export
 # Single provider
 ai-chat-exporter export --provider claude
 # JSON output
 ai-chat-exporter export --format json
 # Both Markdown and JSON
 ai-chat-exporter export --format both
 # Only conversations updated since a date
 ai-chat-exporter export --since 2024-06-01
 # Only conversations in a specific project (case-insensitive substring)
 ai-chat-exporter export --project "learning python"
 # Only conversations outside any project
 ai-chat-exporter export --project none
 # Write to a custom directory
 ai-chat-exporter export --output /path/to/my/notes
 # Preview without writing anything
 ai-chat-exporter export --dry-run
 ```
 Options: `--provider [chatgpt|claude|claude-code|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--dry-run`
 ### `list` — List conversations
 ```bash
 # List all conversations for all providers
 ai-chat-exporter list
 # Single provider
 ai-chat-exporter list --provider chatgpt
 # Filter by project
 ai-chat-exporter list --project "learning python"
 # Only conversations outside any project
 ai-chat-exporter list --project none
 ```
 Fetches and displays all conversations without exporting them. Useful for verifying what the tool can see before running an export.
 ### `joplin` — Sync to Joplin
 ```bash
 # Sync all pending conversations to Joplin
 ai-chat-exporter joplin
 # Preview what would be synced without sending anything
 ai-chat-exporter joplin --dry-run
 # Sync a single provider
 ai-chat-exporter joplin --provider chatgpt
 # Sync only conversations in a specific project
 ai-chat-exporter joplin --project "learning python"
 # Sync only conversations outside any project
 ai-chat-exporter joplin --project none
 ```
 Reads the local export cache and pushes each exported Markdown file to Joplin as a note. Notebooks are created automatically. Re-running is safe — notes are updated (not duplicated).
 **Prerequisites:**
 1. Run `export` first to generate the Markdown files
 2. Open Joplin → Tools → Options → Web Clipper → enable the service
 3. Copy the Authorization token and add `JOPLIN_API_TOKEN=<token>` to your `.env`
 4. Joplin desktop must be open when you run this command
 Options: `--provider [chatgpt|claude|all]`, `--project NAME`, `--dry-run`
 ### `prune` — Delete stale export files
 ```bash
 # Preview what would be deleted
 ai-chat-exporter prune --dry-run
 # Delete (asks for confirmation; -y skips the prompt)
 ai-chat-exporter prune
 ```
 Deletes export files no longer referenced by the cache manifest — leftovers
 from old folder layouts or fixed bugs that would otherwise double-sync into
 Joplin. Refuses to run when the manifest is empty (e.g. right after
 `cache --clear`) so it can never wipe a freshly cleared archive. The `doctor`
 command separately verifies that every manifest entry's file exists on disk.
 ### `cache` — Manage the sync manifest
 ```bash
 # Show statistics
 ai-chat-exporter cache --show
 # Clear all cached entries (forces full re-export next run)
 ai-chat-exporter cache --clear
 # Clear a single provider
 ai-chat-exporter cache --clear --provider claude
 ```
 ---
 ## How the Cache Works
 The cache manifest lives at `cache/manifest.json` (inside the install directory) and records every exported conversation: its title, project, `updated_at` timestamp, output file path, and (after Joplin sync) the Joplin note ID.
 On every `export` run:
 1. Fetch the full conversation list from the provider
 2. Compare each conversation's `updated_at` against the manifest
 3. Export only conversations that are new or have been updated
 4. Write each successfully exported conversation to the manifest **immediately** (not batched)
 On every `joplin` run:
 1. Read the manifest to find conversations not yet synced to Joplin, or re-exported since last sync
 2. Push each pending Markdown file to Joplin (create or update)
 3. Store the Joplin note ID in the manifest so subsequent runs update rather than duplicate
 **This design makes every run inherently resumable.** If the tool is interrupted for any reason — rate limit, network drop, Ctrl+C, crash — simply re-run the same command. It will skip already-processed conversations and continue from where it stopped.
 To force a full re-export: `ai-chat-exporter cache --clear` then re-run export.
 ---
 ## Troubleshooting
 ### `401 Unauthorized`
 Your session token has expired.
 - Run `ai-chat-exporter auth` to get a new token interactively
 - Or manually copy a fresh cookie value into your `.env` file
 Note: Claude's `sessionKey` is an opaque string — the only way to know it's expired is the 401 error. ChatGPT JWTs have an `exp` claim that the `doctor` command can decode and display.
 ### `429 Rate Limited`
 The tool automatically pauses, saves progress, and exits with a clear message showing how many conversations were exported vs remaining. Just re-run the same export command to resume — the cache picks up exactly where it left off.
 ### Joplin: "JOPLIN_API_TOKEN is not set"
 You need to configure the token before running the `joplin` command:
 1. Open Joplin desktop
 2. Go to Tools → Options → Web Clipper
 3. Enable the Web Clipper service
 4. Copy the Authorization token shown on that page
 5. Add `JOPLIN_API_TOKEN=<token>` to your `.env` file
 ### Joplin: "Joplin is not responding"
 Joplin desktop must be running when you run the `joplin` command. The Web Clipper service shuts down when Joplin is closed.
 ### Joplin: "Joplin rejected the API token (HTTP 401)"
 The token in `JOPLIN_API_TOKEN` doesn't match what Joplin expects. Get a fresh token from Joplin → Tools → Options → Web Clipper → Authorization token.
 ### Joplin: note timed out
 If you see a timeout error, Joplin took longer than `JOPLIN_REQUEST_TIMEOUT` seconds (default: 30) to respond. Possible causes:
 - The conversation is very large and Joplin is slow to index it
 - Joplin is busy syncing or loading a large library
 - Joplin has frozen — try restarting it
 To increase the timeout: add `JOPLIN_REQUEST_TIMEOUT=60` to your `.env`.
 ### ChatGPT project conversations not appearing
 Make sure you've added the project IDs to `CHATGPT_PROJECT_IDS` in your `.env`. See [ChatGPT Projects](#chatgpt-projects) for how to find them. Project conversations are not included in the default conversation listing — they must be fetched separately.
 ### Schema warnings in logs (`Unexpected API response shape`)
 The provider's internal API may have changed. Run with `--debug`, sanitize the output (remove any personal content), and check the project's GitHub Issues for known fixes.
 ### Non-text content warnings
 Since v0.4.0, rich content is preserved as typed blocks in the export. ChatGPT voice transcripts render as text and audio assets as `📎 File attached` placeholders with size and duration metadata. Anything the extractor doesn't recognise renders as a visible `> ⚠️ Unsupported content` block naming the type and observed keys, *and* increments a counter in the post-export summary so you can tell whether real content is being silently skipped.
 ### Embedded images and media
 Since v0.6.0, image attachments are downloaded by default into a `media/` folder beside each export and inlined as `![](media/…)`. Audio and other files download only with `EXPORTER_DOWNLOAD_MEDIA=all`. On the next `joplin` run these are uploaded as Joplin resources so they render inside the note. Assets that have expired on the provider's side (older AI-generated images often do) keep their text placeholder and are tallied as `media failed` in the run summary — they're never fatal to the export. Set `EXPORTER_DOWNLOAD_MEDIA=off` to skip downloads entirely.
 ### Tool output / hidden context collapsed
 Since v0.6.0, content that was invisible in the ChatGPT web UI is collapsed by default to one-line placeholders like `> 🔧 Tool output — file_search (15.0 KB) — omitted`. This is ChatGPT's file-retrieval tool re-injecting your attached files on every run — it can be 90% of a conversation's bytes while containing none of the dialogue. Custom Instructions collapse the same way (`> ℹ️ Hidden context`). The post-export summary lists every collapsed message by origin with the total KB omitted. To keep everything, set `EXPORTER_HIDDEN_CONTENT=full` in `.env` or pass `--hidden-content full`.
 ### Empty export / all conversations skipped
 No new or updated conversations since your last run. To verify: `ai-chat-exporter cache --show`. To force a full re-export: `ai-chat-exporter cache --clear`.
 ### Filing a bug report
 1. Run with `--debug`: `ai-chat-exporter export --debug 2>&1 | tee debug.log`
 2. Remove any personal conversation content from `debug.log`
 3. Open a GitHub Issue with the sanitized log and the exact command you ran
 ---
 ## Future Work
 See `FUTURE.md` for the full roadmap. Current priorities:
 - **Watch/scheduled mode** on the way to a headless StartOS service
 ---
 ## Security Notes
 - All exported data is stored **locally only** — nothing is sent anywhere except to your local Joplin instance
 - Exported files and the cache manifest are created with `600` permissions (owner read/write only)
 - `.env` is in `.gitignore` — **never commit it**
 - Session tokens are never logged, printed, or included in error messages
 - The Joplin API token is only ever sent to `localhost` — it never leaves your machine
 - If you accidentally commit `.env`: immediately log out and back in to invalidate the token, then remove it from git history using [BFG Repo Cleaner](https://rtyley.github.io/bfg-repo-cleaner/) or `git filter-branch`
@@ -4,16 +4,18 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "ai-chat-exporter"
-version = "0.1.0"
+version = "0.6.0"
 description = "Export ChatGPT and Claude conversation history to Markdown for personal archival in Joplin"
 requires-python = ">=3.11"
 dependencies = [
    "requests==2.31.0",
    "curl_cffi==0.14.0",
    "click==8.1.7",
    "python-dotenv==1.0.1",
    "rich==13.7.1",
    "python-slugify==8.0.4",
    "PyJWT==2.8.0",
    "browser-cookie3==0.20.1",
 ]
 [project.optional-dependencies]
@@ -1,14 +1,17 @@
 # Editable Git install with no remote (ai-chat-exporter==0.1.0)
 -e /home/jesse/services/ai-chatexport
 certifi==2026.2.25
 cffi==2.0.0
 charset-normalizer==3.4.4
 click==8.1.7
 curl_cffi==0.14.0
 idna==3.11
 iniconfig==2.3.0
 markdown-it-py==4.0.0
 mdurl==0.1.2
 packaging==26.0
 pluggy==1.6.0
 pycparser==3.0
 Pygments==2.19.2
 PyJWT==2.8.0
 pytest==8.1.1
@@ -23,3 +26,4 @@ setuptools==82.0.0
 text-unidecode==1.3
 urllib3==2.6.3
 wheel==0.46.3
 browser-cookie3==0.20.1
@@ -0,0 +1,399 @@
 """Typed content blocks for normalized messages.
 Providers produce ordered lists of blocks; exporters render them. Living outside
 ``src/providers/`` deliberately — blocks are a separate concern from extraction
 or rendering, shared by both layers.
 Block dicts always have ``type`` set to one of the BLOCK_TYPE_* constants.
 Construct via the ``make_*`` helpers; never build dicts by hand. The ``unknown``
 block constructor REQUIRES a corresponding WARNING log + ``LossReport`` tally
 at the call site — see plan §Data-loss visibility.
 """
 import json
 from pathlib import Path
 from typing import Any
 BLOCK_TYPE_TEXT = "text"
 BLOCK_TYPE_CODE = "code"
 BLOCK_TYPE_THINKING = "thinking"
 BLOCK_TYPE_TOOL_USE = "tool_use"
 BLOCK_TYPE_TOOL_RESULT = "tool_result"
 BLOCK_TYPE_CITATION = "citation"
 BLOCK_TYPE_IMAGE_PLACEHOLDER = "image_placeholder"
 BLOCK_TYPE_FILE_PLACEHOLDER = "file_placeholder"
 BLOCK_TYPE_UNKNOWN = "unknown"
 BLOCK_TYPE_HIDDEN_CONTEXT_MARKER = "hidden_context_marker"
 BLOCK_TYPE_COLLAPSED = "collapsed"
 COLLAPSED_KIND_TOOL_DUMP = "tool_dump"
 COLLAPSED_KIND_HIDDEN_CONTEXT = "hidden_context"
 UNKNOWN_REASON_UNKNOWN_TYPE = "unknown_type"
 UNKNOWN_REASON_EXTRACTION_FAILED = "extraction_failed"
 UNKNOWN_REASON_ALL_BLOCKS_FAILED = "all_blocks_failed"
 UNKNOWN_REASON_UNKNOWN_FIELD_IN_KNOWN_TYPE = "unknown_field_in_known_type"
 _OBSERVED_KEYS_LIMIT = 10
 # ---------------------------------------------------------------------------
 # Constructors
 # ---------------------------------------------------------------------------
 def make_text_block(text: str) -> dict | None:
    """Return a text block, or None if the text is empty/whitespace-only.
    Returning None lets callers do ``if block: blocks.append(block)`` and prune
    empty blocks at construction time. See plan §Finalizing the message dict.
    """
    if not isinstance(text, str) or not text.strip():
        return None
    return {"type": BLOCK_TYPE_TEXT, "text": text}
 def make_code_block(code: str, language: str = "") -> dict | None:
    """Return a code block, or None if code is empty."""
    if not isinstance(code, str) or not code.strip():
        return None
    return {"type": BLOCK_TYPE_CODE, "language": language or "", "code": code}
 def make_thinking_block(text: str) -> dict | None:
    """Return a thinking block, or None if empty."""
    if not isinstance(text, str) or not text.strip():
        return None
    return {"type": BLOCK_TYPE_THINKING, "text": text}
 def make_tool_use_block(name: str, input_data: Any, tool_id: str | None = None) -> dict:
    """Return a tool_use block.
    Always returns a block (no None) — tool calls are meaningful even with
    empty inputs.
    """
    return {
        "type": BLOCK_TYPE_TOOL_USE,
        "name": name or "",
        "input": input_data if input_data is not None else {},
        "tool_id": tool_id,
    }
 def make_tool_result_block(
    output: str,
    tool_name: str | None = None,
    is_error: bool = False,
    summary: str | None = None,
 ) -> dict:
    """Return a tool_result block.
    ``summary`` is an optional short human label rendered between header and
    fence (e.g. ChatGPT's ``metadata.reasoning_title`` for execution_output).
    """
    return {
        "type": BLOCK_TYPE_TOOL_RESULT,
        "tool_name": tool_name,
        "output": output if isinstance(output, str) else str(output),
        "is_error": bool(is_error),
        "summary": summary,
    }
 def make_citation_block(
    url: str,
    title: str | None = None,
    snippet: str | None = None,
 ) -> dict | None:
    if not url:
        return None
    return {
        "type": BLOCK_TYPE_CITATION,
        "url": url,
        "title": title,
        "snippet": snippet,
    }
 def make_image_placeholder(
    ref: str,
    source: str = "unknown",
    mime: str | None = None,
 ) -> dict:
    """source ∈ {'user_upload', 'model_generated', 'unknown'}."""
    return {
        "type": BLOCK_TYPE_IMAGE_PLACEHOLDER,
        "ref": ref or "",
        "source": source,
        "mime": mime,
    }
 def make_file_placeholder(
    ref: str,
    filename: str | None = None,
    mime: str | None = None,
    size_bytes: int | None = None,
    duration_seconds: float | None = None,
 ) -> dict:
    return {
        "type": BLOCK_TYPE_FILE_PLACEHOLDER,
        "ref": ref or "",
        "filename": filename,
        "mime": mime,
        "size_bytes": size_bytes,
        "duration_seconds": duration_seconds,
    }
 def make_unknown_block(
    raw_type: str,
    observed_keys: list[str] | None = None,
    reason: str = UNKNOWN_REASON_UNKNOWN_TYPE,
    summary: str | None = None,
 ) -> dict:
    """Construct an unknown block.
    Every call site MUST also emit a WARNING log and increment a LossReport
    tally — see plan §Data-loss visibility. The block surfaces the loss at
    read time; the WARNING surfaces it at run time. Both signals matter.
    """
    keys = list(observed_keys or [])[:_OBSERVED_KEYS_LIMIT]
    return {
        "type": BLOCK_TYPE_UNKNOWN,
        "raw_type": raw_type,
        "observed_keys": keys,
        "reason": reason,
        "summary": summary,
    }
 def make_collapsed_block(
    origin: str,
    content_type: str,
    size_bytes: int,
    kind: str,
 ) -> dict:
    """One-line stand-in for a message omitted by the EXPORTER_HIDDEN_CONTENT policy.
    ``kind`` ∈ {COLLAPSED_KIND_TOOL_DUMP, COLLAPSED_KIND_HIDDEN_CONTEXT}.
    Unlike ``unknown`` blocks (unexpected loss), a collapsed block records an
    intentional policy decision — the content was invisible in the provider's
    web UI and the user chose not to keep it. Every call site MUST tally via
    ``LossReport.record_collapsed`` so the omission stays visible in the
    post-export summary.
    """
    return {
        "type": BLOCK_TYPE_COLLAPSED,
        "origin": origin or "",
        "content_type": content_type or "",
        "size_bytes": int(size_bytes or 0),
        "kind": kind,
    }
 def make_hidden_context_marker(content_type: str) -> dict:
    """A short prepend block that flags the surrounding message as hidden context.
    Driven by the ``metadata.is_visually_hidden_from_conversation`` flag, not by
    content_type matching. The marker tells the reader "this message is
    hidden in the source UI; we're showing it here for archival fidelity."
    """
    return {
        "type": BLOCK_TYPE_HIDDEN_CONTEXT_MARKER,
        "content_type": content_type or "",
    }
 # ---------------------------------------------------------------------------
 # Rendering
 # ---------------------------------------------------------------------------
 def render_blocks_to_markdown(blocks: list[dict]) -> str:
    """Render an ordered list of blocks to a single Markdown string.
    Blocks are joined with one blank line between them. Pure function; no I/O.
    """
    if not blocks:
        return ""
    rendered: list[str] = []
    for block in blocks:
        chunk = _render_one(block)
        if chunk:
            rendered.append(chunk)
    return "\n\n".join(rendered)
 def _render_one(block: dict) -> str:
    btype = block.get("type", "")
    if btype == BLOCK_TYPE_TEXT:
        return block.get("text", "")
    if btype == BLOCK_TYPE_CODE:
        lang = block.get("language") or ""
        code = block.get("code", "")
        fence = _safe_fence(code)
        return f"{fence}{lang}\n{code}\n{fence}"
    if btype == BLOCK_TYPE_THINKING:
        text = block.get("text", "")
        quoted = _blockquote_prefix(text)
        return f"**💭 Reasoning**\n\n{quoted}"
    if btype == BLOCK_TYPE_TOOL_USE:
        name = block.get("name", "")
        input_data = block.get("input", {})
        body_json = json.dumps(input_data, indent=2, sort_keys=False, default=str, ensure_ascii=False)
        fence = _safe_fence(body_json)
        body = f"{fence}json\n{body_json}\n{fence}"
        quoted = _blockquote_prefix(f"🔧 **Tool: {name}**\n{body}")
        return quoted
    if btype == BLOCK_TYPE_TOOL_RESULT:
        output = block.get("output", "")
        is_error = bool(block.get("is_error"))
        tool_name = block.get("tool_name") or ""
        summary = block.get("summary") or ""
        icon = "❌" if is_error else "📤"
        label = "Result (error)" if is_error else "Result"
        if tool_name:
            header = f"{icon} **{label}: {tool_name}**"
        else:
            header = f"{icon} **{label}**"
        fence = _safe_fence(output)
        body = f"{fence}\n{output}\n{fence}"
        if summary:
            inner = f"{header}\n*{summary}*\n{body}"
        else:
            inner = f"{header}\n{body}"
        return _blockquote_prefix(inner)
    if btype == BLOCK_TYPE_CITATION:
        url = block.get("url", "")
        title = block.get("title") or url
        return f"[{title}]({url})"
    if btype == BLOCK_TYPE_IMAGE_PLACEHOLDER:
        ref = block.get("ref", "")
        source = block.get("source", "unknown")
        mime = block.get("mime")
        local_path = block.get("local_path")
        if local_path:
            # Downloaded — render as a real inline image.
            return f"![{source or 'image'}]({local_path})"
        meta_parts = [source] if source else []
        if mime:
            meta_parts.append(mime)
        meta_parts.append("content not preserved in this export")
        meta = ", ".join(meta_parts)
        return f"> 🖼️ **Image attached** — `{ref}` ({meta})"
    if btype == BLOCK_TYPE_FILE_PLACEHOLDER:
        ref = block.get("ref", "")
        filename = block.get("filename")
        label = filename or ref
        mime = block.get("mime")
        size_bytes = block.get("size_bytes")
        duration = block.get("duration_seconds")
        local_path = block.get("local_path")
        meta_parts: list[str] = []
        if mime:
            meta_parts.append(mime)
        size_label = _format_size(size_bytes)
        if size_label:
            meta_parts.append(size_label)
        if isinstance(duration, (int, float)) and duration > 0:
            meta_parts.append(f"{duration:.2f}s")
        if local_path:
            # Downloaded — render as a link to the local copy.
            meta = ", ".join(meta_parts) if meta_parts else ""
            meta_str = f" ({meta})" if meta else ""
            return f"> 📎 **File attached** — [{Path(local_path).name}]({local_path}){meta_str}"
        meta_parts.append("content not preserved in this export")
        meta = ", ".join(meta_parts)
        return f"> 📎 **File attached** — `{label}` ({meta})"
    if btype == BLOCK_TYPE_UNKNOWN:
        raw_type = block.get("raw_type", "?")
        reason = block.get("reason", UNKNOWN_REASON_UNKNOWN_TYPE)
        keys = block.get("observed_keys") or []
        summary = block.get("summary")
        first_line = f"⚠️ **Unsupported content** — type `{raw_type}` ({reason})"
        lines = [first_line]
        if summary:
            lines.append(summary)
        if keys:
            keys_str = ", ".join(f"`{k}`" for k in keys)
            lines.append(f"Keys observed: {keys_str}")
        return _blockquote_prefix("\n".join(lines))
    if btype == BLOCK_TYPE_HIDDEN_CONTEXT_MARKER:
        ctype = block.get("content_type", "")
        return f"> ℹ️ **Hidden context** — `{ctype}`"
    if btype == BLOCK_TYPE_COLLAPSED:
        origin = block.get("origin", "")
        kind = block.get("kind", "")
        if kind == COLLAPSED_KIND_HIDDEN_CONTEXT:
            icon, label = "ℹ️", "Hidden context"
        else:
            icon, label = "🔧", "Tool output"
        size_label = _format_size(block.get("size_bytes"))
        size_part = f" ({size_label})" if size_label else ""
        return (
            f"> {icon} **{label}** — `{origin}`{size_part} — omitted "
            "(EXPORTER_HIDDEN_CONTENT=full to keep)"
        )
    # Defensive: a block of unrecognised local type (shouldn't happen if
    # constructors are used). Render as visible warning rather than dropping.
    return f"> ⚠️ **Internal: unrecognised block type** — `{btype}`"
 # ---------------------------------------------------------------------------
 # Helpers
 # ---------------------------------------------------------------------------
 def _format_size(size_bytes: Any) -> str:
    """Human-readable size ('24.1 KB', '1.05 MB'), or '' when absent/zero."""
    if not isinstance(size_bytes, int) or size_bytes <= 0:
        return ""
    kb = size_bytes / 1024
    return f"{kb:.1f} KB" if kb < 1024 else f"{kb / 1024:.2f} MB"
 def _safe_fence(text: str) -> str:
    """Return a backtick fence longer than the longest run of backticks in ``text``.
    CommonMark requires the closing fence to be at least as long as the opening
    fence. Picking N+1 (where N = longest run in content) ensures the content's
    own backticks are inert. Minimum is 3.
    Verified live against Joplin during planning — see plan
    §Backtick-corruption defense.
    """
    if not isinstance(text, str):
        return "```"
    longest_run = 0
    current_run = 0
    for ch in text:
        if ch == "`":
            current_run += 1
            if current_run > longest_run:
                longest_run = current_run
        else:
            current_run = 0
    fence_len = max(3, longest_run + 1)
    return "`" * fence_len
 def _blockquote_prefix(text: str) -> str:
    """Prefix every line of ``text`` with ``> `` so the whole block renders as a quote.
    Empty source lines become ``>`` (no trailing space) so blockquote continuity
    is preserved without trailing-whitespace noise.
    """
    if not isinstance(text, str):
        return ""
    out_lines: list[str] = []
    for line in text.split("\n"):
        if line == "":
            out_lines.append(">")
        else:
            out_lines.append(f"> {line}")
    return "\n".join(out_lines)
@@ -0,0 +1,101 @@
 """Extract session tokens directly from a local browser's cookie store.
 Default browser is Brave. Uses browser-cookie3, which handles the
 platform-specific cookie decryption (Linux: AES key derived from the OS
 keyring "Safe Storage" secret; Windows: DPAPI; macOS: Keychain). The browser
 must be the one the user is actually logged in with, on the same machine.
 Failure modes worth knowing:
 - Browser not installed / profile not found → BrowserTokenError
 - Cookie DB locked (browser running with exclusive lock; rare on Linux,
  common on Windows) → BrowserTokenError suggesting the browser be closed
 - Logged out / cookie expired → BrowserTokenError naming the missing cookie
 """
 import logging
 logger = logging.getLogger(__name__)
 DEFAULT_BROWSER = "brave"
 SUPPORTED_BROWSERS = ("brave", "chrome", "chromium", "edge", "firefox")
 # ChatGPT splits large session tokens across two cookies to stay under the
 # browser's 4KB cookie limit; small tokens use the unsuffixed name.
 _CHATGPT_COOKIE_0 = "__Secure-next-auth.session-token.0"
 _CHATGPT_COOKIE_1 = "__Secure-next-auth.session-token.1"
 _CHATGPT_COOKIE_SINGLE = "__Secure-next-auth.session-token"
 _CLAUDE_COOKIE = "sessionKey"
 class BrowserTokenError(Exception):
    """Raised when tokens cannot be extracted from the browser profile."""
 def _load_cookies(browser: str, domain: str) -> dict[str, str]:
    """Return {cookie_name: value} for ``domain`` from the given browser."""
    if browser not in SUPPORTED_BROWSERS:
        raise BrowserTokenError(
            f"Unsupported browser {browser!r}. "
            f"Supported: {', '.join(SUPPORTED_BROWSERS)}"
        )
    import browser_cookie3
    loader = getattr(browser_cookie3, browser)
    try:
        jar = loader(domain_name=domain)
    except Exception as e:
        # browser_cookie3 raises BrowserCookieError plus assorted OS/keyring
        # errors. All mean the same thing to the caller: fall back to manual.
        hint = ""
        text = str(e).lower()
        if "lock" in text or "database is locked" in text:
            hint = " Close the browser and try again."
        elif "not find" in text or "no such file" in text:
            hint = " Is the browser installed on this machine?"
        raise BrowserTokenError(
            f"Could not read {browser} cookies for {domain}: {e}.{hint}"
        ) from e
    return {cookie.name: cookie.value or "" for cookie in jar}
 def extract_chatgpt_tokens(browser: str = DEFAULT_BROWSER) -> tuple[str, str | None]:
    """Return (session_token_0, session_token_1_or_None) from the browser.
    Raises BrowserTokenError when no session cookie is present (not logged in,
    or logged out since the last visit).
    """
    cookies = _load_cookies(browser, "chatgpt.com")
    token_0 = cookies.get(_CHATGPT_COOKIE_0, "").strip()
    token_1 = cookies.get(_CHATGPT_COOKIE_1, "").strip() or None
    if token_0:
        logger.info(
            "[browser-tokens] ChatGPT session cookies found in %s (chunked: %s)",
            browser,
            "yes" if token_1 else "no",
        )
        return token_0, token_1
    single = cookies.get(_CHATGPT_COOKIE_SINGLE, "").strip()
    if single:
        logger.info("[browser-tokens] ChatGPT session cookie found in %s (single)", browser)
        return single, None
    raise BrowserTokenError(
        f"No ChatGPT session cookie in {browser} — open https://chatgpt.com "
        "in that browser and log in, then retry."
    )
 def extract_claude_session(browser: str = DEFAULT_BROWSER) -> str:
    """Return the Claude ``sessionKey`` cookie value from the browser."""
    cookies = _load_cookies(browser, "claude.ai")
    session_key = cookies.get(_CLAUDE_COOKIE, "").strip()
    if session_key:
        logger.info("[browser-tokens] Claude sessionKey found in %s", browser)
        return session_key
    raise BrowserTokenError(
        f"No Claude sessionKey cookie in {browser} — open https://claude.ai "
        "in that browser and log in, then retry."
    )
@@ -1,4 +1,4 @@
-"""Local cache manifest for tracking exported conversations."""
+"""Local cache manifest for tracking exported and Joplin-synced conversations."""
 import json
 import logging
@@ -18,11 +18,17 @@ class CacheError(Exception):
 class Cache:
-    """Manages the local JSON manifest of exported conversations.
+    """Manages the local JSON manifest of exported and Joplin-synced conversations.
-    The manifest is the single source of truth for what has been exported.
+    The manifest is the single source of truth for what has been exported and
-    Every run compares the provider's full conversation list against this
+    synced. Every export run compares the provider's full conversation list
-    manifest to determine what is new or updated.
+    against this manifest to determine what is new or updated. The Joplin sync
    run reads it to find conversations not yet pushed to Joplin (or re-exported
    since the last sync).
    Each entry tracks:
      title, project, updated_at, exported_at, file_path,
      joplin_note_id (after first sync), joplin_synced_at (after first sync)
    File security:
    - Permissions: 600 (owner read/write only)
@@ -81,6 +87,7 @@ class Cache:
        self._data[provider][conv_id] = {
            "title": metadata.get("title", ""),
            "project": metadata.get("project"),
            "created_at": metadata.get("created_at", ""),
            "updated_at": metadata.get("updated_at", ""),
            "exported_at": datetime.now(tz=timezone.utc).isoformat(),
            "file_path": metadata.get("file_path", ""),
@@ -150,6 +157,94 @@ class Cache:
        """Return all cached entries for a provider (for --cache --show)."""
        return dict(self._data.get(provider, {}))
    def mark_joplin_synced(self, provider: str, conv_id: str, note_id: str) -> None:
        """Record a successful Joplin sync for a conversation.
        Adds ``joplin_note_id`` and ``joplin_synced_at`` to the manifest entry
        and writes atomically to disk.
        """
        entry = self._data.get(provider, {}).get(conv_id)
        if entry is None:
            logger.warning(
                "[cache] mark_joplin_synced: no cache entry for %s/%s", provider, conv_id[:8]
            )
            return
        entry["joplin_note_id"] = note_id
        entry["joplin_synced_at"] = datetime.now(tz=timezone.utc).isoformat()
        self._save()
    def get_joplin_resources(self, provider: str, conv_id: str) -> dict[str, str]:
        """Return the {relative_media_path: joplin_resource_id} map for a conversation."""
        entry = self._data.get(provider, {}).get(conv_id)
        if not isinstance(entry, dict):
            return {}
        resources = entry.get("joplin_resources")
        return dict(resources) if isinstance(resources, dict) else {}
    def set_joplin_resources(self, provider: str, conv_id: str, resources: dict[str, str]) -> None:
        """Persist the {relative_media_path: joplin_resource_id} map for a conversation."""
        entry = self._data.get(provider, {}).get(conv_id)
        if entry is None:
            return
        entry["joplin_resources"] = dict(resources)
        self._save()
    def get_joplin_pending(self, provider: str) -> list[tuple[str, dict]]:
        """Return (conv_id, entry) pairs that need to be synced to Joplin.
        A conversation is pending when:
        - It has never been synced (no ``joplin_note_id``), OR
        - It was re-exported after the last Joplin sync
          (``exported_at`` > ``joplin_synced_at``).
        Returns:
            List of (conv_id, entry_dict) tuples, where entry_dict includes
            ``file_path``, ``title``, ``project``, and optionally ``joplin_note_id``.
        """
        pending = []
        for conv_id, entry in self._data.get(provider, {}).items():
            if not isinstance(entry, dict):
                continue
            if not entry.get("file_path"):
                continue
            note_id = entry.get("joplin_note_id")
            if not note_id:
                pending.append((conv_id, entry))
                continue
            # Re-sync if the file was re-exported after the last Joplin sync
            exported_at = entry.get("exported_at", "")
            synced_at = entry.get("joplin_synced_at", "")
            if exported_at and synced_at:
                try:
                    from src.utils import _parse_dt
                    if _parse_dt(exported_at) > _parse_dt(synced_at):
                        pending.append((conv_id, entry))
                except Exception:
                    pass
        return pending
    def all_file_paths(self) -> set[str]:
        """Return every non-empty ``file_path`` in the manifest, across providers.
        Used by ``prune`` (files on disk but not here are stale) and by the
        doctor integrity check (files here but not on disk are missing).
        """
        paths: set[str] = set()
        for key, entries in self._data.items():
            if not isinstance(entries, dict) or key in (
                "version",
                "last_run",
                "tos_acknowledged_at",
            ):
                continue
            for entry in entries.values():
                if isinstance(entry, dict) and entry.get("file_path"):
                    paths.add(entry["file_path"])
        return paths
    def last_run(self) -> str | None:
        """Return the ISO8601 timestamp of the last export run, or None."""
        return self._data.get("last_run")
@@ -20,6 +20,12 @@ _CLAUDE_PLACEHOLDER = ""
 # Valid OUTPUT_STRUCTURE values
 VALID_STRUCTURES = {"provider/project/year", "provider/project", "provider/year"}
 # Valid EXPORTER_HIDDEN_CONTENT values (see src/providers/base.py)
 VALID_HIDDEN_CONTENT = {"full", "placeholder", "omit"}
 # Valid EXPORTER_DOWNLOAD_MEDIA values (see src/media.py)
 VALID_DOWNLOAD_MEDIA = {"images", "all", "off"}
 class ConfigError(Exception):
    """Raised when required configuration is missing or invalid."""
@@ -28,6 +34,7 @@ class ConfigError(Exception):
@dataclass
 class Config:
    chatgpt_session_token: str | None
    chatgpt_session_token_1: str | None
    claude_session_key: str | None
    export_dir: Path
    output_structure: str
@@ -35,6 +42,23 @@ class Config:
    log_file: str
    # Decoded ChatGPT JWT expiry (None if token absent or not a JWT)
    chatgpt_token_expiry: datetime | None = field(default=None, repr=False)
    # ChatGPT Project gizmo IDs (g-p-xxx) — project conversations are not
    # included in the default /conversations listing; they must be fetched
    # separately via /backend-api/gizmos/{id}/conversations.
    chatgpt_project_ids: list[str] = field(default_factory=list)
    # Joplin local REST API settings (Web Clipper service)
    joplin_api_token: str | None = None
    joplin_api_url: str = "http://localhost:41184"
    # Policy for content invisible in the provider web UI (retrieval dumps,
    # hidden context): full | placeholder | omit
    hidden_content: str = "placeholder"
    # Session cap: max conversations downloaded per export run (None = unlimited).
    # Every run is resumable, so a capped run just continues next time.
    max_conversations: int | None = None
    # Seconds between consecutive API requests (politeness pacing; 0 disables)
    request_delay: float = 1.0
    # Asset download policy: images | all | off
    download_media: str = "images"
 def load_config() -> Config:
@@ -48,11 +72,36 @@ def load_config() -> Config:
    load_dotenv(override=False)
    chatgpt_token = os.getenv("CHATGPT_SESSION_TOKEN", "").strip() or None
    chatgpt_token_1 = os.getenv("CHATGPT_SESSION_TOKEN_1", "").strip() or None
    claude_key = os.getenv("CLAUDE_SESSION_KEY", "").strip() or None
    export_dir = Path(os.getenv("EXPORT_DIR", "./exports")).expanduser()
    output_structure = os.getenv("OUTPUT_STRUCTURE", "provider/project/year").strip()
-    cache_dir = Path(os.getenv("CACHE_DIR", "~/.ai-chat-exporter")).expanduser()
+    cache_dir = Path(os.getenv("CACHE_DIR", "./cache")).expanduser()
-    log_file = os.getenv("LOG_FILE", "~/.ai-chat-exporter/logs/exporter.log").strip()
+    log_file = os.getenv("LOG_FILE", "./cache/logs/exporter.log").strip()
    # Joplin
    joplin_token = os.getenv("JOPLIN_API_TOKEN", "").strip() or None
    joplin_url = os.getenv("JOPLIN_API_URL", "http://localhost:41184").strip()
    hidden_content = os.getenv("EXPORTER_HIDDEN_CONTENT", "").strip().lower() or "placeholder"
    max_conversations_raw = os.getenv("MAX_CONVERSATIONS_PER_RUN", "").strip()
    request_delay_raw = os.getenv("REQUEST_DELAY", "").strip()
    download_media = os.getenv("EXPORTER_DOWNLOAD_MEDIA", "").strip().lower() or "images"
    # Parse CHATGPT_PROJECT_IDS — comma-separated list of gizmo IDs (g-p-xxx)
    _project_ids_raw = os.getenv("CHATGPT_PROJECT_IDS", "").strip()
    chatgpt_project_ids = [
        pid.strip()
        for pid in _project_ids_raw.split(",")
        if pid.strip() and pid.strip().startswith("g-p-")
    ] if _project_ids_raw else []
    if _project_ids_raw and not chatgpt_project_ids:
        logger.warning(
            "CHATGPT_PROJECT_IDS is set but contains no valid project IDs. "
            "Each ID should start with 'g-p-' (e.g. g-p-68c2b2b3037c8191890036fb4ae3ed9f). "
            "Find your project ID in the browser URL when viewing a project."
        )
    errors: list[str] = []
@@ -63,6 +112,46 @@ def load_config() -> Config:
            f"Must be one of: {', '.join(sorted(VALID_STRUCTURES))}"
        )
    # Validate hidden-content policy
    if hidden_content not in VALID_HIDDEN_CONTENT:
        errors.append(
            f"EXPORTER_HIDDEN_CONTENT '{hidden_content}' is invalid. "
            f"Must be one of: {', '.join(sorted(VALID_HIDDEN_CONTENT))}"
        )
    # Validate media download policy
    if download_media not in VALID_DOWNLOAD_MEDIA:
        errors.append(
            f"EXPORTER_DOWNLOAD_MEDIA '{download_media}' is invalid. "
            f"Must be one of: {', '.join(sorted(VALID_DOWNLOAD_MEDIA))}"
        )
    # Validate session cap
    max_conversations: int | None = None
    if max_conversations_raw:
        try:
            max_conversations = int(max_conversations_raw)
        except ValueError:
            errors.append(
                f"MAX_CONVERSATIONS_PER_RUN '{max_conversations_raw}' is not an integer."
            )
        else:
            if max_conversations < 1:
                errors.append(
                    f"MAX_CONVERSATIONS_PER_RUN must be at least 1 (got {max_conversations})."
                )
    # Validate request pacing
    request_delay = 1.0
    if request_delay_raw:
        try:
            request_delay = float(request_delay_raw)
        except ValueError:
            errors.append(f"REQUEST_DELAY '{request_delay_raw}' is not a number.")
        else:
            if request_delay < 0:
                errors.append(f"REQUEST_DELAY must be >= 0 (got {request_delay}).")
    # Validate and decode ChatGPT JWT
    chatgpt_expiry: datetime | None = None
    if chatgpt_token:
@@ -76,7 +165,7 @@ def load_config() -> Config:
    if not chatgpt_token and not claude_key:
        logger.warning(
            "Neither CHATGPT_SESSION_TOKEN nor CLAUDE_SESSION_KEY is set. "
-            "Run 'python -m src.main auth' to configure credentials."
+            "Run 'ai-chat-exporter auth' to configure credentials."
        )
    # Create and validate output directory
@@ -102,12 +191,20 @@ def load_config() -> Config:
    config = Config(
        chatgpt_session_token=chatgpt_token,
        chatgpt_session_token_1=chatgpt_token_1,
        claude_session_key=claude_key,
        export_dir=export_dir,
        output_structure=output_structure,
        cache_dir=cache_dir,
        log_file=log_file,
        chatgpt_token_expiry=chatgpt_expiry,
        chatgpt_project_ids=chatgpt_project_ids,
        joplin_api_token=joplin_token,
        joplin_api_url=joplin_url,
        hidden_content=hidden_content,
        max_conversations=max_conversations,
        request_delay=request_delay,
        download_media=download_media,
    )
    _log_startup_summary(config)
@@ -125,8 +222,12 @@ def _validate_chatgpt_token(token: str) -> datetime | None:
    try:
        payload = jwt.decode(token, options={"verify_signature": False})
-    except jwt.DecodeError as e:
+    except jwt.DecodeError:
-        logger.warning("CHATGPT_SESSION_TOKEN could not be decoded as JWT: %s", e)
+        # JWE (encrypted JWT, alg=dir) — cannot be decoded without the server key.
        # This is normal for ChatGPT's current token format. Token is valid; expiry unknown.
        logger.debug(
            "CHATGPT_SESSION_TOKEN is an encrypted JWE token — expiry cannot be decoded client-side."
        )
        return None
    exp = payload.get("exp")
@@ -141,7 +242,7 @@ def _validate_chatgpt_token(token: str) -> datetime | None:
    if delta.total_seconds() < 0:
        logger.warning(
            "CHATGPT_SESSION_TOKEN expired at %s. "
-            "Run 'python -m src.main auth' to refresh it.",
+            "Run 'ai-chat-exporter auth' to refresh it.",
            expiry.strftime("%Y-%m-%d %H:%M UTC"),
        )
    elif delta.total_seconds() < 86400:
@@ -178,17 +279,30 @@ def _log_startup_summary(cfg: Config) -> None:
    """Log a single INFO line summarising the active configuration."""
    chatgpt_status = format_token_status(cfg.chatgpt_session_token, cfg.chatgpt_token_expiry)
    claude_status = format_token_status(cfg.claude_session_key)
    joplin_status = "configured" if cfg.joplin_api_token else "not configured"
    logger.info(
        "Config loaded | "
        "ChatGPT: %s | "
        "Claude: %s | "
        "chatgpt_projects: %d | "
        "Joplin: %s | "
        "export_dir=%s | "
        "structure=%s | "
-        "cache_dir=%s",
+        "cache_dir=%s | "
        "hidden_content=%s | "
        "max_conversations=%s | "
        "request_delay=%.1fs | "
        "download_media=%s",
        chatgpt_status,
        claude_status,
        len(cfg.chatgpt_project_ids),
        joplin_status,
        cfg.export_dir,
        cfg.output_structure,
        cfg.cache_dir,
        cfg.hidden_content,
        cfg.max_conversations if cfg.max_conversations is not None else "unlimited",
        cfg.request_delay,
        cfg.download_media,
    )
@@ -6,6 +6,7 @@ from datetime import datetime, timezone
 from pathlib import Path
 from typing import Any
 from src.blocks import render_blocks_to_markdown
 from src.utils import build_export_path, generate_filename
 logger = logging.getLogger(__name__)
@@ -15,6 +16,7 @@ _ROLE_LABELS = {
    "user": ("🧑 Human", "user"),
    "assistant": ("🤖 Assistant", "assistant"),
    "system": ("⚙️ System", "system"),
    "tool": ("🔧 Tool", "tool"),
 }
@@ -125,10 +127,17 @@ class MarkdownExporter:
        # Messages
        for msg in messages:
            role = msg.get("role", "user")
-            content = msg.get("content", "")
+            blocks = msg.get("blocks") or []
            timestamp = msg.get("timestamp")
-            if not content or not content.strip():
+            # Prefer rendering from blocks (v0.4.0+). Backward-compat fallback:
            # if blocks is missing/empty AND content exists, render content as-is.
            if blocks:
                body = render_blocks_to_markdown(blocks)
            else:
                body = msg.get("content", "") or ""
            if not body or not body.strip():
                logger.warning(
                    "[markdown] Skipping empty/whitespace message in conversation %s",
                    conv_id[:8],
@@ -143,7 +152,7 @@ class MarkdownExporter:
            else:
                lines.append("")
-            lines.append(content)
+            lines.append(body)
            lines.append("")
            lines.append("---")
            lines.append("")
@@ -0,0 +1,416 @@
 """Joplin Data API client for importing notes into Joplin desktop."""
 import json
 import logging
 import os
 import re
 from pathlib import Path
 from typing import Any
 import requests
 logger = logging.getLogger(__name__)
 # HTTP timeout for regular API calls (seconds). Notes can be large Markdown
 # files so we allow more time than a typical JSON API call.
 # Override with JOPLIN_REQUEST_TIMEOUT env var if you have very large conversations.
 _REQUEST_TIMEOUT: int = int(os.getenv("JOPLIN_REQUEST_TIMEOUT", "30"))
 class JoplinError(Exception):
    """Raised when the Joplin API returns an error or is unreachable."""
 class JoplinClient:
    """HTTP client for the Joplin local REST API (Web Clipper service).
    Requires Joplin desktop to be running with the Web Clipper service enabled.
    Get your API token from: Joplin → Tools → Options → Web Clipper.
    Args:
        base_url: Joplin API base URL (default: http://localhost:41184).
        token: API authorization token from Joplin Web Clipper settings.
    """
    def __init__(self, base_url: str, token: str) -> None:
        self._base_url = base_url.rstrip("/")
        self._token = token
        # In-memory cache: (parent_id | None, title) → folder ID
        self._notebook_cache: dict[tuple[str | None, str], str] = {}
        self._notebooks_loaded = False
        logger.debug("[joplin] Client initialised with base_url=%s", self._base_url)
    # ------------------------------------------------------------------
    # Connectivity
    # ------------------------------------------------------------------
    def ping(self) -> bool:
        """Return True if the Joplin API is reachable and responding.
        Note: /ping does not require authentication. A successful ping only
        confirms Joplin is running — not that the token is valid. Call
        ``validate_token()`` to confirm authentication separately.
        Raises:
            JoplinError: If the API returns an unexpected non-connection error.
        """
        url = f"{self._base_url}/ping"
        logger.debug("[joplin] GET %s", url)
        try:
            resp = requests.get(url, timeout=5)
            resp.raise_for_status()
            ok = "JoplinClipperServer" in resp.text
            logger.debug("[joplin] ping → %s (body: %r)", "OK" if ok else "unexpected response", resp.text[:80])
            return ok
        except requests.exceptions.ConnectionError:
            logger.debug("[joplin] ping → connection refused at %s", url)
            return False
        except requests.exceptions.Timeout:
            logger.debug("[joplin] ping → timed out after 5s at %s", url)
            return False
        except requests.exceptions.RequestException as e:
            raise JoplinError(f"Joplin ping failed: {e}") from e
    def validate_token(self) -> None:
        """Verify the API token is accepted by Joplin.
        Does a minimal authenticated call (GET /folders?limit=1) and raises
        ``JoplinError`` if authentication fails.
        Raises:
            JoplinError: If the token is rejected (401) or Joplin is unreachable.
        """
        logger.debug("[joplin] Validating API token…")
        self._get("/folders", params={"limit": 1, "fields": "id"})
        logger.debug("[joplin] Token validated OK")
    # ------------------------------------------------------------------
    # Notebooks (folders)
    # ------------------------------------------------------------------
    def list_notebooks(self) -> list[dict]:
        """Return all Joplin notebooks (folders), handling pagination.
        Returns:
            List of folder dicts with at least ``id``, ``title``, and ``parent_id`` keys.
        """
        results: list[dict] = []
        page = 1
        while True:
            logger.debug("[joplin] GET /folders page=%d", page)
            resp = self._get("/folders", params={"page": page, "fields": "id,title,parent_id"})
            items = resp.get("items", [])
            results.extend(items)
            logger.debug("[joplin] /folders page=%d → %d items, has_more=%s", page, len(items), resp.get("has_more"))
            if not resp.get("has_more"):
                break
            page += 1
        return results
    def get_or_create_notebook(self, title: str, parent_id: str | None = None) -> str:
        """Return the Joplin folder ID for ``title`` under ``parent_id``, creating if needed.
        Args:
            title: Notebook display name.
            parent_id: ID of the parent folder, or None for a root notebook.
        Returns:
            Joplin folder ID string.
        """
        if not self._notebooks_loaded:
            self._load_notebook_cache()
        key = (parent_id, title)
        if key in self._notebook_cache:
            folder_id = self._notebook_cache[key]
            logger.debug("[joplin] Notebook cache hit: %r (parent=%s) → %s", title, parent_id, folder_id)
            return folder_id
        # Not found — create it
        logger.info("[joplin] Creating notebook: %r (parent=%s)", title, parent_id)
        data: dict = {"title": title}
        if parent_id:
            data["parent_id"] = parent_id
        resp = self._post("/folders", data)
        folder_id = resp["id"]
        self._notebook_cache[key] = folder_id
        logger.debug("[joplin] Notebook created: %r → %s", title, folder_id)
        return folder_id
    def get_or_create_notebook_path(self, path: list[str]) -> str:
        """Ensure a nested notebook path exists and return the leaf folder ID.
        Creates intermediate notebooks as needed.
        Args:
            path: Ordered list of notebook names, e.g. ["AI-ChatGPT", "No Project"].
        Returns:
            Joplin folder ID of the deepest (leaf) notebook.
        """
        parent_id: str | None = None
        for name in path:
            parent_id = self.get_or_create_notebook(name, parent_id)
        assert parent_id is not None
        return parent_id
    # ------------------------------------------------------------------
    # Notes
    # ------------------------------------------------------------------
    def create_note(self, title: str, body: str, parent_id: str) -> str:
        """Create a new note in the specified notebook.
        Args:
            title: Note title.
            body: Note body (Markdown).
            parent_id: Notebook (folder) ID.
        Returns:
            ID of the created note.
        """
        logger.debug(
            "[joplin] Creating note: %r in notebook %s (%d chars)",
            title, parent_id, len(body),
        )
        resp = self._post("/notes", {"title": title, "body": body, "parent_id": parent_id})
        note_id = resp["id"]
        logger.info("[joplin] Note created: %r → %s", title, note_id)
        return note_id
    def update_note(self, note_id: str, title: str, body: str) -> None:
        """Update the title and body of an existing note.
        Args:
            note_id: Joplin note ID.
            title: New note title.
            body: New note body (Markdown).
        """
        logger.debug(
            "[joplin] Updating note %s: %r (%d chars)",
            note_id, title, len(body),
        )
        self._put(f"/notes/{note_id}", {"title": title, "body": body})
        logger.info("[joplin] Note updated: %r (%s)", title, note_id)
    # ------------------------------------------------------------------
    # Resources (embedded media)
    # ------------------------------------------------------------------
    def create_resource(self, file_path: "Path", title: str | None = None) -> str:
        """Upload a local file as a Joplin resource and return its ID.
        Joplin's ``POST /resources`` is multipart: a ``data`` file part plus a
        ``props`` JSON part. The returned ID is referenced from note bodies as
        ``:/<id>``.
        """
        url = f"{self._base_url}/resources"
        props = json.dumps({"title": title or file_path.name})
        logger.debug("[joplin] POST /resources (%s)", file_path.name)
        try:
            with file_path.open("rb") as fh:
                resp = requests.post(
                    url,
                    params={"token": self._token},
                    files={
                        "data": (file_path.name, fh),
                        "props": (None, props),
                    },
                    timeout=_REQUEST_TIMEOUT,
                )
            resp.raise_for_status()
            resource_id = resp.json()["id"]
            logger.info("[joplin] Resource created: %s → %s", file_path.name, resource_id)
            return resource_id
        except requests.exceptions.ConnectionError as e:
            raise JoplinError(
                "Cannot connect to Joplin. Is Joplin desktop running with Web Clipper enabled?"
            ) from e
        except requests.exceptions.Timeout as e:
            raise JoplinError(_timeout_message("POST", "/resources")) from e
        except requests.exceptions.HTTPError as e:
            raise JoplinError(_http_error_message("POST", "/resources", e)) from e
        except (requests.exceptions.RequestException, KeyError, OSError) as e:
            raise JoplinError(f"Joplin resource upload failed for {file_path.name}: {e}") from e
    # ------------------------------------------------------------------
    # HTTP helpers
    # ------------------------------------------------------------------
    def _get(self, path: str, params: dict | None = None) -> dict[str, Any]:
        url = f"{self._base_url}{path}"
        query = {"token": self._token, **(params or {})}
        logger.debug("[joplin] GET %s params=%s", path, {k: v for k, v in (params or {}).items()})
        try:
            resp = requests.get(url, params=query, timeout=_REQUEST_TIMEOUT)
            logger.debug("[joplin] GET %s → HTTP %d", path, resp.status_code)
            resp.raise_for_status()
            return resp.json()
        except requests.exceptions.ConnectionError as e:
            raise JoplinError(
                "Cannot connect to Joplin. Is Joplin desktop running with Web Clipper enabled?"
            ) from e
        except requests.exceptions.Timeout as e:
            raise JoplinError(_timeout_message("GET", path)) from e
        except requests.exceptions.HTTPError as e:
            raise JoplinError(_http_error_message("GET", path, e)) from e
        except requests.exceptions.RequestException as e:
            raise JoplinError(f"Joplin GET {path} failed: {e}") from e
    def _post(self, path: str, data: dict) -> dict[str, Any]:
        url = f"{self._base_url}{path}"
        logger.debug("[joplin] POST %s", path)
        try:
            resp = requests.post(url, params={"token": self._token}, json=data, timeout=_REQUEST_TIMEOUT)
            logger.debug("[joplin] POST %s → HTTP %d", path, resp.status_code)
            resp.raise_for_status()
            return resp.json()
        except requests.exceptions.ConnectionError as e:
            raise JoplinError(
                "Cannot connect to Joplin. Is Joplin desktop running with Web Clipper enabled?"
            ) from e
        except requests.exceptions.Timeout as e:
            raise JoplinError(_timeout_message("POST", path)) from e
        except requests.exceptions.HTTPError as e:
            raise JoplinError(_http_error_message("POST", path, e)) from e
        except requests.exceptions.RequestException as e:
            raise JoplinError(f"Joplin POST {path} failed: {e}") from e
    def _put(self, path: str, data: dict) -> dict[str, Any]:
        url = f"{self._base_url}{path}"
        logger.debug("[joplin] PUT %s", path)
        try:
            resp = requests.put(url, params={"token": self._token}, json=data, timeout=_REQUEST_TIMEOUT)
            logger.debug("[joplin] PUT %s → HTTP %d", path, resp.status_code)
            resp.raise_for_status()
            return resp.json()
        except requests.exceptions.ConnectionError as e:
            raise JoplinError(
                "Cannot connect to Joplin. Is Joplin desktop running with Web Clipper enabled?"
            ) from e
        except requests.exceptions.Timeout as e:
            raise JoplinError(_timeout_message("PUT", path)) from e
        except requests.exceptions.HTTPError as e:
            raise JoplinError(_http_error_message("PUT", path, e)) from e
        except requests.exceptions.RequestException as e:
            raise JoplinError(f"Joplin PUT {path} failed: {e}") from e
    def _load_notebook_cache(self) -> None:
        logger.debug("[joplin] Loading notebook list from Joplin…")
        notebooks = self.list_notebooks()
        self._notebook_cache = {
            (nb.get("parent_id") or None, nb["title"]): nb["id"]
            for nb in notebooks
        }
        self._notebooks_loaded = True
        logger.debug("[joplin] Notebook cache loaded: %d notebooks", len(self._notebook_cache))
        for (parent_id, title), folder_id in self._notebook_cache.items():
            logger.debug("[joplin]   (%s) %r → %s", parent_id or "root", title, folder_id)
 # ------------------------------------------------------------------
 # Error message helper
 # ------------------------------------------------------------------
 def _timeout_message(method: str, path: str) -> str:
    """Build a clear timeout error message with actionable suggestions."""
    return (
        f"Joplin {method} {path} timed out after {_REQUEST_TIMEOUT}s. "
        "Possible causes:\n"
        "  • The note body is very large and Joplin is slow to process it.\n"
        "  • Joplin is busy (syncing, indexing, or loading a large library).\n"
        "  • Joplin has frozen — try restarting it.\n"
        f"If this happens repeatedly, increase JOPLIN_REQUEST_TIMEOUT in your .env "
        f"(currently {_REQUEST_TIMEOUT}s)."
    )
 def _http_error_message(method: str, path: str, e: requests.exceptions.HTTPError) -> str:
    """Build a human-friendly error message from an HTTP error, with auth hint on 401."""
    resp = e.response
    status = resp.status_code if resp is not None else "?"
    if status == 401:
        return (
            f"Joplin rejected the API token (HTTP 401 on {method} {path}). "
            "Check that JOPLIN_API_TOKEN is correct: "
            "Joplin → Tools → Options → Web Clipper → Authorization token."
        )
    if status == 404:
        return f"Joplin resource not found (HTTP 404 on {method} {path}). The note may have been deleted in Joplin."
    body_snippet = ""
    if resp is not None:
        try:
            body_snippet = f" — {resp.text[:120]}"
        except Exception:
            pass
    return f"Joplin {method} {path} failed: HTTP {status}{body_snippet}"
 # ------------------------------------------------------------------
 # Embedded-media rewriting
 # ------------------------------------------------------------------
 # Markdown image/link targets pointing at the local media/ sibling directory,
 # e.g. ![alt](media/file_x.png) or [name](media/clip.wav).
 _MEDIA_LINK_RE = re.compile(r"(!?\[[^\]]*\])\((media/[^)\s]+)\)")
 def upload_media_and_rewrite(
    body: str,
    note_dir: Path,
    client: "JoplinClient",
    resource_map: dict[str, str],
 ) -> tuple[str, dict[str, str]]:
    """Rewrite local ``media/…`` links to Joplin ``:/resourceId`` references.
    Uploads each referenced file as a Joplin resource (once), rewriting both
    image embeds and file links. ``resource_map`` (relative-path → resource_id)
    is the idempotency record from the cache: known paths are reused, new ones
    are added and returned so the caller can persist them. Missing files are
    left as-is so the link still points at the on-disk copy.
    """
    new_map = dict(resource_map)
    def replace(match: re.Match) -> str:
        label, rel_path = match.group(1), match.group(2)
        resource_id = new_map.get(rel_path)
        if resource_id is None:
            file_path = (note_dir / rel_path).resolve()
            if not file_path.is_file():
                logger.warning("[joplin] Media file missing, leaving link: %s", rel_path)
                return match.group(0)
            resource_id = client.create_resource(file_path)
            new_map[rel_path] = resource_id
        return f"{label}(:/{resource_id})"
    return _MEDIA_LINK_RE.sub(replace, body), new_map
 # ------------------------------------------------------------------
 # Notebook naming helper
 # ------------------------------------------------------------------
 _PROVIDER_DISPLAY = {
    "chatgpt": "AI-ChatGPT",
    "claude": "AI-Claude",
    # Decision 2026-06-12: Claude Code coding projects nest under the same
    # AI-Claude parent, alongside Claude web projects.
    "claude-code": "AI-Claude",
 }
 def notebook_path(provider: str, project: str | None) -> tuple[str, str]:
    """Return (parent_notebook, child_notebook) for the given provider and project.
    The parent is the top-level provider notebook; the child is the project name.
    Examples:
        notebook_path("chatgpt", None)             → ("AI-ChatGPT", "No Project")
        notebook_path("chatgpt", "no-project")     → ("AI-ChatGPT", "No Project")
        notebook_path("claude", "budget-tracker")  → ("AI-Claude", "Budget Tracker")
    """
    parent = _PROVIDER_DISPLAY.get(provider, f"AI-{provider.capitalize()}")
    child = (project or "no-project").replace("-", " ").title()
    return parent, child
@@ -0,0 +1,123 @@
 """Per-export-run tally for content that was dropped or partially extracted.
 Surfaces the loss visibility that the rest of the system promises in its
 output (visible ``unknown`` blocks). The summary emitted at the end of
 each export is the load-bearing operator-facing signal: if a real content
 type starts being silently dropped, this is where it shows up.
 Pass a single instance through ``BaseProvider.normalize_conversation`` and
 read it back in ``src/main.py`` after the export loop. No global state.
 """
 from collections import Counter
 from dataclasses import dataclass, field
 _TOP_N_BREAKDOWN = 5
@dataclass
 class LossReport:
    """Counters for things that didn't render cleanly in an export run."""
    # Type-keyed counters. Values are int counts.
    unknown_blocks: Counter = field(default_factory=Counter)
    extraction_failures: Counter = field(default_factory=Counter)
    filtered_roles: Counter = field(default_factory=Counter)
    # Messages collapsed/omitted by the EXPORTER_HIDDEN_CONTENT policy —
    # intentional, but still surfaced so the omission is never invisible.
    collapsed: Counter = field(default_factory=Counter)
    collapsed_bytes: int = 0
    # Aggregate counters
    messages_rendered: int = 0
    conversations: int = 0
    # Media downloads (EXPORTER_DOWNLOAD_MEDIA): successes / skips / failures
    media_downloaded: int = 0
    media_downloaded_bytes: int = 0
    media_failed: Counter = field(default_factory=Counter)
    # Recording -------------------------------------------------------------
    def record_unknown(self, raw_type: str) -> None:
        self.unknown_blocks[raw_type or "?"] += 1
    def record_extraction_failure(self, raw_type: str) -> None:
        self.extraction_failures[raw_type or "?"] += 1
    def record_filtered_role(self, role: str) -> None:
        self.filtered_roles[role or "?"] += 1
    def record_collapsed(self, origin: str, size_bytes: int = 0) -> None:
        self.collapsed[origin or "?"] += 1
        if isinstance(size_bytes, int) and size_bytes > 0:
            self.collapsed_bytes += size_bytes
    def record_media_downloaded(self, size_bytes: int = 0) -> None:
        self.media_downloaded += 1
        if isinstance(size_bytes, int) and size_bytes > 0:
            self.media_downloaded_bytes += size_bytes
    def record_media_failed(self, reason: str) -> None:
        self.media_failed[reason or "?"] += 1
    def record_message(self) -> None:
        self.messages_rendered += 1
    def record_conversation(self) -> None:
        self.conversations += 1
    # Summary ---------------------------------------------------------------
    def format_summary(self) -> str:
        """Return a multi-line summary table suitable for INFO logging.
        Format pinned by plan §Post-export summary — "(none)" sentinel when a
        counter is empty, top-5 breakdown with "+ N more types" overflow.
        """
        lines: list[str] = ["[export] Run summary:"]
        lines.append(f"  conversations:        {self.conversations}")
        lines.append(f"  messages rendered:    {self.messages_rendered}")
        lines.extend(_format_section("unknown blocks:      ", self.unknown_blocks))
        lines.extend(_format_section("extraction failures: ", self.extraction_failures))
        lines.extend(_format_section("collapsed by policy: ", self.collapsed))
        if self.collapsed:
            lines.append(
                f"                        ≈{self.collapsed_bytes / 1024:.0f} KB omitted "
                "(intentional — EXPORTER_HIDDEN_CONTENT=full to keep)"
            )
        if self.media_downloaded or self.media_failed:
            lines.append(
                f"  media downloaded:     {self.media_downloaded} "
                f"(≈{self.media_downloaded_bytes / 1024:.0f} KB)"
            )
            if self.media_failed:
                total_failed = sum(self.media_failed.values())
                lines.append(f"  media failed:         {total_failed}")
                for reason, count in self.media_failed.most_common(_TOP_N_BREAKDOWN):
                    lines.append(f"                        {reason}={count}")
        lines.append(
            "  filtered roles:       "
            "(filter lifted in v0.4.0 — counter retained for future use, expected 0)"
        )
        if self.filtered_roles:
            for role, count in self.filtered_roles.most_common(_TOP_N_BREAKDOWN):
                lines.append(f"                        {role}={count}")
        return "\n".join(lines)
 def _format_section(label: str, counter: Counter) -> list[str]:
    """Render one counter section: header line + indented breakdown lines."""
    total = sum(counter.values())
    header = f"  {label} {total}"
    if total == 0:
        return [header, "                        (none)"]
    lines = [header]
    most_common = counter.most_common()
    for raw_type, count in most_common[:_TOP_N_BREAKDOWN]:
        lines.append(f"                        {raw_type}={count}")
    if len(most_common) > _TOP_N_BREAKDOWN:
        remainder = len(most_common) - _TOP_N_BREAKDOWN
        lines.append(f"                        + {remainder} more types")
    return lines
@@ -0,0 +1,202 @@
 """Media resolution — download conversation assets next to the export files.
 Walks a normalized conversation's placeholder blocks, downloads each asset via
 the provider, saves it under a ``media/`` directory beside the conversation's
 Markdown file, and annotates the block with a relative ``local_path`` so the
 renderer emits a real inline image / file link instead of a placeholder.
 Policy (``EXPORTER_DOWNLOAD_MEDIA``):
  images (default) — image_placeholder blocks only
  all              — also file_placeholder blocks (voice-mode audio etc.;
                     measured 2026-06-12: 556 clips ≈ 162MB whose transcripts
                     are already in the exports as text)
  off              — leave placeholders untouched
 Failures (expired assets, network) keep the existing placeholder and are
 counted in the run summary — never fatal to the export.
 """
 import logging
 import os
 import tempfile
 from pathlib import Path
 from src.blocks import BLOCK_TYPE_FILE_PLACEHOLDER, BLOCK_TYPE_IMAGE_PLACEHOLDER
 from src.loss_report import LossReport
 from src.providers.base import ProviderError
 from src.utils import build_export_path, generate_filename
 logger = logging.getLogger(__name__)
 MEDIA_IMAGES = "images"
 MEDIA_ALL = "all"
 MEDIA_OFF = "off"
 VALID_MEDIA_POLICIES = {MEDIA_IMAGES, MEDIA_ALL, MEDIA_OFF}
 _EXT_BY_MIME = {
    "image/png": ".png",
    "image/jpeg": ".jpg",
    "image/gif": ".gif",
    "image/webp": ".webp",
    "audio/wav": ".wav",
    "audio/x-wav": ".wav",
    "audio/mpeg": ".mp3",
    "audio/mp4": ".m4a",
    "audio/aac": ".aac",
    "application/pdf": ".pdf",
 }
 def resolve_media_policy() -> str:
    """Read EXPORTER_DOWNLOAD_MEDIA from the environment, defaulting to images."""
    value = os.getenv("EXPORTER_DOWNLOAD_MEDIA", "").strip().lower()
    if not value:
        return MEDIA_IMAGES
    if value not in VALID_MEDIA_POLICIES:
        logger.warning(
            "EXPORTER_DOWNLOAD_MEDIA=%r is invalid (expected images|all|off) "
            "— using 'images'.",
            value,
        )
        return MEDIA_IMAGES
    return value
 def resolve_media(
    normalized: dict,
    provider,
    export_base: Path,
    structure: str,
    policy: str,
    report: LossReport,
 ) -> int:
    """Download assets for one normalized conversation. Returns download count.
    Mutates placeholder blocks in place (adds ``local_path``). Idempotent:
    an asset already on disk is annotated without an API call.
    """
    if policy == MEDIA_OFF:
        return 0
    download = getattr(provider, "download_asset", None)
    if download is None:
        # Provider has no remote assets (e.g. claude-code local transcripts).
        return 0
    wanted_types = {BLOCK_TYPE_IMAGE_PLACEHOLDER}
    if policy == MEDIA_ALL:
        wanted_types.add(BLOCK_TYPE_FILE_PLACEHOLDER)
    targets = [
        block
        for message in normalized.get("messages", [])
        for block in message.get("blocks", [])
        if block.get("type") in wanted_types and block.get("ref")
    ]
    if not targets:
        return 0
    # Same path computation the exporter uses, so media/ lands beside the .md.
    filename = generate_filename(
        normalized.get("title", "Untitled"),
        normalized.get("id", ""),
        normalized.get("created_at") or "2000-01-01",
    )
    conv_dir = build_export_path(
        export_base,
        normalized.get("provider", ""),
        normalized.get("project"),
        normalized.get("created_at") or "2000-01-01",
        filename,
        structure,
    ).parent
    media_dir = conv_dir / "media"
    downloaded = 0
    for block in targets:
        ref = block["ref"]
        file_id = _safe_asset_name(provider, ref)
        if not file_id:
            report.record_media_failed("unparseable-ref")
            continue
        existing = _find_existing(media_dir, file_id)
        if existing is not None:
            block["local_path"] = f"media/{existing.name}"
            continue
        try:
            content, mime, file_name = download(ref)
        except ProviderError as e:
            logger.warning(
                "[media] Could not download %s: %s", ref[:60], e.original
            )
            reason = "expired-or-missing" if "404" in str(e.original) or "not found" in str(
                e.original
            ).lower() else "download-error"
            report.record_media_failed(reason)
            continue
        ext = _pick_extension(mime, file_name)
        target = media_dir / f"{file_id}{ext}"
        try:
            _write_atomic(target, content)
        except OSError as e:
            logger.error("[media] Could not write %s: %s", target, e)
            report.record_media_failed("write-error")
            continue
        block["local_path"] = f"media/{target.name}"
        report.record_media_downloaded(len(content))
        downloaded += 1
    return downloaded
 def _safe_asset_name(provider, ref: str) -> str | None:
    """A stable, filesystem-safe name for the asset (the provider file ID)."""
    parser = getattr(provider, "parse_asset_file_id", None)
    if parser is None:
        from src.providers.chatgpt import parse_asset_file_id as parser
    file_id = parser(ref)
    if not file_id:
        return None
    return "".join(c if c.isalnum() or c in "-_." else "_" for c in file_id)
 def _find_existing(media_dir: Path, file_id: str) -> Path | None:
    """Return an already-downloaded file for this asset, if present."""
    if not media_dir.is_dir():
        return None
    for candidate in media_dir.glob(f"{file_id}.*"):
        if candidate.is_file() and candidate.stat().st_size > 0:
            return candidate
    return None
 def _pick_extension(mime: str | None, file_name: str | None) -> str:
    if mime:
        base_mime = mime.split(";")[0].strip().lower()
        if base_mime in _EXT_BY_MIME:
            return _EXT_BY_MIME[base_mime]
    if file_name:
        suffix = Path(file_name).suffix
        if suffix and len(suffix) <= 8:
            return suffix.lower()
    return ".bin"
 def _write_atomic(target: Path, content: bytes) -> None:
    target.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp_name = tempfile.mkstemp(dir=target.parent, suffix=".tmp")
    try:
        with os.fdopen(fd, "wb") as fh:
            fh.write(content)
        os.chmod(tmp_name, 0o600)
        os.replace(tmp_name, target)
    except OSError:
        try:
            os.unlink(tmp_name)
        except OSError:
            pass
        raise
@@ -1,6 +1,7 @@
 """Abstract base class for AI chat providers."""
 import logging
 import os
 import random
 import time
 from abc import ABC, abstractmethod
@@ -9,8 +10,24 @@ from typing import Any
 import requests
 from src.loss_report import LossReport
 from src.utils import redact_secrets
 # curl_cffi has its own exception hierarchy (rooted at CurlError → OSError),
 # completely separate from requests.exceptions. Import them so _make_request
 # can catch both when a curl_cffi session is in use.
 try:
    from curl_cffi.requests.exceptions import (
        HTTPError as _CurlHTTPError,
        ConnectionError as _CurlConnectionError,
        Timeout as _CurlTimeout,
    )
 except ImportError:
    # Fall back to requests types — catching them twice is harmless.
    _CurlHTTPError = requests.HTTPError  # type: ignore[misc,assignment]
    _CurlConnectionError = requests.ConnectionError  # type: ignore[misc,assignment]
    _CurlTimeout = requests.Timeout  # type: ignore[misc,assignment]
 logger = logging.getLogger(__name__)
 # Request timeouts (connect, read) in seconds
@@ -21,6 +38,62 @@ MAX_RETRIES = 3
 BACKOFF_BASE = 2.0
 BACKOFF_MAX = 60.0
 # EXPORTER_HIDDEN_CONTENT policy — what to do with content that was invisible
 # in the provider's UI (retrieval dumps, hidden-flagged context, agent tool
 # traffic). Shared by all providers.
 HIDDEN_CONTENT_FULL = "full"
 HIDDEN_CONTENT_PLACEHOLDER = "placeholder"
 HIDDEN_CONTENT_OMIT = "omit"
 VALID_HIDDEN_CONTENT_POLICIES = {
    HIDDEN_CONTENT_FULL,
    HIDDEN_CONTENT_PLACEHOLDER,
    HIDDEN_CONTENT_OMIT,
 }
 def resolve_hidden_content_policy() -> str:
    """Read EXPORTER_HIDDEN_CONTENT from the environment, defaulting to placeholder."""
    value = os.getenv("EXPORTER_HIDDEN_CONTENT", "").strip().lower()
    if not value:
        return HIDDEN_CONTENT_PLACEHOLDER
    if value not in VALID_HIDDEN_CONTENT_POLICIES:
        logger.warning(
            "EXPORTER_HIDDEN_CONTENT=%r is invalid (expected full|placeholder|omit) "
            "— using 'placeholder'.",
            value,
        )
        return HIDDEN_CONTENT_PLACEHOLDER
    return value
 # Default polite pacing between consecutive API requests (seconds). Keeps
 # traffic human-paced instead of bursty. REQUEST_DELAY=0 disables.
 DEFAULT_REQUEST_DELAY = 1.0
 def resolve_request_delay() -> float:
    """Read REQUEST_DELAY from the environment, defaulting to DEFAULT_REQUEST_DELAY."""
    raw = os.getenv("REQUEST_DELAY", "").strip()
    if not raw:
        return DEFAULT_REQUEST_DELAY
    try:
        value = float(raw)
    except ValueError:
        logger.warning(
            "REQUEST_DELAY=%r is not a number — using default %.1fs.",
            raw,
            DEFAULT_REQUEST_DELAY,
        )
        return DEFAULT_REQUEST_DELAY
    if value < 0:
        logger.warning(
            "REQUEST_DELAY=%r is negative — using default %.1fs.",
            raw,
            DEFAULT_REQUEST_DELAY,
        )
        return DEFAULT_REQUEST_DELAY
    return value
 # Realistic Chrome User-Agent
 USER_AGENT = (
    "Mozilla/5.0 (X11; Linux x86_64) "
@@ -60,6 +133,9 @@ class BaseProvider(ABC):
                "Accept-Language": "en-US,en;q=0.9",
            }
        )
        # Polite pacing between consecutive requests (REQUEST_DELAY env).
        self._request_delay = resolve_request_delay()
        self._last_request_at: float | None = None
    # ------------------------------------------------------------------
    # Abstract interface — subclasses must implement these
@@ -74,13 +150,36 @@ class BaseProvider(ABC):
        """Return the full conversation detail for a single ID."""
    @abstractmethod
-    def normalize_conversation(self, raw: dict) -> dict:
+    def normalize_conversation(self, raw: dict, loss_report: LossReport | None = None) -> dict:
-        """Transform provider-specific schema to the common normalized schema."""
+        """Transform provider-specific schema to the common normalized schema.
        ``loss_report`` accumulates counts of dropped/unhandled content so the
        export loop can surface a single summary at the end. When None, providers
        construct a throwaway local report (so calling normalize_conversation in
        isolation, e.g. from tests, doesn't crash).
        """
    # ------------------------------------------------------------------
    # Concrete helpers
    # ------------------------------------------------------------------
    def _pace(self) -> None:
        """Sleep so consecutive requests are at least REQUEST_DELAY apart (±25% jitter).
        getattr fallback: tests construct providers via ``__new__``, skipping
        ``__init__`` — those run unpaced.
        """
        delay = getattr(self, "_request_delay", 0)
        if delay <= 0:
            return
        target = delay * random.uniform(0.75, 1.25)
        last = getattr(self, "_last_request_at", None)
        if last is not None:
            wait = target - (time.monotonic() - last)
            if wait > 0:
                time.sleep(wait)
        self._last_request_at = time.monotonic()
    def fetch_all_conversations(self, since: datetime | None = None) -> list[dict]:
        """Fetch every conversation, handling pagination automatically.
@@ -186,6 +285,7 @@ class BaseProvider(ABC):
        while attempt <= MAX_RETRIES:
            attempt += 1
            self._pace()
            start = time.monotonic()
            try:
@@ -271,7 +371,7 @@ class BaseProvider(ABC):
            except ProviderError:
                raise
-            except (requests.ConnectionError, requests.Timeout) as e:
+            except (requests.ConnectionError, requests.Timeout, _CurlConnectionError, _CurlTimeout) as e:
                last_exc = e
                if attempt > MAX_RETRIES:
                    raise ProviderError(
@@ -293,7 +393,7 @@ class BaseProvider(ABC):
                )
                time.sleep(wait)
-            except requests.HTTPError as e:
+            except (requests.HTTPError, _CurlHTTPError) as e:
                raise ProviderError(
                    self.provider_name, f"{method} {url}", e
                ) from e
@@ -311,7 +411,7 @@ class BaseProvider(ABC):
        msg = (
            f"[{self.provider_name}] Authentication failed (401 Unauthorized). "
            "Your session token has likely expired. "
-            "Run 'python -m src.main auth' to refresh your token."
+            "Run 'ai-chat-exporter auth' to refresh your token."
        )
        logger.error(msg)
        raise ProviderError(
@@ -3,25 +3,46 @@
 import logging
 import os
 from curl_cffi import requests as curl_requests
 from src.blocks import (
    UNKNOWN_REASON_EXTRACTION_FAILED,
    UNKNOWN_REASON_UNKNOWN_TYPE,
    make_image_placeholder,
    make_text_block,
    make_thinking_block,
    make_tool_result_block,
    make_tool_use_block,
    make_unknown_block,
 )
 from src.loss_report import LossReport
 from src.providers.base import BaseProvider, ProviderError
 logger = logging.getLogger(__name__)
 BASE_URL = "https://claude.ai/api"
 IMPERSONATE = "chrome120"
 class ClaudeProvider(BaseProvider):
    """Provider for Claude conversations via the internal web API.
-    Authentication: Cookie: sessionKey=<CLAUDE_SESSION_KEY>
+    Uses curl_cffi to impersonate Chrome's TLS fingerprint, bypassing
-    Token: sessionKey cookie value from claude.ai.
+    Cloudflare's bot detection (same issue as chatgpt.com).
-    Typical validity: ~30 days (opaque; expiry cannot be decoded client-side).
+
    Authentication: sessionKey cookie (~30 day lifetime, opaque string).
    Expiry cannot be decoded client-side — a 401 is the only signal.
    """
    provider_name = "claude"
    def __init__(self, session_key: str | None = None) -> None:
-        super().__init__()
+        cf_session = curl_requests.Session(impersonate=IMPERSONATE)
        super().__init__(session=cf_session)  # type: ignore[arg-type]
        # Remove base class User-Agent so curl_cffi uses its Chrome-matched UA
        self._session.headers.pop("User-Agent", None)
        key = session_key or os.getenv("CLAUDE_SESSION_KEY", "").strip()
        if not key:
            raise ProviderError(
@@ -29,19 +50,19 @@ class ClaudeProvider(BaseProvider):
                "init",
                RuntimeError(
                    "CLAUDE_SESSION_KEY is not set. "
-                    "Run 'python -m src.main auth' to configure it."
+                    "Run 'ai-chat-exporter auth' to configure it."
                ),
            )
-        # Set cookie header; never log the key value
+        # Set sessionKey in the cookie jar
        self._session.cookies.set("sessionKey", key, domain="claude.ai", path="/")
        self._session.headers.update(
            {
                "Cookie": f"sessionKey={key}",
                "Referer": "https://claude.ai/",
                "Origin": "https://claude.ai",
            }
        )
        self._org_id: str | None = None  # cached per session
-        logger.debug("[claude] Session initialised (key: [REDACTED])")
+        logger.debug("[claude] Session initialised with Chrome TLS impersonation (key: [REDACTED])")
    def _handle_401(self) -> None:
        msg = (
@@ -50,7 +71,7 @@ class ClaudeProvider(BaseProvider):
            "Note: Claude session keys are opaque — a 401 is the only expiry signal. "
            "To refresh: open claude.ai in Chrome → F12 → Application → Cookies "
            "→ find 'sessionKey' → copy the value. "
-            "Then run 'python -m src.main auth' or update CLAUDE_SESSION_KEY in .env."
+            "Then run 'ai-chat-exporter auth' or update CLAUDE_SESSION_KEY in .env."
        )
        logger.error(msg)
        raise ProviderError(
@@ -151,8 +172,9 @@ class ClaudeProvider(BaseProvider):
        return data
-    def normalize_conversation(self, raw: dict) -> dict:
+    def normalize_conversation(self, raw: dict, loss_report: LossReport | None = None) -> dict:
        """Transform Claude raw schema to the common normalized schema."""
        report = loss_report if loss_report is not None else LossReport()
        conv_id = raw.get("uuid") or raw.get("id", "")
        title = raw.get("name") or raw.get("title") or "Untitled"
        created_at = raw.get("created_at") or raw.get("create_time") or ""
@@ -168,40 +190,37 @@ class ClaudeProvider(BaseProvider):
        # Messages
        raw_messages = raw.get("chat_messages") or raw.get("messages") or []
-        messages = []
+        messages: list[dict] = []
        for msg in raw_messages:
            role = _map_role(msg.get("sender") or msg.get("role", ""))
            if not role:
                continue
-            # Content can be a string or a list of content blocks
+            content_raw = msg.get("content") if "content" in msg else msg.get("text", "")
-            content_raw = msg.get("content") or msg.get("text") or ""
+            blocks = _extract_claude_blocks(content_raw, conv_id, report)
            content, skipped_types = _extract_claude_text(content_raw, conv_id)
            for ctype in skipped_types:
                logger.warning(
                    "[claude] Skipping %s content in conversation %s "
                    "— rich content not yet supported (see FUTURE.md)",
                    ctype,
                    conv_id[:8],
                )
            timestamp = msg.get("created_at") or msg.get("timestamp") or None
-            if content is None:
+            if not blocks:
                logger.debug("[claude] Skipping empty message in conversation %s", conv_id[:8])
                continue
            content_type = "text"
            messages.append(
                {
                    "role": role,
-                    "content": content,
+                    "content_type": content_type,
                    "content_type": "text",
                    "timestamp": timestamp,
                    "blocks": blocks,
                }
            )
        for _ in messages:
            report.record_message()
        report.record_conversation()
        return {
            "id": conv_id,
            "title": title,
@@ -232,43 +251,134 @@ def _map_role(sender: str) -> str | None:
    return mapping.get(sender.lower()) if sender else None
-def _extract_claude_text(
+def _extract_claude_blocks(
-    content: str | list | dict, conv_id: str
+    content: str | list | dict | None, conv_id: str, report: LossReport
-) -> tuple[str | None, list[str]]:
+) -> list[dict]:
-    """Extract plain text from a Claude content field.
+    """Extract typed blocks from a Claude content field.
-    Returns:
+    Defensive dispatch — zero observed cases of rich Claude content in the
-        (text_or_None, list_of_skipped_content_types)
+    user's archive at planning time, so this is theory-only. Real shapes
    will be locked in v0.4.1 once captured. Any unrecognised block type
    surfaces as an `unknown` block + WARNING + tally.
    """
-    skipped: list[str] = []
+    if content is None:
        return []
    if isinstance(content, str):
-        text = content.strip()
+        block = make_text_block(content)
-        return (text if text else None), skipped
+        return [block] if block else []
    if isinstance(content, list):
-        parts: list[str] = []
+        blocks: list[dict] = []
-        for block in content:
+        for item in content:
-            if isinstance(block, str):
+            if isinstance(item, str):
-                parts.append(block)
+                block = make_text_block(item)
-            elif isinstance(block, dict):
+                if block:
-                btype = block.get("type", "text")
+                    blocks.append(block)
-                if btype == "text":
+            elif isinstance(item, dict):
-                    t = block.get("text", "").strip()
+                blocks.extend(_dispatch_claude_block(item, conv_id, report))
-                    if t:
+        return blocks
                        parts.append(t)
                else:
                    skipped.append(btype)
        text = "\n".join(parts).strip()
        return (text if text else None), skipped
    if isinstance(content, dict):
-        btype = content.get("type", "text")
+        return _dispatch_claude_block(content, conv_id, report)
        if btype == "text":
            text = content.get("text", "").strip()
            return (text if text else None), skipped
        else:
            skipped.append(btype)
            return None, skipped
-    return None, skipped
+    return []
 def _dispatch_claude_block(block: dict, conv_id: str, report: LossReport) -> list[dict]:
    """Translate one raw Claude content block into normalized blocks."""
    btype = block.get("type", "text")
    if btype == "text":
        block_obj = make_text_block(block.get("text", "") or "")
        return [block_obj] if block_obj else []
    if btype == "thinking":
        # Claude extended-thinking blocks may use 'thinking' or 'text' field.
        text = block.get("thinking") or block.get("text") or ""
        block_obj = make_thinking_block(text)
        return [block_obj] if block_obj else []
    if btype == "tool_use":
        return [
            make_tool_use_block(
                name=block.get("name", "") or "",
                input_data=block.get("input"),
                tool_id=block.get("id"),
            )
        ]
    if btype == "tool_result":
        # ``content`` may be a string or a list of nested blocks (recursive).
        nested = block.get("content")
        output = _flatten_tool_result_content(nested, conv_id, report)
        return [
            make_tool_result_block(
                output=output,
                tool_name=None,
                is_error=bool(block.get("is_error")),
            )
        ]
    if btype == "image":
        # Source shape is unverified; try the most likely fields.
        source = block.get("source") or {}
        ref = ""
        if isinstance(source, dict):
            ref = (
                source.get("file_uuid")
                or source.get("media_type")
                or source.get("url")
                or ""
            )
        return [make_image_placeholder(ref=ref or "(unknown)", source="user_upload")]
    # Unknown block type
    keys = list(block.keys())
    logger.warning(
        "[claude] Unknown block type %r in conversation %s "
        "— see plan §Data-loss visibility (rendering as unknown block)",
        btype,
        conv_id[:8],
    )
    report.record_unknown(btype or "?")
    return [
        make_unknown_block(
            raw_type=btype or "?",
            observed_keys=keys,
            reason=UNKNOWN_REASON_UNKNOWN_TYPE,
        )
    ]
 def _flatten_tool_result_content(
    nested: object, conv_id: str, report: LossReport
 ) -> str:
    """Flatten Claude tool_result content (string OR list of nested blocks) to text.
    Recurses into nested text blocks; any non-text nested block becomes a
    visible inline marker so non-text content isn't silently dropped.
    """
    if nested is None:
        return ""
    if isinstance(nested, str):
        return nested
    if isinstance(nested, list):
        chunks: list[str] = []
        for item in nested:
            if isinstance(item, str):
                chunks.append(item)
            elif isinstance(item, dict):
                btype = item.get("type", "text")
                if btype == "text":
                    chunks.append(item.get("text", "") or "")
                else:
                    keys = list(item.keys())[:10]
                    report.record_extraction_failure(f"tool_result.{btype}")
                    chunks.append(
                        f"[Unsupported nested {btype} block; keys={keys}]"
                    )
        return "\n".join(c for c in chunks if c)
    if isinstance(nested, dict):
        return _flatten_tool_result_content([nested], conv_id, report)
    return str(nested)
@@ -0,0 +1,476 @@
 """Claude Code session provider — archives local agent transcripts.
 Reads JSONL session files from ``~/.claude/projects/<munged-cwd>/<uuid>.jsonl``
 (override with ``CLAUDE_CODE_DIR``). No tokens, no rate limits, no ToS risk —
 but the data lives in single files Claude Code may clean up, and it contains
 deliverables (reviews, plans, analyses) that exist nowhere else.
 Rendering follows the EXPORTER_HIDDEN_CONTENT policy (decided 2026-06-12):
 prose-only by default. User prompts and assistant text are kept; tool_use /
 tool_result traffic is collapsed to one grouped placeholder per activity run
 (measured: dialogue prose is ~4% of session bytes); thinking blocks are
 dropped (counted in the run summary, no placeholder). ``full`` keeps
 everything.
 Record types in a session file: ``user`` / ``assistant`` carry the dialogue
 (Anthropic-style ``message.content`` block arrays); ``ai-title`` carries the
 evolving session title (last one wins); ``last-prompt``,
 ``file-history-snapshot``, ``attachment``, ``permission-mode``, ``system``
 are harness records and are skipped. Records flagged ``isSidechain`` are
 subagent transcripts; ``isMeta`` are harness-generated user records — both
 skipped.
 """
 import json
 import logging
 import os
 import re
 from collections import Counter
 from datetime import datetime, timezone
 from pathlib import Path
 from src.blocks import (
    COLLAPSED_KIND_TOOL_DUMP,
    UNKNOWN_REASON_UNKNOWN_TYPE,
    make_collapsed_block,
    make_image_placeholder,
    make_text_block,
    make_thinking_block,
    make_tool_result_block,
    make_tool_use_block,
    make_unknown_block,
 )
 from src.loss_report import LossReport
 from src.providers.base import (
    BaseProvider,
    HIDDEN_CONTENT_FULL,
    ProviderError,
    VALID_HIDDEN_CONTENT_POLICIES,
    resolve_hidden_content_policy,
 )
 logger = logging.getLogger(__name__)
 DEFAULT_PROJECTS_DIR = "~/.claude/projects"
 # Harness-injected tags inside user message text. Stripped so exports contain
 # the dialogue, not the CLI plumbing. A record that is nothing but tags
 # (e.g. a /model invocation) ends up empty and is skipped.
 _HARNESS_TAG_RE = re.compile(
    r"<local-command-caveat>.*?</local-command-caveat>"
    r"|<command-name>.*?</command-name>"
    r"|<command-message>.*?</command-message>"
    r"|<command-args>.*?</command-args>"
    r"|<command-contents>.*?</command-contents>"
    r"|<local-command-stdout>.*?</local-command-stdout>"
    r"|<system-reminder>.*?</system-reminder>",
    re.DOTALL,
 )
 # How many distinct tool names to list in a collapsed-activity placeholder.
 _TOOL_NAMES_SHOWN = 4
 def _strip_harness_noise(text: str) -> str:
    if not isinstance(text, str):
        return ""
    return _HARNESS_TAG_RE.sub("", text).strip()
 class ClaudeCodeProvider(BaseProvider):
    """Local-file provider over Claude Code session transcripts."""
    provider_name = "claude-code"
    def __init__(
        self,
        projects_dir: str | Path | None = None,
        hidden_content: str | None = None,
    ) -> None:
        super().__init__()
        self._projects_dir = Path(
            projects_dir or os.getenv("CLAUDE_CODE_DIR", DEFAULT_PROJECTS_DIR)
        ).expanduser()
        self._hidden_content = (
            hidden_content
            if hidden_content in VALID_HIDDEN_CONTENT_POLICIES
            else resolve_hidden_content_policy()
        )
        # conv_id → session file path, populated by _scan()
        self._path_map: dict[str, Path] = {}
    # ------------------------------------------------------------------
    # BaseProvider interface
    # ------------------------------------------------------------------
    def list_conversations(self, offset: int = 0, limit: int = 100) -> list[dict]:
        full = self._scan()
        return full[offset : offset + limit]
    def fetch_all_conversations(self, since: datetime | None = None) -> list[dict]:
        convs = self._scan()
        if since is not None:
            since_aware = since if since.tzinfo else since.replace(tzinfo=timezone.utc)
            convs = [
                c for c in convs
                if datetime.fromisoformat(c["updated_at"]) >= since_aware
            ]
        logger.info(
            "[claude-code] Found %d session(s) under %s", len(convs), self._projects_dir
        )
        return convs
    def get_conversation(self, conv_id: str) -> dict:
        path = self._path_map.get(conv_id)
        if path is None:
            # Direct call without a prior listing (e.g. tests) — scan first.
            self._scan()
            path = self._path_map.get(conv_id)
        if path is None or not path.exists():
            raise ProviderError(
                self.provider_name,
                f"get_conversation({conv_id[:8]})",
                FileNotFoundError(f"No session file for id {conv_id}"),
            )
        records: list[dict] = []
        bad_lines = 0
        for line in path.read_text(encoding="utf-8").splitlines():
            line = line.strip()
            if not line:
                continue
            try:
                records.append(json.loads(line))
            except json.JSONDecodeError:
                bad_lines += 1
        if bad_lines:
            logger.warning(
                "[claude-code] %s: skipped %d unparseable line(s)", path.name, bad_lines
            )
        mtime = datetime.fromtimestamp(path.stat().st_mtime, tz=timezone.utc)
        return {
            "id": conv_id,
            "_path": str(path),
            "_records": records,
            # Listing and normalized updated_at must match, or the cache
            # staleness comparison would re-export every session every run.
            "_mtime_iso": mtime.isoformat(),
        }
    def normalize_conversation(self, raw: dict, loss_report: LossReport | None = None) -> dict:
        report = loss_report if loss_report is not None else LossReport()
        policy = getattr(self, "_hidden_content", None) or resolve_hidden_content_policy()
        conv_id = raw.get("id") or ""
        records: list[dict] = raw.get("_records") or []
        title = _extract_title(records)
        project = _extract_project(records, raw.get("_path"))
        created_at = next(
            (r.get("timestamp") for r in records if r.get("timestamp")), ""
        )
        updated_at = raw.get("_mtime_iso") or next(
            (r.get("timestamp") for r in reversed(records) if r.get("timestamp")), ""
        )
        messages = _extract_messages(records, conv_id, report, policy)
        for _ in messages:
            report.record_message()
        report.record_conversation()
        return {
            "id": conv_id,
            "title": title,
            "provider": self.provider_name,
            "project": project,
            "created_at": created_at or "",
            "updated_at": updated_at or "",
            "message_count": len(messages),
            "messages": messages,
        }
    # ------------------------------------------------------------------
    # Scanning
    # ------------------------------------------------------------------
    def _scan(self) -> list[dict]:
        if not self._projects_dir.is_dir():
            logger.warning(
                "[claude-code] Projects directory %s does not exist", self._projects_dir
            )
            return []
        convs: list[dict] = []
        for proj_dir in sorted(p for p in self._projects_dir.iterdir() if p.is_dir()):
            for session_file in sorted(proj_dir.glob("*.jsonl")):
                try:
                    stat = session_file.stat()
                except OSError:
                    continue
                if stat.st_size == 0:
                    continue
                conv_id = session_file.stem
                self._path_map[conv_id] = session_file
                title, project, created = _read_session_meta(session_file)
                convs.append(
                    {
                        "id": conv_id,
                        "title": title,
                        "project": project,
                        # The --project filter and dry-run table read the
                        # listing dict, not the normalized conversation.
                        "_project_name": project,
                        "created_at": created,
                        "updated_at": datetime.fromtimestamp(
                            stat.st_mtime, tz=timezone.utc
                        ).isoformat(),
                        "_path": str(session_file),
                        "_project_dir": proj_dir.name,
                    }
                )
        return convs
 # ---------------------------------------------------------------------------
 # Internal helpers
 # ---------------------------------------------------------------------------
 def _read_session_meta(path: Path) -> tuple[str, str | None, str]:
    """Light single-pass scan for listing metadata: (title, project, created_at).
    Substring guards keep this cheap — only candidate lines are JSON-parsed.
    The full-fidelity extraction happens later in normalize_conversation.
    """
    title = ""
    project: str | None = None
    created = ""
    try:
        with path.open(encoding="utf-8") as fh:
            for line in fh:
                if '"ai-title"' in line:
                    try:
                        rec = json.loads(line)
                    except json.JSONDecodeError:
                        continue
                    if rec.get("type") == "ai-title" and rec.get("aiTitle"):
                        title = str(rec["aiTitle"])  # last one wins
                    continue
                if (not created or project is None) and (
                    '"timestamp"' in line or '"cwd"' in line
                ):
                    try:
                        rec = json.loads(line)
                    except json.JSONDecodeError:
                        continue
                    if not created and rec.get("timestamp"):
                        created = str(rec["timestamp"])
                    if project is None and rec.get("cwd"):
                        project = Path(rec["cwd"]).name or None
    except OSError as e:
        logger.warning("[claude-code] Could not read %s: %s", path, e)
    return title, project, created
 def _extract_title(records: list[dict]) -> str:
    """Last ai-title record wins; fall back to the first real user prompt."""
    title = ""
    for rec in records:
        if rec.get("type") == "ai-title" and rec.get("aiTitle"):
            title = str(rec["aiTitle"])
    if title:
        return title
    for rec in records:
        if rec.get("type") != "user" or rec.get("isMeta") or rec.get("isSidechain"):
            continue
        content = (rec.get("message") or {}).get("content")
        if isinstance(content, str):
            text = _strip_harness_noise(content)
        elif isinstance(content, list):
            text = " ".join(
                _strip_harness_noise(item.get("text", ""))
                for item in content
                if isinstance(item, dict) and item.get("type") == "text"
            ).strip()
        else:
            text = ""
        if text:
            return text[:80]
    return "Untitled session"
 def _extract_project(records: list[dict], path: str | None) -> str | None:
    """Project = basename of the session's working directory."""
    for rec in records:
        cwd = rec.get("cwd")
        if cwd:
            name = Path(cwd).name
            if name:
                return name
    # Fallback: the munged directory name (cannot be reliably de-munged
    # because '-' is both the path separator and a legal name character).
    if path:
        return Path(path).parent.name.lstrip("-") or None
    return None
 def _extract_messages(
    records: list[dict], conv_id: str, report: LossReport, policy: str
 ) -> list[dict]:
    messages: list[dict] = []
    # Pending collapsed tool activity: name → call count, plus total bytes.
    pending_tools: Counter = Counter()
    pending_bytes = 0
    def flush_pending() -> None:
        nonlocal pending_bytes
        if not pending_tools:
            return
        shown = ", ".join(
            f"{name} ×{count}" for name, count in pending_tools.most_common(_TOOL_NAMES_SHOWN)
        )
        if len(pending_tools) > _TOOL_NAMES_SHOWN:
            shown += ", …"
        calls = sum(pending_tools.values())
        block = make_collapsed_block(
            origin=f"{calls} calls: {shown}",
            content_type="tool_activity",
            size_bytes=pending_bytes,
            kind=COLLAPSED_KIND_TOOL_DUMP,
        )
        if messages and messages[-1]["role"] == "assistant":
            messages[-1]["blocks"].append(block)
        else:
            messages.append(
                {"role": "tool", "content_type": "text", "timestamp": None, "blocks": [block]}
            )
        pending_tools.clear()
        pending_bytes = 0
    for rec in records:
        if rec.get("type") not in ("user", "assistant"):
            continue
        if rec.get("isSidechain") or rec.get("isMeta"):
            continue
        msg = rec.get("message") or {}
        role = msg.get("role") or rec["type"]
        content = msg.get("content")
        blocks: list[dict] = []
        # This record's own tool traffic — merged into pending only after the
        # record's message is appended, so the placeholder lands on it (not on
        # the previous message).
        local_tools: Counter = Counter()
        local_bytes = 0
        if isinstance(content, str):
            text = _strip_harness_noise(content)
            block = make_text_block(text)
            if block:
                blocks.append(block)
        elif isinstance(content, list):
            for item in content:
                if not isinstance(item, dict):
                    continue
                item_type = item.get("type", "")
                if item_type == "text":
                    block = make_text_block(_strip_harness_noise(item.get("text", "")))
                    if block:
                        blocks.append(block)
                elif item_type in ("thinking", "redacted_thinking"):
                    if policy == HIDDEN_CONTENT_FULL:
                        block = make_thinking_block(
                            item.get("thinking") or item.get("text") or ""
                        )
                        if block:
                            blocks.append(block)
                    else:
                        # Decision 2026-06-12: thinking is dropped without a
                        # placeholder, but stays visible in the run summary.
                        report.record_collapsed(
                            "thinking", len(json.dumps(item, default=str))
                        )
                elif item_type == "tool_use":
                    if policy == HIDDEN_CONTENT_FULL:
                        blocks.append(
                            make_tool_use_block(
                                item.get("name", ""), item.get("input"), item.get("id")
                            )
                        )
                    else:
                        name = item.get("name") or "tool"
                        size = len(json.dumps(item, default=str))
                        local_tools[name] += 1
                        local_bytes += size
                        report.record_collapsed(name, size)
                elif item_type == "tool_result":
                    if policy == HIDDEN_CONTENT_FULL:
                        blocks.append(
                            make_tool_result_block(
                                _stringify_tool_result(item.get("content")),
                                is_error=bool(item.get("is_error")),
                            )
                        )
                    else:
                        size = len(json.dumps(item, default=str))
                        local_bytes += size
                        report.record_collapsed("tool_result", size)
                elif item_type == "image":
                    blocks.append(
                        make_image_placeholder(ref="embedded image", source="user_upload")
                    )
                else:
                    logger.warning(
                        "[claude-code] Unknown content block type %r in session %s",
                        item_type,
                        conv_id[:8],
                    )
                    report.record_unknown(f"claude-code.{item_type or '?'}")
                    blocks.append(
                        make_unknown_block(
                            raw_type=f"claude-code.{item_type or '?'}",
                            observed_keys=list(item.keys()),
                            reason=UNKNOWN_REASON_UNKNOWN_TYPE,
                        )
                    )
        if not blocks:
            # Tool-result-only records (and similar): traffic accumulates and
            # is attached to the message that initiated it.
            pending_tools.update(local_tools)
            pending_bytes += local_bytes
            continue
        # Attach previous records' tool activity to the previous message
        # before starting a new one, so the placeholder lands between the
        # dialogue turns it actually occurred between.
        flush_pending()
        messages.append(
            {
                "role": role,
                "content_type": "text",
                "timestamp": rec.get("timestamp"),
                "blocks": blocks,
            }
        )
        pending_tools.update(local_tools)
        pending_bytes += local_bytes
    flush_pending()
    return messages
 def _stringify_tool_result(content) -> str:
    """tool_result content may be a string or a list of text blocks."""
    if isinstance(content, str):
        return content
    if isinstance(content, list):
        parts = []
        for item in content:
            if isinstance(item, dict) and item.get("type") == "text":
                parts.append(item.get("text", ""))
            else:
                parts.append(json.dumps(item, default=str))
        return "\n".join(parts)
    return json.dumps(content, default=str) if content is not None else ""
@@ -50,7 +50,7 @@ def build_export_path(
        created_at: ISO8601 creation timestamp (used for year folder).
        filename: Already-generated filename from generate_filename().
        structure: OUTPUT_STRUCTURE value. One of:
-            "provider/project/year" (default)
+            "provider/project/year" (default) — project and year combined, e.g. no-project.2025/
            "provider/project"
            "provider/year"
@@ -64,14 +64,14 @@ def build_export_path(
    parts: list[str] = [provider]
    if structure == "provider/project/year":
-        parts += [project_slug, year]
+        parts += [f"{project_slug}.{year}"]
    elif structure == "provider/project":
        parts += [project_slug]
    elif structure == "provider/year":
        parts += [year]
    else:
        # Unknown structure — fall back to default
-        parts += [project_slug, year]
+        parts += [f"{project_slug}.{year}"]
    return base_dir.joinpath(*parts) / filename
@@ -8,12 +8,30 @@
    "node-root": {
      "id": "node-root",
      "parent": null,
-      "children": ["node-1"],
+      "children": ["node-uec"],
      "message": null
    },
    "node-uec": {
      "id": "node-uec",
      "parent": "node-root",
      "children": ["node-1"],
      "message": {
        "id": "node-uec",
        "author": {"role": "user"},
        "create_time": null,
        "content": {
          "content_type": "user_editable_context",
          "user_profile": "Preferred name: Jesse",
          "user_instructions": "The user provided the additional info about how they would like you to respond:\n```Always cite sources.```"
        },
        "metadata": {
          "is_visually_hidden_from_conversation": true
        }
      }
    },
    "node-1": {
      "id": "node-1",
-      "parent": "node-root",
+      "parent": "node-uec",
      "children": ["node-2"],
      "message": {
        "id": "node-1",
@@ -28,7 +46,7 @@
    "node-2": {
      "id": "node-2",
      "parent": "node-1",
-      "children": ["node-3"],
+      "children": ["node-mm-user"],
      "message": {
        "id": "node-2",
        "author": {"role": "assistant"},
@@ -39,19 +57,140 @@
        }
      }
    },
-    "node-3": {
+    "node-mm-user": {
-      "id": "node-3",
+      "id": "node-mm-user",
      "parent": "node-2",
-      "children": [],
+      "children": ["node-mm-assistant"],
      "message": {
-        "id": "node-3",
+        "id": "node-mm-user",
        "author": {"role": "user"},
        "create_time": 1704067300.0,
        "content": {
-          "content_type": "image_asset_pointer",
+          "content_type": "multimodal_text",
-          "parts": [{"content_type": "image_asset_pointer", "asset_pointer": "file://some-image"}]
+          "parts": [
            {"content_type": "audio_transcription", "text": "What is the capital of France?", "direction": "in", "decoding_id": null},
            {"content_type": "real_time_user_audio_video_asset_pointer", "frames_asset_pointers": [], "video_container_asset_pointer": null, "audio_asset_pointer": {"content_type": "audio_asset_pointer", "asset_pointer": "sediment://file_user001", "size_bytes": 50000, "format": "wav", "metadata": {"start": 0.0, "end": 2.5}}, "audio_start_timestamp": 1.0}
          ]
        },
        "metadata": {"voice_mode_message": true}
      }
    },
    "node-mm-assistant": {
      "id": "node-mm-assistant",
      "parent": "node-mm-user",
      "children": ["node-mm-user-rev"],
      "message": {
        "id": "node-mm-assistant",
        "author": {"role": "assistant"},
        "create_time": 1704067305.0,
        "content": {
          "content_type": "multimodal_text",
          "parts": [
            {"content_type": "audio_transcription", "text": "The capital of France is Paris.", "direction": "out", "decoding_id": null},
            {"content_type": "audio_asset_pointer", "asset_pointer": "sediment://file_assistant001", "size_bytes": 80000, "format": "wav", "metadata": {"start": 0.0, "end": 3.2}}
          ]
        }
      }
    },
    "node-mm-user-rev": {
      "id": "node-mm-user-rev",
      "parent": "node-mm-assistant",
      "children": ["node-image-only"],
      "message": {
        "id": "node-mm-user-rev",
        "author": {"role": "user"},
        "create_time": 1704067400.0,
        "content": {
          "content_type": "multimodal_text",
          "parts": [
            {"content_type": "real_time_user_audio_video_asset_pointer", "frames_asset_pointers": [], "video_container_asset_pointer": null, "audio_asset_pointer": {"content_type": "audio_asset_pointer", "asset_pointer": "sediment://file_user002", "size_bytes": 30000, "format": "wav", "metadata": {"start": 0.0, "end": 1.5}}, "audio_start_timestamp": 5.0},
            {"content_type": "audio_transcription", "text": "Tell me more please.", "direction": "in", "decoding_id": null}
          ]
        }
      }
    },
    "node-image-only": {
      "id": "node-image-only",
      "parent": "node-mm-user-rev",
      "children": ["node-exec-output"],
      "message": {
        "id": "node-image-only",
        "author": {"role": "user"},
        "create_time": 1704067500.0,
        "content": {
          "content_type": "multimodal_text",
          "parts": [
            {"content_type": "image_asset_pointer", "asset_pointer": "file-service://image001"}
          ]
        }
      }
    },
    "node-exec-output": {
      "id": "node-exec-output",
      "parent": "node-image-only",
      "children": ["node-exec-output-empty"],
      "message": {
        "id": "node-exec-output",
        "author": {"role": "tool", "name": "container.exec", "metadata": {}},
        "create_time": 1704067600.0,
        "content": {
          "content_type": "execution_output",
          "text": "Hello from container.exec\nLine 2 of output"
        },
        "metadata": {
          "aggregate_result": {"status": "success", "messages": []},
          "reasoning_title": "Reading skill documentation"
        }
      }
    },
    "node-exec-output-empty": {
      "id": "node-exec-output-empty",
      "parent": "node-exec-output",
      "children": ["node-system-error"],
      "message": {
        "id": "node-exec-output-empty",
        "author": {"role": "tool", "name": "python", "metadata": {}},
        "create_time": 1704067610.0,
        "content": {
          "content_type": "execution_output",
          "text": ""
        },
        "metadata": {}
      }
    },
    "node-system-error": {
      "id": "node-system-error",
      "parent": "node-exec-output-empty",
      "children": ["node-tether-spinner"],
      "message": {
        "id": "node-system-error",
        "author": {"role": "tool", "name": "web", "metadata": {}},
        "create_time": 1704067620.0,
        "content": {
          "content_type": "system_error",
          "name": "tool_error",
          "text": "Error: Error from browse service: Error calling browse service: 503"
        },
        "metadata": {}
      }
    },
    "node-tether-spinner": {
      "id": "node-tether-spinner",
      "parent": "node-system-error",
      "children": [],
      "message": {
        "id": "node-tether-spinner",
        "author": {"role": "tool", "name": "file_search", "metadata": {}},
        "create_time": 1704067630.0,
        "content": {
          "content_type": "tether_browsing_display",
          "result": "",
          "summary": "",
          "assets": null,
          "tether_id": null
        },
        "metadata": {"command": "spinner", "status": "running"}
      }
    }
  }
 }
@@ -30,6 +30,15 @@
      "sender": "human",
      "created_at": "2024-06-10T14:45:00.000Z",
      "content": "Thank you, that helped!"
    },
    {
      "uuid": "msg-004",
      "sender": "human",
      "created_at": "2024-06-10T14:50:00.000Z",
      "content": [
        {"type": "text", "text": "What about this image?"},
        {"type": "image", "source": {"file_uuid": "claude-image-uuid-1", "media_type": "image/png"}}
      ]
    }
  ]
 }
@@ -0,0 +1,155 @@
 """Unit tests for browser cookie extraction (browser_cookie3 mocked)."""
 import sys
 import types
 from types import SimpleNamespace
 import pytest
 from src.browser_tokens import (
    BrowserTokenError,
    extract_chatgpt_tokens,
    extract_claude_session,
 )
 def _fake_bc3(monkeypatch, cookies_by_domain, raises=None):
    """Install a stub browser_cookie3 module whose ``brave`` loader returns
    SimpleNamespace cookies for the requested domain (or raises)."""
    module = types.ModuleType("browser_cookie3")
    def brave(domain_name=""):
        if raises is not None:
            raise raises
        return [
            SimpleNamespace(name=name, value=value)
            for name, value in cookies_by_domain.get(domain_name, {}).items()
        ]
    module.brave = brave
    monkeypatch.setitem(sys.modules, "browser_cookie3", module)
    return module
 class TestExtractChatGPT:
    def test_chunked_cookies(self, monkeypatch):
        _fake_bc3(monkeypatch, {
            "chatgpt.com": {
                "__Secure-next-auth.session-token.0": "eyJpart0",
                "__Secure-next-auth.session-token.1": "part1rest",
            },
        })
        assert extract_chatgpt_tokens("brave") == ("eyJpart0", "part1rest")
    def test_single_cookie_fallback(self, monkeypatch):
        _fake_bc3(monkeypatch, {
            "chatgpt.com": {"__Secure-next-auth.session-token": "eyJsingle"},
        })
        assert extract_chatgpt_tokens("brave") == ("eyJsingle", None)
    def test_missing_cookie_raises_with_login_hint(self, monkeypatch):
        _fake_bc3(monkeypatch, {"chatgpt.com": {"unrelated": "x"}})
        with pytest.raises(BrowserTokenError, match="log in"):
            extract_chatgpt_tokens("brave")
    def test_loader_failure_wrapped(self, monkeypatch):
        _fake_bc3(monkeypatch, {}, raises=RuntimeError("database is locked"))
        with pytest.raises(BrowserTokenError, match="Close the browser"):
            extract_chatgpt_tokens("brave")
    def test_unsupported_browser_rejected(self):
        with pytest.raises(BrowserTokenError, match="Unsupported browser"):
            extract_chatgpt_tokens("netscape")
 class TestExtractClaude:
    def test_session_key(self, monkeypatch):
        _fake_bc3(monkeypatch, {"claude.ai": {"sessionKey": "sk-ant-sid01-xyz"}})
        assert extract_claude_session("brave") == "sk-ant-sid01-xyz"
    def test_missing_raises(self, monkeypatch):
        _fake_bc3(monkeypatch, {"claude.ai": {}})
        with pytest.raises(BrowserTokenError, match="log in"):
            extract_claude_session("brave")
 class TestAuthFromBrowserCLI:
    def test_writes_env_after_validation(self, monkeypatch, tmp_path):
        """--from-browser extracts, validates live (mocked), and writes .env."""
        from click.testing import CliRunner
        from src.main import cli
        from src.cache import Cache
        cache = Cache(tmp_path / "cache")
        cache.acknowledge_tos()
        _fake_bc3(monkeypatch, {
            "chatgpt.com": {
                "__Secure-next-auth.session-token.0": "eyJtok0",
                "__Secure-next-auth.session-token.1": "tok1",
            },
            "claude.ai": {"sessionKey": "sk-ant-sid01-abc"},
        })
        # Stub out the live validation calls.
        import src.providers.chatgpt as chatgpt_mod
        import src.providers.claude as claude_mod
        monkeypatch.setattr(
            chatgpt_mod.ChatGPTProvider, "_fetch_access_token", lambda self: "at"
        )
        monkeypatch.setattr(
            claude_mod.ClaudeProvider,
            "list_conversations",
            lambda self, offset=0, limit=100: [],
        )
        runner = CliRunner(mix_stderr=True)
        with runner.isolated_filesystem(temp_dir=tmp_path):
            result = runner.invoke(
                cli,
                ["--no-log-file", "auth", "--from-browser"],
                env={"CACHE_DIR": str(tmp_path / "cache")},
            )
            assert result.exit_code == 0, result.output
            from pathlib import Path
            env_text = Path(".env").read_text()
        assert "CHATGPT_SESSION_TOKEN=eyJtok0" in env_text
        assert "CHATGPT_SESSION_TOKEN_1=tok1" in env_text
        assert "CLAUDE_SESSION_KEY=sk-ant-sid01-abc" in env_text
        assert "2 provider(s) configured" in result.output
    def test_validation_failure_leaves_env_untouched(self, monkeypatch, tmp_path):
        from click.testing import CliRunner
        from src.main import cli
        from src.cache import Cache
        from src.providers.base import ProviderError
        cache = Cache(tmp_path / "cache")
        cache.acknowledge_tos()
        _fake_bc3(monkeypatch, {
            "chatgpt.com": {"__Secure-next-auth.session-token.0": "eyJstale"},
            "claude.ai": {},
        })
        import src.providers.chatgpt as chatgpt_mod
        def _fail(self):
            raise ProviderError("chatgpt", "auth", RuntimeError("401"))
        monkeypatch.setattr(chatgpt_mod.ChatGPTProvider, "_fetch_access_token", _fail)
        runner = CliRunner(mix_stderr=True)
        with runner.isolated_filesystem(temp_dir=tmp_path):
            result = runner.invoke(
                cli,
                ["--no-log-file", "auth", "--from-browser", "brave"],
                env={"CACHE_DIR": str(tmp_path / "cache")},
            )
            import pathlib
            env_exists = pathlib.Path(".env").exists()
        assert result.exit_code == 1
        assert "failed live validation" in result.output
        assert not env_exists
@@ -0,0 +1,230 @@
 """Unit tests for the Claude Code session provider."""
 import json
 import pytest
 from src.blocks import (
    BLOCK_TYPE_COLLAPSED,
    BLOCK_TYPE_TEXT,
    BLOCK_TYPE_THINKING,
    BLOCK_TYPE_TOOL_RESULT,
    BLOCK_TYPE_TOOL_USE,
    render_blocks_to_markdown,
 )
 from src.loss_report import LossReport
 from src.providers.claude_code import ClaudeCodeProvider
 def _write_session(tmp_path, records, project_dir="-home-jesse-myproj", name="abc-123"):
    proj = tmp_path / project_dir
    proj.mkdir(parents=True, exist_ok=True)
    f = proj / f"{name}.jsonl"
    f.write_text("\n".join(json.dumps(r) for r in records), encoding="utf-8")
    return f
 def _records():
    """A representative session: title, harness noise, dialogue, tool traffic."""
    return [
        {"type": "ai-title", "aiTitle": "First title", "sessionId": "abc-123"},
        {
            "type": "user",
            "cwd": "/home/jesse/myproj",
            "timestamp": "2026-05-01T10:00:00.000Z",
            "message": {
                "role": "user",
                "content": (
                    "<local-command-caveat>Caveat: ignore</local-command-caveat>"
                    "<command-name>/model</command-name>"
                    "<local-command-stdout>Set model</local-command-stdout>"
                    "Please review my project."
                ),
            },
        },
        {
            "type": "assistant",
            "timestamp": "2026-05-01T10:00:05.000Z",
            "message": {
                "role": "assistant",
                "content": [
                    {"type": "thinking", "thinking": "Let me think about this."},
                    {"type": "text", "text": "I'll review it now."},
                    {"type": "tool_use", "id": "t1", "name": "Read", "input": {"file_path": "/x"}},
                    {"type": "tool_use", "id": "t2", "name": "Read", "input": {"file_path": "/y"}},
                    {"type": "tool_use", "id": "t3", "name": "Bash", "input": {"command": "ls"}},
                ],
            },
        },
        {
            "type": "user",
            "timestamp": "2026-05-01T10:00:08.000Z",
            "message": {
                "role": "user",
                "content": [
                    {"type": "tool_result", "tool_use_id": "t1", "content": "file contents " * 50},
                ],
            },
        },
        {
            "type": "assistant",
            "timestamp": "2026-05-01T10:00:20.000Z",
            "message": {
                "role": "assistant",
                "content": [{"type": "text", "text": "Here is my review: all good."}],
            },
        },
        # Harness records — skipped
        {"type": "file-history-snapshot", "snapshot": {"big": "blob"}},
        {"type": "permission-mode", "mode": "plan"},
        # Subagent and meta records — skipped
        {
            "type": "assistant",
            "isSidechain": True,
            "message": {"role": "assistant", "content": [{"type": "text", "text": "subagent noise"}]},
        },
        {
            "type": "user",
            "isMeta": True,
            "message": {"role": "user", "content": "meta noise"},
        },
        # Later title wins
        {"type": "ai-title", "aiTitle": "Review my project", "sessionId": "abc-123"},
    ]
 class TestClaudeCodeProvider:
    def _provider(self, tmp_path, policy="placeholder"):
        return ClaudeCodeProvider(projects_dir=tmp_path, hidden_content=policy)
    def test_listing_metadata(self, tmp_path):
        _write_session(tmp_path, _records())
        p = self._provider(tmp_path)
        convs = p.fetch_all_conversations()
        assert len(convs) == 1
        c = convs[0]
        assert c["id"] == "abc-123"
        assert c["title"] == "Review my project"  # last ai-title wins
        assert c["_project_name"] == "myproj"  # from cwd basename
        assert c["created_at"] == "2026-05-01T10:00:00.000Z"
        assert c["updated_at"]  # file mtime
    def test_empty_files_skipped(self, tmp_path):
        proj = tmp_path / "-home-x"
        proj.mkdir()
        (proj / "empty.jsonl").write_text("")
        p = self._provider(tmp_path)
        assert p.fetch_all_conversations() == []
    def test_normalize_prose_only_default(self, tmp_path):
        _write_session(tmp_path, _records())
        p = self._provider(tmp_path)
        p.fetch_all_conversations()
        raw = p.get_conversation("abc-123")
        result = p.normalize_conversation(raw)
        assert result["provider"] == "claude-code"
        assert result["title"] == "Review my project"
        assert result["project"] == "myproj"
        roles = [m["role"] for m in result["messages"]]
        assert roles == ["user", "assistant", "assistant"]
        # Harness tags stripped, dialogue kept
        user_text = result["messages"][0]["blocks"][0]["text"]
        assert user_text == "Please review my project."
        assert "Caveat" not in user_text
        # Tool activity collapsed into one grouped placeholder on the
        # assistant message that ran the tools (includes the tool_result
        # bytes from the following user record).
        first_assistant = result["messages"][1]
        types = [b["type"] for b in first_assistant["blocks"]]
        assert types == [BLOCK_TYPE_TEXT, BLOCK_TYPE_COLLAPSED]
        collapsed = first_assistant["blocks"][1]
        assert "3 calls" in collapsed["origin"]
        assert "Read ×2" in collapsed["origin"]
        assert "Bash ×1" in collapsed["origin"]
        assert collapsed["size_bytes"] > 500
        # Thinking dropped without placeholder; sidechain/meta absent
        all_blocks = [b for m in result["messages"] for b in m["blocks"]]
        assert not any(b["type"] == BLOCK_TYPE_THINKING for b in all_blocks)
        assert not any(
            "subagent noise" in (b.get("text") or "") or "meta noise" in (b.get("text") or "")
            for b in all_blocks
        )
    def test_collapsed_counted_in_loss_report(self, tmp_path):
        _write_session(tmp_path, _records())
        p = self._provider(tmp_path)
        p.fetch_all_conversations()
        report = LossReport()
        p.normalize_conversation(p.get_conversation("abc-123"), report)
        assert report.collapsed["Read"] == 2
        assert report.collapsed["Bash"] == 1
        assert report.collapsed["tool_result"] == 1
        assert report.collapsed["thinking"] == 1
        assert report.collapsed_bytes > 500
    def test_full_policy_keeps_everything(self, tmp_path):
        _write_session(tmp_path, _records())
        p = self._provider(tmp_path, policy="full")
        p.fetch_all_conversations()
        result = p.normalize_conversation(p.get_conversation("abc-123"))
        all_blocks = [b for m in result["messages"] for b in m["blocks"]]
        types = {b["type"] for b in all_blocks}
        assert BLOCK_TYPE_THINKING in types
        assert BLOCK_TYPE_TOOL_USE in types
        assert BLOCK_TYPE_TOOL_RESULT in types
        assert BLOCK_TYPE_COLLAPSED not in types
    def test_updated_at_consistent_with_listing(self, tmp_path):
        """Listing and normalized updated_at must match or the cache would
        consider every session stale on every run (mtime vs last timestamp)."""
        _write_session(tmp_path, _records())
        p = self._provider(tmp_path)
        listing = p.fetch_all_conversations()[0]
        normalized = p.normalize_conversation(p.get_conversation("abc-123"))
        assert normalized["updated_at"] == listing["updated_at"]
    def test_title_falls_back_to_first_prompt(self, tmp_path):
        records = [r for r in _records() if r.get("type") != "ai-title"]
        _write_session(tmp_path, records)
        p = self._provider(tmp_path)
        p.fetch_all_conversations()
        result = p.normalize_conversation(p.get_conversation("abc-123"))
        assert result["title"] == "Please review my project."
    def test_unknown_block_type_surfaces(self, tmp_path):
        records = [
            {
                "type": "assistant",
                "timestamp": "2026-05-01T10:00:00.000Z",
                "cwd": "/home/jesse/myproj",
                "message": {
                    "role": "assistant",
                    "content": [
                        {"type": "text", "text": "hello"},
                        {"type": "future_block_xyz", "data": 1},
                    ],
                },
            },
        ]
        _write_session(tmp_path, records)
        p = self._provider(tmp_path)
        p.fetch_all_conversations()
        report = LossReport()
        result = p.normalize_conversation(p.get_conversation("abc-123"), report)
        assert report.unknown_blocks["claude-code.future_block_xyz"] == 1
        rendered = render_blocks_to_markdown(result["messages"][0]["blocks"])
        assert "Unsupported content" in rendered
    def test_collapsed_placeholder_renders_grouped_line(self, tmp_path):
        _write_session(tmp_path, _records())
        p = self._provider(tmp_path)
        p.fetch_all_conversations()
        result = p.normalize_conversation(p.get_conversation("abc-123"))
        rendered = render_blocks_to_markdown(result["messages"][1]["blocks"])
        assert "> 🔧 **Tool output** — `3 calls: Read ×2, Bash ×1`" in rendered
        assert "omitted (EXPORTER_HIDDEN_CONTENT=full to keep)" in rendered
@@ -0,0 +1,277 @@
 """CLI-level tests using Click's CliRunner — no live API calls required."""
 import pytest
 from click.testing import CliRunner
 from src.cache import Cache
 from src.main import _filter_by_project, cli
 # ---------------------------------------------------------------------------
 # _filter_by_project  (T-27)
 # ---------------------------------------------------------------------------
 class TestFilterByProject:
    """Unit tests for the project filter logic used by export/list/joplin."""
    # ChatGPT conversations use the _project_name annotation key
    def _chatgpt(self, conv_id, project_name):
        return {"id": conv_id, "_project_name": project_name}
    # Claude conversations use the project dict key
    def _claude(self, conv_id, project_name):
        proj = {"name": project_name} if project_name else None
        return {"id": conv_id, "project": proj}
    def test_none_filter_keeps_no_project_chatgpt(self):
        convs = [self._chatgpt("a", None), self._chatgpt("b", "Python Course")]
        result = _filter_by_project(convs, "none")
        assert len(result) == 1
        assert result[0]["id"] == "a"
    def test_none_filter_keeps_no_project_claude(self):
        convs = [self._claude("a", None), self._claude("b", "Python Course")]
        result = _filter_by_project(convs, "none")
        assert len(result) == 1
        assert result[0]["id"] == "a"
    def test_name_filter_case_insensitive(self):
        convs = [
            self._chatgpt("a", "Python Course"),
            self._chatgpt("b", "Java Course"),
            self._chatgpt("c", None),
        ]
        result = _filter_by_project(convs, "PYTHON")
        assert len(result) == 1
        assert result[0]["id"] == "a"
    def test_name_filter_substring_match(self):
        convs = [
            self._chatgpt("a", "Python Advanced Course"),
            self._chatgpt("b", "Python Basics"),
            self._chatgpt("c", "JavaScript"),
        ]
        result = _filter_by_project(convs, "python")
        assert len(result) == 2
        assert {c["id"] for c in result} == {"a", "b"}
    def test_no_matches_returns_empty(self):
        convs = [self._chatgpt("a", "Python Course"), self._chatgpt("b", None)]
        result = _filter_by_project(convs, "ruby")
        assert result == []
    def test_none_filter_excludes_all_with_projects(self):
        convs = [self._chatgpt("a", "Project A"), self._chatgpt("b", "Project B")]
        result = _filter_by_project(convs, "none")
        assert result == []
    def test_empty_string_project_treated_as_no_project(self):
        convs = [{"id": "a", "_project_name": ""}, {"id": "b", "_project_name": "Real"}]
        result = _filter_by_project(convs, "none")
        assert len(result) == 1
        assert result[0]["id"] == "a"
    def test_claude_project_string_matched(self):
        # Claude can also have project as a plain string
        convs = [{"id": "a", "project": "python-course"}, {"id": "b", "project": None}]
        result = _filter_by_project(convs, "python")
        assert len(result) == 1
        assert result[0]["id"] == "a"
 # ---------------------------------------------------------------------------
 # export --since validation  (T-25)
 # ---------------------------------------------------------------------------
 class TestExportSinceValidation:
    """Test that --since with an invalid date exits cleanly with an error message."""
    def _pre_populated_cache(self, tmp_path) -> Cache:
        """Create a cache that passes the ToS gate and first-run doctor check."""
        cache = Cache(tmp_path)
        cache.acknowledge_tos()
        cache.mark_exported("chatgpt", "dummy-conv", {"updated_at": "2024-01-01T00:00:00Z"})
        return cache
    def test_invalid_since_date_exits_with_error(self, tmp_path):
        self._pre_populated_cache(tmp_path)
        runner = CliRunner(mix_stderr=True)
        result = runner.invoke(
            cli,
            ["--no-log-file", "export", "--since", "notadate"],
            env={
                "CHATGPT_SESSION_TOKEN": "eyJtesttoken",
                "CACHE_DIR": str(tmp_path),
                "EXPORT_DIR": str(tmp_path / "exports"),
            },
        )
        assert result.exit_code == 1
        assert "Invalid --since date" in result.output
        assert "YYYY-MM-DD" in result.output
    def test_valid_since_date_does_not_error(self, tmp_path):
        """A valid date should not produce the invalid-date error (may fail later on API)."""
        self._pre_populated_cache(tmp_path)
        runner = CliRunner(mix_stderr=True)
        result = runner.invoke(
            cli,
            ["--no-log-file", "export", "--since", "2024-01-01"],
            env={
                "CHATGPT_SESSION_TOKEN": "eyJtesttoken",
                "CACHE_DIR": str(tmp_path),
                "EXPORT_DIR": str(tmp_path / "exports"),
                # This test hits the real (failing) auth endpoint with retries;
                # don't add politeness pacing on top of the backoff sleeps.
                "REQUEST_DELAY": "0",
            },
        )
        assert "Invalid --since date" not in result.output
    def test_max_conversations_zero_rejected(self, tmp_path):
        """--max-conversations uses IntRange(min=1); 0 must be rejected by click."""
        self._pre_populated_cache(tmp_path)
        runner = CliRunner(mix_stderr=True)
        result = runner.invoke(
            cli,
            ["--no-log-file", "export", "--max-conversations", "0"],
            env={
                "CHATGPT_SESSION_TOKEN": "eyJtesttoken",
                "CACHE_DIR": str(tmp_path),
                "EXPORT_DIR": str(tmp_path / "exports"),
            },
        )
        assert result.exit_code == 2
        assert "max-conversations" in result.output
 # ---------------------------------------------------------------------------
 # LossReport summary
 # ---------------------------------------------------------------------------
 class TestLossReportSummary:
    """The LossReport's format_summary() pinned format covers zero, top-5, and overflow cases."""
    def test_zero_summary_uses_none_sentinel(self):
        from src.loss_report import LossReport
        report = LossReport()
        out = report.format_summary()
        assert "[export] Run summary:" in out
        assert "conversations:        0" in out
        assert "messages rendered:    0" in out
        # All three "(none)" sentinels present — never empty parens
        # (unknown blocks, extraction failures, collapsed by policy)
        assert out.count("(none)") == 3
    def test_top_5_breakdown(self):
        from src.loss_report import LossReport
        report = LossReport()
        for raw_type in ("a", "b", "c", "d", "e", "f", "g"):
            report.record_unknown(raw_type)
            if raw_type == "a":
                # Make 'a' the most common
                for _ in range(4):
                    report.record_unknown("a")
        out = report.format_summary()
        # Top entry shown
        assert "a=5" in out
        # Overflow line present (7 types, top 5 + 2 more)
        assert "+ 2 more types" in out
    def test_messages_and_conversations_recorded(self):
        from src.loss_report import LossReport
        report = LossReport()
        report.record_conversation()
        report.record_message()
        report.record_message()
        out = report.format_summary()
        assert "conversations:        1" in out
        assert "messages rendered:    2" in out
 # ---------------------------------------------------------------------------
 # prune command
 # ---------------------------------------------------------------------------
 class TestPrune:
    """prune deletes export files not referenced by the manifest."""
    def _setup(self, tmp_path):
        """Cache with one referenced file; one stale file + empty-dir candidate."""
        cache = Cache(tmp_path / "cache")
        cache.acknowledge_tos()
        export_dir = tmp_path / "exports"
        keep = export_dir / "chatgpt" / "proj.2026" / "keep.md"
        keep.parent.mkdir(parents=True)
        keep.write_text("kept")
        stale = export_dir / "chatgpt" / "proj" / "2026" / "old-layout.md"
        stale.parent.mkdir(parents=True)
        stale.write_text("stale")
        cache.mark_exported("chatgpt", "conv-1", {"file_path": str(keep)})
        return export_dir, keep, stale
    def _invoke(self, tmp_path, *args):
        runner = CliRunner(mix_stderr=True)
        return runner.invoke(
            cli,
            ["--no-log-file", "prune", *args],
            env={
                "CACHE_DIR": str(tmp_path / "cache"),
                "EXPORT_DIR": str(tmp_path / "exports"),
            },
        )
    def test_dry_run_lists_but_keeps_files(self, tmp_path):
        export_dir, keep, stale = self._setup(tmp_path)
        result = self._invoke(tmp_path, "--dry-run")
        assert result.exit_code == 0
        assert "old-layout.md" in result.output
        assert "Dry run" in result.output
        assert stale.exists() and keep.exists()
    def test_yes_deletes_stale_keeps_referenced_sweeps_dirs(self, tmp_path):
        export_dir, keep, stale = self._setup(tmp_path)
        result = self._invoke(tmp_path, "--yes")
        assert result.exit_code == 0
        assert not stale.exists()
        assert keep.exists()
        # Old-layout dirs are now empty and swept
        assert not (export_dir / "chatgpt" / "proj").exists()
    def test_refuses_with_empty_manifest(self, tmp_path):
        """Footgun guard: after cache --clear, prune must not wipe the archive."""
        cache = Cache(tmp_path / "cache")
        cache.acknowledge_tos()
        export_dir = tmp_path / "exports"
        f = export_dir / "chatgpt" / "a.md"
        f.parent.mkdir(parents=True)
        f.write_text("data")
        result = self._invoke(tmp_path, "--yes")
        assert result.exit_code == 1
        assert "Refusing to prune" in result.output
        assert f.exists()
    def test_aborts_without_confirmation(self, tmp_path):
        export_dir, keep, stale = self._setup(tmp_path)
        runner = CliRunner(mix_stderr=True)
        result = runner.invoke(
            cli,
            ["--no-log-file", "prune"],
            input="n\n",
            env={
                "CACHE_DIR": str(tmp_path / "cache"),
                "EXPORT_DIR": str(tmp_path / "exports"),
            },
        )
        assert result.exit_code == 0
        assert "Aborted" in result.output
        assert stale.exists()
@@ -0,0 +1,118 @@
 """Tests for src/config.py — token validation logic (T-14)."""
 import logging
 import time
 import jwt
 import pytest
 from src.config import _validate_chatgpt_token
 class TestValidateChatGPTToken:
    def test_expired_token_logs_warning(self, caplog):
        # T-14: expired JWT must produce a clear warning
        payload = {"exp": int(time.time()) - 3600}  # expired 1 hour ago
        token = jwt.encode(payload, "secret", algorithm="HS256")
        with caplog.at_level(logging.WARNING, logger="src.config"):
            result = _validate_chatgpt_token(token)
        assert any("expired" in r.message.lower() for r in caplog.records)
        assert result is not None  # still returns the expiry datetime
    def test_expiring_within_24h_logs_warning(self, caplog):
        payload = {"exp": int(time.time()) + 3600}  # expires in 1 hour
        token = jwt.encode(payload, "secret", algorithm="HS256")
        with caplog.at_level(logging.WARNING, logger="src.config"):
            _validate_chatgpt_token(token)
        assert any("less than 24 hours" in r.message for r in caplog.records)
    def test_valid_token_no_expiry_warning(self, caplog):
        payload = {"exp": int(time.time()) + 86400 * 5}  # valid for 5 days
        token = jwt.encode(payload, "secret", algorithm="HS256")
        with caplog.at_level(logging.WARNING, logger="src.config"):
            result = _validate_chatgpt_token(token)
        assert not any("expired" in r.message.lower() for r in caplog.records)
        assert result is not None
    def test_token_without_exp_claim_logs_warning(self, caplog):
        payload = {"sub": "user123"}  # no exp
        token = jwt.encode(payload, "secret", algorithm="HS256")
        with caplog.at_level(logging.WARNING, logger="src.config"):
            result = _validate_chatgpt_token(token)
        assert any("'exp'" in r.message or "no 'exp'" in r.message for r in caplog.records)
        assert result is None
    def test_jwe_encrypted_token_returns_none(self, caplog):
        # JWE tokens (alg=dir) cannot be decoded client-side — this is normal for ChatGPT
        jwe_like = "eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIn0.fake.token.data.here"
        with caplog.at_level(logging.DEBUG, logger="src.config"):
            result = _validate_chatgpt_token(jwe_like)
        assert result is None  # cannot decode, but not an error
    def test_non_jwt_string_logs_warning(self, caplog):
        with caplog.at_level(logging.WARNING, logger="src.config"):
            result = _validate_chatgpt_token("notajwttoken")
        assert any("does not look like a JWT" in r.message for r in caplog.records)
        assert result is None
 class TestSessionLimiterConfig:
    """MAX_CONVERSATIONS_PER_RUN and REQUEST_DELAY parsing in load_config."""
    def _load(self, monkeypatch, tmp_path, **env):
        from src.config import load_config
        monkeypatch.setenv("EXPORT_DIR", str(tmp_path / "exports"))
        monkeypatch.setenv("CACHE_DIR", str(tmp_path / "cache"))
        for key in (
            "MAX_CONVERSATIONS_PER_RUN",
            "REQUEST_DELAY",
            "EXPORTER_HIDDEN_CONTENT",
            "EXPORTER_DOWNLOAD_MEDIA",
        ):
            monkeypatch.delenv(key, raising=False)
        for key, value in env.items():
            monkeypatch.setenv(key, value)
        return load_config()
    def test_defaults(self, monkeypatch, tmp_path):
        cfg = self._load(monkeypatch, tmp_path)
        assert cfg.max_conversations is None
        assert cfg.request_delay == 1.0
        assert cfg.hidden_content == "placeholder"
        assert cfg.download_media == "images"
    def test_download_media_valid(self, monkeypatch, tmp_path):
        cfg = self._load(monkeypatch, tmp_path, EXPORTER_DOWNLOAD_MEDIA="all")
        assert cfg.download_media == "all"
    def test_download_media_invalid_raises(self, monkeypatch, tmp_path):
        from src.config import ConfigError
        with pytest.raises(ConfigError, match="EXPORTER_DOWNLOAD_MEDIA"):
            self._load(monkeypatch, tmp_path, EXPORTER_DOWNLOAD_MEDIA="sometimes")
    def test_valid_values(self, monkeypatch, tmp_path):
        cfg = self._load(
            monkeypatch, tmp_path,
            MAX_CONVERSATIONS_PER_RUN="25", REQUEST_DELAY="0.5",
        )
        assert cfg.max_conversations == 25
        assert cfg.request_delay == 0.5
    def test_zero_delay_allowed(self, monkeypatch, tmp_path):
        cfg = self._load(monkeypatch, tmp_path, REQUEST_DELAY="0")
        assert cfg.request_delay == 0.0
    def test_non_integer_cap_raises(self, monkeypatch, tmp_path):
        from src.config import ConfigError
        with pytest.raises(ConfigError, match="MAX_CONVERSATIONS_PER_RUN"):
            self._load(monkeypatch, tmp_path, MAX_CONVERSATIONS_PER_RUN="lots")
    def test_zero_cap_raises(self, monkeypatch, tmp_path):
        from src.config import ConfigError
        with pytest.raises(ConfigError, match="at least 1"):
            self._load(monkeypatch, tmp_path, MAX_CONVERSATIONS_PER_RUN="0")
    def test_negative_delay_raises(self, monkeypatch, tmp_path):
        from src.config import ConfigError
        with pytest.raises(ConfigError, match="REQUEST_DELAY"):
            self._load(monkeypatch, tmp_path, REQUEST_DELAY="-1")
@@ -1,4 +1,4 @@
-"""Unit tests for src/exporters/."""
+"""Unit tests for src/exporters/ and src/blocks.py."""
 import json
 import os
@@ -7,6 +7,23 @@ from pathlib import Path
 import pytest
 from src.blocks import (
    BLOCK_TYPE_TEXT,
    UNKNOWN_REASON_EXTRACTION_FAILED,
    UNKNOWN_REASON_UNKNOWN_TYPE,
    _blockquote_prefix,
    _safe_fence,
    make_code_block,
    make_file_placeholder,
    make_hidden_context_marker,
    make_image_placeholder,
    make_text_block,
    make_thinking_block,
    make_tool_result_block,
    make_tool_use_block,
    make_unknown_block,
    render_blocks_to_markdown,
 )
 from src.exporters.markdown import MarkdownExporter, _yaml_escape, _format_timestamp
 from src.exporters.json_export import JSONExporter
@@ -122,7 +139,7 @@ class TestMarkdownFilenameGeneration:
    def test_year_in_path(self, tmp_path):
        exp = MarkdownExporter(tmp_path)
        path = exp.export(SAMPLE_CONV)
-        assert "/2024/" in str(path)
+        assert ".2024/" in str(path)
    def test_output_structure_provider_project(self, tmp_path):
        exp = MarkdownExporter(tmp_path, output_structure="provider/project")
@@ -199,6 +216,34 @@ class TestJSONExporter:
        assert "  " in raw
 class TestBothFormats:
    """T-38: Markdown and JSON exporters produce matching filenames for the same conversation."""
    def test_both_formats_produce_files(self, tmp_path):
        md_exp = MarkdownExporter(tmp_path)
        json_exp = JSONExporter(tmp_path)
        md_path = md_exp.export(SAMPLE_CONV)
        json_path = json_exp.export(SAMPLE_CONV)
        assert md_path.exists()
        assert json_path.exists()
    def test_both_formats_have_matching_stems(self, tmp_path):
        md_exp = MarkdownExporter(tmp_path)
        json_exp = JSONExporter(tmp_path)
        md_path = md_exp.export(SAMPLE_CONV)
        json_path = json_exp.export(SAMPLE_CONV)
        assert md_path.suffix == ".md"
        assert json_path.suffix == ".json"
        assert md_path.stem == json_path.stem
    def test_both_formats_same_directory(self, tmp_path):
        md_exp = MarkdownExporter(tmp_path)
        json_exp = JSONExporter(tmp_path)
        md_path = md_exp.export(SAMPLE_CONV)
        json_path = json_exp.export(SAMPLE_CONV)
        assert md_path.parent == json_path.parent
 class TestYamlEscape:
    def test_escapes_double_quotes(self):
        assert _yaml_escape('Say "hello"') == 'Say \\"hello\\"'
@@ -222,3 +267,271 @@ class TestFormatTimestamp:
    def test_empty_string(self):
        assert _format_timestamp("") == ""
 # ---------------------------------------------------------------------------
 # Block helpers and rendering
 # ---------------------------------------------------------------------------
 class TestSafeFence:
    def test_minimum_three_backticks(self):
        assert _safe_fence("plain text") == "```"
    def test_four_backticks_when_three_in_content(self):
        assert _safe_fence("here ``` is a fence") == "````"
    def test_five_backticks_when_four_in_content(self):
        assert _safe_fence("here ```` is four") == "`````"
    def test_handles_empty_string(self):
        assert _safe_fence("") == "```"
    def test_handles_run_at_end(self):
        # Trailing run still counted
        assert _safe_fence("text ending in ```") == "````"
 class TestBlockquotePrefix:
    def test_single_line(self):
        assert _blockquote_prefix("hello") == "> hello"
    def test_multi_line(self):
        assert _blockquote_prefix("a\nb\nc") == "> a\n> b\n> c"
    def test_empty_lines_become_naked_quote_marker(self):
        assert _blockquote_prefix("a\n\nb") == "> a\n>\n> b"
    def test_empty_string(self):
        assert _blockquote_prefix("") == ">"
 class TestBlockConstructors:
    def test_make_text_block_returns_none_for_empty(self):
        assert make_text_block("") is None
        assert make_text_block("   ") is None
    def test_make_text_block_returns_dict(self):
        b = make_text_block("hello")
        assert b == {"type": "text", "text": "hello"}
    def test_make_code_block_returns_none_for_empty(self):
        assert make_code_block("") is None
    def test_make_thinking_block_returns_none_for_empty(self):
        assert make_thinking_block("") is None
 class TestRenderBlocks:
    def test_text_block_renders_as_paragraph(self):
        out = render_blocks_to_markdown([make_text_block("Hello world")])
        assert out == "Hello world"
    def test_blocks_separated_by_blank_line(self):
        out = render_blocks_to_markdown(
            [make_text_block("first"), make_text_block("second")]
        )
        assert out == "first\n\nsecond"
    def test_code_block_with_language(self):
        out = render_blocks_to_markdown([make_code_block("print(1)", language="python")])
        assert "```python" in out
        assert "print(1)" in out
    def test_thinking_block_uses_blockquote(self):
        out = render_blocks_to_markdown([make_thinking_block("step 1\nstep 2")])
        assert "**💭 Reasoning**" in out
        assert "> step 1" in out
        assert "> step 2" in out
    def test_tool_use_renders_as_blockquote_with_safe_fence(self):
        out = render_blocks_to_markdown(
            [make_tool_use_block("search", {"query": "test"})]
        )
        assert "> 🔧 **Tool: search**" in out
        # Every line of the body is blockquote-prefixed
        assert "> ```json" in out
        assert "> }" in out
    def test_tool_use_with_multiline_input(self):
        out = render_blocks_to_markdown(
            [make_tool_use_block("complex", {"a": 1, "b": [{"x": "y"}]})]
        )
        # Prefix every line of multi-line JSON
        for line in out.split("\n"):
            assert line.startswith(">") or line == ""
    def test_tool_result_success_uses_outbox_icon(self):
        out = render_blocks_to_markdown([make_tool_result_block("OK")])
        assert "📤 **Result**" in out
        assert "❌" not in out
    def test_tool_result_error_uses_x_icon(self):
        out = render_blocks_to_markdown([make_tool_result_block("oops", is_error=True)])
        assert "❌ **Result (error)**" in out
        assert "📤" not in out
    def test_tool_result_with_tool_name_in_header(self):
        out = render_blocks_to_markdown(
            [make_tool_result_block("done", tool_name="container.exec")]
        )
        assert "📤 **Result: container.exec**" in out
    def test_tool_result_error_with_tool_name(self):
        out = render_blocks_to_markdown(
            [make_tool_result_block("503", tool_name="web", is_error=True)]
        )
        assert "❌ **Result (error): web**" in out
    def test_tool_result_summary_renders_as_italic_line(self):
        out = render_blocks_to_markdown(
            [
                make_tool_result_block(
                    "output",
                    tool_name="container.exec",
                    summary="Reading skill documentation",
                )
            ]
        )
        # Summary line is italic, lives between header and fence,
        # all inside the blockquote prefix.
        assert "> *Reading skill documentation*" in out
        # Order: header before summary before fence
        header_idx = out.index("Result: container.exec")
        summary_idx = out.index("Reading skill documentation")
        fence_idx = out.index("output")
        assert header_idx < summary_idx < fence_idx
    def test_image_placeholder_rendering(self):
        out = render_blocks_to_markdown(
            [make_image_placeholder(ref="file-123", source="user_upload")]
        )
        assert "🖼️ **Image attached**" in out
        assert "`file-123`" in out
        assert "user_upload" in out
        assert "content not preserved" in out
    def test_file_placeholder_with_metadata(self):
        out = render_blocks_to_markdown(
            [make_file_placeholder(ref="sediment://x", mime="audio/wav", size_bytes=10240, duration_seconds=2.5)]
        )
        assert "📎 **File attached**" in out
        assert "audio/wav" in out
        assert "KB" in out
        assert "2.50s" in out
    def test_unknown_block_renders_with_keys(self):
        out = render_blocks_to_markdown(
            [
                make_unknown_block(
                    raw_type="future_x",
                    observed_keys=["foo", "bar"],
                    reason=UNKNOWN_REASON_UNKNOWN_TYPE,
                )
            ]
        )
        assert "⚠️ **Unsupported content**" in out
        assert "future_x" in out
        assert "`foo`" in out
        assert "`bar`" in out
    def test_unknown_extraction_failed_includes_summary(self):
        out = render_blocks_to_markdown(
            [
                make_unknown_block(
                    raw_type="audio_transcription",
                    observed_keys=["asset_pointer"],
                    reason=UNKNOWN_REASON_EXTRACTION_FAILED,
                    summary="expected key 'text' not found",
                )
            ]
        )
        assert "extraction_failed" in out
        assert "expected key 'text' not found" in out
    def test_hidden_context_marker(self):
        out = render_blocks_to_markdown(
            [make_hidden_context_marker("user_editable_context")]
        )
        assert "ℹ️ **Hidden context**" in out
        assert "`user_editable_context`" in out
    def test_safe_fence_prevents_runaway_code_block(self):
        # Content contains an unbalanced opening fence — without _safe_fence
        # this would corrupt downstream rendering.
        evil_content = "before\n```Follow\ntext\nraw is: \"```"
        block = make_code_block(evil_content)
        out = render_blocks_to_markdown([block, make_text_block("after")])
        # The 4-backtick wrap should be present
        assert "````" in out
        # The "after" text should appear OUTSIDE any code block — it follows
        # the closing ```` fence.
        assert out.endswith("after")
    def test_block_order_preserved(self):
        blocks = [
            make_text_block("a"),
            make_image_placeholder(ref="r1", source="user_upload"),
            make_text_block("b"),
        ]
        out = render_blocks_to_markdown(blocks)
        assert out.index("a") < out.index("Image attached")
        assert out.index("Image attached") < out.index("b")
 # ---------------------------------------------------------------------------
 # Markdown exporter with blocks
 # ---------------------------------------------------------------------------
 SAMPLE_CONV_BLOCKS = {
    "id": "blocks12345",
    "title": "Blocks Conversation",
    "provider": "claude",
    "project": None,
    "created_at": "2024-06-10T14:32:00Z",
    "updated_at": "2024-06-10T15:00:00Z",
    "message_count": 1,
    "messages": [
        {
            "role": "assistant",
            "content_type": "text",
            "timestamp": None,
            "blocks": [
                {"type": "text", "text": "Here is the answer."},
                {"type": "tool_use", "name": "search", "input": {"q": "x"}, "tool_id": "t1"},
            ],
        }
    ],
 }
 class TestMarkdownExporterWithBlocks:
    def test_renders_blocks(self, tmp_path):
        exp = MarkdownExporter(tmp_path)
        path = exp.export(SAMPLE_CONV_BLOCKS)
        body = path.read_text()
        assert "Here is the answer." in body
        assert "🔧 **Tool: search**" in body
    def test_falls_back_to_content_when_blocks_missing(self, tmp_path):
        # Backward-compat: messages with `content` only (no `blocks`) still render.
        exp = MarkdownExporter(tmp_path)
        path = exp.export(SAMPLE_CONV)  # SAMPLE_CONV has content only, no blocks
        body = path.read_text()
        assert "Hello, how are you?" in body
    def test_skips_messages_with_neither_blocks_nor_content(self, tmp_path):
        conv = {
            **SAMPLE_CONV_BLOCKS,
            "messages": [
                {"role": "user", "content_type": "text", "timestamp": None, "blocks": []},
                {"role": "assistant", "content_type": "text", "timestamp": None, "blocks": [
                    {"type": "text", "text": "I am here."}
                ]},
            ],
        }
        exp = MarkdownExporter(tmp_path)
        path = exp.export(conv)
        body = path.read_text()
        assert "I am here." in body
@@ -0,0 +1,434 @@
 """Unit tests for src/joplin.py (JoplinClient)."""
 from unittest.mock import MagicMock, patch
 import pytest
 import requests
 from src.joplin import JoplinClient, JoplinError, _http_error_message, _timeout_message, notebook_path
 # ---------------------------------------------------------------------------
 # Helpers
 # ---------------------------------------------------------------------------
 def _make_client() -> JoplinClient:
    return JoplinClient(base_url="http://localhost:41184", token="test-token")
 def _mock_response(json_data=None, text="", status_code=200):
    resp = MagicMock()
    resp.status_code = status_code
    resp.text = text
    resp.json.return_value = json_data or {}
    resp.raise_for_status = MagicMock()
    if status_code >= 400:
        resp.raise_for_status.side_effect = requests.exceptions.HTTPError(
            response=resp
        )
    return resp
 # ---------------------------------------------------------------------------
 # notebook_path helper
 # ---------------------------------------------------------------------------
 class TestNotebookPath:
    def test_no_project(self):
        assert notebook_path("chatgpt", None) == ("AI-ChatGPT", "No Project")
    def test_no_project_string(self):
        assert notebook_path("chatgpt", "no-project") == ("AI-ChatGPT", "No Project")
    def test_project_with_hyphens(self):
        assert notebook_path("chatgpt", "my-project") == ("AI-ChatGPT", "My Project")
    def test_claude_provider(self):
        assert notebook_path("claude", "budget-tracker") == ("AI-Claude", "Budget Tracker")
    def test_multi_word_project(self):
        assert notebook_path("claude", "ai-research-notes") == ("AI-Claude", "Ai Research Notes")
    def test_returns_tuple(self):
        result = notebook_path("chatgpt", "some-project")
        assert isinstance(result, tuple) and len(result) == 2
 # ---------------------------------------------------------------------------
 # ping
 # ---------------------------------------------------------------------------
 class TestPing:
    def test_ping_success(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.return_value = _mock_response(text="JoplinClipperServer")
            assert client.ping() is True
    def test_ping_not_joplin(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.return_value = _mock_response(text="SomeOtherServer")
            assert client.ping() is False
    def test_ping_connection_refused(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.side_effect = requests.exceptions.ConnectionError()
            assert client.ping() is False
    def test_ping_timeout_returns_false(self):
        """Ping timeout is not an error — Joplin just isn't responding."""
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.side_effect = requests.exceptions.Timeout()
            assert client.ping() is False
    def test_ping_invalid_url_raises_joplin_error(self):
        """Non-connection, non-timeout errors (e.g. invalid URL) surface as JoplinError."""
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.side_effect = requests.exceptions.InvalidURL("bad url")
            with pytest.raises(JoplinError):
                client.ping()
 class TestValidateToken:
    def test_validate_token_success(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.return_value = _mock_response(json_data={"items": [], "has_more": False})
            client.validate_token()  # should not raise
    def test_validate_token_401_raises_joplin_error(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.return_value = _mock_response(status_code=401)
            with pytest.raises(JoplinError, match="401"):
                client.validate_token()
 class TestTimeoutMessage:
    def test_includes_timeout_duration(self):
        import src.joplin as joplin_module
        msg = _timeout_message("POST", "/notes")
        assert "POST" in msg
        assert "/notes" in msg
        assert str(joplin_module._REQUEST_TIMEOUT) in msg
    def test_includes_actionable_hints(self):
        msg = _timeout_message("PUT", "/notes/abc")
        assert "JOPLIN_REQUEST_TIMEOUT" in msg
        # Should mention at least one cause
        assert "large" in msg.lower() or "busy" in msg.lower() or "frozen" in msg.lower()
 class TestTimeoutHandling:
    def test_get_timeout_raises_joplin_error_with_clear_message(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.side_effect = requests.exceptions.Timeout()
            with pytest.raises(JoplinError) as exc_info:
                client._get("/folders")
        assert "timed out" in str(exc_info.value).lower()
        assert "JOPLIN_REQUEST_TIMEOUT" in str(exc_info.value)
    def test_post_timeout_raises_joplin_error_with_clear_message(self):
        client = _make_client()
        with patch("requests.post") as mock_post:
            mock_post.side_effect = requests.exceptions.Timeout()
            with pytest.raises(JoplinError) as exc_info:
                client._post("/notes", {"title": "Test"})
        assert "timed out" in str(exc_info.value).lower()
    def test_put_timeout_raises_joplin_error_with_clear_message(self):
        client = _make_client()
        with patch("requests.put") as mock_put:
            mock_put.side_effect = requests.exceptions.Timeout()
            with pytest.raises(JoplinError) as exc_info:
                client._put("/notes/abc", {"title": "Test"})
        assert "timed out" in str(exc_info.value).lower()
    def test_create_note_timeout_propagates(self):
        """Timeout on create_note surfaces as JoplinError, not raw requests exception."""
        client = _make_client()
        with patch("requests.post") as mock_post:
            mock_post.side_effect = requests.exceptions.Timeout()
            with pytest.raises(JoplinError, match="timed out"):
                client.create_note("Big Note", "x" * 100_000, "nb-123")
    def test_update_note_timeout_propagates(self):
        client = _make_client()
        with patch("requests.put") as mock_put:
            mock_put.side_effect = requests.exceptions.Timeout()
            with pytest.raises(JoplinError, match="timed out"):
                client.update_note("note-id", "Big Note", "x" * 100_000)
 class TestHttpErrorMessage:
    def test_401_gives_token_hint(self):
        resp = MagicMock()
        resp.status_code = 401
        resp.text = "Unauthorized"
        e = requests.exceptions.HTTPError(response=resp)
        msg = _http_error_message("GET", "/folders", e)
        assert "401" in msg
        assert "token" in msg.lower()
    def test_404_gives_deleted_note_hint(self):
        resp = MagicMock()
        resp.status_code = 404
        resp.text = "Not Found"
        e = requests.exceptions.HTTPError(response=resp)
        msg = _http_error_message("PUT", "/notes/abc", e)
        assert "404" in msg
        assert "deleted" in msg.lower()
    def test_other_error_includes_status_and_body(self):
        resp = MagicMock()
        resp.status_code = 500
        resp.text = "Internal Server Error"
        e = requests.exceptions.HTTPError(response=resp)
        msg = _http_error_message("POST", "/notes", e)
        assert "500" in msg
 # ---------------------------------------------------------------------------
 # list_notebooks
 # ---------------------------------------------------------------------------
 class TestListNotebooks:
    def test_list_notebooks_single_page(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.return_value = _mock_response(
                json_data={"items": [{"id": "nb1", "title": "ChatGPT - No Project"}], "has_more": False}
            )
            result = client.list_notebooks()
        assert len(result) == 1
        assert result[0]["id"] == "nb1"
    def test_list_notebooks_paginated(self):
        client = _make_client()
        page1 = _mock_response(
            json_data={"items": [{"id": "nb1", "title": "A"}], "has_more": True}
        )
        page2 = _mock_response(
            json_data={"items": [{"id": "nb2", "title": "B"}], "has_more": False}
        )
        with patch("requests.get") as mock_get:
            mock_get.side_effect = [page1, page2]
            result = client.list_notebooks()
        assert len(result) == 2
        assert {nb["id"] for nb in result} == {"nb1", "nb2"}
    def test_list_notebooks_connection_error(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.side_effect = requests.exceptions.ConnectionError()
            with pytest.raises(JoplinError, match="Joplin"):
                client.list_notebooks()
 # ---------------------------------------------------------------------------
 # get_or_create_notebook
 # ---------------------------------------------------------------------------
 class TestGetOrCreateNotebook:
    def test_returns_existing_root_notebook_id(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.return_value = _mock_response(
                json_data={
                    "items": [{"id": "nb-existing", "title": "AI-ChatGPT", "parent_id": ""}],
                    "has_more": False,
                }
            )
            nb_id = client.get_or_create_notebook("AI-ChatGPT")
        assert nb_id == "nb-existing"
    def test_returns_existing_child_notebook_id(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.return_value = _mock_response(
                json_data={
                    "items": [{"id": "nb-child", "title": "No Project", "parent_id": "nb-parent"}],
                    "has_more": False,
                }
            )
            nb_id = client.get_or_create_notebook("No Project", parent_id="nb-parent")
        assert nb_id == "nb-child"
    def test_creates_new_notebook_when_not_found(self):
        client = _make_client()
        with patch("requests.get") as mock_get, patch("requests.post") as mock_post:
            mock_get.return_value = _mock_response(
                json_data={"items": [], "has_more": False}
            )
            mock_post.return_value = _mock_response(
                json_data={"id": "nb-new", "title": "AI-ChatGPT"}
            )
            nb_id = client.get_or_create_notebook("AI-ChatGPT")
        assert nb_id == "nb-new"
        mock_post.assert_called_once()
    def test_creates_child_notebook_with_parent_id(self):
        client = _make_client()
        with patch("requests.get") as mock_get, patch("requests.post") as mock_post:
            mock_get.return_value = _mock_response(
                json_data={"items": [], "has_more": False}
            )
            mock_post.return_value = _mock_response(
                json_data={"id": "nb-child", "title": "My Project"}
            )
            nb_id = client.get_or_create_notebook("My Project", parent_id="nb-parent")
        assert nb_id == "nb-child"
        _, kwargs = mock_post.call_args
        assert kwargs["json"]["parent_id"] == "nb-parent"
    def test_does_not_include_parent_id_for_root(self):
        client = _make_client()
        with patch("requests.get") as mock_get, patch("requests.post") as mock_post:
            mock_get.return_value = _mock_response(json_data={"items": [], "has_more": False})
            mock_post.return_value = _mock_response(json_data={"id": "nb-root", "title": "AI-Claude"})
            client.get_or_create_notebook("AI-Claude")
        _, kwargs = mock_post.call_args
        assert "parent_id" not in kwargs["json"]
    def test_caches_notebook_after_first_load(self):
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.return_value = _mock_response(
                json_data={
                    "items": [{"id": "nb1", "title": "AI-Claude", "parent_id": ""}],
                    "has_more": False,
                }
            )
            # Call twice — GET /folders should only happen once
            client.get_or_create_notebook("AI-Claude")
            client.get_or_create_notebook("AI-Claude")
        assert mock_get.call_count == 1
    def test_different_parent_ids_are_distinct_cache_entries(self):
        """Same title under different parents are different notebooks."""
        client = _make_client()
        with patch("requests.get") as mock_get:
            mock_get.return_value = _mock_response(
                json_data={
                    "items": [
                        {"id": "nb-a", "title": "No Project", "parent_id": "parent-chatgpt"},
                        {"id": "nb-b", "title": "No Project", "parent_id": "parent-claude"},
                    ],
                    "has_more": False,
                }
            )
            id_a = client.get_or_create_notebook("No Project", parent_id="parent-chatgpt")
            id_b = client.get_or_create_notebook("No Project", parent_id="parent-claude")
        assert id_a == "nb-a"
        assert id_b == "nb-b"
 class TestGetOrCreateNotebookPath:
    def test_creates_two_level_path(self):
        client = _make_client()
        with patch("requests.get") as mock_get, patch("requests.post") as mock_post:
            mock_get.return_value = _mock_response(json_data={"items": [], "has_more": False})
            mock_post.side_effect = [
                _mock_response(json_data={"id": "nb-parent", "title": "AI-ChatGPT"}),
                _mock_response(json_data={"id": "nb-child", "title": "No Project"}),
            ]
            leaf_id = client.get_or_create_notebook_path(["AI-ChatGPT", "No Project"])
        assert leaf_id == "nb-child"
        assert mock_post.call_count == 2
        # Second POST should use the parent's ID
        _, kwargs = mock_post.call_args_list[1]
        assert kwargs["json"]["parent_id"] == "nb-parent"
    def test_reuses_existing_parent_for_new_child(self):
        client = _make_client()
        with patch("requests.get") as mock_get, patch("requests.post") as mock_post:
            mock_get.return_value = _mock_response(
                json_data={
                    "items": [{"id": "nb-parent", "title": "AI-Claude", "parent_id": ""}],
                    "has_more": False,
                }
            )
            mock_post.return_value = _mock_response(
                json_data={"id": "nb-child", "title": "Budget Tracker"}
            )
            leaf_id = client.get_or_create_notebook_path(["AI-Claude", "Budget Tracker"])
        assert leaf_id == "nb-child"
        # Only one POST — the parent already existed
        assert mock_post.call_count == 1
        _, kwargs = mock_post.call_args
        assert kwargs["json"]["parent_id"] == "nb-parent"
 # ---------------------------------------------------------------------------
 # create_note
 # ---------------------------------------------------------------------------
 class TestCreateNote:
    def test_create_note_returns_id(self):
        client = _make_client()
        with patch("requests.post") as mock_post:
            mock_post.return_value = _mock_response(
                json_data={"id": "note-123", "title": "My Note"}
            )
            note_id = client.create_note("My Note", "Note body", "nb-456")
        assert note_id == "note-123"
        _, kwargs = mock_post.call_args
        assert kwargs["json"]["title"] == "My Note"
        assert kwargs["json"]["body"] == "Note body"
        assert kwargs["json"]["parent_id"] == "nb-456"
    def test_create_note_connection_error(self):
        client = _make_client()
        with patch("requests.post") as mock_post:
            mock_post.side_effect = requests.exceptions.ConnectionError()
            with pytest.raises(JoplinError, match="Joplin"):
                client.create_note("Title", "Body", "nb-id")
    def test_create_note_http_error(self):
        client = _make_client()
        with patch("requests.post") as mock_post:
            mock_post.return_value = _mock_response(status_code=401)
            with pytest.raises(JoplinError):
                client.create_note("Title", "Body", "nb-id")
 # ---------------------------------------------------------------------------
 # update_note
 # ---------------------------------------------------------------------------
 class TestUpdateNote:
    def test_update_note_calls_put(self):
        client = _make_client()
        with patch("requests.put") as mock_put:
            mock_put.return_value = _mock_response(json_data={"id": "note-123"})
            client.update_note("note-123", "New Title", "New Body")
        mock_put.assert_called_once()
        _, kwargs = mock_put.call_args
        assert kwargs["json"]["title"] == "New Title"
        assert kwargs["json"]["body"] == "New Body"
    def test_update_note_connection_error(self):
        client = _make_client()
        with patch("requests.put") as mock_put:
            mock_put.side_effect = requests.exceptions.ConnectionError()
            with pytest.raises(JoplinError, match="Joplin"):
                client.update_note("note-id", "Title", "Body")
    def test_update_note_http_error(self):
        client = _make_client()
        with patch("requests.put") as mock_put:
            mock_put.return_value = _mock_response(status_code=404)
            with pytest.raises(JoplinError):
                client.update_note("note-id", "Title", "Body")
@@ -0,0 +1,248 @@
 """Tests for media downloads and Joplin resource rewriting."""
 from pathlib import Path
 import pytest
 from src.blocks import (
    make_file_placeholder,
    make_image_placeholder,
    render_blocks_to_markdown,
 )
 from src.loss_report import LossReport
 from src.media import resolve_media, resolve_media_policy
 from src.providers.base import ProviderError
 from src.providers.chatgpt import parse_asset_file_id
 # ---------------------------------------------------------------------------
 # Asset reference parsing
 # ---------------------------------------------------------------------------
 class TestParseAssetFileId:
    def test_plain_sediment(self):
        assert parse_asset_file_id("sediment://file_00000000245c71fda5") == "file_00000000245c71fda5"
    def test_generated_image_with_hash_and_page(self):
        ref = "sediment://8456107fc383a53#file_00000000979c71f685#p_6.png"
        assert parse_asset_file_id(ref) == "file_00000000979c71f685"
    def test_file_service_scheme(self):
        assert parse_asset_file_id("file-service://file-AbCdEf") == "file-AbCdEf"
    def test_unrecognised(self):
        assert parse_asset_file_id("https://example.com/x.png") is None
        assert parse_asset_file_id("") is None
        assert parse_asset_file_id(None) is None
 # ---------------------------------------------------------------------------
 # Media policy
 # ---------------------------------------------------------------------------
 class TestMediaPolicy:
    def test_default(self, monkeypatch):
        monkeypatch.delenv("EXPORTER_DOWNLOAD_MEDIA", raising=False)
        assert resolve_media_policy() == "images"
    def test_valid(self, monkeypatch):
        monkeypatch.setenv("EXPORTER_DOWNLOAD_MEDIA", "all")
        assert resolve_media_policy() == "all"
    def test_invalid_falls_back(self, monkeypatch, caplog):
        monkeypatch.setenv("EXPORTER_DOWNLOAD_MEDIA", "bogus")
        assert resolve_media_policy() == "images"
 # ---------------------------------------------------------------------------
 # resolve_media
 # ---------------------------------------------------------------------------
 class _FakeProvider:
    """Minimal provider exposing download_asset + the ref parser."""
    def __init__(self, assets=None, fail_refs=None):
        self._assets = assets or {}
        self._fail_refs = fail_refs or {}
        self.calls = []
    def parse_asset_file_id(self, ref):
        return parse_asset_file_id(ref)
    def download_asset(self, ref):
        self.calls.append(ref)
        if ref in self._fail_refs:
            raise ProviderError("chatgpt", "download_asset", self._fail_refs[ref])
        return self._assets[ref]  # (content, mime, file_name)
 def _conv_with(blocks):
    return {
        "id": "conv-1",
        "title": "Has Media",
        "provider": "chatgpt",
        "project": None,
        "created_at": "2026-05-20T00:00:00+00:00",
        "messages": [{"role": "user", "blocks": blocks}],
    }
 class TestResolveMedia:
    def test_downloads_image_and_inlines(self, tmp_path):
        ref = "sediment://file_img1"
        provider = _FakeProvider({ref: (b"\x89PNG\r\n", "image/png", "x.png")})
        block = make_image_placeholder(ref=ref, source="user_upload")
        conv = _conv_with([block])
        report = LossReport()
        n = resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
        assert n == 1
        assert report.media_downloaded == 1
        assert block["local_path"] == "media/file_img1.png"
        rendered = render_blocks_to_markdown([block])
        assert rendered == "![user_upload](media/file_img1.png)"
        # File written under the conversation's media/ dir
        written = list(tmp_path.rglob("media/file_img1.png"))
        assert written and written[0].read_bytes() == b"\x89PNG\r\n"
    def test_images_policy_skips_files(self, tmp_path):
        ref = "sediment://file_audio1"
        provider = _FakeProvider({ref: (b"RIFF", "audio/wav", "a.wav")})
        block = make_file_placeholder(ref=ref, mime="audio/wav", size_bytes=1000)
        conv = _conv_with([block])
        report = LossReport()
        n = resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
        assert n == 0
        assert "local_path" not in block
        assert provider.calls == []
    def test_all_policy_downloads_files(self, tmp_path):
        ref = "sediment://file_audio1"
        provider = _FakeProvider({ref: (b"RIFFdata", "audio/wav", "a.wav")})
        block = make_file_placeholder(ref=ref, mime="audio/wav", size_bytes=8)
        conv = _conv_with([block])
        report = LossReport()
        n = resolve_media(conv, provider, tmp_path, "provider/project/year", "all", report)
        assert n == 1
        assert block["local_path"] == "media/file_audio1.wav"
        rendered = render_blocks_to_markdown([block])
        assert "media/file_audio1.wav" in rendered and rendered.startswith("> 📎")
    def test_off_policy_noop(self, tmp_path):
        provider = _FakeProvider({"sediment://file_x": (b"x", "image/png", None)})
        block = make_image_placeholder(ref="sediment://file_x", source="user_upload")
        conv = _conv_with([block])
        report = LossReport()
        assert resolve_media(conv, provider, tmp_path, "provider/project/year", "off", report) == 0
        assert provider.calls == []
    def test_idempotent_uses_disk(self, tmp_path):
        ref = "sediment://file_img1"
        provider = _FakeProvider({ref: (b"\x89PNG", "image/png", "x.png")})
        conv = _conv_with([make_image_placeholder(ref=ref, source="user_upload")])
        report = LossReport()
        resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
        assert len(provider.calls) == 1
        # Second run, fresh blocks: file already on disk → no new API call.
        conv2 = _conv_with([make_image_placeholder(ref=ref, source="user_upload")])
        resolve_media(conv2, provider, tmp_path, "provider/project/year", "images", LossReport())
        assert len(provider.calls) == 1  # unchanged
        assert conv2["messages"][0]["blocks"][0]["local_path"] == "media/file_img1.png"
    def test_failure_keeps_placeholder_and_counts(self, tmp_path):
        ref = "sediment://file_gone"
        provider = _FakeProvider(fail_refs={ref: RuntimeError("Signed URL returned HTTP 404")})
        block = make_image_placeholder(ref=ref, source="model_generated")
        conv = _conv_with([block])
        report = LossReport()
        n = resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
        assert n == 0
        assert "local_path" not in block
        assert report.media_failed["expired-or-missing"] == 1
        # Still renders as a placeholder, not a broken image link
        assert render_blocks_to_markdown([block]).startswith("> 🖼️")
    def test_provider_without_download_asset(self, tmp_path):
        """claude-code has no remote assets — resolve_media must no-op."""
        class NoDownload:
            pass
        block = make_image_placeholder(ref="sediment://file_x", source="user_upload")
        conv = _conv_with([block])
        assert resolve_media(conv, NoDownload(), tmp_path, "provider/project/year", "images", LossReport()) == 0
 # ---------------------------------------------------------------------------
 # Joplin media rewriting
 # ---------------------------------------------------------------------------
 class _FakeJoplin:
    def __init__(self):
        self.uploaded = []
        self._n = 0
    def create_resource(self, file_path, title=None):
        self.uploaded.append(Path(file_path).name)
        self._n += 1
        return f"res{self._n}"
 class TestUploadMediaAndRewrite:
    def _note(self, tmp_path, body):
        media = tmp_path / "media"
        media.mkdir()
        (media / "file_img1.png").write_bytes(b"\x89PNG")
        (media / "clip.wav").write_bytes(b"RIFF")
        return body
    def test_rewrites_image_and_file_links(self, tmp_path):
        from src.joplin import upload_media_and_rewrite
        body = self._note(
            tmp_path,
            "Look: ![user_upload](media/file_img1.png)\n"
            "> 📎 **File attached** — [clip.wav](media/clip.wav) (audio/wav)",
        )
        client = _FakeJoplin()
        new_body, res_map = upload_media_and_rewrite(body, tmp_path, client, {})
        assert "![user_upload](:/res1)" in new_body
        assert "(:/res2)" in new_body
        assert "media/" not in new_body
        assert res_map == {"media/file_img1.png": "res1", "media/clip.wav": "res2"}
        assert len(client.uploaded) == 2
    def test_reuses_known_resource_ids(self, tmp_path):
        from src.joplin import upload_media_and_rewrite
        body = self._note(tmp_path, "![x](media/file_img1.png)")
        client = _FakeJoplin()
        existing = {"media/file_img1.png": "existing-res"}
        new_body, res_map = upload_media_and_rewrite(body, tmp_path, client, existing)
        assert "(:/existing-res)" in new_body
        assert client.uploaded == []  # no re-upload
        assert res_map == existing
    def test_missing_file_left_as_link(self, tmp_path):
        from src.joplin import upload_media_and_rewrite
        (tmp_path / "media").mkdir()
        body = "![x](media/gone.png)"
        client = _FakeJoplin()
        new_body, res_map = upload_media_and_rewrite(body, tmp_path, client, {})
        assert new_body == body
        assert res_map == {}
        assert client.uploaded == []
    def test_no_media_links_untouched(self, tmp_path):
        from src.joplin import upload_media_and_rewrite
        body = "Just text with [a link](https://example.com) and `media/foo` in code."
        client = _FakeJoplin()
        new_body, res_map = upload_media_and_rewrite(body, tmp_path, client, {})
        assert new_body == body
        assert res_map == {}
@@ -1,27 +1,63 @@
 """Unit tests for src/providers/ using fixture files."""
 import json
 import logging
 from pathlib import Path
 import pytest
 from src.blocks import (
    BLOCK_TYPE_FILE_PLACEHOLDER,
    BLOCK_TYPE_HIDDEN_CONTEXT_MARKER,
    BLOCK_TYPE_IMAGE_PLACEHOLDER,
    BLOCK_TYPE_TEXT,
    BLOCK_TYPE_THINKING,
    BLOCK_TYPE_TOOL_RESULT,
    BLOCK_TYPE_TOOL_USE,
    BLOCK_TYPE_UNKNOWN,
    render_blocks_to_markdown,
 )
 from src.loss_report import LossReport
 FIXTURES = Path(__file__).parent / "fixtures"
 # ---------------------------------------------------------------------------
 # Helpers
 # ---------------------------------------------------------------------------
 def _block_types(message: dict) -> list[str]:
    return [b.get("type") for b in (message.get("blocks") or [])]
 def _first_block(message: dict, block_type: str) -> dict | None:
    for b in message.get("blocks") or []:
        if b.get("type") == block_type:
            return b
    return None
 # ---------------------------------------------------------------------------
 # ChatGPT
 # ---------------------------------------------------------------------------
 class TestChatGPTNormalization:
-    """Test ChatGPTProvider.normalize_conversation() using fixture data."""
+    """ChatGPT normalize_conversation block-extraction behavior."""
    def _get_provider(self):
        from src.providers.chatgpt import ChatGPTProvider
        import unittest.mock as mock
        # Bypass __init__ token check
        p = ChatGPTProvider.__new__(ChatGPTProvider)
        import requests
        p._session = requests.Session()
        p._org_id = None
        p._project_ids = []
        p._project_map = {}
        p._project_name_cache = {}
        return p
-    def test_normalizes_with_project(self):
+    def test_normalizes_conversation(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
@@ -29,7 +65,7 @@ class TestChatGPTNormalization:
        assert result["id"] == "chatgpt-conv-001"
        assert result["title"] == "Python Async Tutorial"
        assert result["provider"] == "chatgpt"
-        assert result["project"] == "Learning Python"
+        assert result["project"] is None
        assert result["created_at"] != ""
        assert result["updated_at"] != ""
        assert isinstance(result["messages"], list)
@@ -42,33 +78,144 @@ class TestChatGPTNormalization:
        assert result["project"] is None
        assert result["id"] == "chatgpt-conv-002"
-    def test_extracts_text_messages(self):
+    def test_normalizes_with_project_from_map(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        p._project_map["chatgpt-conv-001"] = "My Research Project"
        result = p.normalize_conversation(raw)
        assert result["project"] == "My Research Project"
    def test_text_message_emits_text_block(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        assert len(result["messages"]) >= 2
        user_msgs = [m for m in result["messages"] if m["role"] == "user"]
-        assert any("async" in m["content"].lower() for m in user_msgs)
+        # The "How does async/await..." message
        async_msgs = [
            m for m in user_msgs
            if any(
                "async" in (b.get("text") or "").lower()
                for b in (m.get("blocks") or [])
            )
        ]
        assert async_msgs, "expected a user message about async/await"
        assert _block_types(async_msgs[0]) == [BLOCK_TYPE_TEXT]
-    def test_skips_non_text_content_with_warning(self, caplog):
+    def test_code_block_preserved_with_language(self):
        import logging
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
-        with caplog.at_level(logging.WARNING):
+        result = p.normalize_conversation(raw)
-            result = p.normalize_conversation(raw)
+
-        # The fixture has an image_asset_pointer node — should be warned about
+        assistant_msgs = [m for m in result["messages"] if m["role"] == "assistant"]
-        assert any(
+        # The first assistant message is the async/await answer with a python fence
-            "image_asset_pointer" in r.message or "rich content" in r.message
+        text_block = _first_block(assistant_msgs[0], BLOCK_TYPE_TEXT)
-            for r in caplog.records
+        assert text_block is not None
-        )
+        assert "```python" in text_block["text"]
    def test_multimodal_voice_user_message(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        # node-mm-user: audio_transcription "What is the capital of France?"
        # + real_time_user_audio_video_asset_pointer wrapping a sediment:// URL
        capital_msgs = [
            m for m in result["messages"]
            if any(
                "capital of france" in (b.get("text") or "").lower()
                for b in (m.get("blocks") or [])
            )
        ]
        assert capital_msgs, "expected the audio_transcription text to surface"
        types = _block_types(capital_msgs[0])
        assert BLOCK_TYPE_TEXT in types
        assert BLOCK_TYPE_FILE_PLACEHOLDER in types
        file_block = _first_block(capital_msgs[0], BLOCK_TYPE_FILE_PLACEHOLDER)
        assert file_block["ref"].startswith("sediment://")
        assert file_block["mime"] == "audio/wav"
        assert file_block["size_bytes"] == 50000
        assert file_block["duration_seconds"] == pytest.approx(2.5)
    def test_multimodal_voice_reverse_order_preserved(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        # node-mm-user-rev has parts in REVERSE order: asset first, transcription second.
        rev_msgs = [
            m for m in result["messages"]
            if any(
                "tell me more" in (b.get("text") or "").lower()
                for b in (m.get("blocks") or [])
            )
        ]
        assert rev_msgs, "expected the reverse-order voice message"
        types = _block_types(rev_msgs[0])
        # Order preserved: file_placeholder before text
        assert types == [BLOCK_TYPE_FILE_PLACEHOLDER, BLOCK_TYPE_TEXT]
    def test_image_only_user_message_renders(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        image_msgs = [
            m for m in result["messages"]
            if any(b.get("type") == BLOCK_TYPE_IMAGE_PLACEHOLDER for b in (m.get("blocks") or []))
        ]
        assert image_msgs, "image-only user message should now render"
    def test_user_editable_context_emits_blocks(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        p._hidden_content = "full"
        result = p.normalize_conversation(raw)
        # The user_editable_context message has user_profile + user_instructions.
        # It should now appear (was silently dropped pre-v0.4.0).
        uec_msgs = [
            m for m in result["messages"]
            if any(
                "Custom Instructions" in (b.get("text") or "")
                for b in (m.get("blocks") or [])
            )
        ]
        assert uec_msgs, "user_editable_context should be visible in output"
        # Hidden context marker should be prepended.
        assert uec_msgs[0]["blocks"][0]["type"] == BLOCK_TYPE_HIDDEN_CONTEXT_MARKER
    def test_user_editable_context_uses_safe_fence(self):
        """The user_instructions value contains embedded triple-backticks; the rendered
        Markdown must use a fence longer than 3 backticks so embedded fences are inert.
        """
        from src.blocks import render_blocks_to_markdown
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        p._hidden_content = "full"
        result = p.normalize_conversation(raw)
        uec_msgs = [
            m for m in result["messages"]
            if any(
                "Custom Instructions" in (b.get("text") or "")
                for b in (m.get("blocks") or [])
            )
        ]
        assert uec_msgs
        rendered = render_blocks_to_markdown(uec_msgs[0]["blocks"])
        # Content has ``` inside, so the wrap fence must be at least 4 backticks.
        assert "````" in rendered, "expected a 4+ backtick safe-fence wrap"
    def test_message_roles_are_valid(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        for msg in result["messages"]:
-            assert msg["role"] in ("user", "assistant", "system")
+            assert msg["role"] in ("user", "assistant", "system", "tool")
    def test_message_count_matches(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
@@ -76,16 +223,312 @@ class TestChatGPTNormalization:
        result = p.normalize_conversation(raw)
        assert result["message_count"] == len(result["messages"])
-    def test_code_fence_preserved(self):
+    def test_loss_report_records_messages(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
-        result = p.normalize_conversation(raw)
+        report = LossReport()
-        all_content = " ".join(m["content"] for m in result["messages"])
+        result = p.normalize_conversation(raw, report)
-        assert "```python" in all_content
+        assert report.messages_rendered == len(result["messages"])
        assert report.conversations == 1
 class TestChatGPTUnknownContent:
    """Unrecognised content types should produce visible unknown blocks + WARNING + tally."""
    def _get_provider(self):
        from src.providers.chatgpt import ChatGPTProvider
        p = ChatGPTProvider.__new__(ChatGPTProvider)
        import requests
        p._session = requests.Session()
        p._org_id = None
        p._project_ids = []
        p._project_map = {}
        p._project_name_cache = {}
        return p
    def _make_unknown_conv(self):
        return {
            "id": "test-unknown",
            "title": "Test",
            "create_time": 1700000000.0,
            "update_time": 1700000001.0,
            "mapping": {
                "root": {"id": "root", "message": None, "parent": None, "children": ["msg1"]},
                "msg1": {
                    "id": "msg1",
                    "message": {
                        "id": "msg1",
                        "author": {"role": "user"},
                        "content": {
                            "content_type": "future_unknown_type_xyz",
                            "some_field": "value",
                        },
                    },
                    "parent": "root",
                    "children": [],
                },
            },
        }
    def test_unknown_content_type_produces_unknown_block(self):
        p = self._get_provider()
        result = p.normalize_conversation(self._make_unknown_conv())
        assert any(
            b.get("type") == BLOCK_TYPE_UNKNOWN
            for m in result["messages"]
            for b in (m.get("blocks") or [])
        )
    def test_unknown_content_type_logs_warning(self, caplog):
        p = self._get_provider()
        with caplog.at_level(logging.WARNING):
            p.normalize_conversation(self._make_unknown_conv())
        assert any("future_unknown_type_xyz" in r.message for r in caplog.records)
    def test_unknown_content_type_increments_loss_report(self):
        p = self._get_provider()
        report = LossReport()
        p.normalize_conversation(self._make_unknown_conv(), report)
        assert report.unknown_blocks["future_unknown_type_xyz"] == 1
 class TestChatGPTHiddenContentPolicy:
    """EXPORTER_HIDDEN_CONTENT: collapse retrieval dumps and hidden context.
    file_search dumps re-inject attached-file text on every tool run (measured
    86% of content bytes on real data) and are NOT hidden-flagged — author.name
    is the discriminator. Hidden-flagged messages (Custom Instructions) are the
    secondary case.
    """
    def _get_provider(self, policy=None):
        from src.providers.chatgpt import ChatGPTProvider
        p = ChatGPTProvider.__new__(ChatGPTProvider)
        import requests
        p._session = requests.Session()
        p._org_id = None
        p._project_ids = []
        p._project_map = {}
        p._project_name_cache = {}
        if policy is not None:
            p._hidden_content = policy
        return p
    def _make_conv(self):
        return {
            "id": "test-collapse",
            "title": "T",
            "create_time": 1700000000.0,
            "update_time": 1700000001.0,
            "mapping": {
                "root": {"id": "root", "message": None, "parent": None, "children": ["dump"]},
                "dump": {
                    "id": "dump",
                    "parent": "root",
                    "children": ["answer"],
                    "message": {
                        "id": "dump",
                        "author": {"role": "tool", "name": "file_search"},
                        "content": {
                            "content_type": "multimodal_text",
                            "parts": ["Full text of an attached file " * 100],
                        },
                    },
                },
                "answer": {
                    "id": "answer",
                    "parent": "dump",
                    "children": ["uec"],
                    "message": {
                        "id": "answer",
                        "author": {"role": "assistant"},
                        "content": {"content_type": "text", "parts": ["The answer."]},
                    },
                },
                "uec": {
                    "id": "uec",
                    "parent": "answer",
                    "children": [],
                    "message": {
                        "id": "uec",
                        "author": {"role": "user"},
                        "content": {
                            "content_type": "user_editable_context",
                            "user_profile": "Preferred name: Jesse",
                            "user_instructions": "Be concise.",
                        },
                        "metadata": {"is_visually_hidden_from_conversation": True},
                    },
                },
            },
        }
    def test_default_policy_is_placeholder(self, monkeypatch):
        from src.blocks import BLOCK_TYPE_COLLAPSED
        monkeypatch.delenv("EXPORTER_HIDDEN_CONTENT", raising=False)
        # No _hidden_content attribute → env fallback → placeholder default.
        result = self._get_provider().normalize_conversation(self._make_conv())
        collapsed = [
            b for m in result["messages"] for b in m["blocks"]
            if b.get("type") == BLOCK_TYPE_COLLAPSED
        ]
        assert len(collapsed) == 2
    def test_placeholder_collapses_file_search_dump(self):
        from src.blocks import BLOCK_TYPE_COLLAPSED, COLLAPSED_KIND_TOOL_DUMP
        result = self._get_provider("placeholder").normalize_conversation(self._make_conv())
        tool_msgs = [m for m in result["messages"] if m["role"] == "tool"]
        assert len(tool_msgs) == 1
        (block,) = tool_msgs[0]["blocks"]
        assert block["type"] == BLOCK_TYPE_COLLAPSED
        assert block["kind"] == COLLAPSED_KIND_TOOL_DUMP
        assert block["origin"] == "file_search"
        assert block["size_bytes"] > 1000
    def test_placeholder_collapses_hidden_flagged_message(self):
        from src.blocks import BLOCK_TYPE_COLLAPSED, COLLAPSED_KIND_HIDDEN_CONTEXT
        result = self._get_provider("placeholder").normalize_conversation(self._make_conv())
        user_msgs = [m for m in result["messages"] if m["role"] == "user"]
        assert len(user_msgs) == 1
        (block,) = user_msgs[0]["blocks"]
        assert block["type"] == BLOCK_TYPE_COLLAPSED
        assert block["kind"] == COLLAPSED_KIND_HIDDEN_CONTEXT
        assert block["origin"] == "user_editable_context"
    def test_placeholder_keeps_normal_messages(self):
        result = self._get_provider("placeholder").normalize_conversation(self._make_conv())
        assistant_msgs = [m for m in result["messages"] if m["role"] == "assistant"]
        assert assistant_msgs[0]["blocks"][0]["type"] == BLOCK_TYPE_TEXT
        assert assistant_msgs[0]["blocks"][0]["text"] == "The answer."
    def test_non_retrieval_tool_authors_not_collapsed(self):
        """web.run / browser / python tool messages are real content — keep."""
        from src.blocks import BLOCK_TYPE_COLLAPSED
        conv = self._make_conv()
        conv["mapping"]["dump"]["message"]["author"]["name"] = "web.run"
        conv["mapping"]["dump"]["message"]["content"] = {
            "content_type": "text",
            "parts": ["Search results summary."],
        }
        result = self._get_provider("placeholder").normalize_conversation(conv)
        tool_msgs = [m for m in result["messages"] if m["role"] == "tool"]
        assert tool_msgs[0]["blocks"][0]["type"] != BLOCK_TYPE_COLLAPSED
    def test_omit_drops_messages_entirely(self):
        result = self._get_provider("omit").normalize_conversation(self._make_conv())
        assert [m["role"] for m in result["messages"]] == ["assistant"]
    def test_full_preserves_v040_behavior(self):
        result = self._get_provider("full").normalize_conversation(self._make_conv())
        tool_msgs = [m for m in result["messages"] if m["role"] == "tool"]
        assert tool_msgs[0]["blocks"][0]["type"] == BLOCK_TYPE_TEXT
        user_msgs = [m for m in result["messages"] if m["role"] == "user"]
        assert user_msgs[0]["blocks"][0]["type"] == BLOCK_TYPE_HIDDEN_CONTEXT_MARKER
    def test_collapsed_tallied_in_loss_report(self):
        report = LossReport()
        self._get_provider("placeholder").normalize_conversation(self._make_conv(), report)
        assert report.collapsed["file_search"] == 1
        assert report.collapsed["user_editable_context"] == 1
        assert report.collapsed_bytes > 1000
        summary = report.format_summary()
        assert "collapsed by policy:  2" in summary
        assert "EXPORTER_HIDDEN_CONTENT=full to keep" in summary
    def test_collapsed_block_renders_one_line_with_size(self):
        from src.blocks import BLOCK_TYPE_COLLAPSED
        result = self._get_provider("placeholder").normalize_conversation(self._make_conv())
        tool_msgs = [m for m in result["messages"] if m["role"] == "tool"]
        rendered = render_blocks_to_markdown(tool_msgs[0]["blocks"])
        assert rendered.startswith("> 🔧 **Tool output** — `file_search` (")
        assert "KB" in rendered
        assert "omitted (EXPORTER_HIDDEN_CONTENT=full to keep)" in rendered
        assert "\n" not in rendered
    def test_invalid_env_value_falls_back_to_placeholder(self, monkeypatch, caplog):
        from src.providers.chatgpt import resolve_hidden_content_policy
        monkeypatch.setenv("EXPORTER_HIDDEN_CONTENT", "bogus")
        with caplog.at_level(logging.WARNING):
            assert resolve_hidden_content_policy() == "placeholder"
        assert any("EXPORTER_HIDDEN_CONTENT" in r.message for r in caplog.records)
 class TestRequestPacing:
    """REQUEST_DELAY politeness pacing between consecutive API requests."""
    def _get_provider(self, delay):
        from src.providers.chatgpt import ChatGPTProvider
        p = ChatGPTProvider.__new__(ChatGPTProvider)
        p._request_delay = delay
        p._last_request_at = None
        return p
    def test_first_request_not_delayed(self, monkeypatch):
        sleeps = []
        monkeypatch.setattr("src.providers.base.time.sleep", lambda s: sleeps.append(s))
        p = self._get_provider(2.0)
        p._pace()
        assert sleeps == []
    def test_consecutive_requests_paced_with_jitter_window(self, monkeypatch):
        sleeps = []
        monkeypatch.setattr("src.providers.base.time.sleep", lambda s: sleeps.append(s))
        monkeypatch.setattr("src.providers.base.random.uniform", lambda a, b: 1.0)
        clock = {"now": 100.0}
        monkeypatch.setattr("src.providers.base.time.monotonic", lambda: clock["now"])
        p = self._get_provider(2.0)
        p._pace()
        clock["now"] = 100.5  # only 0.5s elapsed since last request
        p._pace()
        assert sleeps == [pytest.approx(1.5)]
    def test_no_sleep_when_enough_time_elapsed(self, monkeypatch):
        sleeps = []
        monkeypatch.setattr("src.providers.base.time.sleep", lambda s: sleeps.append(s))
        monkeypatch.setattr("src.providers.base.random.uniform", lambda a, b: 1.0)
        clock = {"now": 100.0}
        monkeypatch.setattr("src.providers.base.time.monotonic", lambda: clock["now"])
        p = self._get_provider(1.0)
        p._pace()
        clock["now"] = 105.0
        p._pace()
        assert sleeps == []
    def test_zero_delay_disables_pacing(self, monkeypatch):
        sleeps = []
        monkeypatch.setattr("src.providers.base.time.sleep", lambda s: sleeps.append(s))
        p = self._get_provider(0)
        p._pace()
        p._pace()
        assert sleeps == []
    def test_pace_noop_without_init_attrs(self):
        """Providers built via __new__ in tests have no pacing attrs — must not crash."""
        from src.providers.chatgpt import ChatGPTProvider
        p = ChatGPTProvider.__new__(ChatGPTProvider)
        p._pace()  # no exception
    def test_resolve_request_delay_default_and_overrides(self, monkeypatch, caplog):
        from src.providers.base import resolve_request_delay, DEFAULT_REQUEST_DELAY
        monkeypatch.delenv("REQUEST_DELAY", raising=False)
        assert resolve_request_delay() == DEFAULT_REQUEST_DELAY
        monkeypatch.setenv("REQUEST_DELAY", "2.5")
        assert resolve_request_delay() == 2.5
        monkeypatch.setenv("REQUEST_DELAY", "0")
        assert resolve_request_delay() == 0.0
        monkeypatch.setenv("REQUEST_DELAY", "abc")
        with caplog.at_level(logging.WARNING):
            assert resolve_request_delay() == DEFAULT_REQUEST_DELAY
        assert any("REQUEST_DELAY" in r.message for r in caplog.records)
 # ---------------------------------------------------------------------------
 # Claude
 # ---------------------------------------------------------------------------
 class TestClaudeNormalization:
-    """Test ClaudeProvider.normalize_conversation() using fixture data."""
+    """Claude normalize_conversation block-extraction behavior."""
    def _get_provider(self):
        from src.providers.claude import ClaudeProvider
@@ -105,55 +548,138 @@ class TestClaudeNormalization:
        assert result["provider"] == "claude"
        assert result["project"] == "StarTOS Packaging"
        assert result["created_at"] == "2024-06-10T14:32:00.000Z"
        assert isinstance(result["messages"], list)
    def test_normalizes_without_project(self):
        raw = json.loads((FIXTURES / "claude_no_project.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        assert result["project"] is None
        assert result["id"] == "claude-conv-002"
-    def test_string_content_extracted(self):
+    def test_string_content_emits_text_block(self):
-        raw = json.loads((FIXTURES / "claude_no_project.json").read_text())
+        raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
-        assert any("Docker" in m["content"] for m in result["messages"])
+        thanks_msgs = [
            m for m in result["messages"]
            if any(
                "thank you" in (b.get("text") or "").lower()
                for b in (m.get("blocks") or [])
            )
        ]
        assert thanks_msgs
-    def test_list_content_extracted(self):
+    def test_list_content_emits_blocks_in_order(self):
        raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        assistant_msgs = [m for m in result["messages"] if m["role"] == "assistant"]
-        assert any("manifest" in m["content"].lower() for m in assistant_msgs)
+        # msg-002 has text + tool_use, in that order.
        assert assistant_msgs
        types = _block_types(assistant_msgs[0])
        assert BLOCK_TYPE_TEXT in types
        assert BLOCK_TYPE_TOOL_USE in types
        # Order preserved
        assert types.index(BLOCK_TYPE_TEXT) < types.index(BLOCK_TYPE_TOOL_USE)
-    def test_non_text_blocks_skipped_with_warning(self, caplog):
+    def test_tool_use_block_fields(self):
        import logging
        raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
        p = self._get_provider()
-        with caplog.at_level(logging.WARNING):
+        result = p.normalize_conversation(raw)
-            result = p.normalize_conversation(raw)
+
-        # The fixture has a tool_use block — should warn
+        assistant_msgs = [m for m in result["messages"] if m["role"] == "assistant"]
        tool_block = _first_block(assistant_msgs[0], BLOCK_TYPE_TOOL_USE)
        assert tool_block["name"] == "search"
        assert tool_block["input"] == {"query": "startOS docs"}
        assert tool_block["tool_id"] == "tool-001"
    def test_image_block_emits_image_placeholder(self):
        raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        msg004 = [
            m for m in result["messages"]
            if any(b.get("type") == BLOCK_TYPE_IMAGE_PLACEHOLDER for b in (m.get("blocks") or []))
        ]
        assert msg004
        img = _first_block(msg004[0], BLOCK_TYPE_IMAGE_PLACEHOLDER)
        assert img["ref"] == "claude-image-uuid-1"
    def test_unknown_block_type_records_loss(self):
        from src.blocks import BLOCK_TYPE_UNKNOWN as _UNK
        raw = {
            "uuid": "test-unknown",
            "name": "T",
            "chat_messages": [
                {
                    "uuid": "m1",
                    "sender": "human",
                    "content": [{"type": "future_block_xyz", "data": "..."}],
                }
            ],
        }
        p = self._get_provider()
        report = LossReport()
        result = p.normalize_conversation(raw, report)
        assert any(
-            "tool_use" in r.message or "rich content" in r.message
+            b.get("type") == _UNK
-            for r in caplog.records
+            for m in result["messages"]
            for b in (m.get("blocks") or [])
        )
        assert report.unknown_blocks["future_block_xyz"] == 1
-    def test_message_count_matches(self):
+    def test_thinking_block(self):
-        raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
+        raw = {
            "uuid": "thinking-test",
            "name": "T",
            "chat_messages": [
                {
                    "uuid": "m1",
                    "sender": "assistant",
                    "content": [
                        {"type": "thinking", "thinking": "Let me reason about this."},
                        {"type": "text", "text": "Here's the answer."},
                    ],
                }
            ],
        }
        p = self._get_provider()
        result = p.normalize_conversation(raw)
-        assert result["message_count"] == len(result["messages"])
+        types = _block_types(result["messages"][0])
        assert BLOCK_TYPE_THINKING in types
        assert BLOCK_TYPE_TEXT in types
-    def test_roles_normalized(self):
+    def test_tool_result_with_nested_text_blocks(self):
-        raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
+        raw = {
            "uuid": "tool-result-test",
            "name": "T",
            "chat_messages": [
                {
                    "uuid": "m1",
                    "sender": "assistant",
                    "content": [
                        {
                            "type": "tool_result",
                            "tool_use_id": "tool-001",
                            "content": [
                                {"type": "text", "text": "search hit 1"},
                                {"type": "text", "text": "search hit 2"},
                            ],
                            "is_error": False,
                        }
                    ],
                }
            ],
        }
        p = self._get_provider()
        result = p.normalize_conversation(raw)
-        for msg in result["messages"]:
+        tool_result = _first_block(result["messages"][0], BLOCK_TYPE_TOOL_RESULT)
-            assert msg["role"] in ("user", "assistant", "system")
+        assert tool_result is not None
        assert "search hit 1" in tool_result["output"]
        assert "search hit 2" in tool_result["output"]
        assert tool_result["is_error"] is False
    def test_human_sender_maps_to_user(self):
        raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
@@ -162,3 +688,188 @@ class TestClaudeNormalization:
        roles = {m["role"] for m in result["messages"]}
        assert "user" in roles
        assert "human" not in roles
    def test_loss_report_messages_recorded(self):
        raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
        p = self._get_provider()
        report = LossReport()
        result = p.normalize_conversation(raw, report)
        assert report.messages_rendered == len(result["messages"])
 # ---------------------------------------------------------------------------
 # v0.4.1 — execution_output, system_error, tether_browsing_display, conv_id
 # ---------------------------------------------------------------------------
 class TestChatGPTToolOutputs:
    """v0.4.1 ChatGPT tool-role content_types map onto tool_result blocks."""
    def _get_provider(self):
        from src.providers.chatgpt import ChatGPTProvider
        p = ChatGPTProvider.__new__(ChatGPTProvider)
        import requests
        p._session = requests.Session()
        p._org_id = None
        p._project_ids = []
        p._project_map = {}
        p._project_name_cache = {}
        return p
    def test_execution_output_emits_tool_result_with_metadata(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        exec_msgs = [
            m for m in result["messages"]
            if any(
                b.get("type") == BLOCK_TYPE_TOOL_RESULT
                and b.get("tool_name") == "container.exec"
                for b in (m.get("blocks") or [])
            )
        ]
        assert exec_msgs, "expected execution_output to render as tool_result"
        block = next(
            b for b in exec_msgs[0]["blocks"] if b.get("type") == BLOCK_TYPE_TOOL_RESULT
        )
        assert block["output"].startswith("Hello from container.exec")
        assert block["is_error"] is False
        assert block["summary"] == "Reading skill documentation"
    def test_execution_output_message_role_is_tool(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        tool_msgs = [m for m in result["messages"] if m["role"] == "tool"]
        assert tool_msgs, "tool-role messages must pass through (filter lifted in v0.4.0)"
    def test_empty_execution_output_skipped(self, caplog):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        with caplog.at_level(logging.DEBUG, logger="src.providers.chatgpt"):
            result = p.normalize_conversation(raw)
        # The empty execution_output (author.name="python") must NOT appear.
        python_msgs = [
            m for m in result["messages"]
            if any(
                b.get("type") == BLOCK_TYPE_TOOL_RESULT and b.get("tool_name") == "python"
                for b in (m.get("blocks") or [])
            )
        ]
        assert not python_msgs, "empty execution_output should be skipped"
        assert any("Skipping empty execution_output" in r.message for r in caplog.records)
    def test_system_error_emits_error_tool_result(self):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        web_err = [
            m for m in result["messages"]
            if any(
                b.get("type") == BLOCK_TYPE_TOOL_RESULT
                and b.get("tool_name") == "web"
                and b.get("is_error") is True
                for b in (m.get("blocks") or [])
            )
        ]
        assert web_err, "system_error should render as tool_result with is_error=True"
        block = next(b for b in web_err[0]["blocks"] if b.get("tool_name") == "web")
        assert "503" in block["output"]
    def test_tether_browsing_display_spinner_skipped(self, caplog):
        raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
        p = self._get_provider()
        with caplog.at_level(logging.DEBUG, logger="src.providers.chatgpt"):
            result = p.normalize_conversation(raw)
        spinner_msgs = [
            m for m in result["messages"]
            if any(
                b.get("type") == BLOCK_TYPE_TOOL_RESULT and b.get("tool_name") == "file_search"
                for b in (m.get("blocks") or [])
            )
        ]
        assert not spinner_msgs, "spinner tether_browsing_display should be skipped"
        assert any("tether_browsing_display spinner" in r.message for r in caplog.records)
    def test_tether_browsing_display_populated_renders_defensively(self):
        """Defensive case (never observed in real data) — populated browse renders."""
        conv = {
            "id": "test-tether",
            "title": "T",
            "create_time": 1700000000.0,
            "update_time": 1700000001.0,
            "mapping": {
                "root": {"id": "root", "message": None, "parent": None, "children": ["m1"]},
                "m1": {
                    "id": "m1",
                    "parent": "root",
                    "children": [],
                    "message": {
                        "id": "m1",
                        "author": {"role": "tool", "name": "browser"},
                        "content": {
                            "content_type": "tether_browsing_display",
                            "result": "Found 3 results about kubernetes ingress.",
                            "summary": "ingress search",
                            "assets": None,
                            "tether_id": None,
                        },
                    },
                },
            },
        }
        p = self._get_provider()
        result = p.normalize_conversation(conv)
        assert any(
            b.get("type") == BLOCK_TYPE_TOOL_RESULT and b.get("tool_name") == "browser"
            for m in result["messages"]
            for b in (m.get("blocks") or [])
        )
 class TestChatGPTConvIdFallback:
    """v0.4.1: live ChatGPT detail responses use conversation_id, not id."""
    def _get_provider(self):
        from src.providers.chatgpt import ChatGPTProvider
        p = ChatGPTProvider.__new__(ChatGPTProvider)
        import requests
        p._session = requests.Session()
        p._org_id = None
        p._project_ids = []
        p._project_map = {}
        p._project_name_cache = {}
        return p
    def test_falls_back_to_conversation_id(self):
        raw = {
            "conversation_id": "live-chatgpt-uuid",
            "title": "T",
            "create_time": 1700000000.0,
            "update_time": 1700000001.0,
            "mapping": {
                "root": {"id": "root", "message": None, "parent": None, "children": []},
            },
        }
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        assert result["id"] == "live-chatgpt-uuid"
    def test_id_takes_precedence_when_both_present(self):
        raw = {
            "id": "from-id",
            "conversation_id": "from-conversation-id",
            "title": "T",
            "create_time": 1700000000.0,
            "update_time": 1700000001.0,
            "mapping": {
                "root": {"id": "root", "message": None, "parent": None, "children": []},
            },
        }
        p = self._get_provider()
        result = p.normalize_conversation(raw)
        assert result["id"] == "from-id"
@@ -0,0 +1,147 @@
 """Tests for src/utils.py — filename generation, path building, redaction."""
 from pathlib import Path
 import pytest
 from src.utils import (
    build_export_path,
    format_token_status,
    generate_filename,
    redact_secrets,
 )
 class TestGenerateFilename:
    def test_basic_format(self):
        name = generate_filename("Hello World", "abc12345def", "2024-06-10T14:00:00Z")
        assert name == "2024-06-10_hello-world_abc12345.md"
    def test_special_chars_slugified(self):
        # T-36: titles with punctuation must produce safe, OS-compatible filenames
        name = generate_filename("What's this?! A test.", "abc12345", "2024-06-01T00:00:00Z")
        assert "?" not in name
        assert "!" not in name
        assert "'" not in name
        assert " " not in name
        assert name.startswith("2024-06-01_")
        assert name.endswith("_abc12345.md")
    def test_unicode_chars_handled(self):
        name = generate_filename("Héllo Wörld", "abc12345", "2024-06-01T00:00:00Z")
        assert " " not in name
        assert name.endswith("_abc12345.md")
    def test_empty_title_becomes_untitled(self):
        name = generate_filename("", "abc12345", "2024-06-01T00:00:00Z")
        assert "untitled" in name
    def test_id_truncated_to_8_chars(self):
        name = generate_filename("Test", "abcdefghijklmnop", "2024-06-01T00:00:00Z")
        assert name.endswith("_abcdefgh.md")
    def test_long_title_truncated(self):
        long_title = "a" * 200
        name = generate_filename(long_title, "abc12345", "2024-06-01T00:00:00Z")
        # Slug is capped at 60 chars by max_length
        slug_part = name.split("_")[1]
        assert len(slug_part) <= 60
    def test_date_comes_from_created_at(self):
        name = generate_filename("Test", "abc12345", "2023-11-25T00:00:00Z")
        assert name.startswith("2023-11-25_")
 class TestBuildExportPath:
    def test_default_structure_provider_project_year(self):
        path = build_export_path(
            Path("/exports"), "claude", "my-project", "2024-06-01T00:00:00Z", "file.md"
        )
        assert str(path) == "/exports/claude/my-project.2024/file.md"
    def test_no_project_uses_no_project_slug(self):
        path = build_export_path(
            Path("/exports"), "chatgpt", None, "2024-06-01T00:00:00Z", "file.md"
        )
        assert "no-project.2024" in str(path)
    def test_provider_project_structure_omits_year(self):
        path = build_export_path(
            Path("/exports"), "claude", "proj", "2024-06-01T00:00:00Z", "file.md",
            structure="provider/project",
        )
        assert "2024" not in str(path)
        assert "proj" in str(path)
    def test_provider_year_structure_omits_project(self):
        path = build_export_path(
            Path("/exports"), "claude", "proj", "2024-06-01T00:00:00Z", "file.md",
            structure="provider/year",
        )
        assert "proj" not in str(path)
        assert "2024" in str(path)
    def test_project_name_with_spaces_is_slugified(self):
        path = build_export_path(
            Path("/exports"), "claude", "My Project Name!", "2024-06-01T00:00:00Z", "file.md"
        )
        assert " " not in str(path)
        assert "!" not in str(path)
 class TestRedactSecrets:
    def test_token_value_redacted(self):
        data = {"token": "supersecret"}
        result = redact_secrets(data)
        assert result["token"] == "[REDACTED]"
    def test_session_key_redacted(self):
        result = redact_secrets({"sessionKey": "abc123"})
        assert result["sessionKey"] == "[REDACTED]"
    def test_non_sensitive_key_unchanged(self):
        result = redact_secrets({"title": "My Chat", "id": "abc123"})
        assert result["title"] == "My Chat"
        assert result["id"] == "abc123"
    def test_nested_dict_redacted(self):
        data = {"user": {"token": "secret", "name": "Alice"}}
        result = redact_secrets(data)
        assert result["user"]["token"] == "[REDACTED]"
        assert result["user"]["name"] == "Alice"
    def test_list_of_dicts(self):
        data = [{"password": "p@ss"}, {"title": "chat"}]
        result = redact_secrets(data)
        assert result[0]["password"] == "[REDACTED]"
        assert result[1]["title"] == "chat"
 class TestFormatTokenStatus:
    def test_none_token_returns_not_set(self):
        assert format_token_status(None) == "[NOT SET]"
    def test_empty_token_returns_not_set(self):
        assert format_token_status("") == "[NOT SET]"
    def test_set_token_no_expiry(self):
        assert format_token_status("sometoken") == "[SET]"
    def test_expired_token(self):
        from datetime import datetime, timezone, timedelta
        expiry = datetime.now(tz=timezone.utc) - timedelta(days=1)
        result = format_token_status("tok", expiry)
        assert "EXPIRED" in result
    def test_expiring_today_shows_hours(self):
        from datetime import datetime, timezone, timedelta
        expiry = datetime.now(tz=timezone.utc) + timedelta(hours=3)
        result = format_token_status("tok", expiry)
        assert "expires in" in result
        assert "h" in result
    def test_expiring_in_days(self):
        from datetime import datetime, timezone, timedelta
        expiry = datetime.now(tz=timezone.utc) + timedelta(days=10, hours=12)
        result = format_token_status("tok", expiry)
        assert "10 days" in result
Author	SHA1	Message	Date
JesseMarkowitz	9e1a8ab7cb	feat: v0.6.0 — collapse policy, session limiter, Claude Code provider, prune, browser auth, media downloads	2026-06-12 18:26:14 -04:00
JesseMarkowitzandClaude Sonnet 4.6	557994f7d9	fix: persist created_at in cache so Joplin note titles get date prefix mark_exported() was discarding created_at from the metadata dict because it wasn't in the hardcoded stored-key list, so the joplin sync always saw an empty date and omitted the prefix. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-05 11:36:21 -04:00
JesseMarkowitzandClaude Sonnet 4.6	e9b2e42893	feat: v0.5.0 — nested Joplin notebooks, date-prefixed note titles, flat year folders Joplin notebooks now use a two-level hierarchy: AI-ChatGPT / <project> and AI-Claude / <project> instead of a single flat title. Note titles are prefixed with the conversation created_at date (YYYY-MM-DD). Export folders collapse provider/project/year into a single provider/project.year directory. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-05 11:05:39 -04:00
JesseMarkowitzandClaude Opus 4.7	68e8d532be	feat: v0.4.1 — ChatGPT tool-output content types and conv_id fix First real-data export against v0.4.0 surfaced 66 unknown blocks across three content types — captured live and added. Added: - execution_output (Code Interpreter / container.exec / python tool output) → tool_result block. output=content.text, tool_name=author.name, is_error=metadata.aggregate_result.status, summary=metadata.reasoning_title - system_error → error tool_result with tool_name=author.name - tether_browsing_display: spinner placeholders (empty result+summary) skip silently with DEBUG log; defensive populated-case branch maps to tool_result (untested in real data) - tool_result block schema: optional `summary` field rendered as italic line between header and fence - tool_result rendering: tool_name appears in header when present (e.g. `📤 Result: container.exec`); existing tool_name=None calls unchanged - _ROLE_LABELS["tool"] = ("🔧 Tool", "tool") Fixed: - chatgpt.normalize_conversation reads `conversation_id` as fallback for `id`. Live API uses conversation_id; fixtures use id. Pre-fix: empty id in YAML frontmatter and missing context in WARNING logs. Tests: 11 new (192 total, 0 failures). Fixture extended with 4 tool-output cases (execution_output success, empty execution_output that should skip, system_error, tether_browsing_display spinner). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-05 09:25:55 -04:00
JesseMarkowitzandClaude Opus 4.7	473d02f71a	feat: v0.4.0 — rich content support with typed blocks and loss visibility Extracts per-message content into a typed `blocks` list (text, code, thinking, tool_use, tool_result, image_placeholder, file_placeholder, unknown) and renders them at exporter write time. Voice transcripts, Custom Instructions, and image references now appear in exports instead of being silently dropped. Foundation: - src/blocks.py: pure block constructors, _safe_fence (fence-corruption defense, verified live in Joplin), _blockquote_prefix, render - src/loss_report.py: per-run tally surfaced as INFO summary at end of export so silently-dropped data becomes visible Providers: - ChatGPT: dispatch on content_type produces typed blocks; voice shapes (audio_transcription, audio_asset_pointer, real_time_user_audio_video_ asset_pointer) locked from live DevTools capture; Custom Instructions bug fix (parts-vs-direct-fields); role filter lifted; hidden-context marker driven by is_visually_hidden_from_conversation flag - Claude: defensive dispatch for text/thinking/tool_use/tool_result/image with recursive nested-block flattening; untested against real rich- content data — fix-forward in v0.4.1 Exporter: - Markdown renders from blocks at write time via render_blocks_to_markdown; backward-compat fallback to content for any pre-v0.4.0 cached data Tests: - 27 new tests across providers, exporters, CLI; fixtures rebuilt with real-shape ChatGPT voice + Custom Instructions cases - 181/181 pass Behavior changes (intentional): - JSON output omits content; consumers should read blocks - Per-conversation message counts increase (Custom Instructions, image- only, tool-only messages now appear) - Existing exports not auto-re-rendered; users wanting fresh output run cache --clear then export Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-04 23:17:18 -04:00
Jesse.MarkowitzandClaude Sonnet 4.6	4798edcea7	docs: update README for chunked ChatGPT session cookies Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-19 22:32:01 -04:00
Jesse.MarkowitzandClaude Sonnet 4.6	19bfdaecbe	fix: v0.2.1 — chunked ChatGPT cookies and Claude project path - Support __Secure-next-auth.session-token.0/.1 split cookies; ChatGPT now issues tokens that exceed the 4KB per-cookie limit and must be sent as two named chunks or the auth endpoint returns no accessToken. Add CHATGPT_SESSION_TOKEN_1 env var; update auth wizard instructions. - Fix Claude conversations exported to wrong directory when project name is present in the listing but absent from the detail endpoint response. Explicitly propagate "project" alongside _-prefixed annotation keys. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-13 22:32:14 -04:00
Jesse.MarkowitzandClaude Opus 4.6	4ccd918eb1	fix: list command shows Claude titles and fits 80-col terminals Claude's list endpoint returns conversations with a `name` field rather than `title`, so every Claude row was falling through to "Untitled". Also set no_wrap + ellipsis overflow and tune column widths so the table renders one row per conversation in Windows Command Prompt (80 cols). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-04-08 14:49:58 -04:00
Jesse.Markowitz	a869e8c7ba	fix for project files written to wrong directory	2026-03-30 15:25:18 -04:00
Jesse.Markowitz	340293ab94	fix for project files not extracted	2026-03-30 13:22:05 -04:00
Jesse.Markowitz	050cd49124	updated to run on Windows and add est capabilities	2026-03-30 11:08:05 -04:00
Jesse.MarkowitzandClaude Sonnet 4.6	304cf4fde4	feat: v0.2.0 — Joplin import, ChatGPT Projects, --project filter Core features: - Add `joplin` command: syncs exported Markdown to Joplin via local REST API - Notebooks auto-created per provider+project (e.g. "ChatGPT - My Project") - Idempotent: notes updated (not duplicated) on re-run; note ID tracked in manifest - Add `--project` filter to `export` and `list` commands (substring or 'none') - Add ChatGPT Projects support via CHATGPT_PROJECT_IDS env var Config: - Add JOPLIN_API_TOKEN, JOPLIN_API_URL, JOPLIN_REQUEST_TIMEOUT - Version now read from importlib.metadata (single source of truth: pyproject.toml) - Bump version to 0.2.0 Quality: - Explicit Timeout handling in JoplinClient with actionable error messages - token validation (validate_token) separate from connectivity (ping) - Remove debug_auth.py, debug_claude.py, and untracked .har file - Add *.har to .gitignore (may contain auth cookies/session tokens) - Update README, CHANGELOG, FUTURE.md to reflect v0.2.0 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-01 06:04:03 -05:00
Jesse.MarkowitzandClaude Sonnet 4.6	23d7c17255	fix: use curl_cffi Chrome TLS impersonation for Claude provider claude.ai has the same Cloudflare TLS fingerprinting protection as chatgpt.com. Apply the same fix: curl_cffi impersonate=chrome120, remove base class User-Agent to avoid JA3/UA mismatch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-02-28 05:34:35 -05:00
Jesse.MarkowitzandClaude Sonnet 4.6	51c806c2c6	fix: set Claude sessionKey in cookie jar instead of raw header Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-02-28 05:33:04 -05:00
Jesse.MarkowitzandClaude Sonnet 4.6	f500c038cb	fix: remove User-Agent override to prevent TLS/UA fingerprint mismatch curl_cffi sets a User-Agent consistent with its JA3 TLS fingerprint. BaseProvider's custom UA (Chrome/121) conflicted with the chrome120 TLS fingerprint, causing Cloudflare to flag the request as a bot. Removing the UA from session headers lets curl_cffi manage its own. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-02-28 05:27:53 -05:00
Jesse.MarkowitzandClaude Sonnet 4.6	bb92ed2731	fix: update debug_auth.py to check accessToken presence by key Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-02-28 05:24:18 -05:00
Jesse.MarkowitzandClaude Sonnet 4.6	5c6dcafa34	fix: use curl_cffi Chrome TLS impersonation to bypass Cloudflare chatgpt.com uses Cloudflare's TLS fingerprinting (JA3/JA4) which blocks Python requests regardless of cookies. curl_cffi impersonates Chrome's exact TLS handshake, making requests indistinguishable from a real browser at the transport layer. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-02-28 05:20:52 -05:00
Jesse.MarkowitzandClaude Sonnet 4.6	d236fdb21a	fix: set session cookie in cookie jar instead of manual header Using self._session.cookies.set() ensures the cookie is sent correctly by the requests session on all calls, including /api/auth/session. Also add sec-fetch-* headers required by chatgpt.com. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-02-27 23:39:38 -05:00
Jesse.MarkowitzandClaude Sonnet 4.6	6a33de682a	fix: implement two-step ChatGPT auth (session cookie → access token) The __Secure-next-auth.session-token cannot be used directly as a Bearer token. It must first be exchanged via GET /api/auth/session (with the token sent as a Cookie) to obtain a short-lived accessToken. This accessToken is then used as the Authorization: Bearer header for all backend-api calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-02-27 23:37:42 -05:00
Jesse.MarkowitzandClaude Sonnet 4.6	b41634d892	fix: load .env in doctor command; handle JWE tokens gracefully Doctor was reading env vars before loading .env, so tokens set in .env were invisible. ChatGPT now uses JWE (encrypted JWT) tokens which PyJWT cannot decode without the server key — treat decode failure as "token set, expiry unknown" rather than a FAIL. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-02-27 23:34:54 -05:00
Jesse.MarkowitzandClaude Sonnet 4.6	8e9ca36b57	docs: add README Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-02-27 23:14:37 -05:00